'mcpbench' is a benchmark test that measures the performance of GPT-5.6 and Claude Opus from the perspective of 'Can an MCP server be built?'

A benchmark test called 'mcpbench' has emerged that measures the performance of various AI models from the perspective of 'Can an MCP client or MCP server be built according to the specifications?'. The author is

Happy MCP special day!!!

2026-07-28 brings stateless by default, multi-round trip elicitations, cimd auth by default and a bunch more cool stuff.

I made a benchmark to track how much of the spec is in the weights :) https://t.co/ltAPJ4Y9GK

— Matt Carey (@mattzcarey) July 28, 2026

MCP stands for 'Model Context Protocol,' and it's a mechanism that acts as a bridge to give AI systems the ability to access external services and data. MCP is a standard developed by Anthropic in November 2024, and its specifications are repeatedly updated to keep up with changes in the AI environment. Version 2026-07-28 was released on July 28, 2026. A major feature of version 2026-07-28 is the elimination of the session system and the transition to a stateless mechanism. The details of the update are summarized in the following article.

The common standard 'MCP' for connecting AI and external tools receives its biggest update ever: What has changed with the elimination of sessions? - GIGAZINE

MCP is a system consisting of an 'MCP server' that actually accesses external resources and executes processing, and an 'MCP client' that receives instructions from an AI agent and sends requests to the MCP server. mcpbench has various AI models perform the 'construction of an MCP client and MCP server' and evaluates the performance of the AI model based on whether or not it is successful.

The following are the test results for creating an MCP client version 2025-11-25 using 'GPT-5.6 Sol', 'GPT-5.6 Luna', 'GPT-5.6 Terra', 'Claude Opus 4.8', 'Claude Sonnet 5', and 'Kimi K2.7 Code'. GPT-5.6 Sol was rated as having the best performance in both 'NO DOCS' (where only client creation instructions are entered) and 'FULL DOCS' (where documentation is entered).

In tests requiring the creation of an MCP client compliant with version 2026-07-28, the score for NO DOCS decreased significantly. However, with FULL DOCS, the score was equivalent to that of version 2025-11-25, indicating that the model lacks knowledge of MCP version 2026-07-28.

In tests to create an MCP server compliant with version 2025-11-25, GET-5.6 Luna achieved the highest score with NO DOCS. With FULL DOCS, Claude Opus 4.8 and Kimi K2.7 Code achieved 100%.

For MCP servers compliant with version 2026-07-28, all AI models achieved a 0% completion rate with NO DOCS. This is thought to be a result of the AI models being unable to adapt to the fact that initialization via 'initialize/initialized' is no longer required in version 2026-07-28. However, all AI models achieved high completion rates when documentation was provided.

The source code for mcpbench is available at the following link.

GitHub - mattzcarey/mcpbench · GitHub
https://github.com/mattzcarey/mcpbench