On this page
Test an MCP server before release with MCP Inspector, contract tests on your tool definitions and a scripted set of realistic tasks, then compare each new version against the previous one in production. Evals tell you the server can do the jobs you planned for; production analytics tell you whether real users, on real AI clients, still can.
A release can pass every check you wrote and still make things worse: a new description can pull a model toward the wrong tool, and a new required field can break an older client. The answer is a loop: test before launch, measure after, and feed what breaks back into the tests.
What should I test before releasing an MCP server?
Test that the server speaks the protocol, that tool definitions did not change by accident, and that a model picks the right tool and completes realistic tasks.
| Check | What it catches | How to run it |
|---|---|---|
| Inspector smoke test | Server does not start, tools missing, auth broken, tool returns an error | MCP Inspector web UI by hand, CLI in CI |
| Contract test | Unreviewed changes to names, schemas, annotations or ordering | Snapshot of tools/list, diffed on every build |
| Task evals | The model cannot finish realistic jobs with your tools | Scripted tasks with verifiers, run against a model |
| Tool selection evals | The model calls the wrong tool or passes bad arguments | Prompts with a known correct tool and argument shape |
Run the first two on every commit, and the evals on every tool change and before each release.
How do I use MCP Inspector to test a server?
MCP Inspector ↗ is the reference tool for testing and debugging MCP servers. One package, @modelcontextprotocol/inspector, provides three clients: a web UI (the default), a scriptable CLI with --cli and a terminal UI with --tui. It connects to stdio servers or to remote servers over HTTP, and it handles both the 2026-07-28 protocol and earlier revisions.
Use the web UI while you build to call tools with your own arguments and read the JSON-RPC exchange. Use the CLI in CI, because its exit codes separate failure classes:
| Exit code | Meaning |
|---|---|
0 | Success |
3 | Server requires authentication (HTTP 401 or 403) |
4 | Server unreachable |
5 | tools/call returned isError: true, or the tool was not found |
6 | --strict found an error-severity tool schema portability problem |
A minimal CI gate lists tools with the strict schema check, then calls one read-only tool against test data:
set -euo pipefail
npx @modelcontextprotocol/inspector --cli node build/index.js \
--method tools/list --strict > /dev/null
npx @modelcontextprotocol/inspector --cli node build/index.js \
--method tools/call --tool-name find_account --tool-arg query=acme-testThe second command exits 5 if the tool reports an error. For a remote server, pass the URL with --transport http and add --stored-auth-only so CI fails fast instead of waiting for a browser sign-in.
What should contract tests check on tool schemas and annotations?
Save the tools/list output as a reviewed snapshot and fail the build when it changes without review. Treat tool definitions like a public API.
Check these fields for every tool:
- Name. A rename breaks saved prompts, client allowlists and dashboards.
- Description. Models read it to choose a tool, so any wording change should trigger the selection evals below.
- inputSchema. Flag new required fields, removed properties and narrowed types.
- outputSchema. If you declare one, the server must return structured results that conform to it. Test real responses against it.
- Annotations. The spec defines four behavior hints in
ToolAnnotations, plus an optionaltitle.
| Annotation | Meaning when true | Default |
|---|---|---|
readOnlyHint | The tool does not modify its environment | false |
destructiveHint | The tool may perform destructive updates (only meaningful when not read-only) | true |
idempotentHint | Repeating the call with the same arguments has no additional effect | false |
openWorldHint | The tool may interact with external entities | true |
The spec calls these hints, not guarantees, and tells clients not to base tool use decisions on annotations from untrusted servers. Your contract test keeps them accurate: a tool that flips readOnlyHint from true to false has changed behavior even if its schema is identical. Also assert a stable order, since the 2026-07-28 revision asks servers to return tools deterministically.
How do I build tool-call evals with realistic tasks?
Write tasks that look like real requests, run each through a model with your server attached, and check the outcome with a verifier. Twenty to fifty tasks covering your main jobs is a practical start.
Anthropic's guide to writing tools for agents ↗ recommends tasks grounded in realistic data, some needing many tool calls, each with a verifiable outcome. It also warns against verifiers so strict that they reject correct answers over formatting.
For each run, record:
- Task success, from the verifier.
- Tool calls per task, to spot redundant or looping calls.
- Tool errors, split into protocol errors and results with
isError: true. - Argument errors, which often point to an unclear parameter name or description.
- Runtime and tokens per task.
Run the set on the current release and the candidate, on every model your users are likely to bring, since tool selection varies by model.
How do I check that models pick the right tool?
Write prompts with a known correct tool, including near misses between similar tools, and score the first tool the model chooses. If you have search_accounts and get_account, include prompts where only one is correct.
When selection fails, fix the definition first. Say when to use the tool and when not to, name parameters unambiguously (account_id, not id), and group related tools under a shared prefix when the list is long. Re-run the full set after any description change, because improving one tool can pull traffic from its neighbor.
What can evals not catch?
Evals cover tasks you thought of, run by models and clients you chose, against data you prepared. Production adds what you cannot fully simulate:
- Real users, who phrase requests differently, bring messy data and stop halfway through jobs.
- Real clients, each of which wraps your tools in its own prompt, handles errors its own way and updates on its own schedule.
- New tasks you never planned for, which have no eval yet.
Evals reduce the chance of a bad release. Production analytics measure whether it happened anyway.
How do I compare server versions after a release?
Tag every deployment with service.version and compare equal windows on either side of the release: same length, weekdays and environment. Read the measures together.
| Synthetic measurement, 3 days each | v2026.10.14 | v2026.10.17 |
|---|---|---|
| Tool calls | 18,400 | 19,100 |
| Error ratio (classified attempts) | 2.1% | 4.6% |
p95 latency, find_account | 1.2 s | 1.3 s |
| Completed jobs / started jobs | 81% | 72% |
| Repeated identical calls per job | 0.3 | 0.9 |
| Distinct users affected by an error | 140 | 410 |
Calls went up while completion fell and repeated calls tripled, which suggests retries rather than growing demand. Users affected matters as much as the ratio: a 1% error rate spread across everyone is a different incident from the same rate concentrated on one account.
Flowlines adds who was affected by a release: the users, AI clients and sessions behind each change in errors or completion. For a step-by-step view of this comparison, see how to measure the impact of a tool change and MCP tool analytics.
Is it a release regression or a change in traffic mix?
Break the change down by segment first. If the error ratio rose inside each segment, suspect the release. If segments stayed flat but traffic shifted toward one that always had more errors, the mix changed.
| Synthetic segment | Error ratio before | Error ratio after | Share of calls before | Share of calls after |
|---|---|---|---|---|
| Client A | 1.5% | 1.6% | 80% | 45% |
| Client B | 6.0% | 6.1% | 20% | 55% |
| All traffic | 2.4% | 4.1% | 100% | 100% |
Overall errors almost doubled, but neither client got worse: in this example, more traffic simply arrived through Client B. Segment by AI client, tool, new versus returning users and account size. A canary release helps too, since comparing two versions at the same moment removes most mix effects.
When should I roll back a release?
Write rollback criteria and name an owner before you ship. These example thresholds are starting points, not standards:
| Signal | Example rollback trigger |
|---|---|
| Error ratio on a core tool | More than double the previous version for 15 minutes, with at least 200 attempts |
| p95 latency on a core tool | 50% above the previous version for 15 minutes |
| Job completion | Down 5 points or more against the matching prior window |
| Users affected | Errors reach users or accounts that had none on the previous version |
| Protocol failures | Any rise in schema or header validation errors after a definition change |
When a write or destructive tool is involved, roll back first and investigate after. For alerts, see MCP error spike alerts and tracking MCP tool latency.
How do I turn production failures into new eval cases?
Every confirmed production failure should become a test. After an incident:
- Find the failing job and its sequence of tool calls.
- Rewrite the request as a task with synthetic records and no personal data.
- Write a verifier for the correct outcome.
- Confirm the task fails on the bad version and passes on the fix.
- Each month, add the common new jobs you see in production.
Checklist
- [ ] Inspector CLI
tools/list --strictpasses in CI - [ ]
tools/listsnapshot reviewed, including annotations - [ ] Task and selection evals at or above the previous version
- [ ] Rollback criteria and owner written down
- [ ] After release: equal windows compared by
service.version, split by client, tool and user group - [ ] Users affected counted, not only the error ratio
- [ ] New failures added to the eval set
For the wider setup, read the MCP observability guide, the OpenTelemetry instrumentation guide and the MCP observability overview.
Sources
Frequently asked questions
How do I test an MCP server before release?
Connect to it with MCP Inspector, run contract tests on the tools/list output, and run a scripted set of realistic tasks through a model with your tools attached. Gate the build on the Inspector CLI exit codes and on your eval pass rate.
What should a contract test check on MCP tools?
Snapshot each tool's name, description, inputSchema, outputSchema and annotations, then fail the build on unreviewed changes. Pay special attention to required fields and to readOnlyHint, destructiveHint, idempotentHint and openWorldHint, because they describe behavior clients may show to users.
Can evals replace production monitoring for an MCP server?
No. Evals run tasks you wrote, with models and clients you chose. Production brings real users, other AI clients and new jobs, so compare each release against the previous one with live data as well.
How do I know whether a release caused a regression?
Compare equal windows before and after the release, tagged with service.version, and break the change down by tool, client and user group. If the rate moved inside each segment, suspect the release; if only the mix of segments changed, suspect traffic.
When should I roll back an MCP server release?
Roll back when a sustained, segment-level change crosses a threshold you wrote down before the release, such as error ratio, p95 latency or job completion on a core tool. Decide the criteria in advance so the call is not made under pressure.