On this page
The MCP server KPIs that show real value are activation (new users who complete a first useful job), retention (users who come back in a later week) and calls per completed job (how much work the agent needs to finish one). Flowlines shows the users, returning users, sessions, use cases and outcomes behind each of them, per tool and per AI client.
Call volume is the number most teams report first, and the one that says least. The MCP analytics guide explains why and defines the base measures: active users, outcome coverage, completion and retry incidence. This guide goes one step further. It turns those measures into the three KPIs a team can review every week, with formulas, synthetic worked examples and the traps in each.
What are the most important KPIs for an MCP server?
Track activation, retention and calls per completed job, and report each one with the coverage it depends on; Flowlines shows all three next to the users and sessions behind them.
| KPI | Formula | Question it answers | Read it with |
|---|---|---|---|
| Activation rate | New users with a completed job within 7 days of their first call, divided by new users in the cohort | Do new users get value quickly? | Time to first completed job |
| Week 1 retention | Users active again in the week after their first week, divided by the cohort | Did the server become part of a workflow? | Cohort size and client mix |
| Returning share | Active users who were also active in an earlier period, divided by active users | How much of this week's usage is repeat usage? | Identity coverage |
| Calls per completed job | Calls inside completed jobs divided by completed jobs, per use case | How efficiently does the agent finish work? | Median and 90th percentile, not only the mean |
| Outcome coverage | Jobs with a known result divided by all attempted jobs | How much of the KPI rests on known outcomes? | Unknown outcomes kept visible |
Three rules apply to every row. Exclude your own team and test scripts. Keep users without a known identity as a separate group instead of dropping them. Compare the same window length, environment and client mix before you call a change an improvement.
Supporting measures still matter, but they explain the KPIs rather than replace them: tool call success rate and p95 latency per tool (see MCP tool analytics), sessions per user, and the share of sessions from each AI client.
How do I measure MCP activation?
Define activation as the first completed job, not the first tool call, and measure the share of new users who reach it within a fixed window. Flowlines shows new users, their first sessions and whether those sessions ended with a known outcome.
A first call proves the server was reachable. It does not prove the user got anything done. Many first sessions are a person trying one prompt to see what happens. The activation event should be the smallest unit of real value for your server: a brief created, a ticket filed, a report returned that the user kept working with.
The funnel below is synthetic. It follows 120 new users who first called a CRM MCP server during one four-week window.
| Step | Users | Share of cohort |
|---|---|---|
| First tool call | 120 | 100% |
| First successful tool call | 96 | 80% |
| First completed job within 7 days | 54 | 45% |
| Second completed job within 28 days | 30 | 25% |
The largest drop sits between a successful call and a completed job. That gap is where tool descriptions, missing tools and confusing results show up, and it is invisible in a call-volume chart. Report time to first completed job beside the rate: a median of two sessions is a different product from a median of six.
What counts as a completed job?
A job counts as completed only when there is evidence: a final tool that succeeded and produced the artifact, an outcome your integration reports, or a follow-up event in your own product. Jobs without that evidence stay unknown, not failed. If unknown outcomes are a large share, fix outcome coverage before you set an activation target.
How do I measure MCP retention?
Group users by the week of their first call and count how many are active again in each later week; Flowlines shows returning users per period and the AI client each one uses.
| First week (synthetic) | New users | Week 1 | Week 2 | Week 3 |
|---|---|---|---|---|
| Week of Sep 7 | 40 | 18 (45.0%) | 14 (35.0%) | 12 (30.0%) |
| Week of Sep 14 | 32 | 13 (40.6%) | 10 (31.3%) | 9 (28.1%) |
| Week of Sep 21 | 48 | 24 (50.0%) | 18 (37.5%) | not yet complete |
| Week of Sep 28 | 36 | 17 (47.2%) | not yet complete | not yet complete |
Read the table along the diagonal, not just the first column. A curve that flattens, here around 30% by week 3, means a stable group made the server part of its work. A curve that keeps falling toward zero means people try it and leave.
Three choices change the result:
- The period. Match it to how often the job happens. A weekly account review should be measured weekly; a code-search server used many times a day can use daily or weekly return.
- What counts as active. A single failed call is not usage. Count a user as active in a period when they complete a job, or at least make a successful call, and state which rule you used.
- Incomplete cohorts. Leave a cell empty until the cohort has lived through the full period. Averaging a two-day-old cohort into week 1 retention drags it down.
Returning share is the simpler weekly number: returning users divided by active users. It answers a different question. A fast-growing server can show a falling returning share while cohort retention is healthy, because most active users are new. MCP server analytics shows both views.
What is calls per completed job, and why does it beat call volume?
Calls per completed job is the number of tool calls inside completed jobs divided by the number of completed jobs; it falls when the agent finds the right path faster and rises when it struggles. Flowlines shows the tool path of each session so you can see where the extra calls go.
Total calls rise for good reasons (more users) and bad ones (retries, loops, unhelpful results). Calls per completed job separates them because it only counts work that ended with a result.
| Use case (synthetic) | Completed jobs | Calls in completed jobs | Mean | Median | 90th percentile |
|---|---|---|---|---|---|
| Prepare a renewal brief | 80 | 320 | 4.0 | 3 | 8 |
| Find the right account | 150 | 270 | 1.8 | 1 | 4 |
| Investigate duplicate records | 20 | 190 | 9.5 | 7 | 21 |
Compute it per use case. Different jobs need different numbers of calls, so a single server-wide average moves whenever the mix of jobs moves. Report the median and the 90th percentile: the mean of 9.5 for duplicate investigations hides a long tail where the agent loops.
Keep calls from unfinished jobs on a separate chart. If they are excluded from this KPI and never looked at, a release that makes failures more expensive can still look like an improvement. For the patterns that inflate this number, such as repeated calls and the wrong tool picked first, see why agents ignore your MCP tools.
Which KPIs should I break down by AI client and tool?
Break all three KPIs down by AI client first, then by tool, because Claude, ChatGPT, Cursor and other clients attempt different jobs and choose tools differently. Flowlines groups users, sessions and tool calls by the client that made them, with an explicit unknown bucket.
| Breakdown | What a difference can mean | Next check |
|---|---|---|
| Activation by client | One client reaches the first completed job less often | Compare the same use case in both clients |
| Retention by client | Users of one client do not come back | Errors and calls per job in that client |
| Calls per job by client | One client needs more attempts for the same job | Tool descriptions as that client presents them |
| Calls per job by tool | One tool appears in long, looping jobs | Its results, errors and repeated calls |
Client names are self-reported and can be missing, so treat them as analytics context, not identity. MCP user analytics covers tracking calls by user and by client, and recognizing the same user across clients.
Is there a benchmark for a good MCP server?
No public benchmark covers activation or retention for MCP servers yet, so set a baseline from your own first month and compare each period against it. Numbers shared in posts or talks rarely state their denominators, test-traffic rules or outcome coverage, which makes them hard to compare with yours.
A baseline is useful when it is specific: the same environment, the same window length, the same clients and the same definition of a completed job. Recompute it when you change any of those, and say so in the report.
Do these KPIs still work after the stateless MCP revision?
Yes, but sessions and clients are measured differently: under the 2026-07-28 revision there is no initialize handshake, and the client describes itself on each request. Activation and retention depend on a stable user identity, which still comes from your own authentication. Calls per completed job depends on how you group calls into a job, which now needs your own grouping rule. Measuring MCP after the stateless spec covers both.
How do I run a weekly MCP KPI review?
Review the three KPIs in the same order every week, with their coverage, and open sessions before you decide anything. This checklist fits in 30 minutes:
- Coverage first. Identity coverage, outcome coverage and telemetry from every server and environment. A quiet week can be an exporter problem.
- Activation. New users, activation rate and time to first completed job, compared with last week and the baseline.
- Retention. The newest complete cohort cell and the returning share.
- Calls per completed job. Median and 90th percentile for the top three use cases.
- By client. Any client whose KPIs moved more than the others.
- Sessions. Open three sessions behind the biggest change: one completed, one unfinished, one unknown.
- One decision. Write down one change to make, its owner, and the KPI that should move.
Key takeaway: report activation, retention and calls per completed job instead of call volume, each with its denominator and coverage. Flowlines shows the users, clients, sessions and outcomes behind every number, and the Free plan includes 100 analyzed MCP sessions per month. For the full monitoring model, read how to monitor MCP servers in production, or see MCP observability for the product walkthrough.
Frequently asked questions
What are the most important KPIs for an MCP server?
Activation (new users who complete a first useful job), retention (users who come back in a later period) and calls per completed job (how much work one finished job takes). Read them with identity coverage and outcome coverage, because a KPI on a partial population can mislead.
How do I measure MCP activation?
Define activation as the first completed job, not the first call. Activation rate is new users who complete a job within a fixed window, such as 7 days after their first call, divided by all new users in the cohort. Report time to first completed job beside it.
How do I measure MCP retention?
Group users by the week of their first call and count how many are active again in each later week. Choose the window to match how often the job happens: a weekly reporting workflow should not be judged on daily return.
What is calls per completed job?
Calls made inside completed jobs divided by the number of completed jobs, reported per use case as a median and a 90th percentile. A falling value usually means the agent finds the right path faster. Count calls from unfinished jobs separately.
Is there a benchmark for a good MCP server?
No public benchmark covers activation or retention for MCP servers yet, and products differ too much for one number to mean much. Set a baseline from your own first month, then compare equal windows, the same clients and the same use cases.
Can Flowlines track MCP server KPIs?
Yes. Flowlines shows distinct and returning users, sessions, use cases, outcomes and calls per tool, by AI client, from production MCP sessions. The Free plan includes 100 analyzed MCP sessions per month.