On this page
Production evaluation asks whether an agent works for real users under real conditions. It combines direct outcomes, deterministic checks, model-based evaluation, behavioral signals and human review. Each source has a different confidence level.
Start with the decision
Do not begin by collecting scores. Begin with the decision the team needs to make: ship a model change, fix a recurring failure, expand a capability, investigate one cohort or decide whether an MCP tool is reliable enough for a workflow.
That decision determines the population, time window, outcome and evidence required.
Use an evidence hierarchy
| Evidence | Best use | Main limitation |
|---|---|---|
| Verified business outcome | Task completion and value | Not available for every intent |
| Deterministic check | Structured correctness, policy and tool state | Covers only specified rules |
| Human review | Nuance and disputed cases | Slow and subject to reviewer variation |
| LLM evaluator | Scalable quality estimates | Model bias, calibration and prompt sensitivity |
| Behavioral signal | Frustration, looping, drift, abandonment | Usually evidence of risk, not ground truth alone |
| Technical status | Availability and execution health | Does not establish user success |
Prefer the strongest available evidence. Keep unknown outcomes unknown. Never relabel missing feedback as positive feedback.
Build a representative evaluation population
Offline test sets are valuable for regressions and known edge cases. Production evaluation adds the workflows nobody predicted. Sample across common intents, high-value tasks, long sessions, low-frequency failures, new users, repeat users, languages, plans and releases.
Record the denominator. "Twelve bad sessions" means something different out of twenty sessions than out of two hundred thousand.
Calibrate evaluators against people
If you use an LLM judge, create a reviewed set with clear rubrics. Measure agreement by intent and failure type, not only overall. Inspect false positives and false negatives. Version the evaluator prompt and model just like application code.
Use model-based evaluation to prioritize review, not to make irreversible decisions without evidence.
Evaluate complete trajectories
Agents act across turns and tools. Score the complete path when the task depends on previous constraints or external actions. A final answer may look correct while the required tool never ran. A tool may fail once, recover, and still produce a successful outcome.
For MCP workflows, evaluate the user intent, tool path, call results, loop count and final outcome together.
Monitor behavioral quality continuously
Watch for changes in issue-free sessions, explicit negative feedback, repeated requests, abandonment, unsupported completion claims, drift and cohort-level gaps. Link each aggregate to examples so the team can distinguish a real pattern from detector noise.
Verify changes after deployment
Attach release, model, prompt and experiment identifiers to sessions. Compare matched production populations and inspect exact examples on both sides. Report confidence limits or at least denominators. Check whether telemetry coverage changed before crediting the release.
A defensible production scorecard
- Task outcome rate by important intent
- Issue-free session rate with a visible denominator
- Quality and safety evaluator results, with calibration notes
- Explicit feedback and abandonment
- Tool-call success, latency, retries and outcome coverage
- Results by user cohort, environment and release
- Human-reviewed examples for the most consequential decisions
Flowlines connects these operating questions to production sessions. It does not replace eval suites or human judgment. It makes the unreviewed population visible and gives evaluators a path back to evidence.
Read the monitoring framework, measure MCP behavior and outcomes, or compare production releases.
Worked example: compare two releases without hiding unknowns
Use this synthetic evaluation sheet for one intent, one environment, and comparable user cohorts:
| Window | Eligible sessions | Known outcomes | Completed | Completion among known |
|---|---|---|---|---|
| Before | 120 | 100 | 80 | 80% |
| After | 125 | 100 | 88 | 88% |
The observed difference is eight percentage points. Outcome coverage is 83.3% before and 80% after. Do not report that the release improved success by eight points without investigating why outcome coverage fell and whether the populations are comparable.
Sample both successful and unsuccessful sessions from each window. Include unknown outcomes as their own stratum. Otherwise the review systematically excludes the sessions your telemetry understands least.
Define a reviewer rubric
For a support agent, ask three separate questions:
- Did the agent address the original request?
- Was a required external action confirmed by supporting evidence?
- Did the final answer accurately describe completion, failure, or remaining work?
Store each answer separately with a short rationale and session reference. A correct explanation of a denied refund can pass answer quality while the requested refund remains uncompleted. Collapsing these into one score hides an important product decision.
Decide what would block the release
Set the decision rule before looking at the result. A new unauthorized action may be a blocker, while a small movement in average response quality may require more review. Use cautious language for observational comparisons. A randomized comparison supports stronger causal conclusions only when assignment, logging, and analysis are valid.
Compare release evidence and review why offline evals are not enough.