Monitoring an AI agent in production means measuring more than uptime and model latency. A complete system connects technical execution to user intent, agent behavior and the final outcome, then lets operators open the sessions behind a recurring pattern.
The five layers to monitor
| Layer | Question | Useful evidence |
|---|---|---|
| Infrastructure | Is the service available? | Errors, saturation, queue depth, dependency health |
| Execution | What did this run do? | Traces, model calls, tokens, tool calls, latency |
| Quality | Does the output meet a known standard? | Evaluators, test cases, human review, policy checks |
| Behavior | What keeps happening in real sessions? | Loops, drift, frustration, false success, abandonment |
| Outcome | Did the user or business goal happen? | Completion events, resolutions, conversions, retained usage |
No one metric represents agent health. A low error rate can coexist with wrong answers. A high evaluator score can coexist with an untested production workflow. A good outcome rate can hide one cohort that consistently fails.
1. Establish traceable identity
Give every run a stable session identifier and, where appropriate, a pseudonymous user identifier. Preserve environment, release, model, prompt or experiment version, agent name and parent-child relationships between model and tool spans.
OpenTelemetry traces ↗ are a portable foundation. Langfuse and LangSmith can provide trace storage and call-level investigation. Flowlines can analyze supported provider data or receive OpenTelemetry without replacing that source system.
2. Define outcomes for important intents
List the workflows users rely on: issue a refund, modify code, answer a compliance question, schedule a meeting. For each one, define the strongest result you can observe. Prefer a verified external event over a model-generated statement.
When direct outcomes are unavailable, label coverage as unknown rather than converting missing data into success.
3. Detect silent failure signals
Start with patterns that have inspectable evidence:
- A completion claim without the required tool result
- Repeated equivalent calls without progress
- Several user corrections or reformulations
- Drift away from the original request or constraint
- Abandonment before a task reaches a completion event
- A quality gap concentrated in a language, plan or use case
Treat each detector as a lead. Review the complete sessions before calling the pattern a root cause.
4. Build alerts around decisions
An alert should contain the affected intent, user scope, change over time, likely impact and links to representative sessions. "Error rate increased" is a symptom. "Refund requests loop after a timeout for users on plan X" is an investigation.
Route product capability gaps to product owners, reliability errors to engineering, and high-value user problems to customer teams. One shared evidence link prevents three teams from debating different dashboard summaries.
5. Compare releases with matched traffic
Record a release or experiment property on every session. Compare similar time windows, intents and cohorts. Check sample size, telemetry coverage and traffic mix. A directional improvement is not proof until the evidence sessions support the explanation.
A production monitoring checklist
- Verify one complete session from request to outcome.
- Confirm user, session, environment and release fields.
- Track availability, errors, latency, tokens and cost.
- Score known quality and safety cases.
- Monitor loops, drift, frustration, abandonment and false success.
- Segment outcomes by intent and user cohort.
- Keep every aggregate linked to the original sessions.
- Recheck the same issue after each relevant change.
Flowlines is the behavior and outcome layer in this stack. It turns production traces into users, intents, recurring issues, affected sessions and next steps while keeping your trace provider in place.
Explore the product workflow, review integrations, or see production comparisons.
Worked example: a refund agent that looks healthy
Consider a synthetic week with 1,000 refund sessions. Infrastructure reports 99.5% successful HTTP responses. That says the service answered; it does not say a refund happened.
Join each session to its refund transaction result. Suppose 720 completed a refund, 180 had a verified rejection or failed attempt, and 100 have no observable outcome. Report completion as 720/900 among sessions with known outcomes, alongside 90% outcome coverage. Do not silently count the missing 100 as successful or unsuccessful.
Now segment by intent and release. If 40 of the 180 failed attempts contain three equivalent refund lookups, inspect a sample of those sessions. A useful hypothesis is that the agent retries an unchanged lookup after a timeout. It is not yet a proven defect in the refund service.
Make the alert actionable
Write the rule around a decision: “Notify engineering when refund sessions repeat the same lookup three times without a transaction result.” Start with a team-agreed observation window and minimum sample, then adjust it from reviewed false positives. Three calls is an illustrative threshold, not a universal default.
A useful notification includes the time window, number of matching sessions, affected users, outcome coverage, and two or three representative session links. Route “refund unavailable for this account” separately from “agent kept retrying after a timeout”; they have different owners.
Verify the source before trusting the chart
Use a test namespace and one synthetic account. Run one completed refund, one expected rejection, one timeout and one abandoned session. Check that each appears once with the correct identity, environment and release. Confirm that a rejection is not mislabeled as an exporter failure, and that abandonment remains unknown when there is insufficient evidence.
Then disconnect the exporter briefly in a safe test environment. Missing telemetry should change coverage or trigger an ingestion-health warning, not improve the agent's reported success rate. Avoid intentionally breaking production dependencies for this check.
Operational handoff
For every alert, record an owner, review cadence, severity and closure condition. A bounded retry change is complete only when the matching pattern decreases in comparable traffic and the previously failing workflow still works. Keep a sample of false positives as regression cases. Review whether tool schemas, authentication or user intent changed before deciding a new spike is a model regression.
See users and cohorts and compare releases to connect this monitoring loop to the people affected.
Last reviewed Sep 7, 2026