The fastest way to debug a production AI agent is to narrow the problem before reading traces. Start with the failed outcome and affected population, identify the recurring behavior, then open the smallest set of complete sessions that can confirm or reject an explanation.
1. State the symptom in user terms
"The agent is bad" is not actionable. Write the observable gap: users asking for refund status repeat the request and leave, the coding agent reports tests passed without a completed test command, or an MCP research workflow calls the same search tool twenty times.
Include the time window, environment, release, intent and affected cohort.
2. Separate technical errors from behavioral failures
Check availability, exceptions, timeouts and dependency failures first. If those are healthy, do not stop. Inspect for false success, intent mismatch, drift, loops, frustration and abandonment.
A recovered tool error may have no user impact. A technically successful trace may contain the actual product failure.
3. Find the pattern before the example
Group sessions by intent, issue, tool path, release and cohort. Measure the denominator and change over time. This prevents an unusual but harmless trace from consuming the investigation.
Once a pattern is real, select several representative sessions: common examples, severe examples, boundary cases and a few successful sessions from the same population.
4. Reconstruct the complete trajectory
Read from the original user intent through every relevant turn and tool result. Ask:
- Did the agent preserve constraints from earlier turns?
- Did each tool receive the intended validated arguments?
- Did the agent interpret errors and empty results honestly?
- Did it repeat an equivalent action without new information?
- Did the final claim match the tool evidence?
- Was a measurable outcome recorded?
For MCP servers, inspect the tools/call spans and the final client-visible result, not only HTTP or transport status.
5. Form a falsifiable explanation
Write a statement the evidence could disprove: "refund sessions loop because the retry policy ignores structured timeout errors" is better than "the model is unreliable." Identify what should differ in successful sessions if the explanation is correct.
6. Change one controllable layer
Choose the smallest relevant change: prompt, tool schema, validation, retry ceiling, progress check, model, retrieval source or product capability. Avoid bundling several fixes when you need to learn which one worked.
7. Verify in matched production traffic
Tag the new release. Compare the same intents and cohorts, check telemetry coverage and inspect examples on both sides. Watch for displacement, where one issue declines but another appears.
Debugging checklist
- User-visible symptom and impact
- Time window, release, intent and cohort
- Technical health checked
- Recurring behavior measured with a denominator
- Complete failure and success sessions reviewed
- Explanation linked to evidence
- One controlled change shipped
- Outcome verified after release
Flowlines shortens this path by grouping production behavior into issues and keeping affected users, use cases and evidence sessions attached. Engineers still use the source trace store for span-level detail; Flowlines shows which traces deserve attention.
See the failure-mode taxonomy or learn how to evaluate production agents.
Worked investigation: “the invoice was sent” but nothing arrived
Start with a synthetic complaint: a user asked an agent to email an invoice. The final answer claimed success, but the user received no message. Keep the initial hypothesis open. This could be a tool failure, incorrect recipient, missing attachment, premature answer or delayed delivery.
Reconstruct the sequence
| Step | Evidence to inspect | What it establishes |
|---|---|---|
| Request | Original user instruction | Intended recipient and invoice |
| Lookup | Validated tool arguments and result | Whether the correct invoice was found |
| Send | Final client-visible send result | Whether delivery was accepted, rejected or never attempted |
| Final answer | Agent's completion statement | What the user was told |
| External check | Delivery receipt, if available | Whether the claimed action actually happened |
Suppose the lookup returned an empty result and no send tool call exists. The defensible finding is “the agent claimed to send an invoice without an observed send action.” Do not jump to “the email provider lost it.” If the send span is missing because telemetry dropped, the conclusion must remain uncertain.
Determine the scope
Search for sessions with the same intent, release and unsupported completion claim. Separate unique sessions from repeated tool calls and unique users from sessions. If one user tried ten times, it is ten affected sessions but one known affected user.
Review examples outside the suspected pattern too. Successful sends help identify the expected path; similar requests with missing telemetry help expose detection blind spots. Never export an entire customer conversation into a ticket when a redacted step and stable session reference are sufficient.
Turn the hypothesis into a regression test
In a test environment, make invoice lookup return no result. Assert that the agent asks for clarification or reports the missing invoice, does not call send with an invented attachment, and does not claim completion. Add a second case where send returns a structured error despite a successful HTTP response.
A prompt change can require a confirmed result before claiming success. A tool-contract change can make “no invoice found” unambiguous. Choose the smallest change supported by the evidence, then test both failure and legitimate success paths.
Verify after the fix
Record the release identifier on new sessions. Compare the same intent and relevant cohorts before and after deployment, including outcome coverage. Inspect remaining failures instead of declaring success from a lower alert count: traffic may have fallen or the detector may no longer recognize the new wording.
Close the issue only when the behavior is addressed and the evidence is sufficient. If external delivery remains unobservable, retain that limitation rather than labelling all accepted sends as delivered.
Inspect failure patterns, prioritize affected users, and compare the release.
Last reviewed Sep 7, 2026