On this page
The fastest way to debug a production AI agent is to narrow the problem before reading traces. Start with the failed outcome and affected population, identify the recurring behavior, then open the smallest set of complete sessions that can confirm or reject an explanation.
1. State the symptom in user terms
"The agent is bad" is not actionable. Write the observable gap: users asking for refund status repeat the request and leave, the coding agent reports tests passed without a completed test command, or an MCP research workflow calls the same search tool twenty times.
Include the time window, environment, release, intent and affected cohort.
2. Separate technical errors from behavioral failures
Check availability, exceptions, timeouts and dependency failures first. If those are healthy, do not stop. Inspect for false success, intent mismatch, drift, loops, frustration and abandonment.
A recovered tool error may have no user impact. A technically successful trace may contain the actual product failure.
3. Find the pattern before the example
Group sessions by intent, issue, tool path, release and cohort. Measure the denominator and change over time. This prevents an unusual but harmless trace from consuming the investigation.
Once a pattern is real, select several representative sessions: common examples, severe examples, boundary cases and a few successful sessions from the same population.
4. Reconstruct the complete trajectory
Read from the original user intent through every relevant turn and tool result. Ask:
- Did the agent preserve constraints from earlier turns?
- Did each tool receive the intended validated arguments?
- Did the agent interpret errors and empty results honestly?
- Did it repeat an equivalent action without new information?
- Did the final claim match the tool evidence?
- Was a measurable outcome recorded?
For MCP servers, inspect the complete tool execution and final client-visible result, not only HTTP or transport status.
5. Form a falsifiable explanation
Write a statement the evidence could disprove: "refund sessions loop because the retry policy ignores structured timeout errors" is better than "the model is unreliable." Identify what should differ in successful sessions if the explanation is correct.
6. Change one controllable layer
Choose the smallest relevant change: prompt, tool schema, validation, retry ceiling, progress check, model, retrieval source or product capability. Avoid bundling several fixes when you need to learn which one worked.
7. Verify in matched production traffic
Tag the new release. Compare the same intents and cohorts, check telemetry coverage and inspect examples on both sides. Watch for displacement, where one issue declines but another appears.
Debugging checklist
- User-visible symptom and impact
- Time window, release, intent and cohort
- Technical health checked
- Recurring behavior measured with a denominator
- Complete failure and success sessions reviewed
- Explanation linked to evidence
- One controlled change shipped
- Outcome verified after release
Flowlines shortens this path by grouping production behavior into issues and keeping affected users, use cases and evidence sessions attached. Engineers still use the source trace store for span-level detail; Flowlines shows which traces deserve attention.
See the failure-mode taxonomy or learn how to evaluate production agents.
Worked investigation: the invoice was sent, but nothing arrived
Start with a synthetic complaint: a user asked an agent to email an invoice. The final answer claimed success, but the user received no message. Keep the initial hypothesis open. This could be a tool failure, an incorrect recipient, a missing attachment, a premature answer, or delayed delivery.
Reconstruct the sequence
| Step | Evidence to inspect | What it establishes |
|---|---|---|
| Request | Original user instruction | Intended recipient and invoice |
| Lookup | Observed lookup result | Whether the correct invoice was found |
| Send | Observed send result | Whether delivery was accepted, rejected, or never attempted |
| Final answer | Agent completion statement | What the user was told |
| External check | Delivery receipt, if available | Whether the claimed action happened |
Suppose the lookup returned no invoice and no send action appears. The defensible finding is that the agent claimed to send an invoice without an observed send action. Do not jump to the conclusion that the email provider lost it. If telemetry is incomplete, the conclusion must remain uncertain.
Turn the hypothesis into a regression test
In a test environment, make invoice lookup return no result. Assert that the agent asks for clarification or reports the missing invoice, does not invent an attachment, and does not claim completion. Add a second case where send returns a domain error despite a successful transport response.
Record the new release identifier, compare the same intent and relevant cohorts, and inspect remaining failures. A lower alert count is not enough if traffic or telemetry coverage also fell.
Inspect failure patterns, prioritize affected users, and compare the release.