Why isn't running evals enough?
Evals test the cases you thought of, on inputs you curated, before you ship. Production is the cases you didn't think of, on real users, over time. Evals are necessary but they're a pre-flight check, not a flight recorder, they can't see drift, real-user intent, or the failure modes that only emerge at scale.
Evals are valuable. They catch regressions on a known test set before a deploy. But they share a structural blind spot: you can only eval what you anticipated. The test set is your hypotheses about how the agent fails, frozen in time.
Production violates those hypotheses constantly. Real users phrase things you didn't script, in volumes that surface rare failures, over weeks where behavior drifts. An eval suite that's green can sit on top of an agent that's quietly degrading for a cohort you never tested.
Behavioral observability is the other half. Evals tell you the agent passed the cases you wrote; Flowlines tells you what's actually happening across every real session, which signals are firing, which cohorts are slipping, which deploy moved the numbers. You need both: the pre-flight check and the flight recorder.
Last updated 2026-05-28