The answer
The honest answer to “is the new prompt better” is rarely obvious from spot-checking outputs. Better on what, for whom? A prompt can improve quality and quietly double cost, or fix one cohort and regress another.
Preserve the prompt or release identifier as a trace property. Then use Groups to compare the sessions that share it and inspect the related issues, users, outcomes, cost, and execution behavior.
Treat a before-and-after difference as evidence, not automatic proof of causality. Open the affected sessions and control for traffic mix, environment, and use case before calling one prompt better.
How do I run a fair prompt comparison?
- Define what a successful outcome is before you change anything.
- Compare windows of the same length with a similar mix of users, clients and tasks.
- Look at outcome rate, retries, escalations and repeat contact, not only a judge score.
- Read a sample of sessions from both versions before deciding.
- Set a rollback threshold in advance so a regression is handled the same way every time.
Published