I asked a team how they evaluated their agent. They had a deck. The deck had four slides about MMLU and one chart of an internal benchmark with twelve examples on it. I asked how the agent was doing. They pointed at the chart. I asked how it was doing in production. There was a pause.
The answer was in their support inbox. Forty tickets a week, half of them screenshots of the agent saying something wrong, the other half of the agent doing something wrong. None of those tickets had ever been turned into an eval. They were rotting in a Slack channel where they’d be deleted in ninety days.
The dataset you already own
Three sources, in order of how cheaply you can mine them:
- Human-edited outputs. Every time a user accepted, then edited, the agent’s draft, you have a labelled example: input → bad output → corrected output. Capture both halves.
- Thumbs-down with comments. The thumb is noise. The comment is gold. Most teams aggregate the thumbs and throw away the text.
- Re-asks. When a user immediately re-prompts after the agent answered, that’s a failure signal too. Log them. Group them. Ten re-asks of the same shape is a category.
Why nobody does this
Because there is no glamour in writing eval cases from a support queue. It feels like data entry. It is data entry. And it’s the only way your evals will ever look like the requests your users actually send, instead of the requests your team made up at offsite.
The team with the synthetic benchmark and the great-looking dashboards is not the team with the working agent. The team with the working agent has two thousand messy real-world cases pulled from production every week, and someone whose job is to keep that pipeline alive.
Your customers wrote your test set. You just have to take it.