How to measure AI quality with evals

1 The bottom line

Turn vague complaints into small eval sets, then pair them with production monitoring so real failures become new evals. Evals only catch what you already know, and monitoring finds the rest.

2 Key takeaways

• Turn each vague complaint into 30 to 40 examples of the failure. A small set with golden answers can be rerun on every new model.
  • For Claude 2 "not following instructions", about 80% of cases were bad JSON output, and the eval now passes at 100% or 99.9% Dianne Penn, Anthropic45:00.
• Read failed transcripts before you write the eval. Each failure type needs a different fix.
  • A "Claude hallucinated" report split into a tool use failure and a search and knowledge synthesis failure Dianne Penn, Anthropic42:55.
• Ten great evals are enough to start. They let the team quantify the goal, track progress and see what is missing.
  • On memory, Cat Wu handed off five evals, how to run them, which ones pass and the prompt that raised the success rate Cat Wu, Anthropic55:14.
• Use evals and production monitoring together. Evals guard known failures, while monitoring shows which traces to inspect.
  • Implicit signals include a ChatGPT user regenerating an answer or a user switching off a code review product Aishwarya Naresh Reganti and Kiriti Badam, OpenAI36:00.
• Log what users do with the AI output. It gives you error analysis for free.
  • In the copilot stage, record how much of the AI draft the human used or changed Aishwarya Naresh Reganti and Kiriti Badam, OpenAI54:10.

Eval are important, production monitoring is important but this notion of only one of them is going to solve things for you, that is completely dismissible.

Aishwarya Naresh Reganti and Kiriti Badam, OpenAI

2.1 Gather real complaints and sessions

Start from what real users and teammates say goes wrong, not from guesses.

• Ask about five trusted people first. Source: Cat Wu, Anthropic54:40
  • A handful of users are much better at saying what makes a model or harness good. Find about five and ask them first.
  • Internally, ask every teammate for their vibe on a new model, for example at team lunches.
  • Then pull data to test the patterns they raise, such as too-abrupt replies or low-quality memories.
• Collect real tasks and count them by category. Source: Roman Ugarte, Grok Bot21:30
  • Gather the tasks users actually give the bot, quantify them by category and improve week over week.
  • One case was a Salesforce dashboard the mouse could not click precisely. A workflow that had failed for seven days worked after one infrastructure fix.
• Expect unpredictable inputs and outputs. Source: Aishwarya Naresh Reganti and Kiriti Badam, OpenAI8:39
  • Users phrase intent in endless ways, and the model is sensitive to prompt phrasing.
  • So build around anticipating and measuring behavior, not around a mapped workflow.
  • Separate what seems possible in a demo from what is consistently good in practice Connor Zwick, Speak16:40.
• Test long sessions, not just short samples. Source: Jack McDermott, ElevenLabs46:40
  • Voice models can degrade the way language models do. Fix drift when it appears.
  • Burt Reynolds' voice morphed into a demonic sound only after half an hour or more of playback, and the clips went viral.

2.2 Read failures to find the real cause

A complaint is a symptom, so look at the sessions until you can name the dominant failure.

• Go through failed trajectories and classify why they failed. Source: Dianne Penn, Anthropic42:55
  • Typical classes are hallucination, overconfidence, wrong tool call, and right document but wrong facts.
  • Each maps to a different fix: tool use, search synthesis or alignment.
• Ask the model why it made the mistake. Source: Cat Wu, Anthropic53:48
  • Ask it to reflect, then use the answer to fix the harness.
  • The cause can be a confusing system prompt, a missed requirement, or a sub-agent whose unchecked work it trusted. The trigger was a model that changed the front end and ran tests without using the UI.
• Bring clear user goals and sample sessions to research. Source: Tara Seshan and Nan Yu, OpenAI18:46
  • Sample sessions show why the model failed.
  • Trust from research is what gets the iteration loop going.

2.3 Turn the pattern into an eval set

Write the failure down as examples with right answers, keep the set small, and rerun it often.

• Write 30 to 40 examples of the failure with a golden answer. Source: Dianne Penn, Anthropic45:00
  • First ask the user for the exact prompt and response, and find the dominant failure.
  • Run the set on every new model version to see whether the pain point is fixed.
  • Keep it on distribution, with cases where the model fails and cases where it should not fail Dianne Penn, Anthropic42:55.
  • The bad JSON eval for Claude 2 now passes at 100% or 99.9%.
• Write expected inputs and ideal outputs before you build. Source: Aishwarya Naresh Reganti and Kiriti Badam, OpenAI48:25
  • PMs and subject matter experts supply the information.
  • This often exposes that team members disagree on how the product should behave. The set will not be comprehensive, but it gives you a start.
• Stop at about ten great evals. Source: Cat Wu, Anthropic55:14
  • You do not need hundreds. Ten good ones quantify the goal and show what is missing.
• Define the user need through an eval, not only a document. Source: Dianne Penn, Anthropic1:01:00
  • An eval shortens the distance to action for researchers.
  • It also lets you measure whether the next model improves the area. You can't improve what you can't measure.

2.4 Add production monitoring and loop

Evals guard what you know, so use live signals to find what you do not.

• Run evals and monitoring side by side. Source: Aishwarya Naresh Reganti and Kiriti Badam, OpenAI36:00
  • Evals encode what must not go wrong, such as an agent that should never offer refunds.
  • Monitoring uses explicit and implicit signals to pick which traces to read, since you cannot read them all at high volume.
  • Codex pairs evals that protect core behavior with A/B tests, user reactions and social media feedback. Code review users annoyed by wrong comments switch the product off.
• Turn each new failure mode you care about into an eval. Source: Aishwarya Naresh Reganti and Kiriti Badam, OpenAI36:00
  • This is how monitoring feeds the eval set.
• Log what humans do with each AI output. Source: Aishwarya Naresh Reganti and Kiriti Badam, OpenAI54:10
  • In the copilot stage, record how much of the draft the human used or changed.
  • Eval metrics only catch errors you already know about. Logging reveals patterns that only appear in production.
• Run a continuous development and calibration loop. Source: Aishwarya Naresh Reganti and Kiriti Badam, OpenAI46:05
  • Development: scope the capability, curate a starter data set, set up the app, design metrics, deploy and run them.
  • Calibration: analyze unexpected behavior, spot error patterns, apply fixes, and add new metrics only when needed.
  • Simple one-off errors, like a badly defined tool, can just be fixed without a new metric.
  • The loop came after an end-to-end customer support agent was shut down because the team made too many hot fixes to track emerging problems.

2.5 Where they disagree

Sources differ on how many evals to write, what an eval is, and when a PRD still helps.

• How many evals to write.
  • Tara Seshan and Nan Yu, OpenAI18:46 say writing evals as much as possible is the most important skill.
  • Cat Wu, Anthropic55:14 says ten great evals are enough to quantify the goal.
• Whether evals replace PRDs.
  • Evals work as shorthand once the problem is clear Dianne Penn, Anthropic48:05.
  • Every Anthropic model release still gets a PRD, mainly for product surfaces, engineering, legal and safety. PRDs also help on ambiguous opportunities like computer use, where pain points are not yet defined.
• Whether to build an LLM judge. Source: Aishwarya Naresh Reganti and Kiriti Badam, OpenAI38:50
  • A PM writing evals does not mean writing a production-grade LLM judge.
  • For complex use cases, judges miss emerging patterns. It can be better to read user signals, fix problems and check for regressions.
  • Public leaderboards like LM Arena only tell you about models, not your product. One client said they did evals but had only checked LM Arena and Artificial Analysis.