Shreya Shankar
Researcher and instructor (co-teachers of the top AI evals course on Maven)
Where to hear and read about Shreya Shankar1 episode or test
Shreya Shankar is a researcher and instructor who co-teaches the top-ranked AI evals course on Maven with Hamel Husain. The thing most worth learning from her is to read real traces yourself, name and count the failures, and only then build narrow pass or fail judges checked against human labels.
Shreya Shankar: the story and the lessons
1Who Shreya Shankar is and what she teaches
Shankar co-teaches the top-ranked evals course on Maven with Hamel Husain. Husain and Shankar have taught over 2,000 PMs and engineers across 500 companies. Students include large parts of the OpenAI and Anthropic teams.
The course is also the highest grossing on Maven. It comes with a 60-page written guide and an AI assistant trained on every lesson, office hour, Discord chat, blog and paper the two have produced.
She researches how builders' standards shift as they review AI outputs. Her 'Who Validates the Validator' study, started in late 2023, found that people's views of good and bad change as they see more outputs. She calls this criteria drift.
Even experienced LLM builders only thought of some failure modes after about 10 outputs. This is the basis for treating judge prompts as living requirements documents.
She distrusts anyone selling an eval technique as entirely new. Open and axial coding come from social science. Husain also points to an eight-year-old Andrew Ng video on error analysis.
What the two added is a curriculum adapted to LLM products. Shankar treats the age of the method as a reason to trust it.
Her stated aim is that everyone can build AI products, not riches. She wants more people teaching evals and asks for real case studies of what people implemented, which keep her and Husain going.
She says her vision is people on the same page about building AI products, not the two becoming billionaires.
2How Shankar teaches error analysis
Shankar starts with about 100 real traces and a quick note on each. Open the observability tool, sample traces from the real application and write a free-form note on what went wrong. Do this before writing any tests.
The number 100 is arbitrary and only makes the task feel doable. On the Nurture Boss leasing assistant, a few traces surfaced a missed human handoff, garbled text-message flow and a hallucinated virtual tour.
She notes only the most upstream error in each trace. Write down the first thing that is wrong and move on, rather than cataloguing every problem. The first two or three notes are painful, then the pace picks up.
Shankar insists humans do the first pass, not an LLM. Handing open coding to an LLM is the number one pitfall. She would bet money that ChatGPT, asked whether the virtual-tour trace has an error, says no.
A domain expert knows the product has no virtual tours, so offering one is a hallucination. The LLM lacks that context. LLMs come in later, for synthesis.
She wants one trusted domain expert leading, not a committee. Husain's 'benevolent dictator' is one person whose taste you trust: a lawyer for legal, a psychiatrist for mental health, often the product manager.
Committees bog down chasing agreement, and the process can become infinitely expensive if you aren't careful.
Shankar stops reading traces at theoretical saturation, not a set count. Keep going until new traces stop changing what you would do next. Some apps need only 40, 60 or even 15, and intuition comes after two or three rounds.
The right number depends on the application and on how practiced the analyst is.
She asks for notes detailed enough to sort later. A note saying 'jank' can't be bucketed by an AI or by you later. 'Did not confirm call transfer with user' can.
The note does not have to be perfect, only give a general idea of what went wrong.
She turns notes into ranked failure modes by counting. Export notes to a CSV, have Claude or ChatGPT propose axial codes, rename them to be specific, then map every note to one and build a pivot table.
On the Nurture Boss data the pivot table showed 17 conversational flow issues. A 'none of the above' category exposes gaps in the categories. Counts guide the choice but severity matters too, since a rarer bad handoff may outweigh a frequent failure.
3Shankar's rules for building LLM judges
She fixes obvious problems directly and builds judges only for the stubborn ones. Many failures are plain prompt or engineering gaps, like never specifying an output format. Shankar typically ends up with between four and seven judges.
Evals have a cost-benefit tradeoff. Jumping straight to writing tests before grounding in real errors is where people go off the rails.
Shankar writes code-based checks first when a failure allows it. Valid JSON, markdown or output length can be tested in code, which is cheaper and needs no evaluation of the checker.
Judges are for subjective failures like whether to hand a user to a human. A judge is a meta eval, so the judge itself must be evaluated.
She scopes each judge to one failure with a pass or fail verdict. Husain's example judge targets only missed human handoff and returns true or false. A 1 to 5 score avoids deciding, and nobody knows what 3.2 versus 3.7 means.
She sees expert-written judge prompts online using a 1 to 7 scale, with people reporting averages like 4.2, and she has to correct the misinformation. She agrees with eval critics on this one point.
She will not trust a judge until it is checked against human labels. Compare human and judge labels in a matrix and iterate on the prompt until the two off-diagonal error types shrink. If an error occurs 10% of the time, a judge that always says pass scores 90% agreement.
The human labels already exist from the earlier coding. Split the data so the judge isn't graded on examples it was tuned on. Some misalignment is fine and can be corrected statistically, but reporting only a bare agreement figure is a bad smell.
She reuses validated judges in tests and production monitoring. Labeled failure traces become unit tests that run on every code push. The same judge can sample 1,000 production traces a day and put the failure rate on a dashboard.
Teams that do this well rarely share it, because the monitoring is their moat. Her advice is to use these artifacts repeatedly to drive product improvements.
Shankar treats a judge prompt as a PRD that runs all the time. It states how the agent should respond in specific cases and checks it constantly, where a manual test checks once. It is built from the team's own data, not only first expectations.
Requirements can't be fixed up front, so she still writes a one-pager and improves it from open codes.
4What Shankar says people get wrong about evals
Shankar says no tool or AI can run evals for you. The top misconception is that you can plug in a tool or have AI eval itself. Generic scores like cosine similarity or a hallucination score are only a weak first stab.
Vendors sell it because people want it, but it does not work, so a human doing application-specific error analysis is still needed.
She argues 'just vibes' teams are quietly doing error analysis. Shankar bets they monitor users, chat counts and chat lengths, and route internal complaints to the developer, who is implicitly doing the analysis.
Coding agents like Claude Code work that way because the developer is the domain expert. Husain warns against generalizing from them, and Shankar notes most teams lack the skill and intensity to dogfood at a level that closes the loop.
She wants A/B tests grounded in error analysis first. An A/B test is an eval, but many teams test hypothetical requirements before looking at data. Real errors turned out different, such as handoff issues and text-message garbling.
'If you're going to do AB tests and they are powered by actual error analysis then that's great.'
Shankar budgets about a week up front, then 30 minutes a week. She spends three to four days with the domain expert on the first round of error analysis and labeling, then builds the spreadsheet and a few LLM judges. After that a weekly cron job repeats the checks.
She often spends more time on data by choice, because she is 'data hungry', but says she doesn't have to. People wrongly assume the upfront cost never ends.
She says remove all friction from looking at data. Looking at data is the highest ROI activity, so build a simple internal annotation app with AI. Nurture Boss built theirs in a few hours.
It showed channels, threads, a hidden system prompt and automated axial-code counts.
Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
A step-by-step method for building AI evals: read real traces, write open codes, group them into failure modes, count them, then build narrow binary LLM judges validated against human labels. The method has been taught to over 2,000 PMs and engineers, takes about a week up front and then roughly 30 minutes per week.
Shankar co-teaches the top-ranked evals course on Maven with Hamel Husain. Husain and Shankar have taught over 2,000 PMs and engineers across 500 companies. Students include large parts of the OpenAI and Anthropic teams.
The course is also the highest grossing on Maven. It comes with a 60-page written guide and an AI assistant trained on every lesson, office hour, Discord chat, blog and paper the two have produced.
She researches how builders' standards shift as they review AI outputs. Her 'Who Validates the Validator' study, started in late 2023, found that people's views of good and bad change as they see more outputs. She calls this criteria drift.
Even experienced LLM builders only thought of some failure modes after about 10 outputs. This is the basis for treating judge prompts as living requirements documents.
She distrusts anyone selling an eval technique as entirely new. Open and axial coding come from social science. Husain also points to an eight-year-old Andrew Ng video on error analysis.
What the two added is a curriculum adapted to LLM products. Shankar treats the age of the method as a reason to trust it.
Her stated aim is that everyone can build AI products, not riches. She wants more people teaching evals and asks for real case studies of what people implemented, which keep her and Husain going.
She says her vision is people on the same page about building AI products, not the two becoming billionaires.
Shankar starts with about 100 real traces and a quick note on each. Open the observability tool, sample traces from the real application and write a free-form note on what went wrong. Do this before writing any tests.
The number 100 is arbitrary and only makes the task feel doable. On the Nurture Boss leasing assistant, a few traces surfaced a missed human handoff, garbled text-message flow and a hallucinated virtual tour.
She notes only the most upstream error in each trace. Write down the first thing that is wrong and move on, rather than cataloguing every problem. The first two or three notes are painful, then the pace picks up.
Shankar insists humans do the first pass, not an LLM. Handing open coding to an LLM is the number one pitfall. She would bet money that ChatGPT, asked whether the virtual-tour trace has an error, says no.
A domain expert knows the product has no virtual tours, so offering one is a hallucination. The LLM lacks that context. LLMs come in later, for synthesis.
She wants one trusted domain expert leading, not a committee. Husain's 'benevolent dictator' is one person whose taste you trust: a lawyer for legal, a psychiatrist for mental health, often the product manager.
Committees bog down chasing agreement, and the process can become infinitely expensive if you aren't careful.
Shankar stops reading traces at theoretical saturation, not a set count. Keep going until new traces stop changing what you would do next. Some apps need only 40, 60 or even 15, and intuition comes after two or three rounds.
The right number depends on the application and on how practiced the analyst is.
She asks for notes detailed enough to sort later. A note saying 'jank' can't be bucketed by an AI or by you later. 'Did not confirm call transfer with user' can.
The note does not have to be perfect, only give a general idea of what went wrong.
She turns notes into ranked failure modes by counting. Export notes to a CSV, have Claude or ChatGPT propose axial codes, rename them to be specific, then map every note to one and build a pivot table.
On the Nurture Boss data the pivot table showed 17 conversational flow issues. A 'none of the above' category exposes gaps in the categories. Counts guide the choice but severity matters too, since a rarer bad handoff may outweigh a frequent failure.
She fixes obvious problems directly and builds judges only for the stubborn ones. Many failures are plain prompt or engineering gaps, like never specifying an output format. Shankar typically ends up with between four and seven judges.
Evals have a cost-benefit tradeoff. Jumping straight to writing tests before grounding in real errors is where people go off the rails.
Shankar writes code-based checks first when a failure allows it. Valid JSON, markdown or output length can be tested in code, which is cheaper and needs no evaluation of the checker.
Judges are for subjective failures like whether to hand a user to a human. A judge is a meta eval, so the judge itself must be evaluated.
She scopes each judge to one failure with a pass or fail verdict. Husain's example judge targets only missed human handoff and returns true or false. A 1 to 5 score avoids deciding, and nobody knows what 3.2 versus 3.7 means.
She sees expert-written judge prompts online using a 1 to 7 scale, with people reporting averages like 4.2, and she has to correct the misinformation. She agrees with eval critics on this one point.
She will not trust a judge until it is checked against human labels. Compare human and judge labels in a matrix and iterate on the prompt until the two off-diagonal error types shrink. If an error occurs 10% of the time, a judge that always says pass scores 90% agreement.
The human labels already exist from the earlier coding. Split the data so the judge isn't graded on examples it was tuned on. Some misalignment is fine and can be corrected statistically, but reporting only a bare agreement figure is a bad smell.
She reuses validated judges in tests and production monitoring. Labeled failure traces become unit tests that run on every code push. The same judge can sample 1,000 production traces a day and put the failure rate on a dashboard.
Teams that do this well rarely share it, because the monitoring is their moat. Her advice is to use these artifacts repeatedly to drive product improvements.
Shankar treats a judge prompt as a PRD that runs all the time. It states how the agent should respond in specific cases and checks it constantly, where a manual test checks once. It is built from the team's own data, not only first expectations.
Requirements can't be fixed up front, so she still writes a one-pager and improves it from open codes.
Shankar says no tool or AI can run evals for you. The top misconception is that you can plug in a tool or have AI eval itself. Generic scores like cosine similarity or a hallucination score are only a weak first stab.
Vendors sell it because people want it, but it does not work, so a human doing application-specific error analysis is still needed.
She argues 'just vibes' teams are quietly doing error analysis. Shankar bets they monitor users, chat counts and chat lengths, and route internal complaints to the developer, who is implicitly doing the analysis.
Coding agents like Claude Code work that way because the developer is the domain expert. Husain warns against generalizing from them, and Shankar notes most teams lack the skill and intensity to dogfood at a level that closes the loop.
She wants A/B tests grounded in error analysis first. An A/B test is an eval, but many teams test hypothetical requirements before looking at data. Real errors turned out different, such as handoff issues and text-message garbling.
'If you're going to do AB tests and they are powered by actual error analysis then that's great.'
Shankar budgets about a week up front, then 30 minutes a week. She spends three to four days with the domain expert on the first round of error analysis and labeling, then builds the spreadsheet and a few LLM judges. After that a weekly cron job repeats the checks.
She often spends more time on data by choice, because she is 'data hungry', but says she doesn't have to. People wrongly assume the upfront cost never ends.
She says remove all friction from looking at data. Looking at data is the highest ROI activity, so build a simple internal annotation app with AI. Nurture Boss built theirs in a few hours.
It showed channels, threads, a hidden system prompt and automated axial-code counts.