Hamel Husain
AI eval consultant and instructor
Where to hear and read about Hamel Husain1 episode or test
Hamel Husain is an AI evals consultant and instructor who, with Shreya Shankar, has taught a method for evaluating AI products to over 2,000 PMs and engineers across 500 companies. The thing most worth learning from him is to read real traces yourself, name and count the failures, and only then automate checks for the stubborn ones.
You don't really know what you want until you see it with these LLMs.
Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
Hamel Husain: the story and the lessons
1Who Hamel Husain is and what he teaches
Husain teaches evals as data analysis, not a new discipline. With Shreya Shankar he has taught over 2,000 PMs and engineers across 500 companies. Their Maven course is the top ranked one on the platform, and students include large parts of the OpenAI and Anthropic teams.
Husain says the word "eval" gets momentum that "error analysis" would not. He sees evals and A/B testing as the same data science applied to a new kind of product.
He consults with clients on reading their own traces. Clients are surprised that his first step is to open individual traces. Husain says they learn a lot and find the problem 100% of the time.
One client sends recruiting emails. The first trace opened with "given your background", and Husain said he deletes such messages. The client was proud the AI used the right name and link, and agreed only after being pushed that the email was generic.
Husain is baffled that so few people teach this approach. He and Shankar are almost alone in teaching a structured method for application-specific evals. He hopes tools adopt more data science thinking.
His course goes beyond the basics into automated evaluators, an improvement flywheel and building your own error analysis interfaces. It comes with a 60-page guide and an AI assistant trained on their lessons, office hours, chats, blogs and papers.
Husain's personal motto is to think like a beginner. He says "Keep learning and think like a beginner." He favors textbooks, and Mitchell's Machine Learning taught him that Occam's razor applies to AI.
He names Claude Code as a favorite product and praises its UX as a polished terminal app. He says it is not just built on vibes.
2Husain's error analysis method
Husain starts by reading about 100 real traces with a note on each. Open the observability tool, sample traces from the live application and write a quick free-form note on the first thing that went wrong. Do this before writing any tests, because the real errors are rarely the ones you imagined.
Any tool works. He demoed it in Braintrust on Nurture Boss, an AI assistant for apartment property managers, where a few traces showed a missed human handoff, garbled text-message flow and a hallucinated virtual tour.
He records only the most upstream error in each trace. Write down the first thing that is wrong and move on. The first two or three notes are painful, then the pace picks up.
Notes should say what happened, such as "did not confirm call transfer with user". A note that just says "jank" cannot be sorted later.
He keeps open coding human because LLMs miss product context. Husain and Shankar call handing open coding to an LLM the number one pitfall. An LLM asked about the virtual tour trace would say it looks fine, because it does not know the product has no virtual tours.
Shankar would bet money on that. LLMs fit later steps such as clustering notes, not the first read.
Husain coined "benevolent dictator" for the one expert who leads review. Pick one domain expert whose taste you trust instead of a committee. That is a lawyer for legal, a psychiatrist for mental health, and often the product manager.
Committees get bogged down chasing agreement. The aim is to keep the process cheap enough that you will actually do it.
He stops reviewing when new traces stop changing what he would do. The 100 trace target is arbitrary and only makes the task feel doable. The formal name is theoretical saturation, and some apps need only 15, 40 or 60.
Intuition for when to stop comes after two or three rounds.
Husain uses an LLM to cluster notes, then counts them. He uploaded a CSV of notes to a Claude project and asked for axial codes. He then rewrote the codes to be specific: tour scheduling, human handoff, formatting, conversational flow and broken follow-up promises.
A Sheets AI formula mapped each note to one category, and a "none of the above" bucket exposes gaps. The pivot table showed 17 conversational flow issues. Counts guide priority but do not decide it, since a rarer failure like a bad handoff may matter more.
Husain trusts the method because it is old. Open and axial coding come from social science. He showed an eight-year-old Andrew Ng video on error analysis.
What he and Shankar added is a curriculum adapted to LLM products.
3How Husain builds and validates LLM judges
Husain reserves automated evals for the pesky failures. Many failures are prompt or engineering gaps, like never specifying output format, so just fix them. Four to seven judges is typical.
He says evals have a cost-benefit tradeoff. Jumping to tests before grounding in real errors is where people go off the rails.
He chooses code checks first and a judge only when needed. Plain code can check valid JSON, markdown or output length, and it is cheaper with no meta-evaluation. A judge fits subjective failures such as when to hand off to a human.
A judge is a meta eval, so the judge itself needs evaluating.
Husain scopes each judge to one failure with pass or fail. His example judge targets only missed human handoff and returns true or false. It lists when a handoff is required: an ignored human request, policy-mandated transfers, sensitive resident issues, missing tool data, and same-day tour requests.
He avoids 1 to 5 scales because "no one knows what 3.2 versus 3.7 means." He calls his prompt deliberately simple, not perfect, and says to write it yourself even if an LLM drafts it.
He checks a judge against human labels before trusting it. Compare human and judge labels in a matrix and iterate until the off-diagonal disagreements shrink. A judge that always says pass scores 90% agreement when the error occurs 10% of the time.
The human labels already exist from open coding. Split the data so the judge is not graded on examples it was tuned on. In his words, once people lose trust in the evals, they lose trust in you.
Husain treats a judge prompt as a PRD that runs constantly. It states how the agent should respond in specific cases and checks it on every run. It has to come from data, because "you don't really know what you want until you see it."
Shankar's criteria drift study found even experienced builders thought of some failure modes only after about 10 outputs. Husain says one-pagers are still worth writing.
He reuses validated judges in CI and live monitoring. Labeled failure traces become unit tests on every code push. The same judge can sample 1,000 production traces a day and put the failure rate on a dashboard.
Teams that do this well rarely share it, because the monitoring is their moat.
4Why Husain says vibes and generic tools fall short
He accepts vibe checks early but not as a permanent method. Without evals, a builder fixes a prompt and hopes nothing else breaks. Vibe checks become unmanageable as the application grows.
Husain says to do them at the start. The problem is relying on them forever.
Husain warns against generalizing from coding agents. Coding agents were the first AI product released widely, and he calls it a mistake to copy their approach. The developer is the domain expert and uses the product all day, so less data and feedback are needed.
Doctors do not dogfood an AI by hunting for its wrong medical advice. Teams that claim "just vibes" still do error analysis in practice.
He wants A/B tests powered by real error analysis. Many teams test hypothetical requirements before looking at data. Real errors, such as handoff issues and garbled text messages, turned out different from what was expected.
He agrees an A/B test is an eval. It just needs to test failures found in real data.
Husain says no tool can run evals for him. The top misconception is that AI can eval itself with a plugged-in tool. Generic scores like cosine similarity or hallucination score are only a first stab.
Vendors sell it because people want it. Humans are still needed, and error analysis is always part of the plan.
He notes opinions on evals swing with the news cycle. People cited Claude Code as proof that vibes beat evals. When Codex was called better, many of the same people turned pro-evals.
A product's success says little about whether evals matter.
5What Husain says evals cost in practice
Husain budgets about a week up front, then 30 minutes weekly. Shankar spends three to four days with the domain expert on the first error analysis, labeling and a few judges. After that a weekly cron job repeats the checks.
Husain says people get overwhelmed by the upfront cost and wrongly assume it never ends.
He removes friction from looking at data with custom tools. AI makes simple internal web apps cheap. Nurture Boss built theirs in a few hours, showing channels, threads, a hidden system prompt and axial code counts.
He calls looking at data the highest ROI activity you can engage in.
He uses evals to cut model costs without losing quality. Some teams have strong evals and a strong product but a high bill from state-of-the-art models. The fix is swapping some calls from the most expensive GPT-5 models to GPT-5 nano or 4 mini.
Evals show whether quality holds after the swap.
Husain accepts an imperfect process that still improves the product. He says there is no one correct way to do evals but many incorrect ones, and the plan depends on resources and product stage.
If an eval turns up something wrong and obvious, fix the application. A passing eval suite is not the goal.
Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
A step-by-step method for building AI evals: read real traces, write open codes, group them into failure modes, count them, then build narrow binary LLM judges validated against human labels. The method has been taught to over 2,000 PMs and engineers, takes about a week up front and then roughly 30 minutes per week.
Husain teaches evals as data analysis, not a new discipline. With Shreya Shankar he has taught over 2,000 PMs and engineers across 500 companies. Their Maven course is the top ranked one on the platform, and students include large parts of the OpenAI and Anthropic teams.
Husain says the word "eval" gets momentum that "error analysis" would not. He sees evals and A/B testing as the same data science applied to a new kind of product.
He consults with clients on reading their own traces. Clients are surprised that his first step is to open individual traces. Husain says they learn a lot and find the problem 100% of the time.
One client sends recruiting emails. The first trace opened with "given your background", and Husain said he deletes such messages. The client was proud the AI used the right name and link, and agreed only after being pushed that the email was generic.
Husain is baffled that so few people teach this approach. He and Shankar are almost alone in teaching a structured method for application-specific evals. He hopes tools adopt more data science thinking.
His course goes beyond the basics into automated evaluators, an improvement flywheel and building your own error analysis interfaces. It comes with a 60-page guide and an AI assistant trained on their lessons, office hours, chats, blogs and papers.
Husain's personal motto is to think like a beginner. He says "Keep learning and think like a beginner." He favors textbooks, and Mitchell's Machine Learning taught him that Occam's razor applies to AI.
He names Claude Code as a favorite product and praises its UX as a polished terminal app. He says it is not just built on vibes.
Husain starts by reading about 100 real traces with a note on each. Open the observability tool, sample traces from the live application and write a quick free-form note on the first thing that went wrong. Do this before writing any tests, because the real errors are rarely the ones you imagined.
Any tool works. He demoed it in Braintrust on Nurture Boss, an AI assistant for apartment property managers, where a few traces showed a missed human handoff, garbled text-message flow and a hallucinated virtual tour.
He records only the most upstream error in each trace. Write down the first thing that is wrong and move on. The first two or three notes are painful, then the pace picks up.
Notes should say what happened, such as "did not confirm call transfer with user". A note that just says "jank" cannot be sorted later.
He keeps open coding human because LLMs miss product context. Husain and Shankar call handing open coding to an LLM the number one pitfall. An LLM asked about the virtual tour trace would say it looks fine, because it does not know the product has no virtual tours.
Shankar would bet money on that. LLMs fit later steps such as clustering notes, not the first read.
Husain coined "benevolent dictator" for the one expert who leads review. Pick one domain expert whose taste you trust instead of a committee. That is a lawyer for legal, a psychiatrist for mental health, and often the product manager.
Committees get bogged down chasing agreement. The aim is to keep the process cheap enough that you will actually do it.
He stops reviewing when new traces stop changing what he would do. The 100 trace target is arbitrary and only makes the task feel doable. The formal name is theoretical saturation, and some apps need only 15, 40 or 60.
Intuition for when to stop comes after two or three rounds.
Husain uses an LLM to cluster notes, then counts them. He uploaded a CSV of notes to a Claude project and asked for axial codes. He then rewrote the codes to be specific: tour scheduling, human handoff, formatting, conversational flow and broken follow-up promises.
A Sheets AI formula mapped each note to one category, and a "none of the above" bucket exposes gaps. The pivot table showed 17 conversational flow issues. Counts guide priority but do not decide it, since a rarer failure like a bad handoff may matter more.
Husain trusts the method because it is old. Open and axial coding come from social science. He showed an eight-year-old Andrew Ng video on error analysis.
What he and Shankar added is a curriculum adapted to LLM products.
Husain reserves automated evals for the pesky failures. Many failures are prompt or engineering gaps, like never specifying output format, so just fix them. Four to seven judges is typical.
He says evals have a cost-benefit tradeoff. Jumping to tests before grounding in real errors is where people go off the rails.
He chooses code checks first and a judge only when needed. Plain code can check valid JSON, markdown or output length, and it is cheaper with no meta-evaluation. A judge fits subjective failures such as when to hand off to a human.
A judge is a meta eval, so the judge itself needs evaluating.
Husain scopes each judge to one failure with pass or fail. His example judge targets only missed human handoff and returns true or false. It lists when a handoff is required: an ignored human request, policy-mandated transfers, sensitive resident issues, missing tool data, and same-day tour requests.
He avoids 1 to 5 scales because "no one knows what 3.2 versus 3.7 means." He calls his prompt deliberately simple, not perfect, and says to write it yourself even if an LLM drafts it.
He checks a judge against human labels before trusting it. Compare human and judge labels in a matrix and iterate until the off-diagonal disagreements shrink. A judge that always says pass scores 90% agreement when the error occurs 10% of the time.
The human labels already exist from open coding. Split the data so the judge is not graded on examples it was tuned on. In his words, once people lose trust in the evals, they lose trust in you.
Husain treats a judge prompt as a PRD that runs constantly. It states how the agent should respond in specific cases and checks it on every run. It has to come from data, because "you don't really know what you want until you see it."
Shankar's criteria drift study found even experienced builders thought of some failure modes only after about 10 outputs. Husain says one-pagers are still worth writing.
He reuses validated judges in CI and live monitoring. Labeled failure traces become unit tests on every code push. The same judge can sample 1,000 production traces a day and put the failure rate on a dashboard.
Teams that do this well rarely share it, because the monitoring is their moat.
He accepts vibe checks early but not as a permanent method. Without evals, a builder fixes a prompt and hopes nothing else breaks. Vibe checks become unmanageable as the application grows.
Husain says to do them at the start. The problem is relying on them forever.
Husain warns against generalizing from coding agents. Coding agents were the first AI product released widely, and he calls it a mistake to copy their approach. The developer is the domain expert and uses the product all day, so less data and feedback are needed.
Doctors do not dogfood an AI by hunting for its wrong medical advice. Teams that claim "just vibes" still do error analysis in practice.
He wants A/B tests powered by real error analysis. Many teams test hypothetical requirements before looking at data. Real errors, such as handoff issues and garbled text messages, turned out different from what was expected.
He agrees an A/B test is an eval. It just needs to test failures found in real data.
Husain says no tool can run evals for him. The top misconception is that AI can eval itself with a plugged-in tool. Generic scores like cosine similarity or hallucination score are only a first stab.
Vendors sell it because people want it. Humans are still needed, and error analysis is always part of the plan.
He notes opinions on evals swing with the news cycle. People cited Claude Code as proof that vibes beat evals. When Codex was called better, many of the same people turned pro-evals.
A product's success says little about whether evals matter.
Husain budgets about a week up front, then 30 minutes weekly. Shankar spends three to four days with the domain expert on the first error analysis, labeling and a few judges. After that a weekly cron job repeats the checks.
Husain says people get overwhelmed by the upfront cost and wrongly assume it never ends.
He removes friction from looking at data with custom tools. AI makes simple internal web apps cheap. Nurture Boss built theirs in a few hours, showing channels, threads, a hidden system prompt and axial code counts.
He calls looking at data the highest ROI activity you can engage in.
He uses evals to cut model costs without losing quality. Some teams have strong evals and a strong product but a high bill from state-of-the-art models. The fix is swapping some calls from the most expensive GPT-5 models to GPT-5 nano or 4 mini.
Evals show whether quality holds after the swap.
Husain accepts an imperfect process that still improves the product. He says there is no one correct way to do evals but many incorrect ones, and the plan depends on resources and product stage.
If an eval turns up something wrong and obvious, fix the application. A passing eval suite is not the goal.