Lenny's Podcast · Sep 25Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
Its leasing assistant traces were the demo dataset: errors included missed human handoffs, garbled text-message flow and a hallucinated virtual tour. A pivot table showed 17 conversational flow issues, and the team built a custom annotation tool in a few hours.
Tips
- Read around 100 real traces and write a note on each▶ 19:41On the Nurture Boss leasing assistant, a few traces surfaced a missed human handoff, garbled text-message flow, and a hallucinated virtual tour.
- Cluster open codes into axial codes with an LLM, then count them▶ 32:20On the Nurture Boss data, the pivot table showed 17 conversational flow issues, alongside human handoff and formatting problems.
- Build a custom data annotation tool to remove review friction▶ 1:29:30The Nurture Boss team built it within a few hours.
