Aishwarya Naresh Reganti
Former Alexa and Microsoft AI researcher
Where to hear and read about Aishwarya Naresh Reganti1 episode or test
Lenny's PodcastWhy most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google & AmazonAishwarya Naresh Reganti is a former AI researcher at Alexa and Microsoft who, with Kiriti Badam, has helped ship over 50 enterprise AI products. The one thing most worth learning from her: start AI products with low autonomy and high human control, then raise autonomy in stages as real user behavior earns trust.
Building is really cheap today. Design is more expensive.
Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google & Amazon
Aishwarya Naresh Reganti: the story and the lessons
1Aishwarya Reganti's background and why AI products differ
Reganti moved from AI research to hands-on enterprise deployments. She was an early AI researcher at Alexa and Microsoft and has published over 35 research papers. With Kiriti Badam she has shipped over 50 enterprise AI products across Amazon, Databricks, OpenAI and Google, in startups and large enterprises.
Their advice comes from deployments, not theory. Badam works on Codex at OpenAI and spent a decade on AI and ML infrastructure at Google and Kumo.
She says AI products break because both input and output are unpredictable. Users can phrase intent in endless ways, and LLMs react differently to small wording changes. Traditional software, like a booking.com flow, maps intent to a fixed path of buttons and forms, and AI replaces that path with a fluid natural-language layer.
You build by anticipating behavior, not by specifying it.
She treats agency and control as a trade-off. Each decision handed to an agent means giving up some control, so the agent must earn that trust. A Berkeley and Databricks paper found about 74 to 75% of enterprises named reliability as their biggest problem.
That worry is why many AI products today are low-autonomy productivity tools.
She argues the winning move is a flywheel, not the first agent. Being first with an agent no longer matters. What matters is having the right flywheels to keep improving over time.
2How Reganti builds autonomy in stages
Start with low agency and high human control. In customer support, V1 routes tickets, V2 is a copilot that drafts replies for humans, and V3 resolves tickets end to end. Coding and marketing assistants follow the same ladder, from suggestions to reviewed blocks to autonomous action.
She compares it to training for a hike: nobody climbs Half Dome on day one. Starting with everything at once makes the complexity hard to control and the failures impossible to tabulate.
Logging what humans do gives error analysis for free. In the copilot stage, the team records how much of each AI draft the human used or changed. That log shows blind spots and feeds the flywheel.
Metrics only catch errors you already know about, while this log reveals patterns that appear only in production.
A narrow first version exposes data problems early. In a support build, V1 only routed tickets, and humans could undo wrong routing. Even that was hard because one retail taxonomy listed shoes, women's shoes and men's shoes at the same level, plus a node last updated in 2019.
A human spots a dead node, but an agent needs that undocumented context handed to it.
She writes expected inputs and outputs down before building. The starter dataset of expected behavior comes from PMs and subject matter experts. Building it often shows the team disagrees on how the product should behave.
The dataset is not comprehensive. It only gets the team started.
She runs a continuous calibration, continuous development loop. The development half is scope the capability, curate data, set up the app, design evaluation metrics, deploy and run them. The calibration half analyzes unexpected behavior, spots error patterns and fixes them, adding a metric only when needed.
It was built after one end-to-end support agent was shut down under too many hot fixes. Air Canada's agent hallucinated a refund policy and the company had to honor it.
Move up a stage when calibration stops surprising you. If a check every day or two shows no new data patterns, little information is left to gain. She admits it comes down to "the vibes" of knowing you are ready.
Recalibrate after a model swap such as GPT-4o to GPT-5, or when users get bolder. A bank's underwriting assistant worked for about three months, then underwriters began asking what previous underwriters did in similar cases, which forced a rebuild.
3Reganti on evals and production monitoring
She says evals and monitoring catch different failures. Evals encode what you already know must never happen, such as an agent offering refunds. Production monitoring uses explicit and implicit signals, like a user regenerating an answer, to show which traces to read.
When monitoring reveals a failure you care about, it becomes a new eval. Even a quick check of 10 must-not-fail questions counts as an evaluation dataset.
She wants the word "eval" defined before anyone argues about it. The term now covers expert notes, PM-written evals, LLM judges and benchmarks. One client said they did evals but had only checked LM Arena and Artificial Analysis, which benchmark models, not their own data.
She says "evaluation metrics" on purpose, since evaluation is a process and the metrics are its dimensions. Her advice: "Don't be obsessed with prescriptions. They're going to change."
Not every error needs a new metric. A badly defined tool that causes a tool-calling error can simply be fixed and left alone. For complex products, an LLM judge can miss newly emerging patterns, so reading user signals and checking for regressions is often simpler.
Badam's Codex team shows the pairing in practice. Codex uses core evals, A/B tests, customer feedback and hands-on testing of each new model, because a customizable coding agent has no fixed set of uses to test. Code review users annoyed by wrong comments may switch the product off.
The team also watches social media for problems and fixes them quickly.
4What Reganti says separates teams that ship working AI
Leaders must get hands-on to rebuild their intuitions. She says AI adoption cannot work bottom up when leaders doubt the technology. The Rackspace CEO blocked 4 to 6 a.m. daily with no meetings to catch up on AI, kept two or three trusted sources and did weekend vibe coding.
The stance she names: "You probably are the dumbest person in the room and you want to learn from everyone."
Frame AI as a 10x tool so experts keep helping. Subject matter experts are needed to define ideal AI behavior, but fear of replacement makes them refuse to talk. The empowering tone has to come from the leader.
She distrusts one-click agents. Enterprise data and systems hide messy taxonomies, duplicate functions and tech debt. Meaningful ROI takes four to six months even with the best data and infrastructure layers.
"If someone's selling you one click agents, it's pure marketing."
She prefers a supervisor with sub-agents over peer-to-peer agents. Agents splitting duties by function and talking in a gossip-style protocol do not work with current models. In customer support, peer-to-peer agents make it hard to control which one replies, and guardrails must go everywhere.
A supervisor directing sub-agents is the pattern that works.
She puts design and taste above building as building gets cheap. She argues implementation will be ridiculously cheap within a few years, so the work shifts to deciding what is worth building. A new hire replaced a task-tracking app the team paid a high fee for with his own vibe-coded one, though she notes the maintenance cost.
The question that matters is whether the product solves a real pain point.
She sees persistence through the pain as the real moat. Anyone can learn anything overnight, but value comes from implementing it and finding what works. Companies in a new area win by learning the non-negotiables, since there is no textbook for it.
Her father's line is her motto: "They told it couldn't be done, but the fool didn't know it, so he did it anyway."
Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google & Amazon
AI products differ from regular software because they are non-deterministic and force a trade-off between agency and control, so they should be built in stages that start with high human control. Drawn from over 50 enterprise AI deployments, the episode gives a continuous calibration, continuous development loop, advice on evals versus production monitoring, and patterns that separate teams that ship working AI from those that hot-fix failing agents.
Reganti moved from AI research to hands-on enterprise deployments. She was an early AI researcher at Alexa and Microsoft and has published over 35 research papers. With Kiriti Badam she has shipped over 50 enterprise AI products across Amazon, Databricks, OpenAI and Google, in startups and large enterprises.
Their advice comes from deployments, not theory. Badam works on Codex at OpenAI and spent a decade on AI and ML infrastructure at Google and Kumo.
She says AI products break because both input and output are unpredictable. Users can phrase intent in endless ways, and LLMs react differently to small wording changes. Traditional software, like a booking.com flow, maps intent to a fixed path of buttons and forms, and AI replaces that path with a fluid natural-language layer.
You build by anticipating behavior, not by specifying it.
She treats agency and control as a trade-off. Each decision handed to an agent means giving up some control, so the agent must earn that trust. A Berkeley and Databricks paper found about 74 to 75% of enterprises named reliability as their biggest problem.
That worry is why many AI products today are low-autonomy productivity tools.
She argues the winning move is a flywheel, not the first agent. Being first with an agent no longer matters. What matters is having the right flywheels to keep improving over time.
Start with low agency and high human control. In customer support, V1 routes tickets, V2 is a copilot that drafts replies for humans, and V3 resolves tickets end to end. Coding and marketing assistants follow the same ladder, from suggestions to reviewed blocks to autonomous action.
She compares it to training for a hike: nobody climbs Half Dome on day one. Starting with everything at once makes the complexity hard to control and the failures impossible to tabulate.
Logging what humans do gives error analysis for free. In the copilot stage, the team records how much of each AI draft the human used or changed. That log shows blind spots and feeds the flywheel.
Metrics only catch errors you already know about, while this log reveals patterns that appear only in production.
A narrow first version exposes data problems early. In a support build, V1 only routed tickets, and humans could undo wrong routing. Even that was hard because one retail taxonomy listed shoes, women's shoes and men's shoes at the same level, plus a node last updated in 2019.
A human spots a dead node, but an agent needs that undocumented context handed to it.
She writes expected inputs and outputs down before building. The starter dataset of expected behavior comes from PMs and subject matter experts. Building it often shows the team disagrees on how the product should behave.
The dataset is not comprehensive. It only gets the team started.
She runs a continuous calibration, continuous development loop. The development half is scope the capability, curate data, set up the app, design evaluation metrics, deploy and run them. The calibration half analyzes unexpected behavior, spots error patterns and fixes them, adding a metric only when needed.
It was built after one end-to-end support agent was shut down under too many hot fixes. Air Canada's agent hallucinated a refund policy and the company had to honor it.
Move up a stage when calibration stops surprising you. If a check every day or two shows no new data patterns, little information is left to gain. She admits it comes down to "the vibes" of knowing you are ready.
Recalibrate after a model swap such as GPT-4o to GPT-5, or when users get bolder. A bank's underwriting assistant worked for about three months, then underwriters began asking what previous underwriters did in similar cases, which forced a rebuild.
She says evals and monitoring catch different failures. Evals encode what you already know must never happen, such as an agent offering refunds. Production monitoring uses explicit and implicit signals, like a user regenerating an answer, to show which traces to read.
When monitoring reveals a failure you care about, it becomes a new eval. Even a quick check of 10 must-not-fail questions counts as an evaluation dataset.
She wants the word "eval" defined before anyone argues about it. The term now covers expert notes, PM-written evals, LLM judges and benchmarks. One client said they did evals but had only checked LM Arena and Artificial Analysis, which benchmark models, not their own data.
She says "evaluation metrics" on purpose, since evaluation is a process and the metrics are its dimensions. Her advice: "Don't be obsessed with prescriptions. They're going to change."
Not every error needs a new metric. A badly defined tool that causes a tool-calling error can simply be fixed and left alone. For complex products, an LLM judge can miss newly emerging patterns, so reading user signals and checking for regressions is often simpler.
Badam's Codex team shows the pairing in practice. Codex uses core evals, A/B tests, customer feedback and hands-on testing of each new model, because a customizable coding agent has no fixed set of uses to test. Code review users annoyed by wrong comments may switch the product off.
The team also watches social media for problems and fixes them quickly.
Leaders must get hands-on to rebuild their intuitions. She says AI adoption cannot work bottom up when leaders doubt the technology. The Rackspace CEO blocked 4 to 6 a.m. daily with no meetings to catch up on AI, kept two or three trusted sources and did weekend vibe coding.
The stance she names: "You probably are the dumbest person in the room and you want to learn from everyone."
Frame AI as a 10x tool so experts keep helping. Subject matter experts are needed to define ideal AI behavior, but fear of replacement makes them refuse to talk. The empowering tone has to come from the leader.
She distrusts one-click agents. Enterprise data and systems hide messy taxonomies, duplicate functions and tech debt. Meaningful ROI takes four to six months even with the best data and infrastructure layers.
"If someone's selling you one click agents, it's pure marketing."
She prefers a supervisor with sub-agents over peer-to-peer agents. Agents splitting duties by function and talking in a gossip-style protocol do not work with current models. In customer support, peer-to-peer agents make it hard to control which one replies, and guardrails must go everywhere.
A supervisor directing sub-agents is the pattern that works.
She puts design and taste above building as building gets cheap. She argues implementation will be ridiculously cheap within a few years, so the work shifts to deciding what is worth building. A new hire replaced a task-tracking app the team paid a high fee for with his own vibe-coded one, though she notes the maintenance cost.
The question that matters is whether the product solves a real pain point.
She sees persistence through the pain as the real moat. Anyone can learn anything overnight, but value comes from implementing it and finding what works. Companies in a new area win by learning the non-negotiables, since there is no textbook for it.
Her father's line is her motto: "They told it couldn't be done, but the fool didn't know it, so he did it anyway."