Dianne Penn
Head of Product for AI Research and Labs, Anthropic
Where to hear and read about Dianne Penn1 episode or test
Dianne Penn is Anthropic's first technical PM and now Head of Product for AI Research and Labs, where she has shaped most model launches since Claude 2. The thing most worth learning from her is to turn vague user complaints into small, repeatable evals and to build products that still work when the next model arrives.
You have to sweat the tokens as much as you sweat the pixels.
Why AI is going vertical (again) | Dianne Penn (Anthropic)
Dianne Penn: the story and the lessons
1Dianne Penn's path as Anthropic's first technical PM
Penn joined as Anthropic's first technical PM when product was five engineers. She joined over three years ago and has helped ship every model from Claude 2 through Fable. She helped incubate Claude Code, MCP, skills and Claude Design, and worked on computer use, tool use and reasoning.
She sits where models and products meet, leading product for the research and Labs teams.
Golden Gate Claude showed Penn's team it could ship research as a product. It came from early 2024 interpretability work: dialing up one model feature made Claude steer every answer back to the Golden Gate Bridge. Engineering, product, design and research built it on Claude.ai within 24 hours.
It reached only about 2,000 people, yet Penn calls it a hidden inflection point. People donated their own time, which was typical of the bottoms-up culture.
Penn spotted long-form code early and pushed a small training change in Opus 3. In 2023 nobody put Anthropic, Claude and coding in one sentence. She saw people using models for long-form code rather than autocomplete and pushed to train Opus 3 to be better at it.
"It ended up being a relatively smaller change from a training perspective but it ended up helping us differentiate in the early days." It drew the first developer fans.
The Opus 3 crunch built research and product trust that still holds. Opus 3 launched in early March 2024, giving a company of under 200 people a reason for users to choose Claude. The final push ran around December, with the team working from their parents' homes.
Several research leads from that time now lead reinforcement learning, character work and alignment. Penn says that foundation helps product and research work well on every production model today.
Penn paired Opus 4.5 with Claude Code so each lifted the other. The team felt Claude Code's magic for many months before Opus 4.5. Then the model could run work end to end in an agent manner, and adoption took off.
"You need frontier products in order to have frontier models and for people to feel the magic of frontier models." Opus 4.5 landed during winter break and drove the Claude Code inflection.
2How Penn turns vague feedback into evals
Penn reads failed transcripts to name the exact failure. "Claude hallucinated" is too vague to give a researcher. She splits it into a tool-use failure, a search and synthesis failure (right document, wrong facts) or an alignment issue.
Each type maps to a different fix, so the classification decides who works on it.
Penn builds an eval from 30 to 40 examples of one failure. For Claude 2's 'bad at following instructions', about 80% of cases were wrong JSON. A 30 to 40 example set fixed it, and the eval now scores 100% or 99.9%.
Each example is a prompt and a response checked against a golden answer. Working evals go into a shared repository and run on every new Claude version.
Penn tests a candidate eval for reproducibility, consistency and size. Before building one she asks whether the problem reproduces, whether it is consistent and whether it is big. A good eval must also be on distribution, covering cases where Claude fails and cases where it should not fail.
Outputs are non-deterministic, so evals describe behavior broadly instead of demanding an exact match.
Penn treats evals as the new PRDs, but still writes PRDs. An eval shortens the distance to action for researchers and shows whether the next model improves. Every model still gets a PRD to align product surfaces, engineering, legal and safety.
PRDs also help explore ambiguous bets like computer use, where no clear user pain points existed. Evals work as shorthand only when the problem is well defined.
Penn asks what changes for users when Claude 8 arrives. Her team plans around the next model, not only the current one. Whatever they build today should still work in that world.
"Let's say Claude 8 comes around, what changes in what users do?" The follow-up is what that means for how the team builds today.
3How Penn runs Labs and plans for sudden jumps
Labs chases bets big enough to be worth 10x, 100x or 1000x. Claude Code, Skills, MCP and Claude Design all came out of Labs. The first question is whether there is a there there, and Ben pushes people toward the 10x and 100x version of an idea.
Prototypes teach the team something even when nothing ships.
Penn keeps Labs pods small and holds prototypes loosely. Pods are small, sometimes one engineer, because large teams on vague ideas slow down. Labs hires people who want zero-to-one work.
"You can be very strongly held opinion about the theme or the area and then more weakly held about the exact prototype." Turning off a bet is hard because owners act like founders, and some bets are revisited one to two model generations later.
Penn has teams experiment in public, not alone. Early on, nearly the whole company tested Claude in one Slack channel. Within about 10 requests someone often hit a new use case.
"Experimentation is not always necessarily an individual sport." She calls experimenting in public Anthropic's secret sauce.
Penn plans for capability jumps nobody can predict. The original scaling-law papers show smooth loss curves but discontinuous jumps in capability. Evals and red teaming help spot jumps, and the gap between what a model can do and what users know is a product overhang.
She asks teams to reason from first principles and change plans when new information arrives. A plan expected months later sometimes becomes possible now, which is how Co-work and skills came about.
Penn treats token spend as an input and experimentation as the goal. She responds to the argument for spending $100,000 a year on tokens by setting goals around experimentation instead. The best ideas come from time spent with each new research model.
She says it is very hard to set a perfect strategy without touching the technology when it moves this fast.
Penn's team built fallback experiences for blocked frontier models. Before the Fable models, Anthropic had no strong fallback experiences. It built fallback systems so users still get a good response from Opus 4 when a safeguard triggers.
The goal is to maximize benefit while minimizing severe downside risk. She leaves policy and export control to the teams that own them.
4How Penn leads, hires and avoids burnout
Penn ships with the technology even as a manager. She keeps owning one or two work streams whenever a model is in flight, which keeps her sense of how fast models are improving. Experienced hires get the same onboarding as early-career PMs: understand users, read consented feedback, talk to customers.
"If you're a manager, you have to be hands-on. You have to spend a portion of your time actually shipping."
Penn hires for first-principles thinking over pattern matching. Her hiring loop has stayed the same for three years. She looks for people who work out the user value for this group with this technology.
Habits from consumer or B2B SaaS can mislead. A PM proves ownership by finding the right feedback and evals, not only by writing a vision doc.
Penn hires low-ego people who care about the team's impact. Anthropic shipped four model series in all of 2024 and more than that in Q2 of this year alone. Decisions at that pace cannot be carried alone.
She asks whether a candidate cares about their own empire or the team's impact. The night before a launch, colleagues who are not the lead owner stay up to review the blog post and improve demos, and after PTO in June she did not return to 3x the work.
Trading taught Penn that the best idea wins regardless of rank. She was the most junior person and the only woman on the high-yield desk.
She now helps teammates of any level pursue good ideas and follow through.
5How Penn uses Claude and where she thinks humans stay valuable
Penn forms her own view first, then uses Claude as a sparring partner. Her thinking and writing are tied together, so she will not let Claude take over the thinking. For routine writing like monthly business reviews, she delegates fully and only verifies.
Her team is improving a skill so she can act only as reviewer and verifier.
Penn built a Claude skill from Crucial Conversations to coach herself. The skill checks whether she is pitching a conversation at the right level of detail and helps her prepare for hard talks. She sometimes ignores its suggestions, but it helps her build trust faster and be more direct.
She is sharing the practice with her managers.
Penn wants Claude to push back, even on pricing. She asked a research version of Opus whether the pricing of the next Claude version was right. She wants a co-worker who pushes back when her ideas are not fully formed.
"A thinking partner doesn't just agree with you. It should add to you."
Penn sees judgment, persistence and proactivity as the human edge. Choosing what to build stays a human call, because AI can build far more than any lab should. She calls judgment an accumulation of nuance and experience that these systems have had less of.
She nudges her four-year-old and eight-year-old toward curiosity, persistence and an opinionated inner voice rather than any AI skill.
Penn goes deep on one or two AI uses and pairs with others. She cannot keep pace with every prototype, so she picks one or two. She finds pairing with an excited colleague beats hunting for the perfect use case alone.
She calls this a product prioritization of your own time and energy.
Why AI is going vertical (again) | Dianne Penn (Anthropic)
Anthropic's first technical PM explains how her team turns vague user feedback into evals, incubates discontinuous bets in Labs, and builds products that stay useful as models improve. The episode shows how a team of five product engineers grew into a leader in coding and agents by spotting early use cases like long-form code and pairing frontier models with frontier products like Claude Code.
Penn joined as Anthropic's first technical PM when product was five engineers. She joined over three years ago and has helped ship every model from Claude 2 through Fable. She helped incubate Claude Code, MCP, skills and Claude Design, and worked on computer use, tool use and reasoning.
She sits where models and products meet, leading product for the research and Labs teams.
Golden Gate Claude showed Penn's team it could ship research as a product. It came from early 2024 interpretability work: dialing up one model feature made Claude steer every answer back to the Golden Gate Bridge. Engineering, product, design and research built it on Claude.ai within 24 hours.
It reached only about 2,000 people, yet Penn calls it a hidden inflection point. People donated their own time, which was typical of the bottoms-up culture.
Penn spotted long-form code early and pushed a small training change in Opus 3. In 2023 nobody put Anthropic, Claude and coding in one sentence. She saw people using models for long-form code rather than autocomplete and pushed to train Opus 3 to be better at it.
"It ended up being a relatively smaller change from a training perspective but it ended up helping us differentiate in the early days." It drew the first developer fans.
The Opus 3 crunch built research and product trust that still holds. Opus 3 launched in early March 2024, giving a company of under 200 people a reason for users to choose Claude. The final push ran around December, with the team working from their parents' homes.
Several research leads from that time now lead reinforcement learning, character work and alignment. Penn says that foundation helps product and research work well on every production model today.
Penn paired Opus 4.5 with Claude Code so each lifted the other. The team felt Claude Code's magic for many months before Opus 4.5. Then the model could run work end to end in an agent manner, and adoption took off.
"You need frontier products in order to have frontier models and for people to feel the magic of frontier models." Opus 4.5 landed during winter break and drove the Claude Code inflection.
Penn reads failed transcripts to name the exact failure. "Claude hallucinated" is too vague to give a researcher. She splits it into a tool-use failure, a search and synthesis failure (right document, wrong facts) or an alignment issue.
Each type maps to a different fix, so the classification decides who works on it.
Penn builds an eval from 30 to 40 examples of one failure. For Claude 2's 'bad at following instructions', about 80% of cases were wrong JSON. A 30 to 40 example set fixed it, and the eval now scores 100% or 99.9%.
Each example is a prompt and a response checked against a golden answer. Working evals go into a shared repository and run on every new Claude version.
Penn tests a candidate eval for reproducibility, consistency and size. Before building one she asks whether the problem reproduces, whether it is consistent and whether it is big. A good eval must also be on distribution, covering cases where Claude fails and cases where it should not fail.
Outputs are non-deterministic, so evals describe behavior broadly instead of demanding an exact match.
Penn treats evals as the new PRDs, but still writes PRDs. An eval shortens the distance to action for researchers and shows whether the next model improves. Every model still gets a PRD to align product surfaces, engineering, legal and safety.
PRDs also help explore ambiguous bets like computer use, where no clear user pain points existed. Evals work as shorthand only when the problem is well defined.
Penn asks what changes for users when Claude 8 arrives. Her team plans around the next model, not only the current one. Whatever they build today should still work in that world.
"Let's say Claude 8 comes around, what changes in what users do?" The follow-up is what that means for how the team builds today.
Labs chases bets big enough to be worth 10x, 100x or 1000x. Claude Code, Skills, MCP and Claude Design all came out of Labs. The first question is whether there is a there there, and Ben pushes people toward the 10x and 100x version of an idea.
Prototypes teach the team something even when nothing ships.
Penn keeps Labs pods small and holds prototypes loosely. Pods are small, sometimes one engineer, because large teams on vague ideas slow down. Labs hires people who want zero-to-one work.
"You can be very strongly held opinion about the theme or the area and then more weakly held about the exact prototype." Turning off a bet is hard because owners act like founders, and some bets are revisited one to two model generations later.
Penn has teams experiment in public, not alone. Early on, nearly the whole company tested Claude in one Slack channel. Within about 10 requests someone often hit a new use case.
"Experimentation is not always necessarily an individual sport." She calls experimenting in public Anthropic's secret sauce.
Penn plans for capability jumps nobody can predict. The original scaling-law papers show smooth loss curves but discontinuous jumps in capability. Evals and red teaming help spot jumps, and the gap between what a model can do and what users know is a product overhang.
She asks teams to reason from first principles and change plans when new information arrives. A plan expected months later sometimes becomes possible now, which is how Co-work and skills came about.
Penn treats token spend as an input and experimentation as the goal. She responds to the argument for spending $100,000 a year on tokens by setting goals around experimentation instead. The best ideas come from time spent with each new research model.
She says it is very hard to set a perfect strategy without touching the technology when it moves this fast.
Penn's team built fallback experiences for blocked frontier models. Before the Fable models, Anthropic had no strong fallback experiences. It built fallback systems so users still get a good response from Opus 4 when a safeguard triggers.
The goal is to maximize benefit while minimizing severe downside risk. She leaves policy and export control to the teams that own them.
Penn ships with the technology even as a manager. She keeps owning one or two work streams whenever a model is in flight, which keeps her sense of how fast models are improving. Experienced hires get the same onboarding as early-career PMs: understand users, read consented feedback, talk to customers.
"If you're a manager, you have to be hands-on. You have to spend a portion of your time actually shipping."
Penn hires for first-principles thinking over pattern matching. Her hiring loop has stayed the same for three years. She looks for people who work out the user value for this group with this technology.
Habits from consumer or B2B SaaS can mislead. A PM proves ownership by finding the right feedback and evals, not only by writing a vision doc.
Penn hires low-ego people who care about the team's impact. Anthropic shipped four model series in all of 2024 and more than that in Q2 of this year alone. Decisions at that pace cannot be carried alone.
She asks whether a candidate cares about their own empire or the team's impact. The night before a launch, colleagues who are not the lead owner stay up to review the blog post and improve demos, and after PTO in June she did not return to 3x the work.
Trading taught Penn that the best idea wins regardless of rank. She was the most junior person and the only woman on the high-yield desk.
She now helps teammates of any level pursue good ideas and follow through.
Penn forms her own view first, then uses Claude as a sparring partner. Her thinking and writing are tied together, so she will not let Claude take over the thinking. For routine writing like monthly business reviews, she delegates fully and only verifies.
Her team is improving a skill so she can act only as reviewer and verifier.
Penn built a Claude skill from Crucial Conversations to coach herself. The skill checks whether she is pitching a conversation at the right level of detail and helps her prepare for hard talks. She sometimes ignores its suggestions, but it helps her build trust faster and be more direct.
She is sharing the practice with her managers.
Penn wants Claude to push back, even on pricing. She asked a research version of Opus whether the pricing of the next Claude version was right. She wants a co-worker who pushes back when her ideas are not fully formed.
"A thinking partner doesn't just agree with you. It should add to you."
Penn sees judgment, persistence and proactivity as the human edge. Choosing what to build stays a human call, because AI can build far more than any lab should. She calls judgment an accumulation of nuance and experience that these systems have had less of.
She nudges her four-year-old and eight-year-old toward curiosity, persistence and an opinionated inner voice rather than any AI skill.
Penn goes deep on one or two AI uses and pairs with others. She cannot keep pace with every prototype, so she picks one or two. She finds pairing with an excited colleague beats hunting for the perfect use case alone.
She calls this a product prioritization of your own time and energy.
More from Anthropic
GuestMike Krieger
Anthropic (IC builder, previously Instagram co-founder and Labs lead)
Anthropic3 sources · 48 tips
GuestAmol Avasare
Head of Growth at Anthropic, previously growth at Mercury and MasterClass
Anthropic1 source · 20 tips
GuestCat Wu
Head of Product for Claude Code and Cowork, Anthropic
Anthropic1 source · 16 tips
GuestFiona Fung
Leads the Claude Code and Cowork teams at Anthropic; former Meta (Facebook Marketplace, Instagram) and Microsoft (Visual Studio, TypeScript) engineering leader
Anthropic1 source · 17 tips