All podcasts / Lenny's Podcast / Summary

Why AI is going vertical (again) | Dianne Penn (Anthropic)

2026-07-26 - 94 min - source - Read full transcript

Lenny Rachitsky (host)Dianne Penn

Key insights

Evals have replaced PRDs as the core artifact of user value for Anthropic's research product managers.

Instead of writing product requirement docs, Penn's team reads failed model transcripts to diagnose the real failure mode (a tool-use miss versus a search/knowledge miss versus an alignment issue), then encodes that pattern as an eval set researchers can act on and measure improvement against. She calls this a form of test-driven development for product managers.

evals-as-prds

Model capability jumps are discontinuous and often invisible until you build an eval to catch them.

Scaling laws produce a smooth loss curve as compute and data increase, but specific capabilities (like reliable arithmetic) emerge suddenly rather than gradually. Because these jumps aren't predictable in advance, Penn argues evals - not intuition - are the mechanism for detecting them, which also makes them essential for safety testing.

evals-as-prds

Frontier products and frontier models unlock each other together, neither succeeds alone.

Penn argues Opus 4.5 would not have had its adoption moment without a product vehicle like Claude Code, and Claude Code's accelerated adoption depended on Opus 4.5's model quality. Her team's internal framing is that you need frontier products to let users feel the magic of frontier models.

anthropic-product-culture

Anthropic Labs incubates discontinuous bets by being firm on the problem area and loose on the exact prototype.

Labs pods are deliberately small, sometimes a single engineer, because large teams pursuing ambiguous, large ideas tend to move slower. Many bets get shelved rather than killed outright, then revisited one or two model generations later once capability catches up; a failed prototype is still treated as valuable learning.

anthropic-product-culture

Golden Gate Claude, a 24-hour stunt reaching roughly 2,000 people, is remembered internally as an identity-forming moment.

In early 2024, Anthropic's researchers found an internal model feature that, when amplified, made Claude reference the Golden Gate Bridge in every response. Engineering, product, design, and research spun up a live experience on claude.ai within 24 hours. Small reach aside, Penn cites it as proof the company could turn research into distinct, fast-shipped user experiences rather than generic chatbot output.

anthropic-product-culture

Tightening safety review on more capable models creates an unintended access gap between labs and everyone else.

Penn describes a recent flagship release that drew heavy scrutiny and got restricted while companies verified it couldn't compromise their systems. She acknowledges this creates an advantage for labs with early internal access to more capable models, and says reducing that gap (not leaning into it) is one of Anthropic's current priorities, alongside building fallback UX systems so restricted models still degrade gracefully for users.

anthropic-product-culture

PM hiring criteria at Anthropic have stayed the same for three years, anchored on first principles thinking over pattern-matching.

Penn's team evaluates candidates on whether they can figure out what to do to hit a goal in a genuinely new situation, rather than repeating tactics that worked in prior consumer or B2B SaaS roles. She applies the same onboarding plan to senior, tenured PM hires as to early-career ones, because everyone has to relearn the job around evals and reading model transcripts.

future-of-product-management

Managers and senior PMs are expected to personally ship with the models, not just direct others who do.

Penn says it's not possible to recognize what a great AI product looks like without having built one yourself, so she carves out time to own one or two work streams personally on every model cycle to keep her intuition for how fast capability is moving, and expects the same of her managers.

ai-native-work-habits

'Sweat the tokens as much as you sweat the pixels' reframes user research for the AI product era.

Where a PM used to walk a user through their screen flow, Penn's team now reads full model transcripts to find the exact trajectory that failed, because the token-level detail (what was asked, what tool was or wasn't called, what document was retrieved) is where the real product opportunity or defect lives.

ai-native-work-habits

Anthropic deliberately trains Claude to push back rather than simply agree, treating disagreement as a core alignment goal.

Penn argues a model that just agrees makes a user's thinking worse, not better; the design goal is a model that behaves like a thinking partner and helps users land on materially better ideas, not one that flatters them into believing they were right all along. She's used this pushback capability personally, including asking a research model to challenge Anthropic's own pricing strategy.

future-of-product-management

Writing quality is currently one of frontier models' more jagged edges, and it's a deliberate, active training investment now.

Penn attributes this less to a knowledge gap (models have studied the best writing in existence) and more to sequencing: recent training cycles prioritized agentic tool use and reasoning, and now that those have improved, writing, tone, and character have become the next rough edge to invest in directly.

future-of-product-management

Durable human value sits in judgment, persistence, and a defended personal point of view, not raw output capability.

Penn splits her own Claude usage by how much personal judgment she wants preserved: she forms her own point of view before bringing Claude in on high-stakes writing, but delegates end-to-end on lower-judgment recurring work like monthly business reviews, becoming a reviewer/verifier instead of the author. She applies the same principle to raising her kids, prioritizing curiosity and 'believing in your own inner voice' over early heavy AI tool use.

life-and-career-advice

Books referenced

How to Raise an Adult - Julie Lythcott-Haims - Penn's go-to personal recommendation on what it actually means to raise capable adults rather than sheltered children.
Incorruptible - Eric Ries - Recent Audible listen; Penn liked its framing of how great companies sustain their values through the metrics they choose, not just revenue.
Crucial Conversations - Kerry Patterson, Joseph Grenny, Ron McMillan, Al Switzler - Penn built a Claude skill around this book's framework to help her prep for difficult management conversations in the moment.

Media referenced

Eric Ries episode of Lenny's Podcast - podcast - Referenced as the companion listen to Penn's take on 'Incorruptible' and building enduring companies.
Fiona Fung episode of Lenny's Podcast - podcast - Fung (manager of Claude Code and Cowork) suggested the burnout question Lenny asks Penn, and is cited on software engineering feeling lonelier now that teams work with agent fleets.
Ben Mann's episode of Lenny's Podcast - podcast - Referenced twice: Mann's line that 'this is the most normal it's ever going to be' and his answer about curiosity and Montessori schooling for kids.
Andrew Ambrosino episode of Lenny's Podcast - podcast - OpenAI's Codex app lead, cited as independently agreeing with Penn that PRDs are not dead.
Fallout - show - Penn's favorite recent watch, an Amazon Prime series based on the video game, binged during time off.

Companies

Anthropic - Penn's employer since 2023; the episode's central subject, covering its product and research culture.
OpenAI - Referenced repeatedly as Anthropic's early competitive benchmark and via its Codex product lead.
WorkOS - Episode sponsor; enterprise-readiness APIs (SSO, SCIM, RBAC, audit logs).
Mercury - Episode sponsor; banking platform Penn's host uses, highlighted for its new 'Command' AI interface.
JPMorgan Chase - Penn's employer before tech, as a high-yield bond trader on the trading floor.
Y Combinator - Gary Tan (YC president) is credited with the 'token maxing' framing discussed at length.
Amazon - Penn's employer immediately prior to Anthropic.

Techniques and frameworks

Evals are the new PRDs - Anthropic's research product management team defines and proves user value through eval sets built from real failure transcripts, not written product requirement docs.
Token maxing - Gary Tan's framing, discussed and reframed by Penn: spending heavily on tokens today to live like a 2028 user, though Penn recasts the real goal as experimentation volume, not spend itself.
Golden Gate Claude - A 24-hour 2024 interpretability demo where a dialed-up 'Golden Gate Bridge' feature made Claude reference the bridge in every response; an early example of turning research into a shipped user experience fast.
Labs incubation model - Anthropic Labs holds a strong opinion about the problem area and a loose one about the exact prototype, runs small pods (sometimes one engineer), and treats a failed bet as valid learning to revisit a model generation later.

Summary

Dianne Penn, Anthropic's first technical product manager and now head of product for its AI research and labs teams, walks through the company's product history from a five-engineer team in 2023 to shipping "more than a year's worth of 2024 model releases within a single quarter" today. She frames two inflection points as the real turning points: training Opus 3 to write long-form code rather than just autocomplete it, and Opus 4.5 arriving alongside Claude Code, since neither the model nor the product would have had its adoption moment alone. She also recounts Golden Gate Claude, a 24-hour 2024 stunt where an amplified interpretability feature made Claude reference the Golden Gate Bridge in every response, as the early moment the company found its identity for turning research into fast, distinctive shipped experiences.

The core of the conversation is how product management itself has changed at Anthropic. Penn's central claim is that "evals are the new PRDs": instead of writing product requirement docs from user interviews, her team reads failed model transcripts to pinpoint the actual failure mode (a missed tool call, a knowledge retrieval miss, an alignment issue) and encodes that as a measurable eval set researchers can act on. She likens this to test-driven development for PMs, and connects it to how emergent capabilities work: scaling laws produce smooth loss curves, but specific abilities jump into existence suddenly and unpredictably, which is why evals rather than intuition are the tool for catching both product opportunities and safety risks. She's explicit that PRDs aren't dead, especially for aligning large stakeholder groups on ambiguous, not-yet-validated ideas like early computer use, but for well-defined problems the eval has become the shorthand artifact.

Penn describes Anthropic Labs' incubation model as holding a strong opinion about the problem area and a loose one about the exact prototype, run by small pods (sometimes a single engineer), where a failed bet is treated as valid learning to revisit a model generation later rather than a wasted effort. On hiring, she says her team's PM evaluation criteria haven't changed in three years and center on first principles thinking over pattern-matched experience from prior consumer or SaaS roles; she applies the same onboarding plan to senior tenured hires as early-career ones, and expects managers to personally ship with the models, not just direct people who do, because you can't recognize a great AI product without having built one yourself.

A recurring theme is Anthropic's explicit design choice to make Claude push back rather than simply agree, which Penn frames as central to both alignment and usefulness: a model that just agrees makes a user's thinking worse, while one that challenges a half-formed idea makes them better. She personally splits her own Claude usage by how much judgment she wants to preserve, forming her own point of view before bringing Claude into higher-stakes writing but delegating routine work like monthly business reviews end to end and acting as reviewer rather than author. She also names writing as one of the current jagged edges of frontier models, attributing it to training sequencing (agentic tool use came first) rather than a knowledge gap, and says it's now an active investment area.

The episode closes with a lightning round covering her book recommendations (a parenting book on raising capable adults, Eric Ries's "Incorruptible" on sustaining company culture through the metrics you choose, and Crucial Conversations, which she turned into a personal Claude coaching skill), her view that durable human value sits in judgment and persistence as capability keeps expanding, and how she's raising her own kids toward curiosity and trusting their own inner voice rather than leaning on AI tools early. She credits Anthropic's low-ego, team-oriented hiring and a strong sense of shared ownership (rather than any individual heroics) as what has kept her from burning out through a year where the company shipped more models in one quarter than it did across all of 2024.

Notable Quotes

"We actually have a saying on the team of evals are the new PRDs." - Dianne Penn

"You have to sweat the tokens as much as you sweat the pixels." - Dianne Penn

"You should come away at the end of the day having better ideas because you've worked with Claude. That should be the hero goal, not just making your ideas 10% better." - Dianne Penn

"No matter how far you go, there's always another level." - Dianne Penn, quoting her grandfather's life motto

"It's not just about raising the IQ of experiences we build - it's about having better conversations with each other, being better managers." - Dianne Penn