Summary
Dan Layfield, Director of Product Management at Diligent, joins host Ashley Stirrup on The Experimentation Edge to trace what fifteen years of A/B testing across Codecademy, Uber Eats, and the Fortune 1000 boardroom actually taught him. He breaks down the Codecademy trial-model rebuild that took four months and several rounds to deliver a 35% conversion lift, why moving on from a losing experiment too early is one of a PM's costliest mistakes, how to escape the B2B feature factory with metrics that genuinely ladder up, why retention should ride a product's natural use case instead of fighting it, and where AI is already replacing weeks of research and analysis. It's a practitioner's guide for product managers, growth leaders, data scientists, and engineers bringing experimentation rigor to both B2C and B2B.
Chapters
00:45 Meet Dan Layfield and Diligent
01:45 Two worlds of experimentation, Codecademy and Uber
03:45 The trial model that lifted conversion 35%
06:20 What to do with a losing experiment
08:50 Two flavors of experimentation
09:45 Reading forty metrics at Uber Eats
13:10 Escaping the B2B feature factory
16:45 Anchoring the North Star to real usage
19:15 Where AI fits in research and analysis
Takeaways
- A losing experiment is often inconclusive, not negative; treat it as a map of the funnel rather than a verdict, and know when a big problem is worth another round.
- Persistence paid off at Codecademy: four months and three to four rounds of trial-model testing produced a 35% conversion increase.
- Separate your two experimentation modes; high-volume CRO chases many small wins, while big, uncertain bets are worth taking multiple shots to de-risk.
- Most B2B product teams are feature factories; the fix is a top-down OKR system, and planning usually breaks in the connections between layers, not inside them.
- Anchor retention and engagement to the product's natural use case, and use AI to synthesize research and simple A/B analysis in hours instead of weeks.
Connect with the Guest
LinkedIn: https://www.linkedin.com/in/layfield/
Website: https://www.diligent.com
Sponsor
GrowthBook is the warehouse-native platform for experimentation, feature flags, and product analytics trusted by AI-native product teams at 3,000+ companies worldwide.
Go to http://growthbook.io