How to run trustworthy experiments

1 The bottom line

Trustworthy experiments start with a written problem and hypothesis, one or two primary metrics, and a checked experiment system. Without those, noise and chance wins look like real learning and teams stack false positives.

2 Key takeaways

• Start from the problem, then write the hypothesis. A written reason lets you diagnose a failed test and try again.
  • Ethan Ethier, Built With Science37:43: about 70% of their experiments fail on the first try, and they iterate on the failures to find winners.
• Choose one or two primary metrics and add a guardrail. Many metrics produce chance wins, and a guardrail shows the harm a bold change might cause.
  • Leon Sasson, Rise34:34: with 20 metrics on a test, one or two will hit significance by chance.
  • Shaw Academy: trial conversion improved without a significant effect on cancellations, which they tracked as a guardrail.
• Change one thing and compare it to the current experience. Then the effect can be traced to the change.
  • HotelTonight: one streamlined variant against the original flow showed a 15% increase in conversions.
  • Jimmy Liu, DoorDash19:50: DashPass lifted order frequency almost 2% in the first 2 or 3 weeks, versus about 0.1% for typical tests.
• Check the experiment system before trusting any result. Real traffic is noisier than most teams expect.
  • Leon Sasson, Rise25:42: always-running A/A tests keep surprising the team with more noise than expected.
• Treat every result as provisional and record what you learned. Effects can fade and context changes.
  • Jason Cohen, WP Engine1:34:35: Shopify's team found about a third of test effects disappeared when checked later against a holdout group.

we have always running AA tests, and sometimes we get surprised that there's a lot more noise than you would expect in the real world.

Leon Sasson, Rise

2.1 Start with the problem and a written hypothesis

Good tests come from a known problem, a reasoned bet and a note of what you will do with the result.

• Pick the problem before the solution Ethan Ethier, Built With Science37:43
  • Find the biggest drop-off in Amplitude.
  • List every reason the page might not convert, such as lack of trust or an unclear value proposition.
  • Agree as a team on the biggest problem, and only then list solutions. Write it all down.
• Look at pages with high traffic and low conversion first Goldelucks
  • Goldelucks saw product pages with significant traffic where many visitors left without buying.
  • Small improvements on such pages apply to a lot of visitors.
• Build hypotheses from past test learnings, not guesses Clarks
  • Clarks used a large experiment database, which suggested that prominent delivery information affects user behavior.
  • They then validated it with an A/B test against the original page.
  • The database-informed hypothesis led to a 2.6% conversion lift.
• Ask what you will do if the result is green or red Tom Verrilli, Whatnot20:45
  • In product review, ask what changes in strategy for each outcome. If the team does not know, they have not thought it through.
  • Whatnot calls this "know then go". It also plays out what happens at 1,000 times the expected usage, before writing code.
• Tag every test with its reason, result and learning Patrick and Pablo, Social19:02
  • Keep these notes as a backlog, so a new hire or an agent can learn from them.
  • Apply the underlying idea, such as reducing onboarding friction, to other apps in different ways.
  • Pablo's point: telling an agent "I have been winning this A/B test" means nothing without the hypothesis and takeaway.

2.2 Decide what counts as a win before launch

Fix the test type, the metrics and the decision rights up front, so the result cannot be argued afterward.

• Label each test "do no harm" or "expected to win" Leon Sasson, Rise23:00
  • Do no harm tests cover directions the team likes, or removals of clutter. Roll them out unless metrics drop.
  • An expected winner that does not move the needle does not earn its place by default.
• Name one or two North Star metrics and decide mainly on those Leon Sasson, Rise34:34
  • Do not decide from a laundry list of metrics.
  • Unacademy set one goal for its carousel test, more clicks on the CTA, which made the winner easy to call Unacademy.
• Add a guardrail metric when the change is bold Shaw Academy
  • A trial reminder could plausibly push more people to cancel before billing.
  • Shaw measured cancellations alongside sign-ups, so they would know early if retention was hurt.
• Agree who decides tradeoffs before the result arrives Cem Kansu, Duolingo42:00
  • When a learning win costs revenue or DAU, no single metric settles it.
  • Teams that understand the principles decide small tradeoffs. Large business hits that serve a strategic goal go to senior people, including the CEO.
  • Duolingo runs around 400 concurrent A/B tests and regularly weighs learning gains against revenue.

2.3 Build the test so one change is isolated

Keep everything else equal, split users evenly and prove the base case before you optimize.

• Hold everything constant except the thing you test TIPSTOP
  • Both TIPSTOP variants highlighted the same key features, so the difference came from how the offer was communicated.
  • Changing the trial message and discount badge is a quick paywall test that needs no new product work.
  • The message and badge changed together, so the test cannot say which drove the 190% jump in direct conversion.
• Split users equally and compare counts on the one metric Unacademy
  • An equal three-way split makes click counts directly comparable.
  • The thin-bordered button got 13k clicks versus 9k for the original and 7k for the variant matching existing UI components.
• Test design guesses instead of trusting them Paktor
  • Paktor expected a heart and an X to beat thumbs up and down. The test showed the opposite.
  • Thumbs raised like clicks by 6% and dislike clicks by 4.3%.
  • Unacademy's variant that matched the design system did worst, so consistency did not make the CTA easier to find Unacademy.
• Test a stripped-out base case before iterating on a flow Jack McDermott and Tanmay Jain, ElevenLabs58:10
  • Hold out a group with only a sign-up form into the zero-state product, for example 10% for 5 days.
  • Measure the incremental retention, engagement and monetization the flow adds.
  • Their team spent too many cycles on onboarding. Simple onboarding with fewer screens to the aha was the bigger win.
• Choose a safe place to test, and start small when setup is costly
  • DoorDash sorted submarkets by profitability, so it could learn without badly hurting the business Jimmy Liu, DoorDash17:17.
  • Running DashPass as an A/B test separated real lift from selection bias, which executives feared Jimmy Liu, DoorDash19:50.
  • Tinder trained its ML paywall model on a couple of features first, with the static paywall as control, and measured conversion and total revenue Shawn Gong, Tinder7:50.

2.4 Check your experiment system

A trustworthy result needs trustworthy tracking, statistics and metric definitions.

• Keep an A/A test running at all times Leon Sasson, Rise25:42
  • It catches tracking and infrastructure problems.
  • User mix shifts by day of week and time of day, so users are not a clean random stream.
  • Rise says it took too long to adopt this.
• Use a standard statistics engine, never your own Leon Sasson, Rise28:28
  • Do not have product managers judge winners in spreadsheets. Pitfalls like sample ratio mismatch are easy to miss.
  • Rise uses Segment for tracking, Amplitude for product analytics and GrowthBook on the warehouse for test statistics.
  • GrowthBook was Rise's biggest move toward standardizing experimentation, and it is cheap if the data is already organized.
• Define each metric cleanly Leon Sasson, Rise28:28
  • Real user journeys are non-linear, with reinstalls and returns, so a metric like trial start rate needs one written definition.
• If you are small, watch the whole funnel Jelte Liebrand, Savvy Navvy1:02:42
  • Without Google-scale traffic you rarely have the sample size for clean A/B tests, and it is easy to read into the numbers what you want.
  • Keep testing, but remember no metric moves in isolation, and a win in one place can quietly break another.
  • At Google with a billion users testing is easy, but a startup can easily fool itself.

2.5 Read results honestly and keep learning

Count unclear tests as the real failures, and revisit old results when the product changes.

• Track the share of inconclusive tests and work to reduce it Giordano Contestabile, Life3609:57
  • Wins get scaled and losses teach something. An inconclusive test means wasted time, a bad setup or a weak hypothesis.
  • Give every loss time for learnings and follow-up action.
  • Revenue is still the core KPI, but Life360 measures the inconclusive share alongside it.
• Retest old ideas when the product or flow changes Patrick and Pablo, Social16:40
  • A result only holds within its context, so revisit big-lift ideas after the configuration changes.
  • Removing the paywall from onboarding failed once, failed again, and then worked once the rest of the app had been refined.

2.6 Where they disagree

Not every team should run routine tests, and some safeguards only fit large products.

• Routine A/B tests of small details may not be worth it Jason Cohen, WP Engine1:34:35
  • A/B testing cannot test strategy or vision. Real wins are rare, so most positive results on details are false positives, even with 95% accurate tools.
  • Teams stack winners for a year and see no change in conversion. Cohen calls it a waste unless you are highly sophisticated.
  • Lenny notes it can pay off at millions-of-users scale. Duolingo's 400 concurrent tests and Liebrand's advice to keep testing point the same way Cem Kansu, Duolingo42:00, Jelte Liebrand, Savvy Navvy1:02:42.
• Leak planning only matters when a test is visible Adam Mosseri, Instagram59:15
  • At Instagram's scale a test cannot be hidden. For controversial design changes the team decides beforehand what to say and whether to talk proactively or reactively.
  • This adds complexity to the development cycle but prevents conflated narratives.
  • A feed redesign test shown to 4% of iOS users was conflated with Reels, recommendations and creator reach complaints, and snowballed through creators and press.