How to judge test results correctly

1 The bottom line

Judge a test on net revenue from matured cohorts and long-term retention, not on early conversion. Early wins often fade through refunds, cancellations and the wrong users.

2 Key takeaways

• Read revenue next to conversion. A small conversion gain can hide a much larger revenue gain.
  • VEED saw 3% higher paid conversion but 33% higher MRR on the same test.
• Wait for cohorts to mature before picking a winner. Refunds and cancellations arrive after the first read.
  • A hard paywall and a no-exit discount both lifted trial starts at Anada Lakra, BoldVoice23:40, but refunds spiked by far more than the gain.
• Check long-term retention before shipping a price change. A higher price can win on new revenue and lose over a year.
  • In several tests, Michal Parizek, Mojo7:47 saw the higher price win on new revenue but lose on modeled long-term revenue, so the baseline stayed.
• Segment every result, losers too. An average can hide who the change helped and who it hurt.
  • A soft paywall won significantly on ARPU in Simply Piano but attracted the wrong audience Yoav Sharon, Simply1:02:00.
  • A failed test worked for suburban users with a dog who had been on the platform a month Giordano Contestabile, Life36010:36.
• Confirm wins at full rollout and in each market. Small tests flatter, and effects differ by region.
  • Mojo stacked regional retests into a 60% ARPU increase Michal Parizek, Mojo10:05.

you might not be looking at your day 31 monetization rate or day 60 kind of net revenue rate, which ultimately is what matter.

Leon Sasson, Rise

2.1 Choose metrics that show value, not just volume

Small conversion lifts can hide much bigger revenue lifts

Moonly conversion39%

Moonly rev/100 installs47%

Moonly ARPPU5%

VEED paid conversion3%

VEED ARPU17%

VEED MRR33%

Read conversion beside revenue, and trust a gap only when the traffic supports it.

• Track revenue metrics together with conversion. See both volume and value in one test.
  • Moonly tracked conversion rate, revenue per 100 installs and ARPPU together.
  • VEED looked at trial conversion, paid conversion, MRR and ARPU, because a paywall change can shift revenue per user and plan mix.
  • Moonly: conversion +39%, revenue per 100 installs +47%, ARPPU +5% Moonly.
  • VEED's ARPU rose 17% in the same test VEED.
• Pair every lift with a significance check. Know the result is reliable before you roll it out.
  • Unacademy reported its click-through lift alongside its significance figure.
  • The lift was 13.5% at 100% significance Unacademy.
  • A gap of 3.67% versus 3.44% is only 0.23 points, and was called a slight win. It needs enough traffic to be trusted Instapage.
• Read results by channel. See where the effect actually lands.
  • After a CTA change, Going saw conversion improve through paid channels.
  • Paid traffic converted better than organic for the first time Going.
• Treat report benchmarks as correlations. Run the test on your own business before copying one.
  • Reports mix selection effects, such as who uses long trials, so one stat is not causal for your app Jacob Eiting and David Barnard, RevenueCat55:50.
  • The median 5.3% growth figure includes apps that owners have abandoned Jacob Eiting and David Barnard, RevenueCat55:50.

2.2 Wait for cohorts to mature

Early metrics are easy to read and often wrong, so judge on net revenue after the cohort has settled.

• Measure net revenue after refunds per exposed user, and give it time. Trial start rate is the easiest day one metric and it misleads.
  • Wait a week for trials to convert, then at least two more weeks for refunds to settle Anada Lakra, BoldVoice23:40.
• Tie fast readouts to day 31 and day 60 results. Early funnel numbers can point the wrong way.
  • Leon Sasson, Rise19:49 built predictive metrics that link quick top-of-funnel readouts to day 31 monetization and day 60 net revenue.
  • A change can raise trial starts while attracting the wrong users. Another can lower the first purchase rate while bringing users who retain longer Leon Sasson, Rise19:49.
• Add a retention counter metric to any pricing test. Check whether buyers return, renew or cancel.
  • If a user who would have bought Plus now buys Platinum, watch whether they come back, renew or cancel Shawn Gong, Tinder9:36.
  • Higher prices usually bring higher cancellation. Mojo modeled a year of revenue using 7-day cancellation rate as a proxy for renewals Michal Parizek, Mojo7:47.
  • Tinder also compares several models against each other and against a paywall with no model, and keeps iterating Shawn Gong, Tinder9:36.
• Do not react to the first backlash. Wait a few weeks, then compare it with the lasting effect on metrics.
  • After a redesign or a move of free features behind a paywall, a vocal minority shows up in the first week or two Francesco Polizzi, Discord46:20.
  • This applies especially to price increases, a view Polizzi shares with Phil Carter from his work at Quizlet Francesco Polizzi, Discord46:20.

2.3 Segment before you decide

Cut the results by user type, and look at behavior, so you learn who a change works for.

• Segment a winning test before rolling it out to everyone. A win on average can come from the wrong users.
  • Months later, Simply saw that the Simply Piano subscribers were mostly touch-only users with no keyboard, who signed up for a year and got little value Yoav Sharon, Simply1:02:00.
  • Deeper segmentation showed kids did find value, so Simply removed the paywall for adult-only accounts and kept it for accounts with kid profiles Yoav Sharon, Simply1:02:00.
• Segment losing tests to find who they did work for. Then build new experiments from those segments.
  • Life360 predicts experiment impact up front, then segments results afterward to understand why Giordano Contestabile, Life36010:36.
  • Someone who knows the user and the product deeply should read the data Giordano Contestabile, Life36010:36.
• Look at what users did, not only the target metrics. A/B tools can make teams lazy.
  • Review user behavior, talk to users and check for confusion, instead of judging only on a few metrics such as day 30 revenue Leon Sasson, Rise33:33.
  • Even a losing test can reveal something, such as most users preferring the new version, which informs the next test Leon Sasson, Rise33:33.
• Reframe a failed test's copy before discarding the idea. The loss may come from the framing.
  • A no-trial toggle gave 20% off for paying upfront, but the copy made the trial look like a loss. It tested negative even with a 30-day money-back guarantee Ethan Ethier, Built With Science57:30.
  • A new arm that makes both options seem pleasant showed a very strong lift over the original after a few weeks Ethan Ethier, Built With Science57:30.

2.4 Verify the call after you make it

Keep checking a decision at rollout, in each market and months later.

• Watch the numbers after full rollout and further down the funnel. A win in a small test may not hold.
  • Anonymous accounts skipped signup and looked great in a small test. At 100% rollout the gains faded, multi-device users could not sync or log in, and support tickets piled up Jelte Liebrand, Savvy Navvy1:01:10.
  • The first results were "through the roof", and the team still has scars from it Jelte Liebrand, Savvy Navvy1:01:10.
• Retest each paywall winner in every key geo. Ship only where it adds incremental revenue.
  • When a test wins with significance in one market, such as Europe, rerun it in the other key markets Michal Parizek, Mojo10:05.
  • Some Mojo changes shipped in some markets and not others, because they showed no lift or a negative one Michal Parizek, Mojo10:05.
• Revisit past decisions at 3 and 6 months. You can change your call once the cohorts have retained.
  • Do it automatically, for tests judged winners and losers alike Leon Sasson, Rise35:15.
  • Some tests judged a loser or flat, and not released, showed a cohort that retained a lot better 3 months later Leon Sasson, Rise35:15.
• Use 3 to 6 month holdouts for effects that change over time. Short and long-term effects can differ.
  • Good fits are leaderboards that take 10 weeks to climb, and push notifications whose novelty fades Cem Kansu, Duolingo22:10.
  • A new push notification can turn negative after 6 weeks Cem Kansu, Duolingo22:10.

2.5 Where they disagree

Teams differ on how much rigor to demand, how long to hold out, and how far to trust short-term data.

• Rigor versus speed. Some sources want a significance figure before rollout, others ship on judgment.
  • Unacademy reports significance beside its lift.
  • Leon Sasson, Rise35:15 does not need publication-level rigor, so a likely winner the team likes can ship without stat sig.
• How long a holdout should run. Long holdouts catch slow effects but are hard to keep clean.
  • Cem Kansu, Duolingo22:10 is "anti" very long holdouts, because teams forget them and bugs pollute the group.
  • Duolingo once ran an ads holdout for a year or two, and now prefers three to six months Cem Kansu, Duolingo22:10.
• Data versus conviction. Some bets cannot be proven by a test in time.
  • Short-term data often does not reflect long-term potential, especially for pricing, where compounding plays out over years Jimmy Liu, DoorDash47:17.
  • No one will run an A/B test for 10 years, so some bets need conviction Jimmy Liu, DoorDash47:17. Mojo instead models a year of revenue from a short-term proxy Michal Parizek, Mojo7:47.