How to measure retention honestly

1 The bottom line

Measure retention as cohort curves of a core action, and trust a curve only when it flattens in the people you want to keep. Spikes, new-user revenue and early clicks all look good while hiding users who leave.

2 Key takeaways

• Trust a retention curve only when it flattens. A flat line means some users found lasting value.
  • Check the share still using the product on day 7, 30 and 90. A curve that drips to zero means "you're toast" Robby Stein, Google28:20.
  • In 2007 Facebook was smaller than MySpace and Friendster but had better retention Brian Balfour, Reforge32:05.
• Cut cohorts by who sticks, then study those users. The average hides the small group where the product works.
  • A horizontal AI workflows startup saw freight forwarders stick for one workflow. It pivoted entirely to a premium product for them Rob Snyder1:10:30.
• Do not read spikes as proof until the cohorts behind them hold. Surges of new users cover up weak retention.
  • Periscope's surges in the US, France, Turkey and the Middle East masked poor retention. It was later shut down Kayvon Beykpour, Twitter1:01:41.
• Compare yourself to your category, and treat benchmarks as rough. Most apps lose users fast, so a low number is often normal.
  • Globally, 26% of first opens come back the next time Steve P. Young, Gamezop2:13.
  • One client's best day one retention in health and fitness was 23%, with 7% on day seven and 4% on day 28 Steve P. Young, Flo Period and Pregnancy Tracker10:26.
• Judge tests on behavior that lasts, not on clicks. A change can lift taps without building a habit.
  • Stash checked whether users still had Auto-Stash active 8 months after the test, and the lift was still visible Stash, Stash.
  • Paktor's winning thumbs design also lifted repeat app usage by 2.3% Paktor, Paktor.

Until you reach a point with your cohorts where there is a plateau, you have more work to do on figuring out the retention of your users.

Sarah Tavel, Pinterest

2.1 Pick the metric

Plot cohorts of a core action at the pace people actually use the product.

• Group users by signup week and track the core action, not monthly actives. Sarah Tavel, Pinterest27:53
  • Track both how many users return and how many complete the core action each week.
  • Look for a plateau, and ideally a smile curve where cohorts grow more active. A leaky bucket means more work to do.
  • If the line never asymptotes, the product is not working, however well designed Peter Deng1:43:52.
  • Instagram's camera-first app Bolt had top design and engineering. It failed its New Zealand or Australia test on retention Peter Deng1:43:52.
  • Many founders had never tracked cohorts before Tavel suggested it. Another operator still sees too little cohort analysis anywhere Shaun Clowes, Confluent1:14:17.
• Match the window to how often people use the product. Uri Levine, Waze20:58
  • Retention is a signal only if the product is used often. A once-a-year tax product cannot be read this way.
  • If usage is weekly, a free product should flatten near 60% and a paid one near 20 to 30%, at scale. Give the team two to three frequency periods to see it flatten Crystal Widjaja, Gojek and Kumu25:20.
  • For high-frequency products, three-month retention of 30, 40 or 50% is a good sign Uri Levine, Waze20:58.
  • Early on with friends and family, retention should be near 80%. Gojek's early days saw 60 to 70% Crystal Widjaja, Gojek and Kumu25:20.
• Track the long horizon, not only day one. Mark Pincus, Zynga44:00
  • Use D1 and D30 as early signals. Low ones will not give a high D365, but a high D30 can still end at zero D365.
  • Ask whether a user would still use this in a year.
  • Zynga tracked day 365 retention, which Pincus thinks no one else did. He says the most valuable companies have the highest D365.
• For subscriptions, track cohort revenue and customer counts together. Dmitry Gurski, Flo35:35
  • Measure the revenue a cohort produces in year one, then year two, then year three.
  • Pair net revenue retention with logo churn, so a healthy revenue number does not hide a shrinking customer base Jason Cohen, WP Engine54:00.
  • A Flo cohort looks like 100, 70, 70, 70 by year, and year four is sometimes above year three. About 25% of subscriptions come from win-back customers.
  • A 20% loss needs a 25% gain to recover. Of over 100 public SaaS companies, about two have NRR under 100%, and the median at IPO is 119% Jason Cohen, WP Engine54:00.

2.2 Set benchmarks, then judge your own trend

Use category numbers for a sanity check, and use cohort-over-cohort change as the real test.

• Check your day 1, 7 and 30 against your category before you worry. Steve P. Young, AppsFlyer Benchmarks3:38
  • Day 7 retention falls across essentially all apps. Many developers who feel behind are at average or better.
  • For a consumer app, D1 near 30 to 40% is solid. Much lower makes you question user intent Albert Cheng, Chess.com29:50.
  • Listory moved early retention from under 10% to over 40% within a couple of months, with gains in day two and week one Yaniv Gilad, Listory2:18.
• Use long-horizon lines for subscription apps. Alex Friedman and Mike Rollins, Flurry33:00
  • Measure return rate by cohort, on a given day, week or month after install.
  • Year-one resubscriber rate shows whether users got value after paying for a full year. Daily actives is weaker because it sits downstream of download volume Josh, Blue Throne17:11.
  • Apps that held over 50% of users past month four had a much longer shelf life Flurry33:00.
  • For year-one resubscribers, 50% is amazing, 40% good, 30% okay and 20% bad. Skip this scale for categories like dating, where retention is much lower Josh, Blue Throne17:11.
• Ask whether each new cohort retains better than the last. Josh Miller, The Browser Company7:58
  • Improving curves despite moving away from the earliest, most passionate adopters are the real signal.
  • Look at the shape of the curve and the size of the pool of possible users, not whether 90-day retention is 40, 30 or 10% Dmitry Gurski, Flo31:25.
  • Arc's D5/D7 retention sits between the low 30s and low 40s depending on cohort, and has been inching up over 12 months.
  • A mass-market product with a low plateau can still build a huge core base Dmitry Gurski, Flo31:25.
• Find out whether retention or acquisition is your bigger lever. Naomi Gleit, Meta26:35
  • Compute daily net growth as new registered users, minus users gone stale (no login for 30 days), plus resurrected users (back after 30 days).
  • Model retention against your addressable market. Assume a typical conversion rate and run the growth curve at several retention rates Scott Ritchie, Pyze5:00.
  • At Facebook the churn and resurrection lines were much larger than the new-user line, so retention became the biggest lever.
  • On a 100 million addressable market, 90% retention reaches capacity, 80% gives about 43 million users, and a weaker case leaves about 17 million Scott Ritchie, Pyze5:00.
  • Thumbtack's model showed extreme sensitivity to the repeat rate of new customers. The team moved resources from top of funnel to lifecycle and cross-sell Dan Hockenmaier, Faire16:23.

2.3 Cut cohorts by who sticks

Find the users who stay, learn why, and compare every segment on the same metrics.

• Find users who are as active on day 90 as on day 7, and talk to them. Sam Chaudhary, ClassDojo16:23
  • Overall retention looked mediocre, so the team segmented cohorts to see who was actually retaining.
  • They phoned those teachers and learned all of them also communicated with parents at home.
  • That led to parent and kid accounts. The multiplayer product brought real product market fit, with viral growth and retention.
  • Look for a second product that pairs with the core one. Strength training users who also use nutrition convert at a much higher rate and are much more retentive 100 days in Ben Gammon, Ladder19:35.
• Plot the retention curve of your top 20% most engaged users. Josh, Blue Throne15:02
  • Pick the engagement metric that fits your app, such as session frequency, session length or daily logins.
  • Find where the curve flattens, ideally at a high level.
  • A curve that drops straight to zero is a warning sign. Cohort retention is the leading indicator of LTV.
• Compare every metric by segment and by acquisition channel. Alex Friedman and Mike Rollins, Flurry19:40
  • Build segments from cohorts, usage level, events, demographics, geography or acquisition source. Compare paying with non-paying and casual with power users.
  • In Google Play Console, open User Acquisition, then Acquisition Reports. It shows retention at day 1, 7, 15 and 30 by channel, so you can compare source quality, not just volume Steve P. Young, App Masters6:37.
  • Flurry's example compares users who reached level three in the US against Germany.
  • The Google Play example app showed 50% day 30 retention.
• Treat what top users do as a clue, not a lever. Dan Hockenmaier, Faire23:42
  • Something unique about those customers drives both the behavior and the retention, so the link is not causal.
  • Look for the real drivers of value and a strong first-run experience instead.
  • He has rarely been able to move users from the worse bucket into the better one.

2.4 Do not get fooled by early spikes

Check the quality of every surge and every early dollar before you build on it.

• Check retention at each spike before you spend on it. Yves Benchimol, WeWard27:05
  • After its TV spike, WeWard checked retention first. It was very high against the market average, so they added ads.
  • Compare the growth graph and retention curves at every spike and non-spike Paul Warren, Margins40:00.
  • Margins' spikes from word of mouth, influencer videos and sharables matched or beat non-spike retention. Readers need real motivation, which may explain the intent.
  • Once you reach around 100,000 users per day, ad revenue starts to cover costs Yves Benchimol, WeWard27:05.
  • With paid traffic, spend the first 30 days collecting data. Paid downloads may not be your customer, so look at who actually stays Chad Mureta, App Empire9:26.
• Treat falling cohort curves as a sign that adjacent users have arrived. Bangaly Kaba, Instagram1:03:14
  • The signal is a drop in cohort quality while nothing in the product has changed.
  • Identify who the next user is, use the product as they would, talk to them and visit them.
  • Instagram's registration conversion fell 15% three months after a high, as it expanded into markets like India and the Philippines.
• Split new revenue from renewal revenue. Steve P. Young, RevenueCat3:06
  • Segment the revenue chart by transaction type. New should outpace renewals. If renewals win, the app may be mostly old users and dying.
  • A lower share of payers on day one can be healthy. It means a large base of existing users can still be monetized Michal Parizek, Mojo13:01.
  • In January one app did $56,000, with about $22,000 to $23,000 from renewals and about $34,000 from new.
  • Mojo gets about 50% of payers on day one, while RevenueCat data shows about 80% across the industry.
• Check older cohorts before you believe hard paywall revenue. Jeff Morris, Tinder4:49
  • Pitch decks lead with 0 to 10 million in a few months, but the cohorts behind them are very new.
  • Monetizing too early can push away people who would have become long-term retainers.
  • Forgotten subscriptions are free revenue, but they dull you to the real task. Deliver enough value every month that people want to keep paying Ravi Mehta, Tinder41:37.

2.5 Test and diagnose on lasting behavior

Judge changes on early habits and week two to four retention, and check where you measure.

• Judge a new feature on early logging and week 2 to 4 retention. Ben Gammon, Ladder18:20
  • At 100 days long-term retention cannot be measured yet, so compress the window.
  • Check whether users log consistently on day one and day two.
  • His reason: "people say stuff that doesn't map to reality." Users ask for things they then do not use.
• Feel the new-user experience and watch D7. Alexander Embiricos, OpenAI56:13
  • A power-user tool tempts teams to build features deep in the adoption journey. Counter that by signing up from scratch regularly.
  • He keeps several paid ChatGPT Pro accounts for this, at about 200 dollars a month each.
• Find the early window that decides who stays. Lior Barak, App Masters0:48
  • On subscription apps, most users who have not returned at least twice in the first seven days delete the app within 30 days. Work out why and how to trigger another visit.
  • Put installs into cohorts and measure how long users take to subscribe. Build retargeting around that window Lior Barak, App Masters0:32.
• Check where you measure, and dissect any drop. Yaniv Gilad, Listory14:43
  • Listory made second day retention its goal, then learned it was measuring it in the wrong place.
  • Every change resets the whole funnel, so an app change may need new messaging Yaniv Gilad, Listory14:10.
  • When a core metric drops 5%, split it by region, device, demographic and use case Robby Stein, Google28:20.
  • A drop in sessions after a release can signal a new crash Alex Friedman and Mike Rollins, Flurry4:50.

2.6 Where they disagree

Fewer day-one payers can signal stronger retention

Mojo50%

Industry (RevenueCat)80%

Sources split on how high retention must be, whether to chase downloads, and whether retention comes before revenue.

• How high must retention be?
  • Annual retention in consumer subscription needs to be north of 60, perhaps 70%, because a retained user pays the same in year two as in year one Casey Winters1:07:55.
  • Annual subscriber retention under 40% does not make a business bad, because win-backs and late conversions keep cohort revenue stable Dmitry Gurski, Flo35:35.
• Fix retention or get more downloads?
  • If your numbers are at or above category average, spend effort on downloads instead of retention anxiety Steve P. Young, AppsFlyer Benchmarks3:38.
  • Needing more downloads is the biggest fallacy. Optimize for the users you already get Steve P. Young and Haim, B7 Dev18:30.
  • One case study went from $1 to $17 a day with 50 downloads a day, and other clients saw revenue per download rise 46% B7 Dev18:30.
  • Download tactics get you from zero to one. At scale, engagement is the lever, and Bain research says 5% more retention can raise profits 25 to 95% Steve P. Young, Gamezop1:27.
• Does retention come before monetization?
  • Players who stay will eventually pay, and early spenders are rare. Build retention first and make monetization depend on it Chuka Ikokwu, Exploding Kittens8:49.
  • Crossy Road's creator cared about retention over monetization because he could always monetize people he retained Steve P. Young, App Masters4:55.
  • The other view is that people who pay pay attention, so stronger monetization does not hurt retention Steve P. Young, Flo Period and Pregnancy Tracker10:06.
  • An app with 1 million downloads a month and 5 million monthly actives saw retention improve after its monetization improved.