CaddieOpen in CaddieDownload for Mac

What 80M users taught Flo about A/B testing — Mike Charniatsou

Shared by Caddie Library · Updated
Sub Club by RevenueCat41:27Open on YouTube
Chapters show their main points

The bottom line

A paywall win counts only if it survives the lifecycle
Flo judges experiments on full-lifecycle revenue, because early paywall lifts often disappear.

A growth win is real only if it lifts the business over the full subscription lifecycle, and the way to find it is to follow evidence. Flo checks results months later, since short-term paywall gains often fade.

Key takeaways

Two guesses lost; evidence-driven rewrite won 23%
V1: design guess-10%
V2: design guess-10%
V3: medical rewrite23%
Design tweaks didn't fix the step; a postmortem and a rewrite with the medical team did.
  • A paywall conversion win can still be a business loss. 15:03
    • A subscription is a lifecycle of trial start, activation and retention, not a one-time event. Onboarding paywalls are only the start of the journey.
    • BoldVoice removed the X button for a hard paywall and saw conversion jump. Then refunds spiked and wiped out the gain.
  • Flo can wait for revenue because its users' needs change over a lifetime. 20:27
    • 40% of all Flo subscriptions happen after six months. Basics like cycle tracking stay free, and paid offers arrive when a new life stage creates a new need.
  • A failed test should be followed up with evidence, not a new explanation. 29:00
    • A perimenopause step lost about 10% twice while the team guessed at design fixes. A postmortem showed 75% of users got an "oops, sorry" screen.
    • The third version, rewritten with the medical team, won 23%. A blackout test that removed the step lost, which confirmed the win.
  • Marketing often spots an unmet user need before product does. 9:52
    • Meta ads about enhancing sex life performed well in the old funnel. Product turned that into its own user goal with tailored questions, and it became a strong revenue driver.
    • Weekly syncs share every ad and funnel change, even ones that seem minor.
  • Borrow the idea behind another team's win, not its solution. 34:49
    • A paywall change that other teams reported as a 10% win was copied pixel for pixel at Flo and did nothing.
    • The Blinkist-style reminder paywall failed at both Luna and Flo. The trial toggle worked for some teams and not for others.

Explanation usually tells you what you want to be true. And I think that the thing that you got to do when the experiment fails is to search for the evidence because the evidence tells you where to look.

Mike Charniatsou

Chapter summaries

0:00 Introducing Mike Charniatsou of Flo

40% of Flo subscriptions come after six months
First six months60%
After six months40%
Because users' needs change over a lifetime, Flo can afford to convert free users much later.

Flo's relationship with its users is built to last a lifetime, and that shapes how it grows.

  • Flo aims to be a lifelong partner to its users. 0:03
    • Mike is a senior growth product manager at Flo, which has nearly 80 million monthly users.
    • User needs differ over time.
    • Flo tracks how those needs evolve and acts when it can add value beyond the core product.
    • Before Flo, Mike worked on web paywalls at Luna.

1:48 Scaling Luna to Nearly $6M ARR

Luna grew from nothing to almost $6M ARR by making sleep stories interactive.

  • Mike joined Luna as its first product hire and grew with it. 1:48
    • He was later promoted to CPO as the team grew.
    • Luna won an App Store design award.
    • The idea was to turn sleep stories, then very popular, into a more interactive experience.
    • Mike is proud of the story as a growth leader.

2:52 Why Mike Left a CPO Role to Build Again at Flo

Mike left a CPO job to build hands-on again, at a scale Luna could not offer.

  • Two reasons drew Mike to Flo: building again, and scale. 2:52
    • With AI growing, he wanted to be an individual contributor who builds things.
    • Flo's web division looked like Luna a year earlier, so he knew what to do.
    • A CPO role is mostly managing people and setting strategy.
    • He called the move a no-brainer.

4:25 Building Web Onboarding Without a Playbook

With no bigger player to copy, both Luna and Flo had to work out web onboarding from first principles.

  • No playbook meant learning every basic of web onboarding from scratch. 4:25
    • Noom was a trailblazer, but no big sleep or meditation app existed to follow.
    • Flo is the biggest in its niche, so it had no leader to copy either.
    • Open questions: the goal of the onboarding, its length and its friction points.
    • Following bigger players saves a small team from paying for the same mistakes.

6:27 The Hidden Cost of Web-to-App Funnels

Web funnels are pitched as a way around store fees, but they are expensive to build.

  • Nobody warned how costly the infrastructure would be. 6:27
    • The pitch was bypassing App Store and Google Play commissions.
    • No single platform existed, so the team made many mistakes.
    • RevenueCat existed then, but without web funnels or web payments.

7:00 How Growth Changes From Early Stage to Flo's Scale

Early-stage growth sets fundamentals, while Flo's growth tunes the details of a funnel.

  • At Flo, progress means refining specific parts of the funnel. 7:00
    • A startup works in broad strokes, deciding how the onboarding journey should work.
    • Flo has a proven product behind every feature, so growth focuses on details.
    • Progress at Luna and progress at Flo have to be judged differently.

8:28 What Ad Budget Scale Unlocks for Experimentation

At scale, tested insight replaces gut feeling
Luna (early stage)Flo (late stage)
Growth focusBroad strokes: fundamentals, onboarding shapeDetails of specific funnel steps
Source of ideasGut feeling, educated guesses, investor tipsMarketing tests many hypotheses at scale
Insight pathMarket insights onlyMarketing validates, then hands to product
Behind each featureNo coherent platformProven product
Flo's ad budget lets marketing test hypotheses at scale and hand product validated insight that Luna could only guess at.

A large ad budget lets marketing test ideas at scale and hand product validated insight.

  • Scale replaces guesswork with tested insight. 8:28
    • Flo's marketing team scans the market and tests many hypotheses at scale.
    • It processes the results and passes them to product.
    • Luna had only market insights, gut feeling, educated guesses and investor tips.
    • Mike calls it a completely different paradigm.

9:54 Turning a Meta Ad Insight Into a New Onboarding Goal

Match the funnel to the ad the user clicked
Marketing can validate what resonates faster than any product funnel, so its signals should feed onboarding design.

Matching the funnel to the ad narrative turns a marketing signal into revenue.

  • The funnel should acknowledge the ad the user clicked. 9:54
    • Extra questions let users explore that narrative without a break.
    • Marketing validates what resonates much faster than a product funnel can.
    • Mike found a user need the existing onboarding had not fully explored.
    • Better tools now let smaller apps build this kind of tailored funnel.

12:35 Easing the Tension Between Marketing and Product Teams

Marketing and product usually clash, and the growth engine gets strong only when they work in sync.

  • Overcoming the marketing and product tension is what builds a strong growth engine. 12:35
    • Marketing thinks the funnel does not convert well.
    • Product thinks marketing brings the wrong audience.
    • Teams acting separately can only be so good.
    • At Flo the relationship is not like a happy marriage.

13:52 How Flo Shares Insights Through Weekly Syncs

Flo shares every change across teams in weekly syncs, because nobody can predict which insight another team will value.

  • Teams share every change, even ones that seem minor. 13:59
    • Marketing reports each ad campaign change, and product reports its own.
    • Sharing often leads to conversations nobody planned.
    • Mike: you never know what insights will be valued by a different team.
    • Flo built this habit on purpose, as a standing practice.

14:36 Why Paywall Conversion Is Not Your Monetization Strategy

A strong growth engine is built around the whole user lifecycle, not one point in the funnel.

  • Monetization has to follow the user's journey, not a single funnel step. 15:24
    • Analyzing experiments long after rollout shows that not all wins turn into business wins.
    • Mike names two common errors: assuming short-term wins become business wins, and treating conversion as a one-time event.
    • He expects most product people to agree.

16:31 When a Hard Paywall Win Led to a Spike in Refunds

Teams are pushed to report wins early, so only late checks show whether a win was real.

  • Incentives reward announcing wins before they are proven. 17:08
    • Teams want to post wins on LinkedIn and show them to management.
    • Flo reviews each rollout at 3, 6 and 12 months.
    • BoldVoice's Anada Lakra described being ready to roll out to everyone before refunds appeared.
    • Barnard's point: such wins come from not exploring the funnel deep enough.

17:45 How to Tell a Short-Term Win From a Business Win

Do a year of uplifts add up to growth?
Flo compares a year's reported relative uplifts with real business growth to separate short-term wins from business wins.

Adding up a year of reported uplifts and comparing them with real growth shows which wins were real.

  • Flo checks whether a year of wins compounded into business growth. 18:25
    • A team may ship tens of experiments with 5, 7 or 10% uplifts.
    • At year end, it asks if the business grew by the same percentage.
    • Barnard asked which KPI counts: LTV, 90-day revenue, retained subscribers or usage. It differs by company.
    • Relative uplifts mean little when you zoom out to the health of the business.

19:19 Balancing Freemium Users and Monetization

Squeezing free users has a limit, and growth leaders should know it.

  • Tightening the free experience works only until the options run out. 20:08
    • Mike: There are so many ways to squeeze the lemons until you run out of all those ways.
    • Barnard cites Duolingo's product chief, who chose not to squeeze freemium too hard, at a cost to the stock.
    • Removing free features to push payment is the tempting shortcut.

20:12 Why 40% of Flo's Subscriptions Come After Six Months

Flo does not force early monetization because it aims to be a lifelong partner.

  • Users' needs and willingness to pay shift over time, and Flo accepts that. 20:32
    • Users fall in and out of needs, so Flo does not push on day one or day seven.
    • Mike: We want our users to keep enjoying the product for free.
    • Barnard notes that apps with high inference costs or early payback pressure lack this luxury.

23:02 Building Monetization Around a Woman's Life Stages

Offers arrive when a new life stage begins
The core stays free for life; payment comes when a stage brings a need Flo can meet.

Flo tracks how needs change and offers a tailored product at the right moment.

  • Knowing the next need lets Flo make a relevant offer at the right time. 23:15
    • Flo is moving from period tracker to guide through life stages.
    • Stages include understanding periods, trying to conceive and following pregnancy.
    • This converts free users into payers 3, 6 or 12 months later.
    • Flo avoids intrusive paywalls that would spoil the relationship.

24:59 Cherishing the Customer Relationship

Caring about user experience is a durable edge, though not every app can afford it.

  • Treating the relationship with care wins in the long run. 25:02
    • Burner's Greg Cohn tracks negative user experiences as a KPI, a similar mindset.
    • Barnard: a knockoff app can be built in a weekend, but not trust.
    • Mike calls this a luxury, not universal advice for apps that need growth from day zero.

26:29 Measuring Free Trial Experiments With Proxy Events

An early proxy lets Flo call trials weeks sooner
Picking a good proxy depends on having enough history to know what predicts conversion 30–40 days out.

Flo judges trial experiments early by proxy events that predict later value.

  • Trials delay the conversion event, so Flo relies on proxies. 26:43
    • Most apps charge upfront, but Flo still offers trials.
    • A good proxy correlates with future user value and allows early calls.
    • A proxy must predict events 30 or 40 days later.
    • Early startups lack the historical data to build good proxies. Mike: in proxies we trust.

28:22 Learning From Failed Experiments With Evidence

After a failed test, check what users actually experienced before running it again.

  • A short postmortem comes before any new attempt. 29:22
    • The first instinct is to blame back copy, slow execution or heavy design.
    • Check whether treatment users saw the change as planned, and where the biggest drop or uplift sat.
    • Flo fails often because its funnels are well optimized and its win rate is slowing.
    • Iterate on new evidence only. Several competing explanations start a vicious cycle of experimenting.

29:41 The Perimenopause Step That Went From a 10% Loss to a 23% Win

The first two runs failed on a flaw in the question, not on the idea behind the step.

  • Evidence from the audience exposed the flaw that guesses had missed. 30:45
    • The step was meant to name the user's perimenopause phase and reinforce the whole experience.
    • The second run only tweaked the design, so it changed nothing.
    • The dead-end message clashed with an ad that promised to explain the user's phase.
    • The medical team rewrote the question. Users with no phase now move on to the next section without an apology.

33:25 Why Flo Runs Blackout Tests

Removing a step tests whether a reported uplift is real and keeps funnels from bloating.

  • Blackout tests are routine at Flo despite the risk of bad news. 33:33
    • They can show that a reported uplift was not real, which nobody wants to find.
    • Funnels keep gaining steps, so removing one is sometimes the better move.
    • Mike cites a line he heard: the biggest app feature is less features.
    • Flo runs them for multiple reasons, not only to validate a win.

34:14 Borrowing Ideas Instead of Copying Solutions

Another team's success is not proof that the same solution works in your context.

  • Teams copy what is visible because it is the easiest strategy. 34:32
    • Many teams borrow the solution before they understand the problem.
    • Flo and Luna both borrowed many ideas. The trouble starts when a solution is treated as guaranteed.
    • Product leaders from other companies pitched their wins as a silver bullet.
    • Mike had no reasonable explanation for the failed copy and would not build strategy on it.
    • David sums it up as borrow the idea, not the solution.

36:51 Why the Blinkist Paywall Failed at Flo and Luna

Borrow the idea; the solution may not transfer
Even a pattern that looks bulletproof failed at both Flo and Luna, while the idea behind it can still be iterated into something that works.

The idea behind the Blinkist paywall can work even where its exact solution does not.

  • The lasting idea is making people comfortable that they will be reminded to cancel. 36:25
    • Apps where it failed iterated on the idea: how to make people comfortable starting a free trial.
    • Mike was surprised, since it looks like a bulletproof pattern.
    • The trial toggle shows the same split. It worked for some teams and not others.
    • Understand the underlying problem first, because borrowed solutions rarely transfer.

37:27 The Problem With Adding Streaks to Everything

Streaks became the default fix for every engagement gap
A gamification feature can't stand in for missing intrinsic motivation, so ask whether forced engagement helps the user or only the dashboard.

Streaks are used as a universal fix, but they cannot replace missing intrinsic motivation.

  • Teams treat a lack of intrinsic motivation as a missing feature. 38:28
    • Duolingo is remarkable, but its success accidentally created a cargo cult of copied mechanics.
    • The test is whether forced engagement adds user value or only lifts product metrics.
    • Typical reasoning: users don't log in daily, so build a streak. Users skip the meditation audio, so add a streak.
    • David saw a streak in ChatGPT and thought, I don't need a streak in my AI tool. It might still suit some cohort.

39:21 Mike's Biggest Win of the Year

Adding the perimenopause phase step to onboarding was Flo's blockbuster experiment.

  • Mike names the phase step as his biggest win. 39:38
    • He calls it unexpectedly a blockbuster.
    • At Flo's scale, a lift this size is huge.
    • It is the same experiment he used to show how to iterate from evidence.
    • He answers the first of David's three closing questions with it.

39:54 Why Partner Mode Failed in Onboarding

A popular feature does not automatically make a good onboarding step.

  • Partner Mode did not help conversion despite its popularity. 40:27
    • It is one of the most requested features, used by more than 10 million pairs.
    • The team expected that putting it in onboarding would boost conversion.
    • Mike says there were multiple reasons it failed, without detailing them.
    • His lesson is that popularity and onboarding fit are separate questions.

40:43 Growth Would Be Easier If....

Mike jokes that growth was far easier in 2013.

  • His answer to the closing prompt is a wish for 2013. 40:50
    • He says life was so much easier then.
    • The prompt was Growth would be easier if…, and his answer was If we were in 2013.
    • David agreed with the sentiment and used it to wrap up.

In the growth library

Mike Charniatsou, Senior growth product manager at Flo, formerly first product hire and CPO at Luna. Mike Charniatsou explains how Flo separates paywall wins from real business wins, learns from failed tests using evidence, and borrows ideas instead of copying solutions. The standout result is an onboarding step that lost about 10% twice, then won 23% once the team fixed the problem the data showed.

Lessons that draw on this:

More from Sub Club · The whole library, and ask it anything

Transcript425 lines
Introducing Mike Charniatsou of Flo 0:00
0:00

Mike Charniatsou: Flo is at a very privileged position. 40% of all subscriptions take place after six months. The nature of the relationship between Flo and our customers is slightly different because we aim to be a lifelong partner to the user, and the user needs differ over time. We just keep track of the user needs, how they evolve. And when we see that there is something of value that we can bring to the user beside our core product offering, this is what lets us convert users from freemium into paying three months, six months, you know, 12 months later.

0:41

David Barnard: Welcome to the Sub Club Podcast, a show dedicated to the best practices for building and growing app businesses. We sit down with the entrepreneurs, investors, and builders behind the most successful apps in the world to learn from their successes and failures. Sub Club is brought to you by RevenueCat. Thousands of the world's best apps trust RevenueCat to power in-app purchases, manage customers, and grow revenue across iOS, Android, and the web.

1:05

You can learn more at revenuecat.com. Let's get into the show. Hello, I'm your host, David Barnard. My guest today is Mike Charniatsou, senior growth product manager at Flo, the women's health app with nearly 80 million monthly users. On the podcast, I talk with Mike about separating paywall wins from business wins, the A/B test that lost twice before delivering a 23% lift, and why you should borrow ideas, not steal solutions.

1:13

Hey Mike, thanks so much for joining me on the podcast today. Mike Charniatsou: Thanks for having me. I've been a long time fan, so great to be here. David Barnard: And we are live, you might notice a different set. I'm not in my home office. We're actually at App Growth Annual in New York City. This is lunchtime, so thank you for skipping lunch to join me today. But super excited to have this chat. And I couldn't miss the opportunity since you were in town for this, to have this conversation.

1:34

And I wanted to start with your experience prior to Flo. So you were at a company called Loona. Give me a quick history of that as your growth in the app space. Mike Charniatsou: I was the first product hire at Loona, so later I was promoted to CPO as the team grew, but it was a company focused on transforming

Scaling Luna to Nearly $6M ARR 1:48
1:54

what was at that time a very popular thing like sleep stories into something of a more interactive experience. It was like an early stage startup and we did lots of great things. We achieved lots of rewards. Loona won the app store design award. David Barnard: Oh, wow. Mike Charniatsou: Yeah. And as a growth leader, I'm really proud of this story because we managed to scale the app from zero to almost 6 million R&R, so over that period of time.

2:23

David Barnard: Nice. And so you grew up at Loona and learned a lot and became chief product officer, and then you transitioned to Flo. And I wanted to bring up Loona to give you an opportunity to talk about what it's like going from chief product officer at a small startup to then joining and being more a part of the team at a much bigger company, why you did it and how that transition has been. Mike Charniatsou: I guess there are two reasons why I decided to do that.

2:51

Reason number one is it was during the time when AI was getting bigger and I felt like there was never more

Why Mike Left a CPO Role to Build Again at Flo 2:52
3:00

interest in time to be an individual contributor, to build things, something that you can do when you're a CEO because you're more focused on managing people, leading people, setting up the strategy, and I wanted to build things. The second reason is that I recognized that Flo web division at that moment of time was where Loona were as a business maybe a year ago or so.

3:29

I mean, I had a very clear idea what needs to be done to hit the business objectives, but I could do that at a scale. So for me, that was a no-brainer. It was the opportunity that I could not just bus up. David Barnard: Yeah, I hear on that. I mean, I probably should be a manager at this point, but I have held onto the individual contributor badge for probably longer than I should have. But I hear you because it's like I'm not spending my days managing people.

3:57

I'm spending my days having conversations, making content, doing stuff, which I enjoy. Mike Charniatsou: It's just fun. It's a long forgotten feeling. David Barnard: So at Loona, you focus a lot on web paywalls, and then that's something you've also been working on a lot at Flo. So I'd love to hear some of the reasoning behind that and some of the successes you had in that. Mike Charniatsou: So when we started exploring the web acquisition, web onboardings with Loona, I guess the web as a thing was just getting traction.

Building Web Onboarding Without a Playbook 4:25
4:31

It was not that popular. There were few companies in the market doing so. And Noom was a great example. I guess they were trailblazer. The biggest challenge for us was Loona was a sleep meditation app. And at that point of time, there was no big player in the niche that we can look up to, like take that playbook and do the same by the playbook. Because when you're a small team, it's good to look up to bigger players because first of all, you can save money not making the same mistakes.

5:08

You know what to do and it can be a very good start. So we had to do and to learn everything from scratch, what's the goal of this onboarding? How long should it be? What are the friction points? We have to figure out all that stuff. When I was approached by the Flo team, I felt like the guys, they were in the same position because as you may know, Flo is the biggest women health app and there is no one bigger than Flo.

5:40

So I mean, there is no playbook for these kind of companies. That's why I took this opportunity. David Barnard: Yeah. And these days there are so many playbooks. People have been talking about web more and more and more, but you were kind of, like you said, figuring it out. There wasn't a playbook. And I think in some ways that's a huge advantage and probably why Flo poached you and approached you because when you have to start something from scratch and there aren't playbooks, you have to make a lot of mistakes and you learn and you figure things out and, I hate the term, but go back to first principles and figure things out.

6:17

And so how's it been at Flo and what are some of the things you've been able to learn and bring to the team? Mike Charniatsou: When you speak about web onboarding flows and web to app thing as a growth channel, the first thing that you hear is like, "Okay,

The Hidden Cost of Web-to-App Funnels 6:27
6:32

that's a good way to bypass App Store and Google Play commissions, all that stuff." And that's what we heard at that moment of time, but nobody told us how expensive it would be to build this thing because you need the infrastructure. And at that point of time, there was no RevenueCat. I mean, there was RevenueCat obviously, but- David Barnard: We didn't have web funnels and web payments and everything. Mike Charniatsou: Yeah. And you had to figure out it from scratch.

6:59

You had to make so many mistakes.

How Growth Changes From Early Stage to Flo’s Scale 7:00
7:02

There was no one coherent platform that allowed you to start and running and run things. And when I moved to Flo, things were slightly different. The main difference between a early stage startup and late stage startup as Flo is that it's how you learn to treat differently the progress that you make at Loona. And I guess most of the guys who work at the early stage startup would agree with me, you paint in white strokes, you're trying to figure out the fundamentals, how this thing should work, how should the onboarding journey look like.

7:42

At Flo, it's different because there was a product feed behind every feature and you had to appreciate how to grow with a more focus to specific details of the funnel.

8:00

David Barnard: I wanted to dive into both Loona and Flo. So what were some of the key moments of success or unlocks at Loona? And then I imagine as we've been discussing, every business is different, but were there anything that did translate and then things that didn't translate, anything that's been working well with web payments at Flo that you didn't even get a chance to try at Loona? Mike Charniatsou: When you're talking about Flo, the first thing that comes to mind is scale.

What Ad Budget Scale Unlocks for Experimentation 8:28
8:28

Product scale, advertising, budget scale, all that kind of things. And there is a wide range of opportunities that come at that scale that were not just accessible to us like when I was part of the Loona team. And scale, advertising scale, allows you to explore more growth opportunities, again, that were also not available to us at Loona.

8:56

For example, lots of great experiments at Flo came from insights that were collected by the marketing team because they were able to scan the market to test

9:08

lots of hypothesis at scale, gather those insights, process those insights, and pass them on to the product team at Flo. At Loona, we operated in a slightly different way. All we had were market insights, gut feeling, indicated guesses, some tips from the investors. It's just a completely different paradigm how you operate. David Barnard: So do you have any examples of the insights from the marketing team and the process of translating those insights into a web funnel or a specific type of paywall or a specific angle of the web funnel?

9:47

Any specific examples to make that more concrete? Mike Charniatsou: Yeah, there is a good one from my time at Flo. The marketing team, user acquisition team, they noticed that

Turning a Meta Ad Insight Into a New Onboarding Goal 9:54
9:57

a specific kind of ads with a specific kind of narratives around enhancing your sex life started to gain traction at Meta. The numbers were pretty good even with the existing funnel. We kind of took that insight and started to dissect it and we figured out that there was a user need that we did not completely explore with the existing onboarding funnels. So what we did, first of all, we created a separate user goal of enhancing sex life and we enriched the onboarding funnel with more questions to tailor

10:43

to that experience so that when the user clicks on the ad and gets into the funnel, we acknowledge that the user has come from a specific ad narrative. And with a specific set of questions, we help them to explore that particular narrative to continue seamlessly. It turned out to be a great revenue driver for us. So I guess the biggest finding here is sharing insights between marketing and product teams is crucial. That's a no-brainer, but many teams kind of disregard that which is not a good thing to do because marketing team has a better insight into things

11:30

that resonate with your audience and they can validate those insights much faster than you ever can do with your product funnel, for example. David Barnard: Yeah, I think something we've been talking more and more about on the podcast because five years ago you had to be at Flo scale or bigger to be able to build out that much complexity of funnels and everything, but it's getting easier and easier with tools and we're trying to make it easier at RevenueCat and things like that.

12:00

But it's so important that gaining those insights from marketing, like you're saying, the ads resonate because it's hitting that specific pain point and need. And then, like you said, it was performing well going into the normal funnel that didn't even highlight that need, but marrying the two where you can go from that whole cohesive experience of identifying the pain point and then reinforcing it along the way, it's just such a powerful combination and it's getting easier for subscale apps to explore that kind of thing.

12:31

Mike Charniatsou: Yeah. I think that when you have that synergy between the marketing team and the product team is probably the

Easing the Tension Between Marketing and Product Teams 12:35
12:38

only way when you can reinforce and make your growth engine stronger. Because if you act separately, you can be just this good, but when there is this synergy between the teams, because usually I guess most of the listeners would agree with me that there's always exist a tension between marketing and product teams because marketing thinks that funnel is not working that well, does not convert that well, and the product folks usually think that marketing does not bring the right audience.

13:12

There is always this kind of tension, but when you find a way to overcome it and get to that synergy, it's when you can build a really strong growth engine. David Barnard: I love it when I talk to product teams and marketing teams that are really well integrated, where they really see it as a cohesive product growth team versus marketing team, product team, and then each has their own little fiefdom. And it sounds like… I mean, is that something you've been able to do at Flo or how does that go?

13:44

How do you achieve that, especially at a company of the scale of Flo? Mike Charniatsou: I'm not going to lie, it's not like a happy marriage.

How Flo Shares Insights Through Weekly Syncs 13:52
13:52

The best way is just figure out how you can seamlessly exchange those insights. The thing that we do at Flo is we have weekly syncs. We keep exchanging everything that we do, even if we think it does not matter because you never know what matters, you never know what insights will be valued by a different team. So we kind of instill this practice that every change that we introduce, every change that the marketing team introduces to the ad campaigns, we just share that info and you never know where the conversation may take you from that point.

14:32

David Barnard: One of the things that came up as we were preparing for the podcast was this line, and I love it, so I'm going to read it here

Why Paywall Conversion Is Not Your Monetization Strategy 14:36
14:37

from my laptop, is, "Your paywall conversion is not your monetization strategy." I really like that mentality. So dig a little deeper in what that means and what you meant by that. Mike Charniatsou: I think there are a few things that many growth teams tend to overestimate, and one of them is that short-term conversion wins smoothly translates to business wins, which is not true. And the second point is that which many growth team got wrong, is that a subscription conversion is a one-time event, which is not.

15:14

There are lots of things like starting a trial, activating a trial, retaining the user, and so on and so forth. So it's a life cycle. What I meant by that is that usually a prerequisite for building a strong growth engine is building a monetization around the user life cycle, not just focusing on

15:37

short-term conversion wins at a certain point of your funnel because onboarding paywalls, for example, is just the beginning of a whole journey for the user.

15:50

And there are so many times when you launch an experiment, you see an uplifting conversion rate at the paywall and you think that it's a thing that's

16:02

going to boost my business to the moon, which is not true because quite often if you keep analyzing the experiments, even after rolling them out for a long time, you'll see that not all of these wins translate to business wins. And I guess most of the product guys would agree with me. David Barnard: Recent example from the podcast, I had Anada Lakra from BoldVoice on, and she shared some experiments

When a Hard Paywall Win Led to a Spike in Refunds 16:31
16:33

they ran where they removed the X button from the paywall to make it a fully hard paywall, and it was crushing it. They were so excited, they were going to roll it out to everyone. And then they started looking at refunds, and refunds spiked massively. And so what looked like just this incredible win that they figured out ended up being a negative drain. It became a drain on revenue because they had so many refunds. And so yeah, to your point, it's like there's so many things that look like wins that you're just not exploring the funnel deep enough.

17:08

Mike Charniatsou: And usually you're very incentivized to report those wins. Like on LinkedIn, for example, to your management, "Hey, we've got this new win, let's roll it out." I guess it's one of those cool things that you can do at Flo is that you have a possibility to analyze all those things three months after the rollout, six months after the rollout, 12 months after rollout, and figure out whether the win that you reported as a win is actually a business win.

17:41

David Barnard: Right. How do you think about business wins then?

How to Tell a Short-Term Win From a Business Win 17:45
17:44

What is the kind of KPI that you would ultimately work against? And I mean, I imagine it's going to be different for each company, but is it just LTV? Is it revenue in 90 days? Is it retained subscribers? Is it active usage of the app? There's so many different ways to think about what is a business win, and it's not always just the immediate financial. Kind of what you were alluding to earlier, sometimes a short-term financial win is at a sacrifice of long-term business goals.

18:14

So what are all the factors that go into how you think about what a business win actually is versus a paywall win versus a onboarding win versus the smaller wins? Mike Charniatsou: I guess over the course of a year, you kind of shape tens of successful experiments with 5, 7, 10% uplift. And at the end of the year, you just do this exercise and check, did your business grow by the same percentage that you reported over the course of the year?

18:46

Did the effect compound over the time or not? And that's probably the best exercise to separate short-term wins from business wins and just ask yourself if you are actually growing or you're just

19:06

reporting those relative uplifts that don't matter when you zoom out and have a look at how healthy your business is. David Barnard: Yeah. And I mean digging deeper into that though, Flo does have a lot of free users in certain countries and does have a freemium experience, right?

Balancing Freemium Users and Monetization 19:19
19:23

And I was actually talking to the chief product officer of Duolingo about this is they have, into the detriment of their stock performance, chosen the long-term business goal of not squeezing the freemium experience too hard. So how do you think about that in balancing the product wins, the paywall wins and things like that without ostracizing? Because you could remove features from the free experience to push people to pay.

19:52

I mean, there's so many ways to just squeeze the lemon more and more, but then like you said, in the long run, is that the business goals? How do you balance all that and think through all of that? Or how does the team? I know that's not all fall on you either. How do you discuss it internally inside Flo? Mike Charniatsou: There are so many ways to squeeze the lemons until you run out of all those ways.

Why 40% of Flo’s Subscriptions Come After Six Months 20:12
20:13

And suddenly there are not that many ways to keep squeezing that lemon. So it's one thing that you should be aware of as a growth leader because Flo is at every privileged position. 40% of all subscriptions take place after six months. David Barnard: Oh, wow. Mike Charniatsou: Yeah. Flo is a subscription-based business, but the nature of the relationship between Flo and our customers is slightly different because we aim to be a lifelong partner to the user, and the user needs differ over time.

20:52

They may easily fall in and out of different needs and the level of willingness to pay may be different and that's okay. So we accept those rules of the game. And by prioritizing this kind of relationship of being a lifelong partner to every woman, we are not forced to figure out how we monetize our customers.

21:21

If we have a look at the life cycle of our customers, as the woman goes through different stages of their life, they may experience different needs. Most of the needs, like tracking your cycle, for example, is covered by Flo without asking the user to pay. David Barnard: I love that point you made earlier about 40% of Flo's subscribers come after six months. I don't think there's many, at least I've heard of, subscription apps that could say that.

21:52

It's a really healthy place as a business, as we were talking about how quickly do you push for monetization. And like you said, not all apps have that luxury if you have a high inference cost, if you're early in your journey and need to get that user acquisition paid back immediately. But as you mature as a business, having that flexibility to be able to nurture a relationship over time and look for that longer term conversion is a really healthy balanced incentive.

22:23

So it's really impressive that Flo's been able to build that way over time. Mike Charniatsou: We want to keep it that way. We want our users to keep enjoying the product for free and make sure that when the user needs change, Flo will still be there. And we just treat it like that. We don't feel the need to try and monetize the user on day one and day seven because we know that one day something may change and we just will be there.

22:58

David Barnard: And then how do you think about everything after that onboarding experience?

Building Monetization Around a Woman’s Life Stages 23:02
23:02

So how do you get people more activated, engaged into the app? What's the journey from downloading the app, getting all the way through onboarding, using it for six months and then subscribing? How do you think about that entire journey? Mike Charniatsou: I think the uniqueness of this journey is mostly rooted in the underlying needs of the user because we already made a step in that direction and we are moving beyond being just a period tracker to a lifelong partner helping the user navigate through different stages of their lives.

23:37

And through their lives, women go through different stages. For example, they want to make sense of how periods affect their lives. They want to get pregnant, they want to follow pregnancy. When you know how the needs of your customers evolve with time, you are able to approach your customers with a tailored-specific product offering at the very right time of their life. And I think this is what Flo is good at. And being a lifelong partner, try to cherish this relationship with the user and not to spoil it with intrusive paywalls.

24:26

So we just keep track of the user needs, how they evolve. And when we see that there is something of value that we can bring to the user beside our core product offering, we do. And this is what lets us convert users from freemium into paying three months, six months, 12 months later. David Barnard: Yeah. Since we're here at App Growth Annual, I'll reference a talk from earlier, Greg Cohn from Burner, he talked about

Cherishing the Customer Relationship 24:59
25:02

negative user experience and they almost track that as a form of KPI to try and minimize those negative user experiences. And it sounds like y'all have a very similar mindset of it's not just about providing value to the user, but it's also about, I love the framing even, cherishing the relationship. "If you are using our app, that's meaningful and we appreciate that." And I think that kind of attitude toward a customer is how you win in the long run, especially again, in the age when anybody can go spin up in a weekend some cheesy knockoff, they're not going to win the market just knocking off the next app.

25:45

Being really in touch with the user and caring about that user experience and not being overly aggressive… You have to find the right time and the right approach to monetization, but the more you can be thoughtful about that and user-friendly about that, the better your chances are long-term to increase that LTV and build the relationship. Mike Charniatsou: Exactly. And once again, it's the luxury that we have as a product team.

26:10

I guess it's not a universal advice, as I mentioned, apps that feel the pressure to launch that growth engine from day zero, but it's a bit different at Flo.

26:23

I mean, we have our own challenges of a different kind, but yeah. David Barnard: How do you think about testing there at Flo?

Measuring Free Trial Experiments With Proxy Events 26:29
26:28

So when you roll something out, and maybe it does create a negative user experience that you don't want for your users, how are you measuring those quantitative and qualitative aspects of the experiments you roll out and making decisions based on? Mike Charniatsou: If we're talking about our web onboardings, Flo is one of the few apps out there in the market that still operates with trials. Most of the companies ask users to pay upfront.

26:55

We don't do that. We give users the opportunity to test the product, but it kind of adds complexity to how you measure experiments, how you evaluate your hypothesis because the actual conversion event, the trial activation event is kind of delayed in time. The best strategy and what we do at Flo is to find the right proxy event that would correlate with the value on the users in the future.

27:26

And yeah, we look at these proxy events that help us to make early calls without sacrificing precision and significance. David Barnard: Yeah. And proxy events are so powerful if you get them right. They're tough to get right though. Mike Charniatsou: If you get them right. Yeah, exactly, because your proxy event has to be informative enough about the probability of an event that will likely take place in 30 days and in 40 days.

27:56

So it's very hard for early stage startups because you can come up with a good proxy if you have lots of historical data when you have lots of data in general. And yeah, I mean when you're in early stage startup, you have to use your gaffling, I guess. But in case of Flo, yeah, in proxies we trust. David Barnard: Inside Flo, and maybe even from your experience before that, Loona, how do you think about failed

Learning From Failed Experiments With Evidence 28:22
28:23

experiments and how to learn from those and what lessons to take away? Mike Charniatsou: It's a tough one. It's a painful one. Yeah, first of all, when your experiment fails, the first thing that you instinctively tend to do is just try to come up with an explanation, know what went wrong, like back copy, slow execution, heavy design, all that kind of stuff. But I guess it's a wrong way of doing that because explanation usually tells you what you want to be true.

29:00

And I think that the thing that you got to do when the experiment fails is to search for the evidence because the evidence tells you where to look. And this is whenever an experiment fails at Flo, which happens quite a lot of times because we are operating with a well-optimized funnels and the win

29:22

rate is slowing down, the first thing you want to do is you want to check if everything went as you planned. Did the user experience and face the change that you introduce in the treatment group? Where was the biggest uplift or where was the biggest drop?

The Perimenopause Step That Went From a 10% Loss to a 23% Win 29:41
29:41

Is it a specific segment or not? And when you do this small postmortem, you try to figure out if there is an evidence for another try, because I don't

29:56

think it's a good practice to iterate because you have a new explanation. You have to iterate when you have new evidence because you may have a number of explanations and it will just launch this vicious cycle of experimenting. And here's a good example from our onboarding team. So in our periscore in boarding funnel, we thought that it could be a good idea to introduce a step where we define what phase of perimenopause the user was in.

30:27

And it made so much sense to us at the very beginning. The hypothesis was strong. We thought that it would just reinforce the whole experience. And the first test came back with almost 10% job in conversion rate. What we did is we did not try to look for the evidence. We just, "Okay, what went wrong? I mean, maybe something was wrong with the design. Let's just do it one more run," and nothing happened. We got the same job and conversion rate.

30:58

And only then we did some analysis, we tried to dissect the audience and we figure out that for some reason, the way the question was written, it did not make easier for the user to answer them. And basically almost 75% of the users were not assigned any perimenopause phase. And what they got was, "Oops, sorry, we were not able to pinpoint your perimenopause phase. We continue your journey a little bit later." It's not the thing that you expect to see in the onboarding funnel when you click on an ad that kind of promise you to understand what phase you're going through.

31:42

That was our piece of evidence. And what we did is we rewrite the question with the help of our medical team and we slightly changed the logic. And in case when we were still unable to define the phase, we just take the users to the next onboarding section without saying to them that, "Oops, we were not…" And it was the third iteration based on the evidence, and it came back with 23% win. David Barnard: Wow. Mike Charniatsou: Yeah.

32:14

So it was too good to be true. We did the false iteration. It was like a blackout test. We removed that new section. The test lost, which proved that there was an actual uplift. So the lesson here is all the iterations that were based on explanation lost, the one that was built on evidence one. And I guess that's a good way of treating and approaching your failed experiments. David Barnard: It's so hard to track, collect enough data, and then to actually go back and have the discipline to analyze the data.

32:53

And like you said, it's so easy to look at a test like that and just be like, "Oh, it failed. Move on." But then you would've missed that 23% win. And I love how deep you went into the numbers and the fact that it was initially a 10% loss, then a 23% gain. And then you even went back and did the blackout test to make sure it was a real 23% gain. And those are the kind of things that are hard to do, but that actually drive those business results in the long run.

33:20

Mike Charniatsou: Yeah. I mean, the thing with blackout is usually not what you want to do because you don't want to find out that there was no actual uplift that you reported.

Why Flo Runs Blackout Tests 33:25
33:33

But it's a pretty common practice of Flo to do blackout tests for multiple reasons. One of them is to validate that there was a nutshell uplift. And the next one is sometimes you just keep adding things to your onboarding funnels. And sometimes a good thing to do is to remove a step. So I just heard this cool idea these days the biggest app feature is less features, I guess. So it kind of resonates with this blackout thing.

34:09

David Barnard: Yeah. So what are your thoughts on seeing something work in one app, or even from your experience, seeing something work

Borrowing Ideas Instead of Copying Solutions 34:14
34:16

really well at Loona and then you come try to Flo and it just falls flat on its face? How do you think about learning from that? And then what experiments do you see and people posting on LinkedIn or whatever that you learn from and try versus just ignoring, keep down the path? Mike Charniatsou: I guess many product teams tend to borrow the solution before understanding the problem, and I guess it's not how it's supposed to be.

34:49

I think borrowing ideas is a good one. We borrowed a lot of ideas at Loona, we borrow a lot of ideas at Flo, but what you should not do is treat the success of another product team as a proof that this thing is going to work in your context for sure. Plus, I guess related to a output-first mentality of most product teams, because copying what's visible is probably the easiest strategy, but there is so much more. And at Flo, I was approached by product leaders from many companies telling me, "Hey, you got to try this one.

35:35

We did this, we tracked 10% uplift in conversion rate. This is your silver bullet. You got to do this." And we did exactly the same thing, pixel to pixel, and it just did not work. I didn't have a reasonable explanation why it happens. I just think that you should not treat it as your strategy, as a way to build things. David Barnard: I love your framing there too. I'm going to steal this from you, is borrow the idea, not the solution.

36:06

And that's exactly where so many people fail. There's trends that come and go, like the Blinkist paywall. And if you just grab the solution, it did work for some people, but then it failed flat on its face for another, but the idea… And then I saw a lot of people who did actually take the idea and iterate on the idea. The idea is that people need to feel comfortable that they'll be reminded to cancel. That makes them more comfortable to start a free trial.

36:32

So the Blinkist paywall did work for a lot of apps because it did that, but then the apps that it didn't work for, they would iterate on the idea. How do we make people feel comfortable starting a free trial? And that's how you take the idea and not just the solution. Mike Charniatsou: The funny thing is that it did not work at Flo at all. David Barnard: Oh, really? Mike Charniatsou: Yeah. David Barnard: I was going to ask you for an example, so maybe that's a perfect example.

Why the Blinkist Paywall Failed at Flo and Luna 36:51
36:52

Mike Charniatsou: I mean, we did tested it at Loona, we did test it at Flo, and it did not work for both of the teams. Although it looks like a bulletproof thing, but the same thing with the trial toggle, it worked for some teams and it's okay when other teams borrow

37:09

the idea, but you don't need to try and borrow the solution because most of the times, it does not work that way. You have to understand the underlying problem first. David Barnard: Yeah. So speaking of borrowing solutions and not ideas, I know you have thoughts on streaks, and we were talking about in preparing for the podcast, I saw

The Problem With Adding Streaks to Everything 37:27
37:33

in ChatGPT, they implemented a streak feature and it was such… Props to them for trying stuff, first of all. Maybe for the right cohort, it would work. But when I saw it, I was like, "Come on. I use AI enough. I don't need a streak in my AI tool." So what are your thoughts on the dueling of code and putting streaks in everything and that kind of thing? Mike Charniatsou: Yeah, a tiny disclaimer, I think that Duolingo is a remarkable product, but I think that their success accidentally created this cargo code when people keep just borrowing mechanics and trying to fix everything with another gamification feature.

38:16

And surprisingly, lots of product teams… Because it all boils down to the issue of intrinsic motivation. Some apps do have it, others don't. And lots of the product teams treat the lack of intrinsic motivation as a missing product feature, and what they're trying to do is like, "Okay, we're going to fix it with a streak feature. Users don't log in every day, let's build a streak. Users don't listen to the meditation audio.

38:50

Let's add a streak." For some reason, it's become like a universal answer to all the problems, which is, I mean, I don't get it.

39:02

And you have to ask yourself, does forcing this engagement contribute to the user value or it contributes to boosting your product metrics? And these are two different stories. David Barnard: That's a great lesson right there. Well, as we wrap up, I'll ask you the three questions I ask every guest now.

Mike’s Biggest Win of the Year 39:21
39:23

First one being, what's your biggest win of the last year? Just an experiment that went especially well, a paywall change, something that was a win? Mike Charniatsou: The experiment with adding the perimenopause phase into the onboarding funnel, determining what perimenopause phase the user is, was unexpectedly a blockbuster experiment for us for obvious reasons. David Barnard: Yeah. I mean, at scale, 23% win is- Mike Charniatsou: It's huge.

39:50

David Barnard: Yeah. And then what about the biggest fail of the last year?

Why Partner Mode Failed in Onboarding 39:54
39:54

Mike Charniatsou: There's so many of them. David Barnard: That's an easier question. Mike Charniatsou: Yeah. I mean, we have this feature, it's called Partner Mode. Circle, it's been one of the most requested features and it's super popular right now. More than 10 million pairs are using it. And we thought, "Okay, if it's so popular, so successful, we are going to definitely boost our conversion rate if we introduce this feature into the onboarding funnel." And it was no good.

40:27

It kind of taught me a lesson that not every good feature, not every popular feature is a good step for your onboarding journey for multiple reasons, but that's the case. David Barnard: Such a great example. And then the last question is a fill in the blank.

Growth Would Be Easier If.... 40:43
40:43

Growth would be easier if… Mike Charniatsou: If we were in 2013. David Barnard: That's a great answer. Mike Charniatsou: It was so much easier in our life. David Barnard: True, true. That's a great one. Well, fun place to wrap up. It's so great chatting with you. Thank you for joining me. Thanks for coming to App Growth Annual and making the trip over here. Mike Charniatsou: Thanks for having me. Yeah, it's an honor.

41:06

Thanks so much. David Barnard: Appreciate it. Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community.