How to run more experiments faster

1 The bottom line

Run more experiments by testing in parallel across surfaces, swinging bigger when traffic is thin, and removing waits in tooling and trial length. Velocity is the growth asset, because most tests fail and only a few big wins drive growth.

2 Key takeaways

• Treat experiments as a portfolio where most tests lose. A few big wins pay for the many losses, so volume matters.
  • Of about 20 tests, 10 are neutral, 5 terrible, 3 interesting and 2 good Patrick and Pablo, Social15:37.
  • Life360 tracks how many tests it ships, what share win and how big the average win is Giordano Contestabile, Life3609:13.
• Run tests in parallel across separate surfaces, not on the same one. Separate surfaces let tests run side by side without interfering.
  • About 10 tests can run across the stack at once Leon Sasson, Rise23:58.
  • Mojo ran test streams in three geo buckets at once Michal Parizek, Mojo19:09.
• Cut the waiting time inside each test cycle. Shorter loops mean more learning per quarter.
  • Duolingo doubled its experimentation velocity by moving from a 14-day trial to 7 days Anmol Tiwari, Duolingo6:43.
  • A third-party paywall platform cut Mojo's cycle from months or weeks to days Michal Parizek, Mojo18:25.
• When traffic is limited, batch changes into one bigger swing. There is not enough traffic for many mini tests.
  • If the batch fails, the team strips it down Ethan Ethier, Built With Science42:45.
• Test ideas instead of debating or forecasting them. Outcomes are hard to predict and building is cheaper now.
  • Tinder's product team lost time speculating for finance forecasts, while small roadmap items often worked big Jeff Morris, Tinder28:15.

maybe you cannot be running 10 parallel tests at the same time, but you can certainly be running 10 tests across the stack at the same time.

Leon Sasson, Rise

2.1 Set a target and a portfolio view

Count your experiments and big swings, and accept that most will lose.

• Set a monthly goal for big swings, assuming most will fail. Source: Claire Vo, ChatPRD21:56
  • Count huge swings per month instead of OKRs on PRs or revenue per headcount.
  • Reward ambition and learning over raw feature velocity.
• Manage three levers: how many tests you ship, what share win, and the size of the average win. Source: Giordano Contestabile, Life3609:13
  • Accept that some tests lose, as long as enough big wins land and the losses teach you something.
• Expect most tests to fail, and scale winners in stages. Source: Patrick and Pablo, Social15:37
  • Run tests small first, then bigger, so you keep adapting.
  • Their studio runs about 100 A/B tests a month across the portfolio.
  • The Flow team reportedly ran over 100 experiments a month Josh, Blue Throne49:16.
• Do not get precious about shipping once your product is mature. Source: Max Schoening, Notion52:50
  • Remind the team that users get more upset when you do not innovate than when you accidentally break something.
  • Increase shots on goal, which cheaper experimentation now makes easier.
  • He did this at GitHub and at Notion. His bar is to make obviously good things and iterate, not hide in a cave until then Max Schoening, Notion55:00.

2.2 Remove the waits

Fix tooling, trial length and release delays before adding more tests.

• Take testing out of the engineering queue. Source: Michal Parizek, Mojo18:25
  • Use a third-party paywall platform so the growth PM can work without waiting on engineering.
  • It cut the cycle of idea, build and launch from months or weeks to days.
  • Parizek says he could not have run that many experiments in such a short time without it, and calls experiment velocity the number one growth asset.
• Rebuild a fragile funnel so you dare to touch it. Source: Luke Martin-Fuller, Visible1:07:11
  • Visible's no-code web funnel was so flaky the team feared it would fall over, which blocked all experiments.
  • They rebuilt it in-house with an internal dashboard, so a new hire can run experiments and variants.
  • The quiz had been unchanged since October 2023, and the rebuild was named the biggest win of the year.
• Shorten the free trial so conversion tests read out sooner. Source: Anmol Tiwari, Duolingo6:43
  • Duolingo moved from a 14-day trial to 7 days, so results arrive in 7 days instead of 14.
  • That doubled their experimentation velocity.
  • It also gave more conversions over time, combined with changes to how many trials a user can get.
• Plan around app review delays. Source: Yves Benchimol, WeWard1:02:02
  • Use technology that avoids a release for specific changes, and keep good contacts at Apple.
  • Ship new features on Android first, since it is lower risk, cheaper and faster Melissa Cash, Pok Pok40:55.
  • Review takes 24 hours, and sometimes five days. WeWard was stuck for more than five days this summer.
  • Google Play also offers more unique testing functionality than the App Store, per Cash.

2.3 Run tests in parallel

Spread tests across surfaces, regions and angles so they do not queue behind each other.

• Split tests across separate surfaces of the app. Source: Leon Sasson, Rise23:58
  • Use onboarding, pre-paywall, paywall, win-back paywall and day one app open as separate surfaces.
  • Set up your analytics so the tests do not interfere with each other.
  • About 10 tests across the stack works, but not 10 on the same surface.
• Run a stream of tests in each geo bucket. Source: Michal Parizek, Mojo19:09
  • Split new users into US, Australia and Canada; Europe; and Latin America.
  • Run each test for one or two weeks, weekly or biweekly, in all three buckets at once.
  • Mojo reached statistical significance in that time per bucket.
• Test ad creative in cheap markets that convert like yours. Source: Yuliya Lennox, Solid Starts28:57
  • Run the same tests in a country such as Indonesia, then bring winners back to the US.
  • Localize with AI in a few hours.
  • Testing around 500 ads a week at US CPMs will bankrupt you. The same results cost $10 a day instead of $1,000 a day.
• Run several comment angles side by side. Source: Nick and Ivan, Superwall15:55
  • Try answering a question then adding a CTA, asking a question and naming the app, telling a story with no CTA, or pointing to a clip about the app.
  • Check which angle converts from profile to click.
  • Pointing people to a video works because many are not ready to install another app but will watch one more video.

2.4 Pick bets that fit your traffic

Match test size to how much traffic and team time you have.

• With limited traffic, batch changes into one bigger swing. Source: Ethan Ethier, Built With Science42:45
  • Test a batch of changes that address one problem instead of many mini tests.
  • If the batch fails, strip it down.
  • Focus on high-leverage funnel drop-offs, not colors or small text tweaks.
• Try cheap copy tests on your main CTA first. Source: Going, Going
  • Change only the wording, especially when it changes what users think they are signing up for.
  • A three-word CTA edit produced a 104% month-over-month increase in trial starts.
  • FLORA says green buttons instead of white have helped, though it gave no figure Weber Wong, FLORA39:39.
• Start paywall A/B testing once you reach about $100K a month. Source: Jonathan Parra, Superwall55:25
  • Larger apps that have not tested much get the biggest returns, since the first test usually pays back immediately.
  • Apps at $10,000 to $50,000 a month also benefit. Conversion can move from about 8%, the average he sees, to 15 to 20% after a few iterations, with no extra ad spend.
• Fund the program, because the first growth test usually fails. Source: Michael Segal and Mark Ungerer, Skylight57:52
  • Plan for the first test to fail and for about eight more to follow.
  • Skylight gave growth work 5% of the roadmap, ran one free trial test, and had no capacity for the follow-ups. They are now hiring their first subscription growth PM.

2.5 Let the numbers decide

Replace forecasts, debates and fear with quick tests, and learn from peers.

• Test instead of predicting the result. Source: Leon Sasson, Rise31:31
  • Have an opinion on direction, but do not try to forecast a 5% lift, 10% or negative.
  • Since building is easier, skip the 20 conversations and just test.
  • Clear winners fail and tiny changes win big.
• Skip Excel forecasting. Source: Jeff Morris, Tinder28:15
  • Do not make the product team spend its time speculating on things that are hard to predict.
  • Tinder was 90% of revenue as a public company and faced pressure to hit every 12-week quarter. Small roadmap items often worked big and covered forecasting misses.
• A/B test aggressive monetization ideas instead of trusting your fear. Source: Michal Parizek, Mojo17:24
  • Measure unusual ideas such as an app-open paywall instead of ruling them out in advance.
  • The weekly app-open paywall drew no noticeable complaints despite expected backlash.
• Share results with peer marketers every week. Source: Yuliya Lennox, Solid Starts32:45
  • Set up weekly calls with three or four adjacent apps and share what you tested and how it went.
  • Results differ by industry, but ideas like a blinking continue button spread this way.
  • PTA, the Flo parent company, shared tests across its portfolio apps, which she credits for faster learning.

2.6 Use AI to raise volume

Automate the shipping loop in stages, and aim AI at velocity without lowering the bar.

• Automate growth experimentation across four loop stages. Source: Amol Avasare, Anthropic34:50
  • Split the loop into identifying opportunities, building, testing against quality and brand bars, and analyzing results.
  • Score how well Claude does at each stage, then hill climb weekly.
  • Start with copy changes and minor UI tweaks, keep a human approving, and encode brand guidelines as a skill to cut review over time.
  • Win rate is roughly that of a junior PM two to three years in. It was not possible before Opus 4.5 and became promising with Opus 4.6.
• Use AI to lift velocity while holding win rate and average win. Source: Giordano Contestabile, Life3609:13
  • Life360 is using AI to raise velocity while keeping the other two levers roughly stable.
  • AI lets you run in two or three weeks experiments that used to take a year Claire Vo, ChatPRD21:56.

2.7 Where they disagree

Test size, test count and the right time to start depend on your product and team.

• Small tweaks versus big swings.
  • Cheap copy and color tests can pay off Going, Going and Weber Wong, FLORA39:39.
  • Ethan Ethier, Built With Science42:45 skips colors and small text tweaks, because limited traffic favors bigger swings.
  • Amol Avasare, Anthropic27:00 says a traditional growth team spends 60 to 70% of its time on small to medium bets and 20 to 30% on large swings.
  • Anthropic flips this to roughly 70/30 or 50/50 toward big swings, and keeps the small optimizations because no one else will. Apply this only when AI is central to your product's value, since he expects that value in two years to be 100 to 1,000 times today's, versus 30 to 50% more for a normal app like grocery delivery.
• Test widely versus pick a few bets.
  • High-volume teams push the number of experiments Josh, Blue Throne49:16.
  • With a tiny team, Skylight57:52 picks one or two bets carefully rather than test widely.
• When to start testing.
  • Jonathan Parra, Superwall55:25 points to about $100K a month as the time to start paywall testing.
  • FLORA tries cheap tweaks early, though its founder admits it does not yet run many A/B tests and plans to, especially around growth loops.