Ronny Kohavi
Experimentation expert, former VP and Technical Fellow at Airbnb, Corporate VP at Microsoft, Director of Data Mining at Amazon
Where to hear and read about Ronny Kohavi1 episode or test
Ronny Kohavi is an experimentation expert who led A/B testing at Amazon, Microsoft and Airbnb and wrote Trustworthy Online Controlled Experiments. The lesson most worth learning from him is to test everything, expect most ideas to fail, and protect the trustworthiness of results so the numbers can be believed.
we are often humbled by how bad we are at predicting the outcome of experiments
The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)
Ronny Kohavi: the story and the lessons
1Ronny Kohavi's path through Amazon, Microsoft and Airbnb
Kohavi has led experimentation at three of the largest tech companies. He was director of data mining and personalization at Amazon, corporate VP running the experimentation platform team at Microsoft, and VP and Technical Fellow of relevance at Airbnb, where he led the search team.
He now advises and teaches full time. He wrote Trustworthy Online Controlled Experiments and teaches a cohort course on Maven.
He built Microsoft's experimentation platform because nobody there ran experiments. His group disbanded a month after he joined, so he picked this project himself.
When he showed Amazon's failure rates, the usual reply was 'we have better PMs here'. Office was shipping on a three-year cycle.
Bing was his beachhead, and its surprising results won over Office. Bing was the first Microsoft group to run experiments at scale and shared its surprising results with the rest of the company. Qi Lu and Satya Nadella backed getting Office to run experiments.
It was hard and slow, but Office then saw that many of its ideas failed too. People who moved out of Bing carried the message that there is a better way to build software.
His book outsold its publisher's forecast and all proceeds go to charity. Cambridge expected a few thousand copies, and it sold over 20,000 in English. It was translated into Chinese, Korean, Japanese and Russian.
It has fewer formulas than most statistics books and focuses on practice and trust. The authors gain nothing from sales.
2Why Kohavi expects most experiments to fail
Failure is the normal outcome of experiments at every company he worked at. Failure rates were about 66% at Microsoft, about 85% at Bing and 92% in Airbnb search relevance. Booking and Google Ads report 80 to 90%.
Every new group believes it is different and is humbled. The longer an area has been optimized, the harder it is to win.
He counts failure per experiment, not per idea. About 10% of experiments are aborted on the first day, usually from a bug or an overlooked problem rather than a bad idea.
After two to four pivots or fixes, an idea can still reach a successful launch.
A cheap idea ranked low in the backlog became Bing's biggest revenue win. Moving the second line of an ad up to the title line raised Bing revenue about 12%, worth over $100 million. Engineers had estimated a couple of hours, and an engineer tired of seeing it in the backlog built it in a couple of days.
The result looked like a bug, so the team hunted for one and replicated the experiment several times. Guardrail metrics did not drop significantly, so users were not hurt.
Kohavi says to be ready to fail eighty percent of the time on big bets. He does not discourage belief in a big idea, only the expectation that it will succeed. A big bet that does win becomes a home run.
Teams often fall in love with a radical redesign and expect it to work.
3How Kohavi balances small bets, big bets and redesigns
Most progress is a staircase of small gains, not a jackpot. About 250 Airbnb search experiments added up to a 6% revenue gain. Bing relevance aims for about 2% a year, with individual gains of 0.1, 0.15 and 0.2.
Bing ads revenue per thousand searches rose through small monthly gains, with some dips for legal reasons such as marking ads more clearly.
He runs a portfolio and tests every code change. Even small bug fixes can have surprising impact. Most organizations put about 70 to 80 percent of effort on the known and the rest on high risk, high reward ideas.
He compares it to investing: 'it's like in stock you need a portfolio'. Google's split was described as about 70 percent search and ads, 20 percent apps and new products, and 10 percent infrastructure.
Kohavi says to ship redesigns in small steps and test each one. Big redesigns usually fail, and after three to six months of building, sunk cost makes teams unwilling to unlaunch. Of 17 bundled changes, maybe four are good, and those are the ones to launch.
He calls the method OFAT, one factor at a time. He is not against large redesigns, but if one cannot be split he tests a few factors at a time.
He does not ship flat results. New code adds maintenance cost, so shipping with no measured value only complicates the code base. The claim that the team will be demotivated is, in his view, wrong.
Legal requirements are the exception: try three variants and ship the one that hurts least. Airbnb's rebrand, with a new logo and redesigned homepage, ran as an experiment with a long-term holdout and was positive in the end.
Bing's social integration bet took too long to abort. It ran about a year and a half, hooked into the Twitter firehose and Facebook, and every experiment came out flat to negative. Kohavi mailed the team's leader statistics showing it was time to abort. The leader chose to continue and the team aborted a few months later.
Kohavi says it took a little long to call the failure, and that whether to keep going is a million dollar question. Netflix's social features and Airbnb's 'friends stayed here' idea also failed.
4Kohavi's rules for trustworthy experiment results
Kohavi checks sample ratio mismatch before reading any result. About 8% of Microsoft experiments had a mismatch. With a 50/50 design and a million users, a 50.2% split should happen about once in half a million experiments.
Common causes are bots that hit variants unevenly, data pipeline filtering and campaign traffic entering mid-site. Third-party tools that added the check reported 6%, 8% and 10% mismatches.
He makes bad results hard to use, because warnings get ignored. A banner was ignored, and a blanked scorecard with an OK button was clicked through anyway. The fix was to underline every number in red, so even screenshots show the problem.
The OK button stays because the metrics can help debug the cause. He says product managers want to see success, a natural bias teams must fight.
He investigates any result that looks too good to be true. Under Twyman's law, if experiments normally move metrics under 1% and one shows 10%, skip the celebratory dinner and investigate. Nine times out of ten a flaw is found.
Real outliers exist, such as the Bing ad title win, which was replicated and triple-checked.
Kohavi says a p-value under 0.05 is not a 5% error rate. With Airbnb search's 8% success rate, a result at p under 0.05 has about a 26% chance of being a false positive. A p-value of 0.02 does not mean a 98% chance the treatment wins.
Getting the number people want requires Bayes' rule and a prior, which historical failure rates supply. The result is the false positive risk.
At Airbnb, borderline wins were rerun before launch. A p-value between 0.01 and 0.05 triggered a replication, and Fisher's or Stouffer's method merged the two runs into a joint p-value. Requiring p below 0.01 plus replication cuts false positives sharply.
This matters because wins shared across the company become institutional knowledge.
He treats trust as the core requirement of a platform. A platform is a safety net for aborting bad launches and an oracle for the key metric after the two-week test. His platform ran built-in checks and stopped to warn when an experiment looked wrong.
Early tools lacked this, which he calls a big mistake. Trust builds up slowly and is easy to lose.
Optimizely lost trust by letting users stop at the first significant p-value. Checking p-values in real time inflates false positives, so aiming for 0.05 could mean about a 30% error rate. Customers saw 'wins' that never showed up in revenue.
A post titled 'Optimizely almost got me fired' described an A/A test that came out significant too often. Kohavi and Ramesh Johari of Stanford pointed out the flaw, and the company fixed it.
5How Kohavi chooses what to optimize
Kohavi says revenue alone is the wrong goal. Many bad changes raise revenue. More ads lift revenue now but hurt users later.
The overall evaluation criterion should be 'causally predictive of the lifetime value of the user', so each change must pass one question: does it help only now, or also later?
At Bing he paired revenue with user experience metrics. Key user metrics were time to a successful click and the share of successful sessions. One framing was a fixed budget of vertical ad pixels, so a change had to earn more within the same space to ship.
Bing ran experiments that raised and lowered ads to map how ads affect churn and the time users need to find a good result.
At Airbnb the goal was a happy guest months later, not a booking. The stay rating shows that, but it arrives months late, so a training set is built to predict it.
Retention rate and time to finish a task are measurable metrics that make the criterion usable.
Amazon's email team showed what a missing countervailing metric costs. The team got credit for every purchase after an email, so they ramped up sends and their metric rose. A side study valued each unsubscribe at a few dollars, and more than half of campaigns turned out net negative.
It led to a new default where the email offers to unsubscribe from that campaign type, such as author emails, which shrank the loss.
6How Kohavi builds an experiment culture and platform
He starts in a team that ships often and agrees on its goal. Bing launched several times a day, while Office once shipped every three years. The first team also needs a clear overall evaluation criterion.
The microsoft.com team picked time on site, but half the room thought more time on a support site was good and half thought it bad. If a team cannot agree on the direction, the metric fails.
He makes platforms self-serve so each experiment costs almost nothing. Teams set up experiments, targets and metrics on a website with no help. Microsoft ran 20,000 to 25,000 experiments a year by 2019, about 100 new treatments started every working day.
Bing allowed up to 10,000 metrics in a scorecard, split into templates of about 2,000, such as one set for UI experiments and one for revenue. Airbnb's weaker analysis tools forced it to hire many data scientists to compensate.
The biggest speedup is an instant scorecard, then variance reduction. Results should arrive within about a day, not after a week's wait for a data scientist. Capping skewed metrics, such as purchases over $1,000 counted as $1,000, lowers variance.
At Airbnb a cap near 30 nights booked per month makes sense, since some agencies book tens or hundreds. CUPED uses pre-experiment data to cut variance and stays unbiased.
He tells new teams to buy part of the platform. At Amazon and Microsoft nothing existed, so they built in-house. Today third-party products are pretty good and trustworthy, and the real question is whether to build 10% or 90%.
He suggests asking where an organization sits on the six axes of a published crawl, walk, run, fly matrix to decide what to build next.
He says to start experimenting at tens of thousands of users. Detecting a 5% change on a retail site needs about 200,000 users. With tens of thousands of users, startups should chase 5 to 10 percent gains, not 1 percent.
Smaller teams should use the early period to build the platform and culture. One-off events such as mergers and acquisitions cannot be tested.
He keeps a searchable archive and reviews surprising results quarterly. Teams search past experiments by keyword before starting new ones. A quarterly meeting covers the most surprising results, where predicted and actual outcomes differ a lot, including losers.
The Windows indexer test improved relevance but drained laptop batteries, and the team documented it for the next design. Public sources such as GoodUI.org, with about 140 patterns, also help pick ideas to test.
7Where Kohavi saw testing beat opinion
Open-in-new-tab won at Microsoft, was forgotten, and had to be reintroduced at Airbnb. It ran around 2008 at Microsoft, first opening Hotmail in a new tab in the UK, then MSN search. Designers pushed back because users had not asked for it.
Airbnb later designs dropped the pattern. Kohavi reintroduced it and saw big improvements, and his lesson is to document winners and remember them.
Untested changes at Airbnb were often worse than assumed. Every change to search relevance was A/B tested before launch, while other teams tested some changes and skipped others. Some changes that launched untested turned out negative when he tested them.
He believes more experiments would have left the company better off, but admits the counterfactual is unknown.
He argues testing matters more during upheaval such as Covid. 'If in peacetime you're wrong two-thirds to 80 percent of the time why would you be suddenly right in wartime.' Results may need replicating later to check they generalize.
Airbnb's big investment in online experiences had unpromising early data and is now a footnote. He thinks Airbnb's revenue would have recovered the same way had it stayed the course.
The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)
Ronny Kohavi explains how to run trustworthy A/B tests, set the right success metric, and build an experiment culture, drawing on Bing, Amazon, Microsoft and Airbnb. A two-line ad title change at Bing raised revenue by about 12 percent (worth over $100 million), yet 66 to 92 percent of experiments fail across his career.
Kohavi has led experimentation at three of the largest tech companies. He was director of data mining and personalization at Amazon, corporate VP running the experimentation platform team at Microsoft, and VP and Technical Fellow of relevance at Airbnb, where he led the search team.
He now advises and teaches full time. He wrote Trustworthy Online Controlled Experiments and teaches a cohort course on Maven.
He built Microsoft's experimentation platform because nobody there ran experiments. His group disbanded a month after he joined, so he picked this project himself.
When he showed Amazon's failure rates, the usual reply was 'we have better PMs here'. Office was shipping on a three-year cycle.
Bing was his beachhead, and its surprising results won over Office. Bing was the first Microsoft group to run experiments at scale and shared its surprising results with the rest of the company. Qi Lu and Satya Nadella backed getting Office to run experiments.
It was hard and slow, but Office then saw that many of its ideas failed too. People who moved out of Bing carried the message that there is a better way to build software.
His book outsold its publisher's forecast and all proceeds go to charity. Cambridge expected a few thousand copies, and it sold over 20,000 in English. It was translated into Chinese, Korean, Japanese and Russian.
It has fewer formulas than most statistics books and focuses on practice and trust. The authors gain nothing from sales.
Failure is the normal outcome of experiments at every company he worked at. Failure rates were about 66% at Microsoft, about 85% at Bing and 92% in Airbnb search relevance. Booking and Google Ads report 80 to 90%.
Every new group believes it is different and is humbled. The longer an area has been optimized, the harder it is to win.
He counts failure per experiment, not per idea. About 10% of experiments are aborted on the first day, usually from a bug or an overlooked problem rather than a bad idea.
After two to four pivots or fixes, an idea can still reach a successful launch.
A cheap idea ranked low in the backlog became Bing's biggest revenue win. Moving the second line of an ad up to the title line raised Bing revenue about 12%, worth over $100 million. Engineers had estimated a couple of hours, and an engineer tired of seeing it in the backlog built it in a couple of days.
The result looked like a bug, so the team hunted for one and replicated the experiment several times. Guardrail metrics did not drop significantly, so users were not hurt.
Kohavi says to be ready to fail eighty percent of the time on big bets. He does not discourage belief in a big idea, only the expectation that it will succeed. A big bet that does win becomes a home run.
Teams often fall in love with a radical redesign and expect it to work.
Most progress is a staircase of small gains, not a jackpot. About 250 Airbnb search experiments added up to a 6% revenue gain. Bing relevance aims for about 2% a year, with individual gains of 0.1, 0.15 and 0.2.
Bing ads revenue per thousand searches rose through small monthly gains, with some dips for legal reasons such as marking ads more clearly.
He runs a portfolio and tests every code change. Even small bug fixes can have surprising impact. Most organizations put about 70 to 80 percent of effort on the known and the rest on high risk, high reward ideas.
He compares it to investing: 'it's like in stock you need a portfolio'. Google's split was described as about 70 percent search and ads, 20 percent apps and new products, and 10 percent infrastructure.
Kohavi says to ship redesigns in small steps and test each one. Big redesigns usually fail, and after three to six months of building, sunk cost makes teams unwilling to unlaunch. Of 17 bundled changes, maybe four are good, and those are the ones to launch.
He calls the method OFAT, one factor at a time. He is not against large redesigns, but if one cannot be split he tests a few factors at a time.
He does not ship flat results. New code adds maintenance cost, so shipping with no measured value only complicates the code base. The claim that the team will be demotivated is, in his view, wrong.
Legal requirements are the exception: try three variants and ship the one that hurts least. Airbnb's rebrand, with a new logo and redesigned homepage, ran as an experiment with a long-term holdout and was positive in the end.
Bing's social integration bet took too long to abort. It ran about a year and a half, hooked into the Twitter firehose and Facebook, and every experiment came out flat to negative. Kohavi mailed the team's leader statistics showing it was time to abort. The leader chose to continue and the team aborted a few months later.
Kohavi says it took a little long to call the failure, and that whether to keep going is a million dollar question. Netflix's social features and Airbnb's 'friends stayed here' idea also failed.
Kohavi checks sample ratio mismatch before reading any result. About 8% of Microsoft experiments had a mismatch. With a 50/50 design and a million users, a 50.2% split should happen about once in half a million experiments.
Common causes are bots that hit variants unevenly, data pipeline filtering and campaign traffic entering mid-site. Third-party tools that added the check reported 6%, 8% and 10% mismatches.
He makes bad results hard to use, because warnings get ignored. A banner was ignored, and a blanked scorecard with an OK button was clicked through anyway. The fix was to underline every number in red, so even screenshots show the problem.
The OK button stays because the metrics can help debug the cause. He says product managers want to see success, a natural bias teams must fight.
He investigates any result that looks too good to be true. Under Twyman's law, if experiments normally move metrics under 1% and one shows 10%, skip the celebratory dinner and investigate. Nine times out of ten a flaw is found.
Real outliers exist, such as the Bing ad title win, which was replicated and triple-checked.
Kohavi says a p-value under 0.05 is not a 5% error rate. With Airbnb search's 8% success rate, a result at p under 0.05 has about a 26% chance of being a false positive. A p-value of 0.02 does not mean a 98% chance the treatment wins.
Getting the number people want requires Bayes' rule and a prior, which historical failure rates supply. The result is the false positive risk.
At Airbnb, borderline wins were rerun before launch. A p-value between 0.01 and 0.05 triggered a replication, and Fisher's or Stouffer's method merged the two runs into a joint p-value. Requiring p below 0.01 plus replication cuts false positives sharply.
This matters because wins shared across the company become institutional knowledge.
He treats trust as the core requirement of a platform. A platform is a safety net for aborting bad launches and an oracle for the key metric after the two-week test. His platform ran built-in checks and stopped to warn when an experiment looked wrong.
Early tools lacked this, which he calls a big mistake. Trust builds up slowly and is easy to lose.
Optimizely lost trust by letting users stop at the first significant p-value. Checking p-values in real time inflates false positives, so aiming for 0.05 could mean about a 30% error rate. Customers saw 'wins' that never showed up in revenue.
A post titled 'Optimizely almost got me fired' described an A/A test that came out significant too often. Kohavi and Ramesh Johari of Stanford pointed out the flaw, and the company fixed it.
Kohavi says revenue alone is the wrong goal. Many bad changes raise revenue. More ads lift revenue now but hurt users later.
The overall evaluation criterion should be 'causally predictive of the lifetime value of the user', so each change must pass one question: does it help only now, or also later?
At Bing he paired revenue with user experience metrics. Key user metrics were time to a successful click and the share of successful sessions. One framing was a fixed budget of vertical ad pixels, so a change had to earn more within the same space to ship.
Bing ran experiments that raised and lowered ads to map how ads affect churn and the time users need to find a good result.
At Airbnb the goal was a happy guest months later, not a booking. The stay rating shows that, but it arrives months late, so a training set is built to predict it.
Retention rate and time to finish a task are measurable metrics that make the criterion usable.
Amazon's email team showed what a missing countervailing metric costs. The team got credit for every purchase after an email, so they ramped up sends and their metric rose. A side study valued each unsubscribe at a few dollars, and more than half of campaigns turned out net negative.
It led to a new default where the email offers to unsubscribe from that campaign type, such as author emails, which shrank the loss.
He starts in a team that ships often and agrees on its goal. Bing launched several times a day, while Office once shipped every three years. The first team also needs a clear overall evaluation criterion.
The microsoft.com team picked time on site, but half the room thought more time on a support site was good and half thought it bad. If a team cannot agree on the direction, the metric fails.
He makes platforms self-serve so each experiment costs almost nothing. Teams set up experiments, targets and metrics on a website with no help. Microsoft ran 20,000 to 25,000 experiments a year by 2019, about 100 new treatments started every working day.
Bing allowed up to 10,000 metrics in a scorecard, split into templates of about 2,000, such as one set for UI experiments and one for revenue. Airbnb's weaker analysis tools forced it to hire many data scientists to compensate.
The biggest speedup is an instant scorecard, then variance reduction. Results should arrive within about a day, not after a week's wait for a data scientist. Capping skewed metrics, such as purchases over $1,000 counted as $1,000, lowers variance.
At Airbnb a cap near 30 nights booked per month makes sense, since some agencies book tens or hundreds. CUPED uses pre-experiment data to cut variance and stays unbiased.
He tells new teams to buy part of the platform. At Amazon and Microsoft nothing existed, so they built in-house. Today third-party products are pretty good and trustworthy, and the real question is whether to build 10% or 90%.
He suggests asking where an organization sits on the six axes of a published crawl, walk, run, fly matrix to decide what to build next.
He says to start experimenting at tens of thousands of users. Detecting a 5% change on a retail site needs about 200,000 users. With tens of thousands of users, startups should chase 5 to 10 percent gains, not 1 percent.
Smaller teams should use the early period to build the platform and culture. One-off events such as mergers and acquisitions cannot be tested.
He keeps a searchable archive and reviews surprising results quarterly. Teams search past experiments by keyword before starting new ones. A quarterly meeting covers the most surprising results, where predicted and actual outcomes differ a lot, including losers.
The Windows indexer test improved relevance but drained laptop batteries, and the team documented it for the next design. Public sources such as GoodUI.org, with about 140 patterns, also help pick ideas to test.
Open-in-new-tab won at Microsoft, was forgotten, and had to be reintroduced at Airbnb. It ran around 2008 at Microsoft, first opening Hotmail in a new tab in the UK, then MSN search. Designers pushed back because users had not asked for it.
Airbnb later designs dropped the pattern. Kohavi reintroduced it and saw big improvements, and his lesson is to document winners and remember them.
Untested changes at Airbnb were often worse than assumed. Every change to search relevance was A/B tested before launch, while other teams tested some changes and skipped others. Some changes that launched untested turned out negative when he tested them.
He believes more experiments would have left the company better off, but admits the counterfactual is unknown.
He argues testing matters more during upheaval such as Covid. 'If in peacetime you're wrong two-thirds to 80 percent of the time why would you be suddenly right in wartime.' Results may need replicating later to check they generalize.
Airbnb's big investment in online experiences had unpromising early data and is now a footnote. He thinks Airbnb's revenue would have recovered the same way had it stayed the course.