Ramesh Johari
Professor at Stanford University, data science and marketplaces researcher, advisor to Airbnb, Uber, Bumble, Stitch Fix and Upwork
Where to hear and read about Ramesh Johari1 episode or test
Ramesh Johari is a Stanford professor who studies online marketplaces and has advised Airbnb, Uber, Bumble, Stitch Fix and Upwork, after starting in data science at oDesk. The thing most worth learning from him is to judge every marketplace decision by the friction it removes and the causal effect it has, and to treat learning as a cost worth paying.
A marketplace business never starts as a marketplace business.
Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor)
Ramesh Johari: the story and the lessons
1Who Ramesh Johari is
Johari pairs Stanford research with hands-on work inside large marketplaces. He teaches data science methods and practice at Stanford, and his research focus is the design and operation of online marketplaces. He has advised or worked with Airbnb, Uber, Stripe, Bumble, Stitch Fix and Upwork.
The list of companies goes on past those six. He describes himself as a professor, not a scaled marketplace entrepreneur, so he offers thoughts rather than a formula.
He entered marketplace data work through oDesk in 2012. He joined oDesk as a research scientist, then directed its data science team. He was looking for experts on how to think about data and marketplaces, and invited Riley Newman, later Airbnb's head of data science, to speak there.
He calls those the early days of the industry and has stayed in touch with Newman since. At Airbnb he later led the review system flows for a while.
Johari defines a marketplace by the friction it removes, not what it lists. Airbnb and Uber do not sell rooms or rides. They sell the removal of transaction costs, such as finding a willing host or a driver right now. Hosts and guests, or drivers and riders, are both the platform's customers.
Economists call these frictions transaction costs, and they are why markets fail. "Uber and Airbnb are selling you the taking away of something."
2Johari's advice on starting a marketplace
Johari says founders fail by acting like a marketplace too early. He sees the most common failure as thinking too much about being a marketplace before being one. Cleaners and car washes are often cited as failed marketplaces, but he doubts the industry is the cause and points to the founders' early mindset.
Telling users you will find all the drivers when you have three is not solving a friction.
Johari tests for a marketplace by asking if both sides have scaled. The smell test is whether you have many buyers and many sellers, only one of the two, or neither. With one side, keep growing it or use it to attract the other. With neither, worry about scaling one side.
Uber turned a subsidized driver base into rider demand with free ride coupons at events and parties. Johari says there is no shame in not being a marketplace, and founders must let the ego go.
He starts with a bespoke value proposition that needs no scale. oDesk first sold trust in remote work, with screenshots and tracking so workers could verify hours and employers could see what they paid for. Matching came later.
UrbanSitter began by letting parents pay sitters by credit card, then used Facebook networks for trusted introductions, and only later charged for finding and contacting sitters.
Johari argues every founder may become a marketplace founder. He says almost no business is untouched by online transactions, so becoming a platform is a choice made after growth. OpenAI looked nothing like a marketplace, yet plugin creators and users now form two sides on its platform.
Lenny's view is that 90% of a marketplace founder's problems are ordinary startup problems, such as how to grow. Johari goes further than that.
Substack added a marketplace layer by solving writers' biggest pain. It began as a publishing tool, then built ways to send readers to writers because writers most need subscribers. For one large newsletter, over 80% of subscribers come from Substack's network.
Johari sees this as the pattern he describes: the marketplace element came after a pain point.
3Monetization traps Johari warns about
Johari warns that early pricing choices can tie your hands later. oDesk took a constant fraction of dollars crossing the platform, which fit its monitoring tools early on. As relationships matured, the value came from the match, yet the platform still took 10%.
A Thumbtack worker delivered from IKEA handed over a business card, and the platform got one lead. After merging with Elance to become Upwork, the company had to rethink pricing for long relationships. "Early commitments to a particular pricing scheme, particular monetization, can really tie your hands."
He cites eBay as a warning about changing rules for mature sellers. eBay's growing list of fine-grained fees strained its relationship with sellers who had built livelihoods on it. For many, eBay was their entire business.
Rewriting the rules felt like breaking a social contract built over a long time. Substack, by contrast, deepened its contract with creators.
Johari treats employees versus contractors as the old market versus firm question. Ronald Coase asked why firms exist if markets match labor efficiently, and answered with transaction costs. Stitch Fix stylists feel magical because the relationship does not feel like a new freelancer each time.
Curation of the labor pool does not have to mean full employment. Sometimes the best fix for a friction is tightly controlled labor, not an open marketplace.
4How Johari wants data science to make decisions
Johari says data science serves a cycle of finding, making and learning from matches. Every marketplace in every vertical faces the same three problems and solves them with algorithms. Search and ranking help find matches, triage helps make them, and ratings and passive signals feed the next round.
At oDesk, with remote hiring, all an employer had was the application. A stay teaches the guest about the host and the host about the guest.
He judges ranking algorithms by outcomes caused, not by predicting the past. At oDesk in 2012 he was asked to predict which applicant would be hired. Ranking applicants by that prediction only reproduced past patterns and did not show value added.
For Airbnb search, the test is whether one algorithm leads to more bookings than another. "Prediction is inherently about correlation. But when we ask people to make decisions, we're asking them to think about causation."
Johari targets promotions by incremental lift, not highest lifetime value. Sending the best promotions to the highest LTV customers is safe but rewards a pattern. The real question is how much more they spend because of the promotion.
"I'm not interested in their absolute LTV. I'm absolutely interested in the difference in their LTV because I sent them this promotion."
He puts data scientists where the biggest friction is. Where the platform sets price, as at Uber, Lyft and DoorDash, pricing needs strong data scientists. Where hosts set prices, as at Airbnb, that work matters less. If finding the needle in the haystack is the biggest friction, put people on search.
Johari is a big believer in experiments but says you can't test everything. Without experimenting there is no way to know the difference a decision makes. Asked about micro-optimizing and local maxima, he replies that you can't experiment your way out of everything.
5Johari on building a culture that values learning
Johari says experiment-heavy teams drift toward small, safe, overlong tests. Teams are risk averse about what to test and how long to run it. Data scientists are often judged on wins per quarter, and small changes win more easily.
He points to the Microsoft paper A/B Testing with Fat Tails, which says to try riskier ideas and run them for less time. Ronald Fisher's experiments were hypothesis driven, not about winners and losers.
He asks data scientists to report what they learned, not only whether a test won. Each test should say what it teaches about a funnel, guest or host preferences, or demand elasticity. The hypotheses can be written into the experiment or launch doc.
Data scientists measured narrowly on impact end up working at the margins of the business. He still holds that measuring impact is critical. "You really want to think not in terms of winning and losing, but learning."
Johari says learning costs something and teams must budget for it. Once a treatment wins, giving samples to control looks silly, like staring at a bill after a big meal. Rewarding only shipped winners tells people time spent testing failures was wasted.
A marketing manager's unauthorized holdout at a real estate platform cost a couple million dollars but revealed what his team was worth. "You have to pay to learn."
He prefers Bayesian priors so every experiment builds on past learning. Standard frequentist analysis ignores everything learned before, so a team may analyze the next button test from scratch after 1,000 earlier ones. A prior built from past tests lets a failed experiment still move how all future tests are read.
That rewards people for contributing to the prior, a positive externality for the business. "Your experiment that failed actually moved our prior."
Johari wants leaders to be quantified rather than just data-driven. At Airbnb the Superhost test showed no business impact, only slightly happier hosts. Its real aim is retaining badged hosts over time, which a short test can't show.
He suggests leadership state competing beliefs about retention, combine them with the test result like a prediction market, and decide whether the bet is still worth making. One significant result should not override what you know about the business.
He expects marketplace changes to create winners and losers. Superhosts are limited in number, so his prior was that bookings would fall, not rise. In one company, steering new suppliers to experienced partners hurt existing partners, and metrics whipsawed month to month.
The question is whether the winners matter more to the business than the losers. "Many of the changes that are most consequential create winners and losers."
6Ramesh Johari on rating and review systems
Johari renorms ratings to fight inflation. Ratings drift upward because leaving a good one is easy and a bad one feels harsh. John Horton's papers show median ratings rising on oDesk, Uber and others.
Making the top rating mean "exceeded expectations" helps. Airbnb asked guests to compare a stay with past ones, which is easier than giving four stars.
He argues plain averages punish newcomers on their first review. A Yelp restaurant with 10,000 reviews won't move, but a first negative can sink a new entrant. Early eBay work found a negative first rating cut immediate expected revenue by 8%.
Averaging with a prior belief lifts a newcomer slightly, and some platforms hide ratings until a few accumulate. He calls rating systems understudied.
Johari counts reviews that were never left as a signal. Steve Tadelis's paper with eBay colleagues on "effective percent positive" counts missing ratings. It predicted a seller's later performance much better than the plain share of positive ratings.
Skipping a review is far easier than leaving a bad one, so silence often hides a poor experience.
Johari led Airbnb's double-blind reviews, and the biggest effect was unplanned. Neither side sees the other's review until they leave their own. The aim was more honest reviews, but the main result was a higher review rate, because the email nudged people to write one to see theirs.
He calls this launch one of the things he is most proud of.
7Johari on AI, hiring and slowing down
Johari says AI makes people more important to data science, not less. An analyst can now generate 10 or 100 explanations, and a campaign test that once had 10 creatives may have 1,000. Someone must choose what to focus on and decide what is good enough.
"Humans have actually become far more important to the productive data science loop, not far less." His Stanford students use LLMs on assignments every week.
He asks candidates what impact their plan would have if it all worked. Many people have not thought past the plan, because we focus on the short term. The answer also shows more of the areas their work could touch.
He would not ask a coding question after November 2022, once AI could help with code.
Johari's motto is to slow down and build mental models. A marketplace founder needs a model of what makes people stay or leave and what makes matches work. He says roadmaps, execution and academic writing have grown faster at the expense of thinking about structure.
He recommends David Freedman's "shoe leather statistics": at oDesk that meant looking at actual jobs and the product before doing data science on it. He also names How to Lie with Statistics by Darrell Huff and Four Thousand Weeks by Oliver Burkeman.
Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor)
Marketplaces sell the removal of friction between two sides, so early on founders should solve a concrete problem and scale one side before calling themselves a marketplace. Data science helps most when it focuses on decisions and causation rather than prediction, with experiments built for learning and rating systems designed with priors.
Johari pairs Stanford research with hands-on work inside large marketplaces. He teaches data science methods and practice at Stanford, and his research focus is the design and operation of online marketplaces. He has advised or worked with Airbnb, Uber, Stripe, Bumble, Stitch Fix and Upwork.
The list of companies goes on past those six. He describes himself as a professor, not a scaled marketplace entrepreneur, so he offers thoughts rather than a formula.
He entered marketplace data work through oDesk in 2012. He joined oDesk as a research scientist, then directed its data science team. He was looking for experts on how to think about data and marketplaces, and invited Riley Newman, later Airbnb's head of data science, to speak there.
He calls those the early days of the industry and has stayed in touch with Newman since. At Airbnb he later led the review system flows for a while.
Johari defines a marketplace by the friction it removes, not what it lists. Airbnb and Uber do not sell rooms or rides. They sell the removal of transaction costs, such as finding a willing host or a driver right now. Hosts and guests, or drivers and riders, are both the platform's customers.
Economists call these frictions transaction costs, and they are why markets fail. "Uber and Airbnb are selling you the taking away of something."
Johari says founders fail by acting like a marketplace too early. He sees the most common failure as thinking too much about being a marketplace before being one. Cleaners and car washes are often cited as failed marketplaces, but he doubts the industry is the cause and points to the founders' early mindset.
Telling users you will find all the drivers when you have three is not solving a friction.
Johari tests for a marketplace by asking if both sides have scaled. The smell test is whether you have many buyers and many sellers, only one of the two, or neither. With one side, keep growing it or use it to attract the other. With neither, worry about scaling one side.
Uber turned a subsidized driver base into rider demand with free ride coupons at events and parties. Johari says there is no shame in not being a marketplace, and founders must let the ego go.
He starts with a bespoke value proposition that needs no scale. oDesk first sold trust in remote work, with screenshots and tracking so workers could verify hours and employers could see what they paid for. Matching came later.
UrbanSitter began by letting parents pay sitters by credit card, then used Facebook networks for trusted introductions, and only later charged for finding and contacting sitters.
Johari argues every founder may become a marketplace founder. He says almost no business is untouched by online transactions, so becoming a platform is a choice made after growth. OpenAI looked nothing like a marketplace, yet plugin creators and users now form two sides on its platform.
Lenny's view is that 90% of a marketplace founder's problems are ordinary startup problems, such as how to grow. Johari goes further than that.
Substack added a marketplace layer by solving writers' biggest pain. It began as a publishing tool, then built ways to send readers to writers because writers most need subscribers. For one large newsletter, over 80% of subscribers come from Substack's network.
Johari sees this as the pattern he describes: the marketplace element came after a pain point.
Johari warns that early pricing choices can tie your hands later. oDesk took a constant fraction of dollars crossing the platform, which fit its monitoring tools early on. As relationships matured, the value came from the match, yet the platform still took 10%.
A Thumbtack worker delivered from IKEA handed over a business card, and the platform got one lead. After merging with Elance to become Upwork, the company had to rethink pricing for long relationships. "Early commitments to a particular pricing scheme, particular monetization, can really tie your hands."
He cites eBay as a warning about changing rules for mature sellers. eBay's growing list of fine-grained fees strained its relationship with sellers who had built livelihoods on it. For many, eBay was their entire business.
Rewriting the rules felt like breaking a social contract built over a long time. Substack, by contrast, deepened its contract with creators.
Johari treats employees versus contractors as the old market versus firm question. Ronald Coase asked why firms exist if markets match labor efficiently, and answered with transaction costs. Stitch Fix stylists feel magical because the relationship does not feel like a new freelancer each time.
Curation of the labor pool does not have to mean full employment. Sometimes the best fix for a friction is tightly controlled labor, not an open marketplace.
Johari says data science serves a cycle of finding, making and learning from matches. Every marketplace in every vertical faces the same three problems and solves them with algorithms. Search and ranking help find matches, triage helps make them, and ratings and passive signals feed the next round.
At oDesk, with remote hiring, all an employer had was the application. A stay teaches the guest about the host and the host about the guest.
He judges ranking algorithms by outcomes caused, not by predicting the past. At oDesk in 2012 he was asked to predict which applicant would be hired. Ranking applicants by that prediction only reproduced past patterns and did not show value added.
For Airbnb search, the test is whether one algorithm leads to more bookings than another. "Prediction is inherently about correlation. But when we ask people to make decisions, we're asking them to think about causation."
Johari targets promotions by incremental lift, not highest lifetime value. Sending the best promotions to the highest LTV customers is safe but rewards a pattern. The real question is how much more they spend because of the promotion.
"I'm not interested in their absolute LTV. I'm absolutely interested in the difference in their LTV because I sent them this promotion."
He puts data scientists where the biggest friction is. Where the platform sets price, as at Uber, Lyft and DoorDash, pricing needs strong data scientists. Where hosts set prices, as at Airbnb, that work matters less. If finding the needle in the haystack is the biggest friction, put people on search.
Johari is a big believer in experiments but says you can't test everything. Without experimenting there is no way to know the difference a decision makes. Asked about micro-optimizing and local maxima, he replies that you can't experiment your way out of everything.
Johari says experiment-heavy teams drift toward small, safe, overlong tests. Teams are risk averse about what to test and how long to run it. Data scientists are often judged on wins per quarter, and small changes win more easily.
He points to the Microsoft paper A/B Testing with Fat Tails, which says to try riskier ideas and run them for less time. Ronald Fisher's experiments were hypothesis driven, not about winners and losers.
He asks data scientists to report what they learned, not only whether a test won. Each test should say what it teaches about a funnel, guest or host preferences, or demand elasticity. The hypotheses can be written into the experiment or launch doc.
Data scientists measured narrowly on impact end up working at the margins of the business. He still holds that measuring impact is critical. "You really want to think not in terms of winning and losing, but learning."
Johari says learning costs something and teams must budget for it. Once a treatment wins, giving samples to control looks silly, like staring at a bill after a big meal. Rewarding only shipped winners tells people time spent testing failures was wasted.
A marketing manager's unauthorized holdout at a real estate platform cost a couple million dollars but revealed what his team was worth. "You have to pay to learn."
He prefers Bayesian priors so every experiment builds on past learning. Standard frequentist analysis ignores everything learned before, so a team may analyze the next button test from scratch after 1,000 earlier ones. A prior built from past tests lets a failed experiment still move how all future tests are read.
That rewards people for contributing to the prior, a positive externality for the business. "Your experiment that failed actually moved our prior."
Johari wants leaders to be quantified rather than just data-driven. At Airbnb the Superhost test showed no business impact, only slightly happier hosts. Its real aim is retaining badged hosts over time, which a short test can't show.
He suggests leadership state competing beliefs about retention, combine them with the test result like a prediction market, and decide whether the bet is still worth making. One significant result should not override what you know about the business.
He expects marketplace changes to create winners and losers. Superhosts are limited in number, so his prior was that bookings would fall, not rise. In one company, steering new suppliers to experienced partners hurt existing partners, and metrics whipsawed month to month.
The question is whether the winners matter more to the business than the losers. "Many of the changes that are most consequential create winners and losers."
Johari renorms ratings to fight inflation. Ratings drift upward because leaving a good one is easy and a bad one feels harsh. John Horton's papers show median ratings rising on oDesk, Uber and others.
Making the top rating mean "exceeded expectations" helps. Airbnb asked guests to compare a stay with past ones, which is easier than giving four stars.
He argues plain averages punish newcomers on their first review. A Yelp restaurant with 10,000 reviews won't move, but a first negative can sink a new entrant. Early eBay work found a negative first rating cut immediate expected revenue by 8%.
Averaging with a prior belief lifts a newcomer slightly, and some platforms hide ratings until a few accumulate. He calls rating systems understudied.
Johari counts reviews that were never left as a signal. Steve Tadelis's paper with eBay colleagues on "effective percent positive" counts missing ratings. It predicted a seller's later performance much better than the plain share of positive ratings.
Skipping a review is far easier than leaving a bad one, so silence often hides a poor experience.
Johari led Airbnb's double-blind reviews, and the biggest effect was unplanned. Neither side sees the other's review until they leave their own. The aim was more honest reviews, but the main result was a higher review rate, because the email nudged people to write one to see theirs.
He calls this launch one of the things he is most proud of.
Johari says AI makes people more important to data science, not less. An analyst can now generate 10 or 100 explanations, and a campaign test that once had 10 creatives may have 1,000. Someone must choose what to focus on and decide what is good enough.
"Humans have actually become far more important to the productive data science loop, not far less." His Stanford students use LLMs on assignments every week.
He asks candidates what impact their plan would have if it all worked. Many people have not thought past the plan, because we focus on the short term. The answer also shows more of the areas their work could touch.
He would not ask a coding question after November 2022, once AI could help with code.
Johari's motto is to slow down and build mental models. A marketplace founder needs a model of what makes people stay or leave and what makes matches work. He says roadmaps, execution and academic writing have grown faster at the expense of thinking about structure.
He recommends David Freedman's "shoe leather statistics": at oDesk that meant looking at actual jobs and the product before doing data science on it. He also names How to Lie with Statistics by Darrell Huff and Four Thousand Weeks by Oliver Burkeman.