Optimizely

Subversive · May 29

How Gamma Grew to $50M in ARR in 2 Years

Noronha joined after its Series A, when it grew from about 100 to almost 500 people, and met his Gamma co-founders and much of the early team there.

Lenny's Podcast · Jul 27

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

It lost trust by letting users stop at the first significant p-value, which inflated false positives to about 30% instead of 5%. A post titled 'Optimizely almost got me fired' described A/A tests coming out significant too often. Kohavi and Ramesh Johari flagged the flaw and the company fixed it.

Tips

• Buy a trustworthy experimentation platform instead of building from scratch▶ 1:06:27Early Optimizely let people stop when p-value hit significance, inflating false positives to an estimated 30% instead of 5%.
• Make trust the core requirement of an experimentation platform▶ 51:45Optimizely's early naive statistics led users to see A/A tests called significant and lose faith in the product.

Lenny's Podcast · Jan 12

How to price your product | Naomi Ionita (Menlo Ventures)

Targeted marketing personas and focused on click-through metrics rather than business outcomes. Naomi was an early buyer.

Lenny's Podcast · Aug 21

The nature of product | Marty Cagan, Silicon Valley Product Group

Cagan cites small Optimizely A/B tests and workflow tweaks as optimization that won't innovate. Relying on them after founders leave marks the end of real discovery.

Tips

• Do real discovery instead of only running low-risk A/B tests▶ 22:50Only about 20% of the features executives prioritize generate any positive return.