CaddieOpen in CaddieDownload for Mac

What your tests and metrics quietly assume

Shared by Caddie Library · Updated
Chapters show their main points
Chapter 27 of 29 · How a field works and shifts
← The growth course
1. Finding Product-Market Fit
Test for real customer pull earlyPick a narrow niche to start inLaunch a simple first version fastLearn from users and customersRead early signs of product-market fit
2. Onboarding and Activation
Structure onboarding as a storyAsk and use onboarding questionsGet users to the aha moment fastDecide how long onboarding should beBuild quiz funnels for paid traffic
3. Paywalls and Pricing
When to choose a hard paywall over freemiumDesign a paywall that convertsPrice annual and monthly plansSet, raise and localize pricesRun free trials and free tastes
4. Business Models and Revenue
Structure subscription tiersKeep a free tier that pays offCharge for usage and AI creditsPair hardware with a subscriptionAdd ads, affiliates and partners
5. Retention and Engagement
Measure retention honestlyBuild habit loops that bring users backUse notifications and lifecycle messagesReduce churn and handle cancellationsBuild social loops and community
6. Brand and Positioning
Position against bigger incumbentsName and describe your productEarn trust in high-stakes categoriesBorrow credibility from celebrities and experts
7. Organic Content and Creators
Find, vet and recruit creatorsPay creators fairly and safelyBrief and manage creatorsFind and copy viral formatsScale one format across many accountsRank in AI search and SEO
8. Paid, App Store and Viral Growth
Make ads that convertTurn organic winners into paid adsStructure and scale Meta campaignsTell if paid acquisition is profitableWin in the App StoreBuild viral and referral loops
9. Experimentation and Analytics
Run trustworthy experimentsJudge test results correctlyRun more experiments fasterWhat metrics to track and how to define them
10. Building AI Products
Build for the models of tomorrowDesign AI interfaces beyond chatDesign AI agents people can trustMeasure AI quality with evalsControl AI costs and pricingBuild products with AI coding tools
11. Teams, Hiring and Funding
Hire people who thrive on ambiguityOrganize small product teamsRun exploratory bets and kill themWork as a team with AI agentsWhen to raise money or sell your company
12. Founder Principles
Pick a mission worth building a company aroundWhy to trust your own judgment over the consensusTurn failure and setbacks into the next stepLearn from rivals and copy what works
13. Choosing Where to Compete
Make low cost your competitive advantageCreate a market instead of fighting for shareWhy scarcity and brand can beat volumeBuild an advantage that feeds itselfAttack a bigger rival where it cannot followWhen to stay focused on the core business
14. Building and Leading the Company
Hire great people and give them roomAlign pay and ownership with the missionKeep your culture and standards as you growBuild candor and fast decision-makingWhy leaders must stay close to the work and customersChoose and prepare a successor
15. Capital and Deals
Fund growth without losing controlAcquire companies and keep what makes them workWhat to do with cash: reinvest, buy back or holdWhy to keep a cash cushion and avoid heavy debtStructure ownership to protect your missionNegotiate deals that keep your leverage
16. Finding Ideas and Thinking
How to find ideas worth writing aboutWhy writing is how you find out what you thinkHow to research so the writing carries feelingHow to read like a writer
17. Structure and Argument
How to open so readers keep goingHow to find the shape before you draftHow to keep readers curious with questions and cluesHow to argue without preachingHow to end so the last page lingers
18. Clear Sentences and Voice
How to write plainly for a reader who knows nothingHow to choose specific details over generic onesHow to sound like yourself on the pageHow to shape rhythm and phrases people remember
19. Characters and Storytelling
How to build characters readers care aboutWhat makes a scene move: want, obstacle and changeHow to write dialogue that sounds real but stays shortHow to make readers feel without telling themHow to turn information into a story
20. Revising, Habits and Readers
How to cut and rewrite your own draftHow to get feedback you can useHow to build a writing routine that lastsHow to get unstuck when a draft stallsHow to use AI without losing your voiceHow to write for one specific reader
21. Why people play games
How to design around the feeling you wantHow to find a game idea with a hookHow to decide who your game is forWhy rewards and motivation need careful designHow to use mystery and curiosity to pull players on
22. Core mechanics and loops
How to build a game around one core verbHow to give each mechanic several jobsHow to add depth with holds, timing and chainsHow to use feedback loops to keep games balancedHow to use randomness without frustrating playersHow to make synergies feel powerful and earned
23. Challenge, difficulty and feel
How to make hard games feel fairHow to tune difficulty with many leversHow to offer assists without breaking your intentHow to make death and retries painlessHow to make movement and controls feel goodHow to add camera and juice that sell impact
24. Teaching players and designing levels
How to teach mechanics without a tutorialHow to introduce mechanics one at a timeHow to guide players without markersHow to build levels around one mechanicHow to design puzzles with a catch and a revelationHow to design dungeons from locks and keys
25. Progression, systems and story
How to gate exploration without trapping playersHow to design unlocks and skill treesHow to let systems create emergent playHow to design enemy AI players can readHow to tell stories through mechanicsHow to make player choices and consequences matter
26. Prototyping, testing and accessibility
How to prototype before you commitHow to scope and cut a game to finish itHow to run playtests that reveal real problemsHow to turn player feedback into the right fixHow to build accessibility in from the start
27. How a field works and shifts
Make steady progress inside a working frameworkSpot a surprising result worth chasingWhen to abandon a model that mostly worksWhat your tests and metrics quietly assumeWin agreement for a new approach
28. Organising a place for discovery
Build a team where disciplines collideFund open-ended research and keep it usefulMove a promising result into real useLearn from missed chances and outside threats
29. Discovering together in a small group
Share work, credit and priority in a groupFind the cause when results behave erraticallyGet a new result noticed and adopted

1 The bottom line

Every standard test, metric and shared term assumes which outcomes can happen, so audit them before trusting the usual reading. Without that audit you miss what you are looking for, argue past colleagues, and forget what past changes dropped.

2 Key takeaways

  • Your instrument decides which outcomes can show up. A standard procedure can quietly commit you to the nonexistence of what you seek.
  • Your metrics are chosen by your framing, not given by nature. A different framing of the same object leads to different measurements and different laws.
    • An Aristotelian watching a swinging stone measured weight, height raised, time to rest and medium resistance. Galileo measured weight, radius, angular displacement and time per swing Kuhn.
  • Shared words do not mean the same thing to every group. Opposite confident answers may come from different training, and new methods that redefine terms meet hostility.
    • Asked whether a helium atom is a molecule, a chemist said yes and a physicist said no Kuhn.
    • The Joule-Lenz Law used settled meanings and was accepted. Ohm's Law required redefining current and resistance and met strong opposition Kuhn.
  • Tidy histories and progress stories hide the real drivers and the losses. What a shift dropped goes unseen unless you list it.
    • Dalton gave three incompatible accounts of his atomism Kuhn.
    • Lavoisier's chemistry stopped asking why metals are so alike, and questions about the qualities of substances returned in the twentieth century Kuhn.

The operations and measurements that a scientist undertakes in the laboratory are not “the given” of experience but rather “the collected with difficulty.”

Thomas S. Kuhn, The Structure of Scientific Revolutions

2.1 Audit the instrument before you accept the reading

Same gas, different proportion, different verdict
Priestley and Lavoisier both trusted the usual dose and misread oxygen; only varying the method revealed it.

A standard test only reports the outcomes it was built to show.

  • Ask which outcomes your standard test or dashboard allows, and which it rules out. Kuhn
    • Choosing an instrument assumes only certain outcomes can occur, so write those assumptions down.
    • Vary the method or the proportions before accepting the usual reading.
    • Priestley and Lavoisier both used the nitric oxide test and misidentified oxygen as ordinary air. Only when Priestley used four times the nitric oxide did the residue nearly vanish.
  • Check where your tests are pointed, and what lies outside that range. Kuhn
    • A standard procedure can commit you to the nonexistence of what you are looking for.
    • Fission was identified late because chemical tests aimed at the upper end of the periodic table.

2.2 Question what your framing makes you measure

One swinging stone, two framings, different measurements
AristotelianGalileo
WeightWeightWeight
Length measureHeight raisedRadius
Motion measureTime to come to restAngular displacement
Extra factorMedium resistanceTime per swing
What it yieldsNo pendulum lawsPendulum laws
The metrics came from each observer's model, not from the stone itself.

What you track is picked because your model says it is promising, so a new framing can show what the old one hid.

  • Treat every metric as a choice, not a neutral fact. Kuhn
    • Ask what your framing makes you track and what it hides.
    • An Aristotelian watching a swinging stone measured weight, height raised, time to rest and medium resistance. Galileo measured weight, radius, angular displacement and time per swing, the data that yield the pendulum laws.
  • Re-express your existing data in the form a new model suggests. Kuhn
    • The way data are recorded decides which regularities show, so rewrite old records under the new framing.
    • Expect a long period of reworking measurements that do not yet fit.
    • Chemists stopped writing that two carbon oxides held 56 per cent and 72 per cent oxygen. They wrote that one weight of carbon combines with 1.3 or 2.6 weights of oxygen, and a 2:1 ratio then stood out across many reactions.

2.3 Check that shared terms mean the same thing

Opposite expert answers can come from different training
ChemistPhysicist
AnswerYesNo
Trained onKinetic gas theoryMolecular spectra
ReasonBehaves like a gas moleculeShows no molecular spectrum
What 'molecule' meansUnit in gas behaviourThing with a molecular spectrum
Before treating a disagreement as a factual dispute, check what each side learned the term from.

One framework can anchor several groups without meaning the same thing to each.

  • When two experts give opposite confident answers, check what each trained on before calling it a factual dispute. Kuhn
    • Look at the examples each specialist learned from.
    • Expect that a change to one application may upend one team and leave others untouched.
    • Asked whether a helium atom is a molecule, a chemist said yes from kinetic gas theory. A physicist said no because it shows no molecular spectrum.
  • Name it openly when a new method redefines terms the team treats as given. Kuhn
    • Ask whether the result only adds information to settled terms, or only works if familiar terms change meaning.
    • State the redefinition instead of letting it surface as unexplained hostility.
    • The Joule-Lenz Law used settled meanings and was accepted. Ohm's Law required redefining current and resistance and met strong opposition.

2.4 Keep honest records and list what each change dropped

Retrospectives make every shift look like a straight line of gains, so keep the real questions and the real losses on file.

  • Record what you actually asked at each stage, not the clean story. Kuhn
    • Histories, onboarding docs and retrospectives get rewritten after each shift. That hides the real driver, which is reformulating the questions.
    • Check original sources and keep honest notes of the questions as they were.
    • Dalton gave three incompatible accounts of his atomism. All implied an early interest in combining proportions that seem to have occurred to him only near the end of his work.
  • Audit what each major change dropped, not only what it gained. Kuhn
    • Winners describe each shift as pure progress, so the losses go unseen.
    • List the questions and capabilities the new approach abandoned. Some return later under a different model.
    • Lavoisier's chemistry stopped asking why metals are so alike, a question phlogistic chemistry had answered. Questions about the qualities of substances returned in the twentieth century.

Sources

Part of How a field works and shifts in the growth library.