CaddieOpen in CaddieDownload for Mac

When to abandon a model that mostly works

Shared by Caddie Library · Updated
Chapters show their main points
Chapter 27 of 29 · How a field works and shifts
← The growth course
1. Finding Product-Market Fit
Test for real customer pull earlyPick a narrow niche to start inLaunch a simple first version fastLearn from users and customersRead early signs of product-market fit
2. Onboarding and Activation
Structure onboarding as a storyAsk and use onboarding questionsGet users to the aha moment fastDecide how long onboarding should beBuild quiz funnels for paid traffic
3. Paywalls and Pricing
When to choose a hard paywall over freemiumDesign a paywall that convertsPrice annual and monthly plansSet, raise and localize pricesRun free trials and free tastes
4. Business Models and Revenue
Structure subscription tiersKeep a free tier that pays offCharge for usage and AI creditsPair hardware with a subscriptionAdd ads, affiliates and partners
5. Retention and Engagement
Measure retention honestlyBuild habit loops that bring users backUse notifications and lifecycle messagesReduce churn and handle cancellationsBuild social loops and community
6. Brand and Positioning
Position against bigger incumbentsName and describe your productEarn trust in high-stakes categoriesBorrow credibility from celebrities and experts
7. Organic Content and Creators
Find, vet and recruit creatorsPay creators fairly and safelyBrief and manage creatorsFind and copy viral formatsScale one format across many accountsRank in AI search and SEO
8. Paid, App Store and Viral Growth
Make ads that convertTurn organic winners into paid adsStructure and scale Meta campaignsTell if paid acquisition is profitableWin in the App StoreBuild viral and referral loops
9. Experimentation and Analytics
Run trustworthy experimentsJudge test results correctlyRun more experiments fasterWhat metrics to track and how to define them
10. Building AI Products
Build for the models of tomorrowDesign AI interfaces beyond chatDesign AI agents people can trustMeasure AI quality with evalsControl AI costs and pricingBuild products with AI coding tools
11. Teams, Hiring and Funding
Hire people who thrive on ambiguityOrganize small product teamsRun exploratory bets and kill themWork as a team with AI agentsWhen to raise money or sell your company
12. Founder Principles
Pick a mission worth building a company aroundWhy to trust your own judgment over the consensusTurn failure and setbacks into the next stepLearn from rivals and copy what works
13. Choosing Where to Compete
Make low cost your competitive advantageCreate a market instead of fighting for shareWhy scarcity and brand can beat volumeBuild an advantage that feeds itselfAttack a bigger rival where it cannot followWhen to stay focused on the core business
14. Building and Leading the Company
Hire great people and give them roomAlign pay and ownership with the missionKeep your culture and standards as you growBuild candor and fast decision-makingWhy leaders must stay close to the work and customersChoose and prepare a successor
15. Capital and Deals
Fund growth without losing controlAcquire companies and keep what makes them workWhat to do with cash: reinvest, buy back or holdWhy to keep a cash cushion and avoid heavy debtStructure ownership to protect your missionNegotiate deals that keep your leverage
16. Finding Ideas and Thinking
How to find ideas worth writing aboutWhy writing is how you find out what you thinkHow to research so the writing carries feelingHow to read like a writer
17. Structure and Argument
How to open so readers keep goingHow to find the shape before you draftHow to keep readers curious with questions and cluesHow to argue without preachingHow to end so the last page lingers
18. Clear Sentences and Voice
How to write plainly for a reader who knows nothingHow to choose specific details over generic onesHow to sound like yourself on the pageHow to shape rhythm and phrases people remember
19. Characters and Storytelling
How to build characters readers care aboutWhat makes a scene move: want, obstacle and changeHow to write dialogue that sounds real but stays shortHow to make readers feel without telling themHow to turn information into a story
20. Revising, Habits and Readers
How to cut and rewrite your own draftHow to get feedback you can useHow to build a writing routine that lastsHow to get unstuck when a draft stallsHow to use AI without losing your voiceHow to write for one specific reader
21. Why people play games
How to design around the feeling you wantHow to find a game idea with a hookHow to decide who your game is forWhy rewards and motivation need careful designHow to use mystery and curiosity to pull players on
22. Core mechanics and loops
How to build a game around one core verbHow to give each mechanic several jobsHow to add depth with holds, timing and chainsHow to use feedback loops to keep games balancedHow to use randomness without frustrating playersHow to make synergies feel powerful and earned
23. Challenge, difficulty and feel
How to make hard games feel fairHow to tune difficulty with many leversHow to offer assists without breaking your intentHow to make death and retries painlessHow to make movement and controls feel goodHow to add camera and juice that sell impact
24. Teaching players and designing levels
How to teach mechanics without a tutorialHow to introduce mechanics one at a timeHow to guide players without markersHow to build levels around one mechanicHow to design puzzles with a catch and a revelationHow to design dungeons from locks and keys
25. Progression, systems and story
How to gate exploration without trapping playersHow to design unlocks and skill treesHow to let systems create emergent playHow to design enemy AI players can readHow to tell stories through mechanicsHow to make player choices and consequences matter
26. Prototyping, testing and accessibility
How to prototype before you commitHow to scope and cut a game to finish itHow to run playtests that reveal real problemsHow to turn player feedback into the right fixHow to build accessibility in from the start
27. How a field works and shifts
Make steady progress inside a working frameworkSpot a surprising result worth chasingWhen to abandon a model that mostly worksWhat your tests and metrics quietly assumeWin agreement for a new approach
28. Organising a place for discovery
Build a team where disciplines collideFund open-ended research and keep it usefulMove a promising result into real useLearn from missed chances and outside threats
29. Discovering together in a small group
Share work, credit and priority in a groupFind the cause when results behave erraticallyGet a new result noticed and adopted

1 The bottom line

Abandon a model when its patches add complexity faster than accuracy and a ready rival can replace it. Dropping a flawed model with nothing behind it leaves the team unable to work.

2 Key takeaways

  • One anomaly is not a reason to drop a model. Every model leaves some problems unsolved.
    • For sixty years Newtonian theory predicted only half the observed motion of the moon's perigee, and in 1750 Clairaut showed the error lay in the mathematics of the application, so the theory stood Thomas S. Kuhn.
  • Watch whether each fix adds more complexity than accuracy. A model in trouble gets patched, and patches break things elsewhere.
    • Ptolemaic astronomers fixed each planetary mismatch with more circles, and Copernicus called the inherited tradition a monster Thomas S. Kuhn.
    • Metal ash weighed more after burning, and phlogiston theory had to explain it with "negative weight" Jenny Uglow.
  • Read the team as well as the data. Competing versions of the core model and noisy method debates show the shared base is gone.
    • By the early 1770s there were nearly as many versions of phlogiston theory as there were pneumatic chemists Thomas S. Kuhn.
  • Test a rival on its facts first, then on how well it binds them. A neat story built on shaky facts is worth nothing.
    • Darwin read the clearest statement of the French chemistry, checked whether its facts held, then asked whether the theory tied them together Jenny Uglow.
  • Choose between the two real options on the table, never against an ideal. No significant model fits perfectly, but one of two can fit better.
    • Neither Priestley's nor Lavoisier's theory matched observations precisely, yet few contemporaries took more than a decade to judge Lavoisier's the better fit Thomas S. Kuhn.

To reject one paradigm without simultaneously substituting another is to reject science itself.

Thomas S. Kuhn, The Structure of Scientific Revolutions

2.1 Read the warning signs

Each patch adds more circles than accuracy
When fixing one mismatch creates errors elsewhere, the framework is failing, as with phlogiston's 'negative weight'.

A failing model shows itself in its patches, in how many versions of it exist, and in how loudly the team argues about method.

  • Track whether each fix adds complexity faster than it adds accuracy. Each patch fixes one discrepancy and creates errors elsewhere Thomas S. Kuhn.
    • When old problems once thought solved start failing, the model is breaking, and retooling is worth its cost.
    • Treat facts that need patches as signs the framework is failing Jenny Uglow.
    • Priestley renamed oxygen "dephlogisticated air" to fit the old theory Jenny Uglow.
    • Ptolemaic astronomers compounded circles until complexity outran accuracy.
  • Ask each team member to state the core model, then count the distinct answers. When nobody can say exactly what the reigning theory is, it has blurred under too many ad hoc articulations Thomas S. Kuhn.
    • Many different answers mean the shared base is gone and work is drifting toward rival camps.
    • Phlogiston theory had nearly as many versions as there were pneumatic chemists by the early 1770s.
  • Listen for rising argument about methods, problems and standards. While trust is high, teams work from shared examples without arguing about rules Thomas S. Kuhn.
    • Open argument about what counts as a legitimate method appears when the model feels insecure.
    • Deep method debates filled early optics, electricity and seventeenth-century chemistry, and returned with the shift from Newtonian to quantum mechanics.
    • Counterpoint: silence about first principles is not proof of agreement.

2.2 Do not act on a single anomaly

Every working model has unsolved problems, so look for patterns and check your own application before blaming the model.

  • Never drop a framework on anomalies alone. Every model leaves unsolved problems Thomas S. Kuhn.
    • Ask whether a rival does better, since rejecting one approach is always a comparison with another.
  • Check whether the error sits in how you applied the model. The fault may be in the working, not in the theory.
    • For sixty years Newtonian theory predicted only half the observed motion of the moon's perigee, and a few proposed changing the inverse square law Thomas S. Kuhn.
    • In 1750 Clairaut showed the error was in the mathematics of the application, and the theory stood Thomas S. Kuhn.

2.3 Test the rival on facts, then on fit

Check a rival's facts before judging its story
Darwin read Fourcroy's clear statement of the French chemistry and tested its facts before he weighed whether the theory hung together.

A replacement earns trust by passing a fact check first and a fit check second.

  • Check the rival's facts before you judge its story. Darwin read the clearest statement of the French chemistry and tested whether its facts held Jenny Uglow.
    • Only then ask whether the theory binds the facts together neatly.
    • Have facts reported in plain prose, with new jargon kept for interpretation, as Keir asked.
    • In November 1788 Darwin urged Watt to read Fourcroy's Introduction, and Watt gave in.
  • Build the replacement from measurement. Lavoisier weighed materials before and after, and built the theory that replaced phlogiston Jenny Uglow.
    • Priestley told Lavoisier of his mercury calx experiment in Paris in autumn 1774.
  • Be wary of a defender who keeps promising a final blow. Priestley did this, and he may have been losing Jenny Uglow.
    • Darwin's reaction to the new chemistry: "thank the Lord, that chemical Faith is not propagated by fire and sword".

2.4 Decide between real alternatives

Switch when a working replacement beats the actual incumbent, and not before.

  • Compare the new approach with the model you actually run, not an ideal. Asking whether one model fits the facts has no precise answer, since all significant models fit only more or less Thomas S. Kuhn.
    • Asking which of two real rivals fits better does have an answer.
    • Neither Priestley's nor Lavoisier's theory matched observations precisely, yet few contemporaries took more than a decade to judge Lavoisier's the better fit.
    • "Verification is like natural selection: it picks out the most viable among the actual alternatives in a particular historical situation".
  • Build the alternative before you declare the old model dead. Dropping a model with nothing to replace it leaves the team with no way to work at all Thomas S. Kuhn.
    • Use failure as a prompt to search, not as a verdict. "Failure of existing rules is the prelude to a search for new ones".

2.5 Where the books differ

Same odd result: failing model or faulty application
Phlogiston (Uglow)Lunar perigee (Kuhn)
AnomalyMetal ash gained weightHalf the perigee motion predicted
Response'Negative weight', renamed oxygenClairaut fixed the maths, 1750
Where the fault layThe modelThe application
Complexity vs accuracyComplexity outran accuracyAccuracy gained, no added complexity
OutcomeReplaced by LavoisierNewtonian theory stood
Judge a fix by what it does: Clairaut's correction added accuracy without complexity, while the phlogiston patches did the reverse.

The same odd result can mean a failing model or a slip in its use, and the books show both.

  • A patch can be a warning sign, and an anomaly can be noise. Both books give cases, so judge them by what the fix does.
    • Uglow's phlogiston case shows patches piling up around a model, with "negative weight" and a renamed oxygen Jenny Uglow.
    • Kuhn's lunar perigee case shows a long-standing anomaly that dissolved once the application was corrected Thomas S. Kuhn.
    • The test is whether a fix adds complexity faster than accuracy, which Clairaut's correction did not Thomas S. Kuhn.

Sources

Part of How a field works and shifts in the growth library.