How to run playtests that reveal real problems

1 The bottom line

Pick testers who match your players, stay silent, and watch what they do rather than what they say. You know every answer, so only other people's behaviour shows where your design fails.

2 Key takeaways

• Test with people who are not you. You know every solution, so you cannot judge your own difficulty.
  • Every playtester, including two professional puzzle designers, failed or skipped a Mind Over Magnet level that Mark thought was trivial. Mark Brown, Game Maker's Toolkit4:57
• Watch what testers do, not what they say. Players are biased and polite when you ask them directly.
  • Mark's nephews both said "I love it" when asked, while his dad's visible frustration showed the real problem. Mark Brown, Game Maker's Toolkit13:52
• Treat a stuck point that several players share as the real problem. A repeated blocker is a design flaw, not bad luck.
  • Nintendo's Breath of the Wild heat map showed about 80 percent of testers following the tower route and 20 percent wandering randomly, and neither matched the intended experience. Mark Brown, Game Maker's Toolkit3:12
• Match your testers to your audience and your question. Different players surface different problems.
  • Some of Mark's best playtests came from his seven and ten year old nephews. Mark Brown, Game Maker's Toolkit13:32
• Test often, and change something between tests. Frequent testing checks that the design works for other people.
  • In Mind Over Magnet's final stretch, Mark ran playtests almost daily with small changes between them, and players began having fun and reacting strongly. Mark Brown, Game Maker's Toolkit9:43

spend more time watching how your playtesters play than listening to what they say.

Mark Brown, Game Maker's Toolkit

2.1 Build something playable and pick the right testers

Test early with a full loop, and choose people who match who the game is for.

• Get a full playable loop before you ask for feedback. Mark Brown, Game Maker's Toolkit6:57
  • Once the spelling and roguelike parts were tied into a complete loop in Word Play, Mark could put it on Discord, take it to schools on an iPad and play it with developers at GDC.
  • Keep testing at every stage, because players get lost, misread controls and make assumptions you never considered. Mark Brown, Game Maker's Toolkit13:06
  • The feedback led to rearranged tiles on the board and bonus points for long words, because a single mega word barely scored.
• Choose testers from your target audience before you act on their feedback. Mark Brown on Valve, Game Maker's Toolkit13:24
  • Hardcore FPS players said the Portal GLaDOS fight needed more action, challenge and skill.
  • Testers who had adapted to Portal's slower, cerebral pace found that version "just frustrated, confused, and dissatisfied".
  • Valve served the hardcore players with optional content, such as advanced chambers and challenge maps.
• Add specialists for the problems friends and family miss. Mark Brown and Arvi Teikari, Game Maker's Toolkit14:17
  • With 200-plus levels, Baba Is You relied on a large tester group to give feedback on stages and find alternative solutions the designer never saw.
  • Ask someone who knows your genre to judge feel. Mark sent his demo to Noel Berry, co-creator of Celeste, who said it felt pretty good. Mark Brown, Game Maker's Toolkit11:12
  • Ask disabled players for feedback, or use consultancies like DAGERS. Mark Brown, Game Maker's Toolkit9:40
  • Baba Is You's testers included the designers behind Ending, A Good Snowman Is Hard to Build, Pipe Push Paradise, Starseed Pilgrim, Minit and The Witness.
  • Even praised games miss things. Spider-Man lacks controller remapping, and XCOM works with keyboard alone but not mouse alone. Mark Brown, Game Maker's Toolkit9:40

2.2 Set up sessions that give clean findings

Control who gets each build, when it goes out and what testers report.

• Give each build to no more than three people. Mark Brown, Game Maker's Toolkit13:08
  • Too many testers at once means too much footage to watch.
  • A small group gives a manageable amount of footage with more distinct issues.
  • Seeing one person's bug or exploit repeat in nine more videos is painful and not useful.
• Send builds on Friday and review them on Monday. Mark Brown, Game Maker's Toolkit12:39
  • Testers play over the weekend, so you are not stuck waiting on footage mid-week.
  • Watch each whole video and note bugs, polish needs, and levels that are too easy, too hard, too fiddly or exploitable.
• Tell testers to report everything, big or small. Mark Brown, Game Maker's Toolkit3:06
  • Collect every report first and sort them afterwards, so testers do not filter out useful ones.
  • The GMTK Discord testing team played Mind Over Magnet with a fine-tooth comb and found nearly 100 issues.
• Let the designers of the level run its playtests. Mark Brown on Valve, Game Maker's Toolkit12:08
  • Valve does not outsource testing to a separate department or a publisher.
  • Watching shows the designers exactly what to fix, motivates them and can spark new ideas.
  • In Half-Life: Alyx, designers saw players cover their real mouths to stop Alyx coughing near a blind zombie, and turned it into a mechanic.

2.3 Watch behaviour, and drop your own knowledge

What testers do is better data than what they say, and your own expertise hides problems.

• Sit with testers, stay silent and read their behaviour. Mark Brown on Valve, Game Maker's Toolkit11:06
  • Give no hints and answer no questions, so the test simulates real play.
  • Body language tells you more about enjoyment than what players say afterward.
  • Sitting next to someone who does not understand your game is humbling and shows where you went wrong. Mark Brown, Game Maker's Toolkit13:52
  • Kim Swift read enjoyment from body language during Portal tests.
• Challenge your assumptions about how skilled players are. Mark Brown, Game Maker's Toolkit16:16
  • After months of play, you hold the same knowledge as the author of the rules.
  • Shigeru Miyamoto reportedly tells new designers to play with left and right hands switched on the controller, to feel like a novice. Mark Brown, Game Maker's Toolkit13:33
  • Mark made two errors himself: overtuned difficulty and mechanics he never taught.
  • He assumed players would crack a clean puzzle at once, so he padded levels with extra steps, traps and red herrings. His dad handed over the controller and asked "can you just do this one?" Mark Brown, Game Maker's Toolkit7:27
• Do not assume players share your genre's conventions, and playtest tutorials a lot. Mark Brown, Game Maker's Toolkit17:14
  • Veteran strategy players link an hourglass to ending a turn, but most people read it as loading.
  • In Total War: Troy playtests, one player spent 40 minutes on the first turn because of the hourglass end turn button. The devs swapped it for an arrow before launch.

2.4 Spot where many players get stuck

Look for repeats across testers, and back your eyes with enough players and careful data.

• Look for the same exact bit stumping several different players. Mark Brown, Game Maker's Toolkit2:12
  • Playtests exist to find where people get stumped and frustrated.
  • A shared blocker points to a design problem, not a few unlucky individuals.
  • Mark's rule from God of War Ragnarok: "if a lot of people are getting stumped by the exact same bit, then it's probably a good idea to step in and fix things."
• Use enough testers to act on trends and ignore outliers. Mark Brown on Valve, Game Maker's Toolkit10:48
  • Each Half-Life 2 chapter had about 100 playtesters, so Valve could see common patterns and ignore a few weird outliers.
  • Lessons like "players don't learn when stressed" and "players don't look up" came from watching many tests.
  • Gabe Newell said that after a couple hundred playtests you develop a much better sense of which design strategies work.
• Record where testers go, and check what they say against what they do. Mark Brown, Game Maker's Toolkit3:12
  • Nintendo plotted where Breath of the Wild playtesters walked as a heat map.
  • Treat play data as one source. Rainbow Six Siege's team is data informed, not data driven. Mark Brown, Game Maker's Toolkit6:54
  • After the landmark redesign, the same map showed players exploring freely while almost all reached key locations.
  • Valve tracked Half-Life 2: Episode One, found a difficulty spike in one battle and patched it. Mark Brown, Game Maker's Toolkit6:54
  • In Slay the Spire, the card Madness appeared in many winning decks only because a late-game event handed it out regularly, not because it was overpowered. Mark Brown, Game Maker's Toolkit6:54
• Retest after big changes, such as final art or a new medium. Mark Brown, Game Maker's Toolkit2:05
  • A puzzle that read clearly in grey cubes can get buried in visual noise once the art goes in.
  • Playtesting also reveals the rules of a technology nobody has designed for yet. Mark Brown on Valve, Game Maker's Toolkit8:12
  • God of War's puzzles were simple in greybox but "a lot harder to read now that all of the pretty visuals have been put in."
  • For Half-Life: Alyx, Valve found players' tolerance for standing around watching people talk was much lower in VR, so it sped up the pace. Mark Brown on Valve, Game Maker's Toolkit8:12

2.5 Stress-test inputs and edge cases

Players hammer controls and change state at the worst moment, so test for it.

• Hit every toggle and binding far faster than a thumb on a stick would. Mark Brown, Game Maker's Toolkit3:26
  • Mechanics that change state can get stuck partway under rapid presses.
  • Check odd rebindings too, because players can turn them into exploits.
  • Mashing the polarity button on one Mind Over Magnet level could leave a platform stuck midway.
  • An anonymous email from a Next Fest demo player revealed that binding move to the scroll wheel let you scroll rapidly and make Uni speed along like a jet engine. Mark Brown, Game Maker's Toolkit3:36
• Flip states at awkward moments, such as just before contact. Mark Brown, Game Maker's Toolkit3:20
  • Physics mechanics have edge cases when the player changes state at the worst instant.
  • Turning magnetism off just before Magnus connected made him bounce off the ceiling at an awkward angle.
• Try to break your own level, then hand it to a first-time player. Mark Brown, Game Maker's Toolkit8:48
  • Look for bypasses that need extra blocking.
  • Have friends and family play, watch what they do, ask for feedback and change the stage.
  • After hours of work you know the level inside out, so a first-timer could easily be confused, lost or find it too demanding.

2.6 Turn findings into changes

Run a steady loop, decide what to keep, check the feeling you wanted and finish with a QA pass.

• Run a weekly loop and treat each design as a hypothesis. Mark Brown on Valve, Game Maker's Toolkit5:24
  • Portal was tested almost every week. Valve discussed results on Monday, applied lessons through the week and tested again on Friday.
  • Set a goal such as a readable, satisfying puzzle, design to hit it, test, then change and repeat until the tests are no longer painful to watch. Mark Brown, Game Maker's Toolkit4:25
  • Give each batch of levels to friends, family, strangers online and fellow puzzle designers, and let the plan change. Mark Brown, Game Maker's Toolkit9:00
  • Valve's wording: "We see our game designs as hypotheses and our playtests as experiments to validate these hypotheses." This turns taste arguments into evidence.
• Keep behaviour that fits your vision and fix behaviour that breaks your emotional goals. Mark Brown, Game Maker's Toolkit9:14
  • Players may misunderstand, ignore or subvert a mechanic, and you cannot know until you test.
  • Dana Nightingale planned Dishonored 2's first clockwork soldier as non-hostile, but about 80 percent of players hid anyway. The team kept that and added Jindosh muttering "Hello? Anyone there?" to validate a perfect stealth run. Dana Nightingale and David Di Giacomo, Game Maker's Toolkit6:18
  • Rocket League's flying cars were kept because they fit the vision.
  • Klei saw players wanted to play Don't Starve together, which led to Don't Starve Together. Criterion saw Burnout Paradise players prefer messing about in the open world, so later DLC leaned into that. Mark Brown, Game Maker's Toolkit4:24
• Check the feeling you wanted, not only whether the game works. Mark Brown, Game Maker's Toolkit12:58
  • Chart the intensity you want, with low points for story beats and high points for combat.
  • Have playtesters describe how intense each area feels, and change the level where the chart is off.
  • In Mass Effect 3, enemies in the catacombs made torchlit fights too intense, so the monsters were scrapped and a cutscene of friendly bombers was added to raise hope.
  • Fujibayashi knew Ultrahand would be a hit when staff excitedly shared videos of their creations during Tears of the Kingdom playtests. Mark Brown, Game Maker's Toolkit6:01
• Finish with a QA group that plays the whole game repeatedly. Mark Brown, Game Maker's Toolkit13:41
  • Mark brought in patrons and insightful demo players for Word Play.
  • They found bugs, typos, and "perks that were overpowered, and perks that were pointlessly weak", and helped shape the difficulty curve and achievements.
  • The group balanced the 160 perks before launch.

2.7 Where designers disagree

Three testers find issues; a hundred confirm patterns

	Mark Brown (GMTK)	Valve (Half-Life 2)
Testers per build	3 or fewer	~100 per chapter
Question answered	What distinct issues exist?	Is this pattern common?
Footage	Manageable to watch fully	Enough to spot trends
Avoids	Same bug in nine more videos	Acting on weird outliers

The sources differ on how many testers to use, whether to take their ideas and how far to trust friends.

• Use a handful of testers, or about a hundred.
  • Mark caps each build at three testers so the footage stays watchable and the findings stay distinct. Mark Brown, Game Maker's Toolkit13:08
  • Valve used about 100 testers per Half-Life 2 chapter to separate trends from outliers. Mark Brown on Valve, Game Maker's Toolkit10:48
  • The two numbers answer different questions: finding distinct issues, and confirming that a pattern is common.
• Ignore testers' suggested fixes, or build on them.
  • Observers should ignore suggested fixes, because testers lack your vision and constraints. Mark Brown on Valve, Game Maker's Toolkit11:06
  • Adam, a developer behind Snipperclips, noticed something in Mind Over Magnet and suggested a puzzle built around it, and Mark accepted. Mark Brown, Game Maker's Toolkit9:00
  • The Valve video also says "You could almost think of the player as another designer." Mark Brown on Valve, Game Maker's Toolkit8:12
• Start with friends and family, or go to strangers.
  • Mark plans to send a demo to family and friends first, then to patrons on the GMTK Discord. Mark Brown, Game Maker's Toolkit11:11
  • Friends and family may be polite, so watch for bias. Mark Brown, Game Maker's Toolkit13:32
  • Strangers who record themselves playing let you watch real behaviour. Mark Brown, Game Maker's Toolkit6:43