Keith Coleman
Where to hear and read about Keith Coleman1 episode or test
Keith Coleman is the Twitter and X product leader who created Birdwatch, now Community Notes, with a team of about five. The thing most worth learning from him is how to build a trusted, internet-scale system by keeping the team tiny and owned by one person, and by holding to a few strict principles of openness and quality.
While it might feel polarized, there's probably like an 80% set of people that agree on quite a lot of things.
An inside look at X’s Community Notes | Keith Coleman & Jay Baxter
Keith Coleman: the story and the lessons
1How Keith Coleman started Community Notes
Coleman joined Twitter mid-election and spent three years on growth. He joined in December 2016, when a debate played out on Twitter daily alongside questionable information. The company was in a turnaround and about a third of employees left each year.
Existing fixes were fact-checkers or internal trust and safety teams deciding what was misleading, and many people did not trust either to make that call.
He gave up managing a PM team to try crazy ideas. After paternity leave he asked his boss Kayvon if he could stop doing his job and work on this full time. Kayvon was stoked. He read all the research he could, then prototyped, then piloted.
He had weighed starting a company but kept coming back to the information quality problem. Running a large PM team took a lot of energy, and he did not see the product change he wanted.
Coleman repeatedly goes from big scope back to a small team. He calls this a "sawtooth leap": start small, grow, then go small again. He says the idea that impact grows with the number of people managed is a myth.
He guesses that had he stayed, he would have produced "16 more pages of OKRs." He loves the zero-to-one experience.
He made the product earn trust in small steps before going public. Mockups went to people across the political spectrum, then an MTurk-style test of note writing, then a public pilot with 1,000 people. Each step was backed by data from the one before.
The team feared the public pilot might be a total dumpster fire. Instead the early notes held gold, and the job became sifting it out. The data also won over internal trust and safety teams.
Elon Musk backed the idea two years before he owned the company. In 2020 a test participant leaked the Figma prototype to a reporter, and Musk, then only a user, replied "Definitely worth trying, IMO." After the acquisition he DM'd that the product was awesome.
Community Notes was the name in Coleman's first mockup, before it became Birdwatch. Musk renamed it back and the change shipped the next day. Coleman's view: "I think descriptive names just makes sense."
The project outlasted four leaders by shipping every week. It ran through Jack, Parag, Elon and Linda as CEO. The team ignored distractions and brought results that convinced each leader to take the next leap.
Cost savings was never the goal. Coleman says even 10,000 human fact-checkers would not work, because they are not trusted, do not scale and are too slow. He describes the work as community service and cares more about the outcome than the credit.
2How Coleman built trust into Community Notes
A note shows only when people who usually disagree rate it helpful. A simple majority vote would produce biased or inaccurate notes, so the algorithm looks for agreement from people who have disagreed in the past. This bootstrapped without ground-truth labels from fact-checkers.
Experts in 2020 expected an open system to be manipulated and to need fact-checker labels. Agreement across divides turned out to be hard to fake.
Coleman accepts publishing only about 8% of proposed notes. The team treats showing a bad note as the worst mistake because it undermines trust. The cutoff is 0.4 on an arbitrary scale, set conservatively after testing which notes users found helpful.
Writers whose notes disagreeing groups rate unhelpful lose the ability to write and must earn it back. Quality was the one thing never to give up.
There is no button to override a note. The company cannot change a note's status. Staff were unsettled at first that a note could go up and the company could not take it down.
The rule: "if there's a problem with a note that's so bad, you want to do something about it's a problem with the system." The fix is to redesign the system.
Coleman published the code and data so outsiders can audit it. Every scoring change is posted on GitHub and the data is published daily. The full run takes about a day on one machine with about 500 GB of RAM, and Vitalik Buterin reran it to verify.
Matrix factorization is normally trained once and served separately. Here it must run as a script on downloaded data, which forced unusual architecture.
Contributors are chosen at random, with no exemptions for anyone. The only entry test is a verified phone number. Coleman refused insider suggestions to start with journalists because he did not want it to be editorial.
Writing unlocks only after ratings show you can spot notes a broad range of people find helpful. His rule: no exemptions for Elon, governments or advertisers.
Coleman puts specific context on the post with zero clicks. Notes are written for the exact claim and shown on the post, not behind a generic link like "Get the facts here." One note can match thousands of posts that share the same image or video.
People who liked a post are notified when a note later appears, and users screenshot and share those alerts.
3What Coleman got wrong first and changed
Showing real names was the wrong default. The first prototypes showed real handles, on the belief that would build trust in a note. The pilot of a few thousand contributors showed the opposite worked better.
Some people hesitated to write on controversial topics under their name. Under a pseudonym, people cross partisan lines more readily and rate helpful a note that criticizes their own side. Other quality mechanisms kept note quality high.
The first algorithm solved the wrong problem. Jay Baxter's first production algorithm was a PageRank variant built to resist manipulation. Pilot data showed polarization and bias were the real problem, and PageRank can amplify bias when one side has more users.
Once bridging was clear, Baxter ran a Kaggle-style bake-off and brought in more engineers. The advice: keep one engineer while requirements are unclear, then add more once goals are clear.
4How Coleman runs a small team
One owner, one senior sponsor and full focus drive the work. Community Notes was the first Thermal project, a program Kayvon created to shield teams from bureaucracy. The team began with Coleman alone, then about five: backend, frontend, ML, design and research.
The decision-maker sits outside the team and must be senior. Coleman takes questions to Musk. With 100% focus, an idea in the morning gives results by afternoon. "A week or two delay totally changes the momentum of a project."
Coleman replaced OKRs and Jira with one running Google Doc. The team meets daily and sets goals per milestone, whether two weeks, a month or three months, not per quarter. The roadmap may change several times in two weeks.
Asana meetings turned into backlog grooming, while in the doc stale items just fall off. Jira is used only to coordinate with other teams. The OKR process took longer than picking a goal and finishing it.
Nobody was assigned to the team; people opted in. People reached out or applied, and Coleman's team interviewed every one. Baxter gave up a low-risk push-notification project that lifted daily users to join.
"If you're going to try something crazy, it's going to be tough if you're just assigning random people to it." After the acquisition, a tiny team scaled Spaces in two or three weeks, work that would have taken a year at the old company.
Coleman would now build a company even leaner. Less process and bureaucracy meant faster launches, more experiments and staff who fix problems outside their own area. He notes opting in matters, since a person may want this pace at one stage of life and not another.
After the layoffs the team audited its systems and removed features whose upkeep cost more than they gained. Baxter's view: "deleting code is more important than writing it a lot of the time."
5What Community Notes achieved and where it goes next
Notes were seen about 30 billion times in 2024. About 95,000 notes were seen about 30 billion times, more than double the year before, which had about 37,000 notes seen 14 billion times. About 950,000 contributors are signed up.
Meta now uses Community Notes as its main fact-checking tool.
Noted posts spread far less without any ranking change. Showing a note cut likes and reposts by 30 to 40% in A-B tests, and outside studies found total reposts drop 50 to 60%. Authors are 80% more likely to delete a post after a note.
Whether a post was noted is not used in the core ranking algorithm. Coleman and Baxter feel mixed about deletions, since the best notes vanish with the post.
Coleman shipped speed and coverage features before the Israel-Hamas surge. Notes on images and videos matched to other posts, plus a speed-up that cut three hours off time to go live, launched weeks before the conflict. About 500 notes appeared in the first three days, with a median of five hours from post to note.
Typical fact checks often take two to four days. Coleman says the system had not been tested at that scale, so the team held its breath.
He lets users show what the product is for. People use notes to debate whether Messi or Ronaldo scored more goals, and to flag spam. The team modeled the Messi debate to estimate each rater's stance, as it does for politics.
Coleman compares this to retweets, which also began as user behavior before becoming core to the product.
Coleman's next steps use AI to help writers and open the algorithm to the public. Supernotes, built by outside researchers, has an LLM generate variants of proposed notes and a simulated jury predict how each would be rated. Coleman wants a product built by the public.
Agents could check that a note is supported by its source, but raters must still verify it themselves. Coleman also imagines using bridging to find agreement on legislation and policy.
An inside look at X’s Community Notes | Keith Coleman & Jay Baxter
Community Notes shows how a tiny, opt-in team with one decision-maker, a bridging-based algorithm and strict principles of openness and trust can build an internet-scale information system. Notes were seen about 30 billion times in 2024, cut reposts by 50 to 60%, and Meta has since adopted the approach.
Coleman joined Twitter mid-election and spent three years on growth. He joined in December 2016, when a debate played out on Twitter daily alongside questionable information. The company was in a turnaround and about a third of employees left each year.
Existing fixes were fact-checkers or internal trust and safety teams deciding what was misleading, and many people did not trust either to make that call.
He gave up managing a PM team to try crazy ideas. After paternity leave he asked his boss Kayvon if he could stop doing his job and work on this full time. Kayvon was stoked. He read all the research he could, then prototyped, then piloted.
He had weighed starting a company but kept coming back to the information quality problem. Running a large PM team took a lot of energy, and he did not see the product change he wanted.
Coleman repeatedly goes from big scope back to a small team. He calls this a "sawtooth leap": start small, grow, then go small again. He says the idea that impact grows with the number of people managed is a myth.
He guesses that had he stayed, he would have produced "16 more pages of OKRs." He loves the zero-to-one experience.
He made the product earn trust in small steps before going public. Mockups went to people across the political spectrum, then an MTurk-style test of note writing, then a public pilot with 1,000 people. Each step was backed by data from the one before.
The team feared the public pilot might be a total dumpster fire. Instead the early notes held gold, and the job became sifting it out. The data also won over internal trust and safety teams.
Elon Musk backed the idea two years before he owned the company. In 2020 a test participant leaked the Figma prototype to a reporter, and Musk, then only a user, replied "Definitely worth trying, IMO." After the acquisition he DM'd that the product was awesome.
Community Notes was the name in Coleman's first mockup, before it became Birdwatch. Musk renamed it back and the change shipped the next day. Coleman's view: "I think descriptive names just makes sense."
The project outlasted four leaders by shipping every week. It ran through Jack, Parag, Elon and Linda as CEO. The team ignored distractions and brought results that convinced each leader to take the next leap.
Cost savings was never the goal. Coleman says even 10,000 human fact-checkers would not work, because they are not trusted, do not scale and are too slow. He describes the work as community service and cares more about the outcome than the credit.
A note shows only when people who usually disagree rate it helpful. A simple majority vote would produce biased or inaccurate notes, so the algorithm looks for agreement from people who have disagreed in the past. This bootstrapped without ground-truth labels from fact-checkers.
Experts in 2020 expected an open system to be manipulated and to need fact-checker labels. Agreement across divides turned out to be hard to fake.
Coleman accepts publishing only about 8% of proposed notes. The team treats showing a bad note as the worst mistake because it undermines trust. The cutoff is 0.4 on an arbitrary scale, set conservatively after testing which notes users found helpful.
Writers whose notes disagreeing groups rate unhelpful lose the ability to write and must earn it back. Quality was the one thing never to give up.
There is no button to override a note. The company cannot change a note's status. Staff were unsettled at first that a note could go up and the company could not take it down.
The rule: "if there's a problem with a note that's so bad, you want to do something about it's a problem with the system." The fix is to redesign the system.
Coleman published the code and data so outsiders can audit it. Every scoring change is posted on GitHub and the data is published daily. The full run takes about a day on one machine with about 500 GB of RAM, and Vitalik Buterin reran it to verify.
Matrix factorization is normally trained once and served separately. Here it must run as a script on downloaded data, which forced unusual architecture.
Contributors are chosen at random, with no exemptions for anyone. The only entry test is a verified phone number. Coleman refused insider suggestions to start with journalists because he did not want it to be editorial.
Writing unlocks only after ratings show you can spot notes a broad range of people find helpful. His rule: no exemptions for Elon, governments or advertisers.
Coleman puts specific context on the post with zero clicks. Notes are written for the exact claim and shown on the post, not behind a generic link like "Get the facts here." One note can match thousands of posts that share the same image or video.
People who liked a post are notified when a note later appears, and users screenshot and share those alerts.
Showing real names was the wrong default. The first prototypes showed real handles, on the belief that would build trust in a note. The pilot of a few thousand contributors showed the opposite worked better.
Some people hesitated to write on controversial topics under their name. Under a pseudonym, people cross partisan lines more readily and rate helpful a note that criticizes their own side. Other quality mechanisms kept note quality high.
The first algorithm solved the wrong problem. Jay Baxter's first production algorithm was a PageRank variant built to resist manipulation. Pilot data showed polarization and bias were the real problem, and PageRank can amplify bias when one side has more users.
Once bridging was clear, Baxter ran a Kaggle-style bake-off and brought in more engineers. The advice: keep one engineer while requirements are unclear, then add more once goals are clear.
One owner, one senior sponsor and full focus drive the work. Community Notes was the first Thermal project, a program Kayvon created to shield teams from bureaucracy. The team began with Coleman alone, then about five: backend, frontend, ML, design and research.
The decision-maker sits outside the team and must be senior. Coleman takes questions to Musk. With 100% focus, an idea in the morning gives results by afternoon. "A week or two delay totally changes the momentum of a project."
Coleman replaced OKRs and Jira with one running Google Doc. The team meets daily and sets goals per milestone, whether two weeks, a month or three months, not per quarter. The roadmap may change several times in two weeks.
Asana meetings turned into backlog grooming, while in the doc stale items just fall off. Jira is used only to coordinate with other teams. The OKR process took longer than picking a goal and finishing it.
Nobody was assigned to the team; people opted in. People reached out or applied, and Coleman's team interviewed every one. Baxter gave up a low-risk push-notification project that lifted daily users to join.
"If you're going to try something crazy, it's going to be tough if you're just assigning random people to it." After the acquisition, a tiny team scaled Spaces in two or three weeks, work that would have taken a year at the old company.
Coleman would now build a company even leaner. Less process and bureaucracy meant faster launches, more experiments and staff who fix problems outside their own area. He notes opting in matters, since a person may want this pace at one stage of life and not another.
After the layoffs the team audited its systems and removed features whose upkeep cost more than they gained. Baxter's view: "deleting code is more important than writing it a lot of the time."
Notes were seen about 30 billion times in 2024. About 95,000 notes were seen about 30 billion times, more than double the year before, which had about 37,000 notes seen 14 billion times. About 950,000 contributors are signed up.
Meta now uses Community Notes as its main fact-checking tool.
Noted posts spread far less without any ranking change. Showing a note cut likes and reposts by 30 to 40% in A-B tests, and outside studies found total reposts drop 50 to 60%. Authors are 80% more likely to delete a post after a note.
Whether a post was noted is not used in the core ranking algorithm. Coleman and Baxter feel mixed about deletions, since the best notes vanish with the post.
Coleman shipped speed and coverage features before the Israel-Hamas surge. Notes on images and videos matched to other posts, plus a speed-up that cut three hours off time to go live, launched weeks before the conflict. About 500 notes appeared in the first three days, with a median of five hours from post to note.
Typical fact checks often take two to four days. Coleman says the system had not been tested at that scale, so the team held its breath.
He lets users show what the product is for. People use notes to debate whether Messi or Ronaldo scored more goals, and to flag spam. The team modeled the Messi debate to estimate each rater's stance, as it does for politics.
Coleman compares this to retweets, which also began as user behavior before becoming core to the product.
Coleman's next steps use AI to help writers and open the algorithm to the public. Supernotes, built by outside researchers, has an LLM generate variants of proposed notes and a simulated jury predict how each would be rated. Coleman wants a product built by the public.
Agents could check that a note is supported by its source, but raters must still verify it themselves. Coleman also imagines using bridging to find agreement on legislation and policy.