LM Arena

Public leaderboards like LM Arena rank models, not your product. One client said they ran evals, but had only checked LM Arena and Artificial Analysis. That is model benchmarking, not evaluating a product on its own data.

The term "eval" now covers data labeling experts writing notes, PMs writing evals, LLM judges, and benchmarks, so it helps to define what you mean before arguing about it. For complex use cases, LLM judges can miss emerging patterns, so reading user signals, fixing problems, and checking for regressions can work better. Quote: "Don't be obsessed with prescriptions. They're going to change."

Lenny's Podcast▶ 38:50