Artificial Analysis

Checking Artificial Analysis is model benchmarking, not evaluating your own product. A client told Aishwarya Naresh Reganti and Kiriti Badam they did evals, but they had only checked LM Arena and Artificial Analysis. That tells you how models compare in general, and nothing about how the product performs on its own data.

Public leaderboards like these only inform model choice. Evals in the product sense mean working from your own user signals: reading them, fixing problems, and checking for regressions. For complex use cases, LLM judges can miss emerging patterns. The advice is to define what you mean by "eval" before arguing about it, since the term now covers data labeling experts writing notes, PMs writing evals, LLM judges, and benchmarks. Quote: "Don't be obsessed with prescriptions. They're going to change."

Lenny's Podcast▶ 38:50