OpenRouter provides the cost and speed data. The rest are benchmarks, weighted by the workload you picked to arrive at quality.
Every score on this page is somebody else’s measurement, carried over unchanged with the date it was taken.
Nothing here is scored by me, and nothing is a vendor’s own claim. Against each source is what it can tell you, what it cannot, and how much of this catalog it actually reaches.
Tells you: Whether a model can make a correct edit through a real coding tool. 225 exercises in six languages, edited through Aider and graded by unit tests.
Does not tell you: Anything about current models: the board stopped updating in October 2025, and carries little weight here for that reason.
Tells you: How reliably a model uses tools across single- and multi-turn tasks. V4 also tests search and memory. Native function-calling results are preferred; prompt-based results are used when those are the only measurements.
Does not tell you: Performance under every agent setup or on the newest models. The tested mode stays with the score; the date shown is the leaderboard update date, not an individual run date.
Tells you: Whether a model can find the right thing in a long prompt. OpenAI’s MRCR v2 with eight near-identical requests hidden in a long chat, scored at 128K tokens, the length nearly every model was tested at.
Does not tell you: Reasoning over a long input, only recall of it. Some of the newest models have not been run yet.
Tells you: One team running every model under one protocol, rather than collected vendor claims: GPQA Diamond, FrontierMath, SimpleQA Verified, SWE-Bench Verified and AIME.
Does not tell you: Anything about your domain. These are exam questions with one right answer; most work is not. AIME is kept on the page but not ranked on — the frontier scores 1.000, so it no longer separates two good models.
Tells you: Conversation skills and creative writing quality.EQ-Bench 4 measures emotional and social judgment in multi-turn chats. Creative Writing v3 assesses creative prose; Longform Writing assesses planning and consistency across eight chapters.
Does not tell you: Factual accuracy, every kind of editing, or performance with real users. These benchmarks use LLM judges and reflect their preferences. Scores come from the live leaderboard data, with distinct scales and coverage; individual run dates are not published.
Tells you: Whether a model works as well outside English. Knowledge questions professionally translated into 16 languages and run by Kaggle; the weakest language is kept beside the average.
Does not tell you: Translation quality as such, or the order among the leaders: the frontier is bunched within two points.
Tells you: How far a model gets on questions written to be at the edge of expert knowledge. Scale’s Diamond set, 1,000 frozen questions, each model run once at high reasoning effort with no tools.
Does not tell you: Anything about models Scale has not run: the board has a dozen, all at the frontier. Not comparable with the original HLE numbers, and Scale flags every row for possible contamination.
Tells you: How faithfully a model turns PDFs and scans into text, tables and fields. The mean of olmOCR-Bench, OmniDocBench and the board’s own extraction set.
Does not tell you: Anything about models released after spring 2026, when it was last updated. It reaches only a dozen models here.
Tells you: Whether a model did exactly what it was told. Rewriting, simplifying, summarizing and story tasks with constraints a script checks rather than a person.
Does not tell you: Anything else LiveBench measures — only its instruction-following category is used. Models join the current release over months, so rows were not all run at the same time.
Tells you: Whether people preferred the answer. Anonymous side-by-side votes turned into an Elo rating, across text, instruction following, web development, vision, documents, search, image generation, image editing, image-to-code, text-to-video and image-to-video.
Does not tell you: Whether the answer was correct. Voters reward tone, length and formatting, and the text board applies style control to damp that but does not remove it. A gap smaller than the confidence interval is not a gap.
Tells you: Which music model listeners preferred. Blind side-by-side votes on vocal and instrumental tracks, published under CC BY 4.0.
Does not tell you: Much about the models here: only the two Lyria models are sold through OpenRouter. The vocal and instrumental boards are fitted separately, and the leaders overlap within their intervals.
Tells you: How many words a model gets wrong. Word error rate on six public English sets — meetings, podcasts, audiobooks, earnings calls and phone speech. Its two sets published by Artificial Analysis, and the headline average that includes them, are left out.
Does not tell you: Anything outside English, long recordings or streaming. It runs open models and a few APIs, not OpenAI’s, Google’s or Deepgram’s, so most of a transcription shortlist has no score from it.
Tells you: Which models exist and what they cost. Context, input and output modalities, tool and schema support, per-provider price including cached reads, uptime, and the measured throughput and time-to-first-token behind each endpoint.
Does not tell you: Quality. It is a marketplace, so the catalog is what OpenRouter routes to — a model served only by its vendor’s own API may be missing. Throughput is a 30-minute window and moves.
Tells you: What the writing is actually like. Stock AI phrases and “not X, but Y” constructions per thousand words, repeated phrasing, vocabulary spread and reading grade — each counted the same way over 17 million words of human writing, so a model’s figure can be read against a person’s.
Does not tell you: Whether the writing is any good. It counts habits, not quality: a model can score well by being terse and dull. Shown on each model rather than folded into the ranking, because too few models have been measured for that.
Tells you: Whether a model can finish harder, longer fixes in unfamiliar codebases. Issues from copyleft repositories, graded by the projects’ own tests and run by Scale.
Does not tell you: A fine ranking of the current frontier: the V2 board’s ten entries sit between 90% and 99%, so it is shown and not ranked on, and the original board is deprecated.
Tells you: 500 real GitHub issues, graded by the maintainers’ own tests. The closest thing to “can it land a change in an existing codebase.”
Does not tell you: The current frontier. Entries are submitted by whoever built the scaffold, so the board rewards teams who bother to submit and runs months behind new releases.
Tells you: Whether an agent follows a written policy while a customer talks to it. Sierra’s own runs: the airline, retail and telecom average, and the newer banking domain, which adds lookups in a knowledge base.
Does not tell you: Results for newer models on the original domains, which stopped in March 2026. Banking is current but hard; the best models pass under half.
Tells you: Whether a model left alone in a terminal finishes the job. A test suite decides, with no partial credit. The board also reports what each run cost and how long a trial took.
Does not tell you: The model on its own. A score belongs to a model and its scaffold — Codex, Claude Code — and swapping the harness moves the number. The board is small, so most of the catalog has no entry.
Tells you: Which voice listeners preferred. Blind side-by-side votes on the same English text, with confidence intervals and vote counts.
Does not tell you: Most of the speech models sold here: it rates a handful of them, and many of the voices it rates are not on OpenRouter. Suspended and low-vote entries are left out; a retired voice keeps its frozen rating. The ratings carry no stated licence.
Tells you: How often a model adds facts that are not in the document it was given. Summaries of thousands of articles, each checked by Vectara’s HHEM model.
Does not tell you: Whether the model knows things, only whether it sticks to what it was handed. Most models land between 90% and 99%, and several recent ones are missing.
Sources considered but not used
For completeness, here are additional sources we considered using for benchmark data, but ultimately decided they were not quite right.
Useful document-understanding tasks, but the published table reviewed in September 2026 stops at July 2025 model releases. Coverage of current models is insufficient; preview and release variants cannot safely share scores.
Excellent, current coverage — but the data is commercially licensed. I do not want this page to depend on republishing or deriving from data without clear redistribution rights.