Methodology

How we measure, and how to read the results.

Every number on this site says what was measured, by whom, on what, and when, and what it does not show.

Our principles

  1. Start with the decision, not the metric. A benchmark is only useful if it helps someone choose.
  2. Rule-based checks first. Where a rule can decide, it does. AI judges are used only where rules cannot decide, and are checked against people's decisions.
  3. Check the checker. Two judges agreeing shows they are consistent, not that they are right.
  4. Report failures, not just averages. A critical failure is shown even when the average looks good.
  5. Test the whole system. Prompt, model, settings and provider together decide the result.
  6. Keep uncertainty visible. Ranges and unresolved cases are shown, not smoothed away.
  7. Keep the failures. Every confirmed failure becomes a permanent test.
  8. Count the cost of a decision, not just the cost of a model call.

Measured by us, or reported by others

Results from our own benchmarks are labelled Measured by Spring Prompt. We ran them, and we keep every model output so each result can be traced back to what the model actually wrote.

Results from other benchmarks are labelled Reported by the source, with its credit and licence. We do not re-run them, and we only publish sources whose licence allows it.

Benchmark pages show each source on its own, in its own units. The overall leaderboard is the one place we combine them: each headline metric becomes a percentile among the same set of current models, and groups are weighted towards our own business benchmarks. The page shows every weight and every part of every score, and what the combined number cannot tell you.

Reading a results table

  • Numbers are shown in the source's own units, with its date.
  • Where a source gives a range (for example a 95% interval), ranks still follow the score, and the range is shown beside it. Where two models' ranges overlap, the gap between them may not be real. A rank is not a verdict that one model is better.
  • Where a model was run in more than one configuration, such as different reasoning efforts, each configuration has its own row and is named in the table. The overall leaderboard and model comparisons use each model's best configuration on each metric. Third-party sources report the settings they ran.
  • A result says nothing about work the benchmark does not test. Each benchmark page lists what it does not measure.

Our own benchmarks

CatalogBench tests product-listing enrichment from feeds, images and supplier copy. ROASBench tests a year of marketing budget decisions in a simulated market. BulletBench tests decisions under a real clock. DeckBench tests turning an analysis into a deck that could be presented as it is. SurveyBench tests planning a user survey and analysing its responses without being misled. Each page explains its method and limits, and our benchmarks, explained sets them side by side.

Where an AI judges part of a result, we name the judge and check it for bias. For CatalogBench we re-scored a calibration run with judges from Google and Anthropic, and the order of the models did not change.

The Spring Prompt Score

The Spring Prompt Score is one number from 0 to 100 from our own benchmarks only. On each benchmark's headline measure, a model's best configuration becomes a percentile: the share of the other models it beats, among the models we tested on CatalogBench or ROASBench. The score is the plain average of those percentiles, every benchmark weighted the same, and a model needs results on at least three of them; each score shows how many it covers. The overall leaderboard also blends in licensed results from other leaderboards, which are not ours to republish; the Spring Prompt Score uses none, so it is open data, with a badge for every model it ranks.

How sure a small test can be

A pass rate from a dozen cases is a rough figure, so we give it as a likely range, and we only call one model better than another when the cases where they differ make it clear. The test case calculator uses the same maths, so you can see how many cases your own question needs.

Where an AI judge decides a check, we count how often it agrees with people's own verdicts, and how often our benchmark judges agree with each other: see judge agreement.

What gets published, and indexed

Pages are built from sealed, versioned releases of our catalogue. A page is only offered to search engines when it has enough evidence to be useful: for example, a comparison needs both models on at least three shared sources. Thinner pages still exist, but are not indexed.

Corrections

If you find an error, tell us. We correct it in the next release and keep the previous release, so every change can be traced.

Data sources

Sources and licences.

Release 2026-11-16-49bdfa045dee

This release publishes results from 13 licensed sources + 5 of our own benchmarks. The overall leaderboard counts a source once it has tested enough of the models it ranks; its page says which ones count today.

SourceCreditLicenceLicence checked
BulletBenchMeasured by Spring PromptSpring Prompt original30 Sep 2026
CatalogBenchMeasured by Spring PromptSpring Prompt original30 Sep 2026
DeckBenchMeasured by Spring PromptSpring Prompt original30 Sep 2026
ROASBenchMeasured by Spring PromptSpring Prompt original30 Sep 2026
SurveyBenchMeasured by Spring PromptSpring Prompt original30 Sep 2026
APEX-AgentsMercor; collected by Epoch AIPublished with permission (Mercor results via Epoch AI)1 Oct 2026
Arena (formerly LMArena)Contains data from the Arena Leaderboard Dataset by Arena, licensed under CC BY 4.0Creative Commons Attribution 4.0 International29 Sep 2026
Artificial AnalysisData sourced from Artificial AnalysisArtificial Analysis commercial data licence1 Oct 2026
Berkeley Function Calling Leaderboard (BFCL) V4Berkeley Function Calling Leaderboard (BFCL) V4 by the UC Berkeley Gorilla team, Apache License 2.0Apache License 2.029 Sep 2026
GDP.pdfSurge AI; collected by Epoch AIPublished with permission (Surge AI results via Epoch AI)1 Oct 2026
Microsoft STATE-BenchSTATE-Bench by Microsoft and STATE-Bench contributors, MIT LicenseMIT License29 Sep 2026
OpenHands IndexOpenHands Index by the OpenHands contributors, Apache License 2.0Apache License 2.029 Sep 2026
Remote Labor IndexScale AI and the Center for AI Safety; collected by Epoch AIPublished with permission (Scale AI and the Center for AI Safety results via Epoch AI)1 Oct 2026
SimpleQA Verified (Epoch AI)Epoch AI, Capabilities & benchmarking (CC BY 4.0)Creative Commons Attribution 4.0 International1 Oct 2026
tau2-benchtau2-bench by Sierra Research and contributors, MIT LicenseMIT License29 Sep 2026
UGI LeaderboardUGI Leaderboard by DontPlanToEnd, Apache License 2.0Apache License 2.029 Sep 2026
Vectara Hallucination LeaderboardVectara Hallucination Leaderboard by Vectara, Apache License 2.0Apache License 2.029 Sep 2026
Vending-Bench 2Andon Labs; collected by Epoch AIPublished with permission (Andon Labs results via Epoch AI)1 Oct 2026
Prices and release data

Where prices and launch dates come from.

DataUsed forSource and termsChecked
Developer list pricesThe price shown for a model wherever we have read it: API pricing, model pages and comparisons. Launch discounts are noted with their end dates.Read by hand from each developer's own pricing page, linked from the model's row on the pricing page7 Oct 2026
OpenRouter pricesThe price where we have no developer price (the typical, median host), the cheapest host, and which models are on sale.OpenRouter's public model and endpoint lists10 Nov 2026
Usage share"What builders actually run" on the release tracker: each model's share of the tokens sent through OpenRouter, week by week.OpenRouter's Data API rankings (CC BY 4.0)8 Nov 2026
Release datesThe release tracker: launch dates since 2024 and how often each line ships.Epoch AI model data (CC BY 4.0), with main lines checked by hand against the developers' own announcements20 Oct 2026