How we measure, and how to read the results.

Our principles

1. Start with the decision, not the metric. A benchmark is only useful if it helps someone choose.
2. Rule-based checks first. Where a rule can decide, it does. AI judges are used only where rules cannot decide, and are checked against people's decisions.
3. Check the checker. Two judges agreeing shows they are consistent, not that they are right.
4. Report failures, not just averages. A critical failure is shown even when the average looks good.
5. Test the whole system. Prompt, model, settings and provider together decide the result.
6. Keep uncertainty visible. Ranges and unresolved cases are shown, not smoothed away.
7. Keep the failures. Every confirmed failure becomes a permanent test.
8. Count the cost of a decision, not just the cost of a model call.

Measured by us, or reported by others

Results from our own benchmarks are labelled Measured by Spring Prompt. We ran them, and we keep every model output so each result can be traced back to what the model actually wrote.

Results from other benchmarks are labelled Reported by the source, with its credit and licence. We do not re-run them, and we only publish sources whose licence allows it.

Benchmark pages show each source on its own, in its own units. The overall leaderboard is the one place we combine them: each headline metric becomes a percentile among the same set of current models, and groups are weighted towards our own business benchmarks. The page shows every weight and every part of every score, and what the combined number cannot tell you.

Reading a results table

• Numbers are shown in the source's own units, with its date.
• Where a source gives a range (for example a 95% interval), ranks still follow the score, and the range is shown beside it. Where two models' ranges overlap, the gap between them may not be real. A rank is not a verdict that one model is better.
• Where a model was run in more than one configuration, such as different reasoning efforts, each configuration has its own row and is named in the table. The overall leaderboard and model comparisons use each model's best configuration on each metric. Third-party sources report the settings they ran.
• A result says nothing about work the benchmark does not test. Each benchmark page lists what it does not measure.

Our own benchmarks

CatalogBench tests product-listing enrichment from feeds, images and supplier copy. ROASBench tests a year of marketing budget decisions in a simulated market. BulletBench tests decisions under a real clock. DeckBench tests turning an analysis into a deck that could be presented as it is. SurveyBench tests planning a user survey and analysing its responses without being misled. Each page explains its method and limits.

Where an AI judges part of a result, we name the judge and check it for bias. For CatalogBench we re-scored a calibration run with judges from Google and Anthropic, and the order of the models did not change.

What gets published, and indexed

Pages are built from sealed, versioned releases of our catalogue. A page is only offered to search engines when it has enough evidence to be useful: for example, a comparison needs both models on at least three shared sources. Thinner pages still exist, but are not indexed.

Corrections

If you find an error, tell us. We correct it in the next release and keep the previous release, so every change can be traced.