Benchmarks / Our benchmarks
Our benchmarks, explained
We run 5 benchmarks of our own. Each one gives models a piece of work a business would hand over, such as product listings, a slide deck or a year of ad budgets, and checks whether the result could be used as it is.
General leaderboards rank models on exams, coding and chat. They are a useful start, but they don't say whether a model's product listing invents a claim or its deck quotes a draft figure. So each of our benchmarks starts from a decision someone has to make, checks with rules wherever a rule can decide, and uses AI judges only where it can't, with the judges checked. Every benchmark page shows real model outputs from the public tasks next to the scores. See how far general leaderboards predict our results, and our methodology. For one number across all five, see the Spring Prompt Score.
At a glance
| Benchmark | Job | Headline measure | Scored by | Public tasks | Models | Leader |
|---|---|---|---|---|---|---|
| CatalogBench | Product listings | Reliably publish-ready, factual brief % of products |
Rules, then an AI judge | 56 products | 28 | GPT-6.1 Sol 75.0% |
| DeckBench | Slide decks | Deck rating rating |
Code, then three AI judges | 6 tasks | 29 | GPT-6 Astra 1,593 |
| SurveyBench | Survey research | Analysis rating rating |
Code, then three AI judges | 9 tasks | 18 | Claude Fable 5.2 1,683 |
| ROASBench | Ad budgets | Score score out of 100 |
A deterministic simulator | One simulated year, 12 monthly plans | 29 | Claude Fable 5.2 61.3 |
| BulletBench | Speed under time pressure | Lightning 10+1 ladder Elo |
Game results against Stockfish | 3 clocks, 8 to 12 games per configuration | 26 | Gemini 3.5 Flash-Lite 892 |
Swipe the table sideways for more columns.
CatalogBench: product listings
Which models can turn a sparse product feed, product photos and supplier copy into a listing that could go live, without inventing anything?
Retailers increasingly let AI write their product listings from a supplier feed and a few photos. The risky part is not the prose: it is attributes that are guessed, claims the pack contradicts or the law requires proof for, and listings that no shopper will find.
Each of the 56 public products is invented, with generated photos that carry real-looking labels. Every feed row hides traps: missing attributes only the photos can answer, a supplier claim the label contradicts, and tempting claims such as “award-winning” with no evidence at all.
How a run works
- 1
The feed row
A sparse supplier row: title, price, a few attributes, and the supplier's marketing text, usable unless the pack says otherwise.
- 2
The photos
One to three product photos, including labels with the real facts: volume, strength, origin, ingredients.
- 3
The listing
The model fills missing attributes and writes the title, highlights, description, alt text, search keywords and category.
- 4
The checks
Rules check attributes and channel limits, a judge answers yes/no questions against the evidence, and a search test checks shoppers can find it.
- 5
Publish-ready?
Only if every blocking check passes. Three runs; the headline counts products that passed in all three.
What a good answer must get right
- Attributes read from the images, not guessed
- Supplier claims checked against the pack and UK rules
- Required UK product information included
- Listings that shoppers can find in search
Method
- Rule-based checks first; judged checks are yes or no
- The judge was checked for bias against Gemini and Claude judges
- Private products are held back so the set can be refreshed
Judged by GPT-6.1 Sol.
What it does not test
- Conversion or sales impact
- Real product photography (a real-photo slice is planned)
- Writing style beyond the listed checks
- Headline measure
- Reliably publish-ready, factual brief, % of products, higher is better
- What the number means
- Share of products whose listing was publish-ready in every one of repeated runs.
- Tasks
- 56 public products, scored; 14 private, held back so the set can be refreshed; 24 for calibration only, never scored
- Runs
- 3 per model
- Sample
- n = 56 products × 3 runs; judging about $1.76 per model, all runs
- Leader today
- GPT-6.1 Sol, 75.0%, of 28 models · Last updated 10 Nov 2026
How the judge is checked
The judge is an OpenAI model, and OpenAI models lead this table, so we checked it for bias under the current rules. Gemini 3.1 Pro and Claude Opus 5.5 judged the same outputs from five models on a calibration set. All three judges gave the OpenAI models the same or nearly the same results, and put the models in the same order on the sales brief; on the factual brief only Gemini differed, ranking Claude Opus 5.5 above GPT-6 Luna. The Claude judge was stricter on Claude Opus 5.5 than the OpenAI judge was. The judges differ mainly in how many claims they flag: Gemini flags the fewest.
Full CatalogBench results, outputs and method →
Download: results.csv · results.json · tasks.json · tasks.csv
DeckBench: slide decks
Given a finished analysis, which models make a deck a manager could present as it is: true to the numbers, telling the right story, and well designed?
Turning an analysis into slides is one of the most common things people ask AI to do at work, and the risk is not ugly slides: it is a number that is not in the analysis, a caveat that falls off, or a draft figure that ends up in front of the board.
Each task is an invented company's analysis written for the benchmark, with planted traps: a caveat that must travel with its number, a figure the memo says not to quote, a metric it warns is misleading, and a finding that contradicts a senior stakeholder's view.
How a run works
- 1
The analysis
An invented company's memo with findings, a recommendation and caveats, two to four data tables, the audience, the decision and a brand kit.
- 2
The deck
The model returns every slide as positioned text, shapes, native charts and tables. We render it to PowerPoint and to images.
- 3
The checks
Code checks the layout and traces every number to the analysis; three judges check the story and rate each slide's design.
- 4
Head to head
On each task the judges compare decks in pairs and pick the one they would rather present. The votes become the rating.
What a good answer must get right
- Every number traced to the analysis
- The story: findings, recommendation and caveats
- Layout and legibility, checked by code
- Design quality, judged slide by slide
Method
- Layout, legibility and numbers are checked by code
- Three judges from three vendors decide each judged check by majority
- No judge compares decks from its own vendor
- Two tasks are held back so the set can be refreshed
Judged by GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro, by majority.
What it does not test
- Your templates, data or brand guidelines
- Decks built in PowerPoint or Google Slides by an agent clicking through the app
- Speaking notes or delivery
- Headline measure
- Deck rating, rating, higher is better
- What the number means
- Head-to-head preference: on each task three judges from three vendors compare decks in pairs and say which they would rather present. A Bradley-Terry rating; a 400-point gap is 10:1 odds. No judge votes on its own vendor's decks.
- Tasks
- 6 public tasks, scored; 2 private, held back so the set can be refreshed
- Runs
- 1 deck per task, 6 tasks
- Leader today
- GPT-6 Astra, 1,593, of 29 models · Last updated 10 Nov 2026
How the judge is checked
A single judge was the strictest of three on unsupported claims (it flagged 24 material claims in a pilot where the other two flagged 12 and 1), and its own model led the pilot. So every judged check is decided by a majority of three judges: GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro. In the pilot they agreed with each other on 97 to 100% of the checks on findings, numbers, recommendations and misleading metrics, and on 89 to 90% of decisions about whether a slide needs work. In head-to-head comparisons a judge never votes on a pair that includes its own vendor's deck. A sample of human ratings is still to come.
Full DeckBench results, outputs and method →
Download: results.csv · results.json · tasks.json · tasks.csv
SurveyBench: survey research
Given a research brief, which models plan a sound user survey, and then draw only the conclusions its responses support?
Survey data rarely says what it seems to at first glance. The risk is not an ugly chart: it is a confident finding from twelve respondents, a satisfaction score asked only of the people who got through, or a stakeholder's hunch written into the question.
Each task is an invented organisation (a meal-kit subscription, a bank, software firms, a council, a GP practice, gyms, a charity) with a brief written for the benchmark. The responses are simulated from respondent-level data with planted effects, so every true number, and every way of being misled, is known before any model sees it.
How a run works
- 1
The brief
An invented organisation's research brief: the decision, four objectives, who to survey, limits on length and sample, and a stakeholder's firm view.
- 2
The survey
The model writes the survey as it would be sent: screener, questions, display logic, sample plan and analysis plan.
- 3
The analysis
Every model then analyses the same fielded survey: crosstabs with bases, fielding notes and open-text answers from simulated respondents.
- 4
Head to head
On each task the judges compare analyses in pairs and pick the one they would rather send to the decision-makers. The votes become the rating.
What a good answer must get right
- Survey questions: leading, double-barrelled, assumptive or ambiguous
- Whether the survey can answer the brief within its limits
- Numbers read from crosstabs, weighted, and tested for significance
- Whether the analysis is misled by weaknesses planted in the data
- Next steps, and what would change the recommendation
Method
- Respondents are simulated, so every true number is known exactly
- Numbers and survey structure are checked by code
- Three judges from three vendors decide each judged check by majority
- No judge compares analyses from its own vendor
- Three tasks are held back so the set can be refreshed
Judged by GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro, by majority.
What it does not test
- Your customers, your data or your survey tool
- Interviews, usability tests or other qualitative research
- Fielding the survey or cleaning raw response files
- Headline measure
- Analysis rating, rating, higher is better
- What the number means
- Head-to-head preference: on each task three judges from three vendors compare analyses of the same survey data in pairs and say which they would rather send to the decision-makers. A Bradley-Terry rating; a 400-point gap is 10:1 odds. No judge votes on its own vendor's analysis.
- Tasks
- 9 public tasks, scored; 3 private, held back so the set can be refreshed
- Runs
- 1 per task
- Leader today
- Claude Fable 5.2, 1,683, of 18 models · Last updated 10 Nov 2026
How the judge is checked
Every judged check is decided by a majority of three judges: GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro. On 12 tasks and seven models, corrected for chance (Fleiss' kappa), they agreed most on next steps (0.81 to 0.85), traps (0.73) and whether a survey faces the precision it was asked for (0.93), and least on double-barrelled questions (0.38), whether every objective can be answered (0.39) and which findings are unsupported (0.49). No judge favoured its own vendor's model. In head-to-head comparisons a judge never votes on a pair that includes its own vendor's analysis. A sample of human ratings is still to come.
Full SurveyBench results, outputs and method →
Download: results.csv · results.json · tasks.json · tasks.csv
ROASBench: ad budgets
Given twelve months of paid-marketing decisions for a simulated brand, which models grow the business and which overspend?
The model runs a year of paid marketing for Northstar Skin, a premium but accessible skincare brand. It controls six channels: Meta prospecting, search, shopping, TikTok, email and remarketing. The score rewards business results, not plans that merely sound good.
How a run works
- 1
The market
One invented skincare brand, Northstar Skin, with fixed economics, audiences, seasonality and shocks.
- 2
The plan
Each month the model returns a plan: budget by channel, campaign types, audiences, ad copy and any discount.
- 3
The simulation
Deterministic rules turn the plan into clicks, customers, refunds, fatigue and saturation.
- 4
The results
The model sees last month's numbers and the state of the business, then plans again. Twelve times.
What a good answer must get right
- Budget allocation across channels, month by month
- Reacting to last month's results
- Staying within budget
Method
- A deterministic simulator scores every plan, so runs are reproducible
- Critical failures (overspend, broken plans) are reported per model
- Scores are not comparable with the 2026 v1 results
What it does not test
- Real ad performance: the market is a deterministic simulation
- Creative quality of the ad copy
- Any brand other than one invented skincare company
- Headline measure
- Score, score out of 100, higher is better
- What the number means
- Average monthly score over a simulated year of paid-marketing plans.
- Tasks
- One simulated year, 12 monthly plans
- Runs
- 1 per model
- Leader today
- Claude Fable 5.2, 61.3, of 29 models · Last updated 9 Nov 2026
Full ROASBench results, outputs and method →
Download: results.csv · results.json
BulletBench: speed under time pressure
When every second of thinking time comes off the clock, which models are fast enough to still make good decisions?
Many products need an answer in about a second: routing, triage, autocomplete, live agents. BulletBench asks which models can still think usefully at that speed. Chess gives a hard, objective score, and the clock makes slow thinking a real cost instead of a free extra.
How a run works
- 1
The position
The model gets the position, the moves so far, its remaining clock and the legal moves.
- 2
The reply
It answers with one move. The whole API round trip, including any hidden reasoning, comes off its clock.
- 3
The engine
Stockfish replies instantly at the model's current ladder level.
- 4
The ladder
Win and the next game is a level up; lose and it is a level down; draw and it stays. The results give a rating with a 95% interval.
What a good answer must get right
- Decision quality under real time pressure
- Response latency at the provider's default settings
- Which reasoning settings are fast enough to use
Method
- Games against a Stockfish ladder; every second of response time comes off the model's clock
- Each model at its default setting and, where offered, its fastest reasoning setting
- Ladder Elo with a 95% interval; it orders models, it is not a FIDE rating
What it does not test
- Any business skill: this is chess against an engine
- Chess strength without a clock
- Headline measure
- Lightning 10+1, ladder Elo, higher is better
- What the number means
- Strength against a Stockfish ladder when every second of response time comes off the model's clock.
- Tasks
- 3 clocks, 8 to 12 games per configuration
- Leader today
- Gemini 3.5 Flash-Lite, 892, of 26 models · Last updated 10 Nov 2026
Full BulletBench results, outputs and method →
Download: results.csv · results.json
Quick answers
What does CatalogBench test?
Which models can turn a sparse product feed, product photos and supplier copy into a listing that could go live, without inventing anything? Scored by rules, then an AI judge. Judged by GPT-6.1 Sol. The headline measure is reliably publish-ready (% of products).
What does DeckBench test?
Given a finished analysis, which models make a deck a manager could present as it is: true to the numbers, telling the right story, and well designed? Scored by code, then three AI judges. Judged by GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro, by majority. The headline measure is deck rating.
What does SurveyBench test?
Given a research brief, which models plan a sound user survey, and then draw only the conclusions its responses support? Scored by code, then three AI judges. Judged by GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro, by majority. The headline measure is analysis rating.
What does ROASBench test?
Given twelve months of paid-marketing decisions for a simulated brand, which models grow the business and which overspend? Scored by a deterministic simulator. The headline measure is overall score (out of 100).
What does BulletBench test?
When every second of thinking time comes off the clock, which models are fast enough to still make good decisions? Scored by game results against Stockfish. The headline measure is ladder Elo (Lightning 10+1).
Are the benchmark tasks public?
Results are scored on public tasks, which each benchmark page shows with real model outputs. CatalogBench, DeckBench and SurveyBench also keep private tasks back, so the set can be refreshed if the public ones leak into training data.
Can I reuse the results?
Yes. Every result from our own benchmarks is free to download as CSV or JSON under Creative Commons Attribution 4.0 International (CC BY 4.0). Credit Spring Prompt (springprompt.com) with a link.