Benchmarks / SurveyBench
Given a research brief, which models plan a sound user survey, and then draw only the conclusions its responses support?
Last updated 12 Oct 2026
9 results · % of tasks, lower is better · ≈ cannot be told apart from the leader. Choose a model to highlight it.Clear highlight
What one task (a survey plan and an analysis) cost at the run date's prices, on a log scale. Up and to the left is better.
Named: the six best and the best for the moneyOther models (hover for names)Best score at each cost
Cheapest first: each model here beats every cheaper one on score.
| # | Model | Mistaken next steps % of tasks, lower is better | Analysis rating rating | Research score points of 100 | Sound research % of tasks | Traps handled % of traps | Cost per task US dollars |
|---|---|---|---|---|---|---|---|
| 1≈ | Claude Opus 5.5Anthropic |
0.0% 0 of 9 |
1,499 | 96.1 | 55.6% 5 of 9 | 95.7% | $0.46 |
| 1≈ | Muse WonderMeta |
0.0% 0 of 9 |
1,460 | 93.6 | 33.3% 3 of 9 | 100.0% | $0.35 |
| 3 | GPT-6.1 SolOpenAI |
11.1% 1 of 9 |
1,431 | 97.0 | 66.7% 6 of 9 | 95.7% | $0.14 |
| 4 | Claude Haiku 5.5Anthropic |
22.2% 2 of 9 |
1,261 | 90.2 | 11.1% 1 of 9 | 94.2% | $0.0185 |
| 5 | Gemini 3.1 Pro PreviewGoogle |
33.3% 3 of 9 |
792 | 71.3 | 0.0% 0 of 9 | 71.0% | $0.33 |
| 6 | Gemini 3.5 Flash-LiteGoogle |
44.4% 4 of 9 |
632 | 55.6 | 0.0% 0 of 9 | 56.5% | $0.0144 |
| 7 | Claude Haiku 4.5Anthropic |
55.6% 5 of 9 |
495 | 64.9 | 0.0% 0 of 9 | 62.3% | $0.0635 |
| 8 | Mistral Large 4Mistral AI |
66.7% 6 of 9 |
797 | 71.2 | 0.0% 0 of 9 | 78.7% | $0.0924 |
| 9 | Mistral Medium 3.5Mistral AI |
77.8% 7 of 9 |
555 | 64.2 | 0.0% 0 of 9 | 59.4% | $0.0730 |
Swipe the table sideways for more columns.
Ranks follow the score as shown, so equal numbers share a rank. Each model runs at its provider's default reasoning setting, once per task. The rating comes from head-to-head comparisons of the analyses, so it keeps separating models after several pass the checks; ranges are 95% intervals from resampling the comparisons. Models whose ranges overlap are not reliably different.
Mistaken next steps, plotted at each model's release date, lower is better. Each model counts once, at its best setting. Scores are today's, not launch day's. Hover or tap a mark for its model.
Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.
Each lab's best so far, stepping whenever one of its releases beats its own earlier models.
AnthropicMetaOpenAIGoogleMistral AI
Every model's survey and analysis for every public task. Open a task to see the brief and the data every model analysed, or a model to read what it wrote, each number against the true value and what the judges flagged.
Whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →
Where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →
Whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →
Whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →
Whether to move general waste collection to every three weeks across the whole district, to do so with extra support for households that need it (a larger bin, or extra collections of nappies and hygiene waste), to pilot it in some areas first, or to drop the proposal. The brief and the data →
Which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →
Whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →
Whether to close all 14 gyms between 11pm and 5am, close overnight only at the quietest gyms, or keep every gym open 24 hours. The brief and the data →
How to spend next year's £45,000 for volunteer support: reimburse volunteers' travel costs, fund a part-time volunteer coordinator with online shift booking, or do both at a smaller scale. The brief and the data →
Share of the set numeric questions answered correctly, against the share of planted traps in the data the analysis handled. Several models get the arithmetic right and still miss what the data cannot show.
Share of tasks failing each check. A task can fail several at once; any one means the survey or the analysis needs fixing before it is used.
| Model | Traps missed | Wrong numbers | Unsupported findings | Mistaken next steps | Wrong recommendation | Leading or double-barrelled | Survey structure | Objectives unanswerable | Stakeholder's view presumed | Precision ignored |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5Anthropic | 33.3% | 0.0% | 22.2% | 0.0% | 0.0% | 0.0% | 11.1% | 0.0% | 0.0% | 0.0% |
| Muse WonderMeta | 0.0% | 0.0% | 11.1% | 0.0% | 0.0% | 44.4% | 33.3% | 11.1% | 11.1% | 22.2% |
| GPT-6.1 SolOpenAI | 33.3% | 0.0% | 0.0% | 11.1% | 11.1% | 0.0% | 11.1% | 0.0% | 0.0% | 0.0% |
| Claude Haiku 5.5Anthropic | 22.2% | 0.0% | 77.8% | 22.2% | 11.1% | 33.3% | 22.2% | 0.0% | 0.0% | 0.0% |
| Mistral Large 4Mistral AI | 100.0% | 22.2% | 100.0% | 66.7% | 44.4% | 33.3% | 11.1% | 11.1% | 22.2% | 55.6% |
| Gemini 3.1 Pro PreviewGoogle | 88.9% | 0.0% | 77.8% | 33.3% | 44.4% | 44.4% | 22.2% | 0.0% | 22.2% | 55.6% |
| Gemini 3.5 Flash-LiteGoogle | 100.0% | 100.0% | 66.7% | 44.4% | 22.2% | 77.8% | 55.6% | 33.3% | 55.6% | 100.0% |
| Mistral Medium 3.5Mistral AI | 100.0% | 100.0% | 100.0% | 77.8% | 44.4% | 55.6% | 11.1% | 11.1% | 33.3% | 100.0% |
| Claude Haiku 4.5Anthropic | 88.9% | 100.0% | 100.0% | 55.6% | 44.4% | 66.7% | 0.0% | 22.2% | 22.2% | 100.0% |
An invented organisation's research brief: the decision, four objectives, who to survey, limits on length and sample, and a stakeholder's firm view.
The model writes the survey as it would be sent: screener, questions, display logic, sample plan and analysis plan.
Every model then analyses the same fielded survey: crosstabs with bases, fielding notes and open-text answers from simulated respondents.
On each task the judges compare analyses in pairs and pick the one they would rather send to the decision-makers. The votes become the rating.
Survey data rarely says what it seems to at first glance. The risk is not an ugly chart: it is a confident finding from twelve respondents, a satisfaction score asked only of the people who got through, or a stakeholder's hunch written into the question.
Each task is an invented organisation (a meal-kit subscription, a bank, software firms, a council, a GP practice, gyms, a charity) with a brief written for the benchmark. The responses are simulated from respondent-level data with planted effects, so every true number, and every way of being misled, is known before any model sees it.
Sound research is a bar: once the best models clear it on most tasks, it stops telling them apart. The head-to-head rating does not run out of room, because an analysis only has to be preferred to another one. Each analysis meets about six others per task; a 400-point gap means the higher-rated model's analysis is preferred about ten times out of eleven.
Every judged check is decided by a majority of three judges: GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro. On 12 tasks and seven models, corrected for chance (Fleiss' kappa), they agreed most on next steps (0.81 to 0.85), traps (0.73) and whether a survey faces the precision it was asked for (0.93), and least on double-barrelled questions (0.38), whether every objective can be answered (0.39) and which findings are unsupported (0.49). No judge favoured its own vendor's model. In head-to-head comparisons a judge never votes on a pair that includes its own vendor's analysis. A sample of human ratings is still to come.