Measured by Spring Prompt

SurveyBench

Name: SurveyBench results
Creator: Spring Prompt

Given a research brief, which models plan a sound user survey, and then draw only the conclusions its responses support?

Last updated 12 Nov 2026

Results dated: 8 Oct 2026 to 12 Nov 2026
Results: 24 configurations of 23 models
Unit: % of tasks
Licence: Spring Prompt original
Judge: GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro, by majority (jp-a06fd450f8ab, jp-ae13ebd1f24b) · bias check
Runs: 1 per task

SurveyBench in brief

What it measures. SurveyBench is a Spring Prompt benchmark. Can a model plan a user survey and read its results without being misled? It scores: Survey questions: leading, double-barrelled, assumptive or ambiguous; Whether the survey can answer the brief within its limits; Numbers read from crosstabs, weighted, and tested for significance; Whether the analysis is misled by weaknesses planted in the data; Next steps, and what would change the recommendation.

Latest results. 12 November 2026, 24 configurations ranked.

Top 3 on Mistaken next steps (% of tasks, lower is better):

Claude Fable 5.2 (Anthropic): 0.0%
Claude Opus 5.5 (Anthropic): 0.0%
DeepSeek-V4.1-Flash-1017 (DeepSeek): 0.0%

The first 4 are too close to tell apart, so read the order among them as a tie.

How to cite. Spring Prompt, "SurveyBench", results dated 12 November 2026. https://springprompt.com/benchmarks/surveybench

Mistaken next steps: Gemini 4 Hydrogen

20 results above zero · % of tasks, lower is better. Choose a model to highlight it.Clear highlight

5 Gemini 3.8 FlashGoogle 11.1%
5 Gemini 4 HydrogenGoogle 11.1%
5 GPT-6.1 SolOpenAI 11.1%
5 GPT-6.1 TerraOpenAI 11.1%
5 GPT-6 LunaOpenAI 11.1%
10 Claude Haiku 5.5Anthropic 22.2%
10 Gemini 4 CarbonGoogle 22.2%
10 Mercury 3Inception 22.2%
10 Muse EmberMeta 22.2%
10 Particle-1Particle AI 22.2%
15 Gemini 3.1 Pro PreviewGoogle 33.3%
15 Gemini 4 ArgonGoogle 33.3%
17 Gemini 3.5 Flash-LiteGoogle 44.4%
17 GLM-5.3 FlashX LegalZ.ai 44.4%
19 Claude Haiku 4.5Anthropic 55.6%

4 models had none: Claude Fable 5.2, Claude Opus 5.5, DeepSeek-V4.1-Flash-1017, Muse Wonder.

Analysis rating against cost

What one task (a survey plan and an analysis) cost at the run date's prices, on a log scale. Up and to the left is better.

Named: the six best and the best for the moneyOther models (hover for names)Best score at each cost

Best per budget

GPT-6 Luna: 1,106 at $0.0088
Particle-1: 1,163 at $0.0140
Claude Haiku 5.5: 1,252 at $0.0185
Muse Ember: 1,393 at $0.0213
GPT-6.1 Sol: 1,446 at $0.14
Muse Wonder: 1,469 at $0.35
Claude Fable 5.2: 1,665 at $1.51

Cheapest first: each model here beats every cheaper one on score.

Full results

SurveyBench: mistaken next steps, % of tasks, lower is better
#	Model	Mistaken next steps % of tasks, lower is better	Analysis rating rating	Research score points of 100	Sound research % of tasks	Traps handled % of traps	Cost per task US dollars	Cost per sound task US dollars
1≈	Claude Fable 5.2Anthropic	0.0% 0 of 9	1,665	97.6	44.4% 4 of 9	98.6%	$1.51	$3.40
1≈	Claude Opus 5.5Anthropic	0.0% 0 of 9	1,445	96.1	55.6% 5 of 9	95.7%	$0.46	$0.82
1≈	DeepSeek-V4.1-Flash-1017DeepSeek	0.0% 0 of 9	1,146	90.9	33.3% 3 of 9	91.3%	$0.0373	$0.11
1≈	Muse WonderMeta	0.0% 0 of 9	1,469	93.6	33.3% 3 of 9	100.0%	$0.35	$1.06
5	Gemini 3.8 FlashGoogle	11.1% 1 of 9	886	77.6	0.0% 0 of 9	78.3%	$0.0943	–
5	Gemini 4 HydrogenGoogle	11.1% 1 of 9	1,117	83.7	11.1% 1 of 9	89.9%	$0.34	$3.03
5	GPT-6.1 SolOpenAI	11.1% 1 of 9	1,446	97.0	66.7% 6 of 9	95.7%	$0.14	$0.21
5	GPT-6.1 TerraOpenAI	11.1% 1 of 9	1,263	92.2	33.3% 3 of 9	94.2%	$0.0569	$0.17
5	GPT-6 LunaOpenAI	11.1% 1 of 9	1,106	91.7	33.3% 3 of 9	94.2%	$0.0088	$0.0264
10	Claude Haiku 5.5Anthropic	22.2% 2 of 9	1,252	90.2	11.1% 1 of 9	94.2%	$0.0185	$0.17
10	Gemini 4 CarbonGoogle	22.2% 2 of 9	1,067	87.0	11.1% 1 of 9	92.8%	$0.31	$2.78
10	Mercury 3Inception	22.2% 2 of 9	753	56.8	0.0% 0 of 9	60.7%	$0.0216	–
10	Muse EmberMeta	22.2% 2 of 9	1,393	86.9	22.2% 2 of 9	98.4%	$0.0213	–
10	Particle-1Particle AI	22.2% 2 of 9	1,163	89.7	22.2% 2 of 9	95.7%	$0.0140	$0.0629
15	Gemini 3.1 Pro PreviewGoogle	33.3% 3 of 9	769	71.3	0.0% 0 of 9	71.0%	$0.33	–
15	Gemini 4 ArgonGoogle	33.3% 3 of 9	954	83.3	0.0% 0 of 9	84.1%	$0.13	–
17	Gemini 3.5 Flash-LiteGoogle	44.4% 4 of 9	658	55.6	0.0% 0 of 9	56.5%	$0.0144	–
17	GLM-5.3 FlashX LegalZ.ai	44.4% 4 of 9	965	59.2	11.1% 1 of 9	83.0%	$0.0693	–
19	Claude Haiku 4.5Anthropic	55.6% 5 of 9	580	64.9	0.0% 0 of 9	62.3%	$0.0635	–
20	Mistral Large 4Mistral AI	66.7% 6 of 9	796	71.2	0.0% 0 of 9	78.7%	$0.0924	–
21	MiniMax M3.1 FlashMiniMax	77.8% 7 of 9	453	28.3	0.0% 0 of 9	66.7%	$0.0198	–
21	Mistral Medium 3.5Mistral AI	77.8% 7 of 9	598	64.2	0.0% 0 of 9	59.4%	$0.0730	–
21	Kimi K3.1 ProMoonshot AI	77.8% 7 of 9	620	39.2	0.0% 0 of 9	93.3%	$0.74	–
24	MiniMax M3.1 FlashMiniMax	100.0% 9 of 9	444	16.4	0.0% 0 of 9	75.0%	–	–

Swipe the table sideways for more columns.

Source: Spring Prompt, springprompt.com/benchmarks/surveybench. Test your own prompt on these models →

Ranks follow the score as shown, so equal numbers share a rank. Each model runs at its provider's default reasoning setting, once per task. The rating comes from head-to-head comparisons of the analyses, so it keeps separating models after several pass the checks; ranges are 95% intervals from resampling the comparisons. Models whose ranges overlap are not reliably different.

Progress over time

Mistaken next steps, plotted at each model's release date, lower is better. Each model counts once, at its best setting. Scores are today's, not launch day's; 2 models without a known release date are left off. Hover or tap a mark for its model.

Best score so far

Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.

Best score by lab

Each lab's best so far, stepping whenever one of its releases beats its own earlier models. The 6 labs with the best scores today; 4 others not drawn.

AnthropicMetaDeepSeekGoogleOpenAIInception

See the surveys and analyses

Every model's survey and analysis for every public task. Open a task to see the brief and the data every model analysed, or a model to read what it wrote, each number against the true value and what the judges flagged.

Why Fernway customers cancel · 2 of 23 sound

Whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Why Tallyroom trials don't convert · 3 of 23 sound

Where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Why Pennywell customers don't use savings pots · 5 of 23 sound

Whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Why Quaymark's customer score fell · 7 of 23 sound

Whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

What Hollinbrook residents think of three-weekly bin collections · 4 of 23 sound

Whether to move general waste collection to every three weeks across the whole district, to do so with extra support for households that need it (a larger bin, or extra collections of nappies and hygiene waste), to pilot it in some areas first, or to drop the proposal. The brief and the data →

What Ledgerline's accountancy practices want built next · 5 of 23 sound

Which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

How Ashgrove patients find online-first booking · 3 of 23 sound

Whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Should Corefit gyms stop opening 24 hours? · 2 of 23 sound

Whether to close all 14 gyms between 11pm and 5am, close overnight only at the quietest gyms, or keep every gym open 24 hours. The brief and the data →

Why Riverside Food Network is losing volunteers · 3 of 23 sound

How to spend next year's £45,000 for volunteer support: reimburse volunteers' travel costs, fund a part-time volunteer coordinator with online shift booking, or do both at a smaller scale. The brief and the data →

More from the results

Right numbers are not the same as a sound reading

Share of the set numeric questions answered correctly, against the share of planted traps in the data the analysis handled. Several models get the arithmetic right and still miss what the data cannot show.

Numbers rightTraps handled

Claude Fable 5.2

100.0% · 98.6%

Claude Haiku 5.5

100.0% · 94.2%

Claude Opus 5.5

100.0% · 95.7%

DeepSeek-V4.1-Flash-1017

100.0% · 91.3%

Gemini 3.1 Pro Preview

100.0% · 71.0%

Gemini 3.8 Flash

100.0% · 78.3%

Gemini 4 Carbon

100.0% · 92.8%

Gemini 4 Hydrogen

100.0% · 89.9%

Muse Ember

100.0% · 98.4%

Muse Wonder

100.0% · 100.0%

MiniMax M3.1 Flash

100.0% · 75.0%

Kimi K3.1 Pro

100.0% · 93.3%

GPT-6.1 Sol

100.0% · 95.7%

GPT-6.1 Terra

100.0% · 94.2%

Particle-1

100.0% · 95.7%

Mistral Large 4

98.9% · 78.7%

GPT-6 Luna

98.0% · 94.2%

Gemini 4 Argon

97.1% · 84.1%

GLM-5.3 FlashX Legal

94.9% · 83.0%

MiniMax M3.1 Flash

90.9% · 66.7%

Gemini 3.5 Flash-Lite

75.5% · 56.5%

Mistral Medium 3.5

75.5% · 59.4%

Claude Haiku 4.5

63.7% · 62.3%

Mercury 3

63.3% · 60.7%

Why the research is not sound

Share of tasks failing each check. A task can fail several at once; any one means the survey or the analysis needs fixing before it is used.

Model	Traps missed	Wrong numbers	Unsupported findings	Mistaken next steps	Wrong recommendation	Leading or double-barrelled	Survey structure	Objectives unanswerable	Stakeholder's view presumed	Precision ignored
Claude Fable 5.2Anthropic	11.1%	0.0%	11.1%	0.0%	0.0%	22.2%	11.1%	0.0%	0.0%	0.0%
Muse WonderMeta	0.0%	0.0%	11.1%	0.0%	0.0%	44.4%	33.3%	11.1%	11.1%	22.2%
GPT-6.1 SolOpenAI	33.3%	0.0%	0.0%	11.1%	11.1%	0.0%	11.1%	0.0%	0.0%	0.0%
Claude Opus 5.5Anthropic	33.3%	0.0%	22.2%	0.0%	0.0%	0.0%	11.1%	0.0%	0.0%	0.0%
Muse EmberMeta	22.2%	11.1%	22.2%	22.2%	22.2%	33.3%	22.2%	0.0%	0.0%	11.1%
GPT-6.1 TerraOpenAI	33.3%	0.0%	0.0%	11.1%	11.1%	11.1%	22.2%	11.1%	11.1%	0.0%
Claude Haiku 5.5Anthropic	22.2%	0.0%	77.8%	22.2%	11.1%	33.3%	22.2%	0.0%	0.0%	0.0%
Particle-1Particle AI	33.3%	0.0%	55.6%	22.2%	11.1%	11.1%	33.3%	11.1%	0.0%	0.0%
DeepSeek-V4.1-Flash-1017DeepSeek	44.4%	0.0%	44.4%	0.0%	11.1%	11.1%	33.3%	11.1%	0.0%	0.0%
Gemini 4 HydrogenGoogle	66.7%	0.0%	33.3%	11.1%	33.3%	44.4%	11.1%	0.0%	22.2%	0.0%
GPT-6 LunaOpenAI	33.3%	11.1%	22.2%	11.1%	11.1%	11.1%	22.2%	0.0%	11.1%	0.0%
Gemini 4 CarbonGoogle	44.4%	0.0%	77.8%	22.2%	22.2%	55.6%	11.1%	0.0%	11.1%	11.1%
GLM-5.3 FlashX LegalZ.ai	66.7%	33.3%	77.8%	44.4%	55.6%	44.4%	44.4%	22.2%	22.2%	22.2%
Gemini 4 ArgonGoogle	77.8%	33.3%	88.9%	33.3%	11.1%	66.7%	0.0%	0.0%	11.1%	11.1%
Gemini 3.8 FlashGoogle	88.9%	0.0%	77.8%	11.1%	0.0%	66.7%	22.2%	0.0%	22.2%	44.4%
Mistral Large 4Mistral AI	100.0%	22.2%	100.0%	66.7%	44.4%	33.3%	11.1%	11.1%	22.2%	55.6%
Gemini 3.1 Pro PreviewGoogle	88.9%	0.0%	77.8%	33.3%	44.4%	44.4%	22.2%	0.0%	22.2%	55.6%
Mercury 3Inception	88.9%	44.4%	44.4%	22.2%	77.8%	55.6%	22.2%	0.0%	22.2%	0.0%
Gemini 3.5 Flash-LiteGoogle	100.0%	100.0%	66.7%	44.4%	22.2%	77.8%	55.6%	33.3%	55.6%	100.0%
Kimi K3.1 ProMoonshot AI	88.9%	77.8%	77.8%	77.8%	100.0%	22.2%	11.1%	0.0%	0.0%	0.0%
Mistral Medium 3.5Mistral AI	100.0%	100.0%	100.0%	77.8%	44.4%	55.6%	11.1%	11.1%	33.3%	100.0%
Claude Haiku 4.5Anthropic	88.9%	100.0%	100.0%	55.6%	44.4%	66.7%	0.0%	22.2%	22.2%	100.0%
MiniMax M3.1 FlashMiniMax	88.9%	100.0%	88.9%	77.8%	88.9%	44.4%	44.4%	44.4%	33.3%	44.4%
MiniMax M3.1 FlashMiniMax	100.0%	88.9%	100.0%	100.0%	100.0%	77.8%	77.8%	66.7%	55.6%	66.7%

How SurveyBench works

1
The brief
An invented organisation's research brief: the decision, four objectives, who to survey, limits on length and sample, and a stakeholder's firm view.
2
The survey
The model writes the survey as it would be sent: screener, questions, display logic, sample plan and analysis plan.
3
The analysis
Every model then analyses the same fielded survey: crosstabs with bases, fielding notes and open-text answers from simulated respondents.
4
Head to head
On each task the judges compare analyses in pairs and pick the one they would rather send to the decision-makers. The votes become the rating.

The task

Survey data rarely says what it seems to at first glance. The risk is not an ugly chart: it is a confident finding from twelve respondents, a satisfaction score asked only of the people who got through, or a stakeholder's hunch written into the question.

Each task is an invented organisation (a meal-kit subscription, a bank, software firms, a council, a GP practice, gyms, a charity) with a brief written for the benchmark. The responses are simulated from respondent-level data with planted effects, so every true number, and every way of being misled, is known before any model sees it.

What makes the research sound

The survey: Within the brief's limits on questions and minutes; every objective answerable; no leading or double-barrelled question; the stakeholder's view tested neutrally; and a sample plan that says plainly when the precision asked for cannot be met.
The numbers: Ten to twelve set questions per task: shares, weighted figures, margins of error, significance tests, differences and counts of open-text answers, each checked against the true value.
The traps: Skewed samples, small bases, questions asked of only some respondents, averages pulled by a few extreme answers, trends broken by a change of method, stated intentions, organised campaigns and reversals within subgroups.
The advice: A recommendation that follows from the data, next steps that would settle what the survey cannot, and none of the mistakes a careless reading leads to.

Why a rating as well

Sound research is a bar: once the best models clear it on most tasks, it stops telling them apart. The head-to-head rating does not run out of room, because an analysis only has to be preferred to another one. Each analysis meets about six others per task; a 400-point gap means the higher-rated model's analysis is preferred about ten times out of eleven.

What it measures

Survey questions: leading, double-barrelled, assumptive or ambiguous
Whether the survey can answer the brief within its limits
Numbers read from crosstabs, weighted, and tested for significance
Whether the analysis is misled by weaknesses planted in the data
Next steps, and what would change the recommendation

What it does not measure

Your customers, your data or your survey tool
Interviews, usability tests or other qualitative research
Fielding the survey or cleaning raw response files

Method

Respondents are simulated, so every true number is known exactly
Numbers and survey structure are checked by code
Three judges from three vendors decide each judged check by majority
No judge compares analyses from its own vendor
Three tasks are held back so the set can be refreshed

How all our benchmarks work

Use the data

Every result on this page is free to reuse under CC BY 4.0: results.csv · results.json. Credit Spring Prompt with a link. All downloads · Embed the chart

Embed this chart

Put SurveyBench's current top 10 in your article. Free to reuse under CC BY 4.0; the code includes the credit link. Add ?n=5 for fewer rows, ?models= with model slugs (as in our model page addresses) for your own comparison, or ?theme=dark.

Interactive chart

<iframe src="https://springprompt.com/embed/surveybench" title="SurveyBench results by Spring Prompt" width="100%" height="456" style="border:0;max-width:720px" loading="lazy"></iframe>
<p><a href="https://springprompt.com/benchmarks/surveybench">SurveyBench results by Spring Prompt</a></p>

Image, for sites that block iframes

<a href="https://springprompt.com/benchmarks/surveybench"><img src="https://springprompt.com/og/benchmarks/surveybench.png" alt="SurveyBench: the top 10 AI models by analysis rating" width="1200" height="630" style="max-width:100%;height:auto"></a>
<p><a href="https://springprompt.com/benchmarks/surveybench">SurveyBench results by Spring Prompt</a></p>

See the image

Checking the judge

Every judged check is decided by a majority of three judges: GPT-6.1 Sol, Claude Opus 5.5 and Gemini 3.1 Pro. On 12 tasks and seven models, corrected for chance (Fleiss' kappa), they agreed most on next steps (0.81 to 0.85), traps (0.73) and whether a survey faces the precision it was asked for (0.93), and least on double-barrelled questions (0.38), whether every objective can be answered (0.39) and which findings are unsupported (0.49). No judge favoured its own vendor's model. In head-to-head comparisons a judge never votes on a pair that includes its own vendor's analysis. A sample of human ratings is still to come.

SurveyBench

SurveyBench in brief

Mistaken next steps: Gemini 4 Hydrogen

Analysis rating against cost

Best per budget

Full results

Progress over time

Best score so far

Best score by lab

See the surveys and analyses

More from the results

Right numbers are not the same as a sound reading

Why the research is not sound

How SurveyBench works

The brief

The survey

The analysis

Head to head

The task

What makes the research sound

Why a rating as well

What it measures

What it does not measure

Method

Use the data

Embed this chart

Interactive chart

Image, for sites that block iframes

Checking the judge