Benchmarks / ROASBench
Measured by Spring Prompt
ROASBench
Given twelve months of paid-marketing decisions for a simulated brand, which models grow the business and which overspend?
- Results dated
- 29 Sep 2026
- Models
- 17
- Unit
- profit per £1 spent
- Licence
- Spring Prompt original
| # | Model | Return on ad spend profit per £1 spent, higher is better | Overall score score out of 100 | Months over budget months of 12 | Cost of a run US dollars |
|---|---|---|---|---|---|
| 1 | Claude Sonnet 5.5Anthropic |
£0.70
|
52.0 | 0 | $0.42 |
| 2 | Claude Opus 5.5Anthropic |
£0.69
|
51.6 | 0 | $1.06 |
| 3 | GPT-6 AstraOpenAI |
£0.67
|
56.6 | 0 | $1.23 |
| 4 | GPT-6 LunaOpenAI |
£0.66
|
53.4 | 0 | $0.0142 |
| 5 | GPT-6 SolOpenAI |
£0.65
|
54.1 | 0 | $0.24 |
| 6 | Gemini 3.8 FlashGoogle |
£0.65
|
51.4 | 0 | $0.16 |
| 7 | Claude Fable 5.1Anthropic |
£0.64
|
51.5 | 0 | $2.49 |
| 8 | Kimi K3Moonshot AI |
£0.50
|
44.1 | 0 | $1.01 |
| 9 | Gemini 3.1 Pro PreviewGoogle |
£0.42
|
43.5 | 0 | $0.52 |
| 10 | Muse Spark 1.3Meta |
£0.29
|
34.1 | 0 | $0.18 |
| 11 | DeepSeek V4 Pro 0423DeepSeek |
£0.26
|
28.2 | 0 | $0.20 |
| 12 | Grok 4.7xAI |
£0.23
|
36.6 | 0 | $0.42 |
| 13 | Qwen3.8 Max (0902)Alibaba |
£0.09
|
25.4 | 2 | $0.73 |
| 14 | Gemini 3.5 Flash LiteGoogle |
£0.01
|
25.4 | 0 | $0.0390 |
| 15 | GLM 5.3Z.ai |
£0.01
|
24.8 | 0 | $0.13 |
| 16 | Mistral Medium 3.5Mistral |
−£0.17
|
15.2 | 3 | $0.15 |
| 17 | Claude Haiku 4.5Anthropic |
−£0.21
|
17.5 | 2 | $0.0298 |
Each model runs at its provider's default reasoning setting. Some providers think at length by default and others barely at all, so this is what you get without tuning.
What it measures
- Budget allocation across channels, month by month
- Reacting to last month's results
- Staying within budget
What it does not measure
- Real ad performance: the market is a deterministic simulation
- Creative quality of the ad copy
- Any brand other than one invented skincare company
Method
- A deterministic simulator scores every plan, so runs are reproducible
- Critical failures (overspend, broken plans) are reported per model
- Scores are not comparable with the 2026 v1 results