Benchmarks / ROASBench
Given twelve months of paid-marketing decisions for a simulated brand, which models grow the business and which overspend?
Last updated 8 Oct 2026
4 results above zero · months of 12, lower is better · ≈ cannot be told apart from the leader. Choose a model to highlight it.Clear highlight
16 models had none: Claude Fable 5.1, Claude Haiku 5.5, Claude Opus 5.5, Claude Sonnet 5.5, DeepSeek-V4-Pro (0423), GLM-5.3, GPT-6 Astra, GPT-6 Luna, GPT-6 Sol, Gemini 3.1 Pro Preview, Gemini 3.5 Flash-Lite, Gemini 3.8 Flash, Grok 4.7, Kimi K3, Mistral Large 4 (high reasoning), Muse Spark 1.3.
Contribution profit after product costs, refunds and media spend, added up month by month. The five highest-scoring models are in colour, as in the ranking.
GPT-6 Astra has the top score, but Claude Fable 5.1 made the most profit ($965k against $599k). The score rewards each month's return on spend and steady plans, not the year's total profit.
| # | Model | Months over budget months of 12, lower is better | Score score out of 100 | Profit simulated US dollars | Profit per $1 spent profit per $1 spent | Cost of a run US dollars | Months losing money months of 12 with negative return |
|---|---|---|---|---|---|---|---|
| 1≈ | Claude Fable 5.1Anthropic |
0
|
51.5 | $965,271 | $0.64 | $2.49 | 1 |
| 1≈ | Claude Haiku 5.5Anthropic |
0
|
24.1 | $32,146 | $0.03 | $0.0313 | – |
| 1≈ | Claude Opus 5.5Anthropic |
0
|
51.7 | $911,566 | $0.69 | $1.06 | 2 |
| 1≈ | Claude Sonnet 5.5Anthropic |
0
|
52.0 | $571,203 | $0.70 | $0.42 | 2 |
| 1≈ | DeepSeek-V4-Pro (0423)DeepSeek |
0
|
28.3 | $301,289 | $0.26 | $0.20 | – |
| 1≈ | Gemini 3.1 Pro PreviewGoogle |
0
|
43.5 | $507,736 | $0.42 | $0.52 | 2 |
| 1≈ | Gemini 3.5 Flash-LiteGoogle |
0
|
25.4 | $11,834 | $0.01 | $0.0390 | – |
| 1≈ | Gemini 3.8 FlashGoogle |
0
|
51.4 | $842,432 | $0.65 | $0.16 | 1 |
| 1≈ | Muse Spark 1.3Meta |
0
|
34.1 | $337,251 | $0.29 | $0.18 | 3 |
| 1≈ | Mistral Large 4 (high reasoning)Mistral AI |
0
|
27.4 | $156,831 | $0.14 | $0.19 | 6 |
| 1≈ | Kimi K3Moonshot AI |
0
|
44.1 | $654,270 | $0.50 | $1.01 | 2 |
| 1≈ | GPT-6 AstraOpenAI |
0
|
56.6 | $599,139 | $0.67 | $1.23 | 2 |
| 1≈ | GPT-6 LunaOpenAI |
0
|
53.4 | $738,879 | $0.66 | $0.0142 | 1 |
| 1≈ | GPT-6 SolOpenAI |
0
|
54.1 | $600,357 | $0.65 | $0.24 | 1 |
| 1≈ | Grok 4.7xAI |
0
|
36.6 | $187,636 | $0.23 | $0.42 | 5 |
| 1≈ | GLM-5.3Z.ai |
0
|
24.8 | $9,718 | $0.01 | $0.13 | – |
| 17≈ | Qwen3.8-Max (0902)Alibaba |
2
|
25.4 | $99,775 | $0.09 | $0.73 | – |
| 17≈ | Claude Haiku 4.5Anthropic |
2
|
17.5 | −$205,001 | −$0.21 | $0.0298 | 9 |
| 19≈ | Mistral Large 4Mistral AI |
3
|
37.0 | $533,401 | $0.45 | $0.0657 | 3 |
| 19≈ | Mistral Medium 3.5Mistral AI |
3
|
15.2 | −$159,882 | −$0.17 | $0.15 | 10 |
Swipe the table sideways for more columns.
Ranks follow the score as shown, so equal numbers share a rank. Each model has one run. The models marked ≈ are within about 5 points of the leader, which the simulator's run-to-run variation can cover, so read them as a tie until repeat runs measure it. For scale: in v1, where 60 configurations each ran two or three times, a model's score moved by a median of 5.0 points between its best and worst run, and by up to 31.0. v1 scored differently, so this is a guide only. Each model runs at its provider's default reasoning setting. Some providers think at length by default and others barely at all, so this is what you get without tuning.
What two models actually decided in the same month of the simulated year, and what happened. Amounts are simulated US dollars.
Budget $70,650 · no discount
Explore the formula, ingredient list and how to use it. Shop directly from Northstar Skin.
Budget $87,670 · no discount
Repair your skin barrier with Northstar's dermatologist-backed formula. 76% gross margin, 94% brand fit for ingredient researchers. Try it today.
Not scored: this ad quotes the brand's internal figures (gross margin, brand fit) to shoppers. The simulator does not read ad copy for sense, so it neither rewarded nor penalised this; a real ad like it should not run.
API cost of the twelve monthly plans, on a log scale.
Named: the six best and the best for the moneyOther models (hover for names)Best score at each cost
Cheapest first: each model here beats every cheaper one on score.
The four parts of the overall score, each out of 100.
| Model | Business | Planning | Audience | Consistency |
|---|---|---|---|---|
| GPT-6 AstraOpenAI | 57.5 | 54.9 | 68.6 | 44.7 |
| GPT-6 SolOpenAI | 52.9 | 54.8 | 69.4 | 42.9 |
| GPT-6 LunaOpenAI | 54.8 | 54.8 | 68.3 | 35.8 |
| Claude Sonnet 5.5Anthropic | 57.2 | 55.1 | 61.8 | 28.5 |
| Claude Opus 5.5Anthropic | 52.3 | 54.7 | 67.7 | 33.8 |
| Claude Fable 5.1Anthropic | 52.3 | 54.6 | 67.7 | 33.3 |
| Gemini 3.8 FlashGoogle | 51.0 | 54.7 | 67.1 | 36.4 |
| Kimi K3Moonshot AI | 40.2 | 54.7 | 65.1 | 28.3 |
| Gemini 3.1 Pro PreviewGoogle | 35.9 | 54.9 | 67.5 | 33.9 |
| Mistral Large 4Mistral AI | 34.3 | 54.4 | 55.8 | 16.7 |
| Grok 4.7xAI | 26.9 | 54.6 | 58.2 | 30.5 |
| Muse Spark 1.3Meta | 25.8 | 54.8 | 51.6 | 26.7 |
| DeepSeek-V4-Pro (0423)DeepSeek | 23.8 | 54.0 | 44.8 | 9.1 |
| Mistral Large 4 (high reasoning)Mistral AI | 19.3 | 55.0 | 51.3 | 9.7 |
| Qwen3.8-Max (0902)Alibaba | 15.6 | 54.8 | 39.7 | 19.3 |
| Gemini 3.5 Flash-LiteGoogle | 12.6 | 54.9 | 45.0 | 21.8 |
| GLM-5.3Z.ai | 12.9 | 51.7 | 51.0 | 14.8 |
| Claude Haiku 5.5Anthropic | 13.5 | 54.8 | 48.4 | 10.6 |
| Claude Haiku 4.5Anthropic | 7.3 | 54.4 | 30.9 | 8.7 |
| Mistral Medium 3.5Mistral AI | 3.4 | 55.2 | 28.5 | 8.7 |
Planning barely separates models: every one scores between 51.7 and 55.2.
Each bar is one month's spend, split by channel, as a share of that model's biggest month (hover for amounts). A red mark under a month means it lost money.
| Model | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | Months losing money |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 2 | ||||||||||||
| GPT-6 Sol | 1 | ||||||||||||
| GPT-6 Luna | 1 | ||||||||||||
| Claude Sonnet 5.5 | 2 | ||||||||||||
| Claude Opus 5.5 | 2 | ||||||||||||
| Claude Fable 5.1 | 1 | ||||||||||||
| Gemini 3.8 Flash | 1 | ||||||||||||
| Kimi K3 | 2 | ||||||||||||
| Gemini 3.1 Pro Preview | 2 | ||||||||||||
| Mistral Large 4 | 3 | ||||||||||||
| Grok 4.7 | 5 | ||||||||||||
| Muse Spark 1.3 | 3 | ||||||||||||
| Mistral Large 4 (high reasoning) | 6 | ||||||||||||
| Claude Haiku 4.5 | 9 | ||||||||||||
| Mistral Medium 3.5 | 10 |
One invented skincare brand, Northstar Skin, with fixed economics, audiences, seasonality and shocks.
Each month the model returns a plan: budget by channel, campaign types, audiences, ad copy and any discount.
Deterministic rules turn the plan into clicks, customers, refunds, fatigue and saturation.
The model sees last month's numbers and the state of the business, then plans again. Twelve times.
The model runs a year of paid marketing for Northstar Skin, a premium but accessible skincare brand. It controls six channels: Meta prospecting, search, shopping, TikTok, email and remarketing. The score rewards business results, not plans that merely sound good.
Each month's score is 50% business, 20% consistency, 18% audience and 12% planning, and the overall score is the average of the twelve months. Business is scored from that month's return on spend, repeat customers, customer value, acquisition cost and refunds, so a model that spends more at a lower return can make more profit over the year and still score lower.
Version 2 replaced v1's AI audience judge with the simulator's deterministic audience model, so every run is reproducible. Scores are not comparable with v1's.