Benchmarks / Microsoft STATE-Bench
Share of shopping assistant tasks completed correctly, averaged over repeated runs.
Data older than 90 days: last updated 1 Jul 2026
4 results · % of tasks, higher is better. Choose a model to highlight it.Clear highlight
| # | Model | Shopping assistant task success (pass@1) % of tasks, higher is better | Price $ per million tokens, in / out |
|---|---|---|---|
| 1 | Claude Opus 4.7 (high reasoning)Anthropic |
54.0%
|
$5 / $25 |
| 2 | GPT-5.4 (high reasoning)OpenAI · best of 2 settings |
53.6%
|
$2.50 / $15 |
| 3 | DeepSeek-V4-Pro (0423)DeepSeek |
48.4%
|
$1.42 / $2.83 |
| 4 | Kimi K2.6Moonshot AI |
47.9%
|
$0.95 / $4 |
Swipe the table sideways for more columns.
Ranks follow the score as shown, so equal numbers share a rank. Each model is shown at its best setting; show every setting. Results as published by Microsoft STATE-Bench; we do not re-run them.
Shopping assistant task success (pass@1), plotted at each model's release date, higher is better. Each model counts once, at its best setting. Scores are today's, not launch day's. Hover or tap a mark for its model.
Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.
Each lab's best so far, stepping whenever one of its releases beats its own earlier models.
AnthropicOpenAIDeepSeekMoonshot AI
Share of shopping assistant tasks completed correctly, averaged over repeated runs.
Not the model alone: it runs inside STATE-Bench's agent loop against simulated users, and versions of the benchmark are not comparable.
We rank a result only when we can tie it to a specific model you can use. These are left out:
STATE-Bench by Microsoft and STATE-Bench contributors, MIT License.