Benchmarks / Microsoft STATE-Bench

Reported by Microsoft STATE-Bench

Microsoft STATE-Bench

Share of shopping assistant tasks completed correctly, averaged over repeated runs.

Data older than 90 days: last updated 1 Jul 2026

Results dated
25 May 2026 to 29 May 2026
Results
5 configurations of 4 models
Unit
% of tasks
Licence
MIT License

Microsoft STATE-Bench in brief

What it measures. Microsoft STATE-Bench is a third-party benchmark, shown here with credit. Measured here: Share of shopping assistant tasks completed correctly, averaged over repeated runs.

Latest results. , 5 configurations ranked.

Top 3 on Shopping assistant task success (pass@1) (% of tasks, higher is better):

  1. Claude Opus 4.7 (high reasoning) (Anthropic): 54.0%
  2. GPT-5.4 (high reasoning) (OpenAI): 53.6%
  3. GPT-5.4 (default reasoning) (OpenAI): 50.3%

How to cite. Microsoft STATE-Bench, via Spring Prompt, results dated 29 May 2026. https://springprompt.com/benchmarks/statebench These results belong to Microsoft STATE-Bench, not to us. Licence: MIT License. Credit Microsoft STATE-Bench first.

Shopping assistant task success (pass@1):

5 results · % of tasks, higher is better. Choose a model to highlight it.Clear highlight

  1. 1 Claude Opus 4.7 (high reasoning)Anthropic 54.0%
  2. 2 GPT-5.4 (high reasoning)OpenAI 53.6%
  3. 3 GPT-5.4 (default reasoning)OpenAI 50.3%
  4. 4 DeepSeek-V4-Pro (0423)DeepSeek 48.4%
  5. 5 Kimi K2.6Moonshot AI 47.9%

Full results

STATE-Bench v0.7: shopping assistant task success (pass@1), % of tasks, higher is better
#ModelShopping assistant task success (pass@1)
% of tasks, higher is better
Price
$ per million tokens, in / out
1 Claude Opus 4.7 (high reasoning)Anthropic
54.0%
$5 / $25
2 GPT-5.4 (high reasoning)OpenAI
53.6%
$2.50 / $15
3 GPT-5.4 (default reasoning)OpenAI
50.3%
$2.50 / $15
4 DeepSeek-V4-Pro (0423)DeepSeek
48.4%
$1.42 / $2.83
5 Kimi K2.6Moonshot AI
47.9%
$0.95 / $4

Swipe the table sideways for more columns.

Source: Microsoft STATE-Bench, via Spring Prompt, springprompt.com/benchmarks/statebench. Test your own prompt on these models →

Ranks follow the score as shown, so equal numbers share a rank. Results as published by Microsoft STATE-Bench; we do not re-run them.

Progress over time

Shopping assistant task success (pass@1), plotted at each model's release date, higher is better. Each model counts once, at its best setting. Scores are today's, not launch day's. Hover or tap a mark for its model.

Best score so far

Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.

46%48%50%52%54%56% Apr 2026JulOct GPT-5.4Claude Opus 4.7

Best score by lab

Each lab's best so far, stepping whenever one of its releases beats its own earlier models.

AnthropicOpenAIDeepSeekMoonshot AI

46%48%50%52%54%56% Apr 2026JulOct AnthropicOpenAIDeepSeekMoonshot AI

What it measures

Share of shopping assistant tasks completed correctly, averaged over repeated runs.

What it does not measure

Not the model alone: it runs inside STATE-Bench's agent loop against simulated users, and versions of the benchmark are not comparable.

4 results from Microsoft STATE-Bench not ranked here · show why

We rank a result only when we can tie it to a specific model you can use. These are left out:

  • Agent systems with memory, not models on their own: 4

STATE-Bench by Microsoft and STATE-Bench contributors, MIT License.