Benchmarks / tau2-bench

Reported by tau2-bench

tau2-bench

Share of banking questions answered from a policy knowledge base tasks completed within policy in all four trials.

Data older than 90 days: last updated 3 Aug 2026

Results dated
5 May 2026 to 3 Aug 2026
Models
21
Unit
% of tasks
Licence
MIT License

tau2-bench in brief

What it measures. tau2-bench is a third-party benchmark, shown here with credit. Measured here: Share of banking questions answered from a policy knowledge base tasks completed within policy in all four trials.

Latest results. , 21 models ranked.

Top 3 on Consistency (pass^4) (% of tasks, higher is better):

  1. Qwen3.8-Max (0902, extra-high reasoning) (Alibaba): 35.1%
  2. Claude Opus 5 (max reasoning) (Anthropic): 32.0%
  3. Grok 4.5 (high reasoning) (xAI): 32.0%

How to cite. tau2-bench, via Spring Prompt, results dated 3 August 2026. https://springprompt.com/benchmarks/tau2-bench These results belong to tau2-bench, not to us. Licence: MIT License. Credit tau2-bench first.

Consistency (pass^4): Claude Opus 5

Top 15 of 21 results · % of tasks, higher is better. Choose a model to highlight it.Clear highlight

  1. 1 Qwen3.8-Max (0902, extra-high reasoning)Alibaba 35.1%
  2. 2 Claude Opus 5 (max reasoning)Anthropic 32.0%
  3. 2 Grok 4.5 (high reasoning)xAI 32.0%
  4. 4 GPT-5.5 (extra-high reasoning)OpenAI 29.9%
  5. 5 Claude Fable 5 (max reasoning)Anthropic 28.9%
  6. 6 GPT-5.6 Sol (extra-high reasoning)OpenAI 27.8%
  7. 7 Claude Opus 4.7 (max reasoning)Anthropic 24.7%
  8. 8 Claude Opus 4.8 (max reasoning)Anthropic 22.7%
  9. 9 GPT-5.4 (extra-high reasoning)OpenAI 21.7%
  10. 10 Muse Spark 1.1 (extra-high reasoning)Meta 20.6%
  11. 11 GPT-5.2 (high reasoning)OpenAI 18.6%
  12. 12 Kimi K3 (max reasoning)Moonshot AI 17.5%
  13. 13 GLM-5.2 (extra-high reasoning)Z.ai 13.4%
  14. 14 Inkling (max reasoning)Thinking Machines 11.3%
  15. 14 Claude Opus 4.5 (high reasoning)Anthropic 11.3%

Full results

tau2 v1.0.1, banking knowledge, alltools retrieval: consistency (pass^4), % of tasks, higher is better
#ModelConsistency (pass^4)
% of tasks, higher is better
Price
$ per million tokens, in / out
1 Qwen3.8-Max (0902, extra-high reasoning)Alibaba
35.1%
$2 / $6
2 Claude Opus 5 (max reasoning)Anthropic
32.0%
$5 / $25
2 Grok 4.5 (high reasoning)xAI
32.0%
$2 / $6
4 GPT-5.5 (extra-high reasoning)OpenAI
29.9%
$5 / $30
5 Claude Fable 5 (max reasoning)Anthropic
28.9%
$10 / $50
6 GPT-5.6 Sol (extra-high reasoning)OpenAI
27.8%
$4 / $20
7 Claude Opus 4.7 (max reasoning)Anthropic
24.7%
$5 / $25
8 Claude Opus 4.8 (max reasoning)Anthropic
22.7%
$5 / $25
9 GPT-5.4 (extra-high reasoning)OpenAI
21.7%
$2.50 / $15
10 Muse Spark 1.1 (extra-high reasoning)Meta
20.6%
$1.25 / $4.25
11 GPT-5.2 (high reasoning)OpenAI
18.6%
$1.75 / $14
12 Kimi K3 (max reasoning)Moonshot AI
17.5%
$3 / $15
13 GLM-5.2 (extra-high reasoning)Z.ai
13.4%
$1.40 / $4.40
14 Inkling (max reasoning)Thinking Machines
11.3%
$1 / $4.05
14 Claude Opus 4.5 (high reasoning)Anthropic
11.3%
$5 / $25
14 Claude Opus 4.6 (max reasoning)Anthropic
11.3%
$5 / $25
17 Gemini 3.1 Pro Preview (high reasoning)Google
9.3%
$2 / $12
18 Grok 4.20 (high reasoning)xAI
8.3%
$1.25 / $2.50
19 Grok 4.1 Fast (high reasoning)xAI
5.2%
–
20 Grok 4 Fast (high reasoning)xAI
4.1%
–
21 Gemini 2.5 Pro (high reasoning)Google
1.0%
$1.25 / $10

Swipe the table sideways for more columns.

Source: tau2-bench, via Spring Prompt, springprompt.com/benchmarks/tau2-bench. Test your own prompt on these models →

Ranks follow the score as shown, so equal numbers share a rank. Results as published by tau2-bench; we do not re-run them.

Progress over time

Consistency (pass^4), plotted at each model's release date, higher is better. Each model counts once, at its best setting. Scores are today's, not launch day's; 1 model without a known release date is left off. Hover or tap a mark for its model.

Best score so far

Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.

-10%0%10%20%30%40% Apr 2025JulOct2026AprJulOct Gemini 2.5 ProGrok 4 FastClaude Opus 4.5GPT-5.2GPT-5.4Claude Opus 4.7GPT-5.5Grok 4.5Qwen3.8-Max (0902)

Best score by lab

Each lab's best so far, stepping whenever one of its releases beats its own earlier models. The 6 labs with the best scores today; 3 others not drawn.

AlibabaxAIAnthropicOpenAIMetaMoonshot AI

0%10%20%30%40% Apr 2025JulOct2026AprJulOct AlibabaxAIAnthropicOpenAIMetaMoonshot AI

What it measures

Share of banking questions answered from a policy knowledge base tasks completed within policy in all four trials.

What it does not measure

Not your policies or customers: the tasks and policies are tau2's own, and the customer is simulated by gpt-5.2.

2 results from tau2-bench not ranked here · show why

We rank a result only when we can tie it to a specific model you can use. These are left out:

  • A custom agent harness, not the standard one: 2

tau2-bench by Sierra Research and contributors, MIT License.