Reported by tau2-bench

tau2-bench

Name: tau2-bench results
Creator: tau2-bench
License: https://github.com/sierra-research/tau2-bench/blob/main/LICENSE

Share of banking questions answered from a policy knowledge base tasks completed within policy, averaged over trials.

Last updated 3 Aug 2026

Results dated: 5 May 2026 to 3 Aug 2026
Models: 21
Unit: % of tasks
Licence: MIT License

Task success (pass^1): Claude Opus 4.8

Top 15 of 21 results · % of tasks, higher is better. Choose a model to highlight it.Clear highlight

1 Qwen3.8-Max (0902, extra-high reasoning)Alibaba 55.2%
2 Claude Opus 5 (max reasoning)Anthropic 48.7%
3 Grok 4.5 (high reasoning)xAI 47.9%
4 GPT-5.6 Sol (extra-high reasoning)OpenAI 46.9%
5 GPT-5.5 (extra-high reasoning)OpenAI 44.6%
6 Muse Spark 1.1 (extra-high reasoning)Meta 40.5%
7 Claude Opus 4.7 (max reasoning)Anthropic 40.2%
8 Claude Fable 5 (max reasoning)Anthropic 39.7%
8 Claude Opus 4.8 (max reasoning)Anthropic 39.7%
10 GPT-5.4 (extra-high reasoning)OpenAI 39.4%
11 Kimi K3 (max reasoning)Moonshot AI 37.1%
11 GLM-5.2 (extra-high reasoning)Z.ai 37.1%
13 GPT-5.2 (high reasoning)OpenAI 32.2%
14 Claude Opus 4.6 (max reasoning)Anthropic 27.3%
15 Gemini 3.1 Pro Preview (high reasoning)Google 26.0%

Full results

tau2 v1.0.1, banking knowledge, alltools retrieval: task success (pass^1), % of tasks, higher is better
#	Model	Task success (pass^1) % of tasks, higher is better	Price $ per million tokens, in / out
1	Qwen3.8-Max (0902, extra-high reasoning)Alibaba	55.2%	$2 / $6
2	Claude Opus 5 (max reasoning)Anthropic	48.7%	$5 / $25
3	Grok 4.5 (high reasoning)xAI	47.9%	$2 / $6
4	GPT-5.6 Sol (extra-high reasoning)OpenAI	46.9%	$4 / $20
5	GPT-5.5 (extra-high reasoning)OpenAI	44.6%	$5 / $30
6	Muse Spark 1.1 (extra-high reasoning)Meta	40.5%	$1.25 / $4.25
7	Claude Opus 4.7 (max reasoning)Anthropic	40.2%	$5 / $25
8	Claude Fable 5 (max reasoning)Anthropic	39.7%	$10 / $50
8	Claude Opus 4.8 (max reasoning)Anthropic	39.7%	$5 / $25
10	GPT-5.4 (extra-high reasoning)OpenAI	39.4%	$2.50 / $15
11	Kimi K3 (max reasoning)Moonshot AI	37.1%	$3 / $15
11	GLM-5.2 (extra-high reasoning)Z.ai	37.1%	$1.40 / $4.40
13	GPT-5.2 (high reasoning)OpenAI	32.2%	$1.75 / $14
14	Claude Opus 4.6 (max reasoning)Anthropic	27.3%	$5 / $25
15	Gemini 3.1 Pro Preview (high reasoning)Google	26.0%	$2 / $12
16	Inkling (max reasoning)Thinking Machines	25.0%	$0.95 / $4.05
17	Claude Opus 4.5 (high reasoning)Anthropic	24.7%	$5 / $25
18	Grok 4.20 (high reasoning)xAI	18.0%	$1.25 / $2.50
19	Grok 4 Fast (high reasoning)xAI	15.7%	–
20	Gemini 2.5 Pro (high reasoning)Google	13.7%	$1.25 / $10
21	Grok 4.1 Fast (high reasoning)xAI	13.1%	–

Swipe the table sideways for more columns.

Ranks follow the score as shown, so equal numbers share a rank. Results as published by tau2-bench; we do not re-run them.

Progress over time

Task success (pass^1), plotted at each model's release date, higher is better. Each model counts once, at its best setting. Scores are today's, not launch day's; 1 model without a known release date is left off. Hover or tap a mark for its model.

Best score so far

Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.

Best score by lab

Each lab's best so far, stepping whenever one of its releases beats its own earlier models. The 6 labs with the best scores today; 3 others not drawn.

AlibabaAnthropicxAIOpenAIMetaZ.ai

What it measures

Share of banking questions answered from a policy knowledge base tasks completed within policy, averaged over trials.

What it does not measure

Not your policies or customers: the tasks and policies are tau2's own, and the customer is simulated by gpt-5.2.

Source

tau2-bench · Methodology

2 results from tau2-bench not ranked here · show why

We rank a result only when we can tie it to a specific model you can use. These are left out:

A custom agent harness, not the standard one: 2

tau2-bench by Sierra Research and contributors, MIT License.