Reported by tau2-bench

tau2-bench

Distyl ButtonAgent: custom agent scaffold, not the standard harness
RAFT-30B-A3B: custom agent scaffold, not the standard harness

Share of banking questions answered from a policy knowledge base tasks completed within policy, averaged over trials.

tau2 v1.0.1, banking knowledge, alltools retrieval: task success (pass^1), % of tasks, higher is better
#	Model	tau2 v1.0.1, banking knowledge, alltools retrieval: task success (pass^1) % of tasks, higher is better
1	Qwen3.8 Max (0902)Alibaba · qwen3.8-max-0902	55.2%
2	Claude Opus 5Anthropic · claude-opus-5	48.7%
3	Grok 4.5xAI · grok-4.5	47.9%
4	GPT-5.6 SolOpenAI · gpt-5.6-sol	46.9%
5	GPT-5.5OpenAI · gpt-5.5	44.6%
6	Muse Spark 1.1Meta · muse-spark-1.1	40.5%
7	Claude Opus 4.7Anthropic · claude-opus-4.7	40.2%
8	Claude Fable 5Anthropic · claude-fable-5	39.7%
8	Claude Opus 4.8Anthropic · claude-opus-4.8	39.7%
10	GPT-5.4OpenAI · gpt-5.4	39.4%
11	Kimi K3Moonshot AI · kimi-k3	37.1%
11	GLM 5.2Z.ai · glm-5.2	37.1%
13	GPT-5.2OpenAI · gpt-5.2	32.2%
14	Claude Opus 4.6Anthropic · claude-opus-4.6	27.3%
15	Gemini 3.1 Pro PreviewGoogle · gemini-3.1-pro-preview	26.0%
16	InklingThinkingmachines · inkling	25.0%
17	Claude Opus 4.5Anthropic · claude-opus-4.5	24.7%
18	Grok 4.20xAI · grok-4.20	18.0%
19	grok-4-fastxAI	15.7%
20	Gemini 2.5 ProGoogle · gemini-2.5-pro	13.7%
21	grok-4-1-fastxAI	13.1%

Results as published by tau2-bench; we do not re-run them.

What it measures

Share of banking questions answered from a policy knowledge base tasks completed within policy, averaged over trials.

Not your policies or customers: the tasks and policies are tau2's own, and the customer is simulated by gpt-5.2.

Failures count against a model: a product with no usable output is a failed listing. They are listed here so you can see why.