Benchmarks / Vectara Hallucination Leaderboard

Reported by Vectara Hallucination Leaderboard

Vectara Hallucination Leaderboard

Share of document summaries that contain something the document does not support.

Last updated 22 Sep 2026

Results dated
22 Sep 2026
Results
108 configurations of 104 models
Unit
% of summaries
Licence
Apache License 2.0

Hallucination rate: Finix-S1-32B

Top 15 of 108 results · % of summaries, lower is better. Choose a model to highlight it.Clear highlight

  1. 1 Finix-S1-32BAnt Group · answers 99.5% 1.8%
  2. 2 GPT-5.4 nanoOpenAI · answers 100.0% 3.1%
  3. 3 Gemini 2.5 Flash-LiteGoogle · answers 99.5% 3.3%
  4. 4 Phi-4Microsoft · answers 80.7% 3.7%
  5. 5 Llama 3.3 70B Instruct TurboMeta · answers 99.5% 4.1%
  6. 6 Snowflake Arctic InstructSnowflake · answers 62.7% 4.3%
  7. 7 Gemma 3 12BGoogle · answers 97.4% 4.4%
  8. 8 Mistral Large 2.1 (2411)Mistral AI · answers 99.9% 4.5%
  9. 9 Qwen3-8BAlibaba · answers 99.9% 4.8%
  10. 10 Nova 2 LiteAmazon · answers 99.6% 5.1%
  11. 10 Nova Pro 1.0Amazon · answers 99.3% 5.1%
  12. 10 Mistral Small 3Mistral AI · answers 97.9% 5.1%
  13. 13 Gemma 4 26B A4BGoogle · answers 99.8% 5.2%
  14. 13 Granite 4.0 H SmallIBM · answers 100.0% 5.2%
  15. 15 Jamba Mini 2AI21 Labs · answers 99.6% 5.3%

Full results

Vectara: hallucination rate, % of summaries, lower is better
#ModelHallucination rate
% of summaries, lower is better		Answer rate
% of documents		Price
$ per million tokens, in / out
1 Finix-S1-32BAnt Group
1.8%
99.5% –
2 GPT-5.4 nanoOpenAI
3.1%
100.0% $0.20 / $1.25
3 Gemini 2.5 Flash-LiteGoogle
3.3%
99.5% $0.10 / $0.40
4 Phi-4Microsoft
3.7%
80.7% $0.07 / $0.14
5 Llama 3.3 70B Instruct TurboMeta
4.1%
99.5% –
6 Snowflake Arctic InstructSnowflake
4.3%
62.7% –
7 Gemma 3 12BGoogle
4.4%
97.4% $0.05 / $0.15
8 Mistral Large 2.1 (2411)Mistral AI
4.5%
99.9% –
9 Qwen3-8BAlibaba
4.8%
99.9% $0.12 / $0.46
10 Nova 2 LiteAmazon
5.1%
99.6% $0.30 / $2.50
10 Nova Pro 1.0Amazon
5.1%
99.3% $0.80 / $3.20
10 Mistral Small 3Mistral AI
5.1%
97.9% $0.05 / $0.08
13 Gemma 4 26B A4BGoogle
5.2%
99.8% $0.10 / $0.30
13 Granite 4.0 H SmallIBM
5.2%
100.0% –
15 Jamba Mini 2AI21 Labs
5.3%
99.6% –
15 DeepSeek-V3.2-ExpDeepSeek
5.3%
96.6% $0.27 / $0.41
17 Qwen3-14BAlibaba
5.4%
99.9% $0.12 / $0.24
18 Nova Micro 1.0Amazon
5.5%
100.0% $0.035 / $0.14
18 DeepSeek-V3.1DeepSeek
5.5%
94.5% $0.55 / $1.65
18 GPT-5.4 miniOpenAI
5.5%
100.0% $0.75 / $4.50
21 GPT-4.1OpenAI
5.6%
99.9% $2 / $8
22 Qwen3-4BAlibaba
5.7%
99.9% –
23 Grok 3xAI
5.8%
93.0% –
24 Qwen3-32BAlibaba
5.9%
99.9% $0.14 / $0.40
25 Nova Lite 1.0Amazon
6.1%
99.9% $0.06 / $0.24
25 DeepSeek-V3DeepSeek
6.1%
97.5% $0.26 / $1.03
27 DeepSeek-V3.2DeepSeek
6.3%
92.6% $0.30 / $0.96
28 Gemma 3 4BGoogle
6.4%
67.3% $0.05 / $0.10
29 GPT-6 SolOpenAI
6.5%
100.0% $2 / $10
30 Trinity Large PreviewArcee AI
6.9%
99.0% –
30 Command R+ (08-2024)Cohere
6.9%
95.0% $2.50 / $10
32 Gemini 2.5 ProGoogle
7.0%
99.1% $1.25 / $10
32 GPT-5.4OpenAI
7.0%
99.9% $2.50 / $15
34 Ministral 3B (2410)Mistral AI
7.3%
99.9% –
35 Gemma 3 27BGoogle
7.4%
98.8% $0.12 / $0.20
35 Gemma 4 31BGoogle
7.4%
100.0% $0.14 / $0.40
35 Ministral 8B (2410)Mistral AI
7.4%
99.9% –
38 Llama 4 ScoutMeta
7.7%
99.0% $0.18 / $0.59
39 Gemini 2.5 FlashGoogle
7.8%
99.0% $0.30 / $2.50
40 Gemini 3.1 Flash-Lite PreviewGoogle
8.2%
99.6% $0.25 / $1.50
40 Llama 4 Maverick 17B 128E Instruct (FP8)Meta
8.2%
100.0% –
42 GPT-5.4 ProOpenAI
8.3%
100.0% $30 / $180
43 GPT-5.2 (low reasoning)OpenAI
8.4%
100.0% $1.75 / $14
44 DeepSeek-V4-Pro (0423)DeepSeek
8.6%
97.2% $1.42 / $2.83
45 GPT-6 AstraOpenAI
8.7%
100.0% $10 / $50
46 MiniMax-M2.5MiniMax
9.1%
98.2% $0.30 / $1.20
47 Qwen3-235B-A22BAlibaba
9.3%
94.9% $0.46 / $1.82
47 Qwen3-Next-80B-A3B-ThinkingAlibaba
9.3%
94.4% $0.15 / $1.20
47 Command ACohere
9.3%
97.6% $2.50 / $10
47 GPT-5.5OpenAI
9.3%
100.0% $5 / $30
47 GLM-4.5-Air (FP8)Z.ai
9.3%
98.1% –
47 GLM-4.7-FlashZ.ai
9.3%
91.6% $0.06 / $0.40
53 Aya Expanse 8BCohere
9.5%
77.5% –
53 GLM-4.6Z.ai
9.5%
94.5% $0.50 / $2
55 Nemotron 3 Nano 30B A3BNVIDIA
9.6%
99.6% $0.05 / $0.20
55 GPT-4o (2024-08-06)OpenAI
9.6%
93.8% $2.50 / $10
57 Jamba Large 1.7AI21 Labs
9.7%
98.9% –
58 Claude Haiku 4.5Anthropic
9.8%
99.5% $1 / $5
59 GLM-5Z.ai
10.1%
99.7% $0.95 / $2.55
60 Claude Sonnet 4Anthropic
10.3%
98.6% $3 / $15
61 Gemini 3.1 Pro PreviewGoogle
10.4%
99.4% $2 / $12
62 Qwen3.5-35B-A3BAlibaba
10.5%
99.8% $0.16 / $1.30
62 Qwen3.5-FlashAlibaba
10.5%
99.8% $0.065 / $0.26
62 GPT-5 nanoOpenAI
10.5%
100.0% $0.05 / $0.40
65 Claude Sonnet 4.6Anthropic
10.6%
99.9% $3 / $15
65 Granite 3.3 8B InstructIBM
10.6%
100.0% –
67 Qwen3.5-Plus (2026-02-15)Alibaba
10.7%
99.8% $0.26 / $1.56
68 Kimi K2.6Moonshot AI
10.8%
99.7% $0.95 / $4
68 GPT-5.2 (high reasoning)OpenAI
10.8%
100.0% $1.75 / $14
70 Claude Opus 4.5Anthropic
10.9%
98.7% $5 / $25
70 Aya Expanse 32BCohere
10.9%
99.8% –
70 GPT-5.1 (low reasoning)OpenAI
10.9%
100.0% $1.25 / $10
73 Qwen3.5-122B-A10BAlibaba
11.2%
99.8% $0.26 / $2.08
74 DeepSeek-R1DeepSeek
11.3%
97.0% $0.70 / $2.50
75 GLM-4.7Z.ai
11.7%
99.8% $0.54 / $1.98
76 Claude Opus 4.1Anthropic
11.8%
92.4% $15 / $75
76 MiniMax-M2.1MiniMax
11.8%
98.5% $0.30 / $1.20
78 Claude Opus 4.7Anthropic
12.0%
98.0% $5 / $25
78 Claude Opus 4Anthropic
12.0%
91.0% –
78 Claude Sonnet 4.5Anthropic
12.0%
95.6% $3 / $15
81 Qwen3.5-27BAlibaba
12.1%
99.8% $0.27 / $2.16
81 GPT-5.1 (high reasoning)OpenAI
12.1%
100.0% $1.25 / $10
83 Claude Opus 4.6Anthropic
12.2%
99.8% $5 / $25
84 Mercury 2Inception
12.3%
100.0% $0.25 / $0.75
85 GPT-5.6 SolOpenAI
12.4%
99.1% $4 / $20
86 MiniMax-M2.7MiniMax
12.9%
99.4% $0.30 / $1.20
86 GPT-5 miniOpenAI
12.9%
99.9% $0.25 / $2
88 Gemini 3 Flash PreviewGoogle
13.5%
99.8% $0.50 / $3
89 Gemini 3 Pro PreviewGoogle
13.6%
99.4% –
90 Kimi K2.5Moonshot AI
14.2%
92.2% $0.57 / $2.85
90 gpt-oss-120bOpenAI
14.2%
99.9% $0.15 / $0.60
92 Mistral Large 3Mistral AI
14.5%
98.8% $0.50 / $1.50
93 Jamba Mini 1.7AI21 Labs
14.7%
99.1% –
93 GPT-5 (minimal reasoning)OpenAI
14.7%
99.9% $1.25 / $10
95 GPT-5 (high reasoning)OpenAI
15.1%
99.9% $1.25 / $10
96 Grok 4.1 Fast (non-reasoning)xAI
17.8%
98.5% –
97 Kimi K2 (0905)Moonshot AI
17.9%
98.6% $0.60 / $2.50
98 o4-mini (high reasoning)OpenAI
18.6%
98.7% $1.10 / $4.40
98 o4-mini (low reasoning)OpenAI
18.6%
98.7% $1.10 / $4.40
100 Grok 4.1 Fast (reasoning)xAI
19.2%
99.7% –
101 Ministral 3 14BMistral AI
19.4%
99.6% $0.20 / $0.20
102 Grok 4 Fast (non-reasoning)xAI
19.7%
99.2% –
103 Grok 4 Fast (reasoning)xAI
20.2%
99.5% –
104 Ministral 3 8BMistral AI
21.7%
99.1% $0.15 / $0.15
105 Mistral Medium 3.1Mistral AI
22.7%
99.7% $0.40 / $2
106 o3-proOpenAI
23.3%
100.0% $20 / $80
107 Phi-4-mini-instructMicrosoft
23.5%
92.5% –
108 Ministral 3 3BMistral AI
24.2%
74.3% $0.10 / $0.10

Swipe the table sideways for more columns.

Ranks follow the score as shown, so equal numbers share a rank. Results as published by Vectara Hallucination Leaderboard; we do not re-run them.

Progress over time

Hallucination rate, plotted at each model's release date, lower is better. Each model counts once, at its best setting. Scores are today's, not launch day's; 17 models without a known release date are left off. Hover or tap a mark for its model.

Best score so far

Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.

0%10%20%30% 2025Jul2026Jul GPT-4o (2024-08-06)Ministral 3B (2410)Mistral Large 2.1 (2411)Phi-4Gemini 2.5 Flash-LiteGPT-5.4 nano

Best score by lab

Each lab's best so far, stepping whenever one of its releases beats its own earlier models. The 6 labs with the best scores today; 10 others not drawn.

OpenAIGoogleMicrosoftMistral AIAlibabaAmazon

2%4%6%8%10% 2025Jul2026Jul OpenAIGoogleMicrosoftMistral AIAlibabaAmazon

What it measures

Share of document summaries that contain something the document does not support.

What it does not measure

Not errors in open questions or other tasks: only summarisation, judged by Vectara's own model (HHEM-2.3), not by people, on news-style documents rather than your data.

Source

Vectara Hallucination Leaderboard · Methodology

Vectara Hallucination Leaderboard by Vectara, Apache License 2.0.