Benchmarks / Vectara Hallucination Leaderboard

Reported by Vectara Hallucination Leaderboard

Vectara Hallucination Leaderboard

Share of documents the model agreed to summarise rather than refusing.

Last updated 22 Sep 2026

Results dated
22 Sep 2026
Results
108 configurations of 104 models
Unit
% of documents
Licence
Apache License 2.0

Answer rate: Phi-4

Top 15 of 108 results · % of documents, higher is better · ≈ cannot be told apart from the leader. Choose a model to highlight it.Clear highlight

  1. 1≈ Nova Micro 1.0Amazon 100.0%
  2. 1≈ Gemma 4 31BGoogle 100.0%
  3. 1≈ Granite 3.3 8B InstructIBM 100.0%
  4. 1≈ Granite 4.0 H SmallIBM 100.0%
  5. 1≈ Mercury 2Inception 100.0%
  6. 1≈ Llama 4 Maverick 17B 128E Instruct (FP8)Meta 100.0%
  7. 1≈ GPT-5.1 (high reasoning)OpenAI 100.0%
  8. 1≈ GPT-5.1 (low reasoning)OpenAI 100.0%
  9. 1≈ GPT-5.2 (high reasoning)OpenAI 100.0%
  10. 1≈ GPT-5.2 (low reasoning)OpenAI 100.0%
  11. 1≈ GPT-5.4 miniOpenAI 100.0%
  12. 1≈ GPT-5.4 nanoOpenAI 100.0%
  13. 1≈ GPT-5.4 ProOpenAI 100.0%
  14. 1≈ GPT-5.5OpenAI 100.0%
  15. 1≈ GPT-5 nanoOpenAI 100.0%
  16. 104 Phi-4Microsoft 80.7%

Full results

Vectara: answer rate, % of documents, higher is better
#ModelAnswer rate
% of documents, higher is better		Price
$ per million tokens, in / out
1≈ Nova Micro 1.0Amazon
100.0%
$0.035 / $0.14
1≈ Gemma 4 31BGoogle
100.0%
$0.14 / $0.40
1≈ Granite 3.3 8B InstructIBM
100.0%
–
1≈ Granite 4.0 H SmallIBM
100.0%
–
1≈ Mercury 2Inception
100.0%
$0.25 / $0.75
1≈ Llama 4 Maverick 17B 128E Instruct (FP8)Meta
100.0%
–
1≈ GPT-5.1 (high reasoning)OpenAI
100.0%
$1.25 / $10
1≈ GPT-5.1 (low reasoning)OpenAI
100.0%
$1.25 / $10
1≈ GPT-5.2 (high reasoning)OpenAI
100.0%
$1.75 / $14
1≈ GPT-5.2 (low reasoning)OpenAI
100.0%
$1.75 / $14
1≈ GPT-5.4 miniOpenAI
100.0%
$0.75 / $4.50
1≈ GPT-5.4 nanoOpenAI
100.0%
$0.20 / $1.25
1≈ GPT-5.4 ProOpenAI
100.0%
$30 / $180
1≈ GPT-5.5OpenAI
100.0%
$5 / $30
1≈ GPT-5 nanoOpenAI
100.0%
$0.05 / $0.40
1≈ GPT-6 AstraOpenAI
100.0%
$10 / $50
1≈ GPT-6 SolOpenAI
100.0%
$2 / $10
1≈ o3-proOpenAI
100.0%
$20 / $80
19 Qwen3-14BAlibaba
99.9%
$0.12 / $0.24
19 Qwen3-32BAlibaba
99.9%
$0.14 / $0.40
19 Qwen3-4BAlibaba
99.9%
–
19 Qwen3-8BAlibaba
99.9%
$0.12 / $0.46
19 Nova Lite 1.0Amazon
99.9%
$0.06 / $0.24
19 Claude Sonnet 4.6Anthropic
99.9%
$3 / $15
19 Ministral 3B (2410)Mistral AI
99.9%
–
19 Ministral 8B (2410)Mistral AI
99.9%
–
19 Mistral Large 2.1 (2411)Mistral AI
99.9%
–
19 GPT-4.1OpenAI
99.9%
$2 / $8
19 GPT-5.4OpenAI
99.9%
$2.50 / $15
19 GPT-5 miniOpenAI
99.9%
$0.25 / $2
19 GPT-5 (high reasoning)OpenAI
99.9%
$1.25 / $10
19 GPT-5 (minimal reasoning)OpenAI
99.9%
$1.25 / $10
19 gpt-oss-120bOpenAI
99.9%
$0.15 / $0.60
34 Qwen3.5-122B-A10BAlibaba
99.8%
$0.26 / $2.08
34 Qwen3.5-27BAlibaba
99.8%
$0.27 / $2.16
34 Qwen3.5-35B-A3BAlibaba
99.8%
$0.16 / $1.30
34 Qwen3.5-FlashAlibaba
99.8%
$0.065 / $0.26
34 Qwen3.5-Plus (2026-02-15)Alibaba
99.8%
$0.26 / $1.56
34 Claude Opus 4.6Anthropic
99.8%
$5 / $25
34 Aya Expanse 32BCohere
99.8%
–
34 Gemini 3 Flash PreviewGoogle
99.8%
$0.50 / $3
34 Gemma 4 26B A4BGoogle
99.8%
$0.10 / $0.30
34 GLM-4.7Z.ai
99.8%
$0.54 / $1.98
44 Mistral Medium 3.1Mistral AI
99.7%
$0.40 / $2
44 Kimi K2.6Moonshot AI
99.7%
$0.95 / $4
44 Grok 4.1 Fast (reasoning)xAI
99.7%
–
44 GLM-5Z.ai
99.7%
$0.95 / $2.55
48 Jamba Mini 2AI21 Labs
99.6%
–
48 Nova 2 LiteAmazon
99.6%
$0.30 / $2.50
48 Gemini 3.1 Flash-Lite PreviewGoogle
99.6%
$0.25 / $1.50
48 Ministral 3 14BMistral AI
99.6%
$0.20 / $0.20
48 Nemotron 3 Nano 30B A3BNVIDIA
99.6%
$0.05 / $0.20
53 Finix-S1-32BAnt Group
99.5%
–
53 Claude Haiku 4.5Anthropic
99.5%
$1 / $5
53 Gemini 2.5 Flash-LiteGoogle
99.5%
$0.10 / $0.40
53 Llama 3.3 70B Instruct TurboMeta
99.5%
–
53 Grok 4 Fast (reasoning)xAI
99.5%
–
58 Gemini 3.1 Pro PreviewGoogle
99.4%
$2 / $12
58 Gemini 3 Pro PreviewGoogle
99.4%
–
58 MiniMax-M2.7MiniMax
99.4%
$0.30 / $1.20
61 Nova Pro 1.0Amazon
99.3%
$0.80 / $3.20
62 o4-mini (high reasoning)OpenAI
99.2%
$1.10 / $4.40
62 Grok 4 Fast (non-reasoning)xAI
99.2%
–
64 Jamba Mini 1.7AI21 Labs
99.1%
–
64 Gemini 2.5 ProGoogle
99.1%
$1.25 / $10
64 Ministral 3 8BMistral AI
99.1%
$0.15 / $0.15
64 GPT-5.6 SolOpenAI
99.1%
$4 / $20
68 Trinity Large PreviewArcee AI
99.0%
–
68 Gemini 2.5 FlashGoogle
99.0%
$0.30 / $2.50
68 Llama 4 ScoutMeta
99.0%
$0.18 / $0.59
71 Jamba Large 1.7AI21 Labs
98.9%
–
72 Gemma 3 27BGoogle
98.8%
$0.12 / $0.20
72 Mistral Large 3Mistral AI
98.8%
$0.50 / $1.50
74 Claude Opus 4.5Anthropic
98.7%
$5 / $25
74 o4-mini (low reasoning)OpenAI
98.7%
$1.10 / $4.40
76 Claude Sonnet 4Anthropic
98.6%
$3 / $15
76 Kimi K2 (0905)Moonshot AI
98.6%
$0.60 / $2.50
78 MiniMax-M2.1MiniMax
98.5%
$0.30 / $1.20
78 Grok 4.1 Fast (non-reasoning)xAI
98.5%
–
80 MiniMax-M2.5MiniMax
98.2%
$0.30 / $1.20
81 GLM-4.5-Air (FP8)Z.ai
98.1%
–
82 Claude Opus 4.7Anthropic
98.0%
$5 / $25
83 Mistral Small 3Mistral AI
97.9%
$0.05 / $0.08
84 Command ACohere
97.6%
$2.50 / $10
85 DeepSeek-V3DeepSeek
97.5%
$0.26 / $1.03
86 Gemma 3 12BGoogle
97.4%
$0.05 / $0.15
87 DeepSeek-V4-Pro (0423)DeepSeek
97.2%
$1.42 / $2.83
88 DeepSeek-R1DeepSeek
97.0%
$0.70 / $2.50
89 DeepSeek-V3.2-ExpDeepSeek
96.6%
$0.27 / $0.41
90 Claude Sonnet 4.5Anthropic
95.6%
$3 / $15
91 Command R+ (08-2024)Cohere
95.0%
$2.50 / $10
92 Qwen3-235B-A22BAlibaba
94.9%
$0.46 / $1.82
93 DeepSeek-V3.1DeepSeek
94.5%
$0.55 / $1.65
93 GLM-4.6Z.ai
94.5%
$0.50 / $2
95 Qwen3-Next-80B-A3B-ThinkingAlibaba
94.4%
$0.15 / $1.20
96 GPT-4o (2024-08-06)OpenAI
93.8%
$2.50 / $10
97 Grok 3xAI
93.0%
–
98 DeepSeek-V3.2DeepSeek
92.6%
$0.30 / $0.96
99 Phi-4-mini-instructMicrosoft
92.5%
–
100 Claude Opus 4.1Anthropic
92.4%
$15 / $75
101 Kimi K2.5Moonshot AI
92.2%
$0.57 / $2.85
102 GLM-4.7-FlashZ.ai
91.6%
$0.06 / $0.40
103 Claude Opus 4Anthropic
91.0%
–
104 Phi-4Microsoft
80.7%
$0.07 / $0.14
105 Aya Expanse 8BCohere
77.5%
–
106 Ministral 3 3BMistral AI
74.3%
$0.10 / $0.10
107 Gemma 3 4BGoogle
67.3%
$0.05 / $0.10
108 Snowflake Arctic InstructSnowflake
62.7%
–

Swipe the table sideways for more columns.

Ranks follow the score as shown, so equal numbers share a rank. Results as published by Vectara Hallucination Leaderboard; we do not re-run them.

Progress over time

Answer rate, plotted at each model's release date, higher is better. Each model counts once, at its best setting. Scores are today's, not launch day's; 17 models without a known release date are left off. Hover or tap a mark for its model.

Best score so far

Every model at its release date. The line moves each time a new release beats every earlier one, and each new best is named where there is room.

60%70%80%90%100%110% 2025Jul2026Jul GPT-4o (2024-08-06)Ministral 3B (2410)Nova Micro 1.0

Best score by lab

Each lab's best so far, stepping whenever one of its releases beats its own earlier models. The 6 labs with the best scores today; 10 others not drawn.

OpenAIAmazonGoogleInceptionMistral AIAlibaba

92%94%96%98%100%102% 2025Jul2026Jul OpenAIAmazonGoogleInceptionMistral AIAlibaba

What it measures

Share of documents the model agreed to summarise rather than refusing.

What it does not measure

Not a quality score: a low rate usually means content filters were triggered, and hallucination rates are measured on answered documents only.

Source

Vectara Hallucination Leaderboard · Methodology

Vectara Hallucination Leaderboard by Vectara, Apache License 2.0.