Benchmarks / CatalogBench
Measured by Spring Prompt
CatalogBench
Which models can turn a sparse product feed, product photos and supplier copy into a listing that could go live, without inventing anything?
- Results dated
- 30 Sep 2026
- Models
- 18
- Unit
- claims per product
- Licence
- Spring Prompt original
- Judge
- openai/gpt-6.1-sol
- Runs
- 3 per model
| # | Model | CatalogBench (sales brief): unsupported claims claims per product, lower is better | Reliably publish-ready % of products | Publish-ready listings % of products | Field accuracy % of missing fields | Reliably publish-ready % of products |
|---|---|---|---|---|---|---|
| 1 | GPT-6.1 SolOpenAI |
0.15
|
69.6% | 74.4% | 94.4% | 66.1% |
| 2 | GPT-6 AstraOpenAI |
0.18
|
71.4% | 74.4% | 94.2% | 66.1% |
| 3 | GPT-6 SolOpenAI |
0.36
|
60.7% | 70.2% | 93.7% | 55.4% |
| 4 | GPT-6 LunaOpenAI |
0.56
|
51.8% | 64.3% | 93.7% | 42.9% |
| 5 | Grok 4.7xAI |
1.58
|
44.6% | 62.5% | 92.9% | 10.7% |
| 6 | GLM 5V TurboZ.aiFailed outputs |
2.58
|
0.0% | 3.0% | 59.6% | 0.0% |
| 7 | DeepSeek V4.1 FlashDeepSeekFailed outputs |
2.79
|
35.7% | 58.3% | 94.4% | 10.7% |
| 8 | Muse Spark 1.3Meta |
3.44
|
37.5% | 57.7% | 92.9% | 7.1% |
| 9 | Claude Sonnet 5.5Anthropic |
3.97
|
21.4% | 35.1% | 91.5% | 1.8% |
| 10 | Claude Opus 5.5Anthropic |
4.36
|
39.3% | 58.3% | 92.2% | 0.0% |
| 11 | Gemini 3.8 FlashGoogle |
7.12
|
16.1% | 33.3% | 92.7% | 1.8% |
| 12 | Kimi K3Moonshot AIFailed outputs |
7.18
|
17.9% | 47.6% | 94.4% | 0.0% |
| 13 | Gemini 3.1 Pro PreviewGoogle |
7.21
|
25.0% | 42.3% | 95.4% | 0.0% |
| 14 | Claude Fable 5.1Anthropic |
7.91
|
16.1% | 35.1% | 94.7% | 0.0% |
| 15 | Gemini 3.5 Flash LiteGoogle |
8.83
|
3.6% | 15.5% | 92.5% | 0.0% |
| 16 | Qwen3.8 Max (0902)AlibabaFailed outputs |
9.09
|
10.7% | 28.0% | 90.3% | 0.0% |
| 17 | Mistral Medium 3.5Mistral |
9.12
|
5.4% | 15.5% | 91.7% | 0.0% |
| 18 | Claude Haiku 4.5Anthropic |
14.3
|
0.0% | 0.6% | 93.7% | 0.0% |
Each model runs at its provider's default reasoning setting. Some providers think at length by default and others barely at all, so this is what you get without tuning.
What it measures
- Attributes read from the images, not guessed
- Supplier claims checked, not repeated
- Required UK product information included
- Listings that shoppers can find in search
What it does not measure
- Conversion or sales impact
- Real product photography (a real-photo slice is planned)
- Writing style beyond the listed checks
Method
- Rule-based checks first; judged checks are yes or no
- The judge was checked for bias against Gemini and Claude judges
- Private products are held back so the set can be refreshed
Checking the judge
The judge is an OpenAI model, and OpenAI models lead this table, so we checked it for bias. Gemini 3.1 Pro and Claude Opus 5.5 judged the same outputs from five models on a calibration set. All three judges put the models in the same order under both briefs. Each was slightly gentler on its own family's marketing copy: the top GPT models moved by 4 to 8 points between judges under the marketing brief, without changing places.
Failures
Failures count against a model: a product with no usable output is a failed listing. They are listed here so you can see why.
- GLM 5V Turbo: invalid JSON (raw line breaks inside text): 138 of 168 attempts.
- DeepSeek V4.1 Flash: invalid JSON: 1 of 168 attempts.
- Kimi K3: invalid JSON: 1 of 168 attempts.
- Qwen3.8 Max (0902): reply cut off at the token limit: 6 of 168 attempts.
Run this on your catalogue. The same checks, on a sample of your own products.Catalogue feed diagnostic →