Benchmarks / CatalogBench

Measured by Spring Prompt

CatalogBench

Which models can turn a sparse product feed, product photos and supplier copy into a listing that could go live, without inventing anything?

Results dated
30 Sep 2026
Models
18
Unit
% of products
Licence
Spring Prompt original
Judge
openai/gpt-6.1-sol
Runs
3 per model

Claims to check:

Top 15 of 18 results · % of products, lower is better. Choose a model to highlight it.Clear highlight

  1. 1 Claude Fable 5.1Anthropic 0.0%
  2. 1 Claude Opus 5.5Anthropic 0.0%
  3. 1 DeepSeek V4.1 FlashDeepSeek 0.0%
  4. 1 Muse Spark 1.3Meta 0.0%
  5. 1 Kimi K3Moonshot AI 0.0%
  6. 1 GPT-6.1 SolOpenAI 0.0%
  7. 1 GPT-6 AstraOpenAI 0.0%
  8. 1 GPT-6 LunaOpenAI 0.0%
  9. 1 GPT-6 SolOpenAI 0.0%
  10. 1 Grok 4.7xAI 0.0%
  11. 11 Claude Sonnet 5.5Anthropic 0.6%
  12. 12 Qwen3.8 Max (0902)Alibaba 1.2%
  13. 13 Gemini 3.8 FlashGoogle 5.4%
  14. 14 Gemini 3.5 Flash LiteGoogle 7.7%
  15. 15 Gemini 3.1 Pro PreviewGoogle 10.1%

Ask for copy that sells, and accuracy drops

Each model's share of products that were publish-ready in all three runs, under the factual brief and under the sales brief.

Factual briefSales brief
GPT-6.1 Sol
75.0% · 67.9%
GPT-6 Astra
75.0% · 69.6%
GPT-6 Sol
66.1% · 67.9%
Grok 4.7
57.1% · 33.9%
Claude Opus 5.5
55.4% · 21.4%
GPT-6 Luna
55.4% · 51.8%
Muse Spark 1.3
48.2% · 10.7%
Gemini 3.1 Pro Preview
44.6% · 16.1%
DeepSeek V4.1 Flash
41.1% · 23.2%
Kimi K3
41.1% · 1.8%
Gemini 3.8 Flash
35.7% · 3.6%
Claude Sonnet 5.5
32.1% · 10.7%
Qwen3.8 Max (0902)
28.6% · 0.0%
Claude Fable 5.1
26.8% · 5.4%
Gemini 3.5 Flash Lite
17.9% · 3.6%
Mistral Medium 3.5
17.9% · 10.7%
Claude Haiku 4.5
1.8% · 0.0%
GLM 5V Turbo
1.8% · 0.0%

Full results

CatalogBench (sales brief): claims to check, % of products, lower is better
#ModelCatalogBench (sales brief): claims to check
% of products, lower is better		Publish-ready, factual brief
% of products		Publish-ready, sales brief
% of products		Cost per 10,000 products
US dollars		Unsupported claims
% of products		Claims to check
% of products		Failed outputs
% of products
1 Claude Fable 5.1Anthropic
0.0%
26.8%5.4%$991.2848.8%0.6%0.0%
1 Claude Opus 5.5Anthropic
0.0%
55.4%21.4%$400.0617.3%0.0%0.0%
1 DeepSeek V4.1 FlashDeepSeekFailed outputs
0.0%
41.1%23.2%$60.7013.1%0.0%0.0%
1 Muse Spark 1.3Meta
0.0%
48.2%10.7%$164.1814.9%0.0%0.0%
1 Kimi K3Moonshot AIFailed outputs
0.0%
41.1%1.8%$468.9516.7%0.0%0.0%
1 GPT-6.1 SolOpenAI
0.0%
75.0%67.9%$83.840.0%0.0%0.0%
1 GPT-6 AstraOpenAI
0.0%
75.0%69.6%$371.201.2%0.0%0.0%
1 GPT-6 LunaOpenAI
0.0%
55.4%51.8%$4.778.9%0.0%0.0%
1 GPT-6 SolOpenAI
0.0%
66.1%67.9%$85.181.8%0.0%0.0%
1 Grok 4.7xAI
0.0%
57.1%33.9%$237.1911.3%0.0%0.0%
11 Claude Sonnet 5.5Anthropic
0.6%
32.1%10.7%$171.9139.3%0.0%0.0%
12 Qwen3.8 Max (0902)AlibabaFailed outputs
1.2%
28.6%0.0%$255.7226.8%0.0%3.6%
13 Gemini 3.8 FlashGoogle
5.4%
35.7%3.6%$102.9231.6%0.0%0.0%
14 Gemini 3.5 Flash LiteGoogle
7.7%
17.9%3.6%$20.4733.3%1.8%0.0%
15 Gemini 3.1 Pro PreviewGoogle
10.1%
44.6%16.1%$354.6426.8%1.2%0.0%
16 Mistral Medium 3.5Mistral
25.6%
17.9%10.7%$80.0549.4%3.6%0.0%
17 GLM 5V TurboZ.aiFailed outputs
28.9%
1.8%0.0%$70.1155.4%11.5%32.1%
18 Claude Haiku 4.5Anthropic
44.6%
1.8%0.0%$61.3873.2%22.6%0.0%

Each model runs at its provider's default reasoning setting. Some providers think at length by default and others barely at all, so this is what you get without tuning.

Real outputs

The same product given to two models, from the published runs. A listing may use the supplier's copy unless the label says otherwise. Claims the law makes the seller prove, such as certifications and safety claims, are allowed but flagged to check.

Quibbly Lighthouse Jigsaw Puzzle, as the model saw it

Quibbly Lighthouse Jigsaw Puzzle

Toys · Sales brief

Supplier copy “Quibbly The Lighthouse, a 1000-piece jigsaw made in the UK from recycled board, with a free poster.”

GPT-6 Astra✓ Publish-ready

Quibbly The Lighthouse Jigsaw Puzzle, 500 Pieces, Ages 10+

  • 500-piece jigsaw featuring a lighthouse above a rocky coastline.
  • Golden sunset, rolling waves and a glowing lantern bring the coastal scene to life.
  • Made from recycled board.
  • Suitable for ages 10 and over.

Piece together a dramatic coastal sunset with Quibbly The Lighthouse, a 500-piece jigsaw puzzle for ages 10 and over. The artwork features a tall lighthouse with a glowing lantern, a small keeper’s house and waves breaking against a rocky shore. Orange and purple clouds sweep across the sky, while golden sunlight reflects on the sea.

Gemini 3.5 Flash Lite✗ Not publish-ready

Quibbly The Lighthouse Jigsaw Puzzle 1000 Pieces

  • Engaging 1000-piece jigsaw puzzle featuring a scenic coastal lighthouse at sunset
  • Crafted from durable recycled board for a sustainable puzzling experience
  • Stunning artwork captures crashing waves, rocky shores, and a vibrant evening sky
  • Designed for puzzle enthusiasts and recommended for ages 10 and up

Immerse yourself in a tranquil coastal escape with the Quibbly The Lighthouse Jigsaw Puzzle. This captivating 1000-piece challenge invites you to piece together a breathtaking seaside scene complete with a majestic lighthouse standing tall against a glowing sunset, dramatic clouds, and gentle ocean waves crashing onto rocky shores.

Why it is not publish-ready
  • Quibbly The Lighthouse Jigsaw Puzzle 1000 PiecesContradicted by the pack. The supplier says 1000, but the lid is printed “500 pieces”.
and 8 more issues found by the judge
Voltwren 20,000 mAh Power Bank, as the model saw it

Voltwren 20,000 mAh Power Bank

Electronics · Factual brief

Supplier copy “Voltwren 20,000 mAh power bank with 65W fast charging for laptops, phones and tablets. Airline approved. USB-C and USB-A ports.”

GPT-6 Astra✓ Publish-ready

Voltwren 20,000 mAh Power Bank, Blue-Grey, USB-C and USB-A

  • 20,000 mAh capacity, with the capacity printed on the casing.
  • One USB-C port and one USB-A port.
  • Blue-grey casing with a smooth finish.
  • Rounded rectangular body with labelled ports on one end.

The Voltwren power bank has a 20,000 mAh capacity and a blue-grey casing with rounded corners and a smooth finish. One USB-C port and one USB-A port sit side by side on the end panel, with their connector types labelled underneath. The broad upper surface carries the Voltwren name and capacity marking.

Gemini 3.1 Pro Preview✓ Publish-ready, after checks

Voltwren 20,000 mAh Power Bank, 65W, USB-C & USB-A, Blue

  • 20,000 mAh battery capacity
  • 65W fast charging supported
  • Equipped with one USB-C and one USB-A port
  • Suitable for charging laptops, phones, and tablets

Keep your devices powered on the go with the Voltwren 20,000 mAh Power Bank. Designed for versatility and convenience, it supports 65W fast charging and is suitable for a range of devices including laptops, mobile phones, and tablets. The power bank is equipped with both a USB-C and a USB-A port, allowing you to charge multiple devices efficiently.

Claims to check before publishing
  • Airline approved for travel
A claim UK rules require the seller to prove. Fine to publish once checked.
Stackwhistle Wooden Stacking Blocks, as the model saw it

Stackwhistle Wooden Stacking Blocks

Toys · Sales brief

Supplier copy “Stackwhistle Stacking Blocks: 12 solid beech blocks with non-toxic water-based paint, FSC certified wood, Montessori approved. 12m+.”

GPT-6 Astra✓ Publish-ready

Stackwhistle Wooden Stacking Blocks, 12 Pieces, Multicoloured, 12m+

  • 12 beech wood blocks for stacking, arranging and building.
  • Bright multicoloured designs include red, blue, green, yellow, orange and purple.
  • Cube-shaped blocks with flat faces for simple stacking play.
  • Suitable for children aged 12 months and over.

Build colourful towers and explore different arrangements with Stackwhistle Wooden Stacking Blocks. This 12-piece set is made from beech wood, with bright colours including red, blue, green, yellow, orange and purple. The cube-shaped blocks have flat faces for stacking, lining up and creating simple constructions, giving children a straightforward way to explore hands-on building play.

Claude Haiku 4.5✗ Not publish-ready

Stackwhistle Wooden Stacking Blocks, 12 Pieces, Multicoloured

  • Set of 12 solid beech wood blocks in vibrant colours for creative stacking and building play
  • FSC-certified sustainable wood with non-toxic water-based paint finish
  • Develops fine motor skills, hand-eye coordination and spatial awareness
  • Suitable from 12 months old; safe for babies and toddlers

Stackwhistle Wooden Stacking Blocks offer hours of creative play and learning for babies and toddlers from 12 months old. This set contains 12 solid beech wood blocks, each finished with vibrant, non-toxic water-based paint in red, blue, yellow, green, orange and purple.

Why it is not publish-ready
  • Suitable from 12 months old; safe for babies and toddlersNot supported by anything. A safety claim that neither the supplier nor the box makes.
and 11 more issues found by the judge
Claims to check before publishing
  • FSC-certified sustainable wood with non-toxic water-based paint finish
  • Montessori-approved learning toy for natural child development
A claim UK rules require the seller to prove. Fine to publish once checked.

More from the results

Publish-ready rate against cost

Cost to enrich 10,000 products at the run date's prices, on a log scale. Up and to the left is better.

0%20%40%60%80% $1$10$100$1,000 Cost per 10,000 products (log scale) Reliably publish-ready, % GPT-6.1 Sol GPT-6 Astra GPT-6 Sol Grok 4.7 Claude Opus 5.5 GPT-6 Luna

Why listings fail: factual brief

Share of products failing each check, averaged over three runs. A listing can fail several at once; any one stops it going live.

ModelNo usable outputUnsupported claimsWrong attributesNot findableWrong categoryChannel rulesUK information
GPT-6.1 SolOpenAI0.0%0.0%16.7%4.8%1.8%0.0%0.0%
GPT-6 AstraOpenAI0.0%1.2%17.3%6.0%1.8%0.0%0.0%
GPT-6 SolOpenAI0.0%1.8%14.3%7.7%3.6%0.0%0.0%
Grok 4.7xAI0.0%11.3%15.5%5.4%2.4%0.0%0.0%
Claude Opus 5.5Anthropic0.0%17.3%15.5%3.0%1.8%0.0%0.0%
GPT-6 LunaOpenAI0.0%8.9%18.4%10.1%1.8%0.0%0.0%
Muse Spark 1.3Meta0.0%14.9%16.7%6.0%4.2%0.0%0.0%
Gemini 3.1 Pro PreviewGoogle0.0%26.8%14.9%7.7%1.8%0.6%0.0%
DeepSeek V4.1 FlashDeepSeek0.0%13.1%20.2%3.6%3.0%3.0%0.0%
Kimi K3Moonshot AI0.0%16.7%24.4%4.2%3.0%0.6%0.0%
Gemini 3.8 FlashGoogle0.0%31.6%15.5%8.3%3.6%0.6%0.0%
Claude Sonnet 5.5Anthropic0.0%39.3%19.1%1.2%3.0%1.2%0.0%
Qwen3.8 Max (0902)Alibaba3.6%26.8%21.4%3.6%1.8%0.6%0.0%
Claude Fable 5.1Anthropic0.0%48.8%11.9%1.8%2.4%0.0%0.0%
Gemini 3.5 Flash LiteGoogle0.0%33.3%34.5%20.2%9.5%1.8%0.0%
Mistral Medium 3.5Mistral0.0%49.4%30.9%14.3%11.3%0.0%0.0%
Claude Haiku 4.5Anthropic0.0%73.2%48.2%8.9%7.7%0.0%0.0%
GLM 5V TurboZ.ai32.1%55.4%13.1%5.4%1.2%1.2%0.0%

Why listings fail: sales brief

The same checks when the model is asked for copy that sells.

ModelNo usable outputUnsupported claimsWrong attributesNot findableWrong categoryChannel rulesUK information
GPT-6.1 SolOpenAI0.0%3.6%19.1%5.4%1.8%0.0%0.0%
GPT-6 AstraOpenAI0.0%2.4%20.2%5.4%1.8%0.0%0.0%
GPT-6 SolOpenAI0.0%3.0%13.7%7.1%3.0%0.6%0.0%
Grok 4.7xAI0.0%26.8%15.5%4.2%2.4%0.0%0.0%
Claude Opus 5.5Anthropic0.0%60.1%15.5%1.8%1.8%0.6%0.0%
GPT-6 LunaOpenAI0.0%8.3%19.1%10.1%0.0%0.0%0.0%
Muse Spark 1.3Meta0.0%57.7%14.3%5.4%3.6%0.0%0.0%
Gemini 3.1 Pro PreviewGoogle0.0%67.3%16.1%6.0%1.8%0.0%0.0%
DeepSeek V4.1 FlashDeepSeek0.6%39.9%20.2%3.6%2.4%1.8%0.0%
Kimi K3Moonshot AI0.6%81.5%20.8%3.0%4.2%0.6%0.0%
Gemini 3.8 FlashGoogle0.0%81.5%11.9%8.9%3.0%0.6%0.0%
Claude Sonnet 5.5Anthropic0.0%59.5%20.8%0.6%3.0%0.6%0.0%
Qwen3.8 Max (0902)Alibaba3.6%84.5%19.6%1.8%2.4%3.0%0.0%
Claude Fable 5.1Anthropic0.0%83.9%14.9%1.8%1.8%0.0%0.0%
Gemini 3.5 Flash LiteGoogle0.0%78.6%35.1%21.4%10.7%1.2%0.0%
Mistral Medium 3.5Mistral0.0%76.8%36.3%10.1%13.1%1.2%0.0%
Claude Haiku 4.5Anthropic0.0%91.7%51.2%6.5%6.5%4.2%0.0%
GLM 5V TurboZ.ai82.1%17.9%3.6%1.8%1.2%0.6%0.0%

How CatalogBench works

  1. 1

    The feed row

    A sparse supplier row: title, price, a few attributes, and the supplier's marketing text, usable unless the pack says otherwise.

  2. 2

    The photos

    One to three product photos, including labels with the real facts: volume, strength, origin, ingredients.

  3. 3

    The listing

    The model fills missing attributes and writes the title, highlights, description, alt text, search keywords and category.

  4. 4

    The checks

    Rules check attributes and channel limits, a judge answers yes/no questions against the evidence, and a search test checks shoppers can find it.

  5. 5

    Publish-ready?

    Only if every blocking check passes. Three runs; the headline counts products that passed in all three.

The task

Retailers increasingly let AI write their product listings from a supplier feed and a few photos. The risky part is not the prose: it is attributes that are guessed, claims the pack contradicts or the law requires proof for, and listings that no shopper will find.

Each of the 56 public products is invented, with generated photos that carry real-looking labels. Every feed row hides traps: missing attributes only the photos can answer, a supplier claim the label contradicts, and tempting claims such as “award-winning” with no evidence at all.

Two briefs

Factual
Write an accurate listing from the evidence.
Sales
Write copy that sells. The same facts, the same checks: persuasive is fine, unsupported is not.

What stops a listing going live

Unsupported claims
A specific, checkable claim with no support. Supplier copy counts as support unless the label or photo contradicts it. Puffery such as “durable” or “soft feel” is not a claim.
Wrong attributes
A wrong, missing or invented value, or a feed/photo conflict that was not flagged.
UK information
Statements UK rules require, such as age warnings.
Not a blocker: claims to check
Supplier claims UK rules require the seller to prove, such as certifications, approvals, safety, health and environmental claims. A listing may repeat them, but we count each as a flag to check before publishing.
Channel rules
Lengths, formats and banned terms.
Category
The wrong category, or the wrong variant in the title.
Findability
Two shopper searches per product against near-identical rival listings; the listing must win on what the shopper asked for.

Reading the results

Content quality (coverage, visual detail, key facts in the title) is scored separately and does not stop a listing going live. Costs are the prices charged on the run date and are shown per 10,000 products, a typical catalogue refresh.

What it measures

  • Attributes read from the images, not guessed
  • Supplier claims checked against the pack and UK rules
  • Required UK product information included
  • Listings that shoppers can find in search

What it does not measure

  • Conversion or sales impact
  • Real product photography (a real-photo slice is planned)
  • Writing style beyond the listed checks

Method

  • Rule-based checks first; judged checks are yes or no
  • The judge was checked for bias against Gemini and Claude judges
  • Private products are held back so the set can be refreshed

Checking the judge

The judge is an OpenAI model, and OpenAI models lead this table, so we checked it for bias under the current rules. Gemini 3.1 Pro and Claude Opus 5.5 judged the same outputs from five models on a calibration set. All three judges gave the OpenAI models the same or nearly the same results, and put the models in the same order on the sales brief; on the factual brief only Gemini differed, ranking Claude Opus 5.5 above GPT-6 Luna. The Claude judge was stricter on Claude Opus 5.5 than the OpenAI judge was. The judges differ mainly in how many claims they flag: Gemini flags the fewest.

Failures

Failures count against a model: a product with no usable output is a failed listing. They are listed here so you can see why.

  • DeepSeek V4.1 Flash: invalid JSON: 1 of 168 attempts.
  • Kimi K3: invalid JSON: 1 of 168 attempts.
  • Qwen3.8 Max (0902): reply cut off at the token limit: 6 of 168 attempts.
  • GLM 5V Turbo: invalid JSON (raw line breaks inside text): 138 of 168 attempts.
Run this on your catalogue. The same checks, on a sample of your own products.Catalogue feed diagnostic →