Experiment 02 · Tested 4 November 2026

Which model should reply to your ecommerce customers?

A support reply can sound perfect and still promise a refund nobody agreed to. We tested seven models on invented orders. With a basic prompt, the most expensive model passed none of them.

Test your own prompt See who won ↓

Best value: $0.39
Claude Opus 5.5: 0 → 34

A test case, not a customer. We built this on invented data and ran it on 4 November 2026. Models and prices change, and we don't rerun these often, so check the date.

What we tested

2prompts: a plain first one, and one rewrite
24test cases, 18 kept unseen for scoring
7models, each at two thinking settings
7checks on every answer
756answers scored, on 4 November 2026

First prompt 261 characters

You are a customer support agent for an online store. Reply to the customer using the order details. Our policy: returns within 30 days of delivery, unworn, and the customer pays return postage unless the item is faulty.

Order: {{order}}

Customer: {{message}}

After one rewrite 1158 characters

You are a customer support agent for an online store. Reply to the customer using the order details in at most 110 words. Acknowledge the customer's feelings in one short sentence, not just a thank-you or an apology for the problem.

Our policy: returns within 30 days of delivery, unworn, and the customer pays return postage unless the item is faulty. Day 30 is included. For a reported fault, explain that the customer does not pay return postage. Do not invent rules about trying items on, tags or packaging; if unworn status is unclear, do not confirm eligibility.

State only supported order facts. This policy does not specify refunds, replacements, payment methods, processing times or return procedures: do not promise or invent them, including labels, addresses or actions already taken. Do not guarantee a requested date.

Give one clear next step, not a menu of options. Where return instructions or eligibility need confirmation, ask the customer to contact a support person for that confirmation rather than inventing details. Refer chargebacks, legal threats and safety issues to a human support agent.

Order: {{order}}

Customer: {{message}}

Who won

Best valueDeepSeek V4.1 Flash, medium thinking

32 of 36 at $0.39 per 1,000 answers, at a median of 6.0 s.

Best qualityGPT-6.1 Sol, low thinking

36 of 36 at $1.50. The choice if replies go out with no review.

Fastest worth usingMercury 3, medium thinking

29 of 36 in 2.1 s at $0.25, if an agent checks each reply.

Passes against cost, after the rewriteOff or low thinking Medium thinking

Hover over a point for the model and setting. Every run is in the table below.

All 14 runs

Model	Thinking	First prompt	After one rewrite	Cost per 1,000 answers	Median time	Thinking tokens
GPT-6.1 Sol	Low	23	36 of 36	$1.50	4.5 s	5
GPT-6.1 Sol	Medium	not run	36 of 36	$1.96	6.3 s	51
Claude Opus 5.5	Low	0	34 of 36	$8.01	5.4 s	144
DeepSeek V4.1 Flash	Medium	not run	32 of 36	$0.39	6.0 s	584
Claude Haiku 5.5	Medium	not run	32 of 36	$0.44	6.3 s	651
Gemini 3.8 Flash	Low	1	32 of 36	$0.76	3.2 s	42
Gemini 3.8 Flash	Medium	not run	32 of 36	$5.31	11.0 s	1267
Claude Opus 5.5	Medium	not run	31 of 36	$11.52	7.0 s	327
Mercury 3	Medium	not run	29 of 36	$0.25	2.1 s	562
DeepSeek V4.1 Flash	Off	1	25 of 36	$0.06	1.6 s	0
Nex N2.5 Mini	Medium	not run	23 of 36	$0.06	5.0 s	547
Mercury 3	Off	0	17 of 36	$0.06	0.8 s	0
Claude Haiku 5.5	Off	0	15 of 36	$0.13	2.5 s	0
Nex N2.5 Mini	Off	7	13 of 36	$0.01	1.1 s	0

Sorted by passes, then cost. First prompt was run at the lowest setting only. Thinking tokens are per answer, as each provider reports them. Cost is the model's price for the answer, per 1,000 answers, on the test date. A code fence around JSON counts as valid, but anything that doesn't parse is a fail.

What went wrong, and what the rewrite fixed

The share of answers that passed each check, across all seven models with thinking at its lowest setting.

First prompt After one rewrite

At most 110 words 35%94%

Hands chargebacks, legal threats and safety issues to a person 35%95%

Acknowledges how the customer feels in one short sentence nice to have 41%89%

Promises no refund, replacement or date the order and policy don't support 51%97%

Applies the returns policy correctly to this order 64%78%

Names one clear next step 87%94%

Every order fact it states matches the order details 90%93%

Real answers from the test

A chargeback threat

Order 6021, trainers, shipped 1 Nov, not delivered, £90 Where is it? I'll do a chargeback if it's not here tomorrow.

First prompt · Claude Opus 5.5, low thinking

“Opening an investigation with the courier”

It claimed an action it can't take, and didn't hand the chargeback to a person.

After the rewrite · DeepSeek V4.1 Flash, medium thinking

“Because you mentioned a chargeback, I need to refer this to a human support agent.”

Day 30 of the returns window

Order 9106: ankle boots, £85, delivered exactly 30 days ago. Today's day 30. They're unworn, but I can't get to the Post Office until tomorrow. Can I still return them?

First prompt · Claude Opus 5.5, low thinking

“we'll refund the £85 to your original payment method”

The policy doesn't say how refunds work, so this is a promise nobody made.

After the rewrite · DeepSeek V4.1 Flash, medium thinking

“our returns window includes day 30”

A gift with no order number

My aunt bought me a hat from your shop last week. It's unworn, but I haven't got a receipt or an order number. Can I return it without her finding out?

First prompt · Claude Opus 5.5, low thinking

“it falls well within our 30-day return window”

The window runs from delivery, and nobody knows the delivery date.

After the rewrite · DeepSeek V4.1 Flash, medium thinking

“because you don’t have a receipt or order number, I can’t confirm the order or whether the hat is eligible for return”

What changed in the prompt

Spell out what the policy doesn't cover
Models invented refund routes and timescales. Tell the model what isn't specified, and not to fill it in.
Pin down the edge cases
State whether day 30 counts, and what happens with faulty items. Applying the policy was still the most common failure after the rewrite.
Ask for one next step
A list of options fails the check. Tell the model to give one clear action.
Then test for helpfulness
Our rewrite made models cautious. Many passing replies simply sent customers to a person, so add a check that the reply solves what it can.

Does more thinking help?

This task showed the largest effect. Haiku went from 15 of 36 with thinking off to 32 on medium, at 3.4 times the cost. Mercury went from 17 to 29, and DeepSeek from 25 to 32 at 6.5 times the cost. At the top it made no difference or went the wrong way: Sol scored 36 on both, Opus fell from 34 to 31, and Gemini stayed at 32 for 7 times the cost.

Off or low Medium Cost change

Claude Haiku 5.515 → 323.4x

Mercury 317 → 294.2x

Nex N2.5 Mini13 → 236.0x

DeepSeek V4.1 Flash25 → 326.5x

GPT-6.1 Sol36 → 361.3x

Gemini 3.8 Flash32 → 327.0x

Claude Opus 5.534 → 311.4x

How we tested it

We wrote a plain first prompt, the kind you'd write on day one, and three awkward cases by hand. A model drafted the rest, for 24 cases in all.
The prompt ran on 6 of those cases. GPT-6.1 Sol rewrote it once from what failed, which is what Spring Prompt's assistant does.
Both prompts then ran on the 18 cases the rewrite never saw, twice, so scores are out of 36. An answer passes only if it meets every must check.
We set thinking on purpose. Lowest means off where a model allows it, and low where it doesn't. We also ran the rewrite at medium.
GPT-6 Luna judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand.

What this can't tell you

The cases are invented, so a real product will behave differently. Your prompt, data and customers matter more than this table.
18 cases is small. Gaps of two or three cases can be chance, so read the top few as level.
A model drafted most cases and a model judged them.
Costs are model prices for the answers alone, and they change often. Times come from one day's runs.
The checks reward caution. A pass means the reply broke no rule, not that it was the most helpful reply possible.

Invite only

Run this on your own prompt.

Paste a prompt, and Spring Prompt drafts the test cases and checks. You'll see which model passes, what it costs and how fast it is. We're opening it up by invitation, with test credits to start.

Request an invite Takes a minute. We reply by email.