Claude Haiku 5.5: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict: ✗ Not sound
Research score: 93 of 100
Analysis rating: 1,412
Head to head, this task: won 7 of 9

Why it is not sound

The survey

A double-barrelled question: Q7

The analysis

Reported a material finding the data does not support

Findings the judges found unsupported

The per-shipment fee is seen as worse than the subscription by most small accounts and by most detractors, but high-volume accounts see it as better. Reinstating a subscription across the board would therefore not be a simple win.Neither small accounts nor detractors have a majority saying the fee is worse: the figures are 49.2% and 47.5%. Nor do high-volume accounts mostly prefer per-shipment pricing: 31.2% say better, while 41.7% say about the same. The recommendation turns a positive balance among high-volume respondents into a majority preference.
Ireland customers are almost uniformly unhappy: 13 of 14 Ireland shippers score 0-6 and none are promoters. This is a serious warning sign, but the base is far too small to size the problem or to say what is driving it. None of the 186 open-text answers mentions Ireland.The open-text records explicitly include the Ireland segment flag in their metadata. Ireland respondents' answers can therefore be isolated, and their stated reasons are available. Not mentioning Ireland explicitly does not make those reasons unknown.
The Q3 promoter share (35.9%) is known to within about plus or minus 4 points at 95% confidence, which is adequate for direction but not for small segment comparisons.The calculated margin of error describes nominal sampling uncertainty, not how precisely the population promoter share is known. An unweighted, self-selected sample with an 11% response rate and a skewed size mix does not establish population accuracy within four points.

Scorecard

Numbers 12 of 12 right

Question		Its answer	True
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?	✓	12.9	12.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?	✓	17.7	17.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?	✓	20.3	20.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?	✓	68.9%	68.9%
How many of the open-text answers mention the per-shipment fee or its cost?	✓	21	21
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?	✓	4.1 points	4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
What percentage of the accounts invited to the Q3 2026 wave completed it?	✓	11%	11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?	✓	49.2%	49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?	✓	4 points	3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?	✓	13	13
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?	✓	1,001	1,001

Traps in the data 8 of 8 handled

✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
✓Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
✓45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
✓521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Do not reverse the per-shipment fee across the board on this evidence. The largest, most consistent problem among detractors is service: tracking gaps and late or missed collections and deliveries. Fund shipment tracking and proactive delay alerts first, because they are the top improvement requested overall (65.6%) and appear in most detractor comments. Separately, test a low-volume plan or subscription for under-20 accounts, where 49% say the fee is worse and 44.5% would be likely to choose a subscription, while keeping per-shipment pricing for 100+ accounts, who mostly prefer it. Defer account managers. Investigate Ireland urgently with a call-back to the Ireland accounts before drawing conclusions. Do not tell the board that loyalty fell 11 points.

What would change it

(1) A matched-mode re-survey (same online method, same segment mix) showing a decline in NPS of more than about 8 points from a pre-fee baseline would make the fee the priority. (2) A subscription pilot for under-20 accounts that shows a clear retention or NPS gain would support reinstating it for that segment. (3) If the Ireland follow-up shows a delivery or carrier problem, Ireland becomes a separate service fix. (4) If tracking and alert experiments show no NPS or retention gain, the service case weakens and fee or pricing changes should be reconsidered.

Findings

The 11-point fall in NPS is not a like-for-like measure of a fall in customer loyalty. The two waves differ in mode (online self-completion in Q3 vs phone calls by account managers in Q1), in response rate (11% online vs 64% of called customers), and in segment mix. Reweighted to the Q1 mix, Q3 NPS is +20.3 against Q1's +24, a gap of about 4 points that is within noise. The board should not be told that loyalty fell 11 points.

Medium confidenceBase 933

Q1 2026 phone NPS +24 (n=412). Q3 2026 online NPS +12.9 (n=521). Q3 reweighted to Q1 segment mix (30/46/24 by shipments a month) gives +20.3. Segment NPS in Q3: under 20 +5, 20-99 +21, 100+ +38. Standard error of the raw 11-point gap is about 5 points, so the raw gap is nominally significant, but mode and mix explain most of it. The phone wave also predates the March 2026 fee, so the fee cannot be isolated from these data.
Service failure, not the fee, is the dominant theme among detractors. Late or missed collections and deliveries, and the inability to see where shipments are, are mentioned in most detractor comments. Tracking and delay alerts are also the most-requested improvement overall. Under-20 accounts, the largest group, are the main source of the fee complaints.

Medium confidenceBase 45

Of the 45 open-text answers from respondents scoring 0-6, 31 (68.9%) mention tracking/visibility or late/missed collections or deliveries. Q7: live tracking 41.3% and delay alerts 34.2%, net 65.6%; 72.9% among 100+ accounts. Among detractors, net tracking or alerts is 69.2%. Coding of open text was done by hand, so counts could move by a few answers under a different coder.
The per-shipment fee is seen as worse than the subscription by most small accounts and by most detractors, but high-volume accounts see it as better. Reinstating a subscription across the board would therefore not be a simple win.

Medium confidenceBase 521

Q5 overall: better 20.0%, same 29.9%, worse 39.0%. Under 20 a month: worse 49.2% vs better 15.0%. 20-99: better 26.6% vs worse 25.3%. 100+: better 31.2% vs worse 14.6%. Detractors: worse 47.5%. Open text: 21 comments mention the fee, almost all from under-20 and 20-99 accounts. The 100+ segment has only 48 respondents.
A subscription option would attract a sizeable minority of customers, concentrated in small accounts. Stated intent overstates real take-up, and the subscription price has not been tested.

Medium confidenceBase 521

Q6 net likely to choose subscription: 39.5% overall; 44.5% under 20 (n=319); 29.9% for 20-99 (n=154); 37.5% for 100+ (n=48). Among those who said the fee is worse: 61.6% likely. Among those who said better: 27.9% likely. Q7 does not offer a subscription as an option, so the 'lower booking fees' answer (22.6%) is the closest proxy.
Ireland customers are almost uniformly unhappy: 13 of 14 Ireland shippers score 0-6 and none are promoters. This is a serious warning sign, but the base is far too small to size the problem or to say what is driving it. None of the 186 open-text answers mentions Ireland.

Low confidenceBase 14

Q9: 14 respondents (2.7%) ship to the Republic of Ireland. Of these, 13 score 0-6 (92.9%) and none score 9-10. NPS -93 versus +16 for the rest. The Ireland respondents' open answers cannot be isolated because the open text does not carry this segment flag beyond the answer itself, so their reasons are unknown.
The small-account NPS fall of 4 points between waves is not statistically distinguishable from zero. Its standard error is about 8 points, and the mode change makes it weaker still.

Low confidenceBase 443

Under 20 a month: Q1 phone NPS +9 (n=124) vs Q3 online NPS +5 (n=319). The standard error of the difference is about 8 points.
The Q3 sample is a self-selected 11% of accounts. Respondents may be more engaged or more aggrieved than the base, so the NPS estimate of +13 (or +17.7 when weighted to the population) may not describe all 4,720 accounts.

High confidenceBase 521

521 completes from 4,720 invited (11.0%). Median completion time was 4m 40s. Reweighting to the population mix moves NPS from +12.9 to +17.7. Detractor accounts across the population, weighted, are about 1,001.
The Q3 promoter share (35.9%) is known to within about plus or minus 4 points at 95% confidence, which is adequate for direction but not for small segment comparisons.

High confidenceBase 521

Standard error of 35.9% at n=521 is 2.1 points; 95% margin is 4.1 points. Segment margins are much wider, e.g. about 13 points for the 100+ segment at n=48.
The 100+ segment's promoter share has not changed significantly between waves. The sample is too small to say whether high-volume customers are more or less loyal.

High confidenceBase 146

100+ promoters: Q1 49.0% (n=98) vs Q3 52.1% (n=48). Difference 3.1 points; pooled z-statistic about 0.35, not significant at 95%.
Account managers for mid-sized customers are not supported as a first investment by these data. Only 14.2% ask for a named account manager, and the 31.2% in the 100+ segment rests on 48 respondents.

Medium confidenceBase 521

Q7 named account manager: 14.2% overall; 15.6% for 20-99 (n=154); 31.2% for 100+ (n=48). Tracking and alerts are preferred over an account manager in every segment.

Caveats

The headline 11-point fall compares a phone wave run by account managers with an online wave. Mode, response rate (64% vs 11%), timing (before vs after the March fee) and segment mix all differ, so the change cannot be attributed to the fee.
Online response was 11%. Non-response bias is unknown; satisfied or dissatisfied customers may have replied at different rates.
The Q1 phone sample was drawn from accounts that account managers chose to call, and those managers may have influenced answers.
Ireland has 14 respondents; the 13-of-14 detractor figure has a very wide confidence interval and the open text cannot be linked to the Ireland segment.
Open-text coding is my own judgement. Some answers were coded to more than one theme, and borderline cases (e.g. Q354, which blends support complaints with chasing the haulier) could be read differently.
Only 45 detractors gave open-text answers, so detractor-level percentages rest on a small base.
Q6 measures stated intent, which typically overstates take-up. The subscription price was not specified.
Segment bases are small for 100+ accounts (n=48), so segment results for that group are indicative only.
Weighted NPS assumes the online respondents within each segment resemble the segment's non-respondents.

Next steps

decisionBrief the commercial director and CEO that the 11-point drop is confounded by mode, response rate and mix; the mix-adjusted change is about 4 points and not significant. Do not use the 11-point figure with the board as a loyalty measure.
decisionFund and scope shipment tracking (status updates at each scan) and proactive delay and missed-collection alerts, starting with pallets and parcels to Scotland and the North, where the open text concentrates complaints.
experimentRun a pricing test: a monthly plan or capped fee for accounts booking under 20 shipments a month, against the current per-shipment model, with a matched control group.
researchCall back all Ireland shipping accounts (about 14 in the sample; more in the full database) to find out why they are unhappy, and check carrier performance on Ireland lanes.
researchRe-run the NPS survey online with a larger, stratified sample and a nonresponse follow-up, keeping the same question wording, and reweight to the population mix.
monitorTrack NPS, the fee-worse share and lateness complaints by segment every quarter, with the open text coded against a fixed frame.
decisionDo not fund account managers for mid-sized customers now. Revisit once tracking and alerts are live and a sample of 100+ and 20-99 accounts has been interviewed.

Open-text themes it coded

Tracking and visibility gaps 42Late or missed collections and deliveries 29Per-shipment fee costs more than the subscription 21Quick and easy booking (praise) 75Customer support slow or hard to reach 6Damaged goods and slow claims 5Neutral or low engagement (no strong view, new users) 22

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

S1
Are you the main contact who books shipments with Quaymark for your business?
One answer
- Yes
- No
Continues if Yes
S2
Has your business booked at least one shipment through Quaymark in the last three months?
One answer
- Yes
- No
- Not sure
Continues if Yes
S3
Is your business registered and operating in the United Kingdom?
One answer
- Yes
- No
Continues if Yes
Q1
On a scale from 0 to 10, how likely are you to recommend Quaymark to a colleague or another business that sends goods?
Scale

Scale 1-11: Not at all likely to Extremely likely
Q2
What is the main reason for the score you just gave?
Open text
Q3
Which of the following have most affected your experience of Quaymark in the last three months? Select all that apply.
Any that apply
- Price or fees
- Speed of getting a price
- Accuracy of the price quoted
- Collection reliability
- Delivery reliability
- Shipment tracking information
- Notice of delays
- Speed of support responses
- Quality of support answers
- Account management contact
- Ease of use of the website
- Nothing has affected my experience
- Other
Q4
Which of these ways of paying for Quaymark would suit your business better?
One answer
- A fee of £3.50 for each shipment, reduced for accounts booking 100 or more a month
- A monthly subscription, from £25 a month for the smallest accounts
- Neither: I have no preference
- Not sure
Q5
What is the main reason for your answer to the previous question?
Open text
Q6
Compared with a monthly subscription, how well does the per-shipment fee fit the way your business books?
Scale

Scale 1-5: Much worse fit to Much better fit
Q7
Please rank these improvements from most to least important for your business. Drag the most important to the top.
Ranking
- Live shipment tracking with updates at each stage
- Proactive alerts when a collection or delivery is likely to be delayed
- A named account manager for your business
- Lower or simpler pricing
- Faster responses from support
Judges: double-barrelled
'Lower or simpler pricing' combines two distinct improvements in one ranked item, so its rank cannot distinguish demand for lower prices from demand for simpler pricing.
The option 'Lower or simpler pricing' joins two different improvements into one rankable item. A respondent who wants simpler pricing, such as a return to the subscription, cannot be told apart from one who wants lower prices.
Q8
In the last three months, roughly how many times did you contact Quaymark support about a shipment?
Number
Q9
To show you are reading carefully, please select 'Somewhat agree' for this statement.
Scale

Scale 1-5: Strongly disagree to Strongly agree
Q10
In a typical month, how many shipments does your business book through Quaymark?
One answer
- Fewer than 20
- 20 to 99
- 100 or more
- Not sure

Sample plan

Invite the main booking contact of each of the 4,720 active UK business accounts (booked at least one shipment in the last three months) by email from the customer database, with the account's size band (from the database) attached to each invitation record so results can be analysed by band without asking respondents to identify themselves. Screen with S1 to S3. Target about 520 completes. Quotas by size band from the database, with at least 173 completes per band: fewer than 20 shipments a month, 20 to 99, and 100 or more. Because the 100 or more band is likely the smallest, invite every account in it and allow the other bands to fill the remaining places; close each band when its quota is reached. Precision: the board's target of plus or minus 5 percentage points at 95% confidence needs about 385 completes per band (worst case, share of 9-10 scores at 50%), so about 1,155 in total. With 520 completes, the whole sample reaches about plus or minus 4.3 points, but each band of about 173 reaches only about plus or minus 7.4 points. Either raise the total to about 1,155 (a response rate of about 25%, which is unlikely from one email), or report bands at plus or minus 7 to 8 points and state this limit in the board report. Quality checks: remove respondents who fail Q9 and those who complete in under 2 minutes; check Q10 against the database band and record mismatches. Expect a low response rate (about 10 to 15%) and check for non-response bias by comparing respondent and non-respondent mix by band, and by Q1 2026 NPS status if available. Send one reminder after 5 working days.