Benchmarks / SurveyBench / Why Quaymark's customer score fell / GPT-6 Luna

Measured by Spring Prompt

GPT-6 Luna: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict: ✓ Sound research
Research score: 97 of 100
Analysis rating: 1,102
Head to head, this task: won 6 of 11

Scorecard

Numbers 12 of 12 right

Question		Its answer	True
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?	✓	12.9	12.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?	✓	17.7	17.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?	✓	20.2	20.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?	✓	70.5%	68.9%
How many of the open-text answers mention the per-shipment fee or its cost?	✓	21	21
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?	✓	4.1 points	4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
What percentage of the accounts invited to the Q3 2026 wave completed it?	✓	11%	11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?	✓	49.2%	49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?	✓	4 points	3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?	✓	13	13
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?	✓	1,001	1,001

Traps in the data 8 of 8 handled

✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
✓Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
✓45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
✓521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Invest first in shipment visibility and reliable proactive delay alerts, while protecting the quick booking experience. Do not make a blanket return to subscription pricing on this survey alone: test an optional subscription, particularly with low-volume customers who report worse value, and assess take-up, retention, contribution margin, and effects on different shipment volumes. Do not add account managers broadly; consider a targeted pilot for the 100+ shipment segment. Treat Ireland as an urgent diagnostic lead, not a settled conclusion. The observed 11-point raw NPS gap should not be presented as a proven fall in loyalty or attributed to the fee change.

What would change it

A controlled pricing test showing that an optional or restored subscription materially improves retention, shipment volume, or customer value without damaging contribution margin would support a broader pricing change. Evidence that tracking and alerts cannot be made sufficiently reliable with partner-haulier data, or that a measured pilot does not improve shipment visibility, delay handling, or customer outcomes, would reduce their priority. Representative Ireland research confirming a distinct and widespread service failure would warrant a dedicated Ireland response. A comparable-mode, repeat survey with stronger response and non-response analysis could change the interpretation of the NPS trend.

Findings

Prioritise better shipment visibility and proactive delay alerts. These are the clearest cross-segment product needs; the evidence supports investing in a focused improvement, though not a promise that it will by itself restore NPS.

High confidenceBase 521

65.6% selected live tracking or late alerts as an improvement (base 521); 41.3% selected live tracking and 34.2% delay alerts. 28.6% were dissatisfied with shipment visibility (base 521), compared with 22.4% dissatisfied with punctuality. Among 44 open-text detractors, 31 (70.5%) mentioned tracking/visibility or late/missed collections or deliveries.
There is a credible case to offer a subscription option, especially to smaller shippers, but this survey alone does not justify reversing the fee for everyone. Test an optional plan and assess its economics before a broad change.

Medium confidenceBase 319

Among respondents booking fewer than 20 shipments monthly, 49.2% said per-shipment pricing was worse (base 319), and 44.5% were likely to choose a subscription. Across all respondents, 39.0% said the fee was worse and 39.5% were likely to choose a subscription (base 521). The open-text responses included 21 explicit fee or fee-cost mentions.
Do not tell the board that customer loyalty definitively fell 11 points. The unweighted NPS difference is 11.1 points, but after matching the Q1 respondent shipment-volume mix, Q3 NPS is 20.2, only 3.8 points below Q1's 24. The survey design cannot establish that the fee caused a change.

High confidenceBase 933

Q3 NPS is 12.9 (base 521) versus Q1 NPS 23.5 (base 412; published rounded value +24). Q3 reweighted to Q1's shipment-volume mix is 20.2. Q1 was conducted by phone by account managers; Q3 was an online email survey, with different samples and response patterns.
Treat Ireland as a signal to investigate, not evidence of a reliable Ireland-wide problem. The observed result is extreme but rests on very few respondents.

Low confidenceBase 14

13 of the 14 respondents who said they ship to the Republic of Ireland scored 0–6 (92.9%; NPS -93). Eleven respondents were not sure whether they shipped to Ireland, and the survey does not establish that the 14 answers represent Ireland customers overall.
Do not prioritise named account managers for all mid-sized customers based on this survey. Consider a targeted service pilot for larger accounts instead.

Medium confidenceBase 521

A named account manager was selected by 15.6% of respondents booking 20–99 shipments monthly (base 154), compared with 31.2% of those booking 100 or more (base 48). In the full sample, 14.2% selected this option (base 521); tracking or delay alerts were selected by 65.6%.

Caveats

The Q3 response rate was 11.0% (521 completions from 4,720 invitations). Respondents may differ from non-respondents, and weighting by shipment-volume band does not correct for other response differences.
Q1 and Q3 used different modes and sampling approaches: Q1 was conducted by phone by account managers; Q3 was an emailed online survey. Mode, interviewer effects, non-response, and sample composition could explain some or all of the observed NPS difference.
The Q3 sample's shipment-volume mix differs from the active-account population: 61.2% of respondents booked fewer than 20 shipments monthly, versus 40% of accounts; 9.2% booked 100 or more, versus 18% of accounts. Weighted estimates address only this measured mix.
NPS measures stated recommendation likelihood, not observed retention or loyalty. Neither the wave comparison nor the self-reported fee answers establish that the pricing change caused the NPS result.
The open-text analysis is qualitative and based on 186 answers, not the full sample. Theme counts should not be treated as population prevalence.
The Ireland result rests on 14 respondents who said they ship to the Republic of Ireland; it is too small and potentially unrepresentative for a firm market-level conclusion.

Next steps

experimentBuild a tracking and delay-alert pilot with a defined subset of hauliers and customers; measure scan freshness, coverage, alert accuracy, and customer outcomes.
experimentTest an optional subscription against per-shipment pricing with customer segments and clear break-even terms; include a holdout and track actual behaviour.
experimentRun a targeted named-account-manager pilot for selected 100+ shipment accounts rather than staffing the whole mid-sized segment.
researchContact the 14 Ireland respondents and review Ireland shipment performance, missed collections, delays, tracking coverage, and complaints using operational data.
monitorRepeat the recommendation measure using a consistent mode and sampling protocol, report segment-weighted results, and analyse response bias.

Open-text themes it coded

Tracking and shipment visibility are inadequate; customers describe chasing hauliers or customers for updates. 42Late or missed collections and deliveries undermine reliability. 29The per-shipment fee is seen as more costly or less predictable than the former subscription. 21Fast, straightforward booking is a clear strength to preserve. 76Slow or difficult customer support is a source of frustration. 8Damage and slow or confusing claims handling are concerns. 5

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

S1
Are you the main person responsible for booking shipments through Quaymark for your business?
One answer
- Yes
- No
Continues if Yes
S2
Has your business booked at least one shipment through Quaymark in the last three months?
One answer
- Yes
- No
- Not sure
Routing Shown only if S1 is: Yes

Continues if Yes
Q1
How likely are you to recommend Quaymark to a colleague at another business?
Scale

Scale 1-11: 0 — Not at all likely to 10 — Extremely likely
Q2
What is the main reason for the score you gave?
Open text
Q3
On average, approximately how many shipments does your business book through Quaymark in a month?
One answer
- Fewer than 20
- 20 to 99
- 100 or more
- Not sure
Q4
Considering the two fee approaches below, which would you prefer Quaymark to offer your business?
One answer
- A fee for each shipment (currently £3.50 per shipment, with a reduced rate for accounts booking 100 or more shipments a month)
- A monthly subscription (previously starting at £25 a month for the smallest accounts)
- A choice between a monthly subscription and a fee for each shipment
- No preference
- Not sure
Q5
Since the fee change in March 2026, what effect, if any, has it had on your business?
One answer
- We have booked fewer shipments through Quaymark
- We are less likely to book shipments through Quaymark, but have not booked fewer so far
- It has increased our costs, but has not changed our booking behaviour
- It has reduced our costs
- It has had no noticeable effect on our costs or booking behaviour
- We are more likely to book shipments through Quaymark
- I joined Quaymark after the fee change, or cannot compare
- Not sure
Q6
Please rank these possible improvements from most valuable to least valuable to your business. Rank 1 as most valuable and rank 4 as least valuable.
Ranking
- Clearer, more detailed shipment tracking
- Proactive alerts when a shipment is delayed
- A named account manager for your business
- A choice between a monthly subscription and a fee for each shipment
Q7
Thinking about shipments you have booked through Quaymark in the last three months, how would you rate the tracking information available to you?
One answer
- Very poor
- Poor
- Neither poor nor good
- Good
- Very good
- I have not used or seen the tracking information
- Not sure
Q8
When you have needed help from Quaymark in the last three months, how easy or difficult was it to get the help you needed?
One answer
- Very difficult
- Difficult
- Neither easy nor difficult
- Easy
- Very easy
- I have not needed help
- I needed help but did not contact Quaymark
- Not sure
Q9
Which, if any, of the following have been problems for your business when using Quaymark in the last three months? Select all that apply.
Any that apply
- The cost of fees
- Understanding or choosing the fee model
- Booking or arranging a shipment
- Getting clear or up-to-date tracking information
- Knowing about shipment delays in time
- Shipment delays or delivery problems
- Getting help or a response from Quaymark
- Other problem
- None of these
Q10
Is there anything else you would like Quaymark to know about your experience or what would improve it?
Open text

Sample plan

Email the survey to the main booking contact at each of the 4,720 active UK business accounts in the customer database. Use the screeners to include only the main booking contact for an account that has booked at least one shipment in the last three months. Target about 520 completed surveys. Monitor completions by the three Q3 account-size bands and, if feasible, use reminders or targeted follow-up to improve representation in underrepresented bands; report actual unweighted base sizes for every band. Weight overall estimates to the account-size distribution of eligible active accounts if reliable frame counts are available; do not weight within-band estimates. The requested precision cannot be guaranteed with 520 completes: even an equal allocation of about 173–174 per band gives a worst-case 95% margin of error of approximately ±7.4 percentage points per band before finite-population correction. To target ±5 points in each band conservatively requires about 385 completes per band, or about 1,155 total; actual requirements may be lower if finite-population corrections based on known band sizes apply. If ±5 points is mandatory, extend fieldwork and set band targets using the eligible population counts rather than presenting the 520-complete results as meeting that precision. The Q1 phone score is a historical benchmark, not a controlled comparison with this online wave.