Gemini 3.1 Pro Preview: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict: ✗ Not sound
Research score: 78 of 100
Analysis rating: 810
Head to head, this task: won 5 of 10

Why it is not sound

The survey

Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

Reported a material finding the data does not support
Missed a trap: Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
Missed a trap: 45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
Missed a trap: 521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
Missed a trap: The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

Findings the judges found unsupported

The reported 11-point NPS drop is a false alarm caused by changing the survey methodology.Reweighting supports a much smaller observed difference, but does not establish that the headline fall is entirely a false alarm or that the true loyalty decline is four points. The method effect and non-response bias have not been measured.
Platform investment must immediately prioritize shipment tracking and proactive delay alerts.The 68.9% figure applies to the 45 detractors who answered Q2, not to all 120 detractors or the stated base of 521. The priority given to tracking and alerts is supported, but this percentage is presented on the wrong base.
The per-shipment fee is highly unpopular among small shippers and risks driving them away.Hypothetical subscription intentions without a specified price do not establish actual switching or churn. Also, large shippers do not largely prefer the fee: 31.2% say it is better, while 41.7% say it is about the same.
There is a severe, localized operational failure with shipments to the Republic of Ireland.Fourteen respondents cannot establish a severe localized operational failure, and recommendation scores do not identify a routing or haulier problem. The small-base caveat does not undo the explicit causal conclusion.

Scorecard

Numbers 12 of 12 right

Question		Its answer	True
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?	✓	13	12.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?	✓	17.8	17.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?	✓	20.3	20.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?	✓	68.9%	68.9%
How many of the open-text answers mention the per-shipment fee or its cost?	✓	21	21
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?	✓	4.1 points	4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
What percentage of the accounts invited to the Q3 2026 wave completed it?	✓	11%	11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?	✓	49.2%	49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?	✓	4 points	3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?	✓	13	13
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?	✓	1,001	1,001

Traps in the data 4 of 8 handled

✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
✗Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
✗45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
✗521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
✗The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Do not invest in account managers for mid-sized customers. Instead, immediately allocate product investment to live tracking and proactive delay alerts, as this is the primary driver of detraction. Commercially, introduce an optional monthly subscription tier targeted specifically at small shippers (<20 shipments/month) to mitigate churn, while retaining the per-shipment fee for larger shippers who prefer it. Concurrently, launch an operational investigation into the haulier network handling shipments to the Republic of Ireland.

What would change it

If technical constraints require more than 6 to 12 months to fix platform tracking, we will need to temporarily invest in manual support or communications for delays. If financial modeling shows an optional subscription for small shippers is unprofitable, alternative retention incentives must be explored.

Findings

The reported 11-point NPS drop is a false alarm caused by changing the survey methodology.

High confidenceBase 521

Q1 2026 was conducted by phone (introducing positive interviewer bias) and leaned heavily toward mid-to-large shippers. Q3 2026 was online, with 61% of respondents being small shippers (who generally score lower). When Q3 is reweighted to match Q1's shipper mix, the NPS is +20, showing the true drop is only roughly 4 points.
Platform investment must immediately prioritize shipment tracking and proactive delay alerts.

High confidenceBase 521

65.6% of respondents selected live tracking or delay alerts as top improvements. Furthermore, 68.9% of detractors explicitly mentioned tracking failures, late deliveries, or the burden of manually chasing hauliers in their open-text feedback.
The per-shipment fee is highly unpopular among small shippers and risks driving them away.

High confidenceBase 319

49.2% of small shippers (<20 shipments a month) say the per-shipment fee is worse for their business, and 44.5% are likely to switch back to a subscription if offered. In contrast, large shippers largely prefer the current fee model.
Adding account managers for mid-sized shippers would be a misdirected investment.

High confidenceBase 154

Only 15.6% of mid-sized customers chose a named account manager as a top improvement, compared to 63% demanding tracking or delay alerts.
There is a severe, localized operational failure with shipments to the Republic of Ireland.

Medium confidenceBase 14

92.9% (13 out of 14) of customers who ship to Ireland scored 0-6 on the recommend question. Despite the small sample size, near-universal dissatisfaction indicates a specific routing or haulier problem.

Caveats

The 11-point NPS drop is heavily influenced by the change from phone interviews to an online survey, and a drastic shift in respondent mix. Direct comparison is statistically flawed without re-weighting.
The insight regarding Ireland shipments relies on only 14 respondents. While the dissatisfaction is stark, the sample size is small.
Open-text analysis is qualitative and based only on those who opted to leave a comment (186 out of 521 respondents).

Next steps

decisionDevelop and launch improved tracking visibility and automated delay notifications.
experimentDesign and test an optional subscription pricing tier for small shippers.
researchInvestigate operational data and haulier performance for shipments to the Republic of Ireland.
monitorStandardize future surveys to be online-only and monitor weighted scores.

Open-text themes it coded

Quick and easy booking process 78Lack of tracking and shipment visibility 42Late deliveries and missed collections 30Dislike of the per-shipment fee / Pricing complaints 21Poor customer support and slow response times 7Damaged goods and difficult claims process 5

The survey it planned

3 screening questions and 5 questions, as the model wrote them.

S1
Are you the main person responsible for booking freight shipments for your business?
One answer
- Yes
- No
Continues if Yes
S2
Has your business booked at least one shipment through Quaymark in the past 3 months?
One answer
- Yes
- No, not in the past 3 months
- Not sure
Continues if Yes
S3
Roughly how many shipments does your business book through Quaymark in a typical month?
One answer
- Fewer than 20
- 20 to 99
- 100 or more
Continues if Fewer than 20; 20 to 99; 100 or more
Q1
Based on your recent experiences, how likely are you to recommend Quaymark to a colleague or another business?
Scale

Scale 1-11: 0 - Not at all likely to 10 - Extremely likely
Q2
What is the primary reason for the score you just gave?
Open text
Q3
In March 2026, Quaymark changed its pricing structure from a monthly subscription to a per-shipment booking fee. Thinking about your business's needs, which pricing model do you prefer?
One answer
- The previous monthly subscription
- The current per-shipment booking fee
- No preference / Not sure
Q4
How has the change to the per-shipment booking fee impacted your overall satisfaction with Quaymark?
One answer
- Significantly increased my satisfaction
- Somewhat increased my satisfaction
- No impact on my satisfaction
- Somewhat decreased my satisfaction
- Significantly decreased my satisfaction
Judges: assumptive
The screeners admit customers who began using Quaymark after the pricing change. They cannot assess a change in their satisfaction from the previous model, but no not-applicable or unsure response is available.
Assumes the respondent was a customer before the March 2026 change. Accounts active in the last three months may have joined after the switch and never used the subscription. There is no 'not applicable' option and no display logic to route them past the question.
Q5
If Quaymark could make one primary improvement to its service, please rank the following options in order of value to your business (1 = Most valuable, 3 = Least valuable).
Ranking
- Enhanced online shipment tracking
- Proactive alerts about shipment delays
- A dedicated account manager for support

Sample plan

Email invitations sent to the main booking contact of all 4,720 active UK accounts. Target 520 total completes. To approach the board's precision requirement of +/- 5% at 95% confidence for the share of 9-10 scores per account size, strict quotas of ~173 completes must be set for each of the three size tiers (<20, 20-99, 100+). Note: Statistically, a sample of 173 per group yields a +/- 5% margin of error only if the true proportion of Promoters is quite high (>85%) or low (<15%); if the share is closer to 50%, the margin of error per subgroup will be approximately +/- 7.5%.