Benchmarks / SurveyBench / Why Quaymark's customer score fell / Gemini 3.8 Flash

Measured by Spring Prompt

Gemini 3.8 Flash: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict: ✗ Not sound
Research score: 84 of 100
Analysis rating: 898
Head to head, this task: won 8 of 12

Why it is not sound

The survey

A double-barrelled question: Q7

The analysis

Reported a material finding the data does not support
Missed a trap: Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
Missed a trap: The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

Findings the judges found unsupported

The primary driver of customer dissatisfaction and churn risk is post-booking operational opacity (lack of tracking and missed/late collections), not the booking experience or pricing.The 68.9% refers to the 45 detractors who answered the open question, not all detractors. The finding omits that denominator and gives a base of 521.
Reversing the per-shipment fee is not warranted for volume shippers; fee resentment is concentrated among low-volume accounts.Describing 44.5% of small accounts as wanting a subscription converts hypothetical, unpriced stated likelihood into established demand. Actual take-up needs testing.
Republic of Ireland shipping suffers from acute operational breakdown, but accounts for a very small share of platform users.Fourteen respondents cannot establish acute operational breakdown across Ireland shipping or Ireland customers' share of platform users. The caveat acknowledges the small base but still asserts that the operational friction is established.

Scorecard

Numbers 12 of 12 right

Question		Its answer	True
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?	✓	12.9	12.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?	✓	17.8	17.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?	✓	20.3	20.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?	✓	68.9%	68.9%
How many of the open-text answers mention the per-shipment fee or its cost?	✓	21	21
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?	✓	4.1 points	4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
What percentage of the accounts invited to the Q3 2026 wave completed it?	✓	11%	11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?	✓	49.2%	49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?	✓	4 points	3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?	✓	13	13
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?	✓	1,001	1,001

Traps in the data 6 of 8 handled

✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
✗Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
✓45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
✓521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
✗The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Do not reverse the per-shipment fee or hire account managers for mid-sized accounts. Invest platform resources into live milestone tracking and automated late collection/delivery alerts. To address fee sensitivity among small shippers without sacrificing transaction revenue, introduce an optional 'Light Shipper' subscription or monthly fee-cap tier.

What would change it

Evidence that churn among accounts booking 20+ shipments is primarily driven by per-shipment fee costs rather than missed SLAs, or proof that haulier partners cannot support API-based milestone/GPS tracking scans, which would force Quaymark to rely on manual account managers to handle tracking chasers.

Findings

The primary driver of customer dissatisfaction and churn risk is post-booking operational opacity (lack of tracking and missed/late collections), not the booking experience or pricing.

High confidenceBase 521

In Q7, 65.6% of respondents selected live tracking or delay alerts as their top improvement (41.3% live tracking, 34.2% delay alerts), including 69.2% of Detractors. In Q4, 28.6% are dissatisfied with visibility (48.4% of Detractors). In open comments, 68.9% of Detractors specifically cited tracking black holes or late/missed collections, whereas booking speed was universally praised as a 'two-minute job'.
Reversing the per-shipment fee is not warranted for volume shippers; fee resentment is concentrated among low-volume accounts.

High confidenceBase 521

Overall, only 39.0% view the fee as worse than the subscription, and only 39.5% say they would be likely to switch back (Q6). However, this divides sharply by volume: 49.2% of <20 shippers say the fee is worse (and 44.5% want a subscription), compared to only 25.3% of 20-99 shippers and 14.6% of 100+ shippers. In Q7, 'Lower booking fees' ranked fourth overall at 22.6%.
Dedicated account managers are not a priority for mid-sized customers, although high-volume shippers show moderate interest.

High confidenceBase 521

Only 15.6% of mid-sized shippers (20-99/month) selected 'A named account manager' as a top improvement in Q7, trailing live tracking (40.3%), delay alerts (35.1%), lower booking fees (19.5%), and quicker support (22.1%). Among 100+ shippers, interest was 31.2%, but tracking remained their highest priority (54.2%).
The apparent 11-point NPS drop from Q1 to Q3 is largely a methodological artifact of survey mode and sample composition rather than true loyalty erosion.

High confidenceBase 521

Q1 was conducted by telephone by Quaymark's own account managers (introducing strong social desirability bias) and over-indexed on large accounts (23.8% 100+ vs 9.2% in Q3). When Q3 is re-weighted to match Q1's customer mix, Q3 NPS is +20.3 (a 3.7-point drop, well within survey margins of error). Among 100+ accounts, NPS was identical (+38 in Q1 and Q3).
Republic of Ireland shipping suffers from acute operational breakdown, but accounts for a very small share of platform users.

Medium confidenceBase 14

Among respondents shipping to Ireland, NPS is -93, with 13 out of 14 (92.9%) scoring 0-6. Qualitative remarks cite 3-to-5 day delays and hauliers failing to collect. However, this represents only 2.7% of surveyed accounts (base of 14).

Caveats

Mode effect invalidates direct Q1-to-Q3 headline tracking: Q1 was an interviewer-led telephone survey conducted by Quaymark's own account managers, whereas Q3 was an anonymous online survey. Phone administration notoriously inflates NPS via social desirability bias.
Unweighted sample skew: Q3 respondents over-represented small shippers (<20 shipments/month: 61.2% in survey vs 40.0% in population) and under-represented large shippers (9.2% in survey vs 18.0% in population). Population-weighted NPS is +18, not +13.
Very small sample for Ireland shippers (n=14): While the extreme dissatisfaction (-93 NPS) highlights real operational friction, findings must be treated qualitatively rather than as a precise metric.
Non-response bias: An 11.0% completion rate means accounts with extreme positive or negative experiences were more motivated to complete the survey.

Next steps

decisionPrioritise product engineering on haulier tracking integrations and proactive SMS/email delay notifications.
researchAudit carrier performance and transit handoffs for Republic of Ireland routes.
experimentModel a hybrid pricing structure offering an optional flat subscription or volume discount specifically tailored to small shippers (<20/month).
monitorEstablish a consistent, quarterly online survey methodology with fixed population weighting.

Open-text themes it coded

Fast, effortless upfront booking experience 78Post-collection tracking opacity and lack of status updates 42Unpunctual, late, or missed haulier collections and delayed deliveries 29Frustration with per-shipment fees eroding margins compared to subscription 21Slow customer service response times and painful freight damage claims 11

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

S1
Are you the person primarily responsible for booking freight shipments (pallets or parcels) for your business on Quaymark?
One answer
- Yes, I make or oversee the bookings
- No, someone else books our shipments
- We do not use Quaymark
Continues if Yes, I make or oversee the bookings
S2
Has your business booked at least one freight shipment through Quaymark in the last three months (since May 2026)?
One answer
- Yes
- No
- Not sure
Continues if Yes
Q1
On average, roughly how many freight shipments (pallets or parcels) does your business book through Quaymark in a typical month?
One answer
- Fewer than 20 shipments a month
- 20 to 99 shipments a month
- 100 or more shipments a month
Q2
How likely are you to recommend Quaymark to another business for their pallet or parcel shipping?
Scale

Scale 1-11: 0 - Not at all likely to 10 - Extremely likely
Q3
What is the primary reason for the score you gave?
Open text
Q4
In March 2026, Quaymark replaced its flat monthly subscription fee with a £3.50 fee per booking (with volume discounts for 100+ bookings). How has this change affected your overall shipping costs with Quaymark?
One answer
- Significantly increased our costs
- Slightly increased our costs
- Made no noticeable difference
- Slightly reduced our costs
- Significantly reduced our costs
- Not sure / I wasn't using Quaymark before March 2026
Q5
Thinking about your business's shipping needs, which pricing model do you personally prefer?
One answer
- A flat monthly subscription (e.g., from £25/month with no per-booking fee)
- A pay-as-you-go fee per shipment booked (e.g., £3.50/shipment, lower at high volume)
- No preference either way
Q6
Please briefly explain why you prefer this pricing model.
Open text

Routing Shown only if Q5 is: A flat monthly subscription (e.g., from £25/month with no per-booking fee); A pay-as-you-go fee per shipment booked (e.g., £3.50/shipment, lower at high volume)
Q7
How satisfied are you currently with Quaymark's online shipment tracking and delay notifications?
Scale

Scale 1-5: 1 - Very dissatisfied to 5 - Very satisfied

Judges: double-barrelled, assumptive
One satisfaction score covers both tracking and delay notifications, which customers may assess differently. It also assumes sufficient experience of both services; neither the screeners nor an inapplicable response option accommodates customers without that experience.
Asks about two features, online tracking and delay notifications, with one rating. It also assumes respondents have used both. Proactive delay alerts may not exist yet, since they are an investment option, and there is no 'not used' option.
Q8
How important is it for your business to have a dedicated, named account manager at Quaymark (as opposed to standard customer support)?
Scale

Scale 1-5: 1 - Not at all important to 5 - Extremely important
Q9
Please rank the following potential platform enhancements in order of value to your business, where 1 is the most valuable and 3 is the least valuable.
Ranking
- Live shipment tracking with proactive alerts when delays occur
- Dedicated account manager for support, rate negotiation, and issue resolution
- Returning to a flat monthly subscription instead of per-shipment fees
Q10
What is the single most important improvement Quaymark could make to better support your freight operations?
Open text

Sample plan

A census email invitation will be deployed to the main booking contacts of all 4,720 active UK accounts with at least one booking in the past 3 months. To meet the board's precision requirement of ±5% margin of error at 95% confidence for the share of promoters (scores 9-10) within each tier (<20, 20-99, and 100+ shipments/month), each tier requires approximately n=250-320 completes (adjusting for finite population size per segment). Achieving this across all three tiers requires a total stratified target of approximately 800-900 completes, or at least n=175 per tier if capped strictly at ~520 completes (which yields ±7.4% margin of error per subgroup). To maximize volume, we will invite all 4,720 accounts, monitor response rates across the three volume strata via Q1/database matching, and use targeted email reminders for under-represented tiers (especially 100+ shipments).