Benchmarks / SurveyBench / Why Quaymark's customer score fell / GPT-6.2 Astra

Measured by Spring Prompt

GPT-6.2 Astra: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict
✓ Sound research
Research score
100 of 100
Analysis rating
1,467
Head to head, this task
won 7 of 9

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?✓1312.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?✓17.817.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?✓20.320.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?✓68.9%68.9%
How many of the open-text answers mention the per-shipment fee or its cost?✓2121
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?✓4.1 points4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What percentage of the accounts invited to the Q3 2026 wave completed it?✓11%11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?✓49.2%49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?✓4 points3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?✓1313
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?✓1,0011,001

Traps in the data 8 of 8 handled

  • ✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
  • ✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
  • ✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
  • ✓Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
  • ✓45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • ✓521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
  • ✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
  • ✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Fund a measured first release of trustworthy tracking and proactive delay alerts. Keep per-shipment pricing available and test an optional, explicitly priced subscription rather than reverting everyone. Defer broad mid-volume account-manager hiring. In parallel, investigate the Ireland signal and address verified carrier lateness rather than expecting notifications to fix it.

What would change it

Shift priority to pricing if invoice and churn evidence, including lapsed accounts, shows material fee-related losses and a real subscription offer improves retention and contribution after cannibalisation. Prioritise carrier remediation over more tracking features if feeds cannot give timely warnings or operational data show missed service is the dominant preventable loss. Invest in managers if a controlled offer to a defined segment yields incremental retained margin above staffing cost and outperforms digital service improvements.

Findings

  1. Prioritise tracking and proactive delay alerts. The strongest demand is to know what is happening after booking, without having to chase a haulier. Protect the speed of booking while fixing the experience that follows it.

    High confidenceBase 521

    Of 521 respondents choosing up to two improvements, 41.3% chose live tracking, 34.2% chose delay alerts and 65.6% chose at least one, counting overlap only once. The combined share was 65.8% of small, 63.0% of medium-volume and 72.9% of large accounts; it was also 58.8% even among promoters. By comparison, 23.0% selected quicker support, 22.6% lower fees and 14.2% a named account manager.

  2. Treat missing information and actual lateness as related but distinct failures. An alert can spare a shipper a wasted afternoon of waiting; it cannot make a missed collection on time.

    Medium confidenceBase 45

    31 of the 45 detractors who explained their score mentioned poor tracking or a late or missed collection or delivery. Q040 described a haulier arriving after warehouse closure with no warning; Q195 learned about missed collections only by chasing. In the closed questions, 48.4% of 120 detractors were dissatisfied with visibility and 44.1% with punctuality, versus 8.0% on each among 187 promoters. This identifies problems to test, not the NPS gain from solving them.

  3. Commercial director: there is a constituency for a subscription option, not a mandate to reverse the fee. Neither the timing of the score change nor the pricing answers establish that the fee caused the drop.

    Medium confidenceBase 521

    39.0% said the fee was worse, 20.0% better and 29.9% about the same; 7.5% had not used the old plan and 3.6% did not know. Worse was especially common below 20 shipments (49.2%, n=319), versus 25.3% at 20–99 (n=154) and 14.6% at 100+ (n=48). At the active-account volume mix, the worse share is about 33%, not 39%. Asked about an alternative rather than a replacement, 39.5% were likely to subscribe and 36.7% unlikely. Interest reached 61.6% among the 203 saying the fee was worse. The 21 fee/cost comments among 186 explanations describe both higher bills and harder budgeting.

  4. Chief executive: tell the board that recorded recommendation NPS was +24 in Q1 and +13 in Q3, not that customer loyalty fell by 11 points. Much of the headline gap is accounted for arithmetically by who answered.

    High confidenceBase 933

    Small accounts, the lowest-NPS band in both waves, made up 124/412 (30.1%) in Q1 and 319/521 (61.2%) in Q3. Giving Q3 the Q1 respondent mix raises Q3 to +20.3, leaving only about a three-point gap to Q1 before addressing mode and selection. Reported band scores moved +9 to +5, +26 to +21 and +38 to +38. At 100+ shipments the promoter shares, 48/98 and 25/48, are not significantly different on a two-sided 5% test; this is not proof of no change.

  5. Operations director: investigate the Republic of Ireland lane promptly, but do not diagnose a general Ireland service failure from this survey. These are businesses sending freight there, not necessarily businesses based there.

    Low confidenceBase 14

    13 of 14 senders were detractors, one was passive and none were promoters (NPS −93); among 496 non-senders, 21.2% were detractors (NPS +16). Of seven senders who gave reasons, five described late or missed service, one poor tracking and one pricing. Their scores and comments concern Quaymark overall, not a verified Ireland shipment.

  6. Do not roll out named account managers to the 20–99 shipment band on this evidence. If personal service is tested, distinguish high-volume accounts from mid-volume accounts and compare it against better shipment information.

    Medium confidenceBase 154

    In the 20–99 band, 15.6% selected a manager versus 63.0% selecting tracking or alerts (n=154). Interest in a manager was higher at 100+ (31.2%, n=48), but 72.9% there chose tracking or alerts. Shipment volume is not necessarily business size or account profitability.

Caveats

  • Q3 had 521 completes from 4,720 invited active accounts. It captures recent bookers who personally book and replied, not former customers who may have left after the fee change. The sample volume mix was 61.2% / 29.6% / 9.2%, versus 40% / 42% / 18% in the database. Weighting the within-band results gives NPS +17.8 and an estimated 21.2% detractors, or about 1,001 accounts, only if respondents represent their band. That is not an observed count.
  • Q1 account managers phoned selected customers (412 interviews from 640 called); Q3 invited all active accounts by email for self-completion. Selection, willingness to criticise a known interviewer and nonresponse can all differ. Q1 volume came from records, Q3 from respondents. Mix adjustment does not make these like-for-like waves, and recommendation intent is not retention or booking behaviour.
  • The ±4.1 percentage points is the conventional unweighted 95% simple-random-sample approximation for 187/521 promoters, without a finite-population correction. It is not an error bound for NPS or protection against nonresponse bias. The significance comparison likewise assumes independent samples.
  • Only 186 of 521 supplied text, including 45 of 120 detractors. Themes overlap; 68.9% counts each qualifying detractor comment once. Q162's terse cost complaint is included as a cost mention. Comments explain experiences, not their prevalence among all accounts.
  • Improvement choices were limited to two and did not specify implementation, price or trade-offs. Subscription intent at an unspecified price is not paid take-up. Associations with recommendation cannot establish which investment improves retention or margin.

Next steps

  • experimentPilot carrier event feeds, timestamped shipment status and late-collection/delivery alerts against the existing experience.
  • experimentCompare account invoices before and after March at comparable volumes, interview lapsed as well as active bookers, and trial a priced subscription alongside pay-per-shipment.
  • researchAudit Republic of Ireland consignments by carrier, route and service level, and follow up sender complaints against specific booking records.
  • decisionDefer manager recruitment; assess whether a limited named-contact trial is justified for high-value accounts with complex cases.
  • monitorEstablish a consistent survey series and pair it with retention and shipment-volume reporting.

Open-text themes it coded

Fast, simple quoting and booking 79Missing, stale or uninformative shipment tracking; having to chase for location or status 42Late or missed collections or deliveries 29Per-shipment fee, increased cost or preference for predictable subscription pricing 21Slow or hard-to-reach customer support 6Damaged freight or difficult claims 5Works adequately, no standout issue or strong view 17Too little experience to judge 5

The survey it planned

2 screening questions and 6 questions, as the model wrote them.

  1. S1

    Are you the main contact for your business’s bookings through Quaymark?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    Has your business booked at least one shipment through Quaymark in the past three months?

    One answer
    • Yes
    • No
    • Not sure

    Routing Shown only if S1 is: Yes

    Continues if Yes

  3. Q1

    On a scale from 0 to 10, how likely are you to recommend Quaymark to a colleague or business contact?

    Scale

    Scale 1-11: 0 — Not at all likely to 10 — Extremely likely

  4. Q2

    What is the main reason for the score you gave?

    Open text
  5. Q3

    If Quaymark could make just one of the following service changes, which, if any, would help your business most?

    One answer
    • Show more up-to-date shipment status information when I check online
    • Notify me when a shipment is expected to be delayed, without my having to check
    • Give my business a named account manager to contact for help
    • An equal benefit from two or more of these
    • None of these would help my business
    • Not sure
  6. Q4

    In March 2026, Quaymark changed from a monthly subscription to a booking fee per shipment. The subscription started at £25 a month for the smallest accounts. The booking fee is £3.50 per shipment, with a lower fee for accounts booking 100 or more shipments a month. Which arrangement would your business prefer?

    One answer
    • Definitely prefer the monthly subscription
    • Somewhat prefer the monthly subscription
    • No preference between the two
    • Somewhat prefer the per-shipment booking fee
    • Definitely prefer the per-shipment booking fee
    • Not enough information to say
  7. Q5

    What matters most to your business when comparing these two ways of paying for Quaymark?

    Open text
  8. Q6

    Thinking about both pricing and service, which one action would you most like Quaymark to take next?

    One answer
    • Bring back the monthly subscription instead of the per-shipment fee
    • Improve online shipment tracking and send alerts about expected delays
    • Provide named account managers for mid-sized customers (20 to 99 shipments a month)
    • Take a different action
    • Keep pricing and service as they are
    • Not sure
Sample plan

Email a unique online link to the main booking contact for each of the 4,720 active UK business accounts; accept one eligible response per account. Keep the invitation neutral about pricing. Before fieldwork, link booking records to the survey by account ID and assign a size band using average monthly Quaymark bookings in the last three months (three-month total divided by three, without rounding): under 20, 20 to under 100, or 100 or more. Record the eligible population count N in each band. Aim for about 520 completes overall (11% of invitations); do not close a band once a target is reached. Monitor returns by band and prioritise reminders to nonrespondents in underrepresented bands. The board’s precision target applies separately to each band, not to the pooled 520. As a planning benchmark, under simple random response within a band, a worst-case 50% share needs about 385 completes before finite-population correction; for a band of N accounts target at least ceiling(N × 384.16 / (N − 1 + 384.16)). Compare the sum of the three targets with 520 before launch. If the targets exceed the budget or achievable response, seek more completes or agree wider band-specific margins; do not promise ±5 points. Inviting every account does not remove voluntary-response bias. Pilot the mobile and desktop questionnaire to confirm completion within six minutes; keep Q2 and Q5 optional. Rotate the substantive action options in Q3 and Q6, keeping the other responses last.