Benchmarks / SurveyBench / Why Quaymark's customer score fell / Mistral Medium 3.5

Measured by Spring Prompt

Mistral Medium 3.5: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict
✗ Not sound
Research score
62 of 100
Analysis rating
540
Head to head, this task
won 0 of 15

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
  • Missed a trap: The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
  • Missed a trap: 45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • Missed a trap: 521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
  • Proposed a mistaken next step: Report the fall to the board as a 11-point decline in customer loyalty

Findings the judges found unsupported

  • The per-shipment fee is a major driver of dissatisfaction, especially for smaller shippers (fewer than 20 shipments/month), where 49.2% say it is worse than the subscription. Reintroducing a subscription option would likely recover a significant portion of these detractors.Fee dissatisfaction among small accounts does not establish that the fee is a major driver of detractor sentiment or that reinstating a subscription would recover significant numbers of detractors. Hypothetical subscription intentions, with no price specified, do not support that behavioural forecast.
  • Tracking and delay alerts are the top unmet needs, with 65.6% of all respondents selecting live tracking or delay alerts as improvements. This is even higher among detractors (69.2%) and passives (69.6%).The reported 72.2% of detractor open-text answers mentioning tracking or delays is incorrect: the supported figure is approximately 69% of the 45 detractors who answered. The Q7 figures and the overall priority of tracking and alerts are supported.
  • The Ireland problem is severe but small in scale: only 2.7% of respondents ship to Ireland, but 92.9% of them are detractors (NPS -93). This is a niche but critical issue requiring urgent attention.The observed Ireland scores are correct, but 14 respondents cannot establish a severe, critical problem across Ireland customers requiring urgent action. The later small-base caveat does not support that strong conclusion.
  • Customer loyalty (NPS) fell by 11 points between Q1 and Q3 (from +24 to +13), driven primarily by the per-shipment fee and tracking issues. The drop is statistically significant at the 95% level for the total sample.The headline score difference does not establish an 11-point decline in customer loyalty or identify its causes. Sample composition and survey method changed. The significance justification also incorrectly uses the margin of error for one wave's promoter percentage instead of the uncertainty in the difference between two NPS estimates.
  • The per-shipment fee is costing Quaymark ~1,088 detractor accounts (weighted to the active account base), with smaller shippers (<20 shipments/month) contributing the majority of this loss.The displayed weighted calculation gives approximately 1,002 detractor accounts, not 1,088. More importantly, detractor status is neither lost business nor evidence that the fee caused it; extrapolation to all active accounts also ignores potential non-response bias.

Scorecard

Numbers 8 of 12 right

QuestionIts answerTrue
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?✓1312.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?✗10.817.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?✗11.220.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?✓72.2%68.9%
How many of the open-text answers mention the per-shipment fee or its cost?✗2821
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?✓4.2 points4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What percentage of the accounts invited to the Q3 2026 wave completed it?✓11.04%11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?✓49.2%49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?✓4 points3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?✓1313
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?✗1,0881,001

Traps in the data 4 of 8 handled

  • ✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
  • ✗Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
  • ✗The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
  • ✓Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
  • ✗45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • ✗521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
  • ✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
  • ✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Prioritise reinstating the subscription option alongside the per-shipment fee, as this directly addresses the primary driver of dissatisfaction (especially for smaller shippers) and could recover a significant portion of detractors. Simultaneously, invest in improving shipment tracking and proactive delay alerts, as these are the top unmet needs across all segments and are strongly linked to detractor sentiment. Address the Ireland issue urgently, as it is a severe but niche problem that could escalate if left unchecked. Account managers are a lower priority and should not be a focus for immediate investment.

What would change it

Evidence that the per-shipment fee is not the primary driver of the NPS drop (e.g., if tracking/delay issues are shown to be the sole cause via controlled experiments or deeper qualitative research) would change the recommendation. Similarly, if the Ireland problem is found to be a one-off issue (e.g., a single haulier) rather than systemic, it may not require a strategic response.

Findings

  1. The per-shipment fee is a major driver of dissatisfaction, especially for smaller shippers (fewer than 20 shipments/month), where 49.2% say it is worse than the subscription. Reintroducing a subscription option would likely recover a significant portion of these detractors.

    High confidenceBase 319

    Q5: 49.2% of <20 shipments/month say per-shipment fee is worse; Q6: 44.5% of <20 shipments/month are fairly/very likely to choose a subscription; NPS for <20 shipments/month fell from +9 (Q1) to +5 (Q3).

  2. Tracking and delay alerts are the top unmet needs, with 65.6% of all respondents selecting live tracking or delay alerts as improvements. This is even higher among detractors (69.2%) and passives (69.6%).

    High confidenceBase 521

    Q7: 65.6% selected live tracking or delay alerts; open-text: 72.2% of detractor answers mention tracking or late/missed collections/deliveries.

  3. The Ireland problem is severe but small in scale: only 2.7% of respondents ship to Ireland, but 92.9% of them are detractors (NPS -93). This is a niche but critical issue requiring urgent attention.

    High confidenceBase 14

    Q9: 2.7% ship to Ireland; Q1: 92.9% of Ireland shippers score 0-6; NPS for Ireland shippers is -93.

  4. Customer loyalty (NPS) fell by 11 points between Q1 and Q3 (from +24 to +13), driven primarily by the per-shipment fee and tracking issues. The drop is statistically significant at the 95% level for the total sample.

    High confidenceBase 521

    NPS Q1: +24; NPS Q3: +13; margin of error for Q3 promoters (35.9%) is ±4.2pp, so the 11-point drop is significant.

  5. For high-volume shippers (100+ shipments/month), NPS is unchanged (+38 in both Q1 and Q3), and the per-shipment fee is seen as better or about the same by 72.9%. This group is not a priority for fee-related changes.

    High confidenceBase 48

    Q1: NPS +38 for 100+; Q3: NPS +38 for 100+; Q5: 31.2% say per-shipment fee is better, 41.7% about the same.

  6. Account managers are a lower priority: only 14.2% of respondents selected this as an improvement, and it is not a top theme in open-text answers.

    Medium confidenceBase 521

    Q7: 14.2% selected named account manager; open-text: no strong theme around account management.

  7. The per-shipment fee is costing Quaymark ~1,088 detractor accounts (weighted to the active account base), with smaller shippers (<20 shipments/month) contributing the majority of this loss.

    Medium confidenceBase 521

    Q1: 23% detractors (0-6) in Q3; weighted to active accounts (40% <20, 42% 20-99, 18% 100+), detractor accounts = 4,720 * (0.4*0.26 + 0.42*0.195 + 0.18*0.146) ≈ 1,088.

Caveats

  • The Q1 2026 wave was conducted by phone with account managers, which may introduce bias (e.g., respondents may have been more positive due to the personal touch). The Q3 2026 wave was online, which may have captured more candid feedback.
  • The Ireland sample is very small (n=14), so findings for this group should be treated with caution despite the extreme NPS score.
  • Open-text answers are from 186 respondents (35.7% of completes), so themes may not fully represent all detractors or passives.
  • The per-shipment fee was introduced in March 2026, so Q1 2026 data reflects the subscription model, while Q3 2026 reflects the fee model. The comparison is valid but may not account for other changes (e.g., haulier performance) between waves.
  • Weighted NPS calculations assume the shipment-a-month mix of the active account base is accurate and representative.

Next steps

  • decisionReintroduce the subscription option as an alternative to the per-shipment fee, with clear messaging on cost savings for frequent shippers.
  • decisionInvest in live tracking and proactive delay alerts, with a focus on real-time updates and clear communication for missed/late collections or deliveries.
  • researchConduct a root-cause analysis of the Ireland problem, including interviews with Ireland shippers and hauliers, to identify and resolve systemic issues.
  • experimentRun a controlled experiment (A/B test) with a subset of small shippers, offering the subscription option to half and monitoring NPS, retention, and revenue impact.
  • monitorMonitor NPS and satisfaction scores for tracking and punctuality monthly, with a focus on small shippers (<20 shipments/month) and Ireland shippers.
  • researchConduct follow-up qualitative research (e.g., interviews or focus groups) with detractors and passives to explore the interplay between fee dissatisfaction and tracking issues.

Open-text themes it coded

Per-shipment fee cost or preference for subscription 25Tracking or visibility issues 50Late or missed collections/deliveries 58Booking speed or ease 107Customer support or claims issues 51

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you the main contact for booking shipments with Quaymark at your business?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Has your business booked at least one shipment through Quaymark in the last three months?

    One answer
    • Yes
    • No

    Continues if Yes

  3. Q1

    On a scale of 0 to 10, how likely are you to recommend Quaymark to a friend or colleague?

    Scale

    Scale 1-11: 0 (Not at all likely) to 10 (Extremely likely)

  4. Q2

    What is the main reason for your score?

    Open text
  5. Q3

    Since the change from a monthly subscription to a per-shipment fee, how do you feel about the new pricing model?

    One answer
    • I strongly prefer the per-shipment fee
    • I slightly prefer the per-shipment fee
    • I am neutral
    • I slightly prefer the monthly subscription
    • I strongly prefer the monthly subscription
    Judges: assumptive

    Eligible contacts may have joined after the pricing change and lack knowledge of the subscription. There is no option for being unable to compare; neutrality is not the same as lack of knowledge.

    Assumes the respondent was a customer under the monthly subscription. Accounts that joined after the change have no basis for comparison, and there is no 'not applicable' or 'don't know' option.

  6. Q4

    Which of the following improvements would most increase your satisfaction with Quaymark? (Select up to 2)

    Any that apply
    • Bring back the monthly subscription instead of the per-shipment fee
    • Improve shipment tracking (e.g., real-time updates, more detailed information)
    • Add proactive delay alerts (e.g., notifications if a shipment is delayed)
    • Assign a dedicated account manager for mid-sized customers
    • None of these
  7. Q5

    How many shipments does your business typically book with Quaymark each month?

    One answer
    • Fewer than 20
    • 20 to 99
    • 100 or more
  8. Q6

    How satisfied are you with the current per-shipment fee of £3.50 (reduced for accounts booking 100 or more a month)?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  9. Q7

    Compared to the previous monthly subscription, do you feel the per-shipment fee is:

    One answer
    • More cost-effective for my business
    • Less cost-effective for my business
    • About the same in cost
    • I don’t know
  10. Q8

    How important is it to you that Quaymark offers the following? (Rate each on a scale of 1 to 5, where 1 is Not at all important and 5 is Extremely important)

    Scale
    • Real-time shipment tracking
    • Proactive delay alerts
    • Dedicated account manager

    Scale 1-5: Not at all important to Extremely important

  11. Q9

    What do you like most about Quaymark?

    Open text
  12. Q10

    What is the one thing Quaymark could do to improve your experience?

    Open text
Sample plan

Invite the main booking contact of all 4,720 active UK business accounts via email. Aim for 520 completes, with no quotas but ensuring representation across account sizes (fewer than 20, 20 to 99, and 100 or more shipments a month). The sample size of 520 will achieve the required precision of ±5 percentage points at 95% confidence for each account size subgroup.