Benchmarks / SurveyBench / Why Quaymark's customer score fell / Mistral Large 4

Measured by Spring Prompt

Mistral Large 4: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict
✗ Not sound
Research score
84 of 100
Analysis rating
815
Head to head, this task
won 9 of 15

Why it is not sound

The survey

  • A research objective the survey could not answer
  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: 521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.

Findings the judges found unsupported

  • The headline 11-point NPS drop is an artefact of changing survey mode (phone to online) and sample mix; the like-for-like fall is only ~4 points when Q3 is reweighted to the Q1 respondent mix. Do not tell the board loyalty fell 11 points.Calling the entire drop an artefact is more definite than the evidence permits. Reweighting establishes a substantial composition effect, but the method effect has not been measured. The remaining roughly four-point difference is size-adjusted, not fully like-for-like.
  • The per-shipment fee is driving dissatisfaction among small shippers, who are the largest segment (40% of accounts). Nearly half (49.2%) of <20/month shippers say the fee is worse than the subscription, and their NPS fell 4 points between waves. 44.5% of small shippers would likely choose a subscription if offered.The fee is a documented complaint among small accounts, but these data do not establish that it caused their NPS decline. Small accounts are also not the largest population segment: they represent 40%, compared with 42% for accounts booking 20–99 shipments.
  • Tracking and delay alerts are the single highest-priority improvement. Two-thirds (65.6%) of all customers selected live tracking or delay alerts as a top-two improvement, and 69.2% of detractors did so. Open-text answers are dominated by complaints about not knowing where shipments are and late deliveries with no warning.The 65.6% figure describes survey respondents, not all customers. The sample is unweighted, has a substantially skewed size mix, and covers only about 11% of invited accounts.
  • Booking speed is a core strength to protect. Over one-third (67 of 186) of open-text answers praise the 2-minute booking process. Any changes to pricing or service must not degrade this experience.The stated count of 67 fast/easy-booking comments is not supported by the supplied answers: 79 mention booking speed or ease, including mixed positive and negative comments. The separate theme count of 72 is also inconsistent with those answers.
  • Ireland shippers are in crisis: 13 of 14 are detractors (NPS -93). The base is tiny so this could be noise, but the uniformity is alarming and warrants immediate investigation before any strategic decision.Calls Ireland shippers 'in crisis' and 'alarming' on a base of 14. It uses the uniformity of answers to undercut its own small-base warning, and says this must be investigated before any strategic decision, which could stall action.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?✓1312.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?✓17.717.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?✓20.320.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?✓68.9%68.9%
How many of the open-text answers mention the per-shipment fee or its cost?✓2121
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?✓4.1 points4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What percentage of the accounts invited to the Q3 2026 wave completed it?✓11%11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?✓49.2%49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?✓4 points3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?✓1313
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?✓1,0011,001

Traps in the data 7 of 8 handled

  • ✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
  • ✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
  • ✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
  • ✓Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
  • ✓45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • ✗521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
  • ✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
  • ✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Prioritise investment in live shipment tracking and proactive delay alerts. This addresses the top improvement priority for 65.6% of customers and the leading cause of detractor sentiment. Do not fully reverse the per-shipment fee; instead, offer a monthly subscription as an optional alternative for small and mid-sized shippers (39.5% overall would likely choose it, rising to 44.5% among small shippers and 61.6% among those who say the fee is worse). Defer account managers for mid-sized customers (only 14.2% want them) but consider a pilot for the 100+ segment (31.2% want them). Immediately commission qualitative research into the Ireland market given the extreme dissatisfaction (13 of 14 detractors).

What would change it

Transactional data showing subscription pricing would materially improve retention or revenue among small shippers; a conjoint study showing customers value lower fees over tracking; qualitative research showing Ireland issues are not systemic; churn data linking tracking dissatisfaction to actual account loss.

Findings

  1. The headline 11-point NPS drop is an artefact of changing survey mode (phone to online) and sample mix; the like-for-like fall is only ~4 points when Q3 is reweighted to the Q1 respondent mix. Do not tell the board loyalty fell 11 points.

    High confidenceBase 933

    Q1 phone NPS +24 (n=412); Q3 online NPS +13 unweighted (n=521) but +20.3 when reweighted to Q1's segment mix (30% small, 46% medium, 24% large); Q3 weighted to population mix (40/42/18) is +17.7.

  2. The per-shipment fee is driving dissatisfaction among small shippers, who are the largest segment (40% of accounts). Nearly half (49.2%) of <20/month shippers say the fee is worse than the subscription, and their NPS fell 4 points between waves. 44.5% of small shippers would likely choose a subscription if offered.

    High confidenceBase 319

    Q5: 49.2% of <20/month say fee is worse vs 14.6% of 100+ shippers; Q6: 44.5% of <20/month likely to choose subscription (61.6% among those who say fee is worse); Q1 Q1 vs Q3 tables: NPS for <20/month segment fell from +9 to +5.

  3. Tracking and delay alerts are the single highest-priority improvement. Two-thirds (65.6%) of all customers selected live tracking or delay alerts as a top-two improvement, and 69.2% of detractors did so. Open-text answers are dominated by complaints about not knowing where shipments are and late deliveries with no warning.

    High confidenceBase 521

    Q7: 41.3% live tracking, 34.2% delay alerts, net 65.6% (72.9% among 100+ shippers); Q4: 28.6% dissatisfied with tracking; 41 of 186 open-text answers mention tracking/visibility failures and 29 mention late/missed collections.

  4. Ireland shippers are in crisis: 13 of 14 are detractors (NPS -93). The base is tiny so this could be noise, but the uniformity is alarming and warrants immediate investigation before any strategic decision.

    Low confidenceBase 14

    Q1 Q3 table, Ships to Ireland column: base 14, 92.9% detractors (0-6), 7.1% passives, 0% promoters; NPS -93.

  5. Booking speed is a core strength to protect. Over one-third (67 of 186) of open-text answers praise the 2-minute booking process. Any changes to pricing or service must not degrade this experience.

    High confidenceBase 186

    Open-text Q2: 67 respondents explicitly praise fast/easy booking (e.g., 'two minutes', 'couple of clicks', 'before my coffee's cooled down').

  6. Large shippers (100+/month) are stable and satisfied (NPS +38 unchanged between waves). They are the only segment where account managers have meaningful demand (31.2% want one), but overall demand for account managers is low (14.2%).

    Medium confidenceBase 146

    Q1 and Q3 tables, 100+ column: NPS +38 both waves; Q7: 31.2% of 100+ want named account manager vs 14.2% overall; Q5: only 14.6% of 100+ say fee is worse.

Caveats

  • Mode effect: Q1 was telephone interviews conducted by account managers; Q3 was online. Phone surveys typically yield satisfaction scores 5-10 points higher due to social desirability bias and rapport. The raw 11-point NPS drop is not like-for-like.
  • Sample mix: The Q3 respondent mix (61% small, 30% medium, 9% large) differs from the population (40%, 42%, 18%) and from Q1 (30%, 46%, 24%). Unweighted comparisons are misleading.
  • Ireland base is only 14 respondents; the -93 NPS has a massive margin of error but the direction is consistent across all 14.
  • Open-text answers are self-selected (186 of 521, 36%) and likely over-represent dissatisfied customers.
  • 7.5% of Q5 respondents joined after March 2026 and cannot compare the fee to the subscription.
  • The survey measures stated preference, not revealed behaviour; customers may say they want a subscription but not switch when offered.

Next steps

  • decisionApprove funding for live tracking and delay alert features
  • experimentDesign and run a subscription-option trial for small shippers (<20/month)
  • researchConduct depth interviews with all 14 Ireland shippers and 20 matched non-Ireland detractors
  • monitorMove to monthly online survey waves with consistent methodology

Open-text themes it coded

Booking speed and ease (positive) 72Tracking and visibility failures 41Late or missed collections and deliveries 29Per-shipment fee cost concerns 21Poor customer support and slow responses 6Damage to goods and slow claims process 5

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you the main person responsible for booking freight shipments for your business on Quaymark?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Approximately how many shipments has your business booked through Quaymark in the last three months?

    One answer
    • None
    • 1 to 19 shipments
    • 20 to 99 shipments
    • 100 or more shipments

    Continues if 1 to 19 shipments; 20 to 99 shipments; 100 or more shipments

  3. Q1

    How likely are you to recommend Quaymark to a colleague or business associate? Please use a scale of 0 to 10, where 0 means 'Not at all likely' and 10 means 'Extremely likely'.

    Scale

    Scale 1-11: 0 - Not at all likely to 10 - Extremely likely

  4. Q2

    What is the main reason for the score you just gave? Please be as specific as possible.

    Open text
  5. Q3

    Before March 2026, Quaymark charged a monthly subscription. Since March 2026, you pay a booking fee of £3.50 per shipment (reduced for accounts booking 100 or more shipments a month). Which pricing approach do you prefer for your business?

    One answer
    • The monthly subscription
    • The per-shipment booking fee
    • No preference
    • It depends on how many shipments I book
  6. Q4

    How has the change from a monthly subscription to a per-shipment booking fee affected your overall shipping costs?

    Scale

    Scale 1-5: Much higher costs to Much lower costs

    Judges: assumptive

    Assumes the business experienced the change from subscription pricing. Eligible accounts may have joined after March 2026 and have no before-and-after cost comparison; no not-applicable option is provided.

    It assumes the respondent was a customer under the subscription before March 2026. Accounts that joined since then have no 'before' to compare, and there is no 'not applicable' option.

  7. Q5

    Since the pricing change in March 2026, has your business changed how much it uses Quaymark?

    One answer
    • I book significantly more shipments through Quaymark
    • I book slightly more shipments through Quaymark
    • No change in how many shipments I book
    • I book slightly fewer shipments through Quaymark
    • I book significantly fewer shipments through Quaymark
    • I have stopped using Quaymark
    Judges: assumptive

    Assumes the business used Quaymark before the pricing change and can compare its usage across that change. The screener also admits accounts that started using Quaymark afterwards, with no applicable response option.

    It assumes the business used Quaymark before the March 2026 pricing change. Newer accounts cannot report a change in usage, and there is no 'not applicable' option.

  8. Q6

    How important is real-time shipment tracking visibility to your business?

    Scale

    Scale 1-5: Not at all important to Extremely important

  9. Q7

    How important are proactive alerts when shipments are delayed to your business?

    Scale

    Scale 1-5: Not at all important to Extremely important

  10. Q8

    How important is having a dedicated account manager to your business?

    Scale

    Scale 1-5: Not at all important to Extremely important

  11. Q9

    Since March 2026, how would you rate the support you have received from Quaymark?

    Scale

    Scale 1-5: Very poor to Excellent

    Judges: assumptive

    Assumes every respondent has received support from Quaymark since March 2026. Booking a shipment does not establish that, and no not-applicable option is provided.

    Assumes the respondent has sought or received support from Quaymark.

  12. Q10

    Is there anything else Quaymark could do to improve your experience? Please share any suggestions.

    Open text
Sample plan

Email invitation to the main booking contact at each of the 4,720 active UK business accounts. Screener confirms respondent is the main booking contact and that the account has booked at least one shipment in the last three months. To achieve the required precision of ±5 percentage points at 95% confidence for the share of promoters (scores 9-10) within each account size group (<20, 20-99, 100+ shipments/month), approximately 385 completes per group are required (total ~1,155). We will implement soft quotas to ensure minimum 385 completes in the 'fewer than 20' and '20-99' groups. Given the likely smaller population of '100+' accounts, this segment may yield fewer completes (e.g., 200-300), resulting in a confidence interval of ±7-8pp rather than ±5pp; results for this group will be reported with appropriate caveats. Target total completes: 1,200 (or all valid responses if higher). Reminder emails will be sent to non-respondents after 7 days; if quotas remain unmet, we will extend fieldwork or use post-stratification weighting for overall estimates.