Benchmarks / SurveyBench / Why Quaymark's customer score fell / GLM 5.3 FlashX Legal

Measured by Spring Prompt

GLM 5.3 FlashX Legal: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict
✗ Not sound
Research score
66 of 100
Analysis rating
997
Head to head, this task
won 9 of 14

Why it is not sound

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: 45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • Missed a trap: 521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.

Findings the judges found unsupported

  • Customers do not, in the main, want the subscription back as their first ask. A sizeable minority would take a subscription if offered, concentrated among small shippers, but the top requested improvement is tracking and delay alerts. Reversing the per-shipment fee across the board is not what the majority are asking for.Saying respondents 'would take' or 'would choose' a subscription turns hypothetical, unpriced stated intentions into expected behaviour. The survey establishes reported likelihood, not actual take-up.
  • Small shippers (fewer than 20 shipments a month) are the least satisfied and the most fee-sensitive, but even they rank tracking and alerts above cheaper fees. NPS in this band fell from +9 to +5 after the fee change. 49.2% say the fee is worse than the subscription and 44.5% would choose a subscription, yet only 24.8% pick lower fees as a top-two improvement while 65.8% pick tracking or alerts. The commercial case for a targeted subscription is in this segment; the case for a blanket reversal is not.The claim that 44.5% of small accounts 'would choose' a subscription overstates hypothetical stated likelihood. A targeted subscription may warrant testing, but its commercial case is not established without pricing, actual take-up and retention evidence.
  • Large accounts (100+ shipments a month) are the most satisfied, have not deteriorated, mostly prefer the per-shipment fee, and show no significant change in promoters. The apparent Q1-to-Q3 movement in the 9-10 share (49.0% to 52.1%) is not statistically significant. They are also the only group where a named account manager is a mainstream ask (31.2%). Do not reverse pricing for this group on this evidence; if account managers are commissioned anywhere, it is here, not in the mid-sized band.Large accounts do not 'mostly prefer' the per-shipment fee: 31.2% say it is better, while 41.7% say it is about the same. A majority being non-negative is not a majority preferring it.
  • Invest in live tracking and proactive delay alerts as the primary product response. This is the only improvement that is the top ask in every size band, among detractors, passives and promoters, and it dominates the reasons given for low scores. Dissatisfaction with shipment visibility (28.6% dissatisfied) is worse than dissatisfaction with punctuality (22.4%). Among detractors who left a comment, 68.9% mention tracking or late/missed collections or deliveries.The evidence attaches n=521 to the 69.2% tracking-or-alerts result among detractors. That percentage has a base of 120 detractors, not all 521 respondents.
  • Weighted to the active-account base, roughly 1,001 of 4,720 accounts would score 0-6. That is the commercial risk pool. It is concentrated in the small-shipper majority (40% of the base at 26.0% detractors), which is also the group most exposed to the per-shipment fee. Mid-sized accounts are the next largest contributor by volume.Size-weighting alone does not establish that approximately 1,001 active accounts would be detractors, given the roughly 11% response rate and unaddressed non-response bias. Nor does a detractor classification establish a behavioural commercial-risk pool. Small accounts are 40% of active accounts, not a majority, and the 20–99 band is larger at 42%.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?✓12.912.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?✓17.717.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?✓20.320.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?✓68.9%68.9%
How many of the open-text answers mention the per-shipment fee or its cost?✓2121
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?✓4.1 points4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What percentage of the accounts invited to the Q3 2026 wave completed it?✓11%11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?✓49.2%49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?✓4 points3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?✓1313
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?✓1,0011,001

Traps in the data 6 of 8 handled

  • ✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
  • ✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
  • ✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
  • ✓Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
  • ✗45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • ✗521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
  • ✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
  • ✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Findings

  1. You cannot tell the board that customer loyalty fell by 11 points between Q1 and Q3. The two waves are not like-for-like: Q1 was phone interviews by account managers (n=412) with a mix of 30% small / 46% mid / 24% large accounts; Q3 was online self-complete (n=521) with 61% small / 30% mid / 9% large. Unweighted NPS moved from +24 to +13 (35.9% promoters minus 23.0% detractors). Re-weighting Q3 to Q1's account mix lifts NPS to +20, so most of the headline 11-point gap is composition, not sentiment. The remaining ~4 points are confounded by mode (account managers interviewing their own customers inflates scores). NPS also measures willingness to recommend, not loyalty. Any board statement of an 11-point loyalty fall would be misleading.

    High confidenceBase 521

    Q3 unweighted NPS +13 (35.9% 9-10, 23.0% 0-6, n=521). Q1 unweighted NPS +24 (40.5% 9-10, 17.0% 0-6, n=412). Q3 re-weighted to Q1 mix: +20.3. Q3 weighted to the 4,720-account mix (40/42/18): +17.7. Q1 band mix 124/190/98 vs Q3 319/154/48 vs population 40/42/18.

  2. Customers do not, in the main, want the subscription back as their first ask. A sizeable minority would take a subscription if offered, concentrated among small shippers, but the top requested improvement is tracking and delay alerts. Reversing the per-shipment fee across the board is not what the majority are asking for.

    High confidenceBase 521

    Q6: 39.5% likely to choose a subscription (n=521). Q5: 39.0% say the fee is worse, 20.0% better, 29.9% the same. Q7: only 22.6% pick lower booking fees as a top-two improvement, vs 65.6% picking live tracking or delay alerts. Among those who say the fee is worse, 61.6% would choose a subscription (n=203).

  3. Small shippers (fewer than 20 shipments a month) are the least satisfied and the most fee-sensitive, but even they rank tracking and alerts above cheaper fees. NPS in this band fell from +9 to +5 after the fee change. 49.2% say the fee is worse than the subscription and 44.5% would choose a subscription, yet only 24.8% pick lower fees as a top-two improvement while 65.8% pick tracking or alerts. The commercial case for a targeted subscription is in this segment; the case for a blanket reversal is not.

    High confidenceBase 319

    Q1 NPS <20/month: +9 (33.1% promoters, 24.2% detractors, n=124). Q3 NPS <20/month: +5 (31.0% promoters, 26.0% detractors, n=319). Q5 worse: 49.2% (n=319). Q6 net likely: 44.5%. Q7 lower fees 24.8% vs tracking/alerts 65.8%.

  4. Large accounts (100+ shipments a month) are the most satisfied, have not deteriorated, mostly prefer the per-shipment fee, and show no significant change in promoters. The apparent Q1-to-Q3 movement in the 9-10 share (49.0% to 52.1%) is not statistically significant. They are also the only group where a named account manager is a mainstream ask (31.2%). Do not reverse pricing for this group on this evidence; if account managers are commissioned anywhere, it is here, not in the mid-sized band.

    Medium confidenceBase 48

    Q3 100+ NPS +38 (52.1% promoters, 14.6% detractors, n=48). Q1 100+ NPS +38 (49.0% promoters, 11.2% detractors, n=98). Two-proportion z on promoters: z=0.35, not significant at 95%. Q5: 31.2% better, 14.6% worse. Q7 account manager 31.2% vs 14.2% overall.

  5. There is no reliable evidence of a distinct Ireland problem. The -93 NPS rests on 14 respondents, 13 of whom scored 0-6. That interval is extremely wide (roughly 69%-99% detractors at 95% confidence). Open-text from Ireland-shippers raises late deliveries, missed collections and the fee — the same issues as UK accounts — with no mention of customs, routes or Ireland-specific failures. Treat the Ireland signal as a lead to investigate, not a finding to act on operationally.

    Low confidenceBase 14

    Q1 Ireland column: base 14, 92.9% scored 0-6, 7.1% scored 7-8, 0.0% scored 9-10, NPS -93. Q9: 2.7% of 521 ship to Ireland (14 respondents). Ireland open-ends (Q033, Q115, Q150, Q295, Q296, Q339) cite lateness, missed collections and fees.

  6. Invest in live tracking and proactive delay alerts as the primary product response. This is the only improvement that is the top ask in every size band, among detractors, passives and promoters, and it dominates the reasons given for low scores. Dissatisfaction with shipment visibility (28.6% dissatisfied) is worse than dissatisfaction with punctuality (22.4%). Among detractors who left a comment, 68.9% mention tracking or late/missed collections or deliveries.

    High confidenceBase 521

    Q7 net tracking or alerts: 65.6% overall; 65.8% / 63.0% / 72.9% by size band; 69.2% of detractors (n=521). Q4 dissatisfied: 28.6%. Q3 dissatisfied: 22.4%. Detractor open-ends mentioning tracking or late/missed: 31 of 45 (68.9%).

  7. Fast booking is the product's proven strength and the main reason promoters score 9-10. Do not disrupt the booking flow to fund or package the other investments. Protect the two-minute quote-and-book experience as a constraint on any roadmap.

    High confidenceBase 521

    Open-text repeatedly cites two-minute booking as the reason for high scores (e.g. Q014, Q023, Q024, Q043, Q110, Q120, Q156, Q200, Q257, Q297, Q340, Q403, Q432, Q503). Q7 'none of these' is highest among promoters (19.8%).

  8. Account managers for mid-sized customers are not supported by this survey. Only 15.6% of 20-99 shippers pick a named account manager as a top-two improvement, barely above the 14.2% total and far below tracking (40.3%) and alerts (35.1%) in that band. The ask is concentrated in the 100+ band.

    High confidenceBase 154

    Q7 named account manager: 11.0% (<20, n=319), 15.6% (20-99, n=154), 31.2% (100+, n=48), 14.2% total (n=521).

  9. The fee change has created a real but segmented commercial issue, not a universal backlash. Mid-sized shippers sit between the two poles (25.3% worse, NPS +21 vs +26 in Q1). Support complaints and damage claims are secondary themes (23.0% want faster support; 13.6% want easier claims) and should not displace tracking/alerts, but they are part of the detractor story.

    Medium confidenceBase 521

    Q5 worse: 49.2% / 25.3% / 14.6% by size band. Q7 support 23.0%, claims 13.6%. Open-text fee mentions: 21 of 186. Support mentions: Q063, Q086, Q108, Q275, Q292, Q354. Claims: Q050, Q455, Q486, Q491, Q514.

  10. Weighted to the active-account base, roughly 1,001 of 4,720 accounts would score 0-6. That is the commercial risk pool. It is concentrated in the small-shipper majority (40% of the base at 26.0% detractors), which is also the group most exposed to the per-shipment fee. Mid-sized accounts are the next largest contributor by volume.

    Medium confidenceBase 4720

    Population 4,720 accounts: 40% <20, 42% 20-99, 18% 100+. Q3 detractor rates 26.0% / 19.5% / 14.6%. Weighted rate 21.2%; 4,720 x 0.212 = 1,001 accounts.

Open-text themes it coded

Fast and easy booking 84Poor tracking and shipment visibility 42Late or missed collections and deliveries 29Per-shipment fee and pricing 29Customer support responsiveness 11Damage claims process 6Named account manager 14

The survey it planned

4 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you the main person at your business who books collections with Quaymark?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Has your business booked at least one shipment through Quaymark in the last three months?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Is the business you book for based in the United Kingdom?

    One answer
    • Yes
    • No

    Continues if Yes

  4. S4

    On average, how many shipments does your business book with Quaymark each month?

    One answer
    • Fewer than 20 shipments a month
    • 20 to 99 shipments a month
    • 100 or more shipments a month
    • Not sure / it varies a lot
  5. Q1

    How likely is it that you would recommend Quaymark to another business?

    Scale

    Scale 1-11: 0 - Not at all likely to 10 - Extremely likely

  6. Q2

    What is the main reason for the score you gave Quaymark? A few words is enough.

    Open text
  7. Q3

    For quality checking, please select 'Fairly dissatisfied' from the list below.

    One answer
    • Very satisfied
    • Fairly satisfied
    • Neither satisfied nor dissatisfied
    • Fairly dissatisfied
    • Very dissatisfied
  8. Q4

    In March 2026 Quaymark changed from a monthly subscription to a per-shipment booking fee of £3.50 (less for accounts booking 100 or more a month). Overall, how has this change affected how you feel about Quaymark?

    One answer
    • Much more positive
    • A little more positive
    • No real change
    • A little more negative
    • Much more negative
    • I was not a customer before the change
    • Don't know
  9. Q5

    If you were choosing for a business like yours, which pricing would you prefer?

    One answer
    • The current per-shipment booking fee
    • The previous monthly subscription (from £25 a month for the smallest accounts)
    • No preference
    • Don't know
  10. Q6

    Since March 2026, which of these, if any, have you done?

    Any that apply
    • Booked fewer shipments with Quaymark
    • Used other freight providers for some shipments I would previously have booked with Quaymark
    • Considered moving most or all of my bookings away from Quaymark
    • None of the above — I have not changed how I use Quaymark
    • I only started using Quaymark after March 2026
    • Don't know
  11. Q7

    Please rank these possible improvements from most valuable to least valuable for your business. Put the most valuable at the top.

    Ranking
    • Better online shipment tracking
    • Automatic alerts when a shipment is delayed
    • A named account manager for your business
    • Lower per-shipment booking fees
  12. Q8

    How satisfied are you with Quaymark's online shipment tracking?

    One answer
    • Very satisfied
    • Fairly satisfied
    • Neither satisfied nor dissatisfied
    • Fairly dissatisfied
    • Very dissatisfied
    • I have not used Quaymark tracking
  13. Q9

    If any of your Quaymark shipments in the last three months were delayed, how well did Quaymark keep you informed?

    One answer
    • I had no delayed shipments
    • Very well informed
    • Fairly well informed
    • Neither well nor poorly informed
    • Fairly poorly informed
    • Very poorly informed
    • Don't know / can't remember
  14. Q10

    Quaymark is considering a named account manager: one dedicated person you could contact for help with bookings, problems and advice. How useful would this be for your business?

    One answer
    • Essential
    • Quite useful
    • Not very useful
    • Not at all useful
    • I already have a named contact and that is enough
    • Don't know
Sample plan

Invite the main booking contact of all 4,720 active UK accounts by email from the customer database. Send from the research team (not the commercial team), with a neutral invitation, a reminder after 4 days and a final reminder after 10 days. Offer a prize draw or a topline results summary to support response. Randomise option order on Q6 and the ranking items in Q7. Set hard quotas on S4 (fewer than 20; 20 to 99; 100 or more shipments a month). The board's precision requirement (share scoring 9-10 within ±5 percentage points at 95% confidence in each account-size band) needs about 385 completes per band (n = 1.96² × 0.25 / 0.05², p=0.5 worst case), so about 1,155 completes in total. About 520 completes would give only roughly ±7pp per band if the three bands were equal in size, and would not meet the stated precision. Fieldwork will therefore target a minimum of 385 completes in each band (about 24% response from 4,720, achievable with three contacts and a sub-6-minute questionnaire). If the 100-plus population in the database is smaller than 385, take a census of that band and report the achievable precision. Use database monthly volume as a fallback for respondents selecting 'Not sure' on S4. Do not weight within-band 9-10 shares; weight any overall NPS back to the account-size mix of the 4,720 active accounts. Exclude duplicates, test accounts and failed attention checks (Q3).