Benchmarks / SurveyBench / Why Quaymark's customer score fell / Gemini 4 Argon

Measured by Spring Prompt

Gemini 4 Argon: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict
✗ Not sound
Research score
84 of 100
Analysis rating
985
Head to head, this task
won 14 of 32

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A double-barrelled question: Q3
  • A double-barrelled question: Q4

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Missed a trap: 521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.

Findings the judges found unsupported

  • To the Chief Executive: Do not tell the board that customer loyalty fell by 11 points between Q1 and Q3. Most of the headline drop (from +24 to +13) is a methodological artefact caused by a shift in sample composition and survey mode, not a genuine collapse in customer sentiment.The reweighted NPS calculations are misstated: Q3 is approximately +17.8 using the active-account mix and +20.3 using the Q1 mix, rather than +17.6 and +20.4. The stated Q1 figure of +23.8 also does not follow from the supplied table.
  • Investing in live shipment tracking and proactive delay alerts is the single highest-impact improvement Quaymark can make across all customer tiers, and is the primary driver of Detractor scores.The open-text result is 31 of 45 detractors who answered, approximately 69%, not 31 of 43 or 72.1%.
  • To the Commercial Director: Do not reverse the per-shipment fee for all customers. Mid-sized and large shippers prefer or are indifferent to the per-shipment fee, while dissatisfaction is concentrated among small shippers (<20 shipments/month)— likely due to a flawed flat subscription price previously, or confusion between platform fees and total shipping costs.There is no evidence that small accounts' dissatisfaction is likely caused by a flawed previous subscription price or confusion between platform fees and total shipping costs.
  • To the Operations Director: There is a severe service failure affecting respondents who ship to the Republic of Ireland, driven by multi-day delivery delays and missed collections rather than pricing, though the survey sample is very small and requires operational validation.The Ireland result warrants investigation, but does not establish a severe failure specifically in Ireland services. The comments are from businesses that ship to Ireland; they do not identify the complained-about shipments as Ireland shipments. Statistical significance does not resolve the small-base and response-selection limitations.
  • Quaymark's core booking flow—getting a quote and booking a collection in around two minutes—is a major competitive strength and the primary reason Promoters recommend the platform.The count praising quick and easy booking is 79 answers, not 71, as the analysis itself reports in its theme summary.

Scorecard

Numbers 11 of 12 right

QuestionIts answerTrue
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?✓12.912.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?✓17.617.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?✓20.420.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?✓72.1%68.9%
How many of the open-text answers mention the per-shipment fee or its cost?✗1721
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?✓4.1 points4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What percentage of the accounts invited to the Q3 2026 wave completed it?✓11%11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?✓49.2%49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?✓3.9 points3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?✓1313
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?✓9991,001

Traps in the data 7 of 8 handled

  • ✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
  • ✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
  • ✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
  • ✓Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
  • ✓45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • ✗521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
  • ✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
  • ✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Prioritise investment in live shipment tracking and proactive delay alerts, and do NOT reverse the per-shipment fee back to a universal subscription or hire account managers for mid-sized (20-99) shippers. Live tracking and delay alerts address the #1 driver of Detractor sentiment across every customer segment (selected by 65.6% overall and 72.9% of 100+ shippers). Keep the per-shipment fee as the default model—since mid-sized and large shippers prefer it—but investigate offering an optional subscription or bundled credit tier specifically aimed at retaining price-sensitive smaller shippers, and audit Ireland partner hauliers immediately.

What would change it

We would reconsider reversing the per-shipment fee or introducing a subscription alternative immediately if billing and churn data from March–August 2026 revealed that: (1) high-margin accounts churned out of the active population before Q3 fielding due to the fee change, or (2) the revenue gained from per-shipment fees among <20 shippers is outweighed by a drop in their booking frequency or retention. We would reconsider account managers for the 100+ segment if unit-economics analysis shows that reducing the 14.6% Detractor rate in that segment via dedicated account management yields a higher ROI than self-serve tracking improvements.

Findings

  1. To the Chief Executive: Do not tell the board that customer loyalty fell by 11 points between Q1 and Q3. Most of the headline drop (from +24 to +13) is a methodological artefact caused by a shift in sample composition and survey mode, not a genuine collapse in customer sentiment.

    High confidenceBase 521

    In Q3 (online), small shippers (<20 shipments/month) made up 61.2% of respondents (319/521) compared to 30.1% in Q1 (124/412), while large shippers (100+ shipments) dropped from 23.8% (98/412) to 9.2% (48/521). Because small shippers have a much lower NPS (+5) than mid-sized (+21) or large shippers (+38), over-sampling them depresses the unweighted total. When Q3 responses are weighted to the true population of 4,720 active accounts (40% <20, 42% 20-99, 18% 100+), Q3 NPS is +17.6. When re-weighted to match the Q1 respondent mix, Q3 NPS is +20.4—only a 3.4-point difference from Q1's +23.8, which is within sampling error and easily explained by switching from phone interviews conducted by account managers (which induce social desirability bias) to an anonymous online survey.

  2. Investing in live shipment tracking and proactive delay alerts is the single highest-impact improvement Quaymark can make across all customer tiers, and is the primary driver of Detractor scores.

    High confidenceBase 521

    Across all 521 respondents, 65.6% selected live tracking (41.3%) or delay alerts (34.2%) as the top improvements in Q7, nearly triple the demand for lower booking fees (22.6%) or quicker support (23.0%). Demand is strong across all volume bands: 65.8% for <20 (base 319), 63.0% for 20-99 (base 154), and 72.9% for 100+ (base 48), as well as 69.2% of Detractors (base 120) and 69.6% of Passives (base 214). In Q4, 28.6% of respondents are dissatisfied with shipment visibility (48.4% among Detractors). In open-text responses (Q2), 72.1% (31/43) of Detractors who commented cited missing tracking or late/missed collections and deliveries.

  3. To the Commercial Director: Do not reverse the per-shipment fee for all customers. Mid-sized and large shippers prefer or are indifferent to the per-shipment fee, while dissatisfaction is concentrated among small shippers (<20 shipments/month)— likely due to a flawed flat subscription price previously, or confusion between platform fees and total shipping costs.

    High confidenceBase 521

    While 39.0% of unweighted respondents say the per-shipment fee is worse (Q5, base 521), this is driven by the over-represented <20 band, where 49.2% say it is worse and only 15.0% say better (base 319). In contrast, among mid-sized shippers (20-99, base 154), 26.6% say the per-shipment fee is better and 39.0% say about the same (only 25.3% worse). Among large shippers (100+, base 48), 31.2% say it is better and 41.7% about the same (only 14.6% worse). Weighted to the true account population, 33.0% find the fee worse versus 54.0% who find it better or the same. Furthermore, only 22.6% selected lower booking fees in Q7 (18.8% of 100+ shippers), and only 17 of 186 open-text responses mentioned the fee.

  4. Adding named account managers should not be pursued for mid-sized customers (20-99 shipments/month), though there is moderate interest among large accounts (100+ shipments/month).

    High confidenceBase 154

    In Q7, only 14.2% of total respondents (base 521) and 15.6% of mid-sized shippers (20-99 shipments/month, base 154) selected a named account manager as a top improvement—ranking fifth out of six options for the mid-sized tier. Interest rises to 31.2% among large accounts (100+ shipments/month, base 48, margin of error ±13.1 percentage points), where it ranks second behind live tracking (54.2%).

  5. To the Operations Director: There is a severe service failure affecting respondents who ship to the Republic of Ireland, driven by multi-day delivery delays and missed collections rather than pricing, though the survey sample is very small and requires operational validation.

    Medium confidenceBase 14

    Of the 14 respondents who ship to Ireland (Q9), 13 (92.9%) are Detractors (scoring 0-6) and 0% are Promoters, yielding an NPS of -93 compared to +16 for non-Ireland shippers (base 496). Even with a small base of 14, a 13/14 Detractor rate vs. 21.2% among non-Ireland shippers is highly statistically significant (Fisher's exact p < 0.0001). Six Ireland shippers provided open-text comments (Q013, Q033, Q115, Q150, Q295, Q296, Q339—seven total, six citing operations): five complained of severe late deliveries or missed collections (e.g., 3-5 day delays on next-day parcels), one cited tracking, and only one cited booking fees.

  6. Quaymark's core booking flow—getting a quote and booking a collection in around two minutes—is a major competitive strength and the primary reason Promoters recommend the platform.

    High confidenceBase 186

    Out of 186 open-text responses to Q2, 71 spontaneously praised the speed, simplicity, and ease of the quote-and-booking process, with dozens explicitly noting it takes 'a couple of minutes'. This holds across all volume bands and is the dominant theme among Promoters (scoring 9-10).

Caveats

  • Unweighted Q3 figures heavily over-represent small shippers (<20 shipments/month are 61.2% of the sample vs. 40% of the population) and under-represent mid-sized (29.6% vs. 42%) and large shippers (9.2% vs. 18%). All strategic decisions must use population-weighted numbers.
  • The Q1 and Q3 waves used fundamentally different methodologies: Q1 was conducted via phone by Quaymark's own account managers (introducing selection bias toward managed accounts and social desirability bias), whereas Q3 was a self-administered online survey sent to all active accounts. Direct comparisons of unweighted totals are invalid.
  • Volume bands in Q1 were taken from database account records, whereas Q3 volume bands were self-reported (Q8); some discrepancy between waves may stem from self-reporting inaccuracy.
  • Subgroup sample sizes for large accounts (100+ shipments/month, n=48, margin of error ±14.1 percentage points) and Republic of Ireland shippers (n=14) are small; point estimates for these subgroups carry wide confidence intervals.
  • The survey only sampled active accounts (at least one shipment booked in the last three months). Customers who churned completely between March and June 2026 due to the pricing change or service failures are excluded (survivorship bias).

Next steps

  • decisionPrioritise engineering and haulier integration roadmap to deliver automated collection/delivery delay alerts and granular in-transit tracking milestones.
  • researchMaintain the per-shipment fee as the standard pricing model and conduct a financial and churn audit of the March 2026 pricing transition across volume bands.
  • experimentTest an optional pre-paid monthly bundle or capped subscription add-on targeted at regular small shippers (e.g., 10–30 shipments/month) rather-than replacing per-shipment pricing.
  • researchConduct an urgent operational audit of Republic of Ireland shipments, examining haulier SLA performance, customs/border delays, and transit times across all active Ireland accounts.
  • monitorStandardise future NPS tracking to use the online survey methodology and weight all reporting to the database population profile.

Open-text themes it coded

Fast, simple quote and booking process 79Poor, vague, or non-existent shipment tracking and visibility 42Late or missed collections and delayed deliveries 29Dissatisfaction with per-shipment fee and preference for monthly subscription 21Slow or unreachable customer support 6Damaged goods and difficult claims process 5General satisfaction, neutral sentiment, or new/infrequent user 22

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Which of the following best describes your involvement in booking pallet or parcel shipments through Quaymark for your business?

    One answer
    • I am the main person who books or manages shipments through Quaymark
    • I share responsibility for booking shipments through Quaymark with others
    • Someone else in my business handles bookings through Quaymark
    • We do not use Quaymark

    Continues if I am the main person who books or manages shipments through Quaymark; I share responsibility for booking shipments through Quaymark with others

  2. S2

    Has your business booked at least one shipment through Quaymark in the last 3 months?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  3. S3

    On average, how many shipments (pallets and parcels combined) has your business booked through Quaymark per month over the last 3 months?

    One answer
    • Fewer than 20 shipments a month
    • 20 to 99 shipments a month
    • 100 or more shipments a month
    • None in the last 3 months

    Continues if Fewer than 20 shipments a month; 20 to 99 shipments a month; 100 or more shipments a month

  4. Q1

    On a scale from 0 to 10, how likely are you to recommend Quaymark to another business that ships pallets or parcels?

    Scale

    Scale 1-11: 0 - Not at all likely to 10 - Extremely likely

  5. Q2

    What are the main reasons for the score you just gave?

    Open text
  6. Q3

    Thinking about your experience with Quaymark over the last 3 months, which of the following have gone well? (Select up to 3)

    Any that apply
    • Speed and ease of getting a quote and booking
    • Partner haulier reliability (on-time collections and deliveries)
    • Online shipment tracking
    • Communication and updates when delays happen
    • Overall shipping prices and fees
    • How the fee structure works (how fees are charged)
    • Customer support when resolving an issue
    • Support from an account manager
    • None of these
    Judges: double-barrelled

    Some options combine distinct experiences into one answer, particularly quote speed/ease and booking speed/ease, and collection and delivery reliability. Selecting an option does not identify which component went well.

    'Speed and ease of getting a quote and booking' covers two distinct steps; 'collections and deliveries' are also two separate operations.

  7. Q4

    And over the last 3 months, which of the following have caused problems or frustration for your business? (Select up to 3)

    Any that apply
    • Speed and ease of getting a quote and booking
    • Partner haulier reliability (late or missed collections and deliveries)
    • Online shipment tracking
    • Communication and updates when delays happen
    • Overall shipping prices and fees
    • How the fee structure works (how fees are charged)
    • Customer support when resolving an issue
    • Lack of support from an account manager
    • None of these
    Judges: double-barrelled

    Some options combine distinct experiences into one answer, particularly quote speed/ease and booking speed/ease, and collection and delivery reliability. Selecting an option does not identify which component caused problems.

    Like Q3, asks about getting a quote and booking together, and collections and deliveries together.

  8. Q5

    When did your business start using Quaymark?

    One answer
    • Before March 2026
    • In or after March 2026
    • Not sure
  9. Q6

    Before March 2026, Quaymark charged a monthly subscription (starting from £25 a month depending on account size). Since March 2026, Quaymark instead charges a booking fee of £3.50 per shipment (reduced for accounts booking 100 or more shipments a month). Which fee structure do you prefer for your business?

    One answer
    • Strongly prefer the monthly subscription
    • Slightly prefer the monthly subscription
    • No preference between the two
    • Slightly prefer the per-shipment booking fee
    • Strongly prefer the per-shipment booking fee
    • Not sure
  10. Q7

    How do your total platform fees under the per-shipment booking fee compare with what your business would pay under a monthly subscription?

    One answer
    • We pay much more now with the per-shipment fee
    • We pay slightly more now with the per-shipment fee
    • We pay about the same under both
    • We pay slightly less now with the per-shipment fee
    • We pay much less now with the per-shipment fee
    • Not sure
  11. Q8

    Please rank the following potential changes to Quaymark from 1 (most valuable to your business) to 4 (least valuable to your business).

    Ranking
    • Replace the per-shipment booking fee with a flat monthly subscription (from £25/month based on volume tier)
    • Upgrade online shipment tracking with more detailed, real-time location updates
    • Send proactive automatic alerts (by email or SMS) as soon as a collection or delivery is delayed
    • Provide a dedicated Quaymark account manager as your single point of contact
  12. Q9

    Thinking about the option you ranked #1 in the previous question, what difference would it make to how much you use Quaymark over the next 6 months?

    One answer
    • We would book a lot more of our shipments through Quaymark
    • We would book slightly more of our shipments through Quaymark
    • We would continue booking about the same amount, but be more satisfied
    • It would stop us reducing or cancelling our use of Quaymark
    • It would make little or no difference to what we book
  13. Q10

    What types of shipments does your business book through Quaymark?

    One answer
    • Mainly pallets
    • Mainly parcels
    • A roughly equal mix of pallets and parcels
Sample plan

Invite the main booking contact at all 4,720 active UK accounts via email, linking S1-Q10 to database records (actual monthly shipment volume tier, Q1 2026 survey participation, support ticket count, and pre/post-March 2026 tenure). Target ~520 completes overall (11% response rate). PRECISION & QUOTA WARNING: Under a simple random sample of infinite population, +/-5 percentage points at 95% confidence (worst-case p=0.50) requires n=385 per stratum (1,155 total), which exceeds the ~520 total complete constraint. However, because Quaymark has a finite population of 4,720 accounts likely skewed heavily toward small shippers and relatively few mid-sized (20-99/mo) and large (100+/mo) accounts, applying the Finite Population Correction (FPC: n >= N*384.16 / (384.16 + (N-1)*0.0025)) drastically reduces required sample sizes in smaller strata (e.g., if N_large = 180, n=123 achieves +/-5%; if N_mid = 450, n=208 achieves +/-5%). We will therefore use disproportionate stratified sampling with targeted reminder waves to maximize completes in the 20-99 and 100+ tiers (e.g., aiming for ~130 completes in 100+, ~190 in 20-99, and ~200 in <20, adjusted to exact database N per tier using FPC), and apply post-stratification weights by population tier size for overall figures.