Benchmarks / SurveyBench / Why Quaymark's customer score fell / Gemini 4 Carbon

Measured by Spring Prompt

Gemini 4 Carbon: Why Quaymark's customer score fell

The decision: whether to reverse the per-shipment fee and bring back the subscription, invest in shipment tracking and proactive delay alerts, or add account managers for mid-sized customers. The brief and the data →

Verdict
✗ Not sound
Research score
93 of 100
Analysis rating
1,076
Head to head, this task
won 10 of 10

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A double-barrelled question: Q3
  • A double-barrelled question: Q4

The analysis

  • Missed a trap: Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What is the Q3 2026 wave's NPS across all respondents (percentage scoring 9-10 minus percentage scoring 0-6), unweighted?✓12.912.9
Weighted to the shipments-a-month mix of all 4,720 active accounts, what is the Q3 2026 wave's NPS?✓17.817.8
Re-weighted to the shipments-a-month mix of the Q1 2026 wave's respondents, what is the Q3 2026 wave's NPS?✓20.320.3
Among respondents who scored 0-6 and answered the open question, what percentage of answers mention tracking (not being able to see where shipments are) or late or missed collections or deliveries?✓68.9%68.9%
How many of the open-text answers mention the per-shipment fee or its cost?✓2121
What is the 95% margin of error, in percentage points, on the Q3 2026 wave's unweighted share of respondents scoring 9-10?✓4.1 points4.12 points
Among accounts booking 100 or more shipments a month, is the change in the share scoring 9-10 between the Q1 2026 and Q3 2026 waves statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What percentage of the accounts invited to the Q3 2026 wave completed it?✓11%11%
Among Q3 2026 respondents booking fewer than 20 shipments a month, what percentage said the per-shipment fee is worse for their business than the subscription?✓49.2%49.2%
Among accounts booking fewer than 20 shipments a month, by how many points did NPS fall between the Q1 2026 and Q3 2026 waves?✓3.9 points3.9 points
How many Q3 2026 respondents who send shipments to the Republic of Ireland scored 0-6?✓1313
Weighted to the shipments-a-month mix of all 4,720 active accounts, about how many accounts would score 0-6?✓1,0011,001

Traps in the data 7 of 8 handled

  • ✓The Q1 wave (+24) was phone interviews by account managers; the Q3 wave (+13) was online. Interviewer-administered scores, especially when the interviewer is the customer's own account manager, run higher, so part of the fall may be the change of method rather than a change in loyalty.
  • ✓Accounts booking fewer than 20 shipments a month are 61% of Q3 respondents against 30% in Q1 and 40% of active accounts, and they score lowest (Q3 NPS +5). Re-weighted to Q1's mix the Q3 NPS is +20.3, and to the active-account mix +17.8, so much of the 11-point fall is a change in who answered.
  • ✓The commercial director believes the fee caused the fall. Among detractors who explained their score, 69% mention tracking or delays and 18% the fee; only 39% of all respondents say the fee is worse than the subscription (49% of small accounts, 15% of the largest).
  • ✗Respondents who ship to Ireland (base 14) are nearly all detractors (NPS -93).
  • ✓45% of small accounts say they would be likely to choose a monthly subscription if offered: a stated intention about a hypothetical, with no price given.
  • ✓521 of 4,720 invited accounts completed (about 11%); unhappy customers and small accounts may be more likely to respond online.
  • ✓Among accounts booking 100 or more a month (bases 98 and 48), the promoter share moved from 49% to 52%, NPS +38 to +38: not significant.
  • ✓The 69% figure is a share of the 45 detractors who answered the open question, not of all respondents or all customers.

The analysis

Recommendation

Prioritise investment in live shipment tracking and proactive late-collection/delivery alerts, while retaining the per-shipment fee model and holding off on hiring mid-market account managers. Specifically, integrate milestone/depot scan feeds from partner hauliers and trigger automated exception alerts when collection slots or delivery ETAs slip—directly addressing the #1 source of detractor pain (65.6% top-two priority; 68.9% of detractor verbatims) and reducing inbound 'where is my shipment?' support load without disturbing the praised two-minute booking experience.

What would change it

Two pieces of internal evidence would alter this decision: (1) Account-cohort churn and net-revenue-retention data from March–August 2026 showing heavy attrition or volume defection after the per-shipment fee launch among formerly active subscribers (hidden from this active-only survey), which would justify introducing an optional priced subscription/bundle tier alongside tracking; or (2) A technical/carrier audit showing partner hauliers cannot supply real-time scan or ETA data via API/EDI, in which case investment would need to shift toward carrier SLA enforcement/re-selection and faster proactive exception support.

Findings

  1. Invest in live shipment tracking and proactive delay alerts rather than reversing the fee change or hiring mid-market account managers: post-booking visibility and uncommunicated collection/delivery delays are the dominant drivers of detraction across every volume tier.

    High confidenceBase 521

    In Q7 (base 521), 65.6% select live tracking (41.3%) or late collection/delivery alerts (34.2%) as a top-two improvement, rising to 69.2% among detractors (base 120) and 69.6% among passives (base 214), and leading across small (65.8%, base 319), mid-sized (63.0%, base 154), and large shippers (72.9%, base 48). Tracking visibility (Q4) is the worst-rated aspect of the service (28.6% dissatisfied overall; 48.3% among detractors), followed by punctuality (Q3: 22.5% dissatisfied overall; 44.2% among detractors). In Q2 open text, 68.9% (31 of 45) of detractor responses cite poor tracking or late/missed collections and deliveries, while 79 of 186 total responses praise the fast two-minute quote-and-booking flow.

  2. The CEO should NOT tell the board that customer loyalty fell by 11 points between Q1 and Q3: roughly two-thirds of the headline gap (+24 to +13) comes from sample-mix distortion, and the remainder is confounded by switching from account-manager phone calls to an online survey.

    High confidenceBase 933

    Small accounts (<20 shipments/month) have much lower NPS (+5 in Q3; +9 in Q1) than large accounts (+38 in both waves), and made up 61.2% of Q3 respondents (319/521) versus 40% of the 4,720 active accounts and just 30.1% of Q1 respondents (124/412). Weighting Q3 to the active-account population raises NPS from +12.9 to +17.8; re-weighting Q3 to Q1's volume mix raises it to +20.3, reducing the 11-point gap to ~3.7 points. Within volume bands, NPS moved only modestly (<20: -3.9 points from +8.9 to +5.0; 20-99: -4.4 points from +25.8 to +21.4; 100+: unchanged at +37.8 vs +37.5, promoter share 49.0% on base 98 vs 52.1% on base 48, not statistically significant). Moreover, Q1 was conducted by phone by customers' own account managers (social desirability and selection bias), whereas Q3 was self-completed online.

  3. Answering the Commercial Director: customers do NOT broadly want a return to mandatory subscriptions, and mid-to-large shippers prefer or accept per-shipment pricing, though dissatisfaction is concentrated among sub-20-shipment accounts.

    Medium confidenceBase 521

    Unweighted in Q5 (base 521), more respondents say the per-shipment fee is better (20.0%) or about the same (29.9%)—49.9% combined—than worse (39.0%). Weighted to the actual active-account mix, 56.2% say better or about the same versus 32.9% worse, because mid-sized (65.6% better/same vs 25.3% worse, base 154) and large accounts (72.9% better/same vs 14.6% worse, base 48) favour or accept the change, while pain concentrates in <20/month shippers (49.2% worse, base 319). Even for an unpriced optional subscription (Q6, base 521), only 39.5% unweighted (37.1% weighted) say they would likely choose it. Lower booking fees rank a distant third in Q7 (22.6% overall; 19.5% in 20-99; 18.8% in 100+), and only 21 of 186 open-text answers (11.3%) mention fees, mostly from passives (13 of 21) rather than detractors (8 of 45).

  4. Mid-sized customers do not prioritise named account managers; hiring account managers for the 20-99 shipment tier would misallocate budget away from self-serve tracking and exception alerts.

    High confidenceBase 154

    In Q7, only 15.6% of mid-sized shippers (20-99 shipments/month, base 154) and 14.2% of all respondents (base 521) select a named account manager, ranking behind live tracking (40.3%), late alerts (35.1%), quicker customer support replies (22.1%), and lower booking fees (19.5%) in the mid-sized tier. Demand for a named account manager only becomes substantial among 100+ shipment accounts (31.2%, base 48), and even they prioritise live tracking far more (54.2%). Open-text comments show customers ring or email Quaymark primarily because online tracking fails to update.

  5. Answering the Operations Director: Quaymark has an acute problem among accounts that ship to the Republic of Ireland, though the segment is too small to drive overall company NPS and requires operational log verification to separate ROI lane failures from general carrier delays.

    Medium confidenceBase 14

    13 of the 14 respondents who ship to Ireland scored 0-6 (92.9% detractors, NPS -93 on base 14 vs 21.2% detractors and NPS +16 among the 496 non-Ireland shippers; Fisher's exact p < 0.0001). However, ROI shippers are only 2.7% of the sample (14/521), shifting total NPS by just ~3 points (+13 vs +16). Furthermore, Q9 identifies whether an account ships to Ireland rather than isolating ROI consignments, and none of the 7 open-text answers from ROI shippers explicitly mention Ireland or customs—5 cite multi-day collection/delivery delays and 1 cites poor tracking.

Caveats

  • Survivorship bias on pricing: Q3 only surveyed active accounts with at least one booking between June and August 2026. Customers who churned outright after the March 2026 fee change are excluded, so Q5 and Q6 may understate total commercial backlash to per-shipment fees.
  • Q1 vs Q3 wave incomparability: Q1 was conducted by telephone by account managers calling 640 of their own customers (over-representing mid/large managed accounts and inviting social-desirability bias), whereas Q3 was a self-administered email survey across all 4,720 active accounts (11.0% response rate) that heavily over-sampled <20/month shippers (61.2% of respondents vs 40% of population) and used self-reported volume (Q8) rather than database records.
  • Unpriced hypothetical in Q6: Asking about subscription uptake without specifying the monthly price or included shipment allowance inflates stated purchase intent (39.5%) among customers hoping to lower their current bill.
  • Small subgroup bases and account-level attribution: The Republic of Ireland subgroup (n=14) and 100+ shipments subgroup (n=48 in Q3) have wide confidence intervals, and Q9 asks whether the business ships to Ireland at all rather than whether dissatisfaction stems specifically from Irish shipments.
  • Booker vs budget-holder perspective: Screeners S1/S2 restricted the survey to operational users who personally book shipments; finance decision-makers may weigh fee predictability vs operational tracking differently (though several bookers did pass on accounts-team feedback).

Next steps

  • decisionCommit product and carrier-ops budget to milestone tracking integration and automated collection/delivery delay alerts, enforcing scan-compliance SLAs on partner hauliers.
  • researchAnalyse database cohort churn, booking frequency, and revenue per account from Q4 2025 through Q3 2026 across volume bands to quantify survivorship bias from the March 2026 fee switch.
  • researchAudit shipment-level on-time collection/delivery, customs delays, and tracking-scan completion for all Republic of Ireland consignments, paired with 8-10 short diagnostic calls with the 13 ROI detractors.
  • experimentIf cohort churn analysis reveals material attrition among regular small/seasonal shippers, test a priced optional monthly bundle/pass (with explicit price points) rather than reverting the entire customer base.
  • monitorStandardise future NPS waves on the online survey instrument, link responses to database volume/route records, and report population-weighted NPS alongside within-band scores to the board.

Open-text themes it coded

Fast, simple quote and booking process 79Poor or missing shipment tracking and having to chase updates 43Missed or late collections and delayed deliveries 29Per-shipment booking fees costing more or harder to budget than subscription 21Neutral, functional satisfaction or too new/infrequent to hold a strong view 22Slow or hard-to-reach customer support 6Freight damage and slow, difficult claims handling 5

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you the main person responsible for booking or managing shipments with Quaymark for your business?

    One answer
    • Yes, I am the main contact
    • I share this responsibility equally with colleagues
    • No, someone else is the main contact

    Continues if Yes, I am the main contact; I share this responsibility equally with colleagues

  2. S2

    Has your business booked at least one pallet or parcel shipment through Quaymark in the last three months?

    One answer
    • Yes
    • No
    • Not sure

    Routing Shown only if S1 is: Yes, I am the main contact; I share this responsibility equally with colleagues

    Continues if Yes

  3. Q1

    On a scale of 0 to 10, how likely are you to recommend Quaymark to another business?

    Scale

    Scale 1-11: 0 - Not at all likely to 10 - Extremely likely

  4. Q2

    What is the main reason for the score you just gave?

    Open text
  5. Q3

    Which of the following have positively contributed to your experience with Quaymark over the last three months? (Select all that apply)

    Any that apply
    • Speed and ease of getting quotes and booking
    • Freight prices from partner hauliers
    • Quaymark's platform and booking fees
    • Reliability of collections and deliveries
    • Online shipment tracking
    • Updates and communication when delays happen
    • Customer support and account management
    • None of these
    Judges: double-barrelled

    The combined option 'Customer support and account management' prevents distinguishing positive experiences with general support from those with account management, which is a separate investment decision.

    'Customer support and account management' joins two distinct things in one option. Account management is a separate investment option in the decision, so a tick cannot be attributed to either. 'Quaymark's platform and booking fees' can be read as the online platform plus fees, or as the platform/booking fee itself.

  6. Q4

    And which of the following have caused frustration or problems for your business over the last three months? (Select all that apply)

    Any that apply
    • Speed and ease of getting quotes and booking
    • Freight prices from partner hauliers
    • Quaymark's platform and booking fees
    • Reliability of collections and deliveries
    • Online shipment tracking
    • Lack of updates or communication when delays happen
    • Customer support and account management
    • None of these
    Judges: double-barrelled

    The combined option 'Customer support and account management' prevents distinguishing problems with general support from those with account management, which is a separate investment decision.

    Same combined options as Q3. 'Customer support and account management' merges two things. 'Platform and booking fees' could mean the software platform or the fee, so a frustration tick cannot be pinned to the fee.

  7. Q5

    How satisfied or dissatisfied are you with Quaymark's current per-shipment booking fee (£3.50 per shipment, or a reduced rate for 100+ shipments a month)?

    One answer
    • Very satisfied
    • Fairly satisfied
    • Neither satisfied nor dissatisfied
    • Fairly dissatisfied
    • Very dissatisfied
    • Not aware of how Quaymark charges platform fees
  8. Q6

    Earlier in 2026, Quaymark replaced its fixed monthly subscription (starting from £25 a month depending on account size) with the current per-shipment booking fee. Which fee structure do you prefer for your business?

    One answer
    • Strongly prefer a fixed monthly subscription
    • Slightly prefer a fixed monthly subscription
    • No preference between the two
    • Slightly prefer a per-shipment booking fee
    • Strongly prefer a per-shipment booking fee
    • Not sure / joined after March 2026
  9. Q7

    We are deciding where to invest next to improve Quaymark. Please rank the following four options from 1 (most valuable to your business) to 4 (least valuable to your business).

    Ranking
    • Option to pay a fixed monthly subscription instead of a per-shipment booking fee
    • More detailed, real-time online shipment tracking
    • Proactive alerts and revised ETAs as soon as a collection or delivery is delayed
    • A dedicated account manager to help manage bookings and resolve issues
  10. Q8

    What single change could Quaymark make to most improve your experience?

    Open text
  11. Q9

    Roughly how many shipments (pallets and parcels combined) does your business book through Quaymark in a typical month?

    One answer
    • Fewer than 20 shipments a month
    • 20 to 99 shipments a month
    • 100 or more shipments a month
  12. Q10

    How long has your business been using Quaymark?

    One answer
    • Since before March 2026
    • Joined in March 2026 or more recently
    • Not sure
Sample plan

Invite all 4,720 active UK customer accounts via email to the main booking contact, using unique survey links tied to CRM account IDs (linking actual 3-month shipment volume, pre/post-March tenure, and Q1 participation flag). Target ~520 completes overall (~11% response rate). Stratify reminders by CRM account size to maximise completes in the mid (20-99/mo) and large (100+/mo) tiers rather than letting small accounts dominate. Methodological note on precision: achieving +/-5 percentage points at 95% confidence (assuming worst-case p=0.5) in each of the three account-size tiers simultaneously requires ~385 completes per tier (~1,155 total) under an infinite population, which exceeds the 520-complete budget. However, because N=4,720 is finite—and the 100+/mo and 20-99/mo tiers are likely smaller finite sub-populations—applying the Finite Population Correction (FPC) substantially reduces required sample sizes in the upper tiers (e.g., if N_tier=250, n=152 achieves +/-5pp; if N_tier=600, n=235 achieves +/-5pp). Allocate soft targets (e.g., ~200 small, ~180 mid, ~140 large, adjusted to exact CRM tier counts using FPC) via targeted email reminders to high-volume tiers, and apply post-stratification weights to match the true 4,720 account-size distribution for overall estimates.