Sign inTest your prompt

Benchmarks / SurveyBench / Why Fernway customers cancel / Gemini 3.8 Flash

Measured by Spring Prompt

Gemini 3.8 Flash: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
66 of 100
Analysis rating
898
Head to head, this task
won 9 of 12

Why it is not sound

The survey

  • Answer bands that overlap or leave gaps
  • A leading question: Q2
  • A leading question: Q8
  • A double-barrelled question: Q2
  • A double-barrelled question: Q3

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: Northern Ireland respondents (base 12) cite delivery problems far more often.
  • Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Findings the judges found unsupported

  • Lack of delivery flexibility ('hard to skip weeks') is the primary driver of customer churn, significantly outpacing price.The survey establishes the most common self-reported cancellation reason, not the primary causal driver of churn. Also, 58.6% expressed hypothetical return intentions; saying they 'would return' turns those intentions into behaviour, notwithstanding the later caveat.
  • A 20% price cut is ineffective at reactivating cancellers and does not address core dissatisfaction.Lower stated return intent does not establish that a price cut is ineffective. Nearly half of price-leavers express likely return, and there is no behavioural or economic test establishing whether that response would be commercially worthwhile.
  • Northern Ireland suffers from an acute operational delivery failure.Eight delivery-related responses among twelve Northern Ireland respondents cannot establish an acute regional operational failure. The small-base caveat does not justify presenting that diagnosis as fact.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 5 of 8 handled

  • ✗Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Commit product resources to build one-tap skipping immediately; reject the 20% across-the-board price cut. Difficulty skipping deliveries is the single largest driver of cancellation (~3,193 of the 9,850 cancellations), and one-tap skipping achieves much higher win-back intent (36.4% vs 25.8% overall; 58.6% vs 22.9% among skip-leavers) without sacrificing 20% of topline margin. Additionally, explore offering a 2-meal/week option to alleviate food waste and fridge overload.

What would change it

Evidence that a 20% price cut dramatically expands customer acquisition volume with positive unit economics, or proof that one-tap skipping causes an unsustainable collapse in weekly active box volume that cannot be offset by retained subscriber lifetime value.

Findings

  1. Lack of delivery flexibility ('hard to skip weeks') is the primary driver of customer churn, significantly outpacing price.

    High confidenceBase 640

    35.5% unweighted (and 32.4% tenure-weighted) cited difficulty skipping weeks as their main reason for cancelling, versus 21.6% unweighted (26.3% weighted) for price. Furthermore, 58.6% of skip-leavers would return for one-tap skipping, compared to only 22.9% for a 20% price cut.

  2. A 20% price cut is ineffective at reactivating cancellers and does not address core dissatisfaction.

    High confidenceBase 640

    Overall, only 25.8% of cancellers say they are likely to return if plans cost 20% less (versus 36.4% for one-tap skipping). Even among those who specifically left due to price, fewer than half (49.3%) say a 20% cut would make them likely to return.

  3. Family plan dissatisfaction is an artifact of sample tenure composition (Simpson's Paradox), not intrinsic plan quality.

    High confidenceBase 640

    The overall satisfaction gap (Family 49.2% vs Couples 53.4%) is not statistically significant (z = 1.07, p > 0.05). When controlling for tenure, Family satisfaction is actually higher than Couples across all cohorts: Under 3 months (69.2% vs 66.1%), 3-12 months (61.6% vs 57.4%), and Over 12 months (42.1% vs 35.2%). Family appeared lower only because 67.1% of Family respondents had tenures over 12 months (where satisfaction drops generally), compared to 31.0% of Couples.

  4. Menu fatigue and recipe repetition become a critical churn factor for mature subscribers.

    High confidenceBase 640

    While only 6.5% of short-tenure (<3 months) customers left due to repetitive recipes, this rose to 25.4% among customers with >12 months tenure. In open text, 40 respondents cited menu repetition, with 25 of those coming from >12-month subscribers.

  5. Northern Ireland suffers from an acute operational delivery failure.

    Medium confidenceBase 12

    66.7% of respondents in Northern Ireland cited delivery problems as their primary reason for cancelling, compared to 8.1% in England, 13.0% in Scotland, and 0.0% in Wales.

Caveats

  • Non-response bias: Completed sample over-represents long-tenure customers (>12 months: 48.0% in sample vs 20.0% in population) and under-represents early churners (<3 months: 21.6% in sample vs 45.0% in population). Early cancellers are more price-sensitive.
  • Intent-to-behavior gap: Stated likelihood to return upon product or price changes generally overestimates actual reactivation rates.
  • Small regional sample size for Northern Ireland (n=12), meaning regional figures indicate a strong signal but lack high statistical precision.

Next steps

  • decisionPrioritise and scope the 4-month build for in-app one-tap week skipping and subscription pausing.
  • researchConduct an emergency operational audit of 3PL logistics and courier partners in Northern Ireland.
  • experimentRun an operational and culinary review to increase recipe rotation and test a 2-recipe/week box tier.
  • decisionBrief the CEO and Head of Product on Simpson's Paradox regarding Family plan satisfaction.

Open-text themes it coded

Friction in skipping or pausing deliveries 47Recipe monotony and lack of variety 40Plan rigidity and demand for lower weekly meal minimums 31Price sensitivity and post-discount price creep 41Inadequate portion sizes 12Excessive or non-recyclable packaging 19Delivery and fulfillment failures 17Unpreventable life circumstance changes 6

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    When did you cancel or stop your Fernway meal-kit subscription?

    One answer
    • Within the last 30 days
    • 1 to 3 months ago
    • 4 to 6 months ago
    • More than 6 months ago
    • I currently have an active Fernway subscription
    • I have never subscribed to Fernway

    Continues if Within the last 30 days; 1 to 3 months ago; 4 to 6 months ago

  2. S2

    How long had you been subscribed to Fernway before you cancelled?

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months
    • Can't recall

    Continues if Less than 3 months; 3 to 12 months; More than 12 months

  3. Q1

    Overall, how satisfied were you with your Fernway subscription during the time you were an active customer?

    Scale

    Scale 1-5: Extremely dissatisfied to Extremely satisfied

  4. Q2

    How satisfied were you with each of the following aspects of Fernway while subscribed?

    Any that apply
    • Recipe variety and taste: Satisfied
    • Ingredient freshness and quality: Satisfied
    • Delivery reliability: Satisfied
    • Value for money: Satisfied
    • Flexibility to skip or change delivery dates: Satisfied
    • None of these satisfied me
    Judges: leading, double-barrelled, ambiguous

    Only satisfied responses are available for individual aspects; neutral and dissatisfied views cannot be recorded separately. Bundled aspects such as recipe variety and taste also require one answer despite potentially different satisfaction levels.

    Every option is positive ('Satisfied'), so there is no way to report dissatisfaction or a neutral view. An unticked item could mean dissatisfied, neutral or not applicable. Several items join two attributes under one answer: variety and taste, freshness and quality, skip or change dates.

  5. Q3

    What was the single primary reason you decided to cancel your Fernway subscription?

    One answer
    • Too expensive / needed to cut household spending
    • Difficult to skip or pause weeks when away or busy
    • Skipping cut-off was too strict (needed 6 days' notice)
    • Food waste because we received boxes we couldn't eat in time
    • Recipes became repetitive or didn't match our tastes
    • Quality or freshness issues with ingredients
    • Delivery or packaging issues
    • Household circumstances changed (e.g. moved, dietary change)
    • Switched to a competitor or regular supermarket shopping
    • Other
    Judges: double-barrelled

    The price option merges 'too expensive' with 'needed to cut household spending', which are two different reasons. Flexibility is split across three options (hard to skip, strict cut-off, food waste) while price is consolidated into one. This makes price look like the leading single reason.

    Several options are double-barrelled, combining distinct concepts (e.g., 'Too expensive / needed to cut household spending', 'Delivery or packaging issues').

  6. Q4

    Which other factors, if any, contributed to your decision to cancel? (Select all that apply)

    Any that apply
    • Price per meal felt too high for our budget
    • Forgot to skip a box before the 6-day deadline
    • App made it too frustrating to manage weekly deliveries
    • Accumulated too many meals in the fridge / freezer
    • Wanted the option to order fewer meals per week
    • Cooking the meals took longer than expected
    • Portion sizes were too small
    • Found cheaper meal alternatives elsewhere
    • No other reasons contributed
  7. Q5

    During your subscription, how frequently did you find Fernway's 6-day advance notice requirement for skipping a delivery difficult to manage?

    One answer
    • Almost every week
    • About once or twice a month
    • A few times during my subscription
    • Never - the 6-day notice was easy to manage
    • I never tried to skip a delivery
  8. Q6

    Did you ever receive and pay for a Fernway box you had intended to skip because you missed the 6-day cut-off?

    One answer
    • Yes, multiple times
    • Yes, once
    • No, but I came close
    • No, never
    • Not applicable
  9. Q7

    Imagine Fernway had offered the following two changes before you cancelled. Which single change would have made you more likely to keep your subscription?

    One answer
    • Option A: Keeping current prices, but allowing you to skip any week with one tap up to 24 hours before delivery
    • Option B: A permanent 20% price reduction across all plans, while keeping the 6-day advance skip deadline
    • Neither change would have kept me subscribed
    • Both changes equally would have kept me subscribed
  10. Q8

    Please rank the following potential improvements from most important (1) to least important (4) in making a meal-kit service right for your household:

    Ranking
    • Lower price per portion (20% reduction)
    • Frictionless flexibility (one-tap skipping without multi-day cut-offs)
    • Volume flexibility (ability to order 2 meals instead of minimum 3-4)
    • Wider recipe selection and culinary variety
    Judges: leading

    Calling the skipping option 'frictionless' gives it an explicitly favourable characterisation that the competing improvements do not receive.

    Only the flexibility option is given positive marketing language ('Frictionless'), which favours it over the plainly worded price option.

  11. Q9

    How likely are you to consider restarting your Fernway subscription in the next 12 months?

    Scale

    Scale 1-5: Definitely will not to Definitely will

  12. Q10

    Which of the following offers would be most persuasive in winning back your business?

    One answer
    • A 20% lower weekly price on your plan
    • A one-tap skip feature that lets you skip any week on short notice (up to 24h before)
    • The ability to order flexible meal counts (e.g. only 2 meals in busy weeks)
    • An indefinite 'pause' mode with no automated restarts
    • None of these would persuade me to return

    Routing Shown only if Q9 is: Definitely will; Probably will; Might or might not

  13. Q11

    Which Fernway plan were you on prior to cancelling?

    One answer
    • Couples plan (2 people)
    • Family plan (4 people)
    • Not sure
  14. Q12

    In your own words, what is the single most important thing Fernway could have done to keep you as a customer?

    Open text
Sample plan

Target population: All UK customers who cancelled a Fernway Couples or Family subscription in the past 6 months. To satisfy the Board's precision requirement of +/- 5% margin of error at a 95% confidence level for each of the three tenure cohorts (<3 months, 3-12 months, >12 months), a minimum of 384 completes per tenure group is statistically required, yielding a minimum total sample of N = 1,155 completes (or N = 600 total with N = 200 per cell yielding +/- 6.9% precision if capped strictly at 600). Sampling execution: Send email invitations census-style to all qualifying cancelled accounts in the database, stratified by verified tenure. Enforce hard quotas of n = 385 completes for each tenure bucket (or n = 200 if constrained by fixed budget/response volume) with soft quotas on Couples vs. Family plan matching the historical churn mix. Questionnaire completion time is calibrated to 4-5 minutes.