Benchmarks / SurveyBench / Why Fernway customers cancel / Gemini 4 Hydrogen

Measured by Spring Prompt

Gemini 4 Hydrogen: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
91 of 100
Analysis rating
1,088
Head to head, this task
won 4 of 10

Why it is not sound

The survey

  • A double-barrelled question: Q11

The analysis

  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 7 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build one-tap skipping (incorporating multi-week holiday pauses and removing dark-pattern confirmation friction) rather than cutting plan prices across the board by 20%. One-tap skipping addresses the largest tenure-weighted churn driver (32.4%, ~3,193 cancellers), commands substantially higher win-back intent than a 20% price cut (36.4% vs 25.8% unweighted; 58.6% vs 22.9% among skip-leavers), and avoids permanently sacrificing 20% of revenue across all retained customers. In parallel, address the <3-month price-churn segment (31.2% of early leavers) with targeted introductory/loyalty offers or a lower-ticket 2-meal/week option rather than a blanket price cut, and immediately verify Northern Ireland delivery performance using operational courier data.

What would change it

Evidence from live financial modelling or controlled A/B testing showing that easier skipping increases skip frequency among active subscribers enough to destroy more contribution margin than a 20% price cut (or targeted discount), OR a win-back/retention experiment showing actual reactivation and 90-day LTV from a price reduction outperforms flexible skipping after accounting for margin loss, OR engineering scoping showing one-tap skip/pause requires substantially more than 4 months while early-tenure price churn escalates.

Findings

  1. Difficulty skipping weeks is the single largest driver of cancellation—even after correcting for tenure bias—and one-tap skipping outperforms a 20% price cut on stated win-back intent.

    High confidenceBase 640

    Unweighted, 35.5% cited 'Hard to skip weeks' vs 21.6% (±3.2 pp) for 'Price' (base 640). Weighted to the cancelled population's tenure mix (45% <3m, 35% 3-12m, 20% >12m), 'Hard to skip weeks' remains #1 at 32.4% (~3,193 of 9,850 cancellers) vs 26.3% for 'Price'. Overall, 36.4% say they would likely return for one-tap skipping vs 25.8% for a 20% price cut; among skip-leavers (base 227), 58.6% would likely return for one-tap skipping vs 22.9% for a 20% price cut (a 35.7 pp gap).

  2. Price is not the primary reason people leave, as the CEO hypothesised, though it is a substantial secondary driver that is masked by the survey's under-representation of early-tenure churners.

    High confidenceBase 640

    Unweighted, Price is #2 at 21.6% (base 640), but it is 31.2% among <3m cancellers (base 138) and 27.2% among 3-12m cancellers (base 195), compared with 13.7% among >12m cancellers (base 307). Because <3m customers are 45% of all cancellers but only 21.6% of respondents, weighting raises Price's share to 26.3%. Even among price-leavers (base 138), only 49.3% say a 20% price cut would likely bring them back, and Q5 open text links price dissatisfaction to post-introductory price jumps and rigid minimum box sizes.

  3. The Family plan is not the problem: its lower aggregate satisfaction is a Simpson's paradox caused by tenure confounding, not plan performance.

    Medium confidenceBase 640

    Across all respondents, the apparent 4.2 pp gap in satisfaction between Couples (53.4%, base 339) and Family (49.2%, base 301) is not statistically significant (z = 1.07, p = 0.29). Within every tenure band, Family satisfaction is actually higher than Couples: <3m (69.2% vs 66.1%; bases 26, 112), 3-12m (61.6% vs 57.4%; bases 73, 122), and >12m (42.1% vs 35.2%; bases 202, 105). Family looks lower overall only because 67.1% of Family respondents had >12m tenure vs 31.0% of Couples, and satisfaction drops sharply with tenure (66.7% <3m to 39.7% >12m).

  4. Menu repetition and lack of variety drive long-term churn and underpin falling satisfaction among tenured customers.

    High confidenceBase 640

    'Recipes got repetitive' rises from 6.5% among <3m cancellers (base 138) to 14.4% at 3-12m (base 195) and 25.4% at >12m (base 307), and is higher on Family (23.6%, base 301) than Couples (13.0%, base 339) due to Family's longer-tenured profile. In Q5 open text (190 answers), 40 respondents cite recipe variety or repetition, 25 of whom were >12m customers.

  5. Northern Ireland shows an acute delivery failure signal, though the survey sample is far too small to quantify the problem without operational data.

    Low confidenceBase 12

    66.7% of Northern Ireland respondents (8 of 12) cited Delivery problems as their main reason for cancelling, compared with 8.1% in England (base 546), 13.0% in Scotland (base 46), and 0.0% in Wales (base 36), and R445 specifically cites late boxes and missing items. Delivery problems are also elevated among <3m churners overall (15.2%, base 138).

  6. Forcing a minimum of three to five meals per week and multi-week pause friction are major unmeasured churn drivers omitted from Q1's precoded list.

    Medium confidenceBase 190

    In Q5 (190 open-text answers), 28 respondents unprompted said they would have stayed with a 1-2 meal per week option or smaller minimum box, and 49 requested skipping/pausing improvements that extend beyond one-tap single-week skipping to multi-week holiday holds, fortnightly plans, later cut-offs, and removing dark-pattern confirm screens.

Caveats

  • Low completion rate (6.5% of 9,850 invited) creates substantial non-response bias risk; respondents who felt strongly about product friction or delivery failures may be over-represented.
  • Severe tenure and plan skew in the achieved sample: >12m customers are 48.0% of respondents vs 20% of all cancellers, while <3m customers are 21.6% vs 45% (and Family is 47.0% vs 34%). Unweighted totals understate price and early delivery problems and overstate skipping and recipe repetition.
  • Stated return likelihood (Q3 and Q4) suffers from hypothetical bias and overstates actual win-back conversion; moreover, the survey does not measure the revenue impact of easier skipping on active customers (lower order frequency) versus a permanent 20% price cut across the entire active base.
  • Q1 forced a single main reason, obscuring multi-factor churn (evident in Q5 combinations like skipping + variety or price + fewer meals) and omitting 'too many meals per week' (28 unprompted Q5 mentions).
  • Very small subgroup bases—especially Northern Ireland (n=12), Wales (n=36), Scotland (n=46), Under 3m Family (n=26), and minor Q1 reasons (Packaging n=31, Other n=28, Portions n=43)—make subgroup percentages volatile.
  • No baseline survey of active (retained) subscribers exists, so we cannot benchmark whether active customers share the same satisfaction drop over tenure or skip friction.

Next steps

  • decisionReject an across-the-board 20% price cut and commit the 4-month product roadmap slot to one-tap skipping, multi-week pause, and removal of multi-screen skip barriers.
  • monitorAudit 6-month courier telemetry in Northern Ireland (and Scotland) for on-time delivery, late/lost boxes, missing-item tickets, and refund rates vs England/Wales.
  • experimentRun a controlled win-back experiment on the ~9,200 non-respondent March-August cancellers comparing a 'one-tap skip/pause' flexible-plan offer against a targeted time-limited price discount (stratified by tenure).
  • experimentModel skip cannibalisation and A/B test low-friction skipping (e.g., removing multi-screen prompts or testing a later cut-off/pause UI) on a cohort of active subscribers before full rollout.
  • researchConduct scoping and unit-economics research on a 2-meal/week minimum plan and expanded menu rotation for >12-month customers.

Open-text themes it coded

Skipping, pausing, and delivery frequency 49Price, price increases, and value for money 42Recipe repetition and limited variety 40Fewer meals per week and smaller minimum boxes 28Excessive or non-recyclable packaging 19Delivery reliability, timing, and missing items 18Portion sizes too small or not adjustable 12External life changes / nothing would have retained 6

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Did you cancel a Fernway meal-kit subscription within the last 6 months?

    One answer
    • Yes, I cancelled within the last 6 months
    • Yes, I cancelled more than 6 months ago
    • No, I currently have an active Fernway subscription
    • No, I have never subscribed to Fernway

    Continues if Yes, I cancelled within the last 6 months

  2. S2

    Roughly how long were you subscribed to Fernway before you cancelled?

    One answer
    • Under 3 months
    • 3 to 12 months
    • Over 12 months
    • I don't remember

    Continues if Under 3 months; 3 to 12 months; Over 12 months

  3. Q1

    Overall, how satisfied or dissatisfied were you with Fernway while you were subscribed?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  4. Q2

    Which aspects of Fernway did you like or feel satisfied with while subscribed? Please select all that apply.

    Any that apply
    • Recipe taste and variety
    • Ingredient quality and freshness
    • Portion sizes
    • Ease and time to cook
    • Delivery reliability and packaging
    • Ability to manage or skip deliveries
    • Price and value for money
    • Customer support
    • None of these
  5. Q3

    Which of the following contributed to your decision to cancel your Fernway subscription? Please select all that apply.

    Any that apply
    • The subscription felt too expensive for my budget
    • I didn't feel it was good value for money
    • The 6-day notice needed to skip or change a box was too long
    • It was too hard to skip or pause weeks when my plans changed
    • Weekly deliveries or meal counts were too inflexible (e.g. wanted fewer meals)
    • I got tired of the recipes or wanted more variety
    • Issues with ingredient quality, freshness, or missing items
    • My household or routine changed (e.g. travelling, cooking less)
    • I only intended to try it short-term or on an introductory offer
    • I switched to another meal-kit or supermarket option
    • Other reason
  6. Q4

    Thinking about everything that led you to cancel, what was the single main reason you cancelled your Fernway subscription?

    One answer
    • Price or budget
    • Not good value for money
    • Difficulty skipping or pausing deliveries / 6-day notice period
    • Wanted fewer meals per week or less frequent deliveries
    • Recipe variety or taste
    • Ingredient quality, freshness, or delivery issues
    • Household or lifestyle change
    • Only wanted to try a short-term or discounted period
    • Switched to a competitor or supermarket
    • Other reason
  7. Q5

    How easy or difficult did you find it to skip a week or manage your deliveries with Fernway's 6-day notice rule?

    Scale

    Scale 1-5: Very difficult to Very easy

    Judges: assumptive

    Assumes respondents tried skipping or managing deliveries under the notice rule. Those who never did are still required to give an ease rating, with no not-applicable option.

    Assumes every respondent tried to skip or change a delivery under the 6-day rule. There is no 'never tried' option, so those who never skipped are forced onto the ease scale.

  8. Q6

    If Fernway were to make improvements to its subscription, how important would each of the following be to you? Please rank from 1 (most important) to 5 (least important).

    Ranking
    • One-tap skipping of any week, closer to delivery
    • Reducing the price of weekly plans by 20%
    • Automatically pausing deliveries for set holiday dates
    • Offering plans with fewer meals per week
    • Offering a wider choice of weekly recipes
  9. Q7

    If you were considering a meal-kit subscription again, which single change to Fernway would make a bigger difference to you?

    One answer
    • Cutting the weekly plan price by 20% (with the current 6-day notice to skip)
    • One-tap skipping of any week closer to delivery (at the current price)
    • Both would matter equally to me
    • Neither would make a difference to me
  10. Q8

    To help us check survey quality, please select 'Family plan (4 people)' from the list below.

    One answer
    • Couples plan (2 people)
    • Family plan (4 people)
    • I am not sure
    • Prefer not to say
  11. Q9

    If Fernway remained exactly as it was when you left, how likely would you be to re-subscribe in the next 6 months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  12. Q10

    Which of the following, if offered, would make you likely to re-subscribe to Fernway? Please select all that apply.

    Any that apply
    • A permanent 20% reduction in plan price
    • A temporary win-back discount (e.g. 30% off for a few weeks)
    • One-tap skipping for any week without a 6-day cutoff
    • Ability to choose fewer meals per week or bi-weekly delivery
    • More recipe variety or new dietary options
    • Improved ingredient freshness or delivery reliability
    • None of these—I would not re-subscribe
  13. Q11

    In your own words, what was your biggest frustration with Fernway, and what—if anything—would encourage you to return?

    Open text
    Judges: double-barrelled

    Asks two separate things (biggest frustration, and what would encourage a return) in a single answer box.

    Asks two distinct questions: biggest frustration AND what would encourage return.

  14. Q12

    Which Fernway plan were you most recently subscribed to?

    One answer
    • Couples plan (for 2 people)
    • Family plan (for 4 people)
    • Both at different times / Not sure
Sample plan

Send email invitations to the full census of UK customers in the Fernway database who cancelled their subscription in the last 6 months (excluding duplicates or anyone who has already re-subscribed, verified via S1), with up to two reminder emails to reduce non-response bias. Target an initial 600 completes split into equal quotas of n = 200 per tenure group from S2 (Under 3 months, 3 to 12 months, Over 12 months), randomising option order in Q2, Q3, Q4, Q6, and Q10. IMPORTANT PRECISION NOTE FOR THE BOARD: With n = 200 per tenure group, the margin of error per group at 95% confidence (worst-case p = 0.50) is +/- 6.9 percentage points (and +/- 4.0 percentage points for the total sample of 600). Achieving the board's requested +/- 5.0 percentage points at 95% confidence for each tenure group requires ~385 completes per group (~1,155 completes total, unless any tenure group's total 6-month churned population is small enough that finite population correction lowers the required n). Either increase the sample target to ~385 per tenure group (if the database population allows) or accept +/- 6.9pp subgroup precision at 600 completes. When reporting overall totals across all churners, apply post-stratification weights so the equal tenure quotas reflect the true tenure distribution in the CRM database.