Benchmarks / SurveyBench / Why Fernway customers cancel / Gemini 4 Carbon
Gemini 4 Carbon: Why Fernway customers cancel
The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 91 of 100
- Analysis rating
- 1,076
- Head to head, this task
- won 5 of 9
Why it is not sound
The survey
- A double-barrelled question: Q2
The analysis
- Reported a material finding the data does not support
- Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
Findings the judges found unsupported
One-tap skipping outperforms a 20% price cut on stated win-back while avoiding permanent revenue cannibalisation across active subscribers.
The statement that a discount 'wins back only 49.3%' converts hypothetical stated likelihood into actual reactivation. The data show that 49.3% of price-reason respondents say they would be likely to return, not that this proportion would return. The stated-intention caveats elsewhere are appropriate but do not support this particular claim.To answer the Operations Director: Northern Ireland shows an acute delivery failure signal, though the tiny survey base requires immediate verification in courier logs.
The finding calls it an 'acute delivery failure signal' from only 8 of 12 respondents. It then argues the small-base warning away with a significance claim (p < 0.0001), which ignores self-selection and multiple regional comparisons. So a 12-person result is presented as a real regional failure.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents gave price as their main reason for cancelling? | ✓ | 21.6% | 21.6% |
| Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason? | ✓ | 26.3% | 26.3% |
| Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap? | ✓ | 58.6% | 58.6% |
| How many of the open-text answers mention recipe variety or repetitive recipes? | ✓ | 40 | 40 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason? | ✓ | 3.2 points | 3.19 points |
| Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason? | ✓ | 3,193 | 3,193 |
| How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes? | ✓ | 25 | 25 |
| Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut? | ✓ | 35.7 points | 35.7 points |
| What percentage of the customers invited completed the survey? | ✓ | 6.5% | 6.5% |
| Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less? | ✓ | 49.3% | 49.3% |
Traps in the data 7 of 8 handled
- ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
- ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
- ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
- ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
- ✓The chief executive believes customers leave mainly because of price.
- ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
- ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
- ✓The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.
The analysis
Build one-tap skipping rather than cutting plan prices by 20%—and ensure the 4-month build covers multi-week holiday pausing, mobile bug fixes, removal of multi-screen confirmation friction, and pre-cutoff skip reminders. Skipping friction is the #1 cancellation driver both unweighted (35.5%) and tenure-weighted (32.4%, ~3,193 cancellers), generates higher stated win-back than a 20% price cut (36.4% vs 25.8% overall; 58.6% vs 22.9% among skip-leavers), and avoids permanently sacrificing 20% of revenue across all retained subscribers. Alongside the build, use targeted retention/step-down offers (rather than a blanket price cut) for early-tenure price sensitivity, expand recipe rotation for >12-month/Family subscribers, and audit Northern Ireland courier performance immediately.
Financial modelling or a controlled holdout experiment showing that: (1) frictionless skipping increases active-customer skip frequency enough that lost weekly orders exceed the LTV saved from lower churn and higher win-back, OR (2) Fernway's contribution margins and price elasticity allow a 20% price cut (or targeted lower-priced 2-meal tier) to generate higher net contribution profit after accounting for revenue loss on existing subscribers.
Findings
-
To answer the CEO: price is not the main reason customers cancel; difficulty skipping weeks is #1, even after correcting for severe tenure skew in the survey sample.
Unweighted, 35.5% cite 'hard to skip weeks' vs 21.6% (±3.2pp) citing price (base 640). Because under-3-month cancellers are under-represented in the sample (21.6% vs 45% of all 9,850 cancellers) and are more price-sensitive (31.2% vs 13.7% among >12-month cancellers), reweighting to population tenure lifts price to 26.3% (~2,588 cancellers), but skipping friction still leads at 32.4% (~3,193 cancellers).
-
One-tap skipping outperforms a 20% price cut on stated win-back while avoiding permanent revenue cannibalisation across active subscribers.
Across all respondents (base 640), 36.4% say they are likely (4-5) to return for one-tap skipping vs 25.8% for a 20% price cut (and ~35.0% vs ~27.1% when reweighted by population cancellation-reason mix). Among skip-reason cancellers (base 227), one-tap skipping beats a 20% price cut by 35.7pp (58.6% vs 22.9%). Even among price cancellers (base 138), a 20% discount wins back only 49.3%, while sacrificing 20% of revenue on every retained box.
-
To answer the Head of Product: the Family plan does not have a satisfaction problem; its lower aggregate score is a Simpson's paradox driven by tenure mix, though long-tenured Family users suffer acute menu fatigue.
Aggregate satisfaction (4-5) on Family (49.2%, base 301) is not statistically different from Couples (53.4%, base 339; 4.2pp gap, z = 1.06, p = 0.29). Within every tenure band, Family satisfaction actually exceeds Couples: <3 months 69.2% vs 66.1%, 3-12 months 61.6% vs 57.4%, and >12 months 42.1% vs 35.2%. Family looks lower overall only because 67.1% of Family respondents (202/301) had >12 months tenure, where satisfaction drops sharply (39.7%) alongside repetitive-recipe churn (25.4% in >12 months; 23.6% on Family vs 13.0% on Couples).
-
To answer the Operations Director: Northern Ireland shows an acute delivery failure signal, though the tiny survey base requires immediate verification in courier logs.
66.7% (8 of 12) of Northern Ireland respondents cancelled primarily due to delivery problems, compared with 8.1% in England (base 546), 13.0% in Scotland (base 46), and 0.0% in Wales (base 36). Despite the small base (n = 12), 8/12 against an ~8% rest-of-UK rate is highly unlikely by chance (p < 0.0001), corroborated qualitatively by reports of late boxes and missing ingredients (R445).
-
Building 'one-tap skip' strictly as a UI shortcut will miss half of the flexibility problem: customers also leave over rigid cut-off deadlines, multi-week holiday caps, and high minimum meal counts per box.
Across 190 open-text responses, 50 cite skipping/pausing friction—not only multi-screen dark patterns, phone-only cancellation, and app crashes (R376, R380, R530, R559), but also early Thursday cut-offs without reminders (R245, R295, R433, R603) and inability to pause for multi-week holidays (R084, R217, R220, R493). Additionally, 29 respondents unprompted said a 1–2 meal/week option (instead of a 3+ meal minimum) would have kept them—an issue absent from Q1's closed list that pushes overwhelmed households into skipping or cancelling.
-
Price-driven churn is concentrated in the first year and is often triggered by cliff-edge introductory discount expiry and mid-life price rises rather than baseline menu pricing alone.
Price accounts for 31.2% of cancellations under 3 months (base 138) and 27.2% at 3-12 months (base 195), falling to 13.7% after 12 months (base 307). Open-text price answers (42 of 190) frequently cite steep post-introductory price jumps, unannounced price creep (e.g., £49 to £67), and lack of loyalty offers for existing subscribers (R077, R124, R158, R296, R340, R356, R500).
Caveats
- Severe sample skew and low completion rate (6.5%; 640 of 9,850): Customers over 12 months tenure make up 48.0% of respondents vs 20% of the cancelled population, and Family makes up 47.0% vs 34%. Unweighted totals overstate long-tenure issues (repetitive recipes) and understate early-tenure price churn.
- Stated win-back intent massively overstates actual reactivation: 'Fairly/very likely' (4-5) in Q3/Q4 typically converts at a fraction of face value (especially 'fairly likely'); Q3 and Q4 were also asked sequentially without randomisation, creating anchoring and order effects.
- Active subscribers were not surveyed: The survey cannot measure how much a 20% price cut would retain current customers vs cannibalise margin, nor how much one-tap skipping would reduce weekly order frequency (skip rate) among active customers who currently tolerate friction.
- Q1 omitted box/meal-count size: 29 of 190 open-text answers cited too many meals per week (wanting 1-2 meals instead of 3+ minimum). Because Q1 lacked this option, some of those respondents likely selected 'hard to skip weeks' or 'price'.
- Small regional sub-samples: Northern Ireland (n = 12), Wales (n = 36), and Scotland (n = 46) have wide margins of error; NI findings are directional triggers for operational log audits, not precise population estimates.
Next steps
- decisionCommit to the one-tap skipping product build, scoping it to include mobile stability, removal of dark-pattern confirmation hurdles, multi-week pause/fortnightly cadence, and 24-hour pre-cutoff push/SMS skip links.
- experimentRun a controlled A/B test on active subscribers before full rollout (e.g., fast-track removing existing multi-screen skip confirmations and adding SMS cutoff reminders) to measure the trade-off between higher skip rates and lower churn.
- experimentTest targeted win-back and early-tenure retention offers (graduated intro-discount step-down at weeks 4–12, or a 2-meal/week down-sell option at cancellation) instead of a 20% blanket price cut.
- researchAudit courier on-time, missing-ingredient, and damage logs across all Northern Ireland deliveries (and Scotland) over the past 6 months.
- monitorIncrease recipe rotation and cuisine/vegetarian variety targeted at >12-month and Family subscribers, and track cohort satisfaction separately by tenure.
Open-text themes it coded
Easier skipping, multi-week pausing, and later cut-off deadlines 49Lower prices, price stability, and loyalty discounts 42Recipe variety and less repetitive menus 40Smaller boxes and flexible meal counts per week (1-2 meal option) 29Less plastic and council-recyclable packaging 19Delivery reliability and complete, undamaged orders 18Larger or customisable portion sizes 12Unavoidable personal or household life changes 6
The survey it planned
3 screening questions and 12 questions, as the model wrote them.
-
S1
Which of the following best describes your household's current status with Fernway?
One answer- I currently have an active Fernway subscription
- I had a Fernway subscription and cancelled it within the last 6 months
- I had a Fernway subscription and cancelled it more than 6 months ago
- I have never had a Fernway subscription
Continues if I had a Fernway subscription and cancelled it within the last 6 months
-
S2
How involved were you in managing your household's Fernway subscription and deciding to cancel it?
One answer- I made the decision on my own
- I shared the decision equally with someone else in my household
- Someone else in my household was mainly responsible
Continues if I made the decision on my own; I shared the decision equally with someone else in my household
-
S3
Roughly how long were you subscribed to Fernway before your most recent cancellation?
One answer- Under 3 months
- 3 to 12 months
- Over 12 months
- I cannot remember
Continues if Under 3 months; 3 to 12 months; Over 12 months
-
Q1
Thinking about your overall experience while subscribed, how satisfied or dissatisfied were you with Fernway?
ScaleScale 1-5: Very dissatisfied to Very satisfied
-
Q2
While subscribed, how satisfied or dissatisfied were you with the taste, quality, and weekly variety of Fernway's recipes and ingredients?
ScaleScale 1-5: Very dissatisfied to Very satisfied
Judges: double-barrelledA single satisfaction rating combines taste, quality and weekly variety, which respondents could rate differently.
Asks about taste, quality and weekly variety together but allows one rating. A respondent could like the taste and still find the variety poor.
-
Q3
Which of the following contributed to your decision to cancel your Fernway subscription? Please select all that apply.
Any that apply- The standard weekly price was too high for our budget
- An introductory discount ended and the full price did not feel worth it
- Having to skip or change deliveries at least 6 days ahead was too restrictive
- It was inconvenient to pause the subscription when our plans changed
- We wanted the option to order fewer meals per week
- Too much food went unused or portion sizes did not suit us
- We were disappointed with recipe variety, taste, or ingredient freshness
- Delivery problems (such as late arrivals or missing/damaged items)
- Our household finances, schedule, or living arrangements changed
- Other reason
-
Q4
Which SINGLE reason was the most important in your decision to cancel?
One answer- The standard weekly price was too high for our budget
- An introductory discount ended and the full price did not feel worth it
- Having to skip or change deliveries at least 6 days ahead was too restrictive
- It was inconvenient to pause the subscription when our plans changed
- We wanted the option to order fewer meals per week
- Too much food went unused or portion sizes did not suit us
- We were disappointed with recipe variety, taste, or ingredient freshness
- Delivery problems (such as late arrivals or missing/damaged items)
- Our household finances, schedule, or living arrangements changed
- Other reason
-
Q5
While subscribed to Fernway, how often did you end up paying for a weekly box you did not want because the 6-day deadline to skip or change your delivery had already passed?
One answer- Never — I always skipped in time or wanted every delivery
- Once
- 2 or 3 times
- 4 or more times
- I did not know I could skip weekly deliveries in the app
-
Q6
Imagine you were choosing a meal-kit subscription today. Please rank the following features from 1 (most important to your household) to 5 (least important to your household).
Ranking- Weekly plan prices that are 20% lower than Fernway's current prices
- One-tap skipping of any delivery week without needing 6 days' advance notice
- Easy pausing of the subscription for several weeks at a time
- Flexibility to order fewer meals per week
- A wider weekly choice of recipes
-
Q7
Suppose Fernway introduced one-tap skipping of any week in the app (removing the 6-day advance notice requirement), while keeping weekly plan prices the same as before. How likely would you be to re-subscribe within the next 3 months?
ScaleScale 1-5: Definitely would not re-subscribe to Definitely would re-subscribe
-
Q8
Suppose instead that Fernway cut weekly plan prices by 20%, while keeping the requirement to skip or change deliveries in the app at least 6 days ahead. How likely would you be to re-subscribe within the next 3 months?
ScaleScale 1-5: Definitely would not re-subscribe to Definitely would re-subscribe
-
Q9
If Fernway could make ONLY ONE of these changes today, which would make you more likely to come back as a subscriber?
One answer- One-tap skipping of any week in the app (at standard plan prices)
- A 20% cut in weekly plan prices (keeping the 6-day notice to skip)
- Something else would matter more to me than either of these
- Nothing — I would not re-subscribe to Fernway under any circumstances
-
Q10
What single change would make you most likely to re-subscribe to Fernway?
Open textRouting Shown only if Q9 is: Something else would matter more to me than either of these
-
Q11
Which Fernway plan were you subscribed to immediately before you cancelled?
One answer- Couples plan (box for two)
- Family plan (box for four)
- Not sure
-
Q12
Since cancelling Fernway, how has your household mainly replaced those weekly meals?
One answer- Switched to another meal-kit subscription
- Bought supermarket groceries to cook from scratch
- Used supermarket ready meals, takeaways, or dining out
- Other
Sample plan
WHO WILL BE INVITED: Email invitations with unique, single-use survey links will be sent to all UK customers in the Fernway database who cancelled in the past 6 months and have not reactivated, stratified by CRM-recorded subscription tenure (<3 months, 3-12 months, >12 months) and plan type (Couples vs Family). Up to two reminder emails will be sent at 3-day and 7-day intervals across varied times of day to reduce non-response bias from less engaged early cancellers. SAMPLE SIZE & QUOTA RESOLUTION: There is a statistical conflict between the brief's 'about 600 completes' and the board's precision target of +/-5 percentage points at 95% confidence WITHIN each of the three tenure groups. Achieving +/-5pp per subgroup requires n=385 completes per tenure band (N=1,155 total completes across 3 equal quotas, or slightly fewer in smaller cohorts such as >12 months if finite population correction applies). If capped at N=600 (equal quotas of n=200 per tenure band), subgroup precision will be +/-6.9pp at 95% confidence (with +/-4.0pp overall). We recommend setting stratified quotas of n=385 per tenure band (N=1,155) if the 6-month cancellation pool and ~10-15% email response rate support it; if N=600 is a hard cap, enforce quotas of n=200 per tenure band and state the +/-6.9pp subgroup margin of error to the board. BIAS & QUALITY CONTROLS: Randomise option order in Q3, Q4, Q6, and Q9 (anchoring 'Other'/'Nothing' last) and randomise the presentation order of Q7 and Q8 (50/50 split) to prevent priming toward price. Exclude speeders (<90 seconds) and cross-check S3/Q11 against CRM records. Rim-weight total-sample estimates back to the true database distribution of 6-month cancellers by tenure and plan type.