Benchmarks / SurveyBench / Why Fernway customers cancel / Claude Fable 5.2

Measured by Spring Prompt

Claude Fable 5.2: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict: ✗ Not sound
Research score: 98 of 100
Analysis rating: 1,683
Head to head, this task: won 6 of 10

Why it is not sound

The analysis

Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Scorecard

Numbers 11 of 11 right

Question		Its answer	True
What percentage of respondents gave price as their main reason for cancelling?	✓	21.6%	21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?	✓	26.3%	26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?	✓	58.6%	58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?	✓	40	40
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?	✓	3.2 points	3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?	✓	3,193	3,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?	✓	25	25
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?	✓	35.7 points	35.7 points
What percentage of the customers invited completed the survey?	✓	6.5%	6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?	✓	49.3%	49.3%

Traps in the data 7 of 8 handled

✓Northern Ireland respondents (base 12) cite delivery problems far more often.
✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
✓'Likely to come back' answers are stated intentions about a hypothetical feature.
✓The chief executive believes customers leave mainly because of price.
✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build skipping; do not make the blanket 20% price cut. Hard-to-skip is the largest single reason for leaving on both unweighted and weighted figures, and skipping has the higher stated win-back appeal (36% against 26% overall; 59% among the roughly 3,200 cancellers it applies to). A price cut moves mainly the price leavers, and half of them say they still would not return.

Scope the build to what leavers describe, not just a button:

one-tap skip that works in the phone app;
a cut-off much closer to delivery;
a multi-week pause that keeps settings;
no phone calls or repeated confirmation screens.

Ship the cheap parts in weeks, not at the end of four months: remove the confirmation screens and send a reminder with a skip link.

Handle price with targeted, tested actions rather than a permanent cut. Price is the leading reason among under-3-month leavers, and many price comments are about rises and lapsing discounts. Test a win-back offer and a price hold or loyalty offer on the people it applies to.

In parallel, open two workstreams the survey points to that neither option fixes: menu rotation for customers past a year, and a two-meal or flexible meal-count option. Operations should check Northern Ireland delivery in their own data.

What would change it

A randomised win-back test in which 20% off brings back clearly more lapsed customers than the skip/pause message, at a margin finance can sustain.
Respondent-level reweighting (tenure by plan) that puts price level with or ahead of skipping, or shows the 20% cut matching skipping on likelihood to return.
Behavioural data showing cancellers rarely tried to skip or rarely received unwanted boxes before leaving, which would mean 'hard to skip' is a stated rather than real cause.
Evidence from current customers that price, not flexibility, is the main risk among those still subscribed.
Finance modelling showing revenue lost to easier skipping exceeds the retention gain.
A finding that most of the skipping friction can be removed in weeks rather than four months, which would free capacity to test both.

Findings

CEO question: the survey does not confirm that price is the main reason people leave. Price is the second most common main reason; difficulty skipping weeks is first. Correcting for the sample's skew towards long-tenure customers narrows the gap but does not reverse the order.

Medium confidenceBase 640

Unweighted: hard to skip 35.5% (227 of 640), price 21.6% (138 of 640, margin ±3.2pp). That 13.9pp gap is statistically clear. Weighted to the true tenure mix of cancellers (45/35/20): skip 32.4% (about 3,190 of 9,850), price 26.3% (about 2,590). Weighting cuts the effective sample to about 450 (margin about ±4pp), so the weighted 6pp lead is directional, not significant (z about 1.7).
One-tap skipping draws more stated win-back interest than a 20% price cut, and each lever mostly works only on its own group. A price cut does little for people who left over skipping, and the reverse is also true.

Medium confidenceBase 640

All respondents: 36.4% likely to return with one-tap skip against 25.8% with 20% off (10.6pp gap, significant at n=640). Skip leavers (n=227): 58.6% against 22.9%, a 35.7pp gap. Price leavers (n=138): 49.3% with the price cut against 24.6% with skip. Even among price leavers, half would not return at 20% off. Re-weighting the reason mix to the true tenure profile gives roughly 35% against 27%; this is an approximation because Q3/Q4 were not tabulated by tenure. These are stated intentions and will overstate real returns for both options.
Price matters most among early leavers, who are 45% of all cancellers but only 22% of the sample. Much of the price complaint is about rises, lapsing introductory discounts and no reward for loyalty, rather than the headline price.

Medium confidenceBase 138

Under 3 months (n=138): price 31.2% (43), skip 26.1% (36); the difference is not significant at this base. Over 12 months (n=307): price 13.7%, skip 38.8%. In the open text, 10 of 42 price comments are about price creep, the introductory offer ending, or offers reserved for new customers (e.g. R124 '£49 to £67 in under a year', R340, R325, R356).
Head of product question: the Family plan is not the problem. Its lower overall satisfaction is not statistically significant and comes from tenure mix: Family respondents are mostly long-tenure, and long-tenure leavers are the least satisfied on either plan. Within every tenure band, Family is at least as satisfied as Couples.

High confidenceBase 640

Overall: Family 49.2% satisfied (n=301), Couples 53.4% (n=339); the 4.2pp gap has z of about 1.1, not significant. Within tenure, Family against Couples: under 3 months 69.2% against 66.1% (Family base only 26), 3-12 months 61.6% against 57.4%, over 12 months 42.1% against 35.2%. None of these is significant individually, but all point the same way. 202 of 301 Family respondents (67%) were over 12 months, against 105 of 339 Couples (31%). This is satisfaction among people who cancelled, not current customers.
The real satisfaction problem is tenure. Customers who leave after more than a year are far less satisfied, and recipe repetition is their distinctive complaint. Neither one-tap skip nor a price cut addresses it.

Medium confidenceBase 307

Satisfied (4-5): under 3 months 66.7% (n=138), 3-12 months 59.0% (n=195), over 12 months 39.7% (n=307). 'Recipes got repetitive' as main reason: 6.5%, 14.4% and 25.4% respectively. Only 19.1% of repetition leavers would return for 20% off and 28.7% for one-tap skip (n=115). In the open text, 40 of 190 answers mention variety or repetition, 25 of them from over-12-month customers. Family's higher repetition share (23.6% against 13.0%) is probably the same tenure effect, but no plan-within-tenure table exists for Q1 to confirm it.
The skipping problem is wider than the number of taps. Leavers describe early cut-offs, no multi-week pause, having to phone or click through retention screens, and a skip function that fails on the phone. A one-tap button that keeps the current cut-off and has no pause would miss much of this.

Medium confidenceBase 190

48 of 190 open-text answers are about skipping, pausing or delivery frequency. Examples: cut-off too early or already locked (R245, R295, R351, R433, R527, R603); wanting a hold of weeks or months with settings kept (R049, R093, R217, R220, R372, R393, R493); phone calls or repeated confirmation screens (R089, R308, R376, R380, R530); app faults (R559, R590). Q3 tested only 'skip any week with one tap', so the 58.6% rests on respondents' own reading of what that includes.
A need the survey did not offer as a reason came up unprompted: fewer meals per week (a two-meal option or a week-to-week flexible number). It overlaps with both 'too expensive' and 'hard to skip' and could be a cheaper fix than a blanket price cut.

Low confidenceBase 190

28 of 190 open-text answers ask for fewer or flexible meals per week (e.g. R059, R098, R222, R328, R361, R459). Respondents quote different minimums (three, four, five), which suggests existing plan options may be poorly understood. Q1 had no option for this, so its true size is unknown; open-text counts show presence, not prevalence.
Operations question: the survey cannot establish a Northern Ireland delivery problem. The signal is striking but rests on 12 people. Treat it as a prompt to check operational data, not as a finding.

Low confidenceBase 12

8 of 12 Northern Ireland respondents gave delivery as their main reason (66.7%), against 50 of 628 elsewhere (8.0%). The gap is unlikely to be chance, but the plausible range on 8 of 12 is roughly 39% to 86%, and there is only one Northern Ireland open-text answer (R445: wrong day, missing items). Scotland at 13.0% (6 of 46) is not distinguishable from England at 8.1%.
Delivery failures look like an early-life problem across the UK: new customers are more than twice as likely as long-standing ones to leave over delivery.

Medium confidenceBase 640

Delivery as main reason: under 3 months 15.2% (21 of 138) against over 12 months 6.5% (20 of 307), a significant difference. Weighted to the tenure mix it is about 11% of cancellers (roughly 1,100), against 9.1% unweighted. 18 open-text answers cite late or missing boxes, missing ingredients or damage. Only 17.2% of delivery leavers would return for one-tap skip and 22.4% for a price cut (n=58).

Caveats

Low response: 640 of 9,850 invited completed (6.5%). People who reply to a win-back survey may differ from those who do not, and weighting cannot correct for that.
The sample is heavily skewed to long-tenure customers (48% over 12 months against 20% of cancellers; 22% under 3 months against 45%) and to the Family plan (47% against 34%). The tables are unweighted, so total-column figures over-represent long-tenure views. Weighting by tenure also brings the plan mix back to about 35% Family, so tenure is the skew that matters.
Tenure weighting reduces the effective sample to roughly 450. The weighted lead of skipping over price (32.4% against 26.3%) is not statistically significant, though the order holds.
Q3 and Q4 are stated intentions about hypothetical changes. They overstate real return rates and are only useful for comparing the two options, not for forecasting. They were not tabulated by tenure, so the weighted comparison (about 35% against 27%) is an approximation.
Q3 tested 'one tap' only. The open text shows the problem also covers cut-off timing, multi-week pauses and phone or retention friction, so appeal depends on what is built.
Q1 forced a single main reason from a fixed list. Several respondents cite price and skipping together (R074, R123, R380, R447, R562), and 'too many meals per week' and life changes had no option.
The survey covers only people who cancelled. It says nothing direct about what keeps current customers or attracts new ones.
The survey has no cost information, so it cannot compare the options financially. A 20% cut applies to every paying customer, and easier skipping will lose some revenue from weeks that are currently delivered unwillingly.
Small bases: Northern Ireland 12, Wales 36, Scotland 46, under-3-months Family 26, packaging leavers 31, 'Other' 28, portion leavers 43. Percentages on these are indicative at best.
Open-text answers came from 190 of 640 respondents (30%), who chose to comment. Theme counts show what issues exist and how people describe them, not how common they are. Theme coding involved judgement at the margins.
Respondents were recalling cancellations up to six months old.

Next steps

decisionApprove the skip/pause build with widened scope (one-tap skip in the phone app, later cut-off, multi-week pause keeping settings, no phone or retention screens). Decline the blanket 20% price cut for now.
experimentShip interim fixes within weeks: remove repeated confirmation and offer screens from the skip flow, add a reminder message with a skip link before cut-off, and fix the phone skip failures.
experimentRun a randomised win-back test on lapsed customers: 20% off, against a 'skip or pause any time' message (once interim fixes are live), against no offer, split by tenure and stated reason where known.
experimentTest a price hold or loyalty offer for existing customers at the end of the introductory discount and at later price rises, instead of a universal cut.
researchRe-run the survey tables with respondent-level weights (tenure by plan), adding Q3/Q4 by tenure and Q1 by plan within tenure.
researchAnalyse behaviour before cancellation and survey current customers: skip attempts, missed cut-offs, boxes delivered then refunded, phone contacts about skipping, and price-change timing against cancellation.
monitorOperations to pull Northern Ireland delivery performance (on-time rate, missing-item and non-delivery claims, refunds, courier) and compare it with Great Britain. Do the same for first-box deliveries across all regions.
researchInvestigate a two-meal or week-to-week flexible meal-count option: confirm what minimums are offered and how they are presented, then size demand among current and lapsed customers.
researchReview menu rotation for customers past 6-12 months: measure how often long-tenure customers are shown the same recipes, then trial a wider rotation.

Open-text themes it coded

Easier skipping, pausing or less frequent deliveries 48Recipe variety / repetitive menu 40Price / value for money 42Price rises, lapsing introductory offers and no loyalty reward (subset of price) 10Fewer meals per week / smaller or flexible plan 28Delivery reliability, missing items and damage 18Packaging waste / recyclability 18Portion sizes too small / adjustable portions 12Life change - nothing would have kept them 6

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

S1
When did you most recently cancel a Fernway subscription?
One answer
- In the last 6 months, and I do not currently have a Fernway subscription
- In the last 6 months, but I have since restarted my subscription
- More than 6 months ago
- I have never cancelled a Fernway subscription
- I have never had a Fernway subscription
Continues if In the last 6 months, and I do not currently have a Fernway subscription; In the last 6 months, but I have since restarted my subscription
S2
Who in your household made the decision to cancel?
One answer
- I did
- I did, jointly with someone else
- Someone else did
Continues if I did; I did, jointly with someone else
Q1
In total, how long were you a Fernway subscriber before you most recently cancelled?
One answer
- Less than 3 months
- 3 to 12 months
- More than 12 months
Q2
Thinking about the whole time you were a subscriber, how satisfied or dissatisfied were you with Fernway overall?
Scale

Scale 1-5: Very dissatisfied to Very satisfied
Q3
How satisfied or dissatisfied were you with the quality of the meals you cooked from your Fernway boxes?
Scale

Scale 1-5: Very dissatisfied to Very satisfied

Judges: assumptive
Assumes the respondent cooked meals from the boxes. Eligible cancellation decision-makers may not have cooked any, and there is no not-applicable option or display logic.
Assumes the respondent cooked meals from the boxes; someone who cancelled before receiving a box or who didn't end up cooking them has no valid option.
Q4
How would you rate the value for money of your Fernway subscription at the price you were paying?
Scale

Scale 1-5: Very poor value to Very good value
Q5
When you were a subscriber, how easy or difficult was it to skip a week's delivery?
One answer
- Very easy
- Fairly easy
- Neither easy nor difficult
- Fairly difficult
- Very difficult
- I never tried to skip a week
Q6
In your own words, what was the main reason you cancelled your Fernway subscription?
Open text
Q7
Which of these, if any, were reasons you cancelled? Please select all that apply.
Any that apply
- The price was too high for me
- My introductory discount ended
- It was too hard to skip a week when I didn't want a delivery
- I wanted to pause my subscription for a longer break
- There were more meals in the box than I needed
- I wasn't happy with the quality of the ingredients
- There wasn't enough variety in the recipes
- The meals took too long to cook
- I had problems with deliveries (late, missing or damaged)
- There was too much packaging
- I switched to another meal-kit company
- My circumstances changed (for example, I moved home or my household changed)
- Another reason not listed here
Q8
And which ONE of these was the main reason you cancelled?
One answer
- The price was too high for me
- My introductory discount ended
- It was too hard to skip a week when I didn't want a delivery
- I wanted to pause my subscription for a longer break
- There were more meals in the box than I needed
- I wasn't happy with the quality of the ingredients
- There wasn't enough variety in the recipes
- The meals took too long to cook
- I had problems with deliveries (late, missing or damaged)
- There was too much packaging
- I switched to another meal-kit company
- My circumstances changed (for example, I moved home or my household changed)
- Another reason not listed here
Q9
Thinking back to when you cancelled, which ONE of these changes, if any, would have been most likely to keep you subscribed?
One answer
- Being able to skip any week's delivery with one tap in the app
- Being able to pause my subscription for a month or more
- Being able to choose fewer meals per box
- A price 20% lower than the standard plan price
- Something else
- None of these – I would have cancelled anyway
Q10
As Fernway is today, how likely or unlikely are you to restart a Fernway subscription in the next three months?
Scale

Scale 1-5: Very unlikely to Very likely

Routing Shown only if S1 is: In the last 6 months, and I do not currently have a Fernway subscription
Q11
If Fernway let you skip any week's delivery with one tap in the app, with prices staying the same as now, how likely or unlikely would you be to restart a subscription in the next three months?
Scale

Scale 1-5: Very unlikely to Very likely

Routing Shown only if S1 is: In the last 6 months, and I do not currently have a Fernway subscription
Q12
If Fernway's plans cost 20% less than their standard price, with skipping a week still needing to be done in the app at least six days ahead, how likely or unlikely would you be to restart a subscription in the next three months?
Scale

Scale 1-5: Very unlikely to Very likely

Routing Shown only if S1 is: In the last 6 months, and I do not currently have a Fernway subscription

Sample plan

WHO IS INVITED: Every customer in the Fernway database who cancelled a subscription in the last six months gets an email invitation with a unique, single-use link. UK residence is not asked because Fernway delivers only in the UK. The unique link lets us append plan type (Couples/Family), cancellation date and recorded tenure from the database without spending questions on them. Q1 self-reported tenure is checked against the database, and the database value is used where they disagree. The invitation is worded neutrally ('help us understand your experience') and mentions neither price nor skipping. Any incentive is a prize draw for a general retail voucher, not Fernway credit, which would attract people already inclined to return. One reminder goes out after 4 days and a second after 9 days, with a two-week field period.

COMPLETES AND QUOTAS: The target is about 600 completes, with a minimum of 200 in each tenure group (under 3 months, 3-12 months, over 12 months). Cancellers probably skew towards short tenure, so reminders are targeted at whichever group is behind, most likely over 12 months. A group closes only once it is well past its minimum.

PRECISION, AND WHY 600 FALLS SHORT OF THE BOARD'S REQUIREMENT: Plus or minus 5 points at 95% confidence within each tenure group needs about 385 completes per group, or about 1,155 in total. With 200 per group the margin is about plus or minus 7 points for a result near 50%, and about plus or minus 6 points for results near 20% or 80%. The total sample of 600 gives about plus or minus 4 points. A finite-population correction helps only if a tenure group is small: 200 completes reach plus or minus 5 only where that group has roughly 420 or fewer cancellers in total. We will check the database counts before launch. Recommendation: because this is an email census with no cost per complete, do not cap at 600. Keep the survey open and aim for 385 or more per tenure group. If that is not reachable, report tenure-group results as plus or minus 7 points and tell the board so.

REPRESENTATIVENESS AND DATA QUALITY: Respondents are compared with all invited cancellers on tenure, plan and cancellation month. Results are weighted to the cancelled population by tenure and plan for total-level figures; tenure-group figures are reported within group. There is no separate attention-check item, to keep within 12 questions. Instead we remove speeders (under one-third of median time), straight-liners across Q2-Q4 and Q10-Q12, and cases where the Q8 main reason was not selected in Q7. Option order in Q7 and Q8 is randomised per respondent, with the same order in both and 'Another reason' fixed last. The first four options in Q9 are randomised. Q11 and Q12 are shown in random order, half seeing the price scenario first. Q6 (open, unprompted) always comes before the reason list so answers are not shaped by the list. Estimated length is 4.5 to 5.5 minutes.