Benchmarks / SurveyBench / Why Fernway customers cancel / Claude Opus 5.5

Measured by Spring Prompt

Claude Opus 5.5: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✓ Sound research
Research score
97 of 100
Analysis rating
1,646
Head to head, this task
won 10 of 11

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 8 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✓The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Prioritise flexibility over a blanket 20% price cut, but do not commit the full four months blind.

Act now on flexibility:

  • Fix the cheap, specific skip faults immediately: later skip deadline, mobile app crash, hidden skip option, repeated confirmation and discount screens, skip caps, phone-only pausing.
  • Ship a minimal version quickly: a skip link in the weekly reminder email or text, and a multi-week pause that keeps preferences.
  • Test that against churn before building the full one-tap feature.

Do not cut list prices by 20%. Price matters to about a quarter of cancellers and about half of those say a cut would bring them back. But a blanket cut gives margin away to every retained customer, while the open text points to targeted fixes. Test those instead:

  • price freezes or caps on rises for existing customers;
  • loyalty or ongoing discounts rather than new-customer-only offers;
  • a cheaper 2-meal or fortnightly plan, which also tackles the large 'too many meals' theme.

Beyond the two options:

  • Treat recipe variety as a priority for long-tenure customers, who are the least satisfied group.
  • Leave the Family plan alone on satisfaction grounds; the apparent gap comes from tenure mix and is not significant.
  • Investigate Northern Ireland delivery through operational data rather than acting on 12 responses.
What would change it

Any of the following would change the recommendation:

  • Behavioural evidence that skipping is not the real problem: few failed skip attempts in product analytics, or quick skip fixes and a minimal skip link that do not reduce cancellation among active customers in an A/B test. That would weaken the case for the full four-month build.
  • A win-back or retention price test showing that a 20% (or smaller) discount reactivates or retains enough customers to cover the margin given up on the whole base. That would strengthen price.
  • Evidence that short-tenure cancellers dominate value lost and are mainly price-driven, which would shift the balance toward price. They are 45% of cancellers and cite price most.
  • A properly designed follow-up showing that 'too many meals' or variety outranks both levers once offered as an answer. That would redirect investment to a smaller plan or menu rotation.

Findings

  1. Difficulty skipping weeks, not price, is the most common main reason for cancelling. This holds after correcting the sample's tenure skew. Price is a real but secondary driver, at roughly a quarter of cancellers.

    Medium confidenceBase 640

    Unweighted, 35.5% said hard to skip weeks (±3.7pp) and 21.6% said price (±3.2pp), base 640. Weighting to the population tenure mix (45/35/20) gives 32.4% for skipping, about 3,190 of 9,850 cancellers, and 26.3% for price. Weighting narrows the gap because short-tenure cancellers cite price more (31.2%, base 138) and are under-represented in the sample. Skipping still leads in the 3-12 month band (36.9% v 27.2%, base 195) and the over-12 month band (38.8% v 13.7%, base 307). Under 3 months it is roughly level (26.1% v 31.2%, base 138).

  2. One-tap skipping is the stronger win-back lever overall. Each lever mainly appeals to the people whose problem it fixes, so neither wins back the whole lapsed base.

    Medium confidenceBase 640

    Likely (4-5) to return across all respondents (base 640): 36.4% with one-tap skip v 25.8% with a 20% price cut. Among skip leavers (base 227): 58.6% with skip v 22.9% with the price cut, a gap of 35.7pp. Among price leavers (base 138): 49.3% with the price cut v 24.6% with skip. Delivery, packaging, portion and 'other' leavers respond weakly to both (7-28%).

  3. A 20% across-the-board cut is a poor fit for what price leavers actually describe. Their complaints are about price rises, introductory discounts ending and no loyalty reward, which points to targeted pricing rather than a blanket cut.

    Low confidenceBase 190

    42 of 190 open answers mention price or value. About 14 of these concern price creep, discounts vanishing after the intro period, new-customer-only offers or wanting loyalty or low-income discounts (e.g. R124 £49 to £67, R296, R325, R356, R500, R340). Several name modest thresholds: about £5 or a tenner less (R449, R181), 15% off (R286), under £40 (R454). A 20% cut would also apply to every retained customer, not just at-risk ones.

  4. There is an unmeasured third driver: too many meals per week, or deliveries too frequent. A smaller or flexible plan (2 meals, fortnightly) could address both skip and price complaints. It is not a Q1 option, so it is hidden inside other answers.

    Low confidenceBase 190

    31 of 190 open answers ask for fewer meals, a 2-meal or 1-meal option, a variable meal count or less frequent deliveries (e.g. R059, R098, R222, R328, R361, R436). Q1 had no such option, so these people were probably coded as skip, price or portions.

  5. The Family plan is not the problem. Its lower headline satisfaction comes from Family respondents being disproportionately long-tenure customers, who are less satisfied on every plan. Within each tenure band Family is as satisfied as Couples or more.

    Medium confidenceBase 640

    Satisfied (4-5): Family 49.2% (base 301) v Couples 53.4% (base 339). The gap of 4.2pp gives z≈1.06, not significant at 95%. Within tenure, Family is higher in every band: under 3 months 69.2% v 66.1% (bases 26/112), 3-12 months 61.6% v 57.4% (73/122), over 12 months 42.1% v 35.2% (202/105). 202 of the 301 Family respondents are over-12-month customers.

  6. Long-tenure customers are the least satisfied, and recipe repetition is their distinctive complaint. Menu variety is a retention issue for the most valuable customers that neither proposed option addresses.

    Medium confidenceBase 307

    Satisfied (4-5): 39.7% for over 12 months (base 307) v 66.7% for under 3 months (base 138). Recipes got repetitive as main reason: 25.4% for over 12 months v 6.5% for under 3 months. 40 of 190 open answers mention variety or repetition, 25 of them from over-12-month customers.

  7. The skip problem is partly a set of specific, fixable faults rather than only a missing feature. Quick fixes may capture much of the benefit before a four-month build.

    Medium confidenceBase 190

    46 of 190 open answers mention skipping or pausing. Specific faults cited: the skip deadline is too early (R245, R295, R351, R433, R603), the app crashes or will not allow skipping on mobile (R559, R590), the skip option is hidden or there are repeated 'are you sure' and discount screens (R376, R527, R530), skips are capped or must be done week by week (R217, R220), and pausing requires a phone call (R089, R380). Others want a multi-week pause that keeps preferences (R049, R372, R393, R493).

  8. The survey cannot confirm a Northern Ireland delivery problem. It is a signal worth checking in operational data, not a finding.

    Low confidenceBase 12

    8 of 12 NI respondents (66.7%) gave delivery as their main reason, against 9.1% overall. The base of 12 gives a 95% interval of roughly 39-86%. Only one NI open answer (R445) exists, and it does mention late and incomplete boxes.

Caveats

  • Only 6.5% of the 9,850 invited cancellers completed (640; 702 started). Lapsed customers who answer a brand's survey are likely more engaged and more open to returning than those who did not, so all 'likely to come back' figures probably overstate real behaviour.
  • The sample is skewed. Over-12-month customers are 48% of the sample v 20% of cancellers, and Family is 47% v 34%. Unweighted totals over-represent long-tenure views (skip, variety) and under-represent short-tenure views (price, delivery). Only tenure weighting was possible because the plan-by-tenure population mix is not available. Plan weighting moves price slightly the other way (Couples 23.6% v Family 19.3%), so weighted price is best read as roughly 25-26%.
  • Q3 and Q4 are stated intentions about hypothetical changes. Stated intent typically overstates actual reactivation by a wide margin. Treat them as relative signals, not forecasts of win-back volumes.
  • Q3 (skip) was always asked before Q4 (price), and the Q3 wording is vivid and specific. Order and framing effects may favour skip. The comparison is directional, not precise.
  • Q1 forced a single main reason and had no option for 'too many meals or too frequent'. Many open answers combine reasons (e.g. skip plus price, variety plus skip), so shares by reason understate overlap.
  • Open-text answers come from 190 of 640 respondents (30%) who chose to write something. Theme counts show what matters to people, not population prevalence. Coding was done by one analyst.
  • Subgroups are small: Northern Ireland has 12 respondents, Under-3-months Family 26, Wales 36, and the Q1 reason groups for portions, packaging and other are each 28-43. No conclusions should be drawn from these alone.
  • Margins of error assume simple random sampling, which this self-selected sample is not. True uncertainty is larger than the quoted ±3-4pp.
  • The survey covers win-back of past cancellers. It says little directly about keeping current customers, which is half of the stated goal and where a blanket price cut is most expensive.

Next steps

  • decisionShip quick skip fixes now: later skip cut-off, fix the mobile skip crash, remove repeated confirmation and discount screens, lift the multi-week skip cap, allow pausing in-app without phoning.
  • experimentA/B test a minimal skip product (a 'skip this week' link in reminder email or text, plus a pause of up to 8 weeks that keeps preferences) with active customers for 6-8 weeks before committing four months to full one-tap skipping.
  • experimentRun targeted win-back offers to lapsed customers in randomised cells: no offer, 10% off, 20% off, and a price freeze or loyalty rate. Stratify by tenure and by stated cancel reason where known.
  • experimentTest a smaller plan (2 meals a week and/or fortnightly delivery) with at-risk and lapsed customers.
  • researchPull delivery-failure, late-delivery and missing-item rates for Northern Ireland postcodes versus Great Britain, and review the courier partner.
  • experimentAnalyse recipe repetition for long-tenure customers (how often the same dish recurs) and trial a broader rotation or 'new this week' slots for over-12-month subscribers.
  • researchFix the exit survey and re-field it. Add 'too many meals or too frequent' to Q1, allow multiple reasons plus a main reason, randomise the order of Q3 and Q4, and weight to population tenure and plan.
  • monitorTrack weekly cancellation reasons, skip attempts and failures, and satisfaction by tenure and plan (with tenure controlled when comparing plans) on a dashboard.

Open-text themes it coded

Easier skipping / pausing / holds 46Price, value, price rises and loyalty or ongoing discounts 42Recipe variety / repetitive menu 40Too many meals: fewer meals per week, flexible meal count or less frequent delivery 31Portions too small / choose portion size 12Delivery reliability, missing or damaged items 18Packaging waste / non-recyclable packaging 18Life change: nothing would have kept them 6

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Which of these best describes you and Fernway right now?

    One answer
    • I have a Fernway subscription that I have never cancelled
    • I cancelled my Fernway subscription in the last six months and have not restarted it
    • I cancelled my Fernway subscription in the last six months and have since restarted it
    • I cancelled my Fernway subscription more than six months ago
    • I have never had a Fernway subscription

    Continues if I cancelled my Fernway subscription in the last six months and have not restarted it

  2. S2

    How long had you been a Fernway subscriber when you cancelled?

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months
    • Not sure

    Routing Shown only if S1 is: I cancelled my Fernway subscription in the last six months and have not restarted it

    Continues if Less than 3 months; 3 to 12 months; More than 12 months; Not sure

  3. Q1

    In your own words, why did you cancel your Fernway subscription?

    Open text
  4. Q2

    Which of these, if any, were reasons you cancelled? Select all that apply.

    Any that apply
    • The price was more than I wanted to spend
    • It wasn't good value for what I got
    • My household budget got tighter
    • I couldn't skip or change a delivery when I needed to
    • Skipping or changing a delivery was too much hassle
    • I didn't need a box every week
    • There was too much food or too many meals
    • The plan size didn't suit my household
    • Not enough variety in the recipes
    • The food quality or freshness wasn't good enough
    • Problems with deliveries (late, missing or damaged items)
    • The recipes took too long or were too hard to cook
    • Too much packaging
    • I switched to another meal kit or food delivery service
    • I only joined for an introductory offer
    • A change in my circumstances (for example, moving home, a new job or health)
    • Something else
  5. Q3

    Which ONE of these was the main reason you cancelled?

    One answer
    • The price was more than I wanted to spend
    • It wasn't good value for what I got
    • My household budget got tighter
    • I couldn't skip or change a delivery when I needed to
    • Skipping or changing a delivery was too much hassle
    • I didn't need a box every week
    • There was too much food or too many meals
    • The plan size didn't suit my household
    • Not enough variety in the recipes
    • The food quality or freshness wasn't good enough
    • Problems with deliveries (late, missing or damaged items)
    • The recipes took too long or were too hard to cook
    • Too much packaging
    • I switched to another meal kit or food delivery service
    • I only joined for an introductory offer
    • A change in my circumstances (for example, moving home, a new job or health)
    • Something else
  6. Q4

    While you were subscribed, did you ever try to skip a week's delivery?

    One answer
    • Yes, and I managed to skip it
    • Yes, but it was too late to skip that week
    • Yes, but I couldn't work out how to do it
    • No, I didn't know I could skip a week
    • No, I never needed to
    • Can't remember
  7. Q5

    While you were subscribed, how often did you receive a box that you would rather not have had that week?

    One answer
    • Never
    • Less than once a month
    • About once a month
    • 2 to 3 times a month
    • Most weeks
    • Can't remember
  8. Q6

    Overall, how satisfied or dissatisfied were you with Fernway while you were subscribed?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  9. Q7

    To show you are reading carefully, please select "Fairly often" for this question.

    One answer
    • Never
    • Rarely
    • Fairly often
    • Very often
  10. Q8

    How satisfied or dissatisfied were you with the value for money of your Fernway plan?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  11. Q9

    How satisfied or dissatisfied were you with how easy it was to skip or change your deliveries?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: assumptive

    One rating covers both skipping and changing deliveries, which respondents could have found differently easy. Everyone sees it, including people who never used these functions or did not know skipping was possible, but there is no not-applicable or don't-know response.

    Shown to everyone, but assumes the respondent tried to skip or change a delivery. Q4 allows 'never needed to' and 'didn't know I could', yet Q9 has no 'never tried / not applicable' option. Those respondents are forced to rate something they never experienced, which distorts the flexibility satisfaction measure.

  12. Q10

    How likely or unlikely are you to subscribe to Fernway again in the next six months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  13. Q11

    Which of these, if any, would make you consider subscribing to Fernway again? Select up to three.

    Any that apply
    • A lower price
    • Skipping any week with one tap, up to the day before delivery
    • Pausing deliveries for several weeks at a time
    • Choosing fewer meals per week
    • Having a box every other week
    • More variety in the recipes
    • Better food quality or freshness
    • More reliable deliveries
    • Quicker or easier recipes
    • Less packaging
    • A discount on my first few boxes back
    • Something else
    • Nothing would make me subscribe again
  14. Q12

    Imagine Fernway made one of the following changes. Which ONE would be more likely to make you subscribe again?

    One answer
    • The price stays the same, and you can skip any week with one tap in the app, up to the day before delivery
    • The price is 20% lower than the price you paid, and skipping a week works as it does now (change your delivery in the app at least six days ahead)
    • Neither would make me more likely to subscribe again
    • Don't know
Sample plan

WHO: Email an invitation to every customer whose Fernway subscription was cancelled in the last six months, taken from the customer database. Exclude people who have opted out of research contact and Fernway staff. Run a de-duplication check. Each invitation link carries a unique ID. The ID appends database tenure at cancellation, plan (Couples or Family), cancellation date and app skip history. Database tenure is used for quotas and analysis. S2 is used only as a check, and 'Not sure' answers are resolved from the database. People who say they have restarted (S1) are screened out and counted. Their number is a useful win-back figure.

PRECISION CONFLICT, FOR THE BOARD TO RESOLVE BEFORE FIELDWORK: The budgeted 600 completes cannot deliver plus or minus 5 points per tenure group. Plus or minus 5 points at 95% confidence (worst case, p=0.5) needs about 385 completes per group. That is about 1,155 in total, with quotas of 385 each for under 3 months, 3-12 months and over 12 months. If any group's total number of cancellers (N) is small, use the finite population correction n = 385 / (1 + 384/N) for that group. For example, N = 1,500 needs about 307. If the budget stays at 600, set equal quotas of 200 per group. That gives about plus or minus 6.9 points per group, and the board must be told this. A proportional 600 would be worse still for the smallest group, likely the over-12-month leavers. Recommended plan: raise the target to about 1,155 with equal quotas. Overall figures are weighted back to the real tenure mix of cancellers.

FIELDWORK: Send one invitation and two reminders over about two weeks. Close each quota when it is full. Offer a neutral incentive, such as a prize draw for a supermarket gift card. Do not offer Fernway credit, because it would bias win-back answers. The survey has 2 screener questions and 12 main questions, about 4 to 5 minutes on mobile.

PROGRAMMING: Q2 and Q11 list order is randomised, with 'Something else' and 'Nothing would…' anchored last. 'Nothing would…' is exclusive. Q11 allows a maximum of 3 choices. Q3 shows only the options chosen at Q2 and is skipped (auto-coded) if only one was chosen. Q12 randomises the order of its first two options. Q1 is open-ended and comes first, so the reasons list cannot prompt the answers.

QUALITY: Remove respondents who fail Q7, speeders (under one third of the median time) and straight-liners.

NEUTRALITY: The questionnaire gives price and flexibility equal, neutral treatment. It is designed to find out why people leave, not to confirm a pre-chosen answer. Results will be reported as they come out.