Benchmarks / SurveyBench / Why Fernway customers cancel / DeepSeek V4.1 Flash 1017

Measured by Spring Prompt

DeepSeek V4.1 Flash 1017: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
90 of 100
Analysis rating
1,221
Head to head, this task
won 11 of 17

Why it is not sound

The analysis

  • Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 6 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Do not make a blanket 20% price cut the primary retention investment. The evidence favours building one-tap skipping, and it should be scoped to include easy pause/multi-week skip and, if feasible, variable meal counts, because open text shows flexibility demand is broader than a single skip action. Treat price as a targeted save offer for price-leavers rather than a universal cut: only about half of price-leavers say 20% off would likely bring them back, and price is not the largest reason overall. In parallel, audit Northern Ireland delivery performance operationally and address menu variety/rotation for long-tenure customers, who drive the low-satisfaction and repetition signal.

What would change it

A behavioural test showing that a 20% price cut produces materially higher actual reactivation and retained revenue than a skip/pause prototype would change the recommendation. So would evidence that the skip-blocked segment is much smaller or harder to reach than stated, that one-tap skipping does not convert to actual reactivation, or that the Northern Ireland delivery issue is systemic and large enough to dominate retention economics.

Findings

  1. The CEO premise is only partly supported: price is a real cancellation driver, but it is not the largest stated reason. Difficulty skipping weeks is the biggest single reason overall.

    High confidenceBase 640

    Q1 total: hard to skip weeks 35.5% vs price 21.6%. Weighted to the tenure mix of all 9,850 cancelled customers, price rises to about 26.3% and hard-to-skip is about 32.4%. Price is more concentrated among shorter-tenure cancellers (31.2% under 3 months, 27.2% 3-12 months, 13.7% over 12 months).

  2. Among customers who cancelled because skipping was hard, one-tap skipping has far stronger stated recovery appeal than a 20% price cut.

    Medium confidenceBase 227

    Q3 hard-to-skip base 227: net likely to return with one-tap skipping 58.6% (fairly 34.4% + very 24.2%). Q4 same base: net likely with 20% lower price 22.9%. Gap 35.7 percentage points.

  3. Across all cancellers, one-tap skipping tests better than a 20% price cut, but both are hypothetical intentions rather than observed behaviour.

    Medium confidenceBase 640

    Q3 total net likely 36.4% vs Q4 total net likely 25.8%. Q3 very likely 14.4%, Q4 very likely 11.4%.

  4. Price leavers are not a monolithic group: a 20% cut would be likely to bring back only about half of them, and only about one in five as 'very likely'.

    Medium confidenceBase 138

    Q4 among Q1 price leavers: net likely 49.3% (fairly likely 27.5% + very likely 21.7%); base 138. Very unlikely 15.2%, fairly unlikely 19.6%, neither 15.9%.

  5. The Family plan is not the problem the head of product suspects; its satisfaction gap vs Couples is small and not statistically significant.

    High confidenceBase 640

    Q2 net satisfied: Couples 53.4% (base 339) vs Family 49.2% (base 301), a 4.2-point gap; two-proportion z about 1.06, not significant at 95%. Within tenure, Family is not lower than Couples: under 3 months 69.2% vs 66.1%, 3-12 months 61.6% vs 57.4%, over 12 months 42.1% vs 35.2%.

  6. Low satisfaction and menu-variety complaints concentrate among longer-tenure customers, which is a retention/product issue rather than a Family-plan issue.

    High confidenceBase 307

    Q2 net satisfied: under 3 months 66.7%, 3-12 months 59.0%, over 12 months 39.7%. Open text: 25 of the 40 recipe-variety/repetition mentions come from over-12-month respondents.

  7. Northern Ireland shows a strong delivery red flag, but the base is too small to treat as proven.

    Low confidenceBase 12

    Q1 Northern Ireland base 12: delivery problems 66.7% (8 of 12) vs 9.1% total. One NI open-text respondent (R445) also cites delivery failure/missing items and packaging. Cannot infer a population rate from n=12.

  8. Open text reinforces that 'flexibility' is broader than one-tap skipping alone: many answers ask for pause/hold, multi-week skip, and fewer or variable meals per week.

    Medium confidenceBase 190

    Among 190 open-text answers, 46 mention skip/pause/hold/schedule flexibility and 31 mention meal-count/plan-size/frequency; these overlap. One-tap skip alone may not satisfy customers who want pause, fortnightly boxes, or two-meal weeks.

  9. The unweighted survey over-represents long-tenure and Family cancellers, so unweighted Q1 understates price and overstates long-tenure product complaints relative to the cancelled customer base.

    High confidenceBase 640

    Sample profile: over 12 months 48.0% vs population 20%; Family 47.0% vs population 34%; under 3 months 21.6% vs population 45%. Weighted price is 26.3% vs unweighted 21.6%.

  10. A blanket 20% price cut is poorly targeted: about 2,591 cancelled customers would cite price as main reason after tenure weighting, and only about half of price leavers say they would likely return at 20% off.

    Medium confidenceBase 640

    Weighted price share 26.3% of 9,850 is about 2,591 customers. Among price leavers, Q4 net likely 49.3%. Weighted skip main reason is about 3,193 customers.

Caveats

  • Completion rate is low: 640 completes from 9,850 invited (6.5%; 702 started). Non-response bias is likely and the results cannot be treated as representative without weighting and response-bias checks.
  • The tables are unweighted, and the achieved sample is skewed vs the cancelled-customer population: over-12-month tenure 48.0% vs 20%, Family plan 47.0% vs 34%, under-3-month tenure 21.6% vs 45%. Unweighted Q1 likely understates price and overstates long-tenure product complaints.
  • Only 190 of 640 respondents gave open text; open-text percentages are not representative and theme coding is subjective. Answers can mention multiple themes.
  • Q3 and Q4 are hypothetical stated-intention questions, not actual reactivation tests. They probably overstate behaviour and cannot by themselves prove that either one-tap skipping or a 20% price cut would work in market.
  • The Northern Ireland delivery finding rests on 12 respondents; 66.7% is 8 people and is too imprecise to conclude a systemic regional delivery problem without operational data.
  • Q2 satisfaction was measured retrospectively 'while you were a customer' and is not a cause-of-churn measure. The Family vs Couples satisfaction gap is not statistically significant.
  • The 95% price margin of error is about 3.2 percentage points without finite-population correction; with the 9,850 population correction it is slightly smaller.
  • Fielding was 1-14 September 2025 for cancellations from March to August 2025; recall and seasonal effects may affect reasons and stated likelihoods.

Next steps

  • decisionProceed with one-tap skipping as the primary retention decision, but scope it with pause/multi-week skip and meal-count flexibility where cheap to add.
  • experimentRun a behavioural price-offer pilot against a skip/pause invitation for price-leavers rather than rolling out a 20% cut to all customers.
  • researchAudit Northern Ireland delivery performance by postcode, courier, late/missing/damaged rates and refunds.
  • researchPrioritise menu rotation and variety for long-tenure customers, and test a fortnightly/two-meal option for high-flexibility needs.
  • monitorRe-weight all future cancellation tracking to the customer database and improve response rates among short-tenure and Family cancellers.

Open-text themes it coded

Recipe variety / repetitive recipes / menu rotation 40Price / value / affordability / discounts 42Skipping / pausing / holding / schedule flexibility 46Plan size / number of meals / weekly frequency 31Delivery / reliability / missing items / late boxes 19Packaging / recycling / plastic waste 19Portion size / hunger / larger servings 12Life circumstances / no product fix 7

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Did you cancel a Fernway subscription in the last six months?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    Do you currently live in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  3. Q1

    Which one of the following was the most important reason you cancelled your Fernway subscription?

    One answer
    • It was too expensive or poor value for money
    • I wanted more flexibility to skip, pause or change weeks
    • The recipes or ingredients did not suit my tastes or diet
    • The portion sizes were not right
    • There was too much packaging or I had environmental concerns
    • I had delivery or customer service problems
    • I/we stopped cooking meal kits or my household circumstances changed
    • I switched to another meal-kit or food service
    • Another reason
    • Don't know / not sure
  4. Q2

    Which, if any, of these also contributed to your decision to cancel your Fernway subscription? Please select all that apply.

    Any that apply
    • It was too expensive or poor value for money
    • I wanted more flexibility to skip, pause or change weeks
    • The recipes or ingredients did not suit my tastes or diet
    • The portion sizes were not right
    • There was too much packaging or I had environmental concerns
    • I had delivery or customer service problems
    • I/we stopped cooking meal kits or my household circumstances changed
    • I switched to another meal-kit or food service
    • Another reason
    • None of these
  5. Q3

    In your own words, what was the main thing Fernway could have done to keep you as a customer?

    Open text
  6. Q4

    Overall, how satisfied were you with Fernway while you were a subscriber?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  7. Q5

    How good value for money did Fernway feel while you were a subscriber?

    Scale

    Scale 1-5: Very poor to Very good

  8. Q6

    Please rank these possible changes by how much each would have made you more likely to stay with Fernway. Put the one that would have helped most at the top.

    Ranking
    • A 20% lower price
    • One-tap skipping of any week
    • Being able to pause the subscription for one or more weeks
    • Choosing fewer or more meals per week
    • More recipe choice
    • Better ingredients or recipe quality
  9. Q7

    If Fernway offered only one of the following and everything else stayed the same, which would make you more likely to consider resubscribing?

    One answer
    • A 20% lower price
    • One-tap skipping of any week
    • Both would make me equally more likely
    • Neither would make me more likely
    • Not sure
  10. Q8

    If Fernway kept its current prices but added one-tap skipping of any week, how likely would you be to consider resubscribing?

    Scale

    Scale 1-5: Very unlikely to Very likely

  11. Q9

    If Fernway cut its plan prices by 20% but did not add one-tap skipping, how likely would you be to consider resubscribing?

    Scale

    Scale 1-5: Very unlikely to Very likely

  12. Q10

    How likely are you to consider resubscribing to Fernway in the next three months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  13. Q11

    Which of the following, if any, would make you consider resubscribing to Fernway? Please select all that apply.

    Any that apply
    • A 20% lower price
    • One-tap skipping of any week
    • Being able to pause the subscription for one or more weeks
    • Choosing fewer or more meals per week
    • More recipe choice
    • Better ingredients or recipe quality
    • Smaller or larger portion sizes
    • Less packaging
    • No minimum commitment
    • Better delivery slots
    • Another change
    • None of these
  14. Q12

    Is there anything else you would like to tell us about why you cancelled or what would bring you back to Fernway? (Optional)

    Open text
Sample plan

Invite every UK customer in the CRM who cancelled an active Fernway subscription in the six months before fieldwork, using a unique link per person, deduplicated and with two reminders. Pre-stratify by CRM tenure band: under 3 months, 3-12 months, over 12 months. Target 600 completes with reporting quotas of 200 per tenure band; if a band is short, invite all and report finite-population corrected intervals. Important: 600 completes cannot meet plus/minus 5 percentage points at 95% confidence for all three tenure bands; a worst-case 50% proportion needs about 384 completes per band (about 1,152 total). At 200 per band the margin is about plus/minus 6.9 points. Therefore either the board accepts about plus/minus 7 points per band, or fieldwork is extended and completes raised to about 1,150 (or until each band reaches 384). Weight overall results back to the eligible population by tenure and plan; do not weight away the reporting quotas.