Benchmarks / SurveyBench / Why Fernway customers cancel / Mistral Large 4

Measured by Spring Prompt

Mistral Large 4: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
75 of 100
Analysis rating
815
Head to head, this task
won 7 of 15

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Findings the judges found unsupported

  • For price-motivated leavers, a 20% price cut is necessary but not sufficient: fewer than half (49.3%) say they would likely return even with the discount, suggesting price is a threshold issue for a minority, not a universal fix.The survey does not establish that a price cut is necessary for price-motivated leavers to return. It tests hypothetical responses to two offers, not necessary conditions for returning; 24.6% of price-motivated leavers also express likely return with easier skipping.
  • There is a severe delivery reliability signal in Northern Ireland: 66.7% of NI respondents (8 of 12) cited delivery problems as their main cancellation reason, compared to 9.1% overall. This is an urgent operational issue, though the base is small.Eight delivery-related responses among 12 Northern Ireland respondents support investigation, not the factual conclusion that there is an urgent regional operational issue. The later validation caveat appropriately limits action but does not establish this finding.
  • Recipe variety/repetition is the second-largest qualitative theme, mentioned by 41 of 190 open-text respondents (22%). It is particularly acute among long-tenure customers: 26 of the 41 mentions come from customers who were with Fernway for over 12 months, indicating menu fatigue drives late-stage churn.The supplied open texts contain 40 recipe-variety mentions, not 41, and 25 of those respondents have over-12-month tenure, not 26. R377, included among the long-tenure examples, is a 3–12-month customer.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4140
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1943,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2625
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 7 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build the one-tap skipping feature; do not implement a 20% across-the-board price cut. The data shows that inflexibility drives churn for a larger segment (35.5% unweighted, ~32.4% weighted) than price (21.6% unweighted, 26.3% weighted), and the stated win-back rate for the skip feature among its target segment (58.6%) is more than double that of the price cut among price-leavers (49.3%). A price cut permanently erodes margin by 20% for all customers to win back a minority, whereas the skip feature is a fixed 4-month investment that addresses the primary barrier without reducing per-customer revenue. However, urgently investigate the Northern Ireland delivery failures (66.7% of NI leavers cite delivery) and consider targeted retention pricing (e.g., loyalty discounts or win-back offers) for the price-sensitive new-customer segment rather than a universal price reduction.

What would change it

The recommendation would change if: (1) a pricing elasticity test shows a 20% price cut would acquire significantly more new customers or reduce churn among high-LTV segments than the skip feature retains; (2) the engineering cost of one-tap skipping exceeds 4 months or proves technically infeasible; (3) analysis of actual delivery data shows the skip feature does not reduce box-return rates or waste; or (4) the NI delivery issue proves systemic and fixable through courier changes alone, removing the need for product investment in that region.

Findings

  1. Inflexibility (hard to skip weeks) is the single largest driver of cancellation, not price. It accounts for 35.5% of cancellations unweighted, and approximately 32.4% when weighted to the actual tenure mix of all 9,850 leavers—roughly 3,194 customers. Price is the main reason for only 21.6% unweighted (26.3% weighted).

    High confidenceBase 640

    Q1 Total column: Hard to skip weeks 35.5% (n=227), Price 21.6% (n=138). Weighted by population tenure (45% under 3m, 35% 3-12m, 20% over 12m): Skip = (0.45×26.1)+(0.35×36.9)+(0.20×38.8)=32.4%. Price = (0.45×31.2)+(0.35×27.2)+(0.20×13.7)=26.3%.

  2. A one-tap skip feature has roughly double the stated win-back potential of a 20% price cut. Among those who left because skipping was hard, 58.6% say they would likely return with one-tap skipping, versus only 22.9% who would return for a 20% price cut—a gap of 35.7 percentage points.

    High confidenceBase 227

    Q3 and Q4 tables, 'Hard to skip weeks' column (n=227). Q3 Net likely (4-5): 58.6%. Q4 Net likely (4-5): 22.9%. Difference: 35.7pp.

  3. For price-motivated leavers, a 20% price cut is necessary but not sufficient: fewer than half (49.3%) say they would likely return even with the discount, suggesting price is a threshold issue for a minority, not a universal fix.

    Medium confidenceBase 138

    Q4 table, 'Price' column (n=138). Net likely (4-5): 49.3%.

  4. The CEO's hypothesis that 'people leave because we're too expensive' is only partially supported. Price is the dominant reason for new customers (31.2% of under-3-month cancellers) but fades with tenure (13.7% of over-12-month cancellers). The sample over-represents long-tenure customers (48% vs 20% in population), which understates price sensitivity in the customer base.

    High confidenceBase 640

    Q1 table by tenure: Under 3 months 31.2% price, Over 12 months 13.7% price. Sample tenure mix (Q6): 21.6% under 3m, 48.0% over 12m vs population 45% and 20% respectively.

  5. The Family plan is not uniquely dissatisfied. While Family satisfaction (4-5) is 49.2% vs Couples 53.4%, this 4.2pp gap is not statistically significant at the 95% level (95% CI for difference: -3.6pp to +11.9pp). Family customers do cite recipe repetition more often (23.6% vs 13.0% for Couples), suggesting a content issue rather than a plan-satisfaction issue.

    Medium confidenceBase 640

    Q2 table: Family satisfied 49.2% (n=301), Couples 53.4% (n=339). Z-test: 1.06 < 1.96. Q1 table: Recipes got repetitive—Family 23.6%, Couples 13.0%.

  6. There is a severe delivery reliability signal in Northern Ireland: 66.7% of NI respondents (8 of 12) cited delivery problems as their main cancellation reason, compared to 9.1% overall. This is an urgent operational issue, though the base is small.

    Low confidenceBase 12

    Q1 table, Northern Ireland column (n=12): Delivery problems 66.7%, Price 0.0%, Hard to skip 16.7%.

  7. Recipe variety/repetition is the second-largest qualitative theme, mentioned by 41 of 190 open-text respondents (22%). It is particularly acute among long-tenure customers: 26 of the 41 mentions come from customers who were with Fernway for over 12 months, indicating menu fatigue drives late-stage churn.

    High confidenceBase 190

    Open-text analysis: 41 respondents mention repetitive recipes, lack of variety, or menu rotation. Of these, 26 are Over 12 months tenure (e.g., R028, R089, R115, R130, R180, R186, R200, R215, R237, R283, R312, R367, R377, R382, R407, R416, R419, R524, R525, R529, R539, R560, R589, R608, R615).

Caveats

  • The sample is heavily skewed toward long-tenure customers (48% over 12 months vs 20% of the cancelling population). This means price sensitivity is likely understated and recipe fatigue overstated in the unweighted results.
  • Completion rate is low (6.5%), indicating potential non-response bias; customers with strong negative experiences (e.g., delivery failures) may be overrepresented.
  • The Northern Ireland delivery finding rests on only 12 respondents; while the signal (66.7%) is extreme, it requires validation before operational changes.
  • Stated intent to return (Q3/Q4) is hypothetical and typically overestimates actual behavior; a 20% price cut may have different real-world elasticity than stated.
  • The survey captures cancellation reasons, not retention drivers for current customers; the skip feature may reduce churn but the data cannot confirm it will not cannibalize full-price weeks.
  • Price is a stated main reason for 21.6%, but many 'Other' or 'Hard to skip' respondents mention price in open text (e.g., R074, R123, R380, R447), suggesting price is a contributing factor for a larger share than the single-choice question captures.

Next steps

  • decisionApprove and build one-tap skipping (4-month product roadmap)
  • researchConduct urgent delivery audit in Northern Ireland
  • experimentRun a pricing experiment (holdout test) for new acquisitions
  • researchAudit recipe rotation and menu variety, especially for Family plan and long-tenure customers
  • monitorRe-weight survey data to population tenure mix and track KPIs monthly
  • decisionInvestigate 'pause' feature as a lower-cost alternative or complement to skip

Open-text themes it coded

Skip/Pause Flexibility (inability to easily skip weeks or pause subscription) 45Recipe Variety/Repetition (menu got repetitive, lacked diversity) 41Price/Affordability (too expensive, cost increases, value for money) 38Portion Sizes (too small, not filling, need larger servings) 12Packaging Waste/Sustainability (plastic, non-recyclable materials, excess packaging) 15Delivery Reliability (late, missing items, wrong day, tracking issues) 15Plan Size/Meal Count (too many meals per week, minimum too high, want fewer options) 25

The survey it planned

3 screening questions and 12 questions, as the model wrote them.

  1. S1

    Have you cancelled a Fernway meal kit subscription in the last 6 months?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Do you currently live in the United Kingdom?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    How long were you subscribed to Fernway before you cancelled your most recent subscription?

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months

    Continues if Less than 3 months; 3 to 12 months; More than 12 months

  4. Q1

    What was the main reason you cancelled your Fernway subscription?

    One answer
    • Price was too high / poor value for money
    • Too much food / food waste
    • Didn't enjoy the recipes / lack of variety
    • Delivery problems (late, damaged, missing items)
    • Too difficult to skip weeks when I needed to
    • Change in my circumstances (moving, budget, family changes)
    • I was just trying it out / trial period ended
    • Found a better alternative
    • Other (please specify)
  5. Q2

    Which of the following contributed to your decision to cancel? Please select all that apply.

    Any that apply
    • Price was too high / poor value for money
    • Too much food / food waste
    • Didn't enjoy the recipes / lack of variety
    • Delivery problems
    • Too difficult to skip weeks
    • Couldn't pause the subscription when needed
    • App was difficult to use
    • Change in my circumstances
    • Other (please specify)
  6. Q3

    Overall, how satisfied were you with your Fernway subscription while you were subscribed?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  7. Q4

    How would you rate the value for money you received from Fernway?

    Scale

    Scale 1-5: Very poor value to Excellent value

  8. Q5

    While you were subscribed, how easy or difficult did you find it to skip a delivery week when you needed to?

    One answer
    • Very easy
    • Somewhat easy
    • Neither easy nor difficult
    • Somewhat difficult
    • Very difficult
    • I never needed to skip a week
  9. Q6

    Which of these mattered more in your decision to cancel?

    One answer
    • The price / value for money
    • The difficulty of skipping weeks when needed
    • Both mattered equally
    • Neither - I left for other reasons
  10. Q7

    If Fernway could only make ONE of these changes, which would be most likely to make you re-subscribe?

    One answer
    • Reduce the price of plans by 20%
    • Allow one-tap skipping of any week in the app
    • Both would be equally likely to bring me back
    • Neither would bring me back
  11. Q8

    How likely are you to re-subscribe to Fernway in the next 3 months?

    Scale

    Scale 1-5: Not at all likely to Very likely

  12. Q9

    What would make you consider re-subscribing to Fernway? Please select all that apply.

    Any that apply
    • 20% lower price
    • One-tap skipping of any week
    • Ability to pause the subscription for a month or more
    • More recipe variety / new cuisines
    • Smaller box sizes (fewer meals per week)
    • More reliable delivery times
    • Improved app experience
    • Nothing would bring me back
  13. Q10

    Which Fernway plan were you subscribed to?

    One answer
    • Couples plan (2 people)
    • Family plan (4 people)
    • I switched between plans
    • Not sure
  14. Q11

    While you were subscribed, how often did you skip a delivery week?

    One answer
    • Every week
    • Most weeks (more than half the time)
    • Some weeks (about half the time)
    • A few weeks (less than half the time)
    • Never
    • I didn't know I could skip weeks
  15. Q12

    Is there anything else Fernway could do to win you back as a customer?

    Open text
Sample plan

Invite every UK customer who cancelled a Fernway subscription in the last 6 months via email from the customer database (census approach). Target 1,200 completes (400 per tenure group: under 3 months, 3-12 months, over 12 months) to achieve the board's required precision of ±5 percentage points at 95% confidence for each tenure subgroup. Note: The initial guidance of 'about 600 completes' would yield only ±7 percentage points for subgroups; 1,200 is the statistical minimum to meet the precision requirement. Apply soft quotas on tenure (S3) to ensure balanced group sizes; if the 'over 12 months' tenure group in the churned population is smaller than 385, report wider confidence intervals for that group. Expected completion time: 4-5 minutes.