Benchmarks / SurveyBench / Why Fernway customers cancel / Mistral Medium 3.5

Measured by Spring Prompt

Mistral Medium 3.5: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
62 of 100
Analysis rating
540
Head to head, this task
won 1 of 15

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A double-barrelled question: Q10

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Missed a trap: Northern Ireland respondents (base 12) cite delivery problems far more often.
  • Missed a trap: Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • Missed a trap: 'Likely to come back' answers are stated intentions about a hypothetical feature.
  • Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.
  • Proposed a mistaken next step: Commit to a Northern Ireland delivery overhaul on the strength of the survey alone

Findings the judges found unsupported

  • Price is not the dominant reason for cancellation; only 21.6% of respondents cited it as their main reason, and this drops to ~19.8% when weighted to the tenure mix of all cancellations.The weighting calculation is wrong: 0.45×31.2 + 0.35×27.2 + 0.20×13.7 = 26.3%, not 19.8%. Weighting increases the price share rather than decreasing it. The stated margin of error also does not account for weighting or non-response bias.
  • Delivery problems are a major issue in Northern Ireland, with 66.7% of respondents there citing it as their main reason for cancelling (vs 8.1% in England).The 66.7% describes eight of twelve Northern Ireland respondents, not an established regional delivery problem. One open-text response does not validate that generalisation. The generic small-base caveat does not adequately qualify the finding or the recommendation to address the region.
  • One-tap skipping would be highly effective for those who left due to inflexibility: 58.6% of respondents who cited 'hard to skip weeks' as their main reason would be likely to return if one-tap skipping were introduced.The 58.6% is hypothetical stated likelihood, not evidence that one-tap skipping would be highly effective or that this proportion would actually return.
  • A 20% price cut would be less effective for those who left due to inflexibility: only 22.9% of respondents who cited 'hard to skip weeks' as their main reason would be likely to return if the price were reduced by 20%.The survey supports lower stated return intentions for a price cut in this group, but does not establish lower actual effectiveness.
  • For those who left due to price, a 20% price cut would be highly effective: 49.3% of respondents who cited price as their main reason would be likely to return.The 49.3% is hypothetical stated likelihood among price leavers, not demonstrated effectiveness or a forecast of actual returns.
  • Recipe variety is a recurring pain point, mentioned in 42 open-text answers (22% of all open-text responses), with 18 of these from long-tenure customers (>12 months).The supplied answers contain 40 recipe-variety mentions, not 42, with 25 from customers of over twelve months, not 18. The finding also conflicts with the analysis's own theme total of 40.
  • Weighted to the tenure mix of all cancellations, ~34.2% of all cancellations would cite 'hard to skip weeks' as their main reason.The stated calculation equals 32.42%, not 34.2%. Skipping remains ahead of price after tenure weighting, but the reported weighted percentage is incorrect.

Scorecard

Numbers 8 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✗19.8%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4240
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓2.9 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✗34.23,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✗1825
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 3 of 8 handled

  • ✗Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✗Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✗'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build the one-tap skipping feature. The evidence strongly suggests that inflexibility (hard to skip weeks) is the primary driver of cancellations (35.5% of respondents, ~34.2% weighted), and 58.6% of these would return if one-tap skipping were introduced. This is far more effective than a 20% price cut (22.9% likely to return for this group). Price is a secondary issue (21.6% of respondents, ~19.8% weighted), and while a price cut would win back some customers (49.3% of price leavers), it would be less targeted and less effective overall. The Family plan is not a unique problem for satisfaction, and delivery issues are localized to Northern Ireland (a smaller segment).

What would change it

Evidence that a 20% price cut would win back a significantly larger share of all cancellations (not just price leavers) than one-tap skipping, or that the cost of implementing one-tap skipping is prohibitive relative to the expected retention gains. For example, if the price cut were shown to win back >40% of all cancellations (vs ~20% for skipping), or if the development cost of skipping were >£X (to be defined by finance).

Findings

  1. Price is not the dominant reason for cancellation; only 21.6% of respondents cited it as their main reason, and this drops to ~19.8% when weighted to the tenure mix of all cancellations.

    High confidenceBase 640

    Q1: Price = 21.6% (base 640). Weighted to tenure (45% under 3m, 35% 3-12m, 20% over 12m): (0.45*31.2 + 0.35*27.2 + 0.20*13.7) = 19.8%. Margin of error: ±2.9pp.

  2. The Family plan is not the primary problem for satisfaction; its net satisfaction (49.2%) is only slightly lower than Couples (53.4%), and the difference is not statistically significant at 95% confidence.

    High confidenceBase 640

    Q2: Net satisfied (4-5) = 49.2% (Family, base 301) vs 53.4% (Couples, base 339). 95% CI for Family: 43.6%-54.8%; Couples: 48.1%-58.7%. Overlap in CIs means no significant difference.

  3. Delivery problems are a major issue in Northern Ireland, with 66.7% of respondents there citing it as their main reason for cancelling (vs 8.1% in England).

    Medium confidenceBase 12

    Q1: Delivery problems = 66.7% (Northern Ireland, base 12) vs 8.1% (England, base 546). Open-text answers from Northern Ireland (R445) also mention delivery and packaging issues.

  4. One-tap skipping would be highly effective for those who left due to inflexibility: 58.6% of respondents who cited 'hard to skip weeks' as their main reason would be likely to return if one-tap skipping were introduced.

    High confidenceBase 227

    Q3: Net likely (4-5) = 58.6% (base 227, main reason = hard to skip weeks).

  5. A 20% price cut would be less effective for those who left due to inflexibility: only 22.9% of respondents who cited 'hard to skip weeks' as their main reason would be likely to return if the price were reduced by 20%.

    High confidenceBase 227

    Q4: Net likely (4-5) = 22.9% (base 227, main reason = hard to skip weeks).

  6. For those who left due to price, a 20% price cut would be highly effective: 49.3% of respondents who cited price as their main reason would be likely to return.

    High confidenceBase 138

    Q4: Net likely (4-5) = 49.3% (base 138, main reason = price).

  7. Recipe variety is a recurring pain point, mentioned in 42 open-text answers (22% of all open-text responses), with 18 of these from long-tenure customers (>12 months).

    High confidenceBase 190

    Q5: 42 mentions of variety/repetition (base 190). 18 from respondents with tenure >12 months (base 190).

  8. Weighted to the tenure mix of all cancellations, ~34.2% of all cancellations would cite 'hard to skip weeks' as their main reason.

    High confidenceBase 640

    Q1: Hard to skip weeks = 35.5% (base 640). Weighted to tenure: (0.45*26.1 + 0.35*36.9 + 0.20*38.8) = 34.2%.

  9. The survey completion rate was low (6.5%), which may introduce non-response bias, particularly if those who left due to price or delivery issues were more or less likely to respond.

    Medium confidenceBase 9850

    640 completes / 9,850 invited = 6.5%.

Caveats

  • The survey has a low completion rate (6.5%), which may bias results if non-respondents differ systematically from respondents (e.g., those who left due to price may be less likely to respond).
  • The sample is unweighted for plan type (Couples vs Family), which may over- or under-represent certain groups. The population is 66% Couples and 34% Family, but the sample is 53% Couples and 47% Family.
  • Open-text answers are qualitative and may not be representative of all respondents, but they provide depth to the quantitative findings.
  • The margin of error for small subgroups (e.g., Northern Ireland, base 12) is large, so findings for these groups should be treated with caution.
  • The survey does not capture the intensity of feelings (e.g., how strongly respondents feel about price vs flexibility), only their stated main reason and likelihood to return.

Next steps

  • experimentRun a controlled experiment: offer one-tap skipping to a random subset of churned customers and measure re-subscription rates vs a control group.
  • experimentConduct a pricing elasticity study: test a 20% price cut with a random subset of churned customers and measure re-subscription rates.
  • decisionInvestigate and address delivery issues in Northern Ireland, as this is a clear outlier (66.7% of cancellations there cite delivery problems).
  • monitorMonitor the impact of one-tap skipping on retention and churn reasons after launch, with a focus on whether 'hard to skip weeks' as a cancellation reason declines.
  • researchExplore recipe variety improvements (e.g., larger rotation, more diverse cuisines) as a secondary initiative, given its prominence in open-text responses (42 mentions).
  • researchRe-weight the survey data for plan type (Couples vs Family) to ensure findings are representative of the population.

Open-text themes it coded

Flexibility (pause/skip) 50Price/value 35Recipe variety/repetition 40Portion sizes 19Delivery problems 19Packaging waste 19Minimum meals/plan flexibility 39

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Did you cancel a Fernway subscription in the last six months?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Are you currently living in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  3. Q1

    How long were you a Fernway subscriber before you cancelled?

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months
  4. Q2

    What was the main reason you cancelled your Fernway subscription? Please select one.

    One answer
    • The price was too high
    • I needed more flexibility (e.g., to skip weeks or pause)
    • The recipes or ingredients didn’t meet my expectations
    • I didn’t have enough time to cook
    • I found a better alternative
    • Other (please specify)
  5. Q3

    Which of the following were reasons for your cancellation? Please select all that apply.

    Any that apply
    • The price was too high
    • I wanted to skip a week but couldn’t do it in time
    • I wanted to pause my subscription but couldn’t
    • I wanted fewer meals per week
    • The recipes were not to my taste
    • The ingredients were not fresh or high quality
    • I didn’t have enough time to cook
    • I found a cheaper alternative
    • I found a more flexible alternative
    • Other (please specify)
  6. Q4

    How important was the ability to skip a week easily (e.g., with one tap) in your decision to cancel?

    Scale

    Scale 1-5: Not at all important to Extremely important

  7. Q5

    How important was the price of Fernway in your decision to cancel?

    Scale

    Scale 1-5: Not at all important to Extremely important

  8. Q6

    If Fernway offered a 20% discount on your plan, how likely would you be to resubscribe?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  9. Q7

    If Fernway allowed you to skip any week with one tap in the app, how likely would you be to resubscribe?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  10. Q8

    Which of the following would most likely make you resubscribe to Fernway? Please select one.

    One answer
    • A 20% discount on my plan
    • The ability to skip any week with one tap
    • The ability to pause my subscription for a month or more
    • Fewer meals per week (e.g., 2 meals instead of 3)
    • More variety in recipes
    • Higher quality ingredients
    • None of these would make me resubscribe
    • Other (please specify)
  11. Q9

    Overall, how satisfied were you with Fernway while you were subscribed?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  12. Q10

    How satisfied were you with the following aspects of Fernway? Please rate each on a scale of 1 to 5, where 1 is very dissatisfied and 5 is very satisfied.

    Scale
    • Recipe variety
    • Ingredient quality
    • Ease of use of the app
    • Flexibility to skip or pause
    • Value for money

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: double-barrelled

    The item 'Flexibility to skip or pause' requires one satisfaction rating for two distinct functions. Respondents could be satisfied with skipping but dissatisfied with pausing, or vice versa.

    The item 'Flexibility to skip or pause' combines two distinct features in one rating. The decision hinges on skipping specifically, so a single score for both cannot separate them.

  13. Q11

    What did you like most about Fernway?

    Open text
  14. Q12

    To ensure we’re getting accurate responses, please select 'Strongly agree' for this question.

    One answer
    • Strongly disagree
    • Disagree
    • Neither agree nor disagree
    • Agree
    • Strongly agree
Sample plan

Invite all UK customers who cancelled a Fernway subscription in the last six months (approximately 600 completes). Ensure representation across tenure groups (under 3 months, 3-12 months, over 12 months) to achieve ±5% precision at 95% confidence for each group.