Benchmarks / SurveyBench / Why Fernway customers cancel / MiniMax M3.1 Flash

Measured by Spring Prompt

MiniMax M3.1 Flash: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
95 of 100
Analysis rating
484
Head to head, this task
won 11 of 12

Why it is not sound

The analysis

  • Got at least one set number wrong

Scorecard

Numbers 10 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✗3025
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 8 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✓The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Proceed with the one-tap skip/pause build as the primary intervention and reject an across-the-board 20% price cut for now. Specify the MVP as genuinely one-tap skipping of any week, with multi-week holiday pausing, retained settings, a visible and later cutoff, mobile access and no repeated confirmation or upsell screens. Use targeted, preferably time-limited price credits for demonstrably price-led customers rather than permanently repricing every plan. Do not label the Family plan or Northern Ireland as a proven root cause, although improve Family menu rotation and audit Northern Ireland delivery immediately. Set realistic expectations: even the stronger option was likely to return only 36.4% of all respondents, so menu freshness, flexible meal quantities, portions and delivery need parallel work. Fund the build against a contribution-margin business case and run concurrent randomized skip and price experiments before making any permanent pricing change.

What would change it

Change the recommendation if actual randomized behaviour shows that a targeted 20% discount produces more incremental retained or reactivated contribution per customer than the skip intervention after engineering, discount and operational costs. Also change course if the skip MVP cannot achieve reliable one-tap completion, cannot support the requested multi-week and holiday use cases, or fails to produce a material retention lift. A well-powered active-subscriber study showing that price is substantially more important than skip friction among customers who are actually at risk would also reopen the pricing-first case.

Findings

  1. CEO answer: price is a material cancellation driver, but the survey does not support the claim that it is the dominant reason people leave. Build skipping as the primary intervention rather than cutting all plan prices by 20%.

    Medium confidenceBase 640

    Among 640 completers, 21.6% gave price as their main reason, increasing to 26.3% after weighting to the population tenure mix; about 2,590 of 9,850 cancellations. Hard to skip weeks was 35.5% unweighted and 32.4% weighted, equivalent to about 3,193 cancellations. The weighted lead for skipping is 6.1 percentage points rather than the unweighted 13.9 points, so price should not be dismissed.

  2. One-tap skipping is substantially more persuasive than a 20% price cut for the customers whose stated problem is skip friction.

    Medium confidenceBase 227

    Among the 227 respondents whose main reason was hard to skip weeks, 58.6% said they would be fairly or very likely to return with one-tap skipping, versus 22.9% with a 20% price cut: a 35.7-point advantage, approximately 133 versus 52 respondents. Across all 640 respondents, the corresponding figures were 36.4% for skipping and 25.8% for the price cut. These are stated intentions rather than observed re-subscriptions.

  3. Use price selectively for price-led customers rather than as a universal plan-price reduction.

    Medium confidenceBase 138

    Among the 138 respondents whose main reason was price, 49.3% said a 20% cut would make them likely to return, compared with 24.6% for one-tap skipping. Thus the discount is more effective in this segment, but even among direct price leavers only 49.3% were likely to return; half were neutral or unlikely. A blanket 20% cut would impose its cost on customers whose problem was not price.

  4. Head of product answer: the Family plan is not shown to be the underlying satisfaction problem, although Family customers report materially more recipe repetition.

    Medium confidenceBase 640

    Net satisfaction was 53.4% among 339 Couples respondents and 49.2% among 301 Family respondents, a 4.2-point Family deficit. The difference is not statistically significant at 95%: two-proportion z is approximately 1.07, p approximately 0.29, with an approximate 95% interval for the Family-minus-Couples gap of -12.0 to +3.6 percentage points. Family satisfaction was not lower within any tenure stratum. However, repetitive recipes were cited by 23.6% of Family respondents versus 13.0% of Couples respondents, pointing to menu rotation rather than a plan-wide satisfaction defect.

  5. Operations answer: Northern Ireland has a serious warning signal for delivery, but this survey cannot confirm a systemic Northern Ireland delivery problem.

    Low confidenceBase 12

    Delivery problems were the main cancellation reason for 8 of 12 Northern Ireland respondents, or 66.7%; the approximate 95% Wilson interval is 39% to 86%. The comparison figures were 13.0% in Scotland, based on 46 respondents, and 8.1% in England, based on 546. With only 12 Northern Ireland respondents and no regional invitation or delivery-volume denominator, this is an operational red flag rather than conclusive population evidence.

  6. Even the better of the two proposed interventions will not address all churn drivers; menu fatigue and meal-count or portion fit need parallel attention.

    Medium confidenceBase 187

    Manual coding of the 187 identifiable Q5 rows supplied, although the metadata reports 190 answers, found 40 mentions of recipe variety or repetition, including 30 from customers of over 12 months; 38 mentions of portions, meal count or plan flexibility; 19 of delivery reliability, missing items or arrival damage; and 19 of packaging or recycling. These themes overlap and should not be treated as population prevalence estimates.

  7. The MVP needs to remove the full skip workflow, not merely add a visible button.

    Medium confidenceBase 187

    Forty-seven of the identifiable Q5 answers mention skipping, pausing, holiday breaks or schedule flexibility. Recurring requests include multi-week pausing, retaining settings and preferences, a cutoff closer to delivery, mobile access, and avoiding hidden navigation, repeated confirmation screens, upsells and phone calls. A one-tap interaction that remains constrained by an early cutoff or cumbersome confirmations would not answer the observed problem.

  8. The survey supports the direction of the decision but not its financial return; no evidence yet shows whether one-tap skipping or discounting produces better incremental contribution from retained and reactivated subscribers.

    High confidenceBase 640

    Only 640 of 9,850 invited customers completed, a 6.5% completion rate, and the sample over-represents customers with over 12 months of tenure at 48% versus 20% of the cancellation population. The study includes only former customers, uses sequential hypothetical questions rather than a choice experiment, and supplies no engineering cost, gross-margin cost, baseline retention, price elasticity or active-subscriber response to either intervention.

Caveats

  • The 6.5% completion rate, 640 of 9,850 invited customers, indicates a self-selected survey rather than a probability sample. The 3.2-point margin of error on price covers sampling error only and is likely to understate total uncertainty from nonresponse and selection bias.
  • The sample materially differs from the cancellation population: over-12-month customers are 48% of respondents versus 20% of all cancellations, under-three-month customers are 21.6% versus 45%, and Family customers are 47% versus 34%. Tenure weighting corrects only the known tenure imbalance; there are no population cross-tabs for plan by tenure or region.
  • The survey covers only former customers, not active subscribers at risk of cancellation. Return intention therefore cannot directly establish how either intervention would retain current subscribers.
  • Q3 and Q4 are sequential hypothetical intentions rather than an actual choice between offers. They provide no price-response curve, reveal no cannibalization or discount cost, and may be affected by question-order and anchoring effects.
  • Q1 forces one main reason and therefore understates multi-causal problems. Q5 supplies important nuance but was answered by only about 30% of completers, and responses that raise price can also reflect repetition, quantities, delivery or packaging.
  • The metadata reports 190 open-text answers, but only 187 identifiable response rows are present in the supplied extract. The counts and respondent coding for themes are based on those 187 visible rows; the three unreported rows could change the open-text totals, including the 40 variety mentions.
  • The Family result is a non-significant estimate with a wide confidence interval, and the available population weights cannot produce a fully adjusted plan-by-tenure satisfaction estimate. It rules out a large simple Family gap in this sample, not every possible Family-specific problem.
  • Northern Ireland has only 12 respondents, a very wide confidence interval and no invitation or delivery-volume denominator. It cannot establish the scale, trend or cause of a regional delivery problem.
  • The one-tap option is not fully specified. Features such as the skip deadline, multi-week pausing, settings retention and treatment of failed or late skips could materially determine whether it solves the observed frustration.
  • No gross-margin model is supplied. The survey therefore cannot determine whether four months of product work or a permanent 20% price reduction produces the better return on investment.

Next steps

  • decisionApprove one-tap skip/pause as the primary initiative and reject a universal 20% list-price reduction; authorise a concurrent price-response test.
  • researchTest the complete skip workflow with active subscribers and recent cancellers, including one-week skips, multi-week holidays, settings retention, cutoff timing and failed-skip recovery.
  • experimentRun a randomized controlled pilot of the skip MVP among eligible active subscribers, with a holdout group.
  • experimentRun an actual win-back experiment among price-led and skip-led former customers using a common audience and arms for skip-first, targeted 10%, 15% and 20% discounts, plus a no-offer control.
  • researchSurvey or analyse active at-risk subscribers, weighted by plan, tenure and region, to separate Family, price, skip, variety, quantity and delivery effects.
  • researchAudit Northern Ireland delivery operations against Great Britain using parcel-level data.
  • decisionPlace menu rotation and flexible meal-count or portion options on the parallel product roadmap.
  • monitorInstrument cancellation reasons, skip funnel errors, delivery exceptions, plan changes and re-subscription offers after launch.

Open-text themes it coded

Skipping, pausing and delivery-cadence flexibility 47Price, affordability and perceived value 42Recipe variety and repetitive recipes 40Portion size, meal count and flexible plan quantity 38Delivery reliability, missing items and arrival condition 19Packaging, recycling and material waste 19Life circumstances outside Fernway's control 5

The survey it planned

3 screening questions and 12 questions, as the model wrote them.

  1. S1

    Did you cancel a Fernway subscription in the last 6 months?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Was the delivery address for your cancelled Fernway subscription in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Were you the person who used or managed the Fernway subscription?

    One answer
    • Yes
    • No

    Continues if Yes

  4. Q1

    Which Fernway plan were you on when you cancelled?

    One answer
    • Couples plan (a recipe box for two people)
    • Family plan (a recipe box for four people)
  5. Q2

    If Fernway offered you the same plans, prices and delivery rules as when you left, how likely would you be to start a subscription again in the next 3 months?

    Scale

    Scale 1-11: Not at all likely to Extremely likely

  6. Q3

    Thinking about your entire subscription, how satisfied were you with Fernway while you were subscribed?

    Scale

    Scale 1-11: Not at all satisfied to Extremely satisfied

  7. Q4

    Which one of these was the main reason you decided to cancel your Fernway subscription?

    One answer
    • The cost of the subscription
    • The flexibility to skip, pause or change deliveries
    • The taste or quality of the ingredients
    • The choice of recipes
    • The number of meals or portion sizes
    • Delivery reliability or timing
    • Packaging
    • Customer service
    • A change in our household or lifestyle
    • A better alternative from another provider
    • Something else
    • I was not sure of a main reason
  8. Q5

    Which of these other factors also played a role in your decision to cancel? Select all that apply. Do not repeat the main reason you selected in the previous question.

    Any that apply
    • The cost of the subscription
    • The flexibility to skip, pause or change deliveries
    • The taste or quality of the ingredients
    • The choice of recipes
    • The number of meals or portion sizes
    • Delivery reliability or timing
    • Packaging
    • Customer service
    • A change in our household or lifestyle
    • A better alternative from another provider
    • Something else
    Judges: assumptive

    Assumes additional factors played a role. Respondents whose main reason was their only reason have no explicit applicable answer; an unselected response cannot distinguish this from nonresponse.

    The question assumes other factors played a role. There is no 'none / no other factors' option, so respondents with a single reason, or those who were not sure of a main reason, cannot answer accurately.

  9. Q6

    Before you cancelled, did you try to change or skip a delivery?

    One answer
    • No—I did not need to change or skip a delivery
    • No—I wanted to, but I did not know Fernway allowed it
    • No—I wanted to, but I did not have time to use the app before the six-day deadline
    • No—the available delivery options did not meet what I needed
    • Yes—I successfully changed or skipped a delivery
    • Yes—I tried, but the six-day deadline had passed
    • I do not remember
  10. Q7

    Thinking only about your decision to cancel, how important was flexibility to skip, pause or change a delivery compared with the cost of the subscription? Use a 5-point scale: 1 means flexibility was much less important than price; 2 means somewhat less important; 3 means about equally important; 4 means somewhat more important; and 5 means much more important.

    Scale

    Scale 1-5: Flexibility was much less important than price to Flexibility was much more important than price

  11. Q8

    While you were subscribed, which of these aspects, if any, did not meet your expectations?

    Any that apply
    • The subscription price
    • The value received for the price
    • The taste or quality of the ingredients
    • The range of recipes
    • The number of meals or portion sizes
    • The ability to skip or change deliveries
    • The reliability or timing of deliveries
    • The packaging
    • The customer service
    • None of these
  12. Q9

    Which one of these changes would have been most likely to keep you subscribed at the time you cancelled?

    One answer
    • A 20% lower subscription price, with the current delivery rules
    • The ability to skip any future delivery week with one tap and no six-day deadline, at the current price
    • Both a 20% lower subscription price and one-tap skipping with no six-day deadline
    • The option to order fewer meals or smaller portions, at the current price
    • Improved recipes, ingredients or both
    • A free trial box before paying again
    • Improved delivery or packaging
    • Something else
    • None of these would have kept me subscribed
    • Not sure
  13. Q10

    Which one of these changes or offers would make you most likely to subscribe again now?

    One answer
    • A 20% lower subscription price, with the current delivery rules
    • The ability to skip any future delivery week with one tap and no six-day deadline, at the current price
    • Both a 20% lower subscription price and one-tap skipping with no six-day deadline
    • The option to order fewer meals or smaller portions, at the current price
    • Improved recipes, ingredients or both
    • A free trial box before paying again
    • Improved delivery or packaging
    • Something else
    • None of these would make me likely to subscribe again
    • Not sure
  14. Q11

    Suppose Fernway offered you the specific option you selected in the previous question. How likely would you be to start a subscription again in the next 3 months?

    Scale

    Scale 1-11: Not at all likely to Extremely likely

    Routing Shown only if Q10 is: A 20% lower subscription price, with the current delivery rules; The ability to skip any future delivery week with one tap and no six-day deadline, at the current price; Both a 20% lower subscription price and one-tap skipping with no six-day deadline; The option to order fewer meals or smaller portions, at the current price; Improved recipes, ingredients or both; A free trial box before paying again; Improved delivery or packaging

  15. Q12

    What is the one change Fernway should make, if any, to give you a reason to consider subscribing again? If no change would, please say so. Please use no more than three sentences.

    Open text
Sample plan

Email every customer record in the Fernway database with a UK delivery address and a cancellation date in the fixed six-month window preceding field launch. Use a unique link and one response per customer, with up to two neutral reminders that do not mention price or flexibility as suspected causes. Derive tenure from subscription start and cancellation dates rather than self-report: under 3 months, 3-12 months inclusive, and over 12 months. The stated sample-size and precision requirements are not simultaneously feasible: approximately 600 completes divided across three groups gives about 200 per group and a worst-case 95% margin of error of approximately ±6.9 percentage points, not ±5. To meet the board requirement, use an initial wave of approximately 600 completes and then top up each tenure group to at least 400 usable completes, approximately 1,200 total; 385 completes per group is the mathematical minimum under simple random sampling. If 600 is an absolute cap, allocate approximately 200 per group and explicitly report the ±6.9-point limitation rather than claim ±5-point precision. Oversample or send additional reminders to underfilled tenure cells, and weight population-level summaries back to the actual tenure and plan distributions while reporting within-tenure estimates with design-adjusted 95% confidence intervals. The precision target applies to headline estimates within each tenure group, not every plan-by-tenure subgroup. Randomize substantive option order except for options such as None and Not sure, and monitor response by plan and cancellation month for differential nonresponse.