Sign inTest your prompt

Benchmarks / SurveyBench / Why Fernway customers cancel / Muse Ember

Measured by Spring Prompt

Muse Ember: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
96 of 100
Analysis rating
1,392
Head to head, this task
won 11 of 14

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A double-barrelled question: Q5

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.19 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 8 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✓The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build one-tap skipping as the priority investment, designed as a true pause: skip any week up to close to packing, skip multiple consecutive weeks in one action, no phone call or repeated confirmations, and remember plan/settings on return. Do not implement a blanket 20% price cut now. In parallel, pilot (a) a flexible 2-meal / variable-meals option, which open text suggests is a distinct need, (b) targeted price measures for short-tenure leavers (intro-price extension, loyalty discount) rather than a universal cut, and (c) a broader recipe rotation for long-tenure customers. Before committing the full four months irrevocably, run a real win-back experiment (below) and an NI delivery audit, because the survey evidence is stated preference from 6.5% of leavers only.

What would change it

A randomised win-back test showing actual resubscription and 6-month retained revenue (net of margin) is higher for a 20% discount than for one-tap skipping; or a finance model showing skipping cannibalises revenue per active customer enough to outweigh win-back; or current-customer evidence that price, not skip friction, drives ongoing retention. Any of these would favour targeted price action over, or alongside, the skip build.

Findings

  1. Hard-to-skip, not price, is the largest cancellation reason, so one-tap skipping should take priority over a blanket 20% price cut. This directly answers the CEO: price is important but not the main driver.

    Medium confidenceBase 640

    Unweighted: hard to skip 35.5% (227/640) vs price 21.6% (138/640, 95% MoE +/-3.19pp). Weighted to population tenure mix (45% under 3m, 35% 3-12m, 20% over 12m): hard to skip 32.4% = about 3,193 of 9,850 cancellers; price 26.3% = about 2,591. Sample over-represents long tenure, which inflates skip and deflates price versus the population.

  2. One-tap skipping beats a 20% cut head-to-head among its target segment and overall, but neither offer strongly wins back a majority of its own segment.

    Medium confidenceBase 227

    Among hard-to-skip leavers (n=227): 58.6% likely to return with one-tap skip vs 22.9% with 20% off, gap 35.7pp. Overall (n=640): 36.4% vs 25.8%. Conversely among price leavers (n=138): 49.3% likely with 20% off vs 24.6% with skip. Even for skip leavers 41.4% are not likely (26.4% unlikely, 15.0% neither) - stated likelihood, not behaviour.

  3. Price sensitivity is concentrated in short-tenure leavers, so a blanket cut would subsidise long-tenure customers who left for other reasons.

    Medium confidenceBase 640

    Price as main reason: 31.2% under 3 months (n=138), 27.2% at 3-12 months (n=195), 13.7% over 12 months (n=307). Hard-to-skip rises the other way: 26.1%, 36.9%, 38.8%. Repetitive recipes: 6.5%, 14.4%, 25.4%. Targeted intro-price/loyalty or cheaper smaller plans fit the data better than a universal 20% cut, whose margin cost on retained customers is not modelled in this survey.

  4. The Family plan is not the problem. Its lower headline satisfaction is a tenure-mix (Simpson's paradox) artefact, answering the head of product.

    Medium confidenceBase 640

    Overall satisfied (4-5): Family 49.2% (n=301) vs Couples 53.4% (n=339), gap 4.2pp, SE of difference about 3.96pp, 95% threshold about 7.8pp - not significant (z~1.06). Within every tenure band Family is higher: under 3m 69.2% vs 66.1%, 3-12m 61.6% vs 57.4%, over 12m 42.1% vs 35.2%. Family respondents are 67% over-12-months (202/301) vs Couples 31% (105/339), and long tenure satisfaction collapses (66.7% under 3m to 39.7% over 12m). Family is also over-sampled (47% of sample vs 34% of population).

  5. Northern Ireland delivery is an urgent operational flag, not a proven prevalence estimate, answering the operations director.

    Low confidenceBase 12

    Delivery problems as main reason: Northern Ireland 66.7% = 8 of only 12 respondents, vs England 8.1% (n=546), Scotland 13.0% (n=46), Wales 0% (n=36). The NI 95% margin is about +/-27pp, so the true share is very imprecise, though even the lower end is well above England. Only 1.9% of sample is NI (n=12) and population region mix is not given, so volume cannot be estimated from this survey. One NI open-text answer (R445) also cites late/missing boxes.

  6. Fatigue with repetitive recipes grows with tenure and is a second, separate long-tenure problem that skipping alone will not fix.

    Medium confidenceBase 190

    Repetitive recipes as main reason rises 6.5% to 25.4% by tenure (above). In open text, 40 of 190 answers mention variety/repetitiveness, and 25 of those 40 are from over-12-month customers, despite over-12-month being 48% of the sample but only 20% of the population. Satisfaction falls from 66.7% (under 3m) to 39.7% (over 12m).

  7. Open text reveals a large unmeasured need for fewer meals / a 2-meal flexible plan, plus packaging, delivery and portion issues, which the forced-choice Q1 understates.

    Low confidenceBase 190

    Among 190 open-text answers (self-selected, multiple themes allowed): variety/repetitiveness 40, skipping/pausing 48, price/value 43, fewer meals/smaller quantity 28, packaging waste 19, delivery/missing 17, portion size 12, life-change/nothing 6. Recurring specifics: minimum 3-4 meals too many, wanting 2 meals or variable meals week-to-week, skip deadlines days before delivery, caps on consecutive skips, and phone/confirmation friction - all design requirements beyond a single skip button.

Caveats

  • Only 640 of 9,850 invited completed (6.5%; 702 started), so nonresponse bias is likely and unmeasurable. Sample skews long-tenure (48% vs 20% of population) and Family plan (47% vs 34%); weighting in this analysis is by tenure only because no tenure-by-plan or region population cross-tab was supplied.
  • All respondents are self-reported leavers (screeners S1/S2); there are no current customers, so effects on retention, box frequency, revenue per active customer and cannibalisation cannot be estimated.
  • Q1 forces a single 'main reason', masking multiple reasons; open text shows many respondents want two things at once (e.g. skip plus cheaper, variety plus fewer meals).
  • Q3/Q4 are hypothetical stated likelihoods with no behaviour, cost or trade-off; 'fairly likely' inflates net likely, and Q3 preceding Q4 may create order effects.
  • Open text is 190 of 640 respondents (29.7%), self-selected; theme counts are mentions among those who wrote, not population prevalences, and coding of borderline answers (e.g. stale, something other than traybakes) involves judgement.
  • Small subgroups have very wide margins: Northern Ireland n=12 (+/- about 27pp), Scotland n=46, Wales n=36. Region population shares are unknown, so regional volumes cannot be weighted.
  • No economics are in the survey: margin impact of a 20% cut across the retained base, build cost, or value of retained customers. Recall is retrospective (cancelled March-August, surveyed September).

Next steps

  • researchReweight the 640 completes by tenure-by-plan (and region if population data exist) and publish weighted shares with confidence intervals for all Q1/Q2/Q3/Q4 cuts
  • experimentRandomised win-back experiment among lapsed customers: early-access one-tap skip vs 20% off vs flexible 2-meal plan vs control, tracking actual resubscribe and 3- and 6-month retention and revenue
  • monitorAudit Northern Ireland (and Scotland) delivery operations: on-time-in-slot rate, missing-item rate, failed/returned boxes, refunds and postcode coverage, from operational records not survey recall
  • experimentPilot a wider recipe rotation / fresher menu cadence for over-12-month customers and measure repeat-cancellation and satisfaction
  • researchInterview short-tenure price leavers, NI leavers and non-respondents, plus current customers, on price thresholds, 2-meal demand and skip-friction specifics
  • decisionSpecify and ship one-tap skip MVP including multi-week pause, late deadline, one-tap reactivation with saved settings, and usage/revenue monitoring

Open-text themes it coded

Recipe variety / repetitive recipes 40Price / value for money 43Skipping, pausing and flexible weeks 48Fewer meals / smaller weekly quantity 28Portion size too small 12Packaging waste / recyclability 19Delivery reliability, timing and missing items 17Life change / no fix 6

The survey it planned

3 screening questions and 11 questions, as the model wrote them.

  1. S1

    Which of the following best describes your Fernway status?

    One answer
    • I cancelled in the last six months and have not rejoined
    • I cancelled in the last six months but have rejoined since
    • I have not cancelled in the last six months

    Continues if I cancelled in the last six months and have not rejoined

  2. S2

    Were you the person who managed your Fernway subscription and decided to cancel it?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    In total, how long were you a Fernway subscriber before you cancelled?

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months

    Continues if Less than 3 months; 3 to 12 months; More than 12 months

  4. Q1

    In your own words, what was the main reason you cancelled your Fernway subscription?

    Open text
  5. Q2

    Which of the following, if any, contributed to your decision to cancel your Fernway subscription? Please select all that apply.

    Any that apply
    • My circumstances changed (for example, moved house, work, diet or family needs)
    • The recipes and meal variety did not suit my tastes
    • Food quality or freshness was not good enough
    • Portion sizes or number of meals were not right for me
    • Cooking time or effort was too much
    • I had too much food waste
    • Delivery times or reliability were poor
    • I could not easily skip, pause or change deliveries when I needed to
    • The price / cost per meal was too high
    • Other (please specify)
  6. Q3

    And which single reason was most important in your decision to cancel?

    One answer
    • My circumstances changed (for example, moved house, work, diet or family needs)
    • The recipes and meal variety did not suit my tastes
    • Food quality or freshness was not good enough
    • Portion sizes or number of meals were not right for me
    • Cooking time or effort was too much
    • I had too much food waste
    • Delivery times or reliability were poor
    • I could not easily skip, pause or change deliveries when I needed to
    • The price / cost per meal was too high
    • Other (please specify)
  7. Q4

    Overall, how satisfied were you with Fernway while you were subscribed?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  8. Q5

    While you were subscribed, how often did you want to skip a week but decided not to because of the six-day deadline or difficulty changing your delivery in the app?

    One answer
    • Never
    • Once
    • 2-3 times
    • 4 or more times
    • I don't remember
    Judges: double-barrelled

    It states as fact that changing a delivery in the app was difficult, which pushes respondents towards the flexibility explanation. It also combines two separate causes (the six-day deadline and app difficulty) into a single count, so the answer cannot be attributed to either.

    Assumes the respondent decided not to skip because of those specific reasons, and asks about two distinct reasons (six-day deadline or difficulty in the app) at the same time.

  9. Q6

    Please rank the following factors in order of importance to you when deciding whether to subscribe to a meal-kit service, with 1 = most important and 6 = least important.

    Ranking
    • Recipe variety and taste
    • Ability to skip or pause any week easily, including at short notice
    • Food quality and freshness
    • Price
    • Portion size and number of meals
    • Delivery convenience and reliability
  10. Q7

    If Fernway reduced the price of both plans by 20% (with no change to skipping), how likely would you be to rejoin at some point in the next six months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  11. Q8

    If Fernway introduced one-tap skipping of any week, including less than six days ahead (with no price cut), how likely would you be to rejoin at some point in the next six months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  12. Q9

    Which of the following would make you most likely to rejoin Fernway?

    One answer
    • One-tap skipping of any week, including less than six days ahead, with no price cut
    • A 20% cut in plan prices, with no change to skipping
    • Both one-tap skipping and a 20% price cut
    • Something else (please specify)
    • Nothing - I would not rejoin Fernway
  13. Q10

    Is there anything else you would like to tell us about why you cancelled Fernway, or what would encourage you to return?

    Open text
  14. Q11

    Which plan were you on when you cancelled?

    One answer
    • Couples plan (for two people)
    • Family plan (for four people)
Sample plan

Invite by email every UK customer in the Fernway database who cancelled in the last six months (census invitation, no pre-selection; no country screener is needed as Fernway delivers only in the UK). Target at least 1,200 eligible completes, with a minimum of 400 completes in each tenure group from S3 (less than 3 months, 3 to 12 months, more than 12 months). At n=400 per group the margin of error for a 50% proportion is plus/minus 4.9 percentage points at 95% confidence (385 per group is the minimum for plus/minus 5 points); the brief's about 600 completes would be only about 200 per group, or plus/minus 6.9 points, and would not meet the board's requirement. Monitor completes by tenure group, send reminders to under-represented groups, do not screen out eligible respondents, and close once every group has at least 400 completes; weight overall estimates back to the database tenure distribution if final group sizes differ.