Benchmarks / SurveyBench / Why Fernway customers cancel / Claude Haiku 5.5

Measured by Spring Prompt

Claude Haiku 5.5: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
86 of 100
Analysis rating
1,412
Head to head, this task
won 6 of 9

Why it is not sound

The survey

  • A double-barrelled question: Q4

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Findings the judges found unsupported

  • A 20% price cut has strong stated appeal only among customers who left over price, and that group is roughly a fifth to a quarter of cancellers. A blanket cut would give away margin on the other three quarters, who mostly would not return for price.The survey does not establish that a blanket cut would give away margin on the other three quarters of cancellers. Cancellers who do not return generate no subscription margin to give away, and the effect on existing customers was not measured.
  • Family cancellers are more likely than Couples to cite repetitive recipes, which points to a menu-rotation issue specific to the Family plan. Hard-to-skip rates are the same for both plans.The unadjusted plan difference does not establish a Family-specific menu problem. Family respondents disproportionately have long tenure, and repetition complaints increase strongly with tenure. No within-tenure comparison of repetition by plan is supplied.
  • Packaging and delivery reliability are under-measured by the closed question. Each is raised in open text more often than its closed-question share suggests.Delivery is mentioned in 18 supplied open-text answers, not 16. Furthermore, the reported 8% is below, not above, the closed-question delivery share of 9.1%. Comparing optional, multiple-theme open responses with a single main-reason question does not establish that either delivery or packaging is under-measured.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 6 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Prioritise the one-tap skip build, scoped to include a multi-week pause and clearer skip rules. Do not cut all plans by 20% on this evidence. The survey shows skipping has the larger stated effect and addresses the most-cited reason, while the price cut's appeal is concentrated among price-driven leavers, who are a minority. If commercial pressure favours a price lever, test a targeted offer to lapsed price-driven customers with a holdout, not a list-price cut. In parallel and at low cost, check Northern Ireland delivery performance, plan a menu-rotation fix for the Family plan, and address packaging. Do not read the Family plan as the cause of dissatisfaction.

What would change it

(a) A holdout test or pilot showing one-tap skipping or pausing does not improve reactivation among lapsed customers, or shows a much smaller effect than stated intent. (b) A holdout price test showing a 20% cut or targeted offer reactivates customers at a rate that more than covers the margin given up, on a per-customer and 12-month basis. (c) Reason-coded CRM data for all 9,850 cancellers, weighted properly, showing price as the majority reason. (d) Evidence from delivery logs that the NI issue is large and systematic, which would raise its priority but not change the feature decision.

Findings

  1. Hard to skip weeks is the largest single reason for cancelling. Price is second, and it is a real but minority driver.

    High confidenceBase 640

    Main reason, unweighted: hard to skip weeks 35.5% (227 of 640), price 21.6% (138), recipes repetitive 18.0% (115). Weighted to the population tenure mix (45/35/20): hard to skip 32.4%, price 26.3%. Price is concentrated in recent cancellers (31.2% under 3 months vs 13.7% over 12 months).

  2. On stated intent, one-tap skipping would win back more cancellers than a 20% price cut across the whole cancelling base (36.4% vs 25.8% likely to return). Among customers who left over skipping, the gap is large (58.6% vs 22.9%).

    Medium confidenceBase 640

    Q3 net likely (4-5), all respondents: 36.4%. Q4 net likely, all respondents: 25.8%. Among hard-to-skip cancellers (base 227): Q3 58.6% (margin about +/-6.4 pts) vs Q4 22.9% (about +/-5.5 pts). The Q3-Q4 comparison is within-person, so the gap is likely real, but both are hypothetical.

  3. A 20% price cut has strong stated appeal only among customers who left over price, and that group is roughly a fifth to a quarter of cancellers. A blanket cut would give away margin on the other three quarters, who mostly would not return for price.

    Medium confidenceBase 138

    Q4 net likely among price cancellers: 49.3% (138 respondents, margin about +/-8.3 pts). Across all respondents, 25.8%. Price cancellers are 21.6% unweighted or 26.3% tenure-weighted of cancellers. No margin or cost data was provided, so the margin cost of a 20% cut could not be quantified.

  4. The Family plan is not the satisfaction problem. Family satisfaction is not significantly lower than Couples, and Family is equal or higher within every tenure band.

    High confidenceBase 640

    Satisfied (4-5): Family 49.2% (301) vs Couples 53.4% (339). Gap 4.2 pts, margin of error about +/-7.8 pts, so not significant. Within tenure: under 3 months Family 69.2% (n=26) vs Couples 66.1% (n=112); 3-12 months Family 61.6% (n=73) vs Couples 57.4% (n=122); over 12 months Family 42.1% (n=202) vs Couples 35.2% (n=105, gap not significant).

  5. Family cancellers are more likely than Couples to cite repetitive recipes, which points to a menu-rotation issue specific to the Family plan. Hard-to-skip rates are the same for both plans.

    Medium confidenceBase 640

    'Recipes got repetitive' as main reason: Family 23.6% (301) vs Couples 13.0% (339), gap 10.6 pts, margin of error about +/-6.0 pts, so significant. Hard to skip: Family 35.2% vs Couples 35.7%.

  6. Recipe variety is a recurring complaint beyond the closed question. It is most common among long-tenure customers, who are also the least satisfied.

    Medium confidenceBase 190

    40 of 190 open-text answers (21%) mention variety or repetition, 25 of them from customers over 12 months. Q1 closed item: 18.0% of 640. Over-12-month customers: 39.7% satisfied (lowest of the three tenure groups).

  7. Customers want more pause and skip flexibility than one-tap skipping of single weeks. Many open-text answers ask for holiday or multi-week pauses that keep their settings, skip windows closer to delivery, and skip limits.

    Medium confidenceBase 190

    46 of 190 open-text answers (24%) mention skipping, pausing, skip deadlines or holds. Examples include R264 ('an actual pause'), R493 (pause up to 8 weeks), R217 (skips capped at a few weeks), R295 (deadline nearly a week before delivery). The Q3 hypothetical asks only about one-tap single-week skipping.

  8. Some customers want a smaller or more flexible plan size (fewer meals or a one-meal week) rather than a lower price. Portion size complaints are a smaller secondary theme.

    Medium confidenceBase 190

    30 of 190 open-text answers (16%) ask for fewer meals or variable meal counts (for example R059, R222, R337, R634). 12 (6%) mention portion size (for example R021, R535, R632). Closed Q1 'portion sizes' is 6.7%.

  9. Packaging and delivery reliability are under-measured by the closed question. Each is raised in open text more often than its closed-question share suggests.

    Medium confidenceBase 190

    Packaging: 19 of 190 open-text answers (10%) vs 4.8% of 640 as main reason. Delivery reliability, including missing items and late or failed drops: 16 of 190 (8%) vs 9.1% of 640 as main reason. Individual answers vary in how much they say about the central issue.

  10. Northern Ireland delivery may be a genuine operational problem, but the base is very small. Delivery was cited by 8 of 12 NI respondents (66.7%), against 8.1% in England. NI is 1.9% of the sample but 8 of 58 delivery-related cancellations (14%).

    Low confidenceBase 12

    Q1 'delivery problems' by region: England 8.1% (546), Scotland 13.0% (46), Wales 0.0% (36), NI 66.7% (12). 95% Wilson interval for NI: about 39% to 88%. Open text from NI includes R445 (boxes not on the promised day, nothing missing). Its direction is consistent with an operational issue, but the sample cannot establish a rate.

  11. The sample is heavily skewed. Long-tenure customers and Family plan customers are over-represented, and recent cancellers are under-represented. Unweighted headline figures therefore overstate repetition and understate price.

    High confidenceBase 640

    Tenure: under 3 months 21.6% of sample vs 45% of population; 3-12 months 30.5% vs 35%; over 12 months 48.0% vs 20%. Plan: Family 47.0% vs 34% of population. Weighting to tenure alone raises price from 21.6% to 26.3%. Plan and tenure cannot be weighted jointly from the tables supplied.

Caveats

  • Response was 6.5% (640 of 9,850 invited). Those who answered may differ systematically from those who did not, for example in how strongly they felt about the cancellation.
  • The sample is skewed on tenure (48% over 12 months vs 20% of cancellers) and on plan (47% Family vs 34%). Only tenure weighting was possible. Plan weighting alone gives price 22.1%. Tenure and plan cannot be weighted jointly from the tables supplied, so the headline figures are approximate.
  • Q3 and Q4 are hypothetical 'likelihood to return' questions. Stated intent typically overstates real return behaviour, and it does not show how many customers would return or for how long. These numbers rank the options; they do not size the return.
  • Q1 gives only one main reason. Customers who left for several reasons, for example price plus repetition, are represented only by the one they chose. The closed list has no option for multi-week pause or for a price increase, so those needs appear only in open text.
  • Open-text analysis covers 190 of 640 respondents (30%), who chose to answer. Theme counts are single-coder judgements. Answers that mention more than one theme are counted under each. Counts are not weighted.
  • Satisfaction is retrospective, asked after cancelling, so it may reflect the cancellation itself.
  • Subgroup bases are small: NI n=12, Wales n=36, Scotland n=46, under-3-month Family n=26, price cancellers n=138. Margins of error are shown where relevant. Regional and small-subgroup findings should not be generalised.
  • No cost, margin, retention or lifetime-value data was supplied. The margin cost of a 20% price cut and the build cost of one-tap skipping cannot be compared from this survey alone.
  • Recall spans up to six months (March to August 2025 cancellations), which may distort reasons for early cancellers.

Next steps

  • decisionApprove the one-tap skip build, scoped as a first release to include a multi-week pause and clear skip-deadline rules. Confirm that the four-month estimate covers the pause feature. If it does not, check whether a skip-only release can ship in under two months.
  • experimentRun a targeted 20% (or smaller) win-back offer for lapsed customers who cited price, with a randomised holdout, and track margin impact.
  • researchRun an operational audit of Northern Ireland deliveries (on-time, missing items, failed drops, courier) for March to August 2025, compared with England and Scotland.
  • experimentTest a menu-rotation change for the Family plan: a larger recipe pool or a faster refresh, and check whether repetition-driven cancellation falls.
  • researchRe-run the cancellation analysis weighted jointly by tenure and plan using the full 9,850 database, and add reason codes for pause, price rise and skip deadline to the cancellation flow.
  • researchRun a short qualitative and prototype study with lapsed customers on skip and pause designs, including what pause length and deadline they need.
  • monitorMonitor delivery and packaging KPIs monthly (on-time rate, missing-item rate, packaging recyclability) by region and plan, and track churn reasons.

Open-text themes it coded

Recipe variety and repetition 40Price, value and discounts 42Skipping and pausing flexibility 46Fewer meals or flexible plan size 30Portion size 12Packaging and waste 19Delivery reliability and missing items 16Life circumstances (not addressable by product) 6

The survey it planned

3 screening questions and 12 questions, as the model wrote them.

  1. S1

    Do you live in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Did you cancel your Fernway subscription in the last six months?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    How long were you a Fernway customer before you cancelled?

    One answer
    • Under 3 months
    • 3 to 12 months
    • Over 12 months

    Continues if Under 3 months; 3 to 12 months; Over 12 months

  4. Q1

    Which one of these was the main reason you cancelled your Fernway subscription?

    One answer
    • Price was too high for what I got
    • I could not skip or pause when my plans changed
    • I did not cook the meals often enough
    • The recipes did not suit my tastes or household
    • Portions were too small or too large
    • Delivery problems (late, damaged or missing items)
    • I switched to another way of eating or cooking
    • Other (please describe)
  5. Q2

    Which of these, if any, also contributed to your decision to cancel? Select all that apply.

    Any that apply
    • Price was too high for what I got
    • I could not skip or pause when my plans changed
    • I did not cook the meals often enough
    • The recipes did not suit my tastes or household
    • Portions were too small or too large
    • Delivery problems (late, damaged or missing items)
    • I switched to another way of eating or cooking
    • Other (please describe)
    • None of these
  6. Q3

    Overall, how satisfied were you with Fernway while you were a subscriber?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  7. Q4

    How satisfied were you with the delivery service: timing, condition of the boxes and reliability?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: double-barrelled

    One satisfaction rating covers delivery timing, box condition and reliability. Respondents could be satisfied with some of these and dissatisfied with others, leaving the rating's meaning unclear.

    One rating covers three separate things: timing, condition of the boxes and reliability. A respondent happy with timing but unhappy with box condition cannot answer accurately.

  8. Q5

    Think about the meal kit service you would choose next. Rank these from most important (1) to least important (6).

    Ranking
    • Price per meal
    • Being able to skip any week
    • Being able to pause for several weeks
    • Being able to order fewer meals in a week
    • Choice of recipes
    • Reliable delivery
  9. Q6

    While you were subscribed, how often did you want to skip a week or pause your plan and find it difficult or impossible to do so?

    One answer
    • Never
    • Once or twice
    • Several times
    • Often
  10. Q7

    Imagine you could have had one of the following changes while you were subscribed. Which would have been more likely to keep you as a customer?

    One answer
    • Option A: the same plan at 20% lower price, with delivery changes still needing six days' notice
    • Option B: the same price, with any week skippable in one tap and no notice needed
    • Both A and B would have been needed to keep me
    • Neither would have kept me
  11. Q8

    To show you are reading carefully, please select 'Somewhat agree' for this question.

    One answer
    • Strongly disagree
    • Somewhat disagree
    • Neither agree nor disagree
    • Somewhat agree
    • Strongly agree
  12. Q9

    How likely would you be to start a Fernway subscription again in the next six months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  13. Q10

    Which of these, if any, would make you more likely to rejoin Fernway? Select all that apply.

    Any that apply
    • A lower price on my plan
    • Being able to skip any week with no notice
    • Being able to pause my subscription for a few weeks
    • Being able to order fewer meals in a week
    • A wider choice of recipes
    • More reliable delivery
    • Nothing would bring me back
  14. Q11

    What, if anything, would need to change for you to consider Fernway again?

    Open text
  15. Q12

    Which plan were you on when you cancelled?

    One answer
    • Couples plan (for two)
    • Family plan (for four)
Sample plan

Invite every UK customer in the Fernway database who cancelled in the six months before fieldwork and has a valid email address. Apply the screener (S1 to S3) and do not filter further. Target 1,152 completes: 384 per tenure group (under 3 months, 3 to 12 months, over 12 months). At 95% confidence and worst-case proportion 0.5, n=384 gives plus or minus 5 points per group. The briefed figure of about 600 completes (roughly 200 per group) gives only about plus or minus 6.9 points per group and does not meet the board's precision requirement. If the database cannot yield 384 completes in a tenure group, report that group's achieved margin of error and do not pool it with the others. Quotas: set tenure from database records and check against S3; close each tenure group when its quota is reached and keep inviting the others. Plan type (Q12) has no hard quota; monitor it against database proportions. Send one reminder at day 3 and one at day 7. Keep all completes in the dataset; run a sensitivity analysis excluding those who fail the attention check (Q8). Do not offer an incentive that could bias answers toward any option.