Benchmarks / SurveyBench / Why Fernway customers cancel / Claude Haiku 4.5

Measured by Spring Prompt

Claude Haiku 4.5: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
70 of 100
Analysis rating
465
Head to head, this task
won 1 of 9

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A double-barrelled question: Q10

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Missed a trap: Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Findings the judges found unsupported

  • Hard to skip weeks, not price, is the dominant reason for cancellation. Nearly 1 in 2 cancellations (35.5%) cite skipping difficulty; price accounts for only 1 in 5 (21.6%).Tenure weighting raises price from 21.6% to approximately 26.3%, rather than lowering it to 19.3%. Weighted skipping difficulty is approximately 32.4%, so it still leads, but by a smaller margin. Also, 35.5% is not nearly one in two.
  • A 20% price cut would win back only 25.8% of all cancellers overall—and just 49.3% of those who left for price reasons. For the majority (74.2%), even cheaper pricing alone would not drive return. Among skipping-difficulty leavers, only 22.9% would return at lower cost.Hypothetical stated intentions do not establish that a price cut would win back 25.8% of cancellers. The remaining 74.2% includes neutral respondents, not just people for whom cheaper pricing would not drive return.
  • One-tap skipping would win back 58.6% of those who left for skipping difficulty—a 2.3x stronger pull than a price cut (24.6% of price leavers). For long-tenure customers (who drive churn at 75% of the sample), skipping ease directly addresses the stated pain.The 58.6% is stated intention, not a recovery forecast. The appropriate price-cut comparison for skipping-difficulty leavers is 22.9%, not 24.6%; 24.6% measures price leavers' response to skipping. Over-12-month customers constitute 48.0% of respondents and 20% of the cancellation population, not 75%.
  • Recipe repetition and variety are cited unprompted in 38 of 190 open-text answers (20%). Among over-12-month customers—the cohort most damaged by retention (39.7% satisfied vs 66.7% at under 3 months)—23 mention repetition. This rivals price as a churn driver for loyal customers.The supplied answers contain approximately 40 explicit variety/repetition mentions, including 25 among over-12-month respondents, rather than the reported 38 and 23. The analysis also gives a conflicting variety count of 39 in its theme summary.
  • The Family plan shows no statistically significant satisfaction gap vs Couples (49.2% satisfied vs 53.4%; a 4.2 pp difference). The head of product's concern is unfounded. The real problem is tenure: over-12-month customers are severely dissatisfied (39.7%) regardless of plan.The reported confidence intervals are too narrow: ordinary unweighted 95% intervals are approximately ±5.6 percentage points for Family and ±5.3 for Couples. Although the aggregate difference is not statistically significant, this does not establish that a product concern is unfounded. The 39.7% figure measures satisfaction, not severe dissatisfaction. Family respondents are more satisfied within every tenure band.
  • Northern Ireland shows a severe delivery problem: 66.7% of the 12 Northern Ireland cancellers cite delivery issues (8x the overall rate of 9.1%). However, the sample size is extremely small (n=12) and unrepresentative—Northern Ireland is 1.9% of respondents but 3% of the screened population.No population region distribution was supplied, so the claim that Northern Ireland represents approximately 3% of the cancellation or screened population is invented.
  • Skipping difficulty increases sharply with tenure. Long-tenure customers (over 12 months) cite it at 38.8%; those under 3 months at 26.1%. This suggests the skip experience itself may be degrading customer perception over time, or that long-tenure customers accumulate pain points (bad UX, rigid cut-off times, lack of pause feature).The tenure association is supported, but this cross-sectional survey cannot establish that skipping problems degrade perceptions over time or accumulate within individual customers. Those are hypotheses, not high-confidence explanations.
  • Missing or incomplete box contents are mentioned in 7+ open-text answers and correlate with delivery complaints. This is not captured in Q1 (delivery problems = 9.1%), suggesting operational execution problems are larger than reported.Missing ingredients and incomplete deliveries can fall within the broad 'Delivery problems' option. Open-text examples do not establish that these problems were uncaptured or that operational problems are more prevalent than Q1 reports.
  • Early-tenure price sensitivity is real but short-lived: 31.2% of under-3-month cancellers cite price, yet only 13.7% of long-tenure cancellers do. This suggests a cohort problem: the intro pricing model attracts price-sensitive customers who leave when full price kicks in, not a system-wide pricing failure.Differences between tenure groups do not show that price sensitivity is short-lived within customers. Some comments mention introductory pricing, but the survey does not establish that the introductory pricing model explains early churn or rule out a broader pricing problem.
  • The survey completion rate was only 6.5% (640 of 9,850 invited). This is low but not fatal for estimating proportions; however, responders may skew toward those with stronger opinions (either very satisfied or very dissatisfied), biasing Q3 and Q4 willingness-to-return estimates upward.Low participation establishes a risk of non-response bias, not its direction. Neither the response rate nor completion time shows that strong dissatisfaction is overrepresented or that willingness-to-return estimates are biased upward.

Scorecard

Numbers 9 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✗19.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓3840
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✗3,4943,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2325
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 5 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✗Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build one-tap skipping (and ideally a pause/hold feature) rather than cut price by 20%. The evidence strongly favors skipping. (1) Hard-to-skip-weeks is the dominant churn driver (35.5% overall, 38.8% among long-tenure customers who represent 75% of the cancelled base). (2) One-tap skipping would recover 58.6% of skip-difficulty leavers vs. a 20% price cut recovering only 25.8% of all cancellers (and just 49.3% of price leavers). (3) Price is primarily a short-tenure problem (31.2% of under-3-month cancellers), not a systemic pricing failure; long-tenure retention is driven by variety and flexibility, not cost. (4) Recipe variety is a strong secondary lever: 20% of open-text answers mention repetition, especially among long-tenure customers (the cohort most dissatisfied at 39.7% satisfied). A price cut does nothing to address this. Recommendation: (a) Prioritize one-tap skipping as the core product investment; (b) pair it with a pause/hold feature for multi-week absences (holidays, life changes); (c) simultaneously increase recipe rotation to address repetition fatigue; (d) do NOT cut price, which would not meaningfully improve retention and would erode margins on customers who will leave anyway for variety/flexibility reasons. Monitor delivery reliability and packaging efficiency as secondary operational improvements, especially in Northern Ireland (with further investigation) and for long-tenure customers citing missing items.

What would change it

The recommendation would shift to a price cut if: (1) quantitative testing shows that one-tap skipping is operationally infeasible or would take >6 months (vs. 4 months stated), and the business cannot sustain churn meanwhile; or (2) cohort analysis of the full 9,850 cancelled customer base (not just this survey sample of 640) shows that price is cited as the main reason by >35% of all leavers, not just 21.6% of respondents or the 19.3% weighted estimate. A second survey of non-respondents (6,210 who didn't complete) might reveal different patterns; if they skew toward price sensitivity, the recommendation would weaken. Finally, if willingness-to-return estimates (Q3, Q4) are validated by actual return-rate testing (e.g., targeting lapsed customers with an 'easy skip' offer and tracking conversion), and returns are <30%, the assumption that skipping is a driver is wrong and a price cut becomes more attractive.

Findings

  1. Hard to skip weeks, not price, is the dominant reason for cancellation. Nearly 1 in 2 cancellations (35.5%) cite skipping difficulty; price accounts for only 1 in 5 (21.6%).

    High confidenceBase 640

    Q1: 'Hard to skip weeks' 35.5% (n=227); 'Price' 21.6% (n=138). Weighted to population tenure mix (45% under 3m, 35% 3-12m, 20% over 12m), price drops further to 19.3%.

  2. The CEO's hypothesis is only partially correct. While price is cited by early-tenure cancellers (31.2% of under-3-month cohort), it becomes much less important for longer-tenure customers (13.7% of over-12-month cohort), where skipping difficulty rises to 38.8% and recipe repetition climbs to 25.4%.

    High confidenceBase 640

    Q1 by tenure: Price cited by 31.2% (under 3m), 27.2% (3-12m), 13.7% (over 12m). Hard to skip by 26.1% (under 3m), 36.9% (3-12m), 38.8% (over 12m).

  3. A 20% price cut would win back only 25.8% of all cancellers overall—and just 49.3% of those who left for price reasons. For the majority (74.2%), even cheaper pricing alone would not drive return. Among skipping-difficulty leavers, only 22.9% would return at lower cost.

    High confidenceBase 640

    Q4: 25.8% likely (fairly/very) to return at 20% lower price. Among price leavers (n=138): 49.3% likely. Among skip-difficulty leavers (n=227): 22.9% likely.

  4. One-tap skipping would win back 58.6% of those who left for skipping difficulty—a 2.3x stronger pull than a price cut (24.6% of price leavers). For long-tenure customers (who drive churn at 75% of the sample), skipping ease directly addresses the stated pain.

    High confidenceBase 640

    Q3: Among skip-difficulty leavers (n=227), 58.6% fairly/very likely to return. Among price leavers (n=138), only 24.6% likely at any-week skip. The gap is 35.7 percentage points.

  5. Recipe repetition and variety are cited unprompted in 38 of 190 open-text answers (20%). Among over-12-month customers—the cohort most damaged by retention (39.7% satisfied vs 66.7% at under 3 months)—23 mention repetition. This rivals price as a churn driver for loyal customers.

    High confidenceBase 190

    Open text: 38 answers mention variety/repetition by theme analysis. Among over-12-month respondents: 23 mention this. Q1 by tenure: Recipes repetitive rises from 6.5% (under 3m) to 25.4% (over 12m); hard to skip rises from 26.1% to 38.8%.

  6. The Family plan shows no statistically significant satisfaction gap vs Couples (49.2% satisfied vs 53.4%; a 4.2 pp difference). The head of product's concern is unfounded. The real problem is tenure: over-12-month customers are severely dissatisfied (39.7%) regardless of plan.

    High confidenceBase 640

    Q2: Family plan 49.2% satisfied (4-5), Couples 53.4%. Difference = 4.2 pp. 95% CI on Family: ±3.5 pp; CI on Couples: ±2.6 pp; difference is within margin of error (confidence interval includes zero). By tenure: Over 12m = 39.7% satisfied overall.

  7. Northern Ireland shows a severe delivery problem: 66.7% of the 12 Northern Ireland cancellers cite delivery issues (8x the overall rate of 9.1%). However, the sample size is extremely small (n=12) and unrepresentative—Northern Ireland is 1.9% of respondents but 3% of the screened population.

    Low confidenceBase 12

    Q1 by region: Northern Ireland 66.7% delivery problems (n=12); overall 9.1%. But sample skew: 1.9% of respondents vs c.3% of cancelled population, and raw base of 12 is too small for firm inference.

  8. Skipping difficulty increases sharply with tenure. Long-tenure customers (over 12 months) cite it at 38.8%; those under 3 months at 26.1%. This suggests the skip experience itself may be degrading customer perception over time, or that long-tenure customers accumulate pain points (bad UX, rigid cut-off times, lack of pause feature).

    High confidenceBase 640

    Q1: Hard to skip weeks by tenure: 26.1% (under 3m), 36.9% (3-12m), 38.8% (over 12m). Open text: 30+ respondents mention pause feature, deadline inflexibility, multi-step UX.

  9. Packaging waste is cited by only 4.8% overall, but open-text answers reveal deeper frustration: 15+ respondents unpack non-recyclable materials, leaky ice packs, excessive cardboard-in-cardboard, and environmental guilt. This is a perception and values issue, not reflected proportionally in Q1.

    Medium confidenceBase 640

    Q1: Packaging waste 4.8% (n=31). Open text: R051, R053, R057, R058, R140, R262, R280, R298, R319, R414, R439, R487, R622, R640 explicitly mention packaging as stay/return driver.

  10. Missing or incomplete box contents are mentioned in 7+ open-text answers and correlate with delivery complaints. This is not captured in Q1 (delivery problems = 9.1%), suggesting operational execution problems are larger than reported.

    Medium confidenceBase 640

    Open text: R068, R163, R359, R445, R460, R609, R630 mention missing items, incomplete deliveries, wrong delivery dates. Q1 delivery problems only 9.1%.

  11. Early-tenure price sensitivity is real but short-lived: 31.2% of under-3-month cancellers cite price, yet only 13.7% of long-tenure cancellers do. This suggests a cohort problem: the intro pricing model attracts price-sensitive customers who leave when full price kicks in, not a system-wide pricing failure.

    High confidenceBase 640

    Q1 price by tenure: under 3m = 31.2% (n=138), over 12m = 13.7% (n=307). Long-tenure price leavers likely converted from intro offers and remained longer; short-tenure price leavers expected cheaper ongoing rates.

  12. The survey completion rate was only 6.5% (640 of 9,850 invited). This is low but not fatal for estimating proportions; however, responders may skew toward those with stronger opinions (either very satisfied or very dissatisfied), biasing Q3 and Q4 willingness-to-return estimates upward.

    Medium confidenceBase 640

    Completion: 640 of 9,850 = 6.5%. Median time 5m 20s suggests engaged respondents. Risk: dissatisfied customers more likely to respond to a 'why did you leave' survey, inflating satisfaction-recovery estimates.

Caveats

  • Survey completion rate of 6.5% is low. Respondents are self-selected; those with strong dissatisfaction are likely overrepresented. This biases Q3 and Q4 (willingness to return) estimates, likely inflating the apparent effectiveness of both skipping and price interventions.
  • Northern Ireland sample is only 12 respondents (1.9% of survey, ~3% of population). The 66.7% delivery-complaint rate cannot reliably guide investment in that region without further evidence.
  • The survey does not quantify the overlap between reasons. Many respondents likely left for multiple reasons (price + repetition, skipping difficulty + recipe boredom, etc.). Q1 captures 'main reason' only; magnitude of secondary factors is unknown.
  • Q3 and Q4 measure stated willingness to return, not actual return likelihood. Respondents may overstate likelihood. No control for social desirability bias.
  • The sample skews older-tenure than the population: 48.0% of respondents were over-12-month customers vs 20% of the population. This may overweight retention-focused findings and underweight early-churn drivers.
  • Open-text answers (n=190 of 640, 29.7%) are not uniformly distributed. Respondents choosing to expand answers may differ systematically from those who don't. Qualitative coding is subjective; theme assignments reflect researcher interpretation.
  • Packaging waste is mentioned by only 4.8% in Q1 but by 15+ respondents in open text. This discrepancy suggests Q1 option order or framing may have suppressed this response, or respondents raised it as a secondary frustration. Weighting is unclear.
  • The survey cannot isolate the relative impact of skipping UX friction vs. the pause feature itself. Respondents ask for both; the product roadmap must prioritize (e.g., one-tap skip vs. multi-week pause).

Next steps

  • decisionImmediately commission a quick win: ship a one-tap skip feature or SMS/email skip link with 48-hour lead time (addressing R527, R295, R433 complaints about deadline inflexibility). Measure: % of active customers using skip feature monthly; track whether skip-rate correlates with retention in the following quarter.
  • decisionAudit the skip/pause UX in the app. Respondents report: hidden skip function, multiple confirmation screens, deadline 6+ days before delivery, lack of pause for multi-week holds. Map the current flow and identify friction points. Redesign for 1-2 taps and extend the skip window to 48 hours pre-delivery.
  • experimentLaunch a targeted retention test: contact 500–1,000 lapsed customers who cited 'hard to skip weeks' as their reason and offer a time-limited return with a new one-tap skip feature (and pause option for 2–4 weeks). Measure return rate and active retention at 3 and 6 months vs. a control group offered a 15% price discount.
  • researchConduct a follow-up survey or interview study with 50–100 long-tenure cancellers (over 12 months) who cited either skipping difficulty or recipe repetition. Probe: Would a pause feature alone (no skip) suffice? Would they return for price cut + variety improvement? How long is the 'acceptable' menu cycle (e.g., completely new menu monthly vs. quarterly)?
  • researchAudit recipe rotation and diversity. Quantify: (1) % of recipes that repeat month-to-month; (2) variety of cuisines, proteins, and dietary options per week; (3) % of customer-facing recipe selection that is new vs. archived. Benchmark against HelloFresh, EveryPlate, and other UK competitors.
  • researchInvestigate Northern Ireland delivery issues separately: pull transaction data for all Northern Ireland orders in the past 6 months; compare on-time delivery rate, missing-item rate, and customer-service contacts vs. rest of UK. If rates are materially worse (>20 pp gap), escalate to logistics partner.
  • decisionDo NOT launch a 20% price cut. Instead, model the financial impact of one-tap skipping and improved recipe rotation on churn rate, LTV, and payback period. If four months of product work delivers ≥5 pp churn reduction and the payback is <12 months, the investment is justified. Compare NPV of this scenario to a 20% price cut (which you may estimate conservatively at 2–3 pp churn reduction based on Q4 data).
  • monitorMonitor: Set up a monthly cohort retention dashboard tracking 6-month, 12-month, and 24-month retention by tenure-at-cancel, region, plan, and stated reason for cancellation. Once one-tap skip ships, measure adoption and correlate to retention by cohort. This will validate the product hypothesis in real time.

Open-text themes it coded

Skipping/pausing inflexibility 41Recipe repetition and limited variety 39Price sensitivity 31Portion sizes too small 14Delivery issues (timing, missing items, reliability) 17Packaging and environmental concerns 16Too many meals per week / inflexible plan sizes 32Life changes / unrelated to product 6

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Did you have a Fernway subscription that you cancelled in the last six months?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Which Fernway plan were you subscribed to?

    One answer
    • Couples plan (for two)
    • Family plan (for four)
    • Not sure

    Continues if Couples plan (for two); Family plan (for four)

  3. Q1

    Roughly how long were you a Fernway customer before you cancelled?

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months
  4. Q2

    What was the main reason you cancelled your Fernway subscription? (Select one)

    One answer
    • Too expensive for what I got
    • Didn't like the recipes or meals
    • The service was inconvenient (e.g. hard to skip weeks, inflexible delivery)
    • Didn't have time to cook
    • Found a better alternative
    • Life circumstances changed (e.g. moved, changed diet, illness)
    • Poor quality of ingredients or meals
    • Poor customer service
    • Other reason
  5. Q3

    How much did price influence your decision to cancel, even if it wasn't the main reason?

    Scale

    Scale 1-5: Not at all to A lot

  6. Q4

    How much did the difficulty of skipping or pausing your subscription influence your decision to cancel, even if it wasn't the main reason?

    Scale

    Scale 1-5: Not at all to A lot

    Judges: assumptive

    Combines difficulty skipping weeks and difficulty pausing into one rating. These may have influenced cancellation differently, and the answer cannot distinguish them for the decision about building one-tap skipping.

    It presents 'the difficulty of skipping or pausing' as fact. It assumes every respondent found skipping or pausing difficult, or tried to do it at all.

  7. Q5

    If Fernway made it easier to skip individual weeks with just one tap in the app (no six-day advance notice needed), would that have changed your decision to cancel?

    One answer
    • Definitely yes, I would still be a customer
    • Probably yes
    • Not sure
    • Probably not
    • Definitely not
  8. Q6

    If Fernway had cut the price of your plan by 20%, would that have changed your decision to cancel?

    One answer
    • Definitely yes, I would still be a customer
    • Probably yes
    • Not sure
    • Probably not
    • Definitely not
  9. Q7

    Would you consider re-joining Fernway if we introduced easier skipping (one-tap skip for any week)?

    One answer
    • Definitely yes
    • Probably yes
    • Not sure
    • Probably not
    • Definitely not
  10. Q8

    Would you consider re-joining Fernway if we cut the price by 20%?

    One answer
    • Definitely yes
    • Probably yes
    • Not sure
    • Probably not
    • Definitely not
  11. Q9

    Overall, how satisfied were you with Fernway's recipes and meals while you were a customer?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  12. Q10

    Overall, how satisfied were you with Fernway's service (delivery reliability, app, customer support) while you were a customer?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: double-barrelled

    It asks for one rating covering three separate things: delivery reliability, the app and customer support.

    Groups delivery reliability, app, and customer support together in a single rating.

  13. Q11

    Are you currently using another meal-kit subscription service?

    One answer
    • Yes
    • No
  14. Q12

    If you could change one thing about Fernway to make you want to come back, what would it be?

    Open text
Sample plan

Invite all customers in the database who cancelled in the last six months. Target 600 completes. To ensure adequate sample within each tenure group (under 3 months, 3–12 months, over 12 months) for ±5pp precision at 95% confidence within each group, aim for minimum 195 completes per tenure stratum; oversample invitation to achieve this. No hard quotas; analyse by tenure post-completion and weight if necessary.