Sign inTest your prompt

Benchmarks / SurveyBench / Why Fernway customers cancel / GPT-6 Luna

Measured by Spring Prompt

GPT-6 Luna: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
85 of 100
Analysis rating
1,102
Head to head, this task
won 4 of 10

Why it is not sound

The survey

  • Answer bands that overlap or leave gaps

The analysis

  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓3840
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2325
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 7 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Choose the skip-flexibility direction over an across-the-board 20% price cut, but stage the four-month build rather than treating stated intent as a forecast. Validate a low-cost prototype or operational workaround first, then build a prominent, reliable one-tap skip that preserves account settings and makes deadlines clear. Consider pause functionality alongside it: many open-text requests are for breaks lasting several weeks, which may exceed a single-week skip. In parallel, test targeted price or loyalty offers with price-sensitive customers rather than reducing every plan price. Address recipe repetition separately, especially for longer-tenure customers. Do not make a Family-plan redesign or Northern Ireland-wide delivery intervention on these data alone.

What would change it

I would reconsider a broad price reduction if a controlled price test showed a materially higher incremental return and retention rate, with contribution margin and customer lifetime value remaining better than the skip-feature case. I would reduce or defer the skip build if a prototype or randomized rollout showed little behavioural lift, or if most skip-reason customers actually need multi-week pauses or variable meal quantities that the proposed feature will not provide. A sufficiently powered plan-by-tenure analysis or current-customer research showing a genuine Family-specific problem would change the plan recommendation; verified route-level delivery failures would change the Northern Ireland response.

Findings

  1. Prioritise making skips genuinely easy over a blanket 20% price cut. Skipping difficulty is the leading stated cancellation reason, and the feature has the strongest stated return appeal among those who named it.

    Medium confidenceBase 640

    Hard to skip weeks was the main reason for 35.5% of respondents (n=227); tenure-weighting this reason gives about 32.4% of the cancelled-customer population, or roughly 3,193 of 9,850. Among this group, 58.6% said they would be likely to return with one-tap skipping, versus 22.9% with a 20% price cut, a 35.7-point difference.

  2. Price is a material churn issue, but the survey does not support the claim that it is the main reason most customers leave or justify a blanket 20% cut on its own.

    Medium confidenceBase 640

    Price was the main reason for 21.6% of respondents (n=640), with a 95% margin of error of about 3.2 percentage points. Tenure-weighting raises the estimate to 26.3%. Among price-reason leavers, 49.3% (n=138) said they would be likely to return at 20% lower prices; that is stated intention, not observed return behaviour.

  3. The Family plan is not demonstrably the satisfaction problem. The small overall satisfaction gap is not statistically significant and reverses within each tenure group; Family’s lower aggregate score is plausibly compositional rather than a plan-specific deficit.

    Medium confidenceBase 640

    Satisfied (4–5) was 49.2% for Family (n=301) and 53.4% for Couples (n=339), a 4.2-point gap that is not significant at 95%. Within under-3-month, 3–12-month, and over-12-month groups, Family satisfaction was respectively 69.2%, 61.6%, and 42.1%, versus 66.1%, 57.4%, and 35.2% for Couples. Family respondents also more often named repetitive recipes as their main reason (23.6% vs 13.0%).

  4. Do not conclude that Northern Ireland has a confirmed delivery-system problem from this survey; treat the signal as a reason to investigate specific routes and orders.

    Low confidenceBase 12

    Eight of the 12 Northern Ireland respondents named delivery problems as their main reason (66.7%), compared with 9.1% overall. The Northern Ireland base is only 12, so each respondent represents 8.3 percentage points; one open-text respondent there also described a delivery issue.

  5. Recipe repetition is a meaningful retention issue, particularly for longer-tenure customers, and will not be solved by either skipping improvements or a price cut alone.

    Medium confidenceBase 190

    Recipes got repetitive was the main cancellation reason for 18.0% overall and 25.4% of over-12-month respondents. In open text, 38 of 190 answers mention recipe variety or repetition; 23 of those are from over-12-month customers.

Caveats

  • Only 640 of 9,850 invitees completed the survey: a 6.5% completion rate. Respondents may differ from nonrespondents, so estimates are not automatically representative of all cancelled customers.
  • The completes overrepresent customers with more than 12 months’ tenure (48.0% vs 20% in the population) and Family customers (47.0% vs 34%). The requested price and skip estimates are weighted for tenure only; no joint population distribution is supplied to correct for plan as well.
  • The 95% margin of error is a simple random-sample calculation and does not account for nonresponse or coverage bias.
  • The return questions are hypothetical stated likelihood, not causal evidence of return, retention, or revenue. The one-tap question also does not establish that a one-tap skip alone solves needs for long pauses, late changes, or variable meal quantities.
  • A single main-reason response understates secondary reasons. Open-text answers show overlapping issues, including price alongside flexibility, and flexibility alongside recipe variety.
  • Northern Ireland estimates are especially unstable because the base is only 12. No population regional mix or delivery-route data is provided.
  • The survey reports satisfaction among people who cancelled, not current-customer satisfaction or a causal comparison of plan performance.

Next steps

  • experimentPrototype a one-tap skip and a multi-week pause flow, including deadline reminders and an easy resume path; test with recently cancelled customers before committing the full four months.
  • experimentRun a randomized, targeted price-offer test among price-reason leavers or at-risk customers; model a 20% permanent cut separately from limited discounts.
  • researchUse tenure-weighted results as the primary population estimate and obtain plan-by-tenure population counts before producing further subgroup estimates.
  • researchReview menu rotation and recipe choice with longer-tenure customers, and test changes independently of the price/skip decision.
  • monitorAudit Northern Ireland deliveries at order and route level before proposing a regional fix.
  • decisionProceed with a staged product decision: reserve the skip-flexibility path, but release full build funding only after the prototype demonstrates behavioural lift and the value case is compared with price-test economics.

Open-text themes it coded

Skip, pause, and subscription flexibility 44Price, value, discounts, or price stability 42Recipe variety or repetitive recipes 38Number of meals, box size, or portion size 42Delivery reliability, missing items, or damaged orders 18Packaging waste or difficult-to-recycle materials 19Life change or circumstances outside Fernway’s offer 6

The survey it planned

1 screening question and 12 questions, as the model wrote them.

  1. S1

    Were you a Fernway subscriber who cancelled your subscription within the past six months?

    One answer
    • Yes
    • No
    • I’m not sure

    Continues if Yes

  2. Q1

    From your first Fernway delivery to the date you cancelled, about how long were you subscribed?

    One answer
    • Less than 3 months
    • 3 months to 12 months inclusive
    • More than 12 months
    • I’m not sure
  3. Q2

    What were the reasons you decided to cancel your Fernway subscription? Select all that apply.

    Any that apply
    • The subscription cost too much
    • I needed more flexibility to skip or pause deliveries
    • I needed a different number of meals or portions
    • The recipes did not suit me
    • The meals or ingredients were not good enough
    • There was too much food or food went to waste
    • Delivery days or times did not suit me
    • There were problems with delivery
    • There were problems with the app or managing my subscription
    • I had a change in circumstances, such as moving, a change in household, or a change in finances
    • I wanted to shop for meals myself
    • Other
    • I don’t remember
  4. Q3

    Which of those was the main reason you decided to cancel?

    One answer
    • The subscription cost too much
    • I needed more flexibility to skip or pause deliveries
    • I needed a different number of meals or portions
    • The recipes did not suit me
    • The meals or ingredients were not good enough
    • There was too much food or food went to waste
    • Delivery days or times did not suit me
    • There were problems with delivery
    • There were problems with the app or managing my subscription
    • I had a change in circumstances, such as moving, a change in household, or a change in finances
    • I wanted to shop for meals myself
    • Other
    • I don’t remember
  5. Q4

    Thinking about your Fernway experience while you were subscribed, overall how satisfied were you?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  6. Q5

    Which parts of Fernway, if any, met or exceeded your expectations while you were subscribed? Select all that apply.

    Any that apply
    • The taste of the meals
    • The variety of recipes
    • The quality of the ingredients
    • The ease of preparing the meals
    • The portion sizes
    • The delivery service
    • The app and subscription management
    • The value for money
    • None of these
    • I don’t remember
  7. Q6

    Before you cancelled, how important would it have been to you to have more flexibility with your subscription, such as being able to skip or pause deliveries or change the number of meals?

    Scale

    Scale 1-5: Not at all important to Extremely important

  8. Q7

    Before you cancelled, how important would an ongoing 20% reduction in your subscription price have been to you?

    Scale

    Scale 1-5: Not at all important to Extremely important

  9. Q8

    Which of the following changes, if any, would have made you more likely to keep your subscription? Select all that apply.

    Any that apply
    • Being able to skip a delivery at any time, including less than six days before delivery
    • Being able to pause my subscription for a while
    • Being able to order fewer meals
    • An ongoing 20% reduction in the subscription price
    • Other
    • None of these
  10. Q9

    If you could restart your subscription with Fernway allowing you to skip any upcoming delivery in one tap, even less than six days before delivery, how likely would you be to restart?

    Scale

    Scale 1-5: Very unlikely to Very likely

  11. Q10

    If you could restart your subscription at an ongoing price 20% lower than the price you paid before cancelling, with the other subscription terms unchanged, how likely would you be to restart?

    Scale

    Scale 1-5: Very unlikely to Very likely

  12. Q11

    Which one change, if any, would be most likely to persuade you to restart your Fernway subscription?

    One answer
    • Being able to skip any upcoming delivery in one tap, even less than six days before delivery
    • Being able to pause my subscription for a while
    • Being able to order fewer meals
    • An ongoing 20% reduction in the subscription price
    • Both more flexible subscription options and an ongoing 20% price reduction
    • Something else
    • None of these would persuade me to restart
  13. Q12

    Is there anything else about why you cancelled, or what might bring you back to Fernway, that you would like us to know?

    Open text
Sample plan

Email every customer in the database who cancelled in the past six months, and invite them to complete the online survey. Aim for about 600 completes, with a target of 200 completes in each tenure group (under 3 months, 3–12 months inclusive, and over 12 months); monitor responses by tenure and use reminders to help balance the groups. Use database tenure where available to assign groups, and compare it with Q1. The requested precision cannot be met with 600 completes: at a 95% confidence level and a worst-case proportion of 50%, 200 completes per group gives approximately ±6.9 percentage points per group. Approximately 385 completes per group, or 1,155 total, are needed for ±5 percentage points before any finite-population correction. If ±5-point precision for each group is required, increase the completes target accordingly; otherwise report the achieved precision and do not present the 600-complete results as meeting that requirement. Weight overall results to the tenure distribution of all eligible cancellations if the response sample differs from it, using available database variables to assess and adjust for non-response where appropriate.