Benchmarks / SurveyBench / Why Fernway customers cancel / GPT-6.2 Sol

Measured by Spring Prompt

GPT-6.2 Sol: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✓ Sound research
Research score
100 of 100
Analysis rating
1,354
Head to head, this task
won 9 of 13

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 8 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✓The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Prioritise easier skipping over a blanket 20% price cut, but do not commit four months of engineering on survey intent alone. First test a lightweight, working skip flow that makes the deadline clear and confirms that no box will ship. Check whether customers also need multi-week pauses or later cut-offs. In parallel, test a targeted price offer for price-led leavers, audit Northern Ireland deliveries, and avoid a Family-plan overhaul on the basis of the raw satisfaction gap.

What would change it

Choose the blanket price cut only if a controlled test shows it delivers more incremental contribution than easier skipping after accounting for margin lost on customers who would have paid full price, changes in paid-box volume, and the build and operating costs of skipping. Re-scope the feature if easier single-week skipping does not improve paid reactivation or retention and testing shows the real blockers are multi-week pauses, cut-offs or minimum meals per box.

Findings

  1. CEO: Price matters, especially early in the subscription, but the survey does not confirm it as the leading reason customers leave.

    Medium confidenceBase 640

    Price was the main reason for 21.6% of 640 respondents, versus 35.5% for hard-to-skip weeks. Weighting to the tenure mix of all cancellers narrows the gap but does not reverse it: 26.3% for price versus 32.4% for skipping, equivalent to about 3,193 of the 9,850 cancellers for skipping. Price led among respondents who left within three months (31.2% price versus 26.1% skipping; base 138).

  2. One-tap skipping is the stronger win-back proposition in this survey. A price offer is worth testing with price-led leavers, not rolling out to everyone.

    Medium confidenceBase 640

    Overall, 36.4% said they were likely to return with one-tap skipping versus 25.8% with a 20% price cut, a 10.6-point gap. Among the 227 skip-led leavers, the gap was 35.7 points (58.6% versus 22.9%). Among the 138 price-led leavers, the price cut led instead (49.3% versus 24.6%). These are stated intentions, not expected conversion rates.

  3. Head of product: The raw satisfaction gap is not a sound reason to treat Family as the problem. Investigate menu repetition separately before making a Family-specific change.

    Medium confidenceBase 640

    Family satisfaction was 49.2% (301 respondents) versus 53.4% for Couples (339). The 4.2-point raw gap was not statistically significant in a two-sided proportions test (p≈0.29). Family respondents were much more likely to have stayed over 12 months (67.1% versus 31.0%), the least satisfied tenure group (39.7% satisfied). Family satisfaction was descriptively higher in every tenure band. Menu repetition was cited as the main reason by 23.6% of Family versus 13.0% of Couples respondents, but it was also more common at longer tenure, so the raw plan gap does not isolate a Family-specific issue.

  4. Operations director: Northern Ireland has a delivery warning worth investigating now, but the survey cannot establish the scale or cause of the problem.

    Low confidenceBase 12

    Eight of 12 Northern Ireland respondents (66.7%) cited delivery as their main reason, compared with 50 of 628 elsewhere (8.0%). R445 mentioned delivery on the promised day and missing contents. The Northern Ireland figure is a share of responding cancellers, not a delivery failure rate.

  5. Design for control over deliveries, not just fewer taps. Test which problems a single-week skip actually solves.

    Medium confidenceBase 190

    Comments describe distinct barriers: too many confirmation screens (R376), limits on consecutive skips (R217), skipping several weeks one at a time (R220), early cut-offs (R295), and needing fewer meals rather than no box (R361). Separately, 40 of the 190 open answers mention recipe variety or repetition, including 25 from respondents with over 12 months' tenure. These comments identify needs to test; they do not estimate their prevalence among all cancellers.

Caveats

  • Only 640 of 9,850 invitees completed the survey (6.5%). Long-tenure customers account for 48.0% of respondents versus 20% of all cancellers, and Family accounts for 47.0% versus 34%. Tenure weighting corrects the tenure mix only; it cannot remove selective response within tenure bands or independently correct for plan and region.
  • The 3.2-point margin of error for price is a nominal 95% sampling margin under simple random sampling. It does not account for nonresponse or self-selection. The satisfaction significance test is likewise a nominal, unadjusted comparison, and a non-significant result does not establish equivalence.
  • Only recent cancellers were surveyed. These results do not measure churn rates by plan or region, the effect on retention among active customers, or the percentage of all deliveries that fail.
  • Q1 forced a single main reason, while the comments show that reasons can overlap. Q3 always came before Q4, so order effects cannot be ruled out. Neither question measures actual reactivation, paid boxes, or margin.
  • The open-text counts are unweighted and describe the people who answered Q5. Themes overlap, and a mention of a need does not establish that a specific feature would have prevented cancellation.

Next steps

  • researchPrototype one-tap skipping with active customers and recent skip-led leavers. Test a single-week skip, consecutive weeks away and a late change of plans, using real fulfilment cut-offs.
  • experimentRandomise eligible leavers to a working skip pilot at the usual price, a 20% offer, or a business-as-usual holdout. Pre-specify skip-led and price-led subgroup comparisons. Separately test skip access against business as usual with active subscribers.
  • decisionSet the investment threshold before the tests and approve the full build or a pricing change only once the observed effects clear it.
  • monitorAudit Northern Ireland delivery records by courier, route and postcode, and compare with comparable routes elsewhere in the UK. Check packing records separately for missing items.
  • researchCompare churn and menu-related complaints by tenure within each plan, and seek follow-up feedback from early leavers and other underrepresented groups using a different contact method.

Open-text themes it coded

Recipe variety or repetitive menus 40Easier skipping, temporary pauses or less frequent deliveries 49Affordability, price stability, discounts or value for money 43Fewer or adjustable meals per box, or a plan for a smaller household 29Larger or adjustable serving sizes 12Less, more recyclable or better-performing packaging 19Delivery reliability, coverage or order completeness and condition 19Life or household change rather than a requested product improvement 6

The survey it planned

1 screening question and 11 questions, as the model wrote them.

  1. S1

    In the past 6 months, have you cancelled a Fernway meal-kit subscription? Please answer yes even if you have since subscribed again. Skipping a delivery without ending your subscription does not count.

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. Q1

    How long had you been on the Fernway subscription you most recently cancelled? Include any weeks you skipped or paused, but do not include earlier subscriptions.

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months
    • Not sure
  3. Q2

    Overall, how satisfied or dissatisfied were you with Fernway while you were on the subscription you most recently cancelled?

    One answer
    • Very dissatisfied
    • Somewhat dissatisfied
    • Neither satisfied nor dissatisfied
    • Somewhat satisfied
    • Very satisfied
    • I don't remember
  4. Q3

    Which of the following, if any, contributed to your decision to cancel that subscription? Select all that apply.

    Any that apply
    • The regular price was more than I wanted to pay
    • A discount or introductory offer ended
    • I had to decide too far in advance whether to skip a delivery
    • Skipping a delivery in the app took too much effort
    • I wanted to pause deliveries for several weeks
    • I wanted fewer meals in each box
    • I didn't like the choice of recipes
    • I was unhappy with the ingredients or the meals
    • I had problems with deliveries
    • I had problems with customer service
    • My household, schedule or cooking habits changed
    • I preferred another meal kit or another way of getting meals
    • I only meant to try Fernway for a short time
    • Another reason
    • I don't remember
  5. Q4

    Of the reasons you selected, which was the main reason you cancelled?

    One answer
    • The regular price was more than I wanted to pay
    • A discount or introductory offer ended
    • I had to decide too far in advance whether to skip a delivery
    • Skipping a delivery in the app took too much effort
    • I wanted to pause deliveries for several weeks
    • I wanted fewer meals in each box
    • I didn't like the choice of recipes
    • I was unhappy with the ingredients or the meals
    • I had problems with deliveries
    • I had problems with customer service
    • My household, schedule or cooking habits changed
    • I preferred another meal kit or another way of getting meals
    • I only meant to try Fernway for a short time
    • Another reason
    • I don't remember
    • No single reason was the main one
  6. Q5

    Do you currently have a Fernway subscription? Please include a subscription that is paused or has a delivery you have skipped.

    One answer
    • Yes
    • No
    • Not sure
  7. Q6

    Imagine Fernway offered the same plan you last had, at its regular price before any discounts. To skip a delivery, you would still need to make the change in the app at least 6 days before delivery. How likely would you be to subscribe again in the next 3 months?

    One answer
    • Definitely would not
    • Probably would not
    • Might or might not
    • Probably would
    • Definitely would

    Routing Shown only if Q5 is: No

  8. Q7

    Imagine Fernway offered the same plan you last had, at its regular price before any discounts, but you could skip any week with one tap in the app without having to decide at least 6 days before delivery. Everything else would stay the same. How likely would you be to subscribe again in the next 3 months?

    One answer
    • Definitely would not
    • Probably would not
    • Might or might not
    • Probably would
    • Definitely would

    Routing Shown only if Q5 is: No

  9. Q8

    Imagine Fernway offered the same plan you last had, but its regular price was 20% lower. To skip a delivery, you would still need to make the change in the app at least 6 days before delivery. Everything else would stay the same. How likely would you be to subscribe again in the next 3 months?

    One answer
    • Definitely would not
    • Probably would not
    • Might or might not
    • Probably would
    • Definitely would

    Routing Shown only if Q5 is: No

  10. Q9

    If Fernway could offer only one of these changes to the plan you last had, which would make you more likely to subscribe again?

    One answer
    • One-tap skipping without the 6-day deadline, at the regular price
    • A 20% lower regular price, with the 6-day deadline still in place
    • Both changes would make me equally more likely to subscribe again
    • Neither change would make me more likely to subscribe again
    • Not sure

    Routing Shown only if Q5 is: No

  11. Q10

    Thinking about your household now, which one of these changes, if any, would make the biggest difference to whether you subscribe again? Assume the rest of the plan stays the same.

    One answer
    • Paying a 20% lower regular price
    • Skipping any week with one tap, without the 6-day deadline
    • Pausing deliveries for several weeks at a time
    • Choosing fewer meals per delivery and paying less per box, with no change to the price per meal
    • None of these would make me more likely to subscribe again
    • Not sure

    Routing Shown only if Q5 is: No

  12. Q11

    Is there anything about why you cancelled, or what might bring you back, that we haven't asked about? Please do not include personal details. This question is optional.

    Open text
Sample plan

Email every unique customer in the UK database who cancelled within six months of the survey launch. Use one link per customer and the most recent qualifying cancellation if there were several. Do not exclude customers who have since rejoined. Aim for about 600 eligible completes, with soft monitoring targets of about 200 per tenure group. Do not cap groups or reject eligible respondents to fill quotas. Prioritise reminders to nonrespondents in groups falling behind. Where records are reliable, calculate tenure from the start and cancellation dates of the relevant subscription; use Q1 as a fallback and retain unknown tenure separately. Use database plan and cancellation date for comparison and nonresponse checks. At 95% confidence, 200 completes per group give a worst-case margin of about ±6.9 percentage points under simple random sampling from a large population. The requested ±5 points would require about 385 per group, or 1,155 total, under those assumptions. Check whether known group population sizes materially reduce the requirement through a finite-population correction. Otherwise agree before launch whether to extend fieldwork to pursue the larger target or report lower precision. Inviting everyone does not eliminate nonresponse bias. Compare response rates and respondent composition with the invitation list by tenure, plan and cancellation recency; calibrate to known totals where justified and report achieved bases, effective sample sizes and intervals with their assumptions. Pilot on mobile to verify a completion time of at most 6 minutes. Randomise the substantive options in Q3, Q9 and Q10, keeping write-in, tie, none and uncertainty options at the end as applicable. In Q3, make 'I don't remember' exclusive. In Q4, display only the substantive reasons selected in Q3 plus 'No single reason was the main one'; skip Q4 if only one reason was selected and carry it forward as the main reason, or record the main reason as unknown if Q3 is 'I don't remember'. Keep Q6 first, then randomly assign the order of Q7 and Q8 before showing Q9.