Gemini 3.1 Pro Preview: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict: ✗ Not sound
Research score: 79 of 100
Analysis rating: 810
Head to head, this task: won 7 of 10

Why it is not sound

The survey

A double-barrelled question: Q2
A double-barrelled question: Q4

The analysis

Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Scorecard

Numbers 11 of 11 right

Question		Its answer	True
What percentage of respondents gave price as their main reason for cancelling?	✓	21.6%	21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?	✓	26.3%	26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?	✓	58.6%	58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?	✓	40	40
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?	✓	3.2 points	3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?	✓	3,193	3,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?	✓	25	25
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?	✓	35.7 points	35.7 points
What percentage of the customers invited completed the survey?	✓	6.5%	6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?	✓	49.3%	49.3%

Traps in the data 6 of 8 handled

✓Northern Ireland respondents (base 12) cite delivery problems far more often.
✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
✓'Likely to come back' answers are stated intentions about a hypothetical feature.
✓The chief executive believes customers leave mainly because of price.
✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build the one-tap skipping feature. The survey disproves the assumption that price is the primary driver of churn; skipping difficulty is the most significant problem (32.4% weighted) and holds higher win-back intent (58.6%) than a price cut. Four months of product work will preserve long-term profit margins, whereas a permanent 20% price cut would broadly cannibalize LTV.

What would change it

If LTV and retention modeling of active subscribers demonstrates that a 20% price cut would yield enough long-term revenue growth to heavily offset the margin loss, or if operational forecasts indicate that one-tap skipping will cause insurmountable supply-chain food waste.

Findings

Price is not the main reason customers leave; difficulty skipping weeks is the primary driver of churn.

High confidenceBase 640

35.5% (unweighted) and 32.4% (weighted to population) cite hard-to-skip weeks, compared to 21.6% (unweighted) and 26.3% (weighted) for price.
One-tap skipping is a stronger win-back driver than a 20% price cut for the customers most sensitive to those respective issues.

High confidenceBase 365

58.6% of skip-cancellers are likely to return for one-tap skipping, whereas only 49.3% of price-cancellers are likely to return for a 20% price cut.
The Family plan is not underperforming; its overall lower satisfaction is a statistical illusion (Simpson's Paradox) caused by higher retention.

High confidenceBase 640

Within every tenure band, Family plan customers are more satisfied than Couples plan customers (e.g., 61.6% vs 57.4% for 3-12 months), but 67% of Family customers are >12 months tenured, where satisfaction naturally dips.
There is a strong signal for severe delivery issues in Northern Ireland, but the survey sample is too small to be conclusive.

Low confidenceBase 12

66.7% of Northern Ireland respondents cited delivery problems as their main reason for leaving, compared to 9.1% overall, but this represents only 8 out of 12 people.
Long-term customers suffer heavily from recipe fatigue, driving churn for our most loyal cohort.

High confidenceBase 307

25.4% of >12-month tenure customers cancel because recipes get repetitive, and 25 open-text answers from this segment specifically ask for broader menu rotation.

Caveats

The survey captures cancelled customers only; we lack data on how a 20% price cut would affect LTV and retention among current active subscribers.
A 6.5% completion rate introduces potential self-selection bias, meaning those with strong negative sentiments may be overrepresented.
Stated intent (likelihood to return) generally overestimates actual future conversion rates.
The Northern Ireland sample size (12 respondents) is statistically insufficient to make operational changes without triangulating with internal data.

Next steps

decisionBuild one-tap skipping in the app
monitorInvestigate Northern Ireland courier performance data
experimentInvestigate offering a 2-meal per week option
researchConduct research into long-term customer menu fatigue

Open-text themes it coded

Difficulty skipping weeks or pausing for holidays 45Repetitive recipes and lack of menu variety 40Too many meals per week / inflexibility of minimums 31Price, expense, and lack of perceived value 41Excessive packaging waste and non-recyclable materials 19Portion sizes are too small 12Delivery issues and missing ingredients 20Life circumstances changed / Not a product fault 6

The survey it planned

2 screening questions and 11 questions, as the model wrote them.

S1
Have you cancelled a Fernway subscription in the past 6 months?
One answer
- Yes
- No
Continues if Yes
S2
How long were you a Fernway subscriber before cancelling?
One answer
- Under 3 months
- 3 to 12 months
- Over 12 months
- I have never been a subscriber
Continues if Under 3 months; 3 to 12 months; Over 12 months
Q1
Overall, how satisfied were you with your Fernway subscription before you cancelled?
Scale

Scale 1-5: Very dissatisfied to Very satisfied
Q2
What was the PRIMARY reason you cancelled your Fernway subscription?
One answer
- It was too expensive
- It was too difficult to skip weeks or pause deliveries
- I wanted fewer meals than the plan allowed
- Recipe variety or quality declined
- My household or personal circumstances changed
- Other
Judges: double-barrelled
The options combine difficulty skipping with difficulty pausing, and declining recipe variety with declining quality. A selected answer cannot distinguish these separate cancellation reasons.
Several options group distinct reasons together (e.g., 'skip weeks or pause deliveries', 'variety or quality'), meaning respondents cannot specify which of the two they meant.
Q3
When thinking about a meal-kit subscription, please rank the following features in order of importance to you (1 being most important, 4 being least).
Ranking
- A low price (strong value for money)
- One-tap flexibility to skip any delivery week
- Flexibility to easily pause the entire subscription
- Flexibility to change the number of meals per week
Q4
How strongly do you agree or disagree: 'I would have kept my Fernway subscription if I could have skipped weeks with one tap, without needing 6 days notice.'
Scale

Scale 1-5: Strongly disagree to Strongly agree

Judges: double-barrelled
Bundles two separate changes into one statement: one-tap skipping and removing the 6-day notice requirement. A respondent who wants only one of them cannot answer accurately.
The statement asks about two distinct feature changes: skipping with 'one tap' (ease) and 'without needing 6 days notice' (timeframe).
Q5
How strongly do you agree or disagree: 'I would have kept my Fernway subscription if the price had been 20% lower.'
Scale

Scale 1-5: Strongly disagree to Strongly agree
Q6
If you had to choose between the following two Fernway improvements, which would you prefer?
One answer
- The current price, but with a new feature to instantly skip any week with one tap
- A 20% cheaper price, but requires at least 6 days notice to skip a week
Q7
To ensure our survey is functioning correctly, please select '2' for this question.
Scale

Scale 1-5: 1 to 5
Q8
How likely are you to subscribe to Fernway again in the future?
Scale

Scale 1-5: Very unlikely to Very likely
Q9
Which of the following would be MOST likely to convince you to restart your subscription?
One answer
- A 20% price reduction
- A one-tap option to instantly skip weeks
- More flexibility to change meal quantities
- Different or improved recipes
- None of these - I will not return
Q10
Which Fernway plan were you subscribed to?
One answer
- Couples plan (for 2 people)
- Family plan (for 4 people)
Q11
What, if anything, could Fernway have done differently to keep you as a customer?
Open text

Sample plan

Email invitation to all customers in the database who cancelled in the last 6 months. NOTE on constraints: To achieve the board's strict precision requirement of +/- 5 percentage points at a 95% confidence level for EACH of the three tenure groups independently, we need approx 384 completes per group, totaling 1,152 completes. If the 'about 600 completes' constraint is a strict budget/time cap, we will enforce equal quotas of 200 per tenure group (<3 months, 3-12 months, >12 months) to total 600, yielding a +/- 6.9 percentage points precision per group instead.