Benchmarks / SurveyBench / Why Fernway customers cancel / GPT-6.2 Astra
GPT-6.2 Astra: Why Fernway customers cancel
The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 98 of 100
- Analysis rating
- 1,467
- Head to head, this task
- won 5 of 11
Why it is not sound
The analysis
- Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents gave price as their main reason for cancelling? | ✓ | 21.6% | 21.6% |
| Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason? | ✓ | 26.3% | 26.3% |
| Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap? | ✓ | 58.6% | 58.6% |
| How many of the open-text answers mention recipe variety or repetitive recipes? | ✓ | 40 | 40 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason? | ✓ | 3.2 points | 3.19 points |
| Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason? | ✓ | 3,193 | 3,193 |
| How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes? | ✓ | 25 | 25 |
| Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut? | ✓ | 35.7 points | 35.7 points |
| What percentage of the customers invited completed the survey? | ✓ | 6.5% | 6.5% |
| Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less? | ✓ | 49.3% | 49.3% |
Traps in the data 7 of 8 handled
- ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
- ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
- ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
- ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
- ✓The chief executive believes customers leave mainly because of price.
- ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
- ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
- ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.
The analysis
Prioritise a low-cost live test of easy skipping over a blanket 20% price cut. Do not yet commit four months of product work. Test an actual skip against an actual discount and current practice: skipping can preserve a subscription while reducing paid boxes, and a blanket cut sacrifices margin even on boxes people would have bought at full price. Keep a targeted price intervention in contention, particularly for early-tenure leavers. Audit Northern Ireland deliveries in parallel rather than waiting for the product decision.
Prefer the price cut if a controlled test at the population tenure mix shows that incremental paid orders and sustained retention more than fund the discount and deliver greater net contribution than skipping after build and operating costs. If price wins only in a segment, target it there. Rescope or stop the skip build if customers can use the pilot but do not return to paid orders, or if later deadlines, multi-week pauses or fewer meals per box account for the benefit rather than one-tap skipping.
Findings
-
Chief executive: price matters, particularly early in the subscription, but the survey does not confirm it is the leading reason people leave.
138 of 640 respondents named price, versus 227 who named difficulty skipping. Applying the canceller population's tenure weights (45%, 35%, 20%) puts price at 26.3% and skipping at 32.4%, narrowing their gap from 13.9 to 6.1 percentage points. The latter share represents about 3,193 of 9,850 cancellers, not 3,193 recoverable customers. Among the 138 customers who left before three months, price led skipping, 31.2% to 26.1%.
-
Test easier skipping first. It has the stronger win-back signal overall, but the two offers appeal to different people.
Of 640 respondents, 36.4% said they were likely to return with one-tap skipping versus 25.8% with a 20% cut. Among the 227 skip-related leavers the comparison was 58.6% versus 22.9%, a 35.7-point advantage. Among the 138 price leavers it reversed: 49.3% for the cut versus 24.6% for skipping. These are separate ratings of hypothetical offers, not purchases or a forced choice.
-
Head of product: do not treat the Family plan as the cause of lower satisfaction. Its apparent disadvantage reverses when customers of similar tenure are compared.
Family satisfaction was 49.2% (301 respondents) versus 53.4% for Couples (339); the 4.2-point deficit is not significant by a two-sided 95% difference-in-proportions test. Over-12-month customers, the least satisfied tenure group, made up 202/301 Family respondents but only 105/339 Couples respondents. Family versus Couples satisfaction was 69.2% versus 66.1% under three months, 61.6% versus 57.4% at 3–12 months, and 42.1% versus 35.2% over 12 months. This explains the reversal without establishing that either plan is better.
-
Operations director: Northern Ireland warrants an immediate delivery audit, but the survey cannot establish the regional failure rate.
8 of 12 Northern Ireland respondents named delivery as their main cancellation reason (66.7%), versus 50 of 628 elsewhere (8.0%). R445 describes late or incomplete boxes. One Northern Ireland response moves the regional estimate by 8.3 points; the denominator is responding cancellers, not deliveries.
-
Do not assume that making the existing skip button faster solves the whole problem. Some customers want fewer dinners in a delivered box, not a week with no box at all.
Among 190 written answers, R376 describes repeated confirmations, R220 wants several weeks off at once, and R245 says the skip window was already closed. R337 instead wants two meals some weeks and four others; R098 wants a two-meal minimum. Meals per box was not a named Q1 option and is distinct from portion size per diner.
-
Investigate menu repetition alongside skipping for established customers: making a week easier to skip does not make that week's menu more attractive.
Repetition was the main reason for 25.4% of the 307 over-12-month respondents, versus 6.5% of the 138 under three months. Of 190 written answers, 40 mentioned variety or repetition; 25 of those 40 came from over-12-month customers. R038 explicitly describes skipping because nothing new was on offer.
Caveats
- 640 of 9,850 invitees completed (6.5%); 640 of 702 starters completing is not the invitation completion rate. Long-tenure cancellers were 48.0% of completes versus 20% of the population; under-three-month cancellers were 21.6% versus 45%. Family was also overrepresented (47.0% versus 34%).
- The weighted figures apply population tenure shares to the three within-tenure percentages. They assume respondents resemble nonrespondents within tenure; they do not separately correct plan mix or unmeasured nonresponse. The overall offer ratings are unweighted; offer-by-tenure results are not supplied.
- The ±3.2-point price margin is the nominal 95% simple-random-sample approximation, 1.96 × sqrt[(138/640)(1−138/640)/640], expressed in percentage points. It does not cover self-selection or recall bias. The plan test likewise describes sampling uncertainty, not bias; non-significance is not proof of equality.
- Only recent UK cancellers were surveyed. There are no active-customer denominators for plan churn, regional delivery failure or retention. Q1 forces one main reason even when several mattered. Q3 and Q4 were answered by the same people; the tables do not show individual preferences or overlap.
- Only 190 of 640 wrote comments. Theme counts describe those answers, can overlap, and are not estimates of demand across all cancellers. Neither stated return intent nor a request for a feature establishes sustained paid orders or profitability.
Next steps
- decisionApprove a bounded pilot, not a full build or a national price change. Agree the investment hurdle with finance before launch.
- experimentRandomise eligible cancellers to a working skip-link offer, a real 20% discount or usual win-back practice. Run a parallel retention test for active customers; stratify by tenure and plan.
- researchObserve customers trying to skip an upcoming box, take a month off and reduce a week's meals. Check failed attempts against app events, reminders and packing cut-offs.
- researchUse respondent-level data to adjust offer ratings for tenure and plan against the customer database; follow up a sample of nonrespondents, especially short-tenure leavers.
- monitorAudit Northern Ireland courier scans, promised versus actual dates, missing-item contacts and refunds against comparable routes and weeks elsewhere.
Open-text themes it coded
Recipe variety or repetition 40Time off or less frequent deliveries: easy skips, pauses, workable deadlines or an alternative to weekly boxes 49Fewer or adjustable meals per box 28Price, affordability, price stability, discounts or value for money 43Larger or adjustable portions per diner 12Delivery or fulfilment reliability: timing, missing items, non-arrival or condition on arrival 18Packaging: waste, recyclability or packaging that leaks or splits 19Changed circumstances; respondent says nothing would have retained them 6
The survey it planned
1 screening question and 9 questions, as the model wrote them.
-
S1
In the past six months, have you cancelled a Fernway subscription? Please do not count skipping a delivery as cancelling.
One answer- Yes
- No
- Not sure
Continues if Yes
-
Q1
Think about the most recent Fernway subscription you cancelled. In your own words, what made you decide to cancel? A few words are fine.
Open text -
Q2
Which of the following, if any, played a part in your decision to cancel that subscription? Select all that apply.
Any that apply- The boxes cost more than I wanted to spend
- I had to decide too far ahead whether to skip a week
- The steps for skipping a delivery were inconvenient
- I wanted to pause deliveries for a while
- I wanted fewer meals in each box
- The recipe choices did not suit me
- I was unhappy with the quality of the ingredients
- Preparing the meals took too much time
- I had problems with deliveries
- My household circumstances changed
- I preferred another way of getting meals
- Another reason
- I don’t remember
-
Q3
Overall, how satisfied or dissatisfied were you with Fernway while you had that subscription?
One answer- Very dissatisfied
- Fairly dissatisfied
- Neither satisfied nor dissatisfied
- Fairly satisfied
- Very satisfied
- I don’t remember
-
Q4
Do you currently have a Fernway subscription?
One answer- Yes
- No
- Not sure
-
Q5
If Fernway offered the same plan at the same price and with the same delivery arrangements as when you cancelled, how likely would you be to subscribe again in the next three months?
One answer- Definitely would not
- Probably would not
- Might or might not
- Probably would
- Definitely would
- Not sure
Routing Shown only if Q4 is: No
-
Q6
Suppose just one thing changed from when you cancelled: the price of every box on your plan was 20% lower. The arrangements for skipping deliveries stayed the same. How likely would you be to subscribe again in the next three months?
One answer- Definitely would not
- Probably would not
- Might or might not
- Probably would
- Definitely would
- Not sure
Routing Shown only if Q4 is: No
-
Q7
Suppose just one thing changed from when you cancelled: you could skip any week with one tap in the app, without having to decide six days ahead. The price of your plan stayed the same. How likely would you be to subscribe again in the next three months?
One answer- Definitely would not
- Probably would not
- Might or might not
- Probably would
- Definitely would
- Not sure
Routing Shown only if Q4 is: No
-
Q8
Thinking back to when you cancelled, which ONE change, if any, would have been most likely to make you keep your subscription? Assume everything else stayed the same.
One answer- A 20% lower price for every box on my plan
- Skipping any week with one tap in the app, without having to decide six days ahead
- Pausing deliveries until I was ready to restart
- Choosing fewer meals per box and paying only for the meals I ordered
- Another change
- None of these changes would have made me stay
- Not sure
-
Q9
What other change would have been most likely to make you stay?
Open textRouting Shown only if Q8 is: Another change
Sample plan
Email a unique survey link to every customer in the UK cancellation database whose most recent cancellation was in the six months before fieldwork; deduplicate by customer and include those who subsequently returned. Link plan at cancellation, dates of that subscription spell and time since cancellation via the survey token. Classify tenure at cancellation as under 3 months, 3 through 12 months inclusive, or over 12 months. Target about 600 eligible completes, aiming for at least 200 per tenure group through targeted reminders to nonrespondents; do not discard extra completes. Pilot on mobile and desktop within the invited frame and confirm completion within six minutes before full rollout. Randomise substantive options in Q2 and Q8, keeping residual answers last and ‘I don’t remember’ exclusive in Q2; counterbalance Q6 and Q7 after Q5. The precision demand is not generally achievable with 600: at 50%, 200 independent random completes per group give about ±6.9 percentage points at 95% confidence. Without a finite-population correction, ±5 points requires 385 per group (1,155 total). Check frame counts before launch: for a group of size N, the simple-random-sample benchmark with finite-population correction is ceiling(384.16 × N / (N + 383.16)). Agree a larger target or wider intervals if needed. These are sampling benchmarks, not guarantees: inviting everyone does not eliminate voluntary-response bias.