Benchmarks / SurveyBench / Why Fernway customers cancel / Qwen4 Max
Qwen4 Max: Why Fernway customers cancel
The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 92 of 100
- Analysis rating
- 1,373
- Head to head, this task
- won 11 of 16
Why it is not sound
The survey
- A double-barrelled question: Q9
The analysis
- Reported a material finding the data does not support
Findings the judges found unsupported
The CEO's hypothesis is not confirmed: inflexibility about weeks, not price, is the largest single stated cause of cancellation — and it stays largest after correcting the sample. Build the flexibility lever, not a blanket price cut.
The quoted ±3.2 percentage-point margin of error for the tenure-weighted price share is too small. Even under an illustrative random-sampling calculation, accounting for the tenure weights gives approximately ±4.2 points before finite-population correction. Such margins also do not capture non-response bias.Price is an entry/early-tenure problem, and the open text says it is mostly price *rises and expiring intro offers*, not the headline level. A permanent 20% cut for everyone is the wrong instrument; fixing the intro-price cliff, repeat increases and new-customer-only discounts is the targeted version.
The claim that price complaints are mostly about increases and expiring introductory offers rather than the headline price contradicts the evidence cited: only 10 of 42 price/value answers concern increases, introductory offers or loyalty unfairness. Many explicitly concern affordability or the price level. These mechanisms warrant testing, not treating as the established dominant explanation.'One-tap skip' under-specifies the requirement. The people who left over weeks wanted multi-week pauses, a later cut-off, reminders and a pause that keeps their account — several had tried to skip and been blocked. Ship the cheap fixes (cut-off, confirmations, bulk skip, skip link in reminders) before or alongside the four-month build.
The answers support investigating deadlines, skip caps, confirmation screens and pause functionality, but they do not establish that changing these would be cheap or quick. No engineering estimates or operational constraints support shipping them ahead of the four-month build as inexpensive fixes.There is a third lever the decision ignores: a smaller / adjustable meal count. A substantial minority do not want to switch off a week, they want less of it — and some would pay more per portion. This is a cheaper, revenue-positive alternative to a 20% cut and should be tested in parallel.
The data support testing smaller or adjustable meal counts, but not calling this a cheaper, revenue-positive alternative. There are no cost, margin or demand measurements. R134 offers to pay more per box for larger servings, not more per portion for a smaller meal-count plan. The assertion that the single-reason question necessarily pushed meal-count complaints into other categories is also unverified.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents gave price as their main reason for cancelling? | ✓ | 21.6% | 21.6% |
| Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason? | ✓ | 26.3% | 26.3% |
| Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap? | ✓ | 58.6% | 58.6% |
| How many of the open-text answers mention recipe variety or repetitive recipes? | ✓ | 40 | 40 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason? | ✓ | 3.2 points | 3.19 points |
| Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason? | ✓ | 3,193 | 3,193 |
| How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes? | ✓ | 25 | 25 |
| Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut? | ✓ | 35.7 points | 35.7 points |
| What percentage of the customers invited completed the survey? | ✓ | 6.5% | 6.5% |
| Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less? | ✓ | 49.3% | 49.3% |
Traps in the data 8 of 8 handled
- ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
- ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
- ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
- ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
- ✓The chief executive believes customers leave mainly because of price.
- ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
- ✓Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
- ✓The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.
The analysis
Do not cut prices 20% across the plans. Build flexibility — but scope it correctly, and stage it. The evidence, once corrected for the sample's heavy long-tenure skew, puts 'hard to skip weeks' ahead of price (32.4% vs 26.3% weighted; ~3,193 vs ~2,590 of the 9,850 cancellers), and the people who left over skipping are far more responsive to a skip fix than to a discount (58.6% vs 22.9%). Price is real but concentrated in the first year and, on the open text, driven by price *rises* and expiring intro offers rather than the sticker price — a blanket cut would give away margin to every retained customer to solve a problem most of them do not have. Concretely: (1) ship the cheap flexibility fixes now — later skip cut-off, remove confirmation/discount interstitials, allow multi-week and bulk skips, raise or remove the consecutive-skip cap, a skip link in the reminder email/SMS, and pause-without-losing-settings; several respondents cancelled only because these blocked them, and this is weeks of work, not four months. (2) Then build the full one-tap in-app skip/pause, with the multi-week and deadline behaviour in the spec from day one. (3) Replace the blanket cut with targeted pricing: end the intro-offer cliff and repeated in-life increases, add a loyalty/existing-customer offer, and use time-limited discounts in win-back only. (4) Pilot a two-meal / adjustable meal-count option — 16% of open answers want less food, not no food, and some would pay more per portion; this is the unaddressed third lever and is revenue-accretive rather than dilutive. (5) Start a menu-rotation workstream for long-tenure customers (satisfaction 39.7% after 12 months; 25 of 40 variety complaints come from that group) — no skip button fixes boredom. Answer the head of product: leave the Family plan alone; its apparent deficit is a tenure composition effect and is not statistically significant. Answer the operations director: the NI finding rests on 8 people; investigate with courier and packing records, not this survey. Before committing the full four months, validate with a randomised win-back test and product analytics, because a 6.5%-response survey of hypothetical intent is a strong steer, not a business case.
Four things would flip or reshape this: (1) A randomised win-back trial in which a 20% (or targeted) discount reactivates materially more cancellers than a flexibility offer, and holds them at 12 weeks at positive contribution margin — stated intent is not behaviour, and price leavers' 49.3% could convert better than skip leavers' 58.6%. (2) Product analytics showing that most skip-related cancellers never actually attempted a skip, or that the binding constraint is the skip cap/cut-off alone — then the four-month build is unnecessary and a configuration change captures most of the value; conversely, if active customers already skip heavily, the revenue lost to skipped weeks could exceed the retention gain. (3) A non-response check showing the 93.5% who did not reply are disproportionately short-tenure price leavers: price's weighted share would rise above skipping's and the ranking reverses. (4) Finance modelling showing the flexibility build cannot pay back at realistic skip rates, or that margin cannot absorb a smaller/2-meal plan — which would push us toward targeted discounting instead. Evidence that Family dissatisfaction is real (a larger, tenure-controlled sample or behavioural data showing higher Family cancel rates at equal tenure) would also reopen the plan question.
Findings
-
The CEO's hypothesis is not confirmed: inflexibility about weeks, not price, is the largest single stated cause of cancellation — and it stays largest after correcting the sample. Build the flexibility lever, not a blanket price cut.
Unweighted Q1 (n=640): hard to skip 35.5% vs price 21.6%. Re-weighted to the population tenure mix (45% <3m / 35% 3-12m / 20% >12m): skip 32.4% vs price 26.3% — gap narrows to 6.1pp but does not reverse. Applied to 9,850 cancellers: ~3,193 skip vs ~2,590 price. Price share MoE ±3.2pp; the weighted gap is directionally robust but not a precise estimate.
-
The raw tables are structurally biased and must never be quoted unweighted to the board: long-tenure and Family customers are roughly 2.4x and 1.4x over-represented. This single fact explains both the 'price is low' and 'Family is bad' readings.
Sample vs population: >12 months 48.0% vs 20%; under 3 months 21.6% vs 45%; Family 47.0% vs 34%. Only 640 of 9,850 invited completed (6.5%); 702 started. Weighting here fixes tenure only — plan and region remain uncorrected.
-
Price is an entry/early-tenure problem, and the open text says it is mostly price *rises and expiring intro offers*, not the headline level. A permanent 20% cut for everyone is the wrong instrument; fixing the intro-price cliff, repeat increases and new-customer-only discounts is the targeted version.
Q1 price: 31.2% of <3m (n=138) and 27.2% of 3-12m (n=195) vs 13.7% of >12m (n=307). 42/190 open answers cite price/value; 10 of those specifically cite rises or loyalty unfairness (R124 £49→£67 in a year, R340 two rises in a year, R077, R296 intro-price jump, R158/R356/R500 new-customer-only offers, R204/R325 no loyalty reward).
-
Each lever mainly moves its own leavers, and skipping addresses the bigger pool. Roughly 1 in 3 cancellers say they would return for easy skipping vs 1 in 4 for a 20% cut — use this as a ranking, not a revenue forecast.
Skip-leavers (n=227): 58.6% likely to return with one-tap skip vs 22.9% with a 20% cut (gap 35.7pp). Price-leavers (n=138): 49.3% likely with the cut vs 24.6% with skip. Tenure-weighted overall stated intent: ~35.0% likely with skip vs ~27.1% with the cut (reason shares weighted, intent-within-reason assumed constant across tenure).
-
Neither lever alone wins back most cancellers, and over half of them left while satisfied. Churn is multi-causal and partly about life fit, so the business case must be built on retention of the current base plus a modest win-back rate — not on a mass return.
63.6% are not 'likely' to return even with one-tap skip; 74.2% not likely with a 20% cut (n=640). Q2: 51.4% of cancellers were satisfied (4-5) while customers. Weighted Q1 remainder: recipes 13.0%, delivery 11.2%, portions 7.6%, packaging 5.3%, other 4.2%. 6/190 open answers say nothing would have kept them (R096, R357, R429, R523, R621, R628).
-
Answer to the head of product: no, the Family plan is not the problem. The 4.2pp satisfaction gap is well inside sampling noise, and within every tenure band Family is actually slightly *more* satisfied than Couples. The real fault line is tenure, not plan — do not spend roadmap on a Family-plan redesign.
Couples 53.4% satisfied (n=339) vs Family 49.2% (n=301); difference 4.2pp with SE ~4.0pp (95% CI roughly -3.6 to +12.0pp), not significant. Within tenure: <3m 66.1% Couples (n=112) vs 69.2% Family (n=26); 3-12m 57.4% (n=122) vs 61.6% (n=73); >12m 35.2% (n=105) vs 42.1% (n=202). Family respondents are 67% >12m (202/301) vs 31% of Couples (105/339) — a textbook composition effect.
-
Long-tenure churn is menu fatigue, and it is the third thing worth fixing alongside flexibility. Satisfaction collapses with tenure while 'repetitive recipes' quadruples; a skip button does not fix boredom.
Q2 satisfied: 66.7% <3m (n=138), 59.0% 3-12m (n=195), 39.7% >12m (n=307). Q1 repetitive recipes: 6.5% / 14.4% / 25.4% respectively (weighted 13.0%). 40/190 open answers cite variety/repetition, 25 of them from >12m customers (e.g. R525 'menu looped back every month', R200, R419, R539, R608).
-
'One-tap skip' under-specifies the requirement. The people who left over weeks wanted multi-week pauses, a later cut-off, reminders and a pause that keeps their account — several had tried to skip and been blocked. Ship the cheap fixes (cut-off, confirmations, bulk skip, skip link in reminders) before or alongside the four-month build.
48/190 open answers ask for skip/pause/cadence flexibility; 31 describe execution friction or deadlines. Multi-week: R217 (skip cap forced cancellation), R220 (skip weeks one at a time), R493 (8-week pause), R372 (months off), R093. Cut-off/locked weeks: R295, R433, R245, R351, R603, R527. Hoops/app failures: R376 (repeated confirmations + discount interstitials), R530, R559 (app crash), R590 (mobile can't skip), R308/R089 (phone calls), R264/R393/R049 (cancel-and-rejoin loses settings).
-
There is a third lever the decision ignores: a smaller / adjustable meal count. A substantial minority do not want to switch off a week, they want less of it — and some would pay more per portion. This is a cheaper, revenue-positive alternative to a 20% cut and should be tested in parallel.
31/190 open answers (16%) ask for fewer or adjustable meals per week (R007, R011, R059, R098, R138, R165, R222, R270, R327, R328, R337, R361, R459, R502, R540, R619, R634 etc.); several link it to price per portion (R328 '£5 a portion for meals I wasn't eating', R436, R103) and R134 would pay more for larger servings. Q1 'portion sizes' is only 6.7% (n=640) because the forced single-reason question pushes these people into skip/price/other.
-
Answer to the operations director: the survey cannot tell you whether Northern Ireland has a delivery problem — 12 respondents is not evidence. The signal is strong enough to justify pulling the operational records, not strong enough to justify spend or a public claim.
NI: 66.7% cite delivery, but that is 8 of 12 respondents (95% MoE roughly ±27pp); NI price 0.0% and portions 0.0% are the same 12 people. England 8.1% delivery (n=546), Scotland 13.0% (n=46), Wales 0.0% (n=36). Delivery is 11.2% of reasons tenure-weighted, and open text splits into two distinct ops issues: late/missing boxes (15 answers, e.g. R170, R311, R630) and missing ingredients (5 answers: R068, R163, R359, R445, R609).
Caveats
- Response rate is 6.5% (640 completes from 9,850 invited). This is a self-selected sliver of cancellers; the tenure/plan skew is measured and corrected, but bias *within* tenure bands (e.g. angry or highly engaged customers being more likely to reply) is unknown and could move the price/skip ranking. Treat all shares as ±3-4pp at best, and the 6.1pp weighted skip-over-price lead as directional.
- Weighting is on tenure only. Family is still 47% of the sample vs 34% of the population, and region is unweighted; a joint tenure×plan weight would shift plan-level figures further.
- Q3 and Q4 are hypothetical stated intent, not behaviour. Both are inflated by acquiescence, and Q3 describes a vivid, effort-free feature while Q4 asks for a price trade-off — some of the 35.7pp gap is question framing, not true preference. Use them to rank levers, never to forecast reactivation or revenue.
- Q1 forces one main reason, so it under-counts compound causes. The open text repeatedly shows price+skip, variety+skip, delivery+packaging in the same answer; 'portion sizes' at 6.7% is almost certainly understated because 'too many meals' has no option and lands in skip/price/other.
- The open text is 190 of 640 (30%), self-selected and unweighted, and skews long-tenure/England. Theme counts are hypothesis-generating and must not be turned into percentages of customers.
- Small bases: Northern Ireland n=12, Wales n=36, Scotland n=46, under-3-months Family n=26. Nothing regional or plan-specific in this survey is decision-grade on its own.
- The survey covers only people who already cancelled (S1/S2), in one six-month window, interviewed 1-6 months later. It says nothing about current customers' churn risk, about how many active customers use or hit the limits of today's skip function, or about recall accuracy.
- No economics are in the data: no margin, LTV, skip-frequency or build-cost figures. A 20% cut is a permanent revenue reduction across the entire retained base; a skip feature is a one-off build cost plus ongoing revenue lost to skipped weeks. The survey cannot adjudicate that trade-off — finance modelling must.
- Statistical significance on the Family/Couples gap is an absence of evidence, not proof of equivalence: with n=339/301 the study could only reliably detect a gap of ~8pp or more.
Next steps
- decisionApprove a staged flexibility roadmap (phase 1: later cut-off, remove confirmation hoops, multi-week/bulk skip, remove skip cap, skip link in reminders, pause retaining settings; phase 2: one-tap in-app skip/pause) and formally decline the blanket 20% price cut; re-scope the four-month estimate against phase 1.
- experimentRandomised win-back trial on the remaining cancellers not in the survey panel: four arms — flexibility/pause messaging, 20% time-limited discount, smaller/2-meal plan offer, holdout control; stratify by tenure and plan.
- researchPull behavioural data for all 9,850 cancellers and the active base: skip attempts vs successes, cap hits, cut-off misses, support contacts about pausing, and weeks between last skip attempt and cancellation; also model revenue forgone per skipped week.
- researchNon-response bias check: compare responders to the full 9,850 on database variables, then run a 3-question incentivised SMS/phone follow-up with ~200 randomly sampled non-responders (main reason, plan, tenure).
- experimentPilot a two-meal and adjustable-meal-count option (and an adjustable portion size) with new and at-risk customers, priced per portion rather than discounted.
- researchAudit NI (and Scotland/Wales) delivery performance from courier and packing records for March-August 2025: on-time-in-slot rate, missing-box rate, missing-ingredient rate, refund rate, by region; compare to England.
- researchMenu-rotation review for long-tenure customers: measure actual dish repeat frequency per customer over 12 weeks, then interview 12-15 >12-month cancellers who cited repetition.
- experimentReplace blanket discounting with a pricing-policy change: cap in-life price increases, extend or smooth the intro-offer transition, and introduce an existing-customer loyalty benefit; test on a holdout basis.
- monitorStand up a monthly churn-reason dashboard from cancellation-flow data plus a rolling short survey, weighted to the population tenure×plan mix, with small-base suppression (n<30).
Open-text themes it coded
Flexibility to skip, pause or slow the weekly rhythm (holidays, travel, unpredictable weeks) 48Skip/pause is hard to execute: hidden or broken UI, confirmation hoops, phone calls, skip caps, early cut-offs, losing settings on rejoin 31Too many meals per week / want a smaller or adjustable plan (minimum three is too much) 31Recipe variety / repetitive menus 40Price / value for money (level, affordability, cost per portion) 42Price rises, expiring intro offers and new-customer-only discounts (loyalty feels penalised) 10Portions too small / want adjustable serving sizes 12Packaging waste and non-recyclable materials 18Delivery reliability: late, wrong-day, damaged or missing boxes 15Missing ingredients / packing accuracy 5Nothing would have kept them — life circumstances changed 6
The survey it planned
4 screening questions and 12 questions, as the model wrote them.
-
S1
Our records show that a Fernway recipe box subscription for your household was cancelled. Which of these describes you best?
One answer- I had a Fernway subscription and I cancelled it
- I had a Fernway subscription and someone else in my household cancelled it
- I currently have an active Fernway subscription
- I have never had a Fernway subscription
Continues if I had a Fernway subscription and I cancelled it; I had a Fernway subscription and someone else in my household cancelled it
-
S2
When was the Fernway subscription cancelled?
One answer- In the last 3 months
- 4 to 6 months ago
- 7 to 12 months ago
- More than 12 months ago
- It has not been cancelled
- Not sure
Continues if In the last 3 months; 4 to 6 months ago
-
S3
From your first box until the day it was cancelled, roughly how long were you subscribed to Fernway in total?
One answer- Less than 3 months
- 3 to 12 months
- More than 12 months
- Not sure
Continues if Less than 3 months; 3 to 12 months; More than 12 months; Not sure
-
S4
Were you the person who mainly looked after the Fernway subscription - choosing meals, changing deliveries, or cancelling?
One answer- Yes, mainly me
- I shared this with someone else in my household
- No, someone else did all of it
- Not sure
Continues if Yes, mainly me; I shared this with someone else in my household
-
Q1
In your own words, why did you cancel your Fernway subscription? Please tell us what mattered most to you.
Open text -
Q2
Which ONE of these was the main reason you cancelled your Fernway subscription?
One answer- The weekly price was too high for what we got
- Skipping or pausing weeks was not flexible enough
- Having to decide at least six days before delivery
- Not being able to order fewer meals in a week
- Too much food or packaging waste
- Recipe quality or taste
- Not enough recipes that suited our diet or tastes
- Ingredients arriving missing, damaged or not fresh
- Delivery problems (late, failed or wrong)
- Our circumstances changed (moving, travel, work, household)
- We switched to another recipe box or food service
- We wanted to shop and cook for ourselves again
- Our household budget got tighter overall
- We only ever intended to try it for a short time
- Something else
- I don't remember
-
Q3
And which of these played any part at all in your decision to cancel? Please select all that apply, including the main reason you just gave.
Any that apply- The weekly price was too high for what we got
- Skipping or pausing weeks was not flexible enough
- Having to decide at least six days before delivery
- Not being able to order fewer meals in a week
- Too much food or packaging waste
- Recipe quality or taste
- Not enough recipes that suited our diet or tastes
- Ingredients arriving missing, damaged or not fresh
- Delivery problems (late, failed or wrong)
- Our circumstances changed (moving, travel, work, household)
- We switched to another recipe box or food service
- We wanted to shop and cook for ourselves again
- Our household budget got tighter overall
- We only ever intended to try it for a short time
- Something else
- None of these - only the main reason I gave
-
Q4
Thinking about the whole time you were subscribed, how satisfied were you with Fernway overall?
ScaleScale 1-11: 0 - Very dissatisfied to 10 - Very satisfied
-
Q5
Thinking about what you paid each week, how would you rate Fernway's value for money?
ScaleScale 1-11: 0 - Very poor value for money to 10 - Very good value for money
-
Q6
While you were subscribed, Fernway let you skip a week by changing your delivery in the app at least six days before it was due. Which of these happened to you? Please select all that apply.
Any that apply- I never wanted to skip or pause a week
- I skipped weeks and found it easy
- I skipped weeks, but it was fiddly or hard to find in the app
- I tried to skip, but the six-day deadline had already passed
- I forgot to change my delivery in time
- I did not know I could skip a week
- I wanted to pause for several weeks, but cancelling seemed the only option
- I wanted fewer meals in a week, but could not reduce the number
- Something else
-
Q7
In the last three months before you cancelled, how many Fernway boxes arrived that you did not want that week? Please enter a number from 0 to 13.
NumberRouting Shown only if Q6 is: I skipped weeks and found it easy; I skipped weeks, but it was fiddly or hard to find in the app; I tried to skip, but the six-day deadline had already passed; I forgot to change my delivery in time; I did not know I could skip a week; I wanted to pause for several weeks, but cancelling seemed the only option; I wanted fewer meals in a week, but could not reduce the number; Something else
-
Q8
This question just checks that the survey is displaying correctly. Please choose 'Blue' below, whatever you might otherwise pick.
One answer- Red
- Blue
- Green
- Yellow
-
Q9
Here are some changes Fernway could make. Please rank them by how much difference each would have made to you while you were a customer. Drag them into order: 1 = would have made the biggest difference, 6 = would have made the least difference.
Ranking- Lower price: 20% off every box, every week
- Skip any week with one tap, up to the day before delivery
- Choose fewer meals in a week (for example 2 instead of 4)
- Pause for a month or more without cancelling
- More recipe choice and better recipes
- Fresher ingredients and more reliable delivery
Judges: double-barrelled'More recipe choice and better recipes' combines breadth and quality, while 'Fresher ingredients and more reliable delivery' combines ingredient freshness and delivery reliability. Each pair receives one rank despite potentially different priorities.
The options 'More recipe choice and better recipes' and 'Fresher ingredients and more reliable delivery' each ask the respondent to rank two distinct concepts at once.
-
Q10
Imagine Fernway made the one change shown below and everything else stayed exactly as it was when you were a customer. How likely would you be to resubscribe within the next three months? [PROGRAMMING NOTE - SHOW EXACTLY ONE VERSION BELOW, ASSIGNED AT RANDOM IN EQUAL THIRDS, BALANCED WITHIN TENURE GROUP AND PLAN. THE RESPONDENT SEES ONLY THEIR VERSION. PRICES PIPED FROM THE CUSTOMER DATABASE.] Version A: Fernway costs 20% less than you paid - £{{price_20pct}} per box instead of £{{price_paid}} - every week, permanently. Version B: You can skip any week with one tap in the app, up to the day before delivery. Prices stay the same. Version C: Nothing changes - same prices, and you still need to change your delivery at least six days ahead.
ScaleScale 1-5: 1 - Not at all likely to 5 - Very likely
-
Q11
Since cancelling Fernway, how do you mainly get the evening meals you used to get from us?
One answer- Another recipe box service (for example Gousto, HelloFresh, Mindful Chef)
- Mostly cooking from scratch with supermarket shopping
- Mostly ready meals, frozen food or meal deals
- Mostly takeaways or eating out
- A mix of the above
- Something else
-
Q12
Which best describes your household's food spending over the past six months?
One answer- We have been cutting back a lot
- We have been cutting back a little
- No real change
- We have been spending more
- Not sure
Sample plan
FRAME AND INVITATION: census email invitation to every account in the customer database with a cancellation date in the last six months (Fernway delivers UK-only, so no geography screen is needed; people who have since moved abroad are kept in, because moving is itself a cancellation reason). Deduplicate to one adult per household; suppress test/duplicate accounts, deceased customers and anyone opted out of research contact, and log the suppression count as a small non-coverage risk. Fieldwork 14 days: invitation plus reminders on day 4 and day 9 to non-openers/non-completers. £5 e-voucher for every complete (an incentive matters here because angry leavers and indifferent leavers otherwise respond at different rates). Neutral invitation wording ('Tell us what went wrong - about 5 minutes'), with no mention of price, discounts or new features, so response does not depend on why someone left. Soft launch n=30 to confirm median completion is under 6 minutes (design estimate ~5 minutes) and that routing and randomisation work.
TENURE STRATA come from the database (first delivery date to cancellation date), not from self-report: under 3 months, 3-12 months, over 12 months. S3 is a validation check only; where self-report and the record disagree by more than one band, the record is used and the case is flagged.
PRECISION - ISSUE TO RESOLVE WITH THE BOARD BEFORE FIELDWORK: +/-5 percentage points at 95% confidence for a proportion inside each tenure group requires n=385 per group (1.96^2 x 0.25 / 0.05^2), i.e. about 1,155 completes, not 600. With 600 split equally (200 per group) the margin is about +/-6.9pp per tenure group and +/-4.0pp overall. Options in order of preference: (1) raise the target to ~1,155 completes if the cancellation frame is large enough - check frame counts first, since at 9% monthly churn the six-month frame may or may not support it; same questionnaire, incentive cost rises from ~£3,000 to ~£5,800. (2) Keep ~600 and have the board formally accept +/-7pp per tenure group. (3) Apply a finite population correction: because we invite the whole frame, a stratum with fewer than ~800 cancellers can reach +/-5pp with fewer completes (e.g. N=500, n=280). What we will not do is report 600 completes as if it met +/-5pp per group. Any cut deeper than tenure (reason x tenure, concept arm x tenure) is +/-10pp or worse and will be labelled directional, not decision-grade.
QUOTAS AND WEIGHTS: target 200 completes per tenure stratum (equal allocation maximises precision for the between-group comparisons the board asked for); monitor plan (Couples/Family) within stratum as a soft interlock. Equal allocation over-represents short-tenure leavers relative to the frame, so apply design weights to the true tenure mix for all headline/overall figures; report per-stratum figures unweighted with confidence intervals. Send targeted top-up reminders to any stratum falling short before closing fieldwork.
FEASIBILITY CONTINGENCY: if a stratum cannot be filled from the six-month frame, extend that stratum to 7-9 months post-cancellation, flag recency, and test for a recency effect before pooling; if it still falls short, publish the achieved margin rather than over-claim.
RANDOMISATION: Q10 has three concept arms (A: permanent 20% price cut; B: one-tap skip up to the day before; C: no change), assigned at random in equal thirds, stratified by tenure and plan: ~200 per arm overall, ~67 per arm per tenure group. This is powered to detect roughly a 12pp difference in top-2-box intent between two arms overall (80% power, alpha .05); arm reads within a tenure group are directional only. Arm A pipes the respondent's real plan price so the discount is concrete and identical in credibility to the feature concept.
BIAS CONTROLS, INCLUDING THE CEO'S PRICE HYPOTHESIS: the unprompted open question (Q1) comes first, before any list or concept mentions price; reason lists are balanced across price, flexibility, product, service and life circumstances with randomised order (fixed 'Something else' / 'None of these'); the price and flexibility concepts are written with equal length and specificity and are tested between-subjects, so nobody rates both and no respondent can signal that 'everything matters'; no item asks 'would a price cut bring you back'. Coding frame and analysis plan are pre-registered before fieldwork, and price and flexibility findings are reported side by side whatever they show. Data quality: exclude Q8 attention failures and speeders (under 2 minutes) from the headline analysis and publish a sensitivity check with them included; report item non-response per question.
NON-RESPONSE AND LIMITS: compare responders with the full invited frame on database variables (tenure, plan, weekly spend, boxes received, times skipped, support contacts, recorded cancellation reason); if materially different, calibrate the weights and state the residual risk. This sample can only speak about winning back leavers and their retrospective experience: it cannot estimate the effect of either change on retaining current subscribers or on acquisition, and stated intent overstates behaviour. Triangulate with behavioural data (share of current subscribers hitting the six-day cut-off, skip frequency versus churn) and follow with a live win-back/retention A/B before committing four months of build or a permanent 20% price cut.