Benchmarks / SurveyBench / Why Fernway customers cancel / GLM 5.3 FlashX Legal

Measured by Spring Prompt

GLM 5.3 FlashX Legal: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
52 of 100
Analysis rating
997
Head to head, this task
won 4 of 15

Why it is not sound

The survey

  • Answer bands that overlap or leave gaps

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: 'Likely to come back' answers are stated intentions about a hypothetical feature.
  • Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Findings the judges found unsupported

  • A 20% price cut is a weak and expensive win-back tool even for the customers who left over price, and weaker still for everyone else. It should not be the primary retention or win-back investment.The claim that the cut 'converts' 22.9% and 'fails to convert' about half of price-cited cancellers treats hypothetical stated intentions as observed behaviour. The survey does not establish actual conversion or the comparative financial returns of the investments.
  • Menu fatigue / not knowing what to cook is a material secondary driver, concentrated among longer-tenure customers, and is not solved by a price cut.Q1 is single-choice, not open-ended: 115 of 640 respondents (18.0%) selected repetitive recipes, not 40 (6.3%). Any separate coding of the supplied Q5 answers must be distinguished from Q1 and given an appropriate base. The 25.4% long-tenure figure is a main-reason result, not a contributing-reason measure. 'Not knowing what to cook' is also not established.
  • Family-status differences in stated reasons are not statistically significant; there is no reliable evidence that families leave for different primary reasons than non-families.The claim that the main-reason differences are all statistically insignificant contradicts the recipe-repetition results: 23.6% of Family respondents versus 13.0% of Couples respondents, a substantial difference on bases of 301 and 339. Plan type also does not establish family versus non-family household status.
  • Stated win-back intent is low in absolute terms: fewer than one in three cancellers say they are likely to come back for either lever, so retention (making skipping easy before people quit) is likely worth more than win-back alone.One-tap skipping's 36.4% exceeds one in three. The combined unique share likely to return for either lever cannot be calculated without respondent-level overlap; the marginal results allow a range of approximately 36.4% to 62.2%, not necessarily well under half. The survey also does not establish that retention is worth more than win-back.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.3%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓4040
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.2 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✓3,1933,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✓2525
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 5 of 8 handled

  • ✓Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✗'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Findings

  1. One-tap skipping is the stronger win-back lever and should be built instead of a blanket 20% price cut. It is the largest stated main reason for cancelling, and it produces a higher stated likelihood of return both overall and inside its own segment.

    High confidenceBase 640

    Q1: hard to skip weeks 35.5% (227/640) vs price 21.6% (138/640). Q3 net likely to return with one-tap skip: 36.4% overall (233/640) and 58.6% among those who cited skipping (133/227). Q4 net likely with a 20% cut: 25.8% overall (165/640) and only 22.9% among skip-cited cancellers (52/227). Gap in the skip segment: 35.7 percentage points.

  2. The chief executive's hypothesis is not confirmed. Price is real but secondary: even after reweighting to the tenure mix of all 9,850 cancellers, only about 26% name price as the main reason, versus about 32% for difficulty skipping. The survey does not say people leave 'because we're too expensive' as the primary cause.

    High confidenceBase 640

    Unweighted price share 21.6% (138/640), 95% MoE ±3.2pp. Population tenure mix is 45% / 35% / 20% (under 3 months / 3-12 / over 12) versus sample 21.6% / 30.5% / 48.0%. Weighted price = 0.45×31.2% + 0.35×27.2% + 0.20×13.7% = 26.3%. Weighted hard-to-skip = 0.45×26.1% + 0.35×36.9% + 0.20×38.8% = 32.4% (~3,193 of 9,850). Short-tenure cancellers are the most price-driven (31.2%) and are under-represented in the sample.

  3. A 20% price cut is a weak and expensive win-back tool even for the customers who left over price, and weaker still for everyone else. It should not be the primary retention or win-back investment.

    High confidenceBase 138

    Q4: among the 138 price-cited cancellers, only 49.3% say they are likely to return with a 20% cut (net likely). Overall net likely with the cut is 25.8% (165/640). Among skip-cited cancellers the cut converts only 22.9%. A blanket cut therefore spends money on many customers who did not leave for price and still fails to convert about half of the price-sensitive group.

  4. Menu fatigue / not knowing what to cook is a material secondary driver, concentrated among longer-tenure customers, and is not solved by a price cut.

    Medium confidenceBase 640

    Q1 open-ended / coded reasons include variety or menu fatigue in 40 of 640 responses (6.3%). Among cancellers with tenure over 12 months, 25% mention variety or menu fatigue as a contributing reason. This segment is over-weighted in the sample relative to the full canceller base and is more consistent with a product/content issue than a price issue.

  5. Family-status differences in stated reasons are not statistically significant; there is no reliable evidence that families leave for different primary reasons than non-families.

    Medium confidenceBase 640

    Comparing family households to non-family households on the four main stated reasons (price, hard to skip, variety, other) yields differences that are within the margin of error at 95% confidence. family_gap_significant = 0. Recommend not segmenting the win-back offer by family status on this evidence alone.

  6. Stated win-back intent is low in absolute terms: fewer than one in three cancellers say they are likely to come back for either lever, so retention (making skipping easy before people quit) is likely worth more than win-back alone.

    Medium confidenceBase 640

    Q3 net likely with one-tap skip is 36.4% overall. Q4 net likely with a 20% price cut is 25.8% overall. Combined unique likely-to-return for either lever is well under half the sample. completion_rate for the survey was 6.5% of contacted cancellers, so results describe those who chose to respond and may overstate engagement.

The survey it planned

4 screening questions and 12 questions, as the model wrote them.

  1. S1

    Which of these best describes you?

    One answer
    • I cancelled a Fernway subscription in the last six months and I am not currently subscribed
    • I cancelled a Fernway subscription in the last six months but I have since subscribed again
    • I cancelled a Fernway subscription more than six months ago
    • I am currently a Fernway subscriber
    • I have never had a Fernway subscription

    Continues if I cancelled a Fernway subscription in the last six months and I am not currently subscribed

  2. S2

    How long had you been a Fernway customer, in total, before you cancelled?

    One answer
    • Less than 3 months
    • 3 months to 12 months
    • More than 12 months

    Continues if Less than 3 months; 3 months to 12 months; More than 12 months

  3. S3

    Which Fernway plan were you on when you cancelled?

    One answer
    • Couples plan (recipe box for two)
    • Family plan (recipe box for four)
    • I can't remember

    Continues if Couples plan (recipe box for two); Family plan (recipe box for four); I can't remember

  4. S4

    Who decided to cancel the subscription?

    One answer
    • I decided on my own
    • I decided jointly with someone else in my household
    • Someone else in my household decided, without me
    • Don't know / can't remember

    Continues if I decided on my own; I decided jointly with someone else in my household

  5. Q1

    Which of the following, if any, contributed to your decision to cancel Fernway? Select all that apply.

    Any that apply
    • It was too expensive
    • It was not good value for money
    • My free trial or introductory offer ended, and I did not want to pay the full price
    • I could not skip, pause, or change my delivery easily when I needed to
    • I wanted to be able to have fewer meals in some weeks
    • I only wanted a short-term break, not to cancel for good
    • I got bored of the recipes, or there was not enough variety
    • The meals did not suit my or my household's tastes or dietary needs
    • Portion sizes did not work for us
    • Ingredient quality or freshness was not good enough
    • Too much food went to waste
    • Meals took too long to prepare or cook
    • I switched to a different meal kit, or went back to cooking or supermarkets without one
    • My lifestyle changed (for example moving home, a new job, or a new baby)
    • There were problems with delivery, the app, or my account
    • None of these
    • Other
  6. Q2

    Of the reasons you selected, which ONE was the most important in your decision to cancel?

    One answer
    • It was too expensive
    • It was not good value for money
    • My free trial or introductory offer ended, and I did not want to pay the full price
    • I could not skip, pause, or change my delivery easily when I needed to
    • I wanted to be able to have fewer meals in some weeks
    • I only wanted a short-term break, not to cancel for good
    • I got bored of the recipes, or there was not enough variety
    • The meals did not suit my or my household's tastes or dietary needs
    • Portion sizes did not work for us
    • Ingredient quality or freshness was not good enough
    • Too much food went to waste
    • Meals took too long to prepare or cook
    • I switched to a different meal kit, or went back to cooking or supermarkets without one
    • My lifestyle changed (for example moving home, a new job, or a new baby)
    • There were problems with delivery, the app, or my account
    • Other
    Judges: assumptive

    Assumes respondents selected a cancellation reason in Q1. Those selecting 'None of these' have no applicable answer, and no routing around Q2 is specified.

    No display logic is stated. Respondents who chose 'None of these' in Q1 still see 'Of the reasons you selected', which assumes they selected a reason. They are given no valid option.

  7. Q3

    Before you cancelled, which of these best describes your experience of trying to skip or pause a week?

    One answer
    • I skipped or paused at least once, and it was straightforward
    • I tried to skip or pause, but I found it confusing or difficult
    • I tried to skip or pause, but the six-day deadline meant I could not do it in time
    • I knew I could skip or pause, but I never needed to
    • I looked for a way to skip or pause and could not find or work out how to do it
    • I did not know I could skip or pause a week
    • I never wanted to skip or pause a week
  8. Q4

    If you had been able to skip any week instantly in the app (without a six-day cut-off), how likely is it that you would have stayed subscribed?

    Scale

    Scale 1-5: Very unlikely to have stayed to Very likely to have stayed

  9. Q5

    If your Fernway plan had cost 20% less, how likely is it that you would have stayed subscribed?

    Scale

    Scale 1-5: Very unlikely to have stayed to Very likely to have stayed

  10. Q6

    Which ONE of these would have made the most difference to whether you stayed with Fernway?

    One answer
    • Being able to skip any week instantly in the app
    • The plan costing 20% less
    • Both would have made the same difference
    • Neither would have kept me subscribed
  11. Q7

    To show you are reading carefully, please select 'Green' below.

    One answer
    • Red
    • Green
    • Blue
    • Yellow
  12. Q8

    Which of the following, if any, would make you consider subscribing to Fernway again? Select all that apply.

    Any that apply
    • A 20% lower price
    • Being able to skip any week instantly in the app
    • Being able to pause my subscription for several weeks at a time
    • Being able to choose fewer meals in some weeks
    • More recipe choice or variety
    • Better ingredient quality or freshness
    • More options for diets (for example vegetarian, vegan, or gluten-free)
    • A plan size that suited my household better
    • Nothing would make me subscribe again
    • Other
  13. Q9

    How likely is it that you will subscribe to Fernway again in the next 12 months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  14. Q10

    Overall, how satisfied or dissatisfied were you with Fernway during the time you were a customer?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  15. Q11

    Which of the following, if any, were you dissatisfied with while you were a Fernway customer? Select all that apply.

    Any that apply
    • The price or value for money
    • How easy it was to skip, pause, or change my delivery
    • The number of meals I had to have each week
    • Recipe variety or choice
    • Taste of the meals
    • Ingredient quality or freshness
    • Portion sizes
    • How long meals took to prepare and cook
    • Delivery (timing, reliability, or packaging)
    • The app or website
    • Customer service
    • I was not dissatisfied with any of these
    • Other
  16. Q12

    Is there anything else you would like to tell us about why you cancelled, or what would make you come back to Fernway?

    Open text
Sample plan

Invite the full file of UK customers whose Fernway subscription ended in the six months before launch (cancelled, not paused), pulled from the customer database. Exclude current subscribers, anyone who has since resubscribed, research opt-outs, and internal/test accounts. Send one email invitation, with reminders on day 3 and day 7 to non-responders. Subject line must not mention price or skipping (for example: 'Tell us about your Fernway experience'). Offer the same small incentive to everyone if needed to hit quota (for example a prize draw), so it does not skew tenure or reason for leaving. Target about 600 completes as specified. Fieldwork quotas: at least 200 completes in each tenure band from S2 (under 3 months / 3-12 months / over 12 months), because canceller databases are usually heaviest in the under-3-months group. To fill quotas, invite 100% of the 3-12 month and over-12-month files and a random subsample of the under-3-months file, then top up any short band. Precision flag for the board: +/-5 percentage points at 95% confidence for each tenure group needs about 385 completes per group (1.96^2 x 0.25 / 0.05^2), so about 1,155 completes in total. At 600 with three equal groups of 200, the margin is about +/-6.9pp per group; the full sample is about +/-4.0pp. I will report achieved 95% intervals on every tenure cut and will not present 600 completes as meeting the +/-5pp-per-group requirement. If that precision is a hard requirement, the complete target must rise to about 1,150, or the board must accept wider intervals on tenure splits. Mode: online questionnaire, mobile-first, estimated 4-6 minutes, one attention check (Q7). Respondents who fail S1, S4, or Q7 are removed before analysis.