Benchmarks / SurveyBench / Why Fernway customers cancel / Gemini 3.5 Flash Lite

Measured by Spring Prompt

Gemini 3.5 Flash Lite: Why Fernway customers cancel

The decision: whether to build one-tap skipping of any week (about four months of product work) or cut the price of the plans by 20%, to win back and keep customers. The brief and the data →

Verdict
✗ Not sound
Research score
64 of 100
Analysis rating
662
Head to head, this task
won 3 of 10

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A double-barrelled question: Q1
  • A double-barrelled question: Q2
  • A double-barrelled question: Q4
  • A double-barrelled question: Q8

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Missed a trap: Northern Ireland respondents (base 12) cite delivery problems far more often.
  • Missed a trap: Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • Missed a trap: The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

Findings the judges found unsupported

  • Difficulty skipping weeks is the single largest driver of churn, cited by over a third of cancelled customers, and vastly outperforms price as a motivator for returning when a targeted fix is applied.The survey identifies the most commonly reported cancellation reason, not the largest causal driver of churn. Higher hypothetical return intent also does not establish that a skipping feature actually outperforms a price cut in winning customers back.
  • The CEO's hypothesis that price is the primary driver of customer loss is partially correct for newer customers but overstated overall, and a 20% price cut fails to win back the majority of even price-sensitive churners.Weighting the displayed tenure-specific price percentages to the population gives approximately 26.3%, not 26.6%. Also, 49.3% expressing likely return does not establish that a price cut would fail to win back a majority in practice.
  • The Head of Product's concern regarding the Family plan is unfounded; dissatisfaction is driven by tenure rather than plan type.The tables support tenure composition as the explanation for the aggregate satisfaction reversal, but do not establish that tenure causes dissatisfaction or that all concerns about the Family plan are unfounded.
  • The Operations Director faces a severe, localized delivery crisis in Northern Ireland that requires immediate operational intervention.Eight delivery-related cancellations among only 12 Northern Ireland respondents do not establish a severe regional delivery crisis requiring immediate intervention. The small-base caveat does not justify this factual conclusion.
  • Recipe monotony is a major secondary churn driver, particularly among long-term customers, compounding the frustration with inflexible subscription management.The supplied comments contain 40 answers mentioning recipe variety or repetition, rather than the claimed 38. The analysis also gives a conflicting count of 36 in its theme summary.

Scorecard

Numbers 9 of 11 right

QuestionIts answerTrue
What percentage of respondents gave price as their main reason for cancelling?✓21.6%21.6%
Weighted to the tenure mix of all customers who cancelled, what percentage would give price as their main reason?✓26.6%26.3%
Among respondents whose main reason was that it was hard to skip weeks, what percentage said they would be likely (fairly or very) to come back if any week could be skipped with one tap?✓58.6%58.6%
How many of the open-text answers mention recipe variety or repetitive recipes?✓3840
What is the 95% margin of error, in percentage points, on the unweighted share of respondents giving price as their main reason?✓3.1 points3.19 points
Is the difference in the share satisfied (4-5) between Family and Couples respondents, across all respondents, statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the tenure mix of all 9,850 customers who cancelled, about how many of them would give 'hard to skip weeks' as their main reason?✗3,4963,193
How many of the open-text answers from respondents who were customers for over 12 months mention recipe variety or repetitive recipes?✗2025
Among respondents whose main reason was that it was hard to skip weeks, by how many percentage points does the share likely to come back with one-tap skipping exceed the share likely to come back with a 20% price cut?✓35.7 points35.7 points
What percentage of the customers invited completed the survey?✓6.5%6.5%
Among respondents whose main reason was price, what percentage said they would be likely (fairly or very) to come back if the plan cost 20% less?✓49.3%49.3%

Traps in the data 5 of 8 handled

  • ✗Northern Ireland respondents (base 12) cite delivery problems far more often.
  • ✓Family-plan respondents look less satisfied overall, but are more satisfied than Couples within every tenure band: Family customers skew to long tenure, and long-tenure customers are less satisfied.
  • ✓Long-tenure customers are 50% of respondents but 20% of cancellers; short-tenure customers, who cite price more, are under-represented.
  • ✓'Likely to come back' answers are stated intentions about a hypothetical feature.
  • ✓The chief executive believes customers leave mainly because of price.
  • ✓Only 702 of 9,850 invited customers started the survey (about 7%); those who respond may differ from those who do not.
  • ✗Only customers who cancelled were surveyed, so the survey cannot show what distinguishes leavers from customers who stay; the reasons are self-reported.
  • ✗The one-tap skipping question (Q3) came immediately before the price-cut question (Q4), so answers to Q4 may have been framed by Q3.

The analysis

Recommendation

Build the one-tap skipping feature. Reject the 20% price cut.

What would change it

Definitive conjoint or pricing elasticity modeling proving that a 20% price cut generates enough incremental volume to offset the massive margin erosion, combined with evidence that skipping is cheap to build or easily solved via a lightweight third-party integration rather than 4 months of heavy product engineering.

Findings

  1. Difficulty skipping weeks is the single largest driver of churn, cited by over a third of cancelled customers, and vastly outperforms price as a motivator for returning when a targeted fix is applied.

    High confidenceBase 640

    35.5% of total respondents (Base: 640) gave 'Hard to skip weeks' as their main reason for cancelling. When offered a one-tap skip feature, 58.6% of this group (Base: 227) reported they would likely return, compared to only 22.9% who would return for a 20% price cut—a massive 35.7 percentage point advantage for product flexibility over price discounting.

  2. The CEO's hypothesis that price is the primary driver of customer loss is partially correct for newer customers but overstated overall, and a 20% price cut fails to win back the majority of even price-sensitive churners.

    High confidenceBase 640

    While 31.2% of customers with under 3 months tenure (Base: 138) cancelled due to price, the overall unweighted average is 21.6% (weighted to population tenure mix: 26.6%). Furthermore, among those who cited price as their main reason (Base: 138), only 49.3% said they would likely return if prices were cut by 20%.

  3. The Head of Product's concern regarding the Family plan is unfounded; dissatisfaction is driven by tenure rather than plan type.

    High confidenceBase 640

    Net satisfaction (4-5) for the Couples plan is 53.4% (Base: 339) and for the Family plan is 49.2% (Base: 301)—a minor difference that is not statistically significant (p > 0.05). Tenure is the real differentiator: net satisfaction drops from 66.7% for under 3 months to 39.7% for over 12 months.

  4. The Operations Director faces a severe, localized delivery crisis in Northern Ireland that requires immediate operational intervention.

    Medium confidenceBase 12

    While overall delivery problems account for only 9.1% of cancellations (Base: 640), 66.7% of respondents from Northern Ireland (Base: 12) cited delivery problems as their primary reason for leaving.

  5. Recipe monotony is a major secondary churn driver, particularly among long-term customers, compounding the frustration with inflexible subscription management.

    High confidenceBase 640

    18.0% of total respondents (Base: 640) cite repetitive recipes as their main cancellation reason, rising to 25.4% among customers with over 12 months tenure (Base: 307). 38 open-text answers explicitly mention menu fatigue.

Caveats

  • The survey is subject to non-response bias, capturing only 6.5% of invited churned customers, meaning highly dissatisfied or completely detached customers may be underrepresented.
  • The sample size for Northern Ireland is extremely small (Base: 12), so while the delivery issue signal is prominent, percentage figures for this region carry a wide margin of error.
  • Stated purchase intent in Q3 and Q4 ('likelihood of coming back') typically overstates actual consumer behavior when faced with friction in reality.

Next steps

  • decisionInitiate product development for one-tap skipping and flexible pausing.
  • monitorConduct an immediate operational audit of courier performance in Northern Ireland.
  • experimentExpand menu variety and rotate recipe pools faster, especially targeting long-term subscribers.

Open-text themes it coded

Difficult skipping/pausing 40Recipe repetition and lack of variety 36Too many meals per week / inflexibility of box size 29Price and value for money concerns 37Delivery issues and missing items 16Packaging waste and non-recyclable materials 16Portion size dissatisfaction 11

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    When did you cancel your Fernway meal-kit subscription?

    One answer
    • In the last 6 months
    • Between 6 and 12 months ago
    • More than 12 months ago
    • I have never subscribed to Fernway

    Continues if In the last 6 months

  2. S2

    For how long were you actively subscribed to Fernway before you cancelled?

    One answer
    • Less than 3 months
    • 3 to 12 months
    • More than 12 months

    Continues if Less than 3 months; 3 to 12 months; More than 12 months

  3. Q1

    What were the primary reasons you decided to cancel your Fernway subscription? (Select up to 3)

    Any that apply
    • It became too expensive / value for money
    • It was too hard to skip weeks or manage my deliveries
    • We got tired of the recipe options / lack of variety
    • We didn't have enough time to cook the meals
    • Food quality or portion size issues
    • Changes in household size or living situation
    • Other (please specify)
    Judges: double-barrelled

    Combined options prevent distinguishing high price from poor value, and food quality problems from portion-size problems.

    Several options combine separate reasons under one tick: 'Food quality or portion size issues', 'too expensive / value for money', and 'too hard to skip weeks or manage my deliveries'. This blurs the reasons for cancelling, which is the core of O1.

  4. Q2

    Thinking back to when you were subscribed, how satisfied were you with the following aspects of Fernway?

    Scale
    • Recipe taste and quality
    • Portion sizes
    • Variety of meals available
    • Ease of managing deliveries (e.g., skipping weeks)
    • Overall value for money

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: double-barrelled

    Recipe taste and quality receive one rating, although respondents could be satisfied with one and dissatisfied with the other.

    'Recipe taste and quality' asks about two distinct attributes on a single scale.

  5. Q3

    How important was the 6-day advance notice requirement for skipping or changing a delivery in your decision to cancel?

    Scale

    Scale 1-5: Not at all important to Extremely important

    Judges: assumptive

    Assumes every respondent knew about the 6-day notice rule and that it played some part in their decision. Those who never tried to skip have no fitting answer.

    Assumes the respondent knew about the 6-day advance notice requirement and that it was a factor in their decision to cancel.

  6. Q4

    When you wanted to skip a week, how frequently did you find it difficult or forget to make the change within the required deadline?

    One answer
    • Very frequently
    • Occasionally
    • Rarely
    • Never / I never tried to skip
    Judges: double-barrelled

    One frequency answer combines finding the change difficult with forgetting the deadline. The final option also combines never experiencing difficulty with never attempting to skip.

    Combines 'found it difficult' and 'forgot' in one answer. The last option also merges 'never had trouble' with 'never tried to skip'.

  7. Q5

    Imagine you were considering returning to a meal-kit service. Rank the following potential changes from 1 (most important to you) to 4 (least important to you).

    Ranking
    • A 20% price reduction on all plans
    • One-tap, instant skipping of any week right up to the day of delivery
    • A wider variety of weekly recipes
    • More flexible plan sizes or fewer meals per week
  8. Q6

    If Fernway reduced its subscription prices by 20%, how likely would you be to restart your subscription?

    Scale

    Scale 1-5: Very unlikely to Very likely

  9. Q7

    If Fernway introduced one-tap instant skipping of any week through the app without strict advance deadlines, how likely would you be to restart your subscription?

    Scale

    Scale 1-5: Very unlikely to Very likely

  10. Q8

    Which of the following, if any, would make you consider returning to Fernway? (Select all that apply)

    Any that apply
    • A 20% price cut
    • One-tap skipping anytime before delivery
    • A special discount code or welcome-back offer
    • New features or improved recipe variety
    • Nothing would convince me to return
    Judges: double-barrelled

    One option combines new features with improved recipe variety, so a selection cannot identify which would attract the respondent. 'New features' is unspecified and could mean substantially different changes to different respondents.

    The option 'New features or improved recipe variety' combines two unrelated changes.

  11. Q9

    Overall, how satisfied were you with your Fernway subscription while you were active?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  12. Q10

    Is there anything else you would like to share about why you left Fernway or what would have made you stay?

    Open text
Sample plan

Email invitations sent to all UK customers who cancelled their Fernway subscription in the last six months, drawn directly from the customer database. Target sample size is 600 completed responses. To achieve the required precision (plus or minus 5 percentage points at 95% confidence) for each tenure group, soft quotas will be monitored during fieldwork to ensure sufficient representation across the three tenure segments: under 3 months, 3-12 months, and over 12 months.