Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / DeepSeek V4.1 Flash 1017

Measured by Spring Prompt

DeepSeek V4.1 Flash 1017: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
85 of 100
Analysis rating
1,221
Head to head, this task
won 7 of 12

Why it is not sound

The survey

  • A research objective no question serves

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.

Findings the judges found unsupported

  • The CFO's £14,500 average materially overstates the addressable savings pool and does not estimate what a rate rise would bring in.The approximately £0.7bn figure is not an upper bound on deposits. Multiplying the weighted likely share by an overall average balance does not establish the balances held by likely respondents, and neither their transfer amounts nor actual uptake is known. The £2.4bn extrapolation estimates external holdings under midpoint assumptions, not an addressable deposit opportunity.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,845239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.3%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓126126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4646
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 7 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✗The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Do not raise the rate to 3.85% as the sole action. Launch an awareness and onboarding push now: a clear in-app explainer, a visible pots entry point, a simplified setup flow, explicit instant-access reassurance, and a payday nudge. The evidence is strongest for unawareness and for the nudge at the current rate. Test the 3.85% rate rise in a controlled pilot or targeted segment alongside the push, rather than rolling it out universally. Do not build a campaign around customers aged 75+; include them in inclusive creative but not as the core audience.

What would change it

A randomised controlled trial showing that the 3.85% rate alone produces materially more funded pots and incremental deposits than the onboarding push, at acceptable cost per pound deposited, would justify prioritising or adding a universal rate rise. Conversely, if the awareness push fails to convert despite high reach, or if observed transfer amounts are much larger than the survey implies, the rate decision should be revisited.

Findings

  1. The main barrier is not rate but not knowing that savings pots exist. A rate rise alone would miss most non-users.

    High confidenceBase 700

    Q1: 39.4% knew about pots, 50.7% did not, 9.9% were not sure (n=700). Weighted to the customer age mix, 66.4% are unaware, about 239,845 of 361,000. In Q2, 126 of 230 open-text answers ask for awareness/explanation or say they did not know what pots are.

  2. The CFO's £14,500 average materially overstates the addressable savings pool and does not estimate what a rate rise would bring in.

    Medium confidenceBase 413

    Q4: 35.7% hold no savings elsewhere; the £14,513 mean is among only the 413 respondents who reported an amount, and the median falls in the £1,000 to less than £5,000 band. Weighted to the non-pot customer age mix, the mean including non-savers is about £6,500, implying roughly £2.4bn held elsewhere across 361,000 customers. Q7 shows 29.9% likely to start at 3.85%, about 105,000 customers weighted, but the survey does not measure how much they would transfer; even if every likely respondent moved their entire average balance, the upper bound is around £0.7bn, likely much lower in practice.

  3. The chief financial officer's claim that 'the rate is the top reason' is true only among the minority who already knew about pots; it is not a population-level diagnosis.

    High confidenceBase 276

    Q3 was asked only of the 276 respondents who knew about pots. Among them, 29.7% chose 'The interest rate is too low'. That is 11.7% of all 700 respondents and 8.8% weighted to the customer age mix.

  4. A payday nudge at the current rate has higher stated intent than a rate rise overall, especially among unaware customers, while a rate rise performs better among already-aware customers. This supports complementary targeting rather than a rate-only decision.

    Medium confidenceBase 700

    Q5 baseline net likely is 17.4%. Q6 nudge at 3.1%: 33.7% likely overall, 42.5% among those who did not know about pots, 18.8% among those who did. Q7 rate rise to 3.85%: 29.9% overall, 24.2% among those who did not know, 37.0% among those who did (n=700).

  5. A campaign built around customers aged 75 or over is not supported. They are a very small segment, the sample is tiny, and stated intent is not higher than average.

    Low confidenceBase 20

    The 75+ group is 2% of the non-pot population, about 7,220 customers, and only n=20 in the survey; Q3 has n=7 for this group. Q4 shows 65% hold £20,000 or more and a sample mean of about £30,000, but Q7 net likely is 25%, Q6 is 25%, and Q5 is 20%. Approximate total balances held elsewhere are about £0.2bn for 75+, versus about £0.9bn for 35-54 and £0.7bn for 55-74 using weighted means.

  6. Setup, navigation and withdrawal-speed concerns are secondary but actionable. The onboarding push should include simple setup and explicit instant-access reassurance.

    Medium confidenceBase 230

    Q3: 10.9% were not sure how to set one up, 7.6% worried about withdrawal speed, and 11.6% had not got round to it (n=276). Q2 open text includes about 42 setup/navigation/findability mentions and 10 withdrawal-speed mentions across 230 answers.

  7. Rate sensitivity is not uniform, but the apparent age difference is not statistically reliable, so age-based rate targeting should not be over-read.

    Medium confidenceBase 276

    Q3 rate-too-low share: 18-34 14.9% (n=47), 35-54 27.5% (n=102), 55-74 37.5% (n=120), 75+ 28.6% (n=7). The 35-54 versus 55-74 difference is 10.0 percentage points, z about 1.6, p about 0.11, not significant at the 95% level; the Q3 margin of error is ±5.4pp.

  8. A meaningful group cannot be converted by either rate or onboarding because they have no spare money; messaging should not imply that saving is possible for everyone.

    Medium confidenceBase 230

    Q3: 13.4% chose 'I don't have money left over to save' (n=276). Q2 open text includes 23 affordability/no-spare-money mentions across 230 answers.

Caveats

  • The 700 completes are unweighted and self-selected from an email invitation; only 6.0% of invited customers completed, so non-response bias is likely.
  • The achieved age mix differs sharply from the customer base: 55-74 is 28.6% of completes but 13% of the population, and 18-34 is 31.1% versus 46%. Weighted figures are more appropriate for population decisions.
  • Q3 is asked only of the 276 respondents who knew about pots before the survey. It cannot be read as the main reason across all non-users.
  • Stated likelihood at Q5-Q7 is hypothetical, not observed behaviour; Q6 and Q7 are separate scenarios and may suffer order and acquiescence effects. The survey does not measure amount transferred, so revenue or deposit uplift cannot be estimated reliably.
  • Q4 uses bands and midpoints, with £50,000+ treated as £75,000 in the supplied mean. The £14,513 average is among 413 reporters only and is skewed; the median is much lower.
  • The 75+ subgroup has only n=20 overall and n=7 for Q3, so any 75+ comparisons are very uncertain.
  • Open-text coding is judgemental; 230 answers were given and respondents can express more than one theme. The counts should be treated as directional, not exact market sizing.

Next steps

  • decisionLaunch the awareness and onboarding push, including in-app prompts, a visible pots entry point, simplified setup, and a payday nudge at the current rate.
  • experimentRun a randomised pilot of 3.85% versus 3.1% among aware and higher-balance segments, with the same onboarding push in both arms.
  • decisionDo not commission a separate 75+ savings campaign; include older customers in the general creative and monitor their response.
  • researchAdd research or behavioural measurement on how much customers would actually transfer into pots and what would trigger the transfer.
  • monitorMonitor competitor savings rates and the conversion of nudge-exposed customers before deciding on a permanent rate change.

Open-text themes it coded

Awareness and explanation need 126Rate or interest too low 45Withdrawal speed and access reassurance 10Setup, navigation and findability 42No spare money or affordability 23Loyalty or inertia with existing provider 16Nothing would change it or not interested in saving 38

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you a Pennywell customer?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    What is your age?

    One answer
    • Under 18
    • 18-34
    • 35-54
    • 55-74
    • 75 or over
    • Prefer not to say

    Continues if 18-34; 35-54; 55-74; 75 or over

  3. S3

    A savings pot is an instant-access savings account inside the Pennywell app. Have you ever opened a savings pot in the Pennywell app?

    One answer
    • Yes
    • No
    • I'm not sure

    Continues if No

  4. Q1

    Before today, were you aware that Pennywell offers savings pots (instant-access savings accounts inside the Pennywell app)?

    One answer
    • Yes
    • No
    • I'm not sure
  5. Q2

    Which of the following, if any, are reasons you have not opened a Pennywell savings pot? Please select all that apply.

    Any that apply
    • I did not know Pennywell offered savings pots
    • The interest rate is too low
    • I do not have enough spare money to save
    • I already have savings elsewhere
    • Setting one up seems like too much effort
    • I do not understand how savings pots work
    • I am worried about access to my money
    • I am worried about security or safety
    • I have no need for a savings pot
    • Other reason
    • None of these
  6. Q3

    How important is the interest rate on a savings pot when you decide whether to open one?

    Scale

    Scale 1-5: Not at all important to Extremely important

  7. Q4

    Please rank these in order of how much each would make you more likely to open a Pennywell savings pot, from most to least influential.

    Ranking
    • A higher interest rate
    • Knowing that savings pots exist
    • A simple set-up process
    • A payday reminder or prompt
    • Reassurance that my money is safe and easy to access
  8. Q5

    How likely are you to open a Pennywell savings pot in the next 3 months at the current rate of 3.1% AER, assuming you had money available to save?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  9. Q6

    How likely are you to open a Pennywell savings pot in the next 3 months if the rate were 3.85% AER instead of 3.1% AER, assuming you had money available to save?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  10. Q7

    How likely are you to open a Pennywell savings pot in the next 3 months if Pennywell showed a prompt in the app and sent you a payday reminder with a one-tap set-up link, at the current rate of 3.1% AER?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  11. Q8

    Approximately how much do you currently hold in savings outside Pennywell (for example, savings accounts, cash ISAs, premium bonds, or other cash savings)?

    One answer
    • £0
    • £1-£499
    • £500-£1,999
    • £2,000-£4,999
    • £5,000-£9,999
    • £10,000-£19,999
    • £20,000-£49,999
    • £50,000 or more
    • Don't know
    • Prefer not to say
  12. Q9

    How much of that money could you realistically move into a Pennywell savings pot within the next 6 months?

    One answer
    • None of it
    • £1-£499
    • £500-£1,999
    • £2,000-£4,999
    • £5,000-£9,999
    • £10,000-£19,999
    • £20,000-£49,999
    • £50,000 or more
    • Don't know
    • Prefer not to say

    Routing Shown only if Q8 is: £1-£499; £500-£1,999; £2,000-£4,999; £5,000-£9,999; £10,000-£19,999; £20,000-£49,999; £50,000 or more

  13. Q10

    To check that you are reading carefully, please select 'Not at all likely' for this question.

    Scale

    Scale 1-5: Not at all likely to Extremely likely

Sample plan

Draw a random sample from the account database of customers aged 18+ who have never opened a savings pot, stratified by the four age bands (18-34, 35-54, 55-74, 75+). Email a unique-link invitation with two reminders; terminate anyone failing S1-S3. Target about 700 completes with quotas of 175 per age band. IMPORTANT PRECISION NOTE: 700 completes cannot meet ±5 percentage points within every age band. At 175 per band, the 95% margin is about ±7.4pp for p=0.5. To meet ±5pp per band requires about 385 per band, i.e. about 1,540 completes. At an assumed 10-12% response, invite roughly 6,000-7,000 customers for 700 completes, or roughly 13,000-15,500 for 1,540 completes. If the 700 budget is fixed, report per-band precision as ±7-8pp and only the overall estimate as about ±3.7pp; do not claim ±5pp by age. Weight overall estimates to the eligible population age profile; keep within-band estimates unweighted.