Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Gemini 4 Hydrogen

Measured by Spring Prompt

Gemini 4 Hydrogen: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
74 of 100
Analysis rating
1,088
Head to head, this task
won 2 of 9

Why it is not sound

The analysis

  • Made a recommendation that does not follow from the data
  • Missed a trap: Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,702239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.2%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓126126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4646
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 7 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✗Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Findings

  1. Low awareness—not the 3.1% AER rate—is the primary barrier across non-pot customers; the CFO's impression that rate settles the decision rests on a Q3 routing artefact.

    High confidenceBase 700

    Only 39.4% unweighted (33.6% age-weighted) knew pots existed (Q1), leaving 66.4% age-weighted (~239,702 of 361,000 non-users) unaware. Q3 was asked only of the 276 aware respondents; while 29.7% (±5.4 pp) of that subgroup cited rate, that represents just 11.7% of the total sample (82/700) and only 8.8% once weighted to the population age mix. Age differences in citing rate at Q3 between 35-54 (27.5%, n=102) and 55-74 (37.5%, n=120) are not statistically significant (z=1.59, p=0.11).

  2. A payday set-up nudge at 3.1% AER outperforms a 3.85% AER rate rise overall and more than doubles stated intent among unaware and younger customers.

    Medium confidenceBase 700

    At 3.1% baseline (Q5), 17.4% are likely to start saving. A payday nudge at 3.1% (Q6) raises stated likelihood to 33.7% (+16.3 pp overall), driven by Q1 'No' (42.5% vs 17.2% at Q5), Q1 'Not sure' (47.8% vs 15.9%), and 18-34s (38.5% vs 17.9%). Raising the rate to 3.85% (Q7) reaches 29.9% overall (+12.5 pp) and helps mainly already-aware users (37.0% vs 18.8% on Q6) and 55-74s (33.5% vs 30.0%), while moving Q1 'No' only to 24.2%.

  3. Even when customers know about or look for pots, poor app findability and a multi-step setup flow cause abandonment, and some doubt withdrawal speed.

    High confidenceBase 230

    In Q2 open-text (n=230), 126 respondents (54.8%, including 46 aged 18-34) asked to be told what pots are and how they work, 40 (17.4%, across both aware and unaware segments) said pots are buried in menus or the setup process is confusing/fiddly, and 10 (4.3%, alongside 7.6% of Q3 respondents) worried about instant access at weekends or emergencies.

  4. We cannot forecast deposit inflows from the £14,513 average external balance; balances are skewed, overstated by sample age mix, and cannot be converted from stated intent alone.

    High confidenceBase 700

    The £14,513 mean excludes the 35.7% holding 'None' (n=250) and 5.3% DK/PNS (n=37); including 'None' lowers the sample mean to ~£9,041, and among the 413 savers elsewhere the median lies in the £1,000-£4,999 band with 55.2% holding under £5,000. Unweighted data also over-represents 55-74s (28.6% vs 13% population) and under-represents 18-34s (31.1% vs 46%, where 45.9% hold None). Q4 includes Cash ISAs (tax-wrapped) while pots are taxable, and 19.6% of aware non-users (plus 17 Q2 comments) cite loyalty/habit with legacy banks.

  5. Do not build a savings campaign around customers aged 75 or over: the segment is tiny, under-sampled, and the least responsive to either the nudge or the 3.85% rate.

    Low confidenceBase 20

    Although 65.0% of the 75+ sample report £20,000+ elsewhere, the base is only n=20 (and n=7 at Q3), giving margins of error around ±20 pp. In the 361,000 non-pot population, 75+ accounts for just 2% (~7,220 customers) compared with 85% aged 18-54 (~306,850) and 13% aged 55-74 (~46,930). Moreover, 75+ shows the lowest Q6/Q7 intent (25.0% likely for both vs 20.0% baseline), and 42.9% of Q3 75+ respondents keep savings elsewhere.

  6. Around a sixth of open-text respondents face hard affordability constraints or disinterest that neither messaging nor a 3.85% rate will overcome.

    Medium confidenceBase 230

    Among all respondents, 35.7% hold no savings elsewhere (45.9% of 18-34s) and 13.4% of Q3 respondents say they have no money left over to save; in Q2, 23 respondents cite zero spare money (bills, rent, debt, low income) and 15 say they are simply not interested in saving in pots.

Open-text themes it coded

Unaware of savings pots and requesting an explanation 126Interest rate perceived as too low or uncompetitive 19

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Which age group are you in?

    One answer
    • Under 18
    • 18-34
    • 35-54
    • 55-74
    • 75 or over

    Continues if 18-34; 35-54; 55-74; 75 or over

  2. S2

    Do you currently have an active Pennywell current account?

    One answer
    • Yes, it is my main bank account
    • Yes, I use it regularly alongside another bank account
    • Yes, I have one, though I rarely use it
    • No, I no longer have a Pennywell account
    • Not sure

    Continues if Yes, it is my main bank account; Yes, I use it regularly alongside another bank account; Yes, I have one, though I rarely use it

  3. Q1

    Before taking this survey, did you know that Pennywell offers instant-access savings pots inside the app?

    One answer
    • Yes, I knew about them and have looked at them in the app
    • Yes, I knew they existed, though I have not looked at them
    • No, I did not know Pennywell offered savings pots
    • Not sure
    • I already have (or have had) a Pennywell savings pot
  4. Q2

    Why have you not opened a Pennywell savings pot? (Select all that apply)

    Any that apply
    • I have not got around to it or have not thought about it
    • I already keep my savings with another bank or building society
    • The interest rate (3.1% AER) is not high enough compared with other accounts
    • I did not know what interest rate it pays or how it works
    • I do not have spare money to put into savings at the moment
    • I prefer keeping my savings separate from my everyday spending app
    • Setting one up looked too complicated or time-consuming
    • I prefer fixed-term savings or investments rather than instant access
    • Other reason

    Routing Shown only if Q1 is: Yes, I knew about them and have looked at them in the app; Yes, I knew they existed, though I have not looked at them

  5. Q3

    If you selected more than one reason above, which ONE is the main reason you have not opened a Pennywell savings pot? (If you only selected one, please confirm it below.)

    One answer
    • I have not got around to it or have not thought about it
    • I already keep my savings with another bank or building society
    • The interest rate (3.1% AER) is not high enough compared with other accounts
    • I did not know what interest rate it pays or how it works
    • I do not have spare money to put into savings at the moment
    • I prefer keeping my savings separate from my everyday spending app
    • Setting one up looked too complicated or time-consuming
    • I prefer fixed-term savings or investments rather than instant access
    • Other reason

    Routing Shown only if Q1 is: Yes, I knew about them and have looked at them in the app; Yes, I knew they existed, though I have not looked at them

  6. Q4

    Pennywell savings pots are instant-access savings accounts inside the app that currently pay 3.1% AER. Thinking about this, how likely would you be to open a Pennywell savings pot in the next 3 months?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  7. Q5

    Still at the current rate of 3.1% AER, how likely would you be to open a Pennywell savings pot if you saw a prompt in the app on payday that let you set one up in a couple of taps?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  8. Q6

    If Pennywell increased the savings pot interest rate from 3.1% AER to 3.85% AER (with the current in-app set-up, without a payday prompt), how likely would you be to open one in the next 3 months?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  9. Q7

    To help us check data quality, please select 'Mostly agree' from the options below.

    One answer
    • Strongly disagree
    • Mostly disagree
    • Neither agree nor disagree
    • Mostly agree
    • Strongly agree
  10. Q8

    If you were considering opening a Pennywell savings pot, which ONE of the following would make the biggest difference to you?

    One answer
    • Increasing the interest rate from 3.1% AER to 3.85% AER
    • An in-app prompt on payday to set up a pot in a couple of taps
    • Clearer information in the app about how savings pots work
    • None of these would make me open a Pennywell savings pot
  11. Q9

    Roughly how much do you currently hold in savings accounts outside of Pennywell? (Please include instant-access, notice, or Cash ISA savings, and exclude pensions, property, and stocks or shares investments.)

    One answer
    • £0 (I do not currently have savings outside Pennywell)
    • £1 to £999
    • £1,000 to £4,999
    • £5,000 to £14,999
    • £15,000 to £49,999
    • £50,000 or more
    • Prefer not to say
  12. Q10

    If Pennywell offered a savings pot that suited your needs, roughly how much of those outside savings would you realistically consider moving into it?

    One answer
    • £0 (I would keep existing savings where they are and only add new money)
    • £1 to £999
    • £1,000 to £4,999
    • £5,000 to £14,999
    • £15,000 or more
    • Not sure
    • Prefer not to say

    Routing Shown only if Q9 is: £1 to £999; £1,000 to £4,999; £5,000 to £14,999; £15,000 to £49,999; £50,000 or more

    Judges: assumptive

    The £0 option assumes respondents would add new money. Someone who would neither move existing savings nor add new money has no accurately worded £0 response.

    Assumes the respondent does not already have a savings pot.

Sample plan

Target population: Pennywell adult customers (aged 18+) who have never opened a savings pot (~361,200 customers, 86% of the 420,000 customer base). Sampling method: Stratified random sample drawn from the Pennywell account database across the four required age bands (18-34, 35-54, 55-74, 75 or over), invited by email with one reminder after 3-5 days. Precision and sample size trade-off: Achieving +/- 5 percentage points at 95% confidence within each of the four age bands requires n=385 completes per band (about 1,540 completes total, subject to finite population correction if the 75 or over customer base in the database is small). Under the current 700-complete constraint, set equal quotas of 175 completes per age band (18-34, 35-54, 55-74, 75 or over), yielding +/- 7.4 percentage points margin of error per age band at 95% confidence and +/- 3.7 percentage points overall; recommend expanding to ~1,540 completes if +/- 5 pp per band is mandatory for the board. Assuming a 5-8% email completion rate (and lower base size or response rate in 75+), send invitations in waves starting with ~10,000-14,000 emails (or ~25,000 if targeting 1,540 completes) and dynamically top up slower-filling age quotas. Apply post-stratification weights by the true database age distribution when reporting overall totals (O1-O3).