Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Particle-1

Measured by Spring Prompt

Particle-1: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
87 of 100
Analysis rating
1,193
Head to head, this task
won 6 of 14

Why it is not sound

The survey

  • A research objective no question serves

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • Awareness, not rate, is the largest addressable barrier among non-users: 60.6% did not know pots existed, equivalent to about 239,800 of 361,000 weighted non-pot customers; in open text 126 of 230 spontaneous answers asked what pots are or said nobody had told them.The headline pairs the unweighted 60.6% unaware figure with a weighted population count of about 239,800. That count corresponds to approximately 66.4%, not 60.6%. The evidence section distinguishes these, but the headline's equivalence is incorrect.
  • The CFO's £14,500 average overstates the addressable balance base: it is an unweighted mean among 413 reporters with median £1k-<£5k, and the customer-weighted external savings pool is closer to £2.5bn. Even under generous intent assumptions, rate-rise balances are an upper bound of roughly £0.7-0.9bn and extra interest cost about £5-7m/year at +0.75pp.Applying the overall stated-likelihood percentage to estimated external savings does not establish an upper bound on deposits. The tables do not show how likelihood relates to balances or what proportion of those balances customers would transfer. Consequently, the £0.7–0.9bn deposit range and associated £5–7m interest-cost estimate are not supported as bounds, even with the stated caveats.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,845239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.3%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓126126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4646
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 8 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Do both, but not as a blanket rate rise based on the Q3 result alone. Launch the awareness/onboarding and payday set-aside push now, including a clear in-app explainer and simplified/setup findability fixes; run the 3.85% rate as a controlled pilot crossed with and without the nudge. Treat 75+ as a small targeted test, not the main campaign.

What would change it

A controlled test showing that 3.85% alone produces substantially higher actual pot openings, funded balances and payback than the nudge/awareness route would justify moving faster to a full rate rise. Conversely, if the nudge/awareness route lifts actual openings at low cost or the rate pilot fails to fund enough balances, avoid the full rate rise. A targeted 75+ test showing strong incremental balances and payback could justify a dedicated older/high-balance campaign.

Findings

  1. Awareness, not rate, is the largest addressable barrier among non-users: 60.6% did not know pots existed, equivalent to about 239,800 of 361,000 weighted non-pot customers; in open text 126 of 230 spontaneous answers asked what pots are or said nobody had told them.

    High confidenceBase 700

    Q1: Yes 39.4%, No 50.7%, Not sure 9.9% (base 700). Weighted No/Not sure using customer age mix = 66.4% of 361,000 = about 239,845. Open-text awareness mentions = 126/230.

  2. The CFO's 'rate is the top reason' finding is conditional on already knowing about pots: it is the top reason among aware non-users, but it describes only a minority of all non-users.

    High confidenceBase 276

    Q3 asked only of 276 Q1=Yes respondents: 29.7% chose 'interest rate too low'. This is 11.7% of all 700 respondents and 8.8% weighted to the customer age mix. It cannot describe the 60.6% who were unaware.

  3. A payday set-aside nudge at the current rate is at least as promising overall as a rate rise alone on stated intent, and much stronger among unaware customers; rate rise matters more among those already aware.

    Medium confidenceBase 700

    Q6 nudge at 3.1%: 33.7% likely vs Q5 current 17.4% and Q7 3.85% 29.9% (base 700). Among Q1=No, Q6 likely 42.5% vs Q7 24.2%. Among aware, Q7 likely 37.0% vs Q5 18.1%. Q6 and Q7 were separate hypotheticals, not a combined test.

  4. A rate rise alone would not be a complete answer, but it has real conversion potential among aware non-users, supporting a controlled rate test alongside awareness/onboarding rather than a blanket rate rise.

    Medium confidenceBase 276

    Among Q1=Yes respondents (base 276), Q7 likely 37.0% vs Q5 18.1%, and 29.7% name rate as main reason at Q3. Overall Q7 likely rises only to 29.9% from 17.4% at Q5.

  5. The CFO's £14,500 average overstates the addressable balance base: it is an unweighted mean among 413 reporters with median £1k-<£5k, and the customer-weighted external savings pool is closer to £2.5bn. Even under generous intent assumptions, rate-rise balances are an upper bound of roughly £0.7-0.9bn and extra interest cost about £5-7m/year at +0.75pp.

    Low confidenceBase 413

    Q4: 59.0% have any external savings; 413 reported an amount; unweighted mean £14,513, median £1k-<£5k. Weighted age mix gives roughly £2.5bn across 361,000. Q7 29.9% likely applied to all external savings gives ~£0.7-0.9bn upper bound; +0.75pp on that is ~£5-7m/year. Stated intent will overstate actual transfers.

  6. The 75+ segment has high balances but is too small and uncertain to justify a main campaign built around it; it should be treated as a targeted segment test only.

    Low confidenceBase 20

    75+ is 2% of customers and only 20 survey respondents; Q4 shows 65% with £20k+ externally (13/20), but Q1 aware only 35%, Q7 likely only 25%, and Q3 base only 7.

  7. Open text identifies execution barriers that awareness/onboarding must solve: setup complexity and findability are mentioned far more often than withdrawal-speed worries, though both are present.

    Medium confidenceBase 230

    Q2 open text (230 answers): 39 respondents mention setup/findability/hidden menus/too many steps; 10 mention withdrawal speed or instant-access reassurance.

Caveats

  • Tables are unweighted and the achieved sample skews older than the customer base: 55-74 is 28.6% of completes vs 13% of the population, 18-34 is 31.1% vs 46%. Weighted estimates are used only where stated.
  • Only 700 of 11,600 invited completed (6.0%); non-response and self-selection may bias awareness, rate sensitivity and balance estimates. The 86.2% completion among starters is high, but starters are self-selected.
  • Q3 was asked only of the 276 respondents who already knew about pots. Its 'rate is top reason' result cannot be generalised to the majority who were unaware.
  • Q5-Q7 are stated likelihoods, not actual opening or funding behaviour; they are likely to overstate conversion. There is no cell testing nudge and rate rise together.
  • Q4 uses recalled bands and midpoint assumptions; the £14,513 mean is among 413 reporters, not all non-users, and the median is only £1k-<£5k. Use actual balance/transaction data before final economics.
  • The 75+ group is only 20 respondents (7 asked Q3), so its high-balance pattern is directionally interesting but statistically fragile.
  • Open text was answered by 230 of 700 and is illustrative, not a prevalence estimate; theme counts overlap where answers raise multiple issues.

Next steps

  • decisionLaunch in-app awareness, a simple pots explainer, and payday set-aside nudges; make pots visible from the home/accounts area and simplify setup.
  • experimentRun a controlled A/B/factorial test: 3.1% vs 3.85% crossed with and without payday nudge, measuring actual openings and funded balances.
  • researchConduct usability research on setup flow and withdrawal reassurance, then fix the largest drop-off points.
  • researchValidate external and transferable balances using account/transaction data or a targeted offer test rather than relying on Q4 bands.
  • experimentTest a targeted high-balance older/75+ outreach separately from the main campaign.

Open-text themes it coded

Unaware or wants an explanation/promotion of pots 126Rate or interest too low / needs to be more competitive 45Setup complexity, findability, menus or lack of obvious app entry point 39No spare money / affordability / bills and debt 23Happy with existing provider, habit, loyalty, or not interested in saving pots 28Withdrawal speed or instant-access reassurance 10

The survey it planned

3 screening questions and 9 questions, as the model wrote them.

  1. S1

    Are you a Pennywell customer?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    Have you ever opened a Pennywell savings pot?

    One answer
    • Yes
    • No
    • Not sure

    Continues if No

  3. S3

    What is your age?

    One answer
    • Under 18
    • 18-34
    • 35-54
    • 55-74
    • 75 or over

    Continues if 18-34; 35-54; 55-74; 75 or over

  4. Q1

    Before today, were you aware that Pennywell offers savings pots?

    One answer
    • Yes
    • No
    • Not sure
  5. Q2

    Which of these are reasons you have not opened a Pennywell savings pot? Please select all that apply.

    Any that apply
    • The interest rate is too low
    • I don't know enough about how savings pots work
    • I don't know how to set one up
    • I haven't got round to it
    • I don't have enough spare money to save
    • I prefer to keep my savings with another provider
    • I am worried about being able to access my money
    • Another reason
    • Prefer not to say

    Routing Shown only if Q1 is: Yes

  6. Q3

    Pennywell savings pots currently pay 3.1% AER. Which one of the following would make you most likely to open a savings pot?

    One answer
    • A higher interest rate than the current 3.1% AER
    • A clear in-app prompt explaining what savings pots are
    • A set-up nudge on payday
    • More information about how quickly I can access my money
    • None of these
    • Don't know
  7. Q4

    How important is the interest rate when you decide where to keep savings?

    Scale

    Scale 1-5: Not at all important to Extremely important

  8. Q5

    Pennywell savings pots currently pay 3.1% AER. If the rate were 3.85% AER, how likely would you be to open a savings pot?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  9. Q6

    Approximately how much do you currently hold in cash savings outside Pennywell? Please include current accounts, savings accounts, cash ISAs and Premium Bonds, but not investments or pensions.

    One answer
    • £0
    • £1-£499
    • £500-£1,999
    • £2,000-£4,999
    • £5,000-£9,999
    • £10,000-£19,999
    • £20,000-£49,999
    • £50,000 or more
    • Prefer not to say
  10. Q7

    How much of those savings outside Pennywell could you access within one working day without penalty?

    One answer
    • £0
    • £1-£499
    • £500-£1,999
    • £2,000-£4,999
    • £5,000-£9,999
    • £10,000-£19,999
    • £20,000 or more
    • Don't know
    • Prefer not to say
    Judges: assumptive

    Asks about 'those savings outside Pennywell' but has no display logic, so respondents who answered £0 or 'Prefer not to say' at Q6 still see it, which assumes they hold savings elsewhere.

    Assumes the respondent has savings outside Pennywell; it is asked even if the respondent selects £0 in Q6.

  11. Q8

    If you opened a Pennywell savings pot, how much would you be likely to move into it in the first month?

    One answer
    • £0
    • £1-£99
    • £100-£499
    • £500-£999
    • £1,000-£2,499
    • £2,500-£4,999
    • £5,000 or more
    • Don't know
    • Prefer not to say
  12. Q9

    This question checks that you are reading carefully. Please select 'Disagree'.

    One answer
    • Strongly agree
    • Agree
    • Neither agree nor disagree
    • Disagree
    • Strongly disagree
Sample plan

Invite a random sample from the Pennywell account database of customers aged 18+ who have never opened a savings pot. Verify eligibility in S1-S3. Precision requirement: ±5 percentage points at 95% confidence for each of four age bands requires about 385 completes per band at p=0.5, i.e. about 1,540 completes total. The requested ~700 completes cannot meet ±5pp per age band: 175 per band gives about ±7.4pp at p=0.5. Recommended plan: stratified random sample by age band with equal quotas of 385 completes per band (18-34, 35-54, 55-74, 75+), total ~1,540, with extra invitations for 75+ if response is lower. If the 700 limit is fixed, use 175 per band and report ±7.4pp per band; do not claim ±5pp. Use database age and pot status for stratification; weight overall results to the Pennywell customer age profile and report age-band results unweighted. Exclude attention-check failures from analysis and replace to quota if fieldwork allows.