Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Muse Ember

Measured by Spring Prompt

Muse Ember: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict: ✗ Not sound
Research score: 94 of 100
Analysis rating: 1,392
Head to head, this task: won 8 of 13

Why it is not sound

The survey

A research objective no question serves

Scorecard

Numbers 12 of 12 right

Question		Its answer	True
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?	✓	39.4%	39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?	✓	29.7%	29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?	✓	11.7%	11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?	✓	8.8%	8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?	✓	239,700	239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?	✓	55.2%	55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?	✓	29.9%	29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?	✓	5.4 points	5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?	✓	126	126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?	✓	46	46
What percentage of the customers invited completed the survey?	✓	6%	6%

Traps in the data 8 of 8 handled

✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Do not raise the rate to 3.85% as the primary move now. Run the awareness and onboarding push first at 3.1%: plain-English in-app explanation of what pots are, a prominent home-screen entry point, a simplified set-up flow with withdrawal-speed reassurance, and a payday set-aside nudge, aimed at the weighted 66% who do not know about pots and especially at 18-34s. In parallel, run a randomised test of nudge versus rate rise (ideally 2x2) measuring actual pot openings and balances transferred, segmented by awareness and existing savings elsewhere, before committing Pennywell to a permanent 0.75pp rate increase. If a rate test is run, prioritise knowers who already hold savings elsewhere, the only group where a rate message showed intent above the nudge.

What would change it

Evidence from a randomised behavioural test that a rise to 3.85% produces incremental actual openings and transferred balances, concentrated in customers with material external balances, at a cost per pound saved that the COO/CFO judge profitable after the 0.75pp interest cost, and that this incremental effect exceeds the nudge's; or evidence that the nudge/explanation does not convert non-knowers in practice. Also, a properly boosted sample showing a large, profitable 75+ opportunity would change targeting, but on current n=20 it does not.

Findings

The rate is not the top barrier in the whole non-user population. It is the top stated reason only among the minority who already knew about pots, so it does not settle the case for a rate rise.

Medium confidenceBase 276

Q1 Yes 39.4% of 700, No 50.7%, Not sure 9.9%. Q3 was asked only of the 276 knowers: 29.7% chose rate too low (82/276, 95% margin 5.4pp). That is 11.7% of all 700, and 8.8% weighted to the population age mix (18-34 46%, 35-54 39%, 55-74 13%, 75+ 2%). Weighted unawareness is 66.4%, about 239,700 of 361,000 non-users.
At the current 3.1% rate, an in-app payday set-aside nudge beats a rise to 3.85% on stated intent overall, with opposite effects for knowers and non-knowers. Neither converts a majority.

Medium confidenceBase 700

Q5 at 3.1% likely 17.4% (700). Q6 nudge at 3.1% likely 33.7%; Q7 rise to 3.85% likely 29.9%. Non-knowers (Q1 No, n=355): nudge 42.5% vs rise 24.2%. Knowers (n=276): nudge 18.8% vs rise 37.0%. Not sure (n=69): nudge 47.8% vs rise 30.4%.
The £14,500 average savings elsewhere cannot be used to estimate rate-rise inflow. Most savers hold small amounts, over a third hold none, and the mean is driven by a top-coded tail.

Medium confidenceBase 413

Q4 (700): 35.7% none, 59.0% any elsewhere. Among 413 reporting an amount, 55.2% hold under £5,000 (228/413), median band £1,000-£5,000, mean £14,513 from band midpoints counting £50,000+ as £75,000. Only 12.6% of the sample, about 9.4% age-weighted, hold £20,000+. No question asked amount that would move at 3.85%.
Do not build the campaign around customers aged 75+. They are a tiny population segment with a tiny survey base, and their stated intent is not the highest.

Low confidenceBase 20

75+ is 20 respondents (2.9% of sample, 2% of population, about 7,220 of 361,000), 13/20 reporting £20,000+ elsewhere, Q3 base only 7. Q6 and Q7 likely both 25.0% (5/20), below the total. Any 75+ estimate has a margin of roughly ±20pp.
Unawareness and poor findability/setup, not price alone, are the dominant open-text barriers, especially for younger customers.

Medium confidenceBase 230

126 of 230 open-text answers say they did not know about pots/what they are or want Pennywell to tell them, including 46 from 18-34 respondents. A further 40 answers cite difficulty finding pots or fiddly/confusing set-up, including knowers who tried and abandoned set-up. 230 is a self-selected subset of 700, so this shows salience, not population prevalence.
Age differences in citing the rate should not be targeted on this evidence. The 35-54 vs 55-74 difference is not statistically significant at 95%.

Medium confidenceBase 222

Among Q3 respondents, rate too low: 35-54 27.5% (28/102), 55-74 37.5% (45/120), difference about 10pp, z about 1.6, not significant at 95%. The sample is also age-skewed vs population: 18-34 31.1% vs 46% population, 55-74 28.6% vs 13%, so unweighted totals overstate older, more rate-sensitive and higher-balance behaviour.

Caveats

Only 700 of 11,600 invited completed (6.0%); 812 started. Non-respondents are uncharacterised beyond age weighting, so population inference may be biased in unknown directions.
Sample age mix differs materially from population (18-34 31.1% vs 46%, 55-74 28.6% vs 13%); weighted estimates are model-dependent and still rest on small older bases.
Q3 was asked only of knowers (276), so its percentages cannot be read as population shares without re-basing and weighting.
Q5-Q7 are stated likelihoods to hypothetical prompts, not observed openings or transfers; they do not measure amount, persistence, or profitability.
Q4 balances are banded and self-reported; the £14,513 mean uses midpoints and counts £50,000+ as £75,000, and is sensitive to that top-coding.
75+ evidence rests on 20 respondents in total and 7 for Q3; no reliable segment inference is possible.
Open text is 230 self-selected answers from 700; theme counts show what respondents volunteered, not prevalence in the population. Coding of awareness, findability and setup borders involves judgement.
No data on competitor rates, cost of raising the rate by 0.75pp, cannibalisation, or margin per balance, all needed to judge a rate rise.

Next steps

decisionApprove and launch awareness/onboarding push plus payday nudge at 3.1%, holding the rate pending test results
experimentRandomised experiment: nudge x rate (3.1% vs 3.85%) with actual opening and balance-transfer outcomes
experimentFix findability and set-up friction: home-screen button, fewer steps, plain-English withdrawal timing
researchResearch amount that would move and competitor rate benchmark, with boosted non-respondent and older-customer samples if targeting is considered
monitorMonitor cost per acquired pot, balance per pot, and margin at 3.1% vs 3.85% by age and prior awareness

Open-text themes it coded

Did not know about pots / what they are, or want Pennywell to tell them 126Rate too low / want a higher rate 46No spare money / nothing left to save 23Hard to find pots or set-up is fiddly/confusing 40Worried about how quickly money can be withdrawn 10Loyal to savings elsewhere / would not move savings 18Not interested in saving this way / nothing would change their mind 15

The survey it planned

2 screening questions and 9 questions, as the model wrote them.

S1
Are you a Pennywell current account customer?
One answer
- Yes
- No
Continues if Yes
S2
What is your age?
One answer
- 18-34
- 35-54
- 55-74
- 75 or over
- Under 18
Continues if 18-34; 35-54; 55-74; 75 or over
Q1
Before starting this survey, had you heard of savings pots in the Pennywell app?
One answer
- Yes, I had heard of them
- No, I had not heard of them
- Not sure
Q2
Why have you not opened a Pennywell savings pot? Please select all that apply.
Any that apply
- I did not know how to open one
- I did not think the interest rate was good enough
- I do not need to save at the moment
- I prefer to keep my savings elsewhere
- I have been meaning to open one but have not got round to it
- I am not interested in saving inside the Pennywell app
- Something else
Routing Shown only if Q1 is: Yes, I had heard of them
Q3
Savings pots are instant-access savings accounts inside the Pennywell app, currently paying 3.1% AER. If you were shown how to set one up in the app and reminded on payday, how likely would you be to open one in the next month at 3.1% AER?
Scale

Scale 1-5: Extremely unlikely to Extremely likely
Q4
If savings pots paid 3.85% AER instead of 3.1% AER, and you were shown how to set one up in the app and reminded on payday, how likely would you be to open one in the next month?
Scale

Scale 1-5: Extremely unlikely to Extremely likely
Q5
Please rank the following in order of importance when deciding whether to open a savings pot, from most important (1) to least important (5).
Ranking
- The interest rate
- How easy it is to set up
- A reminder on payday
- Having a specific savings goal
- Keeping savings in one app
Q6
Approximately how much do you currently hold in savings outside Pennywell, for example with other banks, building societies, or in cash ISAs?
One answer
- £0
- £1 to £999
- £1,000 to £4,999
- £5,000 to £9,999
- £10,000 to £24,999
- £25,000 or more
- Prefer not to say
Q7
Where do you hold savings outside Pennywell? Please select all that apply.
Any that apply
- Instant-access account
- Cash ISA
- Fixed-term or notice account
- Other
Routing Shown only if Q6 is: £1 to £999; £1,000 to £4,999; £5,000 to £9,999; £10,000 to £24,999; £25,000 or more
Q8
Have you ever opened a Pennywell savings pot?
One answer
- Yes
- No
Continues if No
Q9
This is a quality-control question. Please select 'Yes' to show you are reading carefully.
One answer
- Yes
- No

Sample plan

Invite by email a stratified random sample from the account database frame of customers who have never opened a pot (86% of 420,000 = 361,200), stratified by database age band with equal allocation and disproportionate oversampling of the smaller 55-74 and 75 or over bands, inviting in waves with reminders until each quota fills; pot status is taken from the database and only re-checked at Q8 after awareness is measured, so the screener does not prime Q1. Target 1,540 completes (385 per age band: 18-34, 35-54, 55-74, 75 or over), which gives plus or minus 5 percentage points at 95% confidence per band at p=0.5 (1.96 x sqrt(0.25/385) = 0.05). The brief's 700 completes cannot meet this: 175 per band gives plus or minus 7.4 points, so the board must fund 1,540 or accept wider precision; at an assumed 15% response, invite about 10,300 initially with top-ups for under-filled bands. Weight overall estimates back to the frame age distribution; report unweighted per-band results; exclude Q8 'Yes' and Q9 attention failures.