Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / GPT-6 Luna

Measured by Spring Prompt

GPT-6 Luna: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict: ✗ Not sound
Research score: 82 of 100
Analysis rating: 1,102
Head to head, this task: won 4 of 9

Why it is not sound

The survey

A research objective no question serves

The analysis

Got at least one set number wrong
Reported a material finding the data does not support

Findings the judges found unsupported

There is no statistically significant evidence here that 55–74-year-olds are more likely than 35–54-year-olds to cite rate as their main reason. Treat the apparent difference as a lead for further testing, not a proven segment distinction.The stated ±2.8 percentage-point margin of error for the overall Q3 rate share is incorrect. For 82 of 276 respondents, the simple-random-sample approximate 95% margin is ±5.4 percentage points. The conclusion that the difference between the two age groups is not significant at 95% is supported.

Scorecard

Numbers 10 of 12 right

Question		Its answer	True
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?	✓	39.4%	39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?	✓	29.7%	29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?	✓	11.7%	11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?	✓	8.8%	8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?	✓	239,845	239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?	✓	55.2%	55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?	✓	29.9%	29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?	✗	2.8 points	5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?	✓	128	126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?	✗	41	46
What percentage of the customers invited completed the survey?	✓	6%	6%

Traps in the data 8 of 8 handled

✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Prioritise an awareness and onboarding push, including a clear in-app explanation, visible route to create a pot, simpler setup guidance, and a carefully designed payday nudge tested against a control. Do not raise the rate to 3.85% solely because rate was the most common Q3 answer or because respondents stated higher intent in Q7. Treat a rate increase as a separate commercial decision: model its cost and test whether it produces incremental funded balances. Do not target customers aged 75+ alone; include a broader age mix, with particular attention to testing 55–74 customers.

What would change it

I would be more supportive of moving the rate to 3.85% if a controlled test or credible behavioural evidence showed a material, sustained increase in net new funded balances—especially balances transferred from elsewhere—and the incremental margin after interest cost, retention effects, and operational costs met Pennywell's hurdle. I would change the campaign targeting if larger, representative research or live campaign results showed a clearly stronger profitable response among 75+ customers than among other age groups.

Findings

Awareness and onboarding are the clearest immediate opportunity. Only 39.4% knew about pots before the survey; age-weighted to the customer population, an estimated 66.4%—about 240,000 never-pot customers—did not know or were unsure.

Medium confidenceBase 700

Q1: 276 of 700 respondents said Yes; age-weighted No/Not sure is 66.4% using population age shares. The open-text coding also found 128 of 230 answers asking for information or saying they did not know what pots were.
The rate is a meaningful barrier for customers already aware of pots, but it does not settle the decision to raise it. It was the most common single reason in Q3, yet that question was only asked of aware respondents; the reason represents 11.7% of all respondents and an age-weighted 8.8% of the never-pot customer population.

Medium confidenceBase 276

82 of 276 aware respondents chose 'The interest rate is too low' (29.7%); 82 of all 700 is 11.7%. Age-weighting the age-specific joint shares of awareness and this Q3 answer gives 8.8%.
A payday set-aside prompt merits a controlled test before a broad rollout. Stated likelihood was higher for the prompt at the current rate than for either the current-rate question or the 3.85% rate scenario, but these are hypothetical, sequential survey questions—not evidence of incremental deposits.

Medium confidenceBase 700

Likely (4–5): 33.7% for the payday prompt at 3.1% (Q6), 17.4% at 3.1% without the prompt (Q5), and 29.9% for 3.85% (Q7); each base is 700.
Do not build the campaign around customers aged 75 or over on these results alone. They report substantial savings elsewhere, but the subgroup is very small and their stated likelihood is not notably higher; the much larger 55–74 group is also worth including in tests.

Low confidenceBase 20

Among 20 respondents aged 75+, 85% reported savings elsewhere and 65% reported £20,000 or more; likely intent was 20% at 3.1%, 25% with the prompt, and 25% at 3.85%. Among 200 aged 55–74, 66.5% reported savings elsewhere and 21.5% reported £20,000 or more.
The survey cannot establish how much money a rate rise would bring into Pennywell. The reported average is not an estimate of transferable balances or incremental inflows, so it should not be used as a business case for the rate change.

High confidenceBase 413

The £14,513 mean is based on 413 respondents who reported an amount, estimated from band midpoints and treating £50,000+ as £75,000. The median is in the £1,000–under £5,000 band; the survey did not measure what respondents would actually transfer.
There is no statistically significant evidence here that 55–74-year-olds are more likely than 35–54-year-olds to cite rate as their main reason. Treat the apparent difference as a lead for further testing, not a proven segment distinction.

Medium confidenceBase 222

Q3 rate reason: 37.5% of 55–74 respondents (45/120) versus 27.5% of 35–54 respondents (28/102); the difference is not significant at 95%. The overall Q3 rate share has an approximate 95% margin of error of ±2.8 percentage points.

Caveats

Only 700 of 11,600 invited customers completed the survey (6.0%). Random selection from the database does not remove the risk that people who chose to respond differ from nonrespondents.
The tables are unweighted, and the achieved age mix differs from the customer population. Age-weighted estimates use the supplied population age shares, but cannot correct nonresponse differences within age groups.
Q3 was asked only of the 276 respondents who already knew about pots. Its 29.7% rate response is conditional on awareness; it is not the share of all customers naming rate as a barrier.
Q5–Q7 are stated intentions asked sequentially, not randomized choices or observed behaviour. Differences between them do not prove that a prompt or a rate change causes new accounts or balances.
Q4 is self-reported in broad bands. The £14,513 mean uses band-midpoint assumptions and a £75,000 value for the open-ended top band; it is based only on 413 respondents reporting an amount and says nothing about how much they would transfer.
The 75+ subgroup has only 20 respondents, and the 75+ Q3 base is only 7. Its percentages are highly unstable and should not support a targeted campaign decision without more evidence.
The 95% margin of error is a simple-random-sample approximation for the Q3 base; it does not account for nonresponse or other sources of survey error.
The 128 open-text awareness mentions are manual thematic coding of 230 answers. They illustrate issues but are not a population estimate.

Next steps

decisionApprove an awareness and onboarding pilot with a clear in-app explanation, prominent entry point, guided setup, and transparent withdrawal-time information.
experimentRandomize eligible never-pot customers to a payday set-aside prompt, an awareness-only message, and a holdout.
researchBuild a unit-economics model for the 3.85% rate before committing to a rate change.
experimentIf commercially feasible, test rate messaging or a rate offer in a controlled, legally and operationally appropriate pilot.
monitorMonitor the pilot by age group and gather more evidence before any 75+-specific campaign.

Open-text themes it coded

Unaware of pots, unsure what they are, or asking Pennywell to explain or announce them 127Among 18–34-year-olds: unaware of pots, unsure what they are, or asking Pennywell to explain or announce them 41A higher or more competitive interest rate 44App discoverability, simpler setup, or step-by-step guidance 40Reassurance about how quickly withdrawals reach the current account 10No spare money to save or current financial constraints 23Not interested in saving through pots or no inducement would change their mind 15Habit, trust, or existing savings held with another provider 19

The survey it planned

2 screening questions and 8 questions, as the model wrote them.

S1
Which age group are you in?
One answer
- Under 18
- 18–34
- 35–54
- 55–74
- 75 or over
- Prefer not to say
Continues if 18–34; 35–54; 55–74; 75 or over
S2
Have you ever opened a Pennywell savings pot?
One answer
- Yes
- No
- Not sure
Continues if No
Q1
Before today, were you aware that Pennywell offers savings pots—instant-access savings accounts within the app?
One answer
- Yes
- No
- Not sure
Q2
Before today, what were the reasons you had not opened a Pennywell savings pot? Select all that apply.
Any that apply
- The interest rate did not seem attractive
- I did not know how to find or open a pot
- I have not had savings to put aside
- I prefer to keep my savings with another provider
- I do not understand how pots work
- I have concerns about access to my money or withdrawals
- I do not need a separate place for savings
- I intended to open one but did not get around to it
- I have not had a particular reason
- Other reason
Routing Shown only if Q1 is: Yes
Q3
Roughly how much do you currently hold in savings accounts with providers other than Pennywell? Please exclude current-account balances, cash kept at home, pensions and investments.
One answer
- None
- Less than £1,000
- £1,000 to £4,999
- £5,000 to £9,999
- £10,000 to £24,999
- £25,000 to £49,999
- £50,000 or more
- Not sure
- Prefer not to say
Q4
Imagine Pennywell kept the savings pot rate at 3.1% AER and sent you an in-app message explaining pots, together with a payday reminder and a direct link to set one up. How likely would you be to open a pot in the next three months?
Scale

Scale 1-11: 0 — Not at all likely to 10 — Extremely likely
Q5
Imagine Pennywell raised the savings pot rate to 3.85% AER, but sent no new in-app messages or payday set-up reminder. How likely would you be to open a pot in the next three months?
Scale

Scale 1-11: 0 — Not at all likely to 10 — Extremely likely
Q6
Imagine Pennywell raised the savings pot rate to 3.85% AER and sent you an in-app message explaining pots, together with a payday reminder and a direct link to set one up. How likely would you be to open a pot in the next three months?
Scale

Scale 1-11: 0 — Not at all likely to 10 — Extremely likely
Q7
Which of the following, if any, would be most likely to make you open a Pennywell savings pot? Select one.
One answer
- A clear in-app explanation that savings pots exist
- A payday reminder with a direct link to make setting up a pot easier
- Raising the savings pot rate to 3.85% AER
- An in-app explanation and payday set-up reminder, without changing the rate
- Raising the rate to 3.85% AER as well as an in-app explanation and payday set-up reminder
- None of these would make me open a pot
- Not sure
Q8
If you opened a Pennywell savings pot, what share of the savings you hold in accounts with other providers would you consider moving into it?
One answer
- None
- Less than 25%
- 25% to 49%
- 50% to 74%
- 75% to 100%
- Not sure
Routing Shown only if Q3 is: Less than £1,000; £1,000 to £4,999; £5,000 to £9,999; £10,000 to £24,999; £25,000 to £49,999; £50,000 or more

Sample plan

Invite a random sample of customers recorded in the account database as never having opened a pot, stratified by age band. Aim for about 700 completed surveys, with age quotas of approximately 175 in each band (18–34, 35–54, 55–74, 75 or over); set invitation numbers by band using expected response rates and invite more customers in bands likely to respond less. Weight the combined results back to the age distribution of eligible Pennywell customers, and report unweighted age-band results alongside weighted overall results. The requested precision is not achievable with about 700 completes: 175 per band gives a worst-case 95% margin of error of about ±7.4 percentage points. To achieve approximately ±5 points in every age band requires about 385 completes per band, or 1,540 total, before allowing for weighting or other design effects. If ±5-point precision for each band is mandatory, the completes target must therefore increase. Randomise the order of Q4–Q6 for each respondent to reduce order effects.