Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / GPT-6.2 Sol
GPT-6.2 Sol: Why Pennywell customers don't use savings pots
The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 90 of 100
- Analysis rating
- 1,354
- Head to head, this task
- won 8 of 11
Why it is not sound
The survey
- No screening question
- Answer bands that overlap or leave gaps
The analysis
- Got at least one set number wrong
Scorecard
Numbers 11 of 12 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)? | ✓ | 39.4% | 39.4% |
| Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'? | ✓ | 29.7% | 29.7% |
| As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3? | ✓ | 11.7% | 11.7% |
| Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3? | ✓ | 8.77% | 8.8% |
| Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)? | ✓ | 239,302 | 239,702 |
| Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000? | ✓ | 55.2% | 55.2% |
| What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER? | ✓ | 29.9% | 29.9% |
| Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'? | ✓ | 5.4 points | 5.39 points |
| How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots? | ✓ | 126 | 126 |
| How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots? | ✗ | 43 | 46 |
| What percentage of the customers invited completed the survey? | ✓ | 6.03% | 6% |
Traps in the data 8 of 8 handled
- ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
- ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
- ✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
- ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
- ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
- ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
- ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
- ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.
The analysis
Approve a controlled awareness and onboarding pilot at 3.1% AER: use a prominent in-app explanation, a direct setup link, clear withdrawal information and an optional, dismissible payday nudge. Keep it accessible across ages rather than building the campaign around customers aged 75 or over. Do not commit to a bank-wide increase now. Test 3.85% separately and in combination with onboarding, and use retained external funding and full repricing costs to decide whether to roll out either intervention or both.
Roll out 3.85% if a controlled live test shows that it brings in enough incremental, retained external funding to clear Finance's hurdle after interest on existing eligible balances and other costs. Choose both only if the combined offer adds enough value over the better single option to justify its extra cost. Scale onboarding alone if it lifts retained funded use and the rate does not pay back. Revisit a 75+ campaign only if a larger study or live test demonstrates meaningful incremental response, not just larger balances.
Findings
-
Start with an awareness and onboarding pilot at 3.1% AER. The survey gives a stronger case for testing that first than for raising the rate across the board.
Only 276 of 700 respondents knew pots existed. After age weighting, an estimated 66.4% of never-users did not know or were unsure, equivalent to about 239,300 customers if respondents represent their age bands. Stated likelihood was 17.4% at 3.1%, 33.7% with a pay-in nudge at 3.1%, and 29.9% at 3.85%. The age-weighted figures were 17.4%, 35.1% and 28.9%, respectively. These are signals for what to test, not predicted conversion rates.
-
The rate being the top reason does not settle the pricing decision. It makes 3.85% worth testing, particularly as a way to reach customers who already know about pots.
Rate was the most common single answer among aware respondents, but only 82 of 276 chose it (29.7%): 11.7% of all 700 completes and 8.77% after age weighting. Among the 276 aware respondents, 37.0% said they were likely to start at 3.85%, versus 18.1% at baseline and 18.8% with the nudge. Among the 355 who said they did not know about pots, the nudge drew 42.5% likely versus 24.2% for the higher rate. The Q3 difference between ages 35–54 and 55–74 (28/102 versus 45/120) was not significant in a two-sided test at the 95% level (p ≈ 0.11).
-
Do not use the £14,500 figure to forecast how much a rate rise would bring in. The survey cannot estimate incremental deposits.
The £14,513 mean is a band-midpoint estimate for the 413 respondents reporting positive external savings, not a mean for all never-users. It assumes £75,000 for the open-ended top band. Among those 413, 228 (55.2%) reported less than £5,000; across all 700, 250 (35.7%) reported none. Q7 measures intention to start saving, not whether money would come from another provider, how much would move, or how long it would stay. For scale only, if the rate caused 1% of the 361,000 never-users to bring in and retain £1,000 each that they otherwise would not have brought in, that would be £3.61m. Neither assumption is estimated by this survey.
-
Do not build the campaign around customers aged 75 or over. Include them in accessible onboarding and gather more evidence before targeting them on the strength of their balances.
13 of 20 respondents aged 75 or over reported at least £20,000 elsewhere, but the group is only 2% of never-users, only seven answered Q3, and just five said they were likely to start in each of the nudge and higher-rate scenarios. External balances are not evidence of willingness to switch. By contrast, ages 18–34 account for 46% of never-users; 171 of 218 in that band did not know about pots or were unsure, and 38.5% reported likely use with the nudge.
-
Make the prompt lead to a clear, easy setup rather than simply announcing that pots exist.
126 of 230 open-text answers mentioned not knowing about pots or wanting Pennywell to explain them, including 43 answers from ages 18–34. Other answers described buried navigation, abandoned setup and uncertainty about withdrawal timing. Some respondents said they had no spare cash or preferred to leave savings with a provider they trusted, so neither clearer onboarding nor a higher rate will remove every barrier.
Caveats
- The invitation sample was random, but only 700 of 11,600 invitees completed. Ages 55–74 made up 28.6% of completes versus 13% of the target population; ages 18–34 made up 31.1% versus 46%. Age weighting corrects this mix, not differences between respondents and nonrespondents within age bands. The awareness count is an extrapolation, not a database count.
- Q3 was a forced single choice shown only to respondents who already knew about pots. The 8.77% weighted figure is the joint share who were aware and selected rate; it is not the share whose behaviour might respond to a higher rate. The ±5.4 percentage-point margin is a nominal binomial sampling margin and does not cover nonresponse or measurement bias.
- By Q5, the survey had already explained pots to everyone. Q5–Q7 are fixed-order hypothetical ratings, not observed behaviour. Q6 describes nudging on any incoming payment, not specifically payday; the survey did not isolate awareness messaging or test the higher rate and onboarding together. The reported intent lifts should not be added.
- External balances are self-reported bands with an open-ended top band, and some customers did not disclose an amount. There is no observed transfer behaviour, and the supplied tables do not show how external balances and rate intent overlap at respondent level.
- Open-text codes can overlap and describe the 230 people who answered Q2, not prevalence across all customers. The 75+ and aware-by-age results have particularly small bases. Existing pot holders were screened out, so the survey does not measure how they would respond or what repricing their balances would cost.
Next steps
- experimentRun a customer-level randomised test with four arms: current experience at 3.1%, onboarding and payday nudge at 3.1%, 3.85% without the extra onboarding push, and both. Make the applicable rate clear in each arm.
- decisionBenchmark comparable UK instant-access rates and build the pricing business case before committing to a rollout.
- researchUsability-test the explainer, setup and withdrawal journey with never-users across ages, deliberately including more customers aged 75 or over. Where permitted, compare survey respondents and nonrespondents on relevant account characteristics.
Open-text themes it coded
Did not know about pots or what they were, or wanted Pennywell to tell them about pots 126Wants a higher or more competitive rate, or says the return is not worth the effort 45Hard to find in the app or needs a more visible in-app entry point or prompt 33Setup feels confusing or laborious, or needs practical guidance on getting started 16Wants reassurance about how quickly money can be withdrawn 10Does not have spare money to save 23Existing-provider habit, trust or reluctance to switch from current savings arrangements 17Not interested in saving, saving in pots or saving through an app 15
The survey it planned
0 screening questions and 8 questions, as the model wrote them.
-
Q1
Before starting this survey, did you know that Pennywell offered savings accounts called savings pots in its app?
One answer- Yes
- No
- Not sure
-
Q2
Which of the following, if any, are reasons you haven't opened a Pennywell savings pot? Select all that apply. If you select ‘Not sure’, do not select any other answers.
Any that apply- I didn't know where to find savings pots in the app
- I wasn't sure how to open one
- I thought opening one would take too much time or effort
- I hadn't got round to opening one
- I didn't know what interest rate they paid
- The interest rate wasn't high enough for me
- I already had somewhere to save that suited me
- I didn't have money available to save
- I prefer to keep my savings separate from my day-to-day bank account
- Another reason
- Not sure
Routing Shown only if Q1 is: Yes
-
Q3
About how much do you currently have in savings outside Pennywell that you can withdraw without giving notice or paying a penalty? Include easy-access savings accounts and easy-access cash ISAs. Count only your share of any joint savings. Don't include investments or money set aside for day-to-day spending.
One answer- £0
- More than £0 but less than £500
- £500 to less than £2,000
- £2,000 to less than £5,000
- £5,000 to less than £10,000
- £10,000 to less than £25,000
- £25,000 or more
- Not sure
- Prefer not to say
-
Q4
Imagine this situation: Pennywell offers a savings pot in the app. It pays 3.1% AER (annual equivalent rate), and you can withdraw your money at any time. You can find it in the savings tab, but the app doesn't show you a prompt or a payday reminder to set one up. In this situation, how likely would you be to open a savings pot in the next three months?
One answer- Very unlikely
- Fairly unlikely
- Neither likely nor unlikely
- Fairly likely
- Very likely
- Not sure
-
Q5
Imagine this situation: Pennywell offers a savings pot in the app. It pays 3.85% AER (annual equivalent rate), and you can withdraw your money at any time. You can find it in the savings tab, but the app doesn't show you a prompt or a payday reminder to set one up. In this situation, how likely would you be to open a savings pot in the next three months?
One answer- Very unlikely
- Fairly unlikely
- Neither likely nor unlikely
- Fairly likely
- Very likely
- Not sure
-
Q6
Imagine this situation: Pennywell offers a savings pot in the app. It pays 3.1% AER (annual equivalent rate), and you can withdraw your money at any time. You can find it in the savings tab. The app also shows you a prompt with a link to set one up. If your pay goes into Pennywell, the app also shows you a set-up reminder on payday. In this situation, how likely would you be to open a savings pot in the next three months?
One answer- Very unlikely
- Fairly unlikely
- Neither likely nor unlikely
- Fairly likely
- Very likely
- Not sure
-
Q7
Imagine this situation: Pennywell offers a savings pot in the app. It pays 3.85% AER (annual equivalent rate), and you can withdraw your money at any time. You can find it in the savings tab. The app also shows you a prompt with a link to set one up. If your pay goes into Pennywell, the app also shows you a set-up reminder on payday. In this situation, how likely would you be to open a savings pot in the next three months?
One answer- Very unlikely
- Fairly unlikely
- Neither likely nor unlikely
- Fairly likely
- Very likely
- Not sure
-
Q8
In the situation you just read about, how much, if anything, would you move from your existing savings outside Pennywell into a savings pot in the next three months? If you wouldn't move any money or don't have savings outside Pennywell, select £0.
One answer- £0
- More than £0 but less than £500
- £500 to less than £2,000
- £2,000 to less than £5,000
- £5,000 to less than £10,000
- £10,000 to less than £25,000
- £25,000 or more
- Not sure
- Prefer not to say
Sample plan
Use the account database to establish eligibility before invitation: current Pennywell customers aged 18 or over with no record of ever opening a savings pot, including a pot they later closed. No respondent-facing screener is needed. Invite a stratified random sample by email, using database age bands of 18–34, 35–54, 55–74 and 75 or over. Use a neutral invitation that does not mention savings pots, rates or prompts. For the 700-complete plan, target 175 completes per band. Draw random reserve batches within each band and release them as needed based on response rates; for example, a 10% eligible completion rate would require about 1,750 invitations per band. Use four survey versions: randomly assign respondents within each age band in equal proportions to exactly one of Q4, Q5, Q6 and Q7; the other three are not displayed. Everyone then sees Q8. Each respondent sees four or five questions. Randomise the substantive Q2 options, keeping ‘Another reason’ and ‘Not sure’ last and making ‘Not sure’ exclusive. Test routing and completion time on mobile before launch. The 700-complete target and the board's precision requirement cannot both be met under standard planning assumptions: 175 completes per band gives about ±7.4 percentage points at 95% confidence for a proportion near 50%, before weighting. If ±5 points is non-negotiable, seek approval for at least 385 effective completes per band (1,540 effective completes total), allowing for any weighting losses. That target applies to whole-band proportions, not to routed subgroups, experimental arms or contrasts.