Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Mistral Large 4

Measured by Spring Prompt

Mistral Large 4: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
64 of 100
Analysis rating
815
Head to head, this task
won 5 of 15

Why it is not sound

The survey

  • A research objective no question serves
  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • Missed a trap: Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • Proposed a mistaken next step: Raise the pots rate to 3.85% for all customers now
  • Proposed a mistaken next step: Size the deposit opportunity by multiplying the mean balance held elsewhere by the number of customers

Findings the judges found unsupported

  • Customers aged 75+ do hold large balances (65% have £20k+, vs 4-9% of younger groups), but they are only 2% of the non-pot population (~7,200 customers). Their awareness of pots is moderate (35% knew), and stated intent at 3.85% (25%) is no higher than other groups. A campaign targeting them alone would reach too few people to drive material volume.Only 20 respondents were aged 75 or over. Their reported balances cannot establish that this customer group generally holds large balances. Nor does their small population share establish that a campaign could not drive material deposit volume; actual deposits were not measured.
  • Stated intent to open a pot at 3.85% AER is 29.9% (~108,000 of 361,000 customers). Customers report holding an average of £14,513 in savings elsewhere (median £1k-£5k). If the 29.9% who say they are likely moved just 10-25% of their existing savings, this would represent £157m-£391m in new deposits, though this is speculative as the survey did not measure intended deposit amounts.The deposit estimate applies the mean balance among respondents with savings elsewhere to all hypothetical likely adopters, without evidence about those adopters' balances or willingness to transfer money. The mean is also skewed by large balances. Calling the estimate speculative does not establish the claimed deposit opportunity.
  • Open-text responses reinforce the quantitative findings: 128 of 230 respondents (56%) explicitly state they did not know about pots or ask Pennywell to explain/tell them about them. 47 of these are aged 18-34. The next most common themes are rate too low (46 mentions) and usability/friction in finding or setting up pots (28 mentions).The reported coding counts do not match the supplied answers. For example, there are 45 rate mentions, rather than 46, and 38 answers mentioning difficulty finding or setting up pots, rather than 28. The analysis also gives inconsistent rate totals between this finding and its theme summary.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,845239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.3%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓128126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4746
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 6 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✗The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✗Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Run both, but sequence them as an integrated campaign: launch the awareness and onboarding push (in-app prompts, home-screen placement, payday nudge) immediately to close the 66% awareness gap, and simultaneously raise the rate to 3.85% AER to convert the 29.7% of aware customers who cite rate as their main barrier. The payday nudge at the current rate already outperforms the rate rise in stated intent (33.7% vs 29.9%), and it is especially effective for the unaware segment (42.5% likely). Raising the rate alone would fail to reach the two-thirds of customers who do not know pots exist, while awareness alone would leave the rate objection unaddressed for the aware minority. Do not build a dedicated campaign around customers aged 75+: they hold large balances but represent only 2% of the non-pot population (~7,200 customers) and their stated intent is no higher than other groups.

What would change it

The recommendation would change if: (1) A/B testing shows the payday nudge alone converts unaware customers to active savers at a rate comparable to the combined intervention, making the rate rise unnecessary; (2) Further research shows that the 29.9% likely at 3.85% would deposit negligible amounts (e.g., <£500 average), making the deposit value too low to justify the interest cost of the rate rise; (3) Competitive analysis shows 3.85% is still uncompetitive vs best-buy tables, requiring a higher rate to move the 'rate' segment.

Findings

  1. Unawareness is the dominant barrier: two-thirds of non-pot customers (66.4% weighted, ~240,000 of 361,000) did not know savings pots existed before the survey. Among those who did know, 'rate too low' was the top reason (29.7%), but this represents only 8.8% of all non-pot customers weighted (11.7% unweighted).

    High confidenceBase 700

    Q1: 60.6% answered No/Not sure (50.7% No, 9.9% Not sure). Weighted to population age mix: 66.4% unaware (18-34: 78.5% unaware; 35-54: 61.1%; 55-74: 40.0%; 75+: 65.0%). Q3 base (n=276) shows rate is top reason at 29.7%, but 276/700 = 39.4% knew, so 39.4% × 29.7% = 11.7% of total. Weighted: 8.8%.

  2. The payday nudge (behavioural prompt) generates higher stated intent than the rate rise alone. At the current 3.1% rate, a payday set-aside suggestion produces 33.7% likely-to-use (Q6), compared to 29.9% if the rate rises to 3.85% with no nudge (Q7), and only 17.4% at 3.1% with no nudge (Q5). The nudge is particularly effective for the unaware segment (42.5% likely if unaware vs 18.8% if aware).

    High confidenceBase 700

    Q5 (3.1%, no nudge): 17.4% net likely. Q6 (3.1%, payday nudge): 33.7% net likely. Q7 (3.85%, no nudge mentioned): 29.9% net likely. Q6 by awareness: Knew=18.8%, Didn't know=42.5%, Not sure=47.8%.

  3. The rate is a real but secondary barrier. Among the minority who know about pots (39.4%), 'rate too low' is the single biggest reason (29.7%), ahead of inertia (19.6%) and lack of spare cash (13.4%). However, the difference between age groups 35-54 (27.5%) and 55-74 (37.5%) is not statistically significant (z=1.58, p>0.05), and the 95% margin of error on the 29.7% figure is ±5.4pp.

    High confidenceBase 276

    Q3: Rate too low 29.7% (n=82 of 276), Keep elsewhere 19.6%, No money left 13.4%. Statistical test: p1=0.275 (n=102), p2=0.375 (n=120), pooled SE=0.063, z=1.58 < 1.96. Margin of error: 1.96×√(0.297×0.703/276) = 5.4pp.

  4. Customers aged 75+ do hold large balances (65% have £20k+, vs 4-9% of younger groups), but they are only 2% of the non-pot population (~7,200 customers). Their awareness of pots is moderate (35% knew), and stated intent at 3.85% (25%) is no higher than other groups. A campaign targeting them alone would reach too few people to drive material volume.

    Medium confidenceBase 20

    Q4: 75+ £20k+ = 65.0% (n=13 of 20). Population: 2% of 361,000 = 7,220. Q1: 35.0% knew (n=7 of 20). Q7: 25.0% likely (n=5 of 20). Q4 weighted £20k+ for 75+: 0.02×65% = 1.3% of total base hold £20k+ in this age group.

  5. Stated intent to open a pot at 3.85% AER is 29.9% (~108,000 of 361,000 customers). Customers report holding an average of £14,513 in savings elsewhere (median £1k-£5k). If the 29.9% who say they are likely moved just 10-25% of their existing savings, this would represent £157m-£391m in new deposits, though this is speculative as the survey did not measure intended deposit amounts.

    Low confidenceBase 700

    Q7: 29.9% net likely (18.0% fairly + 11.9% very). Q4 note: Mean £14,513 among 413 reporting an amount. 361,000 × 29.9% = 107,939 customers. 107,939 × £14,513 = £1.57bn total savings held by this group; 10-25% = £157m-£391m.

  6. Open-text responses reinforce the quantitative findings: 128 of 230 respondents (56%) explicitly state they did not know about pots or ask Pennywell to explain/tell them about them. 47 of these are aged 18-34. The next most common themes are rate too low (46 mentions) and usability/friction in finding or setting up pots (28 mentions).

    High confidenceBase 230

    Q2 open-text coding: Awareness/education theme = 128 respondents (list in themes). Rate theme = 46 respondents. Usability theme = 28 respondents. 18-34 awareness mentions = 47 (66% of the estimated 71 18-34 respondents in the open-text sample).

Caveats

  • Stated intent overestimates actual behaviour: 29.9% say they are likely to open a pot at 3.85%, but historical conversion from survey intent is typically 10-30% of stated intent.
  • The survey did not ask how much money customers would deposit into a pot, so deposit value estimates (£157m-£391m) rely on assumptions about the proportion of existing savings moved.
  • Savings balances are self-reported in bands; the £14,513 mean uses midpoints and counts '£50,000 or more' as £75,000, which may under- or over-state true mean depending on the tail.
  • The sample is skewed older than the population (28.6% aged 55-74 vs 13% in population; 31.1% aged 18-34 vs 46% in population). Weighted figures are provided where relevant, but unweighted tables are shown in the data.
  • Base sizes for 75+ are very small (n=20 overall, n=7 for Q3), so age-specific findings for this group are directional only.
  • Q3 (reasons for not using) was asked only of the 39.4% who knew about pots, so it cannot speak to the motivations of the 60.6% who were unaware.
  • The survey tested the rate rise and the payday nudge separately; the combined effect of doing both is unknown and may be synergistic or substitutive.

Next steps

  • decisionLaunch in-app awareness campaign: home-screen banner, 'pots' shortcut on main navigation, and educational tooltip explaining instant access and rate
  • experimentImplement payday nudge: prompt to set aside £X when salary is paid into current account
  • decisionRaise savings pots rate from 3.1% to 3.85% AER
  • experimentRun a 4-week pilot with combined intervention (awareness push + rate rise + payday nudge) on a random 10% of non-pot customers before full rollout
  • researchCommission further research on deposit amounts: ask likely adopters (Q7 4-5) how much they would move from existing savings
  • decisionSimplify pot set-up flow: reduce steps, add progress indicator, and test one-tap creation

Open-text themes it coded

Lack of awareness / need for explanation (didn't know pots existed, want Pennywell to tell/explain them) 128Rate too low / not competitive / need higher interest 45Usability friction / can't find pots in app / setup too complex 28Inertia / keep savings with existing provider / trust / habit 13Liquidity concerns / worried about withdrawal speed / access 10No spare money / cannot save 22Not interested / nothing would change / don't want to save 15

The survey it planned

3 screening questions and 9 questions, as the model wrote them.

  1. S1

    Are you a Pennywell current account customer?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Have you ever opened a savings pot in the Pennywell app?

    One answer
    • Yes, I have opened one
    • No, I have never opened one
    • Not sure

    Continues if No, I have never opened one; Not sure

  3. S3

    What is your age?

    One answer
    • 18-34
    • 35-54
    • 55-74
    • 75 or over

    Continues if 18-34; 35-54; 55-74; 75 or over

  4. Q1

    Before today, were you aware that Pennywell offers savings pots inside the app?

    One answer
    • Yes
    • No
  5. Q2

    You said you were aware of savings pots but haven't opened one. What is the main reason you haven't opened a savings pot yet?

    One answer
    • I haven't got around to it
    • I don't know how to set one up
    • The interest rate is too low
    • I don't have spare money to save right now
    • I don't trust it or worry about the safety of my money
    • I prefer to keep my savings with another bank or building society
    • Other (please specify)

    Routing Shown only if Q1 is: Yes

  6. Q3

    Pennywell savings pots are instant-access savings accounts inside the app, currently paying 3.1% AER interest. Now that you know about them, how likely are you to open a savings pot in the next 3 months?

    Scale

    Scale 1-5: Very unlikely to Very likely

    Routing Shown only if Q1 is: No

  7. Q4

    If the interest rate on Pennywell savings pots increased to 3.85% AER (from the current 3.1%), how likely would you be to open a savings pot?

    Scale

    Scale 1-5: Very unlikely to Very likely

    Judges: ambiguous

    No timeframe is specified for opening a pot. Respondents could answer about immediate uptake or opening one eventually, unlike Q3's explicit three-month window.

    No timeframe is given for opening a pot. Q3 specifies 'in the next 3 months', so respondents may read the period differently and the answers are not comparable with the Q3 baseline.

  8. Q5

    If you received a prompt in the Pennywell app on payday suggesting you move some money into a savings pot, how likely would you be to follow that suggestion?

    Scale

    Scale 1-5: Very unlikely to Very likely

  9. Q6

    Do you currently hold savings in accounts with other banks or building societies (not including your Pennywell current account)?

    One answer
    • Yes
    • No
  10. Q7

    Roughly how much do you currently have in total across all your savings accounts with other providers?

    One answer
    • Less than £1,000
    • £1,000 to £4,999
    • £5,000 to £9,999
    • £10,000 to £24,999
    • £25,000 to £49,999
    • £50,000 or more
    • Prefer not to say

    Routing Shown only if Q6 is: Yes

  11. Q8

    If you opened a Pennywell savings pot, roughly how much would you expect to put into it in the first month?

    One answer
    • Nothing
    • Less than £500
    • £500 to £1,999
    • £2,000 to £4,999
    • £5,000 or more
    • Prefer not to say
  12. Q9

    How much do you trust Pennywell to keep money saved in a savings pot safe and secure?

    Scale

    Scale 1-5: Not at all to Completely

Sample plan

Invite a random sample of Pennywell customers aged 18+ who have never opened a savings pot (identified from the account database) via email to complete the online questionnaire. Target 1,600 completed interviews with hard quotas of 400 completes in each age band (18-34, 35-54, 55-74, 75 or over). Note: The brief mentions 'about 700 completes', but to achieve the board's precision requirement of ±5 percentage points at 95% confidence for each of the four age bands, a minimum of 385 completes per band is required (total n≈1,540); 1,600 is recommended to allow for non-response and weighting. Quotas will be monitored in real-time and invitations closed by band when targets are met.