Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Claude Haiku 5.5

Measured by Spring Prompt

Claude Haiku 5.5: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
94 of 100
Analysis rating
1,412
Head to head, this task
won 6 of 9

Why it is not sound

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • A rate rise to 3.85% is most likely to move customers who already know about pots. For customers who do not know about pots, a prompt at the current rate moves stated intent much more than a rate rise does.The claim that the overall Q6–Q7 difference is not statistically significant cannot be established from these marginal tables. These are paired responses, and significance requires the respondent-level pattern of changes; an independent-samples calculation does not resolve it.
  • The 'keep it with another provider' barrier is real for older customers. Customers aged 55 and over hold most of the large balances elsewhere, and a rate rise alone is unlikely to move them.Older respondents account for most reported large balances in the unweighted sample, but this does not hold after weighting to the customer age mix. The tables also do not establish that a rate rise alone is unlikely to move older customers generally; the open text establishes loyalty for some respondents, not that general behavioural prediction.
  • Balances held elsewhere are concentrated, so the £14,500 mean misstates what a rate rise could attract.The approximately £8,600 mean across all 700 respondents implicitly assigns zero balances to respondents answering Don't know or Prefer not to say. Their balances are unknown, so this is not an established whole-sample mean.
  • The 75 or over group is too small to justify a campaign. It is also the group most likely to be unaware of pots, and it needs a careful, Consumer Duty-compliant approach.The 75+ group is not the group most likely to be unaware of pots: 65% answered No or Not sure, compared with approximately 78.5% of 18–34s.
  • The sample under-represents the 18-34 group, who are the least aware. Unweighted figures overstate awareness.The 18–34 group does not have the lowest Q5 net-likely score. Its score is 17.9%, compared with 16.4% for 35–54s. It does have the lowest Q7 score.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,800239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.2%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓126126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4646
What percentage of the customers invited completed the survey?✓6.03%6%

Traps in the data 8 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Do not settle this with a blanket rate rise on current evidence. Run the awareness and onboarding push now, and test the rate in parallel on a randomised basis. The push addresses the majority who did not know about pots (about two-thirds of customers, weighted). Its stated-intent lift for that group (prompt 42.5% vs rate rise 24.2%) is larger than the rate rise. Among customers who know about pots, the rate is the lever (37.0% at 3.85% vs 18.8% for the prompt), so the rate question should be answered with a test on that group before any change for all customers. Before the push goes live, fix the findability and set-up problems the open text raises, and make the explanation of what pots are clear inside the app. Do not build a campaign around the 75+ group on this evidence.

What would change it

(1) A randomised test showing the 3.85% rate lifts opened pots and balances enough, net of the cost of the higher rate on all pot balances, to clear Pennywell's return threshold, would support the rate rise. (2) A test showing the awareness push does not raise pot openings among the unaware, or that its effect is much smaller than the stated intent suggests, would shift the weight towards rate. (3) Evidence that the 3.85% rate attracts existing savers from other providers in a cost-effective way would strengthen the case. (4) A re-run with a weighted, larger sample of 75+ customers showing high intent and high balances, and a fair-value and vulnerability review, would be needed before any campaign for that group.

Findings

  1. Lack of awareness is the largest barrier. Most customers who have never opened a pot did not know they could, and the open-text answers say the same thing.

    High confidenceBase 700

    Q1: 39.4% knew (unweighted), so 60.6% said No or Not sure. Weighted to the population age mix, 33.6% knew and 66.4% did not (about 239,800 of 361,000 customers). Awareness is lowest at 18-34 (21.6% knew, 78.5% No or Not sure). Among 230 open-text answers, 126 (55%) say the respondent did not know about pots or what they are, or want Pennywell to explain them. Q2 was asked after the survey described pots, so these answers show that even a description did not make pots clear to many respondents.

  2. The interest rate is the top stated reason for not opening a pot, but only among customers who already knew about pots. It cannot be read as the reason for the whole base, and it does not settle the rate decision.

    High confidenceBase 276

    Q3 (asked only of the 276 who knew): rate too low 29.7% (±5.4pp), ahead of keeping savings with another provider (19.6%) and no money to save (13.4%). As a share of all 700 respondents this is 11.7%. Among the 355 who did not know about pots, Q3 was not asked, so their main barrier is unmeasured. The open text suggests awareness and explanation come first for this group.

  3. A rate rise to 3.85% is most likely to move customers who already know about pots. For customers who do not know about pots, a prompt at the current rate moves stated intent much more than a rate rise does.

    Medium confidenceBase 631

    Net likely to save (4-5): knew about pots, Q7 at 3.85% 37.0% vs Q6 prompt at 3.1% 18.8% (n=276, difference 18.2pp, z≈4.9). Did not know about pots, Q6 prompt 42.5% vs Q7 rate rise 24.2% (n=355, difference 18.3pp, z≈5.3). Overall: Q5 current rate 17.4%, Q6 prompt 33.7%, Q7 rate rise 29.9%. The overall difference between Q6 and Q7 is small (3.8pp, not significant), so the overall average hides the split.

  4. The rate rise lifts stated intent across the whole base by about 12.5 points, from 17.4% to 29.9%. This is a stated-intent result, and it does not show how much money would move or what it would cost.

    Medium confidenceBase 700

    Q5 at 3.1%: net likely 17.4% (n=700). Q7 at 3.85%: net likely 29.9% (n=700). Difference 12.5pp, statistically clear (z≈5.6 treating the samples as independent). The survey does not show the cost of raising the rate, which would apply to all pot balances, including those of existing pot users, who were excluded from the sample.

  5. The 'keep it with another provider' barrier is real for older customers. Customers aged 55 and over hold most of the large balances elsewhere, and a rate rise alone is unlikely to move them.

    Medium confidenceBase 200

    Q4: 21.5% of 55-74s and 65% of 75+ (n=20) hold £20,000 or more elsewhere, against 4.1% of 18-34s. Among those who knew about pots, 42.9% of 75+ (n=7) and 23.3% of 55-74s (n=120) say they keep savings with another provider. Also, 'interest rate too low' is 37.5% for 55-74s and 27.5% for 35-54s. The 35-54 vs 55-74 difference is not significant (z≈1.6, p≈0.11).

  6. Balances held elsewhere are concentrated, so the £14,500 mean misstates what a rate rise could attract.

    Medium confidenceBase 413

    Among the 413 respondents who gave an amount, the mean is £14,513, but it uses band midpoints, with £50,000+ counted as £75,000, and the median is in the £1,000-£5,000 band. 55.2% of those who gave an amount hold under £5,000, and 12.6% of all respondents hold £20,000 or more. Across all 700 respondents, the mean is about £8,600 (14,513 × 413/700), because 35.7% hold nothing elsewhere. Also, holding money elsewhere does not mean it would move into pots.

  7. The 75 or over group is too small to justify a campaign. It is also the group most likely to be unaware of pots, and it needs a careful, Consumer Duty-compliant approach.

    Low confidenceBase 20

    The 75+ base is 20 (Q4 and Q5-Q7) and 7 (Q3). Their share of the population is 2%, about 7,200 customers. Q5 net likely at 3.1% is 20% and Q6 is 25%, but the margin on n=20 is roughly ±20pp. 65% of 75+ respondents did not know about pots (60% No, 5% Not sure), and 65% hold £20,000+ elsewhere. Targeting older customers with high balances needs fair-value and vulnerability review.

  8. The sample under-represents the 18-34 group, who are the least aware. Unweighted figures overstate awareness.

    High confidenceBase 700

    Sample age mix: 18-34 31.1%, 35-54 37.4%, 55-74 28.6%, 75+ 2.9%. Population: 46%, 39%, 13%, 2%. Awareness is 39.4% unweighted and 33.6% weighted. The stated-intent results may also be biased, since the 18-34 group has the lowest net likely scores at Q5 and Q7 but the highest unaware share.

  9. Many customers who know about pots say they cannot find them, cannot work out how to set one up, or are worried about how quickly they can get their money out. These are fixable product problems that a push alone would not solve.

    Medium confidenceBase 230

    Open text (230 answers): 27 say they cannot find pots in the app, 13 describe a set-up that is too complicated, 10 raise access or withdrawal speed, and 23 say they have no spare money. Q3 shows 7.6% of those who knew about pots cite speed of access. These counts come from one coder and are indicative.

Caveats

  • Q2 open text was answered by 230 of 700 respondents. Those who answered may differ from those who did not, and the themes are one coder's reading, not a validated coding frame.
  • Q2 was asked after the survey described savings pots, so the high unawareness in open text is measured after a description. It may still understate how many customers would notice pots in the app.
  • Q5, Q6 and Q7 are stated intentions, asked of the same people in a fixed order. Q6 may have primed Q7 or the reverse. The order effect is untested. Stated intent normally overstates real take-up.
  • Q6 describes a payday prompt that does not yet exist. Its real effect depends on how it is built and how often it appears.
  • The survey covers only customers who have never opened a pot. Existing pot holders, whose balances a rate rise would also raise in cost, are not in the sample, so the cost of a rate rise cannot be read from it.
  • The sample is unweighted and under-represents 18-34s (31% vs 46%) and over-represents 55-74s (29% vs 13%). The 75+ base is 20 respondents, too small for reliable percentages.
  • Q4 balances are banded, and the mean uses band midpoints with an assumed £75,000 for the top band. Balances elsewhere are not the same as money that would move into pots.
  • Q3 is asked only of those who knew about pots (n=276). The main barrier for the unaware majority is not measured.
  • The Q3 margin of error (±5.4pp) ignores finite-population and design effects. The 35-54 vs 55-74 difference is not significant at 95%.
  • Completion is 6.0% of invitations (700 of 11,600). Non-response may be systematic, for example from people who are less engaged with the app.

Next steps

  • decisionApprove the awareness and onboarding push: an in-app explainer of what pots are, a 'did you know' prompt, and a payday set-up nudge. Fix the pots entry point and simplify set-up as a prerequisite.
  • experimentRun a randomised rate test: 3.1% vs 3.85% for customers who already know about pots (and for the push cohort), with a holdout, for at least one full pay cycle.
  • researchBuild a cost model for the rate rise with finance, covering existing pot balances, expected new balances, and switching from other providers. Use it to answer the CFO's question about how much a rate rise could bring in.
  • researchTest question order and the framing of the prompt (Q6 vs Q7 order, and the wording of the payday prompt) in a follow-up survey with randomised order, and add a behavioural measure.
  • researchRe-run the survey with a larger, weighted sample of 18-34 and 75+ customers, plus existing pot holders, before any targeting by age.
  • decisionReview any campaign aimed at older customers with large balances against Consumer Duty fair-value and vulnerability requirements before it is designed.
  • monitorMonitor pot openings, time to first deposit, set-up abandonment, and withdrawal complaints after the push and any rate change.

Open-text themes it coded

Did not know pots existed or what they are; wants Pennywell to explain or tell them 126Wants a higher or more competitive rate 44Cannot find or locate pots in the app 27Set-up is complicated or fiddly 13Worried about speed of access or withdrawal 10No spare money to save 23Happy with savings elsewhere or loyal to another provider 18Not interested in saving 15

The survey it planned

2 screening questions and 9 questions, as the model wrote them.

  1. S1

    Are you aged 18 or over?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Have you ever opened a savings pot in the Pennywell app?

    One answer
    • Yes, I have opened one
    • No, I have never opened one
    • Not sure

    Continues if No, I have never opened one

  3. Q1

    Before today, had you heard of savings pots in the Pennywell app?

    One answer
    • Yes, I knew about them and had looked at them in the app
    • Yes, I knew about them but had not looked at them
    • No, I had not heard of them
    • Not sure
  4. Q2

    Why haven't you opened a savings pot? Select all that apply.

    Any that apply
    • I don't have spare money to save
    • I already save somewhere else
    • I didn't know how savings pots worked
    • The interest rate isn't high enough
    • I don't need to save at the moment
    • It isn't clear how to get my money back out
    • I haven't got round to it
    • I'm not sure they are safe
    • I didn't think about it

    Routing Shown only if Q1 is: Yes, I knew about them and had looked at them in the app; Yes, I knew about them but had not looked at them

  5. Q3

    Imagine each of these were offered to you. Rank them from 1 (most likely to get you to open a savings pot) to 4 (least likely).

    Ranking
    • A higher interest rate on savings pots
    • A reminder in the app to save
    • A prompt to move money into a pot when you are paid
    • A quicker way to set up a savings pot
  6. Q4

    Savings pots currently pay 3.1% AER. If they paid 3.85% AER instead, how much would that make you more likely to open a savings pot?

    Scale

    Scale 1-5: Not at all to A great deal

  7. Q5

    How likely are you to open a savings pot in the next six months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  8. Q6

    Across all your savings accounts with other providers, roughly how much do you hold in total?

    One answer
    • None
    • Under £1,000
    • £1,000 to £4,999
    • £5,000 to £9,999
    • £10,000 to £24,999
    • £25,000 to £49,999
    • £50,000 or more
    • Prefer not to say
  9. Q7

    Which types of savings do you hold with other providers? Select all that apply.

    Any that apply
    • Easy-access savings account with a high street bank
    • Easy-access savings account with an online bank or app
    • Fixed-rate bond
    • Cash ISA
    • Regular saver account
    • Premium Bonds
    • Other savings product

    Routing Shown only if Q6 is: Under £1,000; £1,000 to £4,999; £5,000 to £9,999; £10,000 to £24,999; £25,000 to £49,999; £50,000 or more

  10. Q8

    How are you usually paid?

    One answer
    • Monthly, on a set date
    • Every two weeks
    • Weekly
    • Irregularly or on invoice
    • I am not currently paid
  11. Q9

    What is your age?

    One answer
    • 18 to 34
    • 35 to 54
    • 55 to 74
    • 75 or over
    • Prefer not to say
Sample plan

Invite by email a random sample drawn from the account database of Pennywell customers aged 18 or over who have never opened a savings pot (about 361,000 customers, 86% of 420,000). Exclude staff and test accounts. Set quotas by age band taken from the database: 385 completes in each of 18-34, 35-54, 55-74 and 75 or over, giving 1,540 completes. This is above the brief's c.700. At 700 completes, each band would have about 175 respondents, giving a worst-case margin of about ±7.4 points per band, which does not meet the board's ±5-point requirement. Overall results at 700 would be about ±3.7 points. If the budget is capped at 700, report the per-band margins as they are and tell the board the requirement is not met for any band. The 75 or over band is likely to be small in the base, so oversample invitations to that band and keep inviting until its quota is full. Close each band's quota when it is reached. Report overall figures weighted to the age profile of the never-opener base, and report band figures unweighted within band. Check self-reported age (Q9) against the database age and flag mismatches.