Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Mistral Medium 3.5

Measured by Spring Prompt

Mistral Medium 3.5: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
63 of 100
Analysis rating
540
Head to head, this task
won 3 of 13

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Missed a trap: Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • Missed a trap: The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • Missed a trap: Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • Proposed a mistaken next step: Size the deposit opportunity by multiplying the mean balance held elsewhere by the number of customers

Findings the judges found unsupported

  • Awareness is the primary barrier: 60.6% of respondents did not know about savings pots before the survey (50.7% No + 9.9% Not sure at Q1). This is the largest single barrier to adoption.The population estimate is incorrectly labelled weighted. Age weighting gives about 65.7% No or Not sure, approximately 237,000 customers. The unweighted 60.6% multiplied by 361,000 is approximately 219,000, not 224,000.
  • Among those who knew about pots (Q1: Yes), the interest rate is the top reason for not using them (29.7% at Q3), but it is not the dominant barrier overall. Only 11.8% of all non-pot users (weighted) both knew about pots and cited the rate as the main reason.About 11.7% of respondents knew about pots and cited the rate; age weighting reduces this to approximately 9.1%, not 11.8%. The assertion that the other listed reasons are similarly or more common is not supported by Q3, where each is less common.
  • An awareness and onboarding push (e.g., in-app prompts and payday nudges) would have a larger impact on intent than a rate rise alone. 33.7% of all respondents would be likely to use pots with onboarding at 3.1% (Q6), vs. 29.9% with a rate rise to 3.85% (Q7).Q6 measures stated intent under a payday suggestion, not the effect of the complete awareness and onboarding push. Its descriptive advantage over Q7 does not establish that the intervention would have a larger impact; significance of the paired difference cannot be assessed from these marginal tables.
  • Customers aged 75+ hold disproportionately large savings elsewhere (65% hold £20,000+), but they are a tiny segment (2% of non-pot users). Targeting them would reach few customers and miss the majority of potential deposits.The 75+ result is treated as a high-confidence segment finding despite only 20 responses. A small share of customers does not establish a small share of potential deposits, so the claim that targeting them would miss the majority of potential deposits is unsupported.
  • The average savings held elsewhere by non-pot users is ~£14,500 (mean of reported amounts). If a rate rise to 3.85% AER convinced all rate-sensitive users to move their savings, it could theoretically bring in ~£1.7 billion (11.8% of 361,000 * £14,500). However, this is an upper-bound estimate and ignores other barriers (e.g., habit, trust).The calculation shown produces about £618 million, not £1.7 billion. It also uses an incorrectly weighted rate-sensitive share and applies the mean among respondents with positive external savings to a different subgroup without supporting cross-tabulation. Assuming all savings move is not a defensible upper bound, and the same calculation cannot also be called a more realistic estimate.
  • The difference in the share choosing 'The interest rate is too low' between 35-54 (27.5%) and 55-74 (37.5%) is not statistically significant at the 95% level (margin of error: ~5.6 percentage points).The non-significance conclusion is correct, but the stated 5.6-point margin of error is not appropriate for this age-group comparison. The approximate 95% margin for the difference is about 12 percentage points; overlapping individual confidence intervals are not the relevant significance test.
  • Open-text answers reveal that lack of awareness is the dominant theme (128 mentions), with many respondents explicitly stating they had never heard of pots or wanted Pennywell to explain them. This is especially true for 18-34 (42 mentions).The awareness counts are internally inconsistent: the finding reports 128 mentions, while the theme summary reports 217 answers. The latter is not supported by the supplied responses, and the reported 42 awareness mentions among 18–34s also does not match the supplied answers.

Scorecard

Numbers 8 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✗11.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✗224,000239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✗50.6%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.6 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓128126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✗4246
What percentage of the customers invited completed the survey?✓6.03%6%

Traps in the data 5 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✗Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✗The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✗Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Run the awareness and onboarding push (in-app prompts and payday nudges) first. This addresses the largest barrier (unawareness, 60.6% of non-pot users) and has a higher impact on intent (33.7% likely to use pots with onboarding at 3.1%) than a rate rise alone (29.9% at 3.85%). A rate rise should be considered only if the onboarding push fails to drive adoption, or as a secondary lever to capture rate-sensitive users (11.8% of non-pot users). Do not prioritize a campaign targeting 75+ customers, as they are a tiny segment (2%) and their savings are unlikely to move without a rate rise.

What would change it

Evidence that the onboarding push fails to drive adoption (e.g., <10% of non-pot users open a pot within 3 months) or that rate-sensitive users are a larger segment than estimated (e.g., >20% of non-pot users cite rate as the main barrier in follow-up research).

Findings

  1. Awareness is the primary barrier: 60.6% of respondents did not know about savings pots before the survey (50.7% No + 9.9% Not sure at Q1). This is the largest single barrier to adoption.

    High confidenceBase 700

    Q1: 50.7% No, 9.9% Not sure. Base: 700. Weighted to population: ~60.6% of 361,000 = ~224,000 unaware customers.

  2. Among those who knew about pots (Q1: Yes), the interest rate is the top reason for not using them (29.7% at Q3), but it is not the dominant barrier overall. Only 11.8% of all non-pot users (weighted) both knew about pots and cited the rate as the main reason.

    High confidenceBase 276

    Q3: 29.7% chose 'The interest rate is too low' (base: 276). Weighted to population: 11.8%. Other reasons (e.g., savings elsewhere, no money to save) are similarly or more common.

  3. Raising the rate to 3.85% AER would increase intent to use pots: 29.9% of all respondents would be likely (4-5) to start saving in a pot at this rate (vs. 17.4% at 3.1%). However, this is still a minority.

    High confidenceBase 700

    Q7: 29.9% likely (base: 700). Q5: 17.4% likely at 3.1%. The uplift is ~12.5 percentage points.

  4. An awareness and onboarding push (e.g., in-app prompts and payday nudges) would have a larger impact on intent than a rate rise alone. 33.7% of all respondents would be likely to use pots with onboarding at 3.1% (Q6), vs. 29.9% with a rate rise to 3.85% (Q7).

    High confidenceBase 700

    Q6: 33.7% likely with onboarding (base: 700). Q7: 29.9% likely with rate rise (base: 700). Onboarding outperforms rate rise by ~3.8 percentage points.

  5. Customers aged 75+ hold disproportionately large savings elsewhere (65% hold £20,000+), but they are a tiny segment (2% of non-pot users). Targeting them would reach few customers and miss the majority of potential deposits.

    High confidenceBase 20

    Q4: 65% of 75+ hold £20,000+ (base: 20). Population: 2% of 361,000 = ~7,220 customers. Q4: 12.6% of all respondents hold £20,000+ (base: 700).

  6. The average savings held elsewhere by non-pot users is ~£14,500 (mean of reported amounts). If a rate rise to 3.85% AER convinced all rate-sensitive users to move their savings, it could theoretically bring in ~£1.7 billion (11.8% of 361,000 * £14,500). However, this is an upper-bound estimate and ignores other barriers (e.g., habit, trust).

    Medium confidenceBase 413

    Q4: Mean savings elsewhere = £14,513 (base: 413). Weighted rate-sensitive users: 11.8% of 361,000 = ~42,600. 42,600 * £14,500 = ~£618 million (more realistic, as not all would move).

  7. The difference in the share choosing 'The interest rate is too low' between 35-54 (27.5%) and 55-74 (37.5%) is not statistically significant at the 95% level (margin of error: ~5.6 percentage points).

    High confidenceBase 222

    Q3: 35-54 = 27.5% (base: 102), 55-74 = 37.5% (base: 120). Margin of error: 5.6 percentage points. Overlap in confidence intervals.

  8. Open-text answers reveal that lack of awareness is the dominant theme (128 mentions), with many respondents explicitly stating they had never heard of pots or wanted Pennywell to explain them. This is especially true for 18-34 (42 mentions).

    High confidenceBase 230

    Open-text: 128 mentions of unawareness (base: 230). 18-34: 42 mentions (base: 230).

Caveats

  • The survey was fielded to a random sample of 11,600 non-pot users, but only 700 completed it (6.03% response rate). Non-respondents may differ systematically from respondents, introducing bias.
  • The open-text answers (230) are not representative of all respondents and may overrepresent those with strong opinions or awareness gaps.
  • The mean savings held elsewhere (£14,513) is skewed by high balances in older age groups. The median falls in the £1,000–£5,000 band, suggesting most users hold less.
  • The estimate of £1.7 billion in potential deposits from a rate rise is theoretical and assumes all rate-sensitive users would move their savings, which is unlikely due to other barriers (e.g., habit, trust).
  • The margin of error for Q3 (5.6 percentage points) means small differences between age groups may not be statistically significant.
  • The weighted estimates assume the survey sample is representative of the full population of non-pot users by age, which may not hold for other characteristics (e.g., savings behavior).

Next steps

  • decisionLaunch the awareness and onboarding push (in-app prompts and payday nudges) for all non-pot users.
  • monitorMonitor the impact of the onboarding push on pot adoption and savings inflows.
  • researchConduct a follow-up survey with non-pot users who did not respond to the onboarding push to understand remaining barriers.
  • experimentIf the onboarding push fails to drive adoption, test a rate rise to 3.85% AER with a subset of non-pot users (e.g., those who cited rate as the main barrier in Q3).
  • decisionDo not prioritize a campaign targeting 75+ customers, as they are a tiny segment (2%) and their savings are unlikely to move without a rate rise.

Open-text themes it coded

Unawareness of savings pots 217Rate is too low 38Setup is confusing or hard to find 47Savings held elsewhere (habit/trust) 30No money to save 46Withdrawal speed concerns 13

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Do you currently have a Pennywell current account?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Have you ever opened a savings pot in the Pennywell app?

    One answer
    • Yes
    • No
    • I don’t know

    Continues if No

  3. S3

    Are you aged 18 or over?

    One answer
    • Yes
    • No

    Continues if Yes

  4. Q1

    Before today, were you aware that Pennywell offers savings pots (instant-access savings accounts inside the app)?

    One answer
    • Yes
    • No
  5. Q2

    What is the main reason you have not opened a savings pot in the Pennywell app? Please select one.

    One answer
    • I didn’t know they existed
    • The interest rate is too low
    • I don’t need or want a savings pot
    • I already have savings elsewhere
    • I find the app confusing or hard to use
    • I don’t trust savings pots in a bank app
    • Other (please specify)

    Routing Shown only if Q1 is: No

  6. Q3

    If you knew about savings pots, what is the main reason you have not opened one? Please select one.

    One answer
    • The interest rate is too low
    • I don’t need or want a savings pot
    • I already have savings elsewhere
    • I find the app confusing or hard to use
    • I don’t trust savings pots in a bank app
    • Other (please specify)

    Routing Shown only if Q1 is: Yes

  7. Q4

    How important is the interest rate when you choose where to save money? Please rate on a scale of 1 to 5, where 1 is 'Not at all important' and 5 is 'Extremely important'.

    Scale

    Scale 1-5: Not at all important to Extremely important

  8. Q5

    If Pennywell increased the interest rate on savings pots from 3.1% to 3.85% AER, how likely would you be to open a savings pot? Please rate on a scale of 1 to 5, where 1 is 'Not at all likely' and 5 is 'Extremely likely'.

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  9. Q6

    How much do you currently have saved in other savings accounts or products (e.g., ISAs, fixed-rate bonds, other bank savings)? Please select the closest range.

    One answer
    • £0
    • £1 - £500
    • £501 - £2,000
    • £2,001 - £5,000
    • £5,001 - £10,000
    • £10,001 - £20,000
    • £20,001 - £50,000
    • £50,001 or more
    • I don’t know
  10. Q7

    If Pennywell made it easier to set up a savings pot (e.g., with in-app prompts or a nudge on payday), how likely would you be to open one? Please rate on a scale of 1 to 5, where 1 is 'Not at all likely' and 5 is 'Extremely likely'.

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  11. Q8

    Which of the following would most encourage you to open a savings pot in the Pennywell app? Please select up to two.

    Any that apply
    • A higher interest rate (e.g., 3.85% AER)
    • Easier set-up (e.g., in-app prompts or a nudge on payday)
    • More information about how pots work
    • A guarantee that my money is safe
    • None of these would encourage me
    • Other (please specify)
  12. Q9

    What is your age?

    One answer
    • 18-34
    • 35-54
    • 55-74
    • 75 or over
  13. Q10

    This is an attention check. Please select 'Yes' to confirm you are paying attention.

    One answer
    • Yes
    • No
Sample plan

Random sample of 700 Pennywell customers aged 18 or over who have never opened a savings pot, invited via email. Quotas will be set to ensure at least 150 completes per age band (18-34, 35-54, 55-74, 75 or over) to achieve ±5% precision at 95% confidence for each group.