Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Gemini 3.5 Flash Lite

Measured by Spring Prompt

Gemini 3.5 Flash Lite: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
48 of 100
Analysis rating
662
Head to head, this task
won 1 of 10

Why it is not sound

The survey

  • Presumed the stakeholder's view instead of testing it
  • A research objective no question serves
  • Ignored the precision the brief asked for, which the sample cannot deliver
  • Answer bands that overlap or leave gaps
  • A leading question: Q4
  • A double-barrelled question: Q4

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • Missed a trap: The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • Missed a trap: 30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • Missed a trap: Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • Proposed a mistaken next step: Raise the pots rate to 3.85% for all customers now
  • Proposed a mistaken next step: Size the deposit opportunity by multiplying the mean balance held elsewhere by the number of customers

Findings the judges found unsupported

  • Older customers hold the largest savings balances elsewhere, but their volume within Pennywell's customer base is very small.The finding treats the balance pattern among customers aged 75 or over as established, with high confidence, despite a base of only 20. The population share is supported, but the survey does not reliably establish this group's balance profile.
  • While interest rate is cited as the top barrier by those who know about pots, raising the rate alone has limited reach compared to fixing awareness and onboarding.It treats stated likelihood at a hypothetical rate and with a hypothetical nudge as measured 'adoption' reach, with high confidence and no caveat that stated intentions overstate behaviour. The figures are also unweighted for the age skew.

Scorecard

Numbers 9 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✗12.5%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✗218,526239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.4%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓139126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✗3846
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 4 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✗Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✗The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✗30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✗Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Do both: run a comprehensive awareness and onboarding push alongside raising the rate to 3.85% AER. While the CFO is correct that the rate is the top stated barrier among those who know about pots, over half of customers are totally unaware pots exist. Fixing app discoverability and nudging on payday unlocks the broader mass-market audience, while the rate increase removes the barrier for financially motivated savers.

What would change it

Definitive usage tracking showing that awareness campaigns alone drive zero incremental deposits without a rate hike, or conversely, financial modeling proving that the margin compression from 3.85% outweighs any net interest income generated from brought-in balances.

Findings

  1. Low awareness of savings pots is the single biggest barrier to adoption, affecting over 60% of customers, particularly younger cohorts.

    High confidenceBase 700

    50.7% of all respondents stated they did not know about savings pots (Q1), rising to 67.0% among 18-34 year olds. Furthermore, 139 out of 230 open-text comments explicitly requested an explanation or stated they were unaware.

  2. While interest rate is cited as the top barrier by those who know about pots, raising the rate alone has limited reach compared to fixing awareness and onboarding.

    High confidenceBase 276

    Among those who knew about pots, 29.7% cited the interest rate as the main barrier (Q3). However, raising the rate to 3.85% increases the net likely adoption to only 29.9% (Q7), whereas an in-app payday nudge alone lifts intent to 33.7% (Q6).

  3. Older customers hold the largest savings balances elsewhere, but their volume within Pennywell's customer base is very small.

    High confidenceBase 20

    65.0% of respondents aged 75 or over hold £20,000+ elsewhere (Q4), but this age group accounts for only 2% of the total customer population (361,000 base).

Caveats

  • The survey was fielded exclusively among customers who have never opened a savings pot, meaning findings reflect non-users rather than active pot holders.
  • Survey completion rate was low (6.0% of invitees), introducing potential self-selection bias where respondents with strong opinions (either negative about rates or completely unaware) were more likely to complete.
  • Savings amounts held elsewhere (Q4) are self-reported and grouped into bands, requiring midpoint estimations for averages.

Next steps

  • decisionRedesign app navigation to bring savings pots out of the 'More' menu and onto the primary accounts dashboard.
  • experimentImplement an A/B test for the in-app payday savings nudge combined with a simple onboarding walkthrough.
  • researchModel the net revenue impact of raising the savings rate to 3.85% AER against the estimated average external balance of £14,513 per converting customer.

Open-text themes it coded

Unaware of savings pots existence 123Interest rate too low / uncompetitive 44App UX / navigation / setup friction 26No spare money / living paycheck to paycheck 20Loyalty or habit with existing external provider 10

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Have you ever opened a savings pot in your Pennywell app?

    One answer
    • Yes, I have opened a savings pot
    • No, I have never opened a savings pot
    • I am not sure

    Continues if No, I have never opened a savings pot

  2. S2

    Which age group do you fall into?

    One answer
    • 18-34
    • 35-54
    • 55-74
    • 75 or over
    • Prefer not to say

    Continues if 18-34; 35-54; 55-74; 75 or over

  3. Q1

    Before taking this survey, did you know that Pennywell offered savings pots inside the app?

    One answer
    • Yes, I was fully aware of them
    • I had a vague idea they might exist, but wasn't sure
    • No, I had no idea they existed
  4. Q2

    What are the main reasons you haven't opened a savings pot with Pennywell yet? (Select all that apply)

    Any that apply
    • I didn't know they existed
    • The current interest rate (3.1% AER) is too low
    • I already keep my savings with another bank/provider
    • It's too much hassle to set up or move money
    • I prefer to keep all my money in my main current account
    • I simply haven't got around to it yet
    • Other (please specify)
  5. Q3

    Pennywell's current savings pots pay 3.1% AER instant access. If this rate were raised to 3.85% AER, how likely would you be to open a savings pot?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

  6. Q4

    Imagine if Pennywell made it effortless to set up a pot and automatically sweep leftover money into it on payday, while keeping the interest rate at 3.1% AER. How likely would you be to use a savings pot?

    Scale

    Scale 1-5: Not at all likely to Extremely likely

    Judges: leading, double-barrelled

    Calling set-up 'effortless' pushes a favourable assessment. One rating combines easier set-up with automatic payday transfers, so their separate influence cannot be identified. 'Leftover money' does not specify what money would be transferred.

    'Effortless' is a loaded, promotional word that inflates likelihood. The question also bundles two features (easy set-up and automatic payday sweep) into one answer.

  7. Q5

    Which of the following factors would most influence you to finally open a Pennywell savings pot? (Please choose your top factor)

    One answer
    • A higher interest rate (e.g., 3.85% AER or more)
    • Better visibility and reminders inside the app
    • Automated saving features (like round-ups or payday sweeps)
    • Trust and reputation of Pennywell
    • None of these would make me open one
  8. Q6

    Roughly how much total savings do you currently hold across all other bank accounts, savings accounts, ISAs, or other financial providers?

    One answer
    • £0
    • £1 - £999
    • £1,000 - £4,999
    • £5,000 - £9,999
    • £10,000 - £24,999
    • £25,000 - £49,999
    • £50,000 or more
    • Prefer not to say
  9. Q7

    What interest rate (AER) do you estimate your current primary savings provider pays you?

    One answer
    • Under 2.0%
    • 2.0% - 3.0%
    • 3.1% - 4.0%
    • 4.1% - 5.0%
    • Over 5.0%
    • I have no idea / Don't know
    Judges: assumptive

    Respondents with no savings still see this question, which assumes they have a primary savings provider. 'Primary' could mean the provider holding the most savings or the one used most often. The rate bands also leave gaps, such as rates above 3.0% but below 3.1%.

    It assumes everyone has a primary savings provider. There is no option for having no savings elsewhere, even though Q6 allows £0.

  10. Q8

    If Pennywell offered a savings rate matching or beating the top rates on the market, what proportion of the savings you hold elsewhere would you realistically move to a Pennywell pot?

    One answer
    • All or almost all of it
    • A large portion (around half or more)
    • A small portion
    • None of it
    • I don't have savings elsewhere
    Judges: ambiguous

    'Top rates on the market' specifies neither a rate nor comparable account conditions. 'All or almost all' overlaps with 'around half or more', so the response categories do not identify distinct proportions.

    The options overlap: 'A large portion (around half or more)' includes 'All or almost all of it'. 'Top rates on the market' is also left undefined.

  11. Q9

    To ensure you are reading carefully, please select 'A large portion' from the options below.

    One answer
    • All or almost all of it
    • A large portion
    • None of it
  12. Q10

    Is there anything else you would like to share about why you haven't used Pennywell savings pots, or what would encourage you to start?

    Open text
Sample plan

Email invitations sent to a randomized sample of Pennywell customers who have never opened a savings pot (drawn from the 360,400 eligible customers out of 420,000 total). Target is approximately 700 completed responses. Quotas will be applied across four age bands (18-34, 35-54, 55-74, 75 or over) with a minimum of n=150 completes per age band to ensure subgroup precision within +/- 5 percentage points at a 95% confidence level.