Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Gemini 4 Argon

Measured by Spring Prompt

Gemini 4 Argon: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
85 of 100
Analysis rating
985
Head to head, this task
won 11 of 28

Why it is not sound

The survey

  • A double-barrelled question: Q4
  • A double-barrelled question: Q10

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • Proposed a mistaken next step: Size the deposit opportunity by multiplying the mean balance held elsewhere by the number of customers

Findings the judges found unsupported

  • To the CFO ('Doesn't the rate being the top reason settle it?'): No. Low awareness, not the interest rate, is the primary barrier. 'The interest rate is too low' was only the top answer among the 39.4% minority who already knew about pots, equating to just 11.7% of all respondents (and 8.8% when weighted to the true customer age mix). In contrast, 60.6% of respondents (66.3% weighted, or ~239,500 customers) did not know savings pots existed.The open-text counts are inaccurate and inconsistent with the analysis's closing theme summary. The supplied answers contain 45 higher-rate mentions, not 37. This understates the qualitative rate theme, although the main awareness conclusion remains supported.
  • To the CFO ('How much could a rate rise bring in?'): Multiplying the £14,513 mean balance by the customer base vastly overstates potential inflows. Over a third of customers have no external savings, the mean is heavily skewed by older respondents, and among those with external savings the median balance is £1,000–£5,000. Realistic inflows from a rate rise to 3.85% are likely to sit in the £80m to £240m range rather than the ~£600m+ implied by unweighted averages.The £80m–£240m inflow forecast relies on invented historical conversion discounts and an unsupported assumption that converts stated intentions into full-balance transfers. No historical conversion evidence or joint relationship between savings balances and rate intent was supplied. Despite warning against mean-based sizing, the calculation still extrapolates an estimated mean balance across customers.
  • To the Head of Marketing ('Should we build a campaign around 75+ customers?'): No. While 75+ customers hold larger external balances, they make up only 2% of the non-pot customer base (7,220 people), our survey sample for this group is too small to rely on (Base: 20), and they show the lowest responsiveness to both nudges and rate rises. If targeting existing balances, focus on the 35–54 and 55–74 segments instead.The assertion that customers aged 75 or over have larger balances and the lowest responsiveness treats estimates from only 20 respondents as established segment characteristics. The sample supports describing these observations cautiously, not a high-confidence conclusion about that population.
  • Poor app discoverability and a confusing setup flow are actively blocking conversion, even among customers who know about pots or look for them after being prompted.The stated count of 27 discoverability or setup complaints is not supported by the supplied answers, which contain substantially more explicit examples. It also conflicts with the analysis's later count of 43. The underlying discoverability and setup finding is supported.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,487239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.2%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓124126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4646
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 7 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✗The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Launch the awareness and onboarding push first—combining home-screen visibility, a simplified setup flow, and the payday set-up nudge—before committing to an across-the-board rate rise to 3.85% AER. Two-thirds (66.3% weighted) of non-users do not know pots exist, and the payday nudge at 3.1% generates greater overall adoption intent (35.1% weighted) than raising the rate to 3.85% (28.9% weighted) at zero additional interest margin cost. If the business specifically needs to attract larger existing balances held at incumbent banks by 35–74-year-olds, test the 3.85% rate (or a targeted promotional bonus rate) in a controlled A/B experiment *after* or *alongside* fixing awareness and app discoverability, rather than raising the base rate blindly.

What would change it

We would shift to recommending an immediate rate rise to 3.85% AER alongside the awareness push if: (1) Treasury/Finance confirms that the primary commercial goal is maximising total £ deposit volume from large-balance transferrers (55–74s) rather than growing the active saver base, AND the net interest margin model shows that incremental deposits from aware rate-sensitive customers more than offset the 75bps cost across existing pot balances; or (2) a live in-app A/B test of the awareness and payday nudge shows that actual pot-opening and funding conversion at 3.1% falls below 5% because aware customers drop off upon seeing the 3.1% rate.

Findings

  1. To the CFO ('Doesn't the rate being the top reason settle it?'): No. Low awareness, not the interest rate, is the primary barrier. 'The interest rate is too low' was only the top answer among the 39.4% minority who already knew about pots, equating to just 11.7% of all respondents (and 8.8% when weighted to the true customer age mix). In contrast, 60.6% of respondents (66.3% weighted, or ~239,500 customers) did not know savings pots existed.

    High confidenceBase 700

    Q1 (Base: 700): 50.7% answered 'No' and 9.9% 'Not sure' to knowing about pots (60.6% total; 66.3% when age-weighted to the 361,000 non-pot population). Q3 (Base: 276 aware respondents): 29.7% (82 respondents, ±5.4pp MoE) chose 'The interest rate is too low', which is 11.7% of all 700 respondents (8.8% weighted). In open text (Q2, Base: 230), 124 respondents cited lack of awareness or needing an explanation, versus 37 mentioning a higher rate.

  2. An in-app payday savings nudge at the current 3.1% AER generates slightly higher overall saving intent than raising the rate to 3.85% AER (33.7% vs 29.9% unweighted; 35.1% vs 28.9% age-weighted), because the nudge strongly activates the unreached majority who were previously unaware of pots, whereas a rate hike mainly appeals to the older, already-aware minority.

    Medium confidenceBase 700

    Q5, Q6, Q7 (Base: 700): Baseline intent at 3.1% is 17.4% (17.3% weighted). A payday nudge at 3.1% (Q6) lifts likely intent to 33.7% (35.1% weighted), outperforming a rate rise to 3.85% at Q7 (29.9% unweighted, 28.9% weighted). Among those unaware of pots (Q1 'No', Base: 355), the nudge lifts intent to 42.5% compared to 24.2% for the rate rise. Among those already aware (Q1 'Yes', Base: 276), the rate rise lifts intent to 37.0% while the nudge has almost no effect (18.8% vs 18.1% baseline).

  3. To the CFO ('How much could a rate rise bring in?'): Multiplying the £14,513 mean balance by the customer base vastly overstates potential inflows. Over a third of customers have no external savings, the mean is heavily skewed by older respondents, and among those with external savings the median balance is £1,000–£5,000. Realistic inflows from a rate rise to 3.85% are likely to sit in the £80m to £240m range rather than the ~£600m+ implied by unweighted averages.

    Medium confidenceBase 700

    Q4 (Base: 700): 35.7% hold no external savings, and 5.3% said Don't know/Prefer not to say. Only 59.0% (413 respondents; 56.4% age-weighted) report an amount. While the mean among the 413 reporting an amount is £14,513 (£8,563 across all 700 unweighted), age-weighting to the actual customer base reduces the overall average to ~£6,550 per customer (~£11,600 per saver). Of the 413 reporting savings, 55.2% hold under £5,000 (25.4% under £1,000; 29.8% £1,000 to <£5,000). At Q7, 28.9% (weighted) say they are likely to save at 3.85% (11.6% 'Very likely', 17.3% 'Fairly likely', lifting baseline intent by +11.6pp). Applying historical conversion discounts (e.g., 10%–30% of stated intent translating to full balance transfers) yields ~£80m–£240m.

  4. To the Head of Marketing ('Should we build a campaign around 75+ customers?'): No. While 75+ customers hold larger external balances, they make up only 2% of the non-pot customer base (7,220 people), our survey sample for this group is too small to rely on (Base: 20), and they show the lowest responsiveness to both nudges and rate rises. If targeting existing balances, focus on the 35–54 and 55–74 segments instead.

    High confidenceBase 700

    Population data: 75+ is 2% of 361,000 (7,220 customers), whereas 55–74 is 13% (46,930) and 35–54 is 39% (140,790). Q4: 75+ respondents (Base: 20, very low precision, ±21pp MoE) report 65.0% holding £20k+, but 55–74 (Base: 200) has 21.5% holding £20k+ (~10,090 customers) and 35–54 (Base: 262) has 8.8% holding £20k+ (~12,390 customers). Furthermore, 75+ intent barely moves from baseline (20.0% at Q5) to 25.0% under either the payday nudge (Q6) or 3.85% rate rise (Q7).

  5. Poor app discoverability and a confusing setup flow are actively blocking conversion, even among customers who know about pots or look for them after being prompted.

    High confidenceBase 276

    Q3 (Base: 276 aware): 10.9% cited 'I wasn't sure how to set one up' (rising to 17.0% among 18–34s, Base: 47) and 11.6% 'I haven't got round to it'. In Q2 open text (Base: 230), 27 respondents specifically complained that pots are buried in menus (e.g., hidden under 'More') or that the multi-screen setup process is too fiddly and confusing, causing them to abandon creation.

  6. A notable minority of aware non-users hold back because of habitual inertia with traditional banks or fear that they cannot withdraw money instantly in an emergency.

    High confidenceBase 276

    Q3 (Base: 276): 19.6% chose 'I keep my savings with another bank or provider' (rising to 23.3% among 55–74s) and 7.6% chose 'I'm worried about how quickly I could get my money out'. Open text (Q2) includes 18 mentions of sticking with incumbent banks out of habit/trust and 9 mentions requiring explicit reassurance that withdrawals are instant 24/7.

Caveats

  • Sample age skew: Older customers were heavily over-represented among respondents (55–74s were 28.6% of the sample vs 13% of the target population; 18–34s were 31.1% vs 46% of the population). Because older respondents have higher awareness, larger external balances, and higher rate sensitivity, unweighted totals overstate both rate sensitivity and potential deposit inflows.
  • Non-response bias: Only 6.0% of invited customers completed the survey (700 of 11,600). Respondents who complete an email survey about savings may be more financially engaged than the average non-user.
  • Stated intent vs. actual behaviour: Questions Q5–Q7 measure stated likelihood on a hypothetical scale, which routinely overstates real-world adoption and balance transfers, and Q6/Q7 were asked after Q1–Q5 had already educated respondents about pots.
  • Small subgroup bases: The 75+ age group has only 20 respondents overall (±21.9pp MoE) and just 7 respondents at Q3, making point estimates for that segment noisy. Similarly, 18–34 at Q3 has a base of 47 (±14.3pp MoE).
  • No joint-effect measurement: The survey tested the payday nudge at 3.1% (Q6) and the rate rise to 3.85% (Q7) separately, so we cannot directly observe the additive lift or cannibalisation of doing both from the survey alone.

Next steps

  • decisionRedesign pot discoverability and simplify the pot creation flow (move pots out from under the 'More' menu onto the main account screen, reduce setup steps, and state clearly that withdrawals are instant 24/7).
  • experimentRun a controlled A/B experiment testing the payday set-up nudge and in-app awareness prompts at 3.1% AER vs. control across a representative sample of the 361,000 non-users.
  • experimentBuild a deposit economics model and run a pricing test (e.g., 3.1% vs 3.85% promotional/introductory rate for new pot openers) targeted at 35–74-year-olds.
  • researchConduct rapid usability testing (5–8 participants across age bands) on the revised pot creation screens and payday nudge copy.

Open-text themes it coded

Unaware of savings pots and need clear explanation / communication 126Higher / more competitive interest rate needed 45Better app discoverability and simpler setup flow 43No spare money / cannot afford to save 23Prefer existing bank / building society (habit and trust) 17Not interested in saving / using savings pots 15Need reassurance on instant / fast withdrawals 10

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Do you currently have an active current account with Pennywell?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Which of the following have you ever opened or used inside the Pennywell app? Please select all that apply.

    Any that apply
    • A Pennywell Debit Card
    • A Pennywell Savings Pot
    • A Pennywell Joint Account
    • A Pennywell Mortgage
    • None of the above

    Routing Shown only if S1 is: Yes

    Continues if A Pennywell Debit Card; A Pennywell Joint Account; None of the above

  3. S3

    Which age group do you belong to?

    One answer
    • Under 18
    • 18 to 34
    • 35 to 54
    • 55 to 74
    • 75 or over

    Continues if 18 to 34; 35 to 54; 55 to 74; 75 or over

  4. Q1

    Before taking this survey today, did you know that Pennywell offers instant-access 'Savings Pots' inside the app?

    One answer
    • Yes, I knew how they worked
    • Yes, I had seen or heard of them, but didn't know the details
    • No, I did not know Pennywell offered Savings Pots
    • Not sure
  5. Q2

    Do you know what interest rate (AER) Pennywell Savings Pots currently pay?

    One answer
    • Under 2.0% AER
    • Around 2.5% AER
    • Around 3.1% AER
    • Around 3.85% AER
    • 4.0% AER or more
    • I don't know what rate they pay

    Routing Shown only if Q1 is: Yes, I knew how they worked; Yes, I had seen or heard of them, but didn't know the details

  6. Q3

    Why have you not opened a Pennywell Savings Pot so far? Please select all that apply.

    Any that apply
    • I don't have spare money to put into savings right now
    • The interest rate is lower than what I can get elsewhere
    • I didn't know what interest rate they paid
    • I haven't got around to setting one up yet
    • I wasn't sure where to find them or how to open one in the app
    • I prefer to keep my savings with my existing savings provider
    • I didn't know if I could withdraw my money instantly
    • I don't trust app-only banks with my savings
    • Other reason

    Routing Shown only if Q1 is: Yes, I knew how they worked; Yes, I had seen or heard of them, but didn't know the details

  7. Q4

    In your own words, what is the single main thing that has stopped you from opening a Pennywell Savings Pot, or what would have encouraged you to open one?

    Open text
    Judges: double-barrelled

    It combines the main historical barrier with a hypothetical encouragement in one response. Respondents can answer different parts, so the answers do not consistently measure one construct.

    It asks for two different things in one answer box: the main barrier, or what would have encouraged opening. Answers will mix the two and can't be told apart.

  8. Q5

    Now that you know Pennywell offers instant-access Savings Pots inside the app paying 3.1% AER (where you can add or withdraw money at any time), how likely are you to open one in the next 3 months?

    Scale

    Scale 1-5: 1 - Very unlikely to 5 - Very likely

  9. Q6

    If Pennywell sent you an optional prompt in the app on payday to set up a Savings Pot in two taps and move spare cash across (paying 3.1% AER), how likely would you be to open one?

    Scale

    Scale 1-5: 1 - Very unlikely to 5 - Very likely

  10. Q7

    If Pennywell Savings Pots paid 3.85% AER (instead of 3.1% AER), how likely would you be to open one in the next 3 months?

    Scale

    Scale 1-5: 1 - Very unlikely to 5 - Very likely

  11. Q8

    Roughly how much money do you currently hold in easy-access or instant-access savings or cash accounts with OTHER banks or building societies (outside Pennywell)?

    One answer
    • £0 (I don't have easy-access savings elsewhere)
    • £1 to £499
    • £500 to £1,999
    • £2,000 to £4,999
    • £5,000 to £9,999
    • £10,000 to £24,999
    • £25,000 to £49,999
    • £50,000 or more
    • Prefer not to say
  12. Q9

    Think about the savings you hold outside Pennywell. What interest rate are you currently getting on the main account where you keep those easy-access savings?

    One answer
    • Less than 2.0% AER
    • 2.0% to 3.0% AER
    • 3.1% to 3.79% AER
    • 3.8% to 4.49% AER
    • 4.5% AER or more
    • I don't know the rate

    Routing Shown only if Q8 is: £1 to £499; £500 to £1,999; £2,000 to £4,999; £5,000 to £9,999; £10,000 to £24,999; £25,000 to £49,999; £50,000 or more

  13. Q10

    Roughly how much of your external savings (in £) would you consider moving into a Pennywell Savings Pot over the next 3 months if the rate stays at 3.1% AER, and how much if the rate increased to 3.85% AER? Which best describes what you would do?

    One answer
    • I would move most or all of my external savings at the current 3.1% AER rate
    • I would move some of my external savings at 3.1% AER, and more if it paid 3.85% AER
    • I would NOT move savings at 3.1% AER, but I WOULD move most or all of my savings at 3.85% AER
    • I would NOT move savings at 3.1% AER, but I WOULD move some of my savings at 3.85% AER
    • I would not move any of my external savings at either 3.1% AER or 3.85% AER

    Routing Shown only if Q8 is: £1 to £499; £500 to £1,999; £2,000 to £4,999; £5,000 to £9,999; £10,000 to £24,999; £25,000 to £49,999; £50,000 or more

    Judges: double-barrelled

    It requests amounts under two rates but permits only one combined categorical answer. The options omit moving the same partial amount at both rates and force anyone selecting partial movement at 3.1% to say they would move more at 3.85%, favouring the rate-rise hypothesis. It asks for amounts in pounds but supplies only undefined 'some' and 'most or all' categories.

    The stem asks for two amounts in pounds, one at 3.1% and one at 3.85%, but allows only one combined answer, and none of the options is in pounds. The combinations are also incomplete: there is no option for 'some at 3.1% and no more at 3.85%'. Some respondents are therefore forced into an answer that doesn't fit.

Sample plan

POPULATION & SAMPLING FRAME: The target population is the ~361,200 Pennywell customers (86% of 420,000) who have never opened a savings pot. Invite a stratified random sample via email directly from the account database (filtering out the 14% who already hold or have held a pot, plus S2 as a double-check against false recall/database lag; anyone selecting 'A Pennywell Savings Pot' or 'A Pennywell Mortgage' [foil] in S2 is screened out).

SAMPLE SIZE vs PRECISION CONSTRAINT (CRITICAL CALLOUT FOR THE BOARD): Standard sample size for ±5 percentage points margin of error at 95% confidence for a single group is n=384. Across all 4 age bands (18-34, 35-54, 55-74, 75+), achieving ±5pp per band strictly requires ~1,536 completes (384 per band). With the brief's constraint of ~700 total completes:

  • We will use disproportionate stratified random sampling with equal quotas of n=175 completes per age band (Total N=700). At n=175 per band, the maximum margin of error (at p=50%) is ±7.4 percentage points at 95% confidence (and ±5.2pp when a proportion is around 15% or 85%). Overall results (N=700) will have a margin of error of ±3.7pp.
  • If the Board strictly requires ±5.0pp worst-case precision inside every single age band, we must recommend increasing the fieldwork cap from 700 to 1,540 completes (385 per age band), which costs almost nothing extra since we are emailing our own customer database.

WEIGHTING: Because we sample equal numbers per age band (especially oversampling the 75+ segment of an app-only bank), overall population estimates (O1-O3) will be post-stratification weighted back to the true age distribution of Pennywell's non-pot customer base.