Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Qwen4 Max

Measured by Spring Prompt

Qwen4 Max: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✗ Not sound
Research score
66 of 100
Analysis rating
1,373
Head to head, this task
won 2 of 15

Why it is not sound

The survey

  • A research objective no question serves

The analysis

  • Made a recommendation that does not follow from the data
  • Missed a trap: Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.8%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,800239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.2%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓126126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4646
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 7 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✗Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Findings

  1. Awareness, not price, is the binding constraint for most non-users. About two-thirds of the 361,000 customers who have never opened a pot do not know pots exist, so no rate change can reach them. An awareness and onboarding push is the only lever that addresses this group and should be the backbone of the decision.

    High confidenceBase 700

    Q1: No 50.7% + Not sure 9.9% = 60.6% of 700 unweighted; age-weighted to the database (18-34 46%, 35-54 39%, 55-74 13%, 75+ 2%) = 66.4%, about 239,800 of 361,000. Awareness is lowest where the customers are: 21.6% of 18-34 knew (n=218) vs 60.0% of 55-74 (n=200); weighted awareness 33.6% vs unweighted 39.4%. 126 of 230 open texts ask to be told what pots are, including 46 of 72 from 18-34.

  2. The CFO's premise is a base error. 'Rate is the top reason' holds only inside the 39.4% who already knew about pots, and even there it is 29.7%, not a majority. Across all non-users rate is the main stated barrier for roughly 1 in 11 (8.8% weighted, about 32,000 customers) - a real but minority segment.

    High confidenceBase 276

    Q3 shown only to Q1=Yes (base 276). 29.7% of 276 = 82 people = 11.7% of 700. Weighted: 18-34 21.6%x14.9%=3.2%; 35-54 38.9%x27.5%=10.7%; 55-74 60.0%x37.5%=22.5%; 75+ 35.0%x28.6%=10.0%; total 8.8% (about 31,700 of 361,000). Among aware 18-34 (n=47) rate is 14.9%, below 'no money left over' (21.3%). Margin of error on 29.7% is +/-5.4pp (24.3%-35.1%).

  3. The two levers work on different people: the payday nudge moves the unaware, the rate rise moves the already-aware. This argues for sequencing (awareness first, targeted pricing second), not choosing one lever on an average.

    High confidenceBase 700

    Net likely (4-5): Q1=No (n=355) Q5 17.2% -> Q6 42.5% (+25.3pp) but Q7 24.2% (+7.0pp). Not sure (n=69) 15.9% -> 47.8% nudge (+31.9pp) vs 30.4% rate. Q1=Yes (n=276): nudge 18.1% -> 18.8% (+0.7pp), rate 18.1% -> 37.0% (+18.9pp). Pooled 424 unaware/not-sure: 17.0% baseline, 43.4% nudge, 25.2% rate.

  4. On stated intent the nudge beats the rate rise overall, and the gap widens after age weighting. But these are hypothetical intentions in fixed order, so treat the ranking as a hypothesis to test behaviourally, not a forecast of funded pots.

    Medium confidenceBase 700

    n=700: Q5 (3.1%) 17.4%; Q6 (3.1% + payday prompt) 33.7%; Q7 (3.85%) 29.9%. Age-weighted: 17.4%, 35.1%, 28.9% - lead grows from 3.8pp to 6.2pp because responsive 18-34s (Q6 38.5%) are under-represented. Q5->Q6->Q7 fixed order, no rotation, no combined arm; Q2 defined pots before Q5, so the 17.4% baseline is already post-awareness and Q6 conflates explanation with prompting.

  5. The £14,500 figure cannot be multiplied by anything to size the opportunity. It is a top-coded midpoint mean of a skewed distribution from an older-skewed sample. Population-weighted, savings elsewhere average about £6,500 per non-user, not £14,500.

    Medium confidenceBase 413

    Q4 mean £14,513 from band midpoints with £50,000+ coded £75,000, among only 413 of 700 (59.0%) giving an amount; median in £1,000-<£5,000; 55.2% of the 413 under £5,000; 35.7% of all 700 hold nothing; 12.6% hold £20,000+. Age-weighted rebuild gives about £6,550 per non-user (£11,600 among the 56.5% with any savings), roughly £2.4bn across 361,000 vs £5.2bn from naive 361,000 x £14,500. Stock is not transferable flow: much sits in ISAs and long-standing relationships.

  6. Balances and responsiveness point in opposite directions, so a rate-led inflow forecast built on big balances will disappoint. Those most likely to act on a prompt have the least money; those with money are most attached to their incumbent and least moved by a nudge.

    Medium confidenceBase 700

    18-34 (46% of population, n=218): 45.9% no savings elsewhere, 4.1% £20,000+, yet Q6 net likely 38.5% (highest). 55-74 (13%, n=200): 21.5% £20,000+, Q6 30.0%, Q7 33.5%, open texts dominated by rate thresholds and loyalty (P016 Leeds, P339 Barclays, P451 Lloyds, P592 building society, P639 Halifax, P649 Nationwide). 16 open texts cite habit/trust in an incumbent.

  7. Do not build a campaign around the 75-or-overs. The 'big balances' finding rests on 20 respondents (13 in the £20,000+ bands) representing 2% of the population, and they are among the least responsive to either lever.

    High confidenceBase 20

    Q8 75+ = 2.9% of sample (n=20) vs 2% of population (about 7,200). Q4 75+: 65.0% £20,000+ = 13 people; Q3 base 7, so 42.9% 'savings elsewhere' is 3 people. Q6 25.0% and Q7 25.0% (5 and 5 respondents). Five of seven open texts (P271, P439, P468, P695, P697) ask what pots are. Age for prefer-not-say was backfilled from account records, adding noise to a tiny cell.

  8. There is no statistically reliable evidence that rate sensitivity differs between 35-54 and 55-74, so do not build separate pricing or targeting on that contrast.

    High confidenceBase 222

    Q3 'rate too low': 35-54 27.5% (28 of 102) vs 55-74 37.5% (45 of 120); difference 10.0pp, pooled two-proportion z = 1.59, p = 0.11, not significant at 95%. Overall Q3 rate share 29.7% +/-5.4pp (n=276), before any design effect from weighting. The 75+ cell (n=7) supports no comparison.

  9. Discovery and set-up friction will leak value from any awareness push and are cheap to fix. Some customers knew about pots, tried to open one and abandoned. Promoting a feature people cannot find or complete buys complaints, not deposits.

    Medium confidenceBase 276

    Aware (n=276): 10.9% 'wasn't sure how to set one up', 11.6% 'haven't got round to it'; aware 18-34 (n=47) setup 17.0%. Open text: 29 say pots are buried/hard to find (P088, P150, P164, P180, P299, P471, P484, P565, P580, P621, P661) and 13 describe set-up friction including abandonment (P039 'gave up', P048 'bailed', P141 'abandoned it', P483 'five-screen process', P661 'spent ten minutes looking... quit'). P088, P150, P164, P471, P661 all answered Yes at Q1.

  10. 3.85% may not clear the threshold for the people the rate rise targets. The rate-sensitive group talks in 4-5% terms and beating their incumbent by a clear margin, so 75bp risks paying more on the whole book while still losing switchers. Qualitative signal, not a price test.

    Low confidenceBase 230

    45 of 230 open texts mention rate. Thresholds: P093 'something like 5%', P178 'at least 4%', P640 'over 4% elsewhere', P449 'a full percentage point more than Nationwide', P499 'near the top of the best-buy tables', P352 'a couple of percent'. 16 of the 45 come from people who had never heard of pots and ask for an explanation first (P054, P070, P194, P384, P427, P669) - rate is a post-information reaction, not a pre-existing barrier.

  11. A substantial minority is unreachable by either lever: no spare money, or no interest in saving in an app. Cap expectations and deprioritise them rather than spend prompts on them.

    Medium confidenceBase 700

    Q3 (n=276): 13.4% 'no money left over' (21.3% among aware 18-34, n=47). Open text: 23 say nothing spare (P053, P058, P080, P089, P110, P201, P218, P267, P336, P346, P362, P395, P401, P481, P519, P522, P611, P626, P631, P648, P678, P688, P699) and 15 are not interested in saving at all. 'Very unlikely' barely moves: 24.9% Q5, 20.7% Q6, 19.0% Q7 (n=700).

  12. Instant-access reassurance is a low-cost copy fix that removes a stated barrier for a small but firm group; put it in onboarding creative rather than treating it as a product change.

    Medium confidenceBase 276

    Q3 (n=276): 7.6% worried about speed of access (10.0% of 55-74; 14.3% of 75+ but n=7). Ten open texts ask for certainty: P017, P036, P063, P210, P231, P338, P371, P541, P675, P681 - e.g. P231 'at 2am on a Sunday', P371 'another bank once took three working days'.

Open-text themes it coded

Did not know pots exist / want Pennywell to explain what they are 50Rate too low / explicit rate threshold or best-buy comparison 18No spare money to save 23Pots hard to find in the app 11Set-up friction or abandoned attempt 5Habit, loyalty or trust in existing provider 6Need certainty of instant access 10

The survey it planned

1 screening question and 10 questions, as the model wrote them.

  1. S1

    First, a quick question so we can make sure this survey is right for you. Which of these age bands are you in?

    One answer
    • Under 18
    • 18 to 34
    • 35 to 54
    • 55 to 74
    • 75 or over
    • Prefer not to say

    Continues if 18 to 34; 35 to 54; 55 to 74; 75 or over

  2. Q1

    Before you opened this survey, did you know that Pennywell offers 'savings pots' in its app? A savings pot is a separate instant-access savings account you can open inside the Pennywell app. Please answer about what you knew before today, not what you have just read.

    One answer
    • Yes – I knew about savings pots and understood what they were
    • I had heard of savings pots but did not really know what they were
    • No – I did not know Pennywell offered savings pots
    • Not sure / can't remember
  3. Q2

    You said you had heard of savings pots but have never opened one. Which of the following are reasons why you have not opened one? Please select up to 3 that matter most to you.

    Any that apply
    • I already save elsewhere and do not want to move it
    • The interest rate is lower than I can get elsewhere
    • I did not know what interest rate savings pots pay
    • I do not have spare money to put aside
    • I keep meaning to but have not got round to it
    • Setting one up seems like too much effort or too many steps
    • I do not know how to open one
    • I prefer to keep savings away from my everyday spending account
    • I am not comfortable keeping savings with an app-only bank
    • I am unsure my money would be safe or easy to withdraw
    • I do not need to save / I am not interested in saving
    • Other reason (please write in)
    • Don't know / can't remember

    Routing Shown only if Q1 is: Yes – I knew about savings pots and understood what they were; I had heard of savings pots but did not really know what they were

  4. Q3

    As far as you know, what interest rate do Pennywell savings pots currently pay? Please pick the closest answer, even if you are guessing.

    One answer
    • Under 1% AER
    • 1% to under 2% AER
    • 2% to under 3% AER
    • 3% to under 4% AER
    • 4% AER or more
    • I have no idea

    Routing Shown only if Q1 is: Yes – I knew about savings pots and understood what they were; I had heard of savings pots but did not really know what they were

  5. Q4

    Here is a short description of savings pots so that everyone is answering about the same thing: a savings pot is a separate instant-access savings account inside the Pennywell app. You can open one in the app in about a minute, move money in and out whenever you like, and there are no fees and no minimum balance. Eligible deposits are protected by the FSCS up to £85,000 per person. Savings pots currently pay 3.1% AER (variable, so it can change). With pots paying 3.1% AER, how likely or unlikely would you be to open a savings pot in the next 3 months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  6. Q5

    Now imagine Pennywell savings pots paid 3.85% AER (variable) instead of 3.1% AER. Everything else about pots stays exactly the same. At 3.85% AER, how likely or unlikely would you be to open a savings pot in the next 3 months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  7. Q6

    If you could only have one of these, which would make you more likely to open a Pennywell savings pot?

    One answer
    • Pots paying 3.85% AER, but you have to find them in the app yourself and set them up yourself
    • Pots paying 3.1% AER, with a reminder in the app on payday that takes you straight to a set-up taking under a minute
    • Neither – I would not open a pot in either case
    • Don't know
  8. Q7

    Thinking about all the cash savings you hold outside Pennywell – savings accounts, cash ISAs, credit unions and similar – but not pensions, investments or property: roughly how much do you have in total? Please give your best estimate in pounds, to the nearest £100. Enter 0 if you have no cash savings outside Pennywell. Leave the box blank if you would rather not say.

    Number
  9. Q8

    Of the cash savings you hold outside Pennywell, how much could you move into an instant-access account within a week without losing interest or paying a penalty?

    One answer
    • None of it
    • Up to about a quarter of it
    • About half of it
    • About three-quarters of it
    • Almost all of it
    • I have no cash savings outside Pennywell
    • Don't know
  10. Q9

    If you did open a Pennywell savings pot in the next 3 months, how much would you expect to put into it to start with? Please give your best estimate in pounds. Enter 0 if you would not put any money in. Leave the box blank if you would rather not say.

    Number
  11. Q10

    In a typical week, how often do you open the Pennywell app?

    One answer
    • Most days
    • About once a week
    • Two or three times a month
    • Less than once a month
    • I hardly ever or never open the app
    • Don't know
    Judges: ambiguous

    The response categories leave no clear answer for customers opening the app two or three days a week: this is neither 'Most days' nor 'About once a week'. Such customers must interpret or stretch a category, making the frequency measure unreliable.

    The question asks about a 'typical week', but several options are monthly frequencies ('Two or three times a month', 'Less than once a month'). These cannot be answered within a weekly frame, so the reference period is unclear.

Sample plan

FRAME: Pennywell current-account holders aged 18+ whose account record shows no savings pot ever opened (approx. 86% of 420,000 = c.361,000 customers). Eligibility by pot status is taken from the account database, NOT from a self-report question, because asking 'have you opened a pot?' before Q1 would destroy the awareness measure. Refresh the extract on launch day, invite a stratified random sample by age band (S1), and at analysis drop/flag anyone whose record shows a pot opened between extract and response, plus closed accounts and staff accounts. PRECISION CONFLICT – MUST BE RESOLVED BEFORE FIELDWORK: ±5pp at 95% confidence within each of four age bands requires n=385 per band (1.96²×0.25/0.05² = 384.2), i.e. c.1,540 completes, plus a buffer for exclusions = target 1,600. The stated 700 completes cannot deliver this: with equal quotas of 175 per band the worst-case margin is ±7.4pp, and with proportionate allocation the 55-74 and 75+ bands would be far worse. Recommendation to the board in writing: either fund c.1,600 completes, or accept ±7.4pp per band (700 completes, 175 per band) and treat age-band differences as indicative only. Do not silently switch to proportionate allocation. QUOTAS (recommended option): hard quota of 385 completes in each of 18-34, 35-54, 55-74, 75+; no substitution between bands; monitor daily. FEASIBILITY CHECK: before launch, count non-openers by band in the database. An app-only bank is likely to have a thin 75+ base; if fewer than c.4,000 eligible 75+ customers exist, invite all of them (census), expect a low response rate, and report the achieved confidence interval for that band honestly rather than claiming ±5pp. Consider a paper/telephone top-up for 75+ only if the board approves a mode change (note it breaks the online-only constraint and introduces mode effects). INVITATIONS: assume 8-12% usable response from a customer email list; invite c.4,000-5,000 randomly selected non-openers per band (c.18,000 total) in waves, personalised email from Pennywell with survey hosted on a Pennywell-branded domain, no incentive or a small non-cash prize draw disclosed as unrelated to answers, two reminders at day 4 and day 9, fieldwork 12-14 days, closes on quota. WEIGHTING: within-band estimates are unweighted; for all-customer totals apply design weights so the age-band profile matches the non-opener population (report the design effect and the resulting wider CI for totals). Optionally rake on tenure and average balance if response is skewed. Report unweighted bases everywhere. QUALITY: no attention-check item is used, to protect the 10-question budget for substantive measures; instead screen out completes under 90 seconds, duplicate account IDs, straight-lined scale pairs with identical verbatims, and implausible savings values (winsorise, do not delete). BIAS SAFEGUARDS (given the CFO's stated prior): pre-register the analysis plan and decision thresholds in the O2 entry before fieldwork; the questionnaire measures awareness before any rate is mentioned, asks perceived rate (Q3) before revealing the actual rate, and includes a forced trade-off (Q6) so 'rate' cannot win by default; the analyst reports all pre-registered comparisons including those that contradict the rate-rise hypothesis. Timing: c.4 minutes median (10 closed/numeric items, no long open-ends), piloted with 15-20 customers before launch.