Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / GPT-6.1 Sol

Measured by Spring Prompt

GPT-6.1 Sol: Why Pennywell customers don't use savings pots

The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →

Verdict
✓ Sound research
Research score
100 of 100
Analysis rating
1,650
Head to head, this task
won 9 of 9

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of respondents said they knew, before the survey, that they could open savings pots (Yes at Q1)?✓39.4%39.4%
Among respondents who were asked Q3, what percentage chose 'The interest rate is too low'?✓29.7%29.7%
As a percentage of all respondents, how many knew about pots and chose 'The interest rate is too low' at Q3?✓11.7%11.7%
Weighted to the age mix of all customers who have never opened a pot, what percentage of them would know about pots and choose 'The interest rate is too low' at Q3?✓8.77%8.8%
Weighted to the age mix of all 361,000 customers who have never opened a pot, about how many of them did not know they could open savings pots (No or Not sure at Q1)?✓239,700239,702
Among respondents who reported an amount held in savings elsewhere (excluding None, Don't know and Prefer not to say), what percentage hold less than £5,000?✓55.2%55.2%
What percentage of all respondents said they would be likely (fairly or very) to start saving in a pot if the rate rose to 3.85% AER?✓29.9%29.9%
Among respondents asked Q3, is the difference in the share choosing 'The interest rate is too low' between those aged 35-54 and those aged 55-74 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
What is the 95% margin of error, in percentage points, on the share of respondents asked Q3 who chose 'The interest rate is too low'?✓5.4 points5.39 points
How many of the open-text answers say the respondent did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓126126
How many of the open-text answers from respondents aged 18-34 say they did not know about savings pots or what they are, or want Pennywell to tell them about pots?✓4646
What percentage of the customers invited completed the survey?✓6%6%

Traps in the data 8 of 8 handled

  • ✓Q3 was asked only of the 276 respondents who knew about pots. 'The interest rate is too low' leads there (30%), but that is 12% of all respondents (about 9% weighted), while 61% of all respondents did not know pots existed (No or Not sure).
  • ✓Customers aged 55 or over are 15% of non-users but 31% of respondents; they know about pots more and cite the rate more, so unweighted figures overstate awareness and the rate reason (weighted, awareness falls from 39% to 34%).
  • ✓The mean amount held elsewhere (£14,513) is pulled up by a few large balances; the median is in the £1,000 to less than £5,000 band and 55% of those reporting an amount hold less than £5,000. Many hold nothing elsewhere.
  • ✓30% say they would be likely to start saving in a pot at 3.85% (17% at the current 3.1%); these are stated intentions about a hypothetical rate.
  • ✓Respondents aged 75 or over (base 20) show 65% holding £20,000 or more elsewhere, against 13% overall.
  • ✓Only 700 of 11,600 invited customers completed (about 6%); those who respond to an email about savings may be more engaged with saving than those who do not.
  • ✓The chief financial officer believes customers don't use pots because the rate is uncompetitive.
  • ✓Among those asked Q3, the rate reason is 10 points higher for 55-74s (38%, base 120) than for 35-54s (27%, base 102), but the difference is not significant at 95%.

The analysis

Recommendation

Proceed with a controlled awareness-and-onboarding push at 3.1%, including a visible route to pots, plain-English instant-access information and an optional payment-triggered setup nudge. Do not approve a blanket rise to 3.85% on this survey alone. In parallel, test the actual rate proposition against the onboarding push and their combination, subject to pricing, regulatory and operational approval. The survey supports two different mechanisms: discovery and activation among previously unaware customers, and rate competitiveness among already-aware customers. It does not establish their actual incremental deposits or economics. Keep 75-plus customers included with accessible communication, but do not make them the main campaign audience on this evidence. A 0.75-percentage-point rise adds roughly £7.50 annual interest per £1,000 of balance maintained for a year; the business case must include the cost on existing and otherwise-acquired balances, not just deposits caused by the increase.

What would change it

Recommend the rate rise, or both interventions, if a controlled behavioural test demonstrates sufficient incremental external deposits and sustained balances to clear the CFO's funding-value and profitability thresholds after additional interest, implementation and campaign costs. Evidence that 3.85% remains below customers' relevant alternatives would weaken that proposal. If onboarding improves awareness but not funded use or net deposits, shift investment toward competitiveness or other barriers. A 75-plus-led campaign would require a larger, representative segment study and behavioural evidence showing a material, reachable and economically attractive opportunity, not just high balances among 20 respondents.

Findings

  1. Make awareness and discoverability the first intervention to test. Most surveyed non-users did not know, or were unsure, that pots existed; adjusting for the customer age mix makes this gap larger.

    Medium confidenceBase 700

    At Q1, 276/700 knew about pots and 424/700 did not know or were unsure. Age-weighting gives approximately 33.6% aware and 66.4% unaware or unsure: about 239,700 of the 361,000 non-users if respondents represent customers within each age band. Among 18-34s, 78.4% were unaware or unsure; this age group constitutes 46% of the customer population. Awareness or explanation appears in 126/230 open answers, including 46 from 18-34s.

  2. CFO: no, the rate being the top listed reason does not settle the decision. It is a meaningful barrier among customers already aware of pots, not a diagnosis of all non-users.

    High confidenceBase 276

    Only the 276 respondents who already knew about pots received Q3. Of these, 82 selected the rate: 29.7%, with a nominal 95% margin of error of approximately ±5.4 percentage points. These 82 represent 11.7% of all 700 respondents, or 8.77% after age-weighting. Rate concerns also appear in 45 open answers, including answers that combine them with awareness or provider inertia. The 35-54 versus 55-74 difference at Q3, 28/102 versus 45/120, is not significant at 95% using a conventional two-proportion test.

  3. The payday-style nudge has the stronger overall stated-intent signal, while the rate rise has a stronger signal among those already aware. Test both mechanisms rather than assuming one universal barrier.

    Medium confidenceBase 700

    Likely to start saving: 17.4% at 3.1%, 33.7% with the payment-triggered nudge at 3.1%, and 29.9% at 3.85%, all based on 700 respondents. Age-weighted figures are approximately 17.4%, 35.1% and 28.9%. Among those previously unaware, nudge intent is 42.5% versus 24.2% for the higher rate; among those previously aware, it is 18.8% versus 37.0%. These are hypothetical responses, not measured conversion or causal treatment effects.

  4. CFO: this survey cannot provide a defensible forecast of deposits attracted by a rate rise. The £14,513 average is neither an all-customer average nor an amount available to transfer.

    High confidenceBase 700

    The £14,513 figure covers only 413 respondents reporting positive savings elsewhere, uses band midpoints, and assigns £75,000 to every £50,000-plus answer. Another 250 respondents reported no savings elsewhere. Of the 413 positive savers, 228, or 55.2%, hold less than £5,000. Including the 250 zero balances gives an unweighted midpoint estimate of about £9,041 among 663 respondents with a known balance, still not a population or inflow estimate. Q7 provides intention to open, not transfer amounts, transfer timing or incremental deposits.

  5. Marketing: do not build the main campaign around customers aged 75 or over. Their balances justify exploratory work, but the segment is small and the survey evidence is too thin for a campaign strategy.

    Low confidenceBase 20

    Among just 20 respondents aged 75-plus, 17 reported positive savings elsewhere and 13 reported £20,000 or more. Only seven received Q3. Higher-rate and nudge intent are both 25%, representing five people each. This age group is only 2% of the non-user population, approximately 7,220 customers. Customers aged 35-54 and 55-74 are much larger segments, approximately 140,790 and 46,930 respectively.

  6. An awareness message needs a clear route to opening a pot and credible access information; promotion alone will leave important friction unresolved.

    Medium confidenceBase 230

    Forty open answers describe difficulty finding pots or setup friction; ten request reassurance about withdrawal speed. Among the 276 aware respondents at Q3, 10.9% selected setup uncertainty, 11.6% not getting round to it and 7.6% access concerns. Examples include abandoned setup attempts and pots being buried in menus. These reports identify issues to investigate, not verified interface defects affecting every customer.

  7. Do not assume every non-user is a convertible saver. Use optional, dismissible prompts and distinguish lack of money from lack of awareness.

    Medium confidenceBase 700

    At Q4, 250/700 reported no savings elsewhere. Twenty-three open answers explicitly describe having no spare money, and 15 express disinterest in saving or using pots. Eighteen mention an existing provider, loyalty, habit or provider-specific rate comparison. Neither better onboarding nor 3.85% necessarily removes these barriers.

Caveats

  • Only 700 of 11,600 invitees completed: 6.0%. Random invitation does not make respondents representative. Nonresponse may relate to savings, engagement or interest in pots; age-weighting cannot correct those differences.
  • The sample is older than the population: 55-74s are 28.6% of completes but 13% of non-users, while 18-34s are 31.1% versus 46%. Population-facing estimates use the supplied age mix and assume representativeness within age bands.
  • The weighted unaware-customer count is an approximate projection, not a database count. Calculations use reconstructed integer counts consistent with the rounded tables where possible.
  • Q3 was asked only of previously aware respondents and forced one main reason. Its ranking excludes unaware customers and does not capture all overlapping barriers. The weighted rate-of-all answer measures the joint event of awareness and selecting rate, not every customer who might care about rates.
  • The statistical-significance answer uses an independent two-proportion test: approximately z=1.59 and p=0.11. The ±5.4-point margin is a nominal normal-approximation sampling margin for 82/276; it does not cover nonresponse, measurement error or weighting.
  • Q5-Q7 are hypothetical questions in a fixed order, after respondents were introduced to pots. Q5 is therefore not an untreated awareness baseline. Q6 measures a payment-triggered suggestion, not an isolated awareness campaign, and its wording is broader than a salary-only payday nudge.
  • The same respondents answered Q5-Q7. Marginal tables do not show individual switching or support a paired significance test. Intent differences are not causal conversion estimates, and no question measures the combination of a rate rise and nudges.
  • The open-text themes are overlapping qualitative codes of 230 answers, not representative prevalence estimates for all 700 respondents or all customers. General requests for explanation are distinguished from explicit reports of setup or navigation friction.
  • Savings amounts are banded, exclude pensions and investments, and include cash ISAs. Midpoint estimates are sensitive to the assumed £75,000 value for the open-ended top band. Cash ISA balances are not automatically transferable on equivalent tax terms.
  • There is no respondent-level link here between balance, intention, existing provider rate and amount willing to transfer. Stated willingness to open a pot does not imply moving all external savings or adding new deposits to Pennywell.
  • The 75-plus results rest on 20 completes; each respondent represents five percentage points. Some age classifications use account records rather than a stated Q8 age. Age alone is not a reliable proxy for savings needs or app accessibility.
  • Some rate-focused open answers request 4%, 5%, or a margin above their existing provider. They support investigating competitiveness, not a conclusion that exactly 3.85% is sufficient.

Next steps

  • decisionApprove development and a limited rollout of clear pots discovery, simplified onboarding and optional payment-triggered nudges, with a holdout group.
  • experimentRun a powered, randomized 2-by-2 test of onboarding/nudges on or off and 3.1% versus 3.85%, with appropriate approval for any rate differentiation.
  • researchBuild the deposit and profitability model from behavioural test data and actual balance flows.
  • researchConduct usability sessions covering unaware customers, abandoned setup attempts and access concerns, including older customers.
  • researchValidate the population opportunity and competitive proposition using respondent-level analysis and targeted follow-up.
  • monitorTrack outcomes beyond campaign clicks and initial pot opening.

Open-text themes it coded

Unaware of pots, unsure what they are, or requesting Pennywell explain them 126Higher or more competitive interest rate needed 45Reported setup friction or difficulty finding pots 40Reassurance about immediate access and withdrawal speed 10No spare money or financial circumstances prevent saving 23Disinterest in saving or using pots 15Existing provider preference, habit or provider-specific rate comparison 18

The survey it planned

1 screening question and 10 questions, as the model wrote them.

  1. S1

    Are you currently a Pennywell customer aged 18 or over?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. Q1

    Before starting this survey, had you heard of Pennywell savings pots?

    One answer
    • Yes, and I knew what they were
    • Yes, but I did not know what they were
    • No
    • Not sure
  3. Q2

    Before starting this survey, what were your reasons for not opening a Pennywell savings pot? Select all that apply.

    Any that apply
    • I did not know enough about how pots worked
    • I did not know where to find pots in the app
    • I thought setting one up would take too much time or effort
    • I meant to open one but had not got around to it
    • The interest rate was too low
    • I did not know the interest rate
    • I preferred to keep my savings with another provider
    • I preferred to keep my savings separate from my everyday banking
    • I was concerned about the safety of my money
    • I did not have money available to save
    • I did not need another savings account
    • Another reason
    • No particular reason
    • Not sure

    Routing Shown only if Q1 is: Yes, and I knew what they were; Yes, but I did not know what they were

  4. Q3

    Approximately how much money do you personally hold outside Pennywell in savings accounts that let you withdraw your money immediately without a charge? Include savings held in current accounts, but exclude investments, pensions, and accounts with withdrawal restrictions or charges. For joint accounts, count only your share.

    One answer
    • £0
    • £1–£499
    • £500–£999
    • £1,000–£2,499
    • £2,500–£4,999
    • £5,000–£9,999
    • £10,000–£24,999
    • £25,000–£49,999
    • £50,000 or more
    • Not sure
    • Prefer not to say
  5. Q4

    What annual interest rate does the account holding the largest share of those savings currently pay? If you know its AER, use that. AER means annual equivalent rate and allows savings rates to be compared.

    One answer
    • No interest
    • Above 0% but below 3.1%
    • 3.1% to below 3.85%
    • 3.85% to below 4.5%
    • 4.5% or more
    • Not sure
    • Prefer not to say

    Routing Shown only if Q3 is: £1–£499; £500–£999; £1,000–£2,499; £2,500–£4,999; £5,000–£9,999; £10,000–£24,999; £25,000–£49,999; £50,000 or more

  6. Q5

    What is your age?

    One answer
    • 18–34
    • 35–54
    • 55–74
    • 75 or over
    • Prefer not to say
  7. Q6

    Imagine Pennywell offers the following: Savings pots are instant-access savings accounts inside the Pennywell app. You can withdraw your money whenever you want. They pay 3.1% AER. Pots are available under a tab in the app, with no additional prompts or payday reminder to help you find and set up a pot. How likely would you be to open a pot and put money into it within the next 30 days?

    Scale

    Scale 1-5: Very unlikely to Very likely

  8. Q7

    Imagine Pennywell offers the following: Savings pots are instant-access savings accounts inside the Pennywell app. You can withdraw your money whenever you want. They pay 3.1% AER. In-app prompts show you where to find pots and how to set one up. A reminder when your pay arrives invites you to set up a pot. You can ignore these prompts and reminders. How likely would you be to open a pot and put money into it within the next 30 days?

    Scale

    Scale 1-5: Very unlikely to Very likely

  9. Q8

    Imagine Pennywell offers the following: Savings pots are instant-access savings accounts inside the Pennywell app. You can withdraw your money whenever you want. They pay 3.85% AER. Pots are available under a tab in the app, with no additional prompts or payday reminder to help you find and set up a pot. How likely would you be to open a pot and put money into it within the next 30 days?

    Scale

    Scale 1-5: Very unlikely to Very likely

  10. Q9

    Imagine Pennywell offers the following: Savings pots are instant-access savings accounts inside the Pennywell app. You can withdraw your money whenever you want. They pay 3.85% AER. In-app prompts show you where to find pots and how to set one up. A reminder when your pay arrives invites you to set up a pot. You can ignore these prompts and reminders. How likely would you be to open a pot and put money into it within the next 30 days?

    Scale

    Scale 1-5: Very unlikely to Very likely

  11. Q10

    Thinking specifically about the option with 3.85% AER, in-app prompts explaining where to find pots and how to set one up, and a reminder when your pay arrives: approximately how much of your existing instant-access savings outside Pennywell would you realistically move into a pot within the next 30 days? Do not include money already held at Pennywell or money you would save from future income.

    One answer
    • £0
    • £1–£499
    • £500–£999
    • £1,000–£2,499
    • £2,500–£4,999
    • £5,000–£9,999
    • £10,000–£24,999
    • £25,000–£49,999
    • £50,000 or more
    • Not sure
    • Prefer not to say

    Routing Shown only if Q3 is: £1–£499; £500–£999; £1,000–£2,499; £2,500–£4,999; £5,000–£9,999; £10,000–£24,999; £25,000–£49,999; £50,000 or more

Sample plan

Use the account database to identify current adult customers who have never opened a pot, including customers who previously opened and closed one as ineligible. Recheck eligibility before invitation. Do not mention savings pots in the email invitation; describe this as a five-minute survey about banking. The database supplies pot-history eligibility, avoiding a pot-history screener that would reveal the product before Q1. Randomly sample independently within the four database age bands, targeting 175 completed responses per band, 700 overall. Invite randomly selected batches and send the same reminders across bands; determine invitation volumes from observed response rates rather than assuming a response rate. If the oldest band is small, invite all eligible customers in that band and report any shortfall. Use secure individual survey links and permit one response per customer. Q5 provides self-reported age; use database age for sampling and, with appropriate data-use permission, for age analysis when Q5 is unanswered. Weight overall estimates to the age distribution of eligible never-pot customers, not all 420,000 customers. At 700 completes, the requested age-band precision is not achievable in general: 175 per band gives approximately ±7.4 percentage points at 95% confidence for a proportion near 50%, before weighting or other design effects. Approximately 385 completes per age band, or 1,540 overall, are needed for ±5 percentage points under simple random sampling without a finite-population correction. Seek approval for that larger sample if the precision requirement is mandatory; otherwise explicitly agree the weaker precision. Assess finite-population corrections using actual eligible counts. Sampling confidence intervals do not cover nonresponse or measurement bias. Keep Q1–Q5 in the listed order; present Q6–Q9 as four separate screens in uniformly random order, retaining their IDs, followed by Q10. These are alternative scenarios, not cumulative changes. Randomise Q2's substantive options, keeping 'Another reason', 'No particular reason' and 'Not sure' last; make the last two exclusive. Label all five scale points: Very unlikely, Unlikely, Neither likely nor unlikely, Likely, Very likely. Pilot the programmed questionnaire for comprehension and a completion time below five minutes.