Benchmarks / SurveyBench / Why Pennywell customers don't use savings pots / Gemini 4 Neon
Gemini 4 Neon: Why Pennywell customers don't use savings pots
The decision: whether to run an awareness and onboarding push for savings pots (in-app prompts and a set-up nudge on payday), to raise the pots rate to 3.85% AER, or both. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 29 of 100
- Analysis rating
- 629
- Head to head, this task
- won 0 of 5
Why it is not sound
The survey
- A double-barrelled question: Q2
The analysis
- Produced no usable analysis
Scorecard
The analysis
This model produced no usable analysis for this task.
The survey it planned
3 screening questions and 10 questions, as the model wrote them.
-
S1
Which of the following age groups are you in?
One answer- Under 18
- 18-34
- 35-54
- 55-74
- 75 or over
- Prefer not to say
Continues if 18-34; 35-54; 55-74; 75 or over
-
S2
Do you currently have an active Pennywell current account that you access using the Pennywell app?
One answer- Yes
- No
- Not sure
Continues if Yes
-
S3
Do you, or does anyone in your immediate household, work for Pennywell?
One answer- Yes
- No
Continues if No
-
Q1
Before receiving this survey, were you aware that Pennywell offers savings pots inside the Pennywell app?
One answer- Yes, I knew about them, and I have already opened one
- Yes, I knew about them, but I have never opened one
- No, I did not know Pennywell offered savings pots
- Not sure
-
Q2
Why have you not opened a Pennywell savings pot? Please select all that apply.
Any that apply- I have not got round to setting one up
- I did not know enough about how they work or what interest rate they pay
- I thought the interest rate was too low compared with other accounts
- I already keep my savings with another bank or building society
- I prefer to keep my savings separate from my everyday spending account
- I do not have spare money to put into savings at the moment
- I only use Pennywell as a secondary account, not my main bank
- I was not sure how easily I could access my money
- Other
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one
Judges: double-barrelledThe option 'I did not know enough about how they work or what interest rate they pay' combines two different barriers: understanding of set-up/function and knowledge of the rate. These are the very factors O2 needs to separate, so selections of this option cannot be attributed to either one.
The option 'I did not know enough about how they work or what interest rate they pay' groups two different reasons into one.
-
Q3
Before today, roughly what interest rate (AER) did you think Pennywell savings pots paid?
One answer- 0% (no interest)
- More than 0%, under 2.0% AER
- 2.0% to under 3.0% AER
- 3.0% to under 4.0% AER
- 4.0% to under 5.0% AER
- 5.0% AER or more
- I had no idea what interest rate they paid
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one
-
Q4
Roughly how much money do you currently hold outside Pennywell in instant-access (easy-access) savings accounts or set aside as savings in another current account? Please exclude Cash ISAs, fixed-term accounts, investments, and pensions.
One answer- £0 (I have no savings of this kind outside Pennywell)
- £1 to £499
- £500 to £1,999
- £2,000 to £4,999
- £5,000 to £9,999
- £10,000 to £24,999
- £25,000 or more
- Don't know
- Prefer not to say
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one; No, I did not know Pennywell offered savings pots; Not sure
-
Q5
Roughly what interest rate (AER) do you earn on the largest part of those easy-access savings outside Pennywell?
One answer- 0% (for example, sitting in a current account)
- More than 0%, under 2.0% AER
- 2.0% to under 3.0% AER
- 3.0% to under 3.5% AER
- 3.5% to under 4.0% AER
- 4.0% to under 4.5% AER
- 4.5% AER or more
- Don't know
Routing Shown only if Q4 is: £1 to £499; £500 to £1,999; £2,000 to £4,999; £5,000 to £9,999; £10,000 to £24,999; £25,000 or more
-
Q6
Pennywell savings pots are instant-access savings accounts inside the Pennywell app that currently pay 3.1% AER. Now that you know about them, how likely are you to open a Pennywell savings pot in the next 3 months?
ScaleScale 1-5: Very unlikely to Very likely
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one; No, I did not know Pennywell offered savings pots; Not sure
-
Q7
Imagine the Pennywell app showed a prompt about savings pots and offered a quick set-up option when your salary, pension, or regular income arrives, with the interest rate remaining at 3.1% AER. How likely would you be to open a Pennywell savings pot in the next 3 months?
ScaleScale 1-5: Very unlikely to Very likely
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one; No, I did not know Pennywell offered savings pots; Not sure
-
Q8
Imagine Pennywell raised the interest rate on savings pots from 3.1% AER to 3.85% AER, with no other changes to the app. How likely would you be to open a Pennywell savings pot in the next 3 months?
ScaleScale 1-5: Very unlikely to Very likely
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one; No, I did not know Pennywell offered savings pots; Not sure
-
Q9
To check that this survey is displaying properly, please select 'Somewhat agree' from the options below.
One answer- Strongly disagree
- Somewhat disagree
- Neither agree nor disagree
- Somewhat agree
- Strongly agree
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one; No, I did not know Pennywell offered savings pots; Not sure
-
Q10
Which of the following best describes how you use your Pennywell current account?
One answer- It is my main current account (my salary, pension, or main income is paid into it)
- I use it regularly for everyday spending, but my main income is paid into another bank
- I use it occasionally (for example, for certain bills or spending)
- I rarely or never use it
Routing Shown only if Q1 is: Yes, I knew about them, but I have never opened one; No, I did not know Pennywell offered savings pots; Not sure
Sample plan
Sampling frame: Pennywell customers aged 18+ who have never opened a savings pot (~361,200 customers, 86% of the 420,000 customer base), pulled from the account database immediately before launch. Use a neutral email subject and invitation ('Help us improve your Pennywell app experience') that does NOT mention savings pots or interest rates, preventing self-selection bias and avoiding priming Q1 awareness. Do not screen on savings pots in the screener for the same reason; anyone who opened a pot between sample pull and survey completion is filtered out via Q1 and excluded from the target base. CRITICAL PRECISION FLAG: The board's precision target of +/-5 percentage points at 95% confidence in EACH of the four age bands (18-34, 35-54, 55-74, 75 or over) requires ~385 completes per band (~1,540 completes total, assuming a large population per band) and is mathematically unachievable with ~700 completes. A simple random sample of 700 would leave the 75+ band severely underpowered (likely <35 completes, +/-16 pp or worse in an app-only bank). Recommended approach: use a stratified random sample by database age band. Under the ~700-complete cap, set equal quotas of 175 completes per age band (18-34, 35-54, 55-74, 75+), which yields +/-7.4 percentage points at 95% confidence per band (narrower if finite population correction applies to a small 75+ customer base), and apply post-stratification weights back to the true customer-base age distribution for overall Pennywell totals. However, because emailing Pennywell's own database carries minimal marginal cost, recommend to stakeholders that they expand the target to ~1,540 completes (385 per age band, or a full census invite of 75+ if fewer than ~3,000 exist) if +/-5 pp per band is a firm requirement; alternatively, formally sign off on +/-7.4 pp per band at n=700. Invite volumes: assuming a 6-8% net response rate after excluding attention-check failures (Q9), speeders (<45 seconds), and recent pot openers (Q1), invite ~11,000-14,000 customers in controlled waves (~2,750-3,500 per age band for 700 completes; ~24,000-30,000 for 1,540 completes), with one reminder email after 4 days. Randomise option order in Q2 (keeping 'Other' last) and randomise the presentation order of Q7 and Q8 (after Q6) to eliminate order bias.