Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Gemini 4 Carbon

Measured by Spring Prompt

Gemini 4 Carbon: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
73 of 100
Analysis rating
1,076
Head to head, this task
won 4 of 9

Why it is not sound

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first
  • Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Findings the judges found unsupported

  • Answer to the practice manager: No—patients overall do not prefer online-first, and the system is only working well for under-45s. Once adjusted for the practice's adult age profile, preference for phone booking (45.0%) outweighs preference for the online form (40.7%), even before accounting for the 42% of adults without a mobile number on record who could not receive this SMS survey.Age-weighted in-person preference is approximately 6.7%, not 7.0%; combined phone and in-person preference is approximately 51.7%, not 52.0%.
  • Answer to the GP partner: Booking satisfaction is markedly lower than last year's 71%, standing at 50.8% unweighted (±4.2 pp) and 46.3% age-weighted among those who tried to book, driven by severe dissatisfaction among patients aged 65 and over—though the exact percentage-point drop cannot be taken at face value because the survey mode and sample changed completely.The analysis describes satisfaction as lower than last year and subsequently calls the difference a fall mixing genuine decline with methodological differences. These surveys cannot establish a genuine decline, its size, or that older patients drove a decline. A caveat about the exact percentage-point drop does not resolve this.
  • An over-75s-only age threshold for phone and reception support would leave behind 65–74-year-olds and a large share of 45–64-year-olds. Patients aged 65–74 struggle almost as severely with online-first as over-75s, and 88% of open-text requests for phone booking came from under-65s.The small older subgroup samples support concern and further investigation, but do not establish with high confidence that 65–74-year-olds struggle almost as severely as over-75s. The proportion of phone-request comments coming from under-65s also reflects the heavily age-skewed respondent pool, rather than establishing the population distribution of unmet support needs.
  • Roughly a quarter of adult patients—representing about 4,392 registered adults when age-weighted—needed an appointment in the last three months but did not even try to book, primarily because of slow reply times and perceived lack of appointment availability rather than digital inability among under-65s.The weighted extrapolation is a permissible calculation, but presenting approximately 4,392 registered adults as an established population finding with high confidence is unsupported by this low-response, mobile-only survey. Age weighting does not correct coverage and nonresponse bias.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9497
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1113
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 5 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Do not keep online-first as it is, and do not restrict phone and reception access solely to the narrow over-75/disability/interpreter criteria as written. Instead, restore telephone booking and active in-person reception booking alongside the online form—keeping online available 24/7 as the fast, convenient option rather than a mandatory gatekeeper, while ensuring priority/ring-fenced phone and face-to-face support (with interpreter and BSL access) for patients aged 65+ (not just 75+), disabled patients, patients with limited English, and patients without mobile access without requiring patients to prove eligibility. Simultaneously, fix the two operational failures driving digital frustration so that most under-65s choose online voluntarily: cut the online request form down to a short, accessible, screen-reader-compliant form and commit to automatic receipt confirmation plus a same-working-day response SLA.

What would change it

If staffing and call-volume modelling (or a 6-week pilot) shows that opening telephone booking to all patients recreates severe morning phone queues that block vulnerable callers or pull staff away from delivering same-working-day online replies, step back from an unrestricted morning phone line to a dedicated, unpoliced phone and reception pathway expanded to all 65+s, disabled patients, interpreter/BSL users, non-mobile patients, and anyone self-declaring difficulty going online—provided that shortening the form and hitting same-day replies lifts 45–64 satisfaction above 65%. Conversely, if reception capacity easily absorbs demand once online turnaround and form length are fixed, maintain open multi-channel choice permanently.

Findings

  1. Answer to the practice manager: No—patients overall do not prefer online-first, and the system is only working well for under-45s. Once adjusted for the practice's adult age profile, preference for phone booking (45.0%) outweighs preference for the online form (40.7%), even before accounting for the 42% of adults without a mobile number on record who could not receive this SMS survey.

    High confidenceBase 886

    Unweighted Q6 shows 46.2% prefer online vs 40.7% phone and 4.6% in person (base 886), only because 18–44s and 45–64s make up 90.1% of respondents vs 76.0% of registered adults. Age-weighted across registered adults (44% 18–44, 32% 45–64, 13% 65–74, 11% 75+), phone preference rises to 45.0% and in-person to 7.0% (52.0% non-online combined), while online falls to 40.7%. Online is preferred by a majority only among 18–44s (58.4%, base 462), falling to 38.0% among 45–64s (base 337), 16.7% among 65–74s (base 54), and 6.1% among 75+s (base 33).

  2. Answer to the GP partner: Booking satisfaction is markedly lower than last year's 71%, standing at 50.8% unweighted (±4.2 pp) and 46.3% age-weighted among those who tried to book, driven by severe dissatisfaction among patients aged 65 and over—though the exact percentage-point drop cannot be taken at face value because the survey mode and sample changed completely.

    Medium confidenceBase 549

    At Q2, 50.8% of triers were satisfied and 33.9% dissatisfied (base 549). Satisfaction falls sharply with age: 64.7% in 18–44s (base 272), 44.0% in 45–64s (base 209), 17.8% in 65–74s (60.0% dissatisfied, base 45), and 13.0% in 75+s (73.9% dissatisfied, base 23). However, October 2025's 71% figure (base 412) came from a waiting-room paper survey that captured only patients who had succeeded in getting an in-person appointment, whereas September 2026's SMS survey includes failed booking attempts and excludes patients without mobile phones.

  3. An over-75s-only age threshold for phone and reception support would leave behind 65–74-year-olds and a large share of 45–64-year-olds. Patients aged 65–74 struggle almost as severely with online-first as over-75s, and 88% of open-text requests for phone booking came from under-65s.

    High confidenceBase 886

    Among 65–74s, only 17.8% of triers were satisfied (base 45), 77.8% prefer phone (63.0%) or reception (14.8%) over online (16.7%, base 54), and 44.4% of deterred 65–74s said they are not confident going online (base 18)—higher even than 75+s (36.4%, base 11). Among 45–64s, more prefer phone (49.0%) than online (38.0%, base 337). In Q3 open text (base 246), 94 respondents asked for phone booking, of whom 83 were aged under 65 and only 11 were aged 65 or over.

  4. Roughly a quarter of adult patients—representing about 4,392 registered adults when age-weighted—needed an appointment in the last three months but did not even try to book, primarily because of slow reply times and perceived lack of appointment availability rather than digital inability among under-65s.

    High confidenceBase 886

    At Q4, 24.3% of all respondents (base 886; 25.7% age-weighted across 17,100 adults = 4,392 patients) did not try to book despite needing an appointment, rising to 33.3% among both 65–74s (base 54) and 75+s (base 33), and 35.1% among those who made no booking attempt at all in the period (base 313). At Q5 (base 215), top barriers were expecting to wait too long for a reply (43.7%; 51.2% in 45–64s) and not thinking they would get an appointment (28.8%), alongside form difficulty (16.7%) and low online confidence (9.3% overall, but 44.4% in 65–74s and 36.4% in 75+s). In Q3, 45 respondents cited waits of 24 hours to 5 days for a reply and 29 criticised the lengthy, repetitive form.

  5. Patients with disabilities experience lower booking satisfaction and face diverse physical, sensory, and cognitive barriers that require both non-digital channels (phone and face-to-face reception) and accessibility fixes to the online form itself.

    High confidenceBase 102

    Disabled triers reported 40.0% satisfaction vs 52.3% among non-disabled triers (and only 10.8% 'very satisfied' vs 22.9%, Q2 bases 65 and 484; gap p = 0.063). Across all disabled respondents (Q6 base 102), 50.0% prefer phone and 15.7% prefer in-person reception ( nearly five times the non-disabled rate of 3.2%), while only 23.5% prefer online (vs 49.1% non-disabled). Open-text answers from 21 disabled respondents detail screen-reader unlabelled buttons (JAWS), hand tremors (Parkinson's, MS, stroke, arthritis), sight loss, deafness/BSL needs, dementia, learning disabilities, selective mutism, and chronic fatigue requiring save-and-return or proxy booking.

  6. Answer to the PPG chair (part 1—sex): Women who tried to book reported lower satisfaction than men by 7.8 percentage points, but this difference is not statistically significant at the 95% confidence level, and the tables do not show whether women were deterred from trying more often.

    Medium confidenceBase 540

    At Q2, 47.5% of female triers (base 303) vs 55.3% of male triers (base 237) were satisfied (two-proportion z = 1.79, p = 0.074). Q4, Q5, and Q6 were not cross-tabulated by sex, and Q2 was not cross-tabulated by age within sex.

  7. Answer to the PPG chair (part 2—interpreters): The survey reached very few patients needing an interpreter and lacks satisfaction crosstabs for them, so overall experience evidence is thin—but even within this tiny group, preference for in-person reception booking is overwhelming.

    Low confidenceBase 13

    Only 13 respondents (1.5% of 886) reported needing an interpreter, and Q1, Q2, and Q4 were not broken out for this group. However, at Q6 (base 13), 61.5% (8 of 13) prefer to book in person at reception and 23.1% (3 of 13) by phone, vs just 7.7% (1 of 13) online (compared with 3.8% in-person and 46.7% online among non-interpreter patients, base 873). Open-text comments (A112 Polish, A756 Urdu) highlight fear of miscommunicating symptoms in written English and dependence on family members.

Caveats

  • 42% coverage exclusion: 7,260 of the 17,100 registered adults (42%) have no mobile number on record and could not be invited. Those excluded are disproportionately likely to be older, digitally excluded, or vulnerable, meaning even age-weighted estimates almost certainly overstate online preference and satisfaction and understate unmet need.
  • Low response rate and strong age skew: Only 9.0% of invited patients completed the survey (886 of 9,840), with 15.3% drop-off among starters (160 of 1,046—potentially affected by screen-reader, literacy, or language barriers). Respondents aged 65+ represent 24.0% of registered adults (13% 65–74, 11% 75+) but only 9.8% of the unweighted sample (n=54 and n=33).
  • Small subgroup bases: Subgroups for over-75 triers (n=23), deterred 65–74s (n=18) and 75+s (n=11), and especially patients needing an interpreter (n=13 overall; no Q2/Q4 breakdown) are very small and carry wide margins of error; furthermore, an English-only SMS survey systematically under-represents patients with limited English.
  • Incomparable historical baseline: October 2025's 71% satisfaction figure came from a waiting-room paper survey (n=412) restricted to patients who had successfully booked and attended in person, so the fall to 50.8% (46.3% age-weighted) mixes genuine decline with sample, mode, and survivorship differences.
  • Adult-only scope and missing cross-breaks: Parents/carers booking for the ~3,900 patients under 18 were excluded (Q1 asked only about booking 'for yourself'), and cross-tabulations of Q4/Q5/Q6 by sex and Q1/Q2/Q4 by interpreter need were not produced.

Next steps

  • decisionReopen phone and walk-in reception booking alongside online requests, with interpreter/LanguageLine and BSL video-relay workflows at reception and no punitive gatekeeping for patients who struggle online, ensuring 65–74s and non-mobile patients are fully covered.
  • decisionRedesign and accessibility-audit the online request form: strip out repetitive pages for routine requests, add instant SMS/email receipt confirmation, fix WCAG/JAWS screen-reader labels, text contrast/sizing, save-and-return, proxy booking consent, and add GP-continuity preference fields.
  • experimentTest a triage-and-staffing rota that guarantees same-working-day responses to online requests submitted before afternoon cut-off while tracking morning call wait times across a 6-week operational pilot.
  • researchRun targeted non-SMS research with excluded groups—waiting-room/postal paper surveys for patients without mobile numbers on record, interpreter-assisted phone/community outreach with Polish, Urdu, and other non-English speakers, and re-tabulate existing survey data for sex-by-age and Q4/Q5 sex differences.
  • monitorMonitor weekly booking channel split, call abandonment, online turnaround time, DNA rates, and a 2-question post-booking satisfaction pulse across age, disability, and interpreter flags.

Open-text themes it coded

Restore phone booking / speak to staff by phone 94Faster replies and receipt confirmation for online requests 45Satisfied with the current online form / works well as is 45Shorter, simpler, less repetitive online form 29Accessibility and format adjustments for disabled patients 21Continuity of care / booking with a regular GP 15In-person help at reception or speaking directly to a person 7Language, translated forms, BSL, and interpreter support 3No suggestions / rarely need appointments 32

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Are you currently registered as a patient at Ashgrove Medical Group? (If you are helping an adult family member or friend who is registered here to complete this survey, with their permission, please answer every question from their point of view.)

    One answer
    • Yes – I am registered here and answering for myself
    • Yes – I am answering on behalf of an adult patient registered here
    • No – neither I nor the person I am helping is registered here

    Continues if Yes – I am registered here and answering for myself; Yes – I am answering on behalf of an adult patient registered here

  2. S2

    Which age group are you in? (If you are answering on behalf of someone else, please select their age group.)

    One answer
    • Under 18
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over

    Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over

  3. Q1

    In April 2026, Ashgrove Medical Group changed how appointments are booked so that most requests start with an online form, with reception phones kept for patients who cannot go online. Since April 2026, have you tried to book or needed an appointment at the practice?

    One answer
    • Yes – I used the online request form
    • Yes – I phoned reception
    • Yes – I asked at the reception desk in person
    • Yes – I used more than one of these ways (online, phone, or in person)
    • Yes – I needed an appointment, but did not end up requesting one
    • No – I have not needed an appointment since April 2026
  4. Q2

    Overall, how satisfied or dissatisfied have you been with booking an appointment at Ashgrove Medical Group since April 2026?

    One answer
    • Very satisfied
    • Fairly satisfied
    • Neither satisfied nor dissatisfied
    • Fairly dissatisfied
    • Very dissatisfied

    Routing Shown only if Q1 is: Yes – I used the online request form; Yes – I phoned reception; Yes – I asked at the reception desk in person; Yes – I used more than one of these ways (online, phone, or in person); Yes – I needed an appointment, but did not end up requesting one

  5. Q3

    Since April 2026, has there been a time when you needed an appointment at the practice, but were put off trying to book or gave up before completing your request?

    One answer
    • Yes – I needed an appointment and did not try to book at all
    • Yes – I started trying to book, but gave up before completing it
    • No – whenever I needed an appointment, I completed my request

    Routing Shown only if Q1 is: Yes – I used the online request form; Yes – I phoned reception; Yes – I asked at the reception desk in person; Yes – I used more than one of these ways (online, phone, or in person); Yes – I needed an appointment, but did not end up requesting one

  6. Q4

    What put you off booking, or caused you to give up? (Please select all that apply.)

    Any that apply
    • The online request form was confusing or had too many questions
    • I do not have internet access, mobile data, or a suitable device
    • The online form was closed or unavailable at the time I needed it
    • I could not get through on the phone or waited too long in the queue
    • Reception staff told me to use the online form, which I could not do
    • A health condition or disability made using the booking system difficult
    • Reading or writing in English on the online form was difficult for me
    • I did not feel comfortable typing private medical details into an online form
    • I was unsure when or how the practice would reply to my request
    • Other reason

    Routing Shown only if Q3 is: Yes – I needed an appointment and did not try to book at all; Yes – I started trying to book, but gave up before completing it

  7. Q5

    Without help from someone else, how easy or difficult is it for you to fill in an online form to request a GP appointment?

    One answer
    • Very easy
    • Fairly easy
    • Neither easy nor difficult
    • Fairly difficult
    • Very difficult
    • I cannot use an online form at all without help
  8. Q6

    When you need to ask for an appointment at Ashgrove Medical Group in the future, which ONE way would you most prefer to use?

    One answer
    • Fill in the practice's online request form (on a phone, tablet, or computer)
    • Phone the practice reception
    • Speak to a receptionist in person at the practice
    • Book directly through the NHS App
    • I have no preference
  9. Q7

    Which of the following would do the most to make booking an appointment easier for you? (Please select up to three.)

    Any that apply
    • A shorter, simpler online request form
    • Keeping the online request form open for longer hours (including evenings and weekends)
    • Clearer updates on when the practice will reply to an online request
    • Being able to choose whether the practice replies by text message or phone call
    • Being able to phone reception in the morning to book an appointment
    • A dedicated phone line reserved for patients who cannot go online, are aged 75 or over, have a disability, or need language support
    • In-person help at the reception desk to book an appointment
    • Online forms in other languages, or an easy way to ask for an interpreter
    • Nothing – booking an appointment is already easy for me
  10. Q8

    In your own words, what one change would make the biggest difference to how you book an appointment at Ashgrove Medical Group? (Optional)

    Open text
  11. Q9

    Do you have any physical or mental health conditions, disabilities, or illnesses lasting or expected to last 12 months or more?

    One answer
    • Yes
    • No
    • Prefer not to say
  12. Q10

    Does your condition or illness reduce your ability to carry out day-to-day activities?

    One answer
    • Yes, a lot
    • Yes, a little
    • No, not at all
    • Prefer not to say

    Routing Shown only if Q9 is: Yes

  13. Q11

    Do you ever need an interpreter (including British Sign Language) when speaking with GP practice staff or clinicians?

    One answer
    • Yes – for a spoken language other than English
    • Yes – for British Sign Language (BSL)
    • No
    • Prefer not to say
  14. Q12

    Without help from someone else, how easy or difficult is it for you to read and fill in forms written in English?

    One answer
    • Very easy
    • Fairly easy
    • Fairly difficult
    • Very difficult
    • I cannot read or write in English
    • Prefer not to say
Sample plan

INVITATION & FIELDWORK: Send an SMS invitation with a unique one-use survey link to all registered adult patients (aged 18+, approx. 17,100 total) who have a mobile number on record, with one reminder SMS after 5 days to non-responders. CRITICAL SAMPLE SIZE & QUOTA RECONCILIATION: Achieving +/-5 percentage points at 95% confidence within EACH of the four age bands (18-44, 45-64, 65-74, 75+) requires n=325 to 365 completes per band after applying the finite population correction to Ashgrove's 17,100 adult register—totalling ~1,380 to 1,420 completes across all four bands. If fieldwork is strictly capped at ~900 completes (225 equal quota per age band), the margin of error per age band widens to +/-6.4 percentage points at 95% confidence. We therefore set minimum quotas of 225 per age band within a 900 cap, and strongly recommend lifting the cap to 1,400 completes (~330 each in 65-74 and 75+, ~370 each in 18-44 and 45-64) to meet the partners' +/-5pp precision target. MITIGATING EXCLUSION BIAS: Because an SMS-to-web survey systematically under-represents patients who cannot go online, lack a mobile phone, have severe disabilities, or have limited English, S1 explicitly permits proxy completion by carers/family members, and overall practice-level estimates will be rim-weighted back to the practice clinical register profile by age band, sex, and recorded interpreter/accessibility flags (and we advise a small parallel telephone/reception-paper check of ~50-100 patients without mobile numbers so non-digital voices are not missed).