Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Gemini 4 Hydrogen

Measured by Spring Prompt

Gemini 4 Hydrogen: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
67 of 100
Analysis rating
1,088
Head to head, this task
won 3 of 9

Why it is not sound

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first

Findings the judges found unsupported

  • Booking satisfaction is substantially lower than last year's 71% benchmark (50.8% unweighted, 46.3% age-weighted among triers), though mode differences mean the exact magnitude of the drop is imprecise.The analysis reports a drop in satisfaction despite the surveys having different sampling methods and respondent populations. These differences prevent interpreting the figures as a change, not merely estimating its magnitude imprecisely.
  • A quarter of respondents needed care in the last three months but did not try to book, driven primarily by expected wait times and access pessimism rather than self-resolving illness.Projecting the age-weighted result to approximately 4,392 adults treats respondents as representative within age groups despite substantial coverage and nonresponse bias. Age weighting alone does not establish this population count.
  • Much of the working-age demand to restore phone booking stems from fixable failures in the online workflow—multi-day response delays, a long repetitive form, and accessibility defects—rather than hostility to digital access itself.The comments support themes of slow replies, form complexity and accessibility problems, but do not establish that much working-age demand for phone booking is caused by these problems rather than a preference for speaking to someone. Nor do they show that fixing the online workflow would remove that demand.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9497
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1113
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 5 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Findings

  1. Patients do not broadly prefer online-first booking, and online-first is not working well overall: unweighted preference splits 46.2% online vs 45.3% phone or in-person, and age-weighting reverses this to 40.7% online vs 51.7% phone or in-person (45.0% phone, 6.7% in person) even before accounting for patients without mobile numbers.

    High confidenceBase 886

    Q6 (n=886): 46.2% prefer online, 40.7% phone, 4.6% in person, 6.5% no preference. Weighted to the 17,100 adult practice list (44% 18-44, 32% 45-64, 13% 65-74, 11% 75+): 40.7% online, 45.0% phone, 6.7% in person. Only 18-44s show an online majority (58.4%).

  2. Booking satisfaction is substantially lower than last year's 71% benchmark (50.8% unweighted, 46.3% age-weighted among triers), though mode differences mean the exact magnitude of the drop is imprecise.

    Medium confidenceBase 549

    Q2 (n=549 triers): 50.8% fairly/very satisfied (95% CI 46.6%-55.0%, MoE +/-4.2 pp) vs 33.9% dissatisfied. Age-weighted satisfaction is 46.3%. Last year's 71% came from an October 2025 waiting-room paper survey (n=412) of successful attenders, whereas September 2026 used an SMS-to-web survey including unsuccessful/frustrated triers.

  3. A quarter of respondents needed care in the last three months but did not try to book, driven primarily by expected wait times and access pessimism rather than self-resolving illness.

    High confidenceBase 886

    Q4 (n=886): 24.3% (age-weighted 25.7%, ~4,392 of 17,100 adults) needed an appointment but did not try to book. In Q5 (n=215), top barriers were expecting too long a wait for a reply (43.7%) and believing they would not get an appointment (28.8%), alongside pharmacy/NHS 111 diversion (22.8%) and form difficulty (16.7%).

  4. Older adults (65+, not just 75+) and disabled patients face severe booking exclusion; restricting assisted access strictly to over-75s would miss the 65-74 cohort whose satisfaction and avoidance mirror over-75s.

    Medium confidenceBase 549

    Q2 satisfaction: 64.7% for 18-44 (n=272) vs 44.0% for 45-64 (n=209), dropping sharply to 17.8% for 65-74 (n=45) and 13.0% for 75+ (n=23), and 40.0% for disabled triers (n=65) vs 52.3% non-disabled (n=484). Both 65-74 (n=54) and 75+ (n=33) report 33.3% non-attempt at Q4, with 77.8% of 65-74s preferring phone (63.0%) or in-person (14.8%) at Q6.

  5. There is no statistically significant difference in booking satisfaction between female and male respondents at the 95% confidence level, though women score directionally lower.

    Medium confidenceBase 540

    Q2 (n=540 female/male): 47.5% of women (n=303) vs 55.3% of men (n=237) were satisfied; the 7.8 pp difference is not statistically significant (two-proportion z=1.79, p=0.074). Q4/Q5 are not cross-tabulated by sex.

  6. Patients who need an interpreter are too poorly represented in this English-language SMS survey to quantify their booking difficulty, but directional and open-text evidence points to high exclusion from online-only booking.

    Low confidenceBase 13

    Q10 (n=886): only 13 respondents (1.5%) needed an interpreter, and Q2 has no interpreter breakdown. Among those 13 at Q6, 61.5% (8 respondents) prefer booking in person at reception and only 7.7% (1) prefer the online form; open-text answers (A112 Polish, A756 Urdu) describe fear of writing medical details wrongly in English and reliance on relatives.

  7. Much of the working-age demand to restore phone booking stems from fixable failures in the online workflow—multi-day response delays, a long repetitive form, and accessibility defects—rather than hostility to digital access itself.

    High confidenceBase 246

    Q3 open-text (246 answers): 94 respondents ask for phone booking (79 aged 18-64, 11 aged 65+), while 45 call for faster replies (often reporting 24-72+ hour waits), 29 want a shorter/simpler form, 22 cite disability/accessibility/language barriers (e.g. broken JAWS screen-reader buttons, small text, no save-and-resume, BSL/translation gaps), and 45 explicitly praise the online form when it works.

Caveats

  • Sampling frame coverage bias: 42% of registered adult patients (~7,260 of 17,100) have no mobile number on record and were excluded. Offline and older patients are almost certainly more dissatisfied and less digital-ready than survey respondents.
  • Non-response and age skew: only 9.0% of SMS recipients completed the survey (886 of 9,840; 15.3% drop-off among 1,046 starters). Adults aged 65+ make up 24% of registered adults but only 9.8% of the unweighted sample (n=87), giving wide margins of error on 65-74 (n=45 in Q2) and 75+ (n=23 in Q2) subgroups.
  • Mode and survivorship incompatibility with October 2025 baseline: last year's 71% figure came from paper questionnaires handed out in the waiting room (n=412) to patients who succeeded in getting an appointment, whereas 2026 is an SMS-to-web survey capturing people who struggled or gave up.
  • Severe under-coverage of limited-English speakers: the survey was administered in English only, yielding just 13 respondents (1.5%) needing an interpreter and no Q2 satisfaction breakdown for this group, so quantitative estimates for language-barrier patients are unreliable.
  • Age-weighted estimates assume respondents within each age bracket represent non-respondents and patients without mobiles in the same bracket, which likely understates true dissatisfaction and non-digital preference among older and digitally excluded patients.

Open-text themes it coded

Restore or offer phone booking 94Faster response to online requests 45Satisfied with current online form 45Shorten or simplify the online form 29Accessibility, disability, and language support 22Continuity of care (same GP) 15In-person help or human contact (non-phone or unspecified) 6No suggestions or rarely uses practice 32

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Are you currently registered as a patient at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    Which age group are you in?

    One answer
    • Under 18
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over

    Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over

  3. Q1

    Since April 2026, have you booked or tried to book an appointment at Ashgrove Medical Group?

    One answer
    • Yes
    • No
  4. Q2

    Overall, how satisfied or dissatisfied are you with the process of booking an appointment since April 2026?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Routing Shown only if Q1 is: Yes

  5. Q3

    How did you try to book an appointment the last time you contacted the practice?

    One answer
    • Filled in the online request form myself
    • Called the practice by phone
    • Visited the reception desk in person
    • Someone else booked on my behalf
    • Other

    Routing Shown only if Q1 is: Yes

    Judges: assumptive

    Q1 establishes that the respondent has tried booking since April 2026, but not that their most recent contact with the practice was a booking attempt. Q3 assumes that it was.

    As a single-choice question, it assumes the respondent only used one method the last time they tried to book, whereas they might have tried multiple (e.g., trying online first, then calling).

  6. Q4

    Since April 2026, has there been a time when you needed an appointment at Ashgrove Medical Group, but decided not to try to book one?

    One answer
    • Yes
    • No
    • Not sure
  7. Q5

    Why did you decide not to try to book an appointment? Please select all that apply.

    Any that apply
    • The online form felt too difficult or confusing
    • I do not have reliable internet access or a suitable device
    • The online form was closed or unavailable when I checked
    • I wanted to speak to someone by phone and did not think I could
    • Wait times for a reply or appointment seemed too long
    • Reading or English language barriers made booking difficult
    • I did not want to write details of my health problem on an online form
    • My health problem got better or I used another service (such as a pharmacy or NHS 111)
    • Other

    Routing Shown only if Q4 is: Yes

  8. Q6

    In the future, how would you most prefer to book an appointment at Ashgrove Medical Group?

    One answer
    • Filling in an online request form
    • Calling a reception phone line
    • Asking in person at the reception desk
    • I have no preference
  9. Q7

    Ashgrove Medical Group is reviewing its booking system. Which of these options would you prefer the practice to offer?

    One answer
    • Keep online-first booking as it is (online form for most patients; phone line kept for those who cannot go online)
    • Add a dedicated phone line and in-person reception help only for patients who need them (over-75s, disabled patients, and patients with limited English), while keeping online-first for everyone else
    • Restore a morning phone booking line open to all patients, alongside the online form
    • Not sure
  10. Q8

    Which of the following, if any, would make booking an appointment easier for you? Please select all that apply.

    Any that apply
    • A morning phone line open to anyone who wants to book by phone
    • A dedicated phone line and in-person help for older patients, disabled patients, or patients who need language support
    • A shorter or simpler online request form
    • Clearer instructions or help on how to use the online form
    • Keeping the online form open for longer hours
    • Clearer information on when the practice will reply to a request
    • Translated booking forms or interpreter support when booking
    • None of these — booking is already easy for me
    • Other
  11. Q9

    Is there anything else you would like to tell the practice about booking appointments, or anything that would make it easier for you?

    Open text
  12. Q10

    Do you have any physical or mental health conditions, disabilities, or impairments that make it harder to use an online form or phone booking system?

    One answer
    • Yes
    • No
    • Prefer not to say
  13. Q11

    Do you usually need an interpreter, or help with English, when booking or attending a GP appointment?

    One answer
    • Yes, always or usually
    • Yes, sometimes
    • No
    • Prefer not to say
  14. Q12

    Which statement best describes how you manage online forms?

    One answer
    • I can easily fill in an online form on my own
    • I can fill in an online form, though I sometimes find it difficult
    • I can only fill in an online form with help from someone else
    • I cannot fill in an online form or do not have internet access or a suitable device
    • Prefer not to say
Sample plan

Invite all ~17,100 registered adult patients (aged 18+) with a mobile number on record via SMS containing the survey link. Set soft quotas across the four required age bands (18-44, 45-64, 65-74, 75+) and send targeted SMS reminders to underrepresented age bands (especially 65-74 and 75+) to ensure balanced subgroup sizes. Note on precision vs. sample size: under the ~900-complete constraint (~225 completes per age band), the overall margin of error at 95% confidence is +/-3.3 percentage points, but each individual age band will have a margin of error of roughly +/-6.0 to +/-6.5 percentage points (even after finite population correction for smaller strata like 75+). Achieving +/-5 percentage points at 95% confidence within every age band requires ~340 to 384 completes per band (~1,400 to 1,500 total completes), so the practice should either expand total completes to ~1,450 or accept ~+/-6.5% band precision at n=900. Note on mode bias: an SMS-to-online survey will systematically under-represent patients without smartphones/internet access, patients with severe visual/cognitive disabilities, over-75s, and patients with limited English—the very groups central to the decision. To mitigate this, provide an optional assisted phone or paper completion route at reception/via outbound calls for a sample of 75+ and limited-English patients if feasible, and weight final overall results back to the practice's true adult age distribution from the patient register.