Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Gemini 4 Hydrogen
Gemini 4 Hydrogen: How Ashgrove patients find online-first booking
The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 67 of 100
- Analysis rating
- 1,088
- Head to head, this task
- won 3 of 9
Why it is not sound
The analysis
- Reported a material finding the data does not support
- Made a recommendation that does not follow from the data
- Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
- Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
- Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first
Findings the judges found unsupported
Booking satisfaction is substantially lower than last year's 71% benchmark (50.8% unweighted, 46.3% age-weighted among triers), though mode differences mean the exact magnitude of the drop is imprecise.
The analysis reports a drop in satisfaction despite the surveys having different sampling methods and respondent populations. These differences prevent interpreting the figures as a change, not merely estimating its magnitude imprecisely.A quarter of respondents needed care in the last three months but did not try to book, driven primarily by expected wait times and access pessimism rather than self-resolving illness.
Projecting the age-weighted result to approximately 4,392 adults treats respondents as representative within age groups despite substantial coverage and nonresponse bias. Age weighting alone does not establish this population count.Much of the working-age demand to restore phone booking stems from fixable failures in the online workflow—multi-day response delays, a long repetitive form, and accessibility defects—rather than hostility to digital access itself.
The comments support themes of slow replies, form complexity and accessibility problems, but do not establish that much working-age demand for phone booking is caused by these problems rather than a preference for speaking to someone. Nor do they show that fixing the online workflow would remove that demand.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process? | ✓ | 50.8% | 50.8% |
| Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied? | ✓ | 46.3% | 46.3% |
| What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one? | ✓ | 24.3% | 24.3% |
| Weighted to the age mix of registered adults, what percentage would prefer to book by phone? | ✓ | 45% | 45% |
| What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book? | ✓ | 4.2 points | 4.18 points |
| Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers ask to be able to book by phone, or for a phone booking line? | ✓ | 94 | 97 |
| How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line? | ✓ | 11 | 13 |
| Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one? | ✓ | 4,392 | 4,393 |
| Among respondents aged 18 to 44, what percentage would prefer to book using the online request form? | ✓ | 58.4% | 58.4% |
| What percentage of the patients sent the text message completed the survey? | ✓ | 9% | 9% |
Traps in the data 5 of 7 handled
- ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
- ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
- ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
- ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
- ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
- ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
- ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.
The analysis
Findings
-
Patients do not broadly prefer online-first booking, and online-first is not working well overall: unweighted preference splits 46.2% online vs 45.3% phone or in-person, and age-weighting reverses this to 40.7% online vs 51.7% phone or in-person (45.0% phone, 6.7% in person) even before accounting for patients without mobile numbers.
Q6 (n=886): 46.2% prefer online, 40.7% phone, 4.6% in person, 6.5% no preference. Weighted to the 17,100 adult practice list (44% 18-44, 32% 45-64, 13% 65-74, 11% 75+): 40.7% online, 45.0% phone, 6.7% in person. Only 18-44s show an online majority (58.4%).
-
Booking satisfaction is substantially lower than last year's 71% benchmark (50.8% unweighted, 46.3% age-weighted among triers), though mode differences mean the exact magnitude of the drop is imprecise.
Q2 (n=549 triers): 50.8% fairly/very satisfied (95% CI 46.6%-55.0%, MoE +/-4.2 pp) vs 33.9% dissatisfied. Age-weighted satisfaction is 46.3%. Last year's 71% came from an October 2025 waiting-room paper survey (n=412) of successful attenders, whereas September 2026 used an SMS-to-web survey including unsuccessful/frustrated triers.
-
A quarter of respondents needed care in the last three months but did not try to book, driven primarily by expected wait times and access pessimism rather than self-resolving illness.
Q4 (n=886): 24.3% (age-weighted 25.7%, ~4,392 of 17,100 adults) needed an appointment but did not try to book. In Q5 (n=215), top barriers were expecting too long a wait for a reply (43.7%) and believing they would not get an appointment (28.8%), alongside pharmacy/NHS 111 diversion (22.8%) and form difficulty (16.7%).
-
Older adults (65+, not just 75+) and disabled patients face severe booking exclusion; restricting assisted access strictly to over-75s would miss the 65-74 cohort whose satisfaction and avoidance mirror over-75s.
Q2 satisfaction: 64.7% for 18-44 (n=272) vs 44.0% for 45-64 (n=209), dropping sharply to 17.8% for 65-74 (n=45) and 13.0% for 75+ (n=23), and 40.0% for disabled triers (n=65) vs 52.3% non-disabled (n=484). Both 65-74 (n=54) and 75+ (n=33) report 33.3% non-attempt at Q4, with 77.8% of 65-74s preferring phone (63.0%) or in-person (14.8%) at Q6.
-
There is no statistically significant difference in booking satisfaction between female and male respondents at the 95% confidence level, though women score directionally lower.
Q2 (n=540 female/male): 47.5% of women (n=303) vs 55.3% of men (n=237) were satisfied; the 7.8 pp difference is not statistically significant (two-proportion z=1.79, p=0.074). Q4/Q5 are not cross-tabulated by sex.
-
Patients who need an interpreter are too poorly represented in this English-language SMS survey to quantify their booking difficulty, but directional and open-text evidence points to high exclusion from online-only booking.
Q10 (n=886): only 13 respondents (1.5%) needed an interpreter, and Q2 has no interpreter breakdown. Among those 13 at Q6, 61.5% (8 respondents) prefer booking in person at reception and only 7.7% (1) prefer the online form; open-text answers (A112 Polish, A756 Urdu) describe fear of writing medical details wrongly in English and reliance on relatives.
-
Much of the working-age demand to restore phone booking stems from fixable failures in the online workflow—multi-day response delays, a long repetitive form, and accessibility defects—rather than hostility to digital access itself.
Q3 open-text (246 answers): 94 respondents ask for phone booking (79 aged 18-64, 11 aged 65+), while 45 call for faster replies (often reporting 24-72+ hour waits), 29 want a shorter/simpler form, 22 cite disability/accessibility/language barriers (e.g. broken JAWS screen-reader buttons, small text, no save-and-resume, BSL/translation gaps), and 45 explicitly praise the online form when it works.
Caveats
- Sampling frame coverage bias: 42% of registered adult patients (~7,260 of 17,100) have no mobile number on record and were excluded. Offline and older patients are almost certainly more dissatisfied and less digital-ready than survey respondents.
- Non-response and age skew: only 9.0% of SMS recipients completed the survey (886 of 9,840; 15.3% drop-off among 1,046 starters). Adults aged 65+ make up 24% of registered adults but only 9.8% of the unweighted sample (n=87), giving wide margins of error on 65-74 (n=45 in Q2) and 75+ (n=23 in Q2) subgroups.
- Mode and survivorship incompatibility with October 2025 baseline: last year's 71% figure came from paper questionnaires handed out in the waiting room (n=412) to patients who succeeded in getting an appointment, whereas 2026 is an SMS-to-web survey capturing people who struggled or gave up.
- Severe under-coverage of limited-English speakers: the survey was administered in English only, yielding just 13 respondents (1.5%) needing an interpreter and no Q2 satisfaction breakdown for this group, so quantitative estimates for language-barrier patients are unreliable.
- Age-weighted estimates assume respondents within each age bracket represent non-respondents and patients without mobiles in the same bracket, which likely understates true dissatisfaction and non-digital preference among older and digitally excluded patients.
Open-text themes it coded
Restore or offer phone booking 94Faster response to online requests 45Satisfied with current online form 45Shorten or simplify the online form 29Accessibility, disability, and language support 22Continuity of care (same GP) 15In-person help or human contact (non-phone or unspecified) 6No suggestions or rarely uses practice 32
The survey it planned
2 screening questions and 12 questions, as the model wrote them.
-
S1
Are you currently registered as a patient at Ashgrove Medical Group?
One answer- Yes
- No
- Not sure
Continues if Yes
-
S2
Which age group are you in?
One answer- Under 18
- 18 to 44
- 45 to 64
- 65 to 74
- 75 or over
Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over
-
Q1
Since April 2026, have you booked or tried to book an appointment at Ashgrove Medical Group?
One answer- Yes
- No
-
Q2
Overall, how satisfied or dissatisfied are you with the process of booking an appointment since April 2026?
ScaleScale 1-5: Very dissatisfied to Very satisfied
Routing Shown only if Q1 is: Yes
-
Q3
How did you try to book an appointment the last time you contacted the practice?
One answer- Filled in the online request form myself
- Called the practice by phone
- Visited the reception desk in person
- Someone else booked on my behalf
- Other
Routing Shown only if Q1 is: Yes
Judges: assumptiveQ1 establishes that the respondent has tried booking since April 2026, but not that their most recent contact with the practice was a booking attempt. Q3 assumes that it was.
As a single-choice question, it assumes the respondent only used one method the last time they tried to book, whereas they might have tried multiple (e.g., trying online first, then calling).
-
Q4
Since April 2026, has there been a time when you needed an appointment at Ashgrove Medical Group, but decided not to try to book one?
One answer- Yes
- No
- Not sure
-
Q5
Why did you decide not to try to book an appointment? Please select all that apply.
Any that apply- The online form felt too difficult or confusing
- I do not have reliable internet access or a suitable device
- The online form was closed or unavailable when I checked
- I wanted to speak to someone by phone and did not think I could
- Wait times for a reply or appointment seemed too long
- Reading or English language barriers made booking difficult
- I did not want to write details of my health problem on an online form
- My health problem got better or I used another service (such as a pharmacy or NHS 111)
- Other
Routing Shown only if Q4 is: Yes
-
Q6
In the future, how would you most prefer to book an appointment at Ashgrove Medical Group?
One answer- Filling in an online request form
- Calling a reception phone line
- Asking in person at the reception desk
- I have no preference
-
Q7
Ashgrove Medical Group is reviewing its booking system. Which of these options would you prefer the practice to offer?
One answer- Keep online-first booking as it is (online form for most patients; phone line kept for those who cannot go online)
- Add a dedicated phone line and in-person reception help only for patients who need them (over-75s, disabled patients, and patients with limited English), while keeping online-first for everyone else
- Restore a morning phone booking line open to all patients, alongside the online form
- Not sure
-
Q8
Which of the following, if any, would make booking an appointment easier for you? Please select all that apply.
Any that apply- A morning phone line open to anyone who wants to book by phone
- A dedicated phone line and in-person help for older patients, disabled patients, or patients who need language support
- A shorter or simpler online request form
- Clearer instructions or help on how to use the online form
- Keeping the online form open for longer hours
- Clearer information on when the practice will reply to a request
- Translated booking forms or interpreter support when booking
- None of these — booking is already easy for me
- Other
-
Q9
Is there anything else you would like to tell the practice about booking appointments, or anything that would make it easier for you?
Open text -
Q10
Do you have any physical or mental health conditions, disabilities, or impairments that make it harder to use an online form or phone booking system?
One answer- Yes
- No
- Prefer not to say
-
Q11
Do you usually need an interpreter, or help with English, when booking or attending a GP appointment?
One answer- Yes, always or usually
- Yes, sometimes
- No
- Prefer not to say
-
Q12
Which statement best describes how you manage online forms?
One answer- I can easily fill in an online form on my own
- I can fill in an online form, though I sometimes find it difficult
- I can only fill in an online form with help from someone else
- I cannot fill in an online form or do not have internet access or a suitable device
- Prefer not to say
Sample plan
Invite all ~17,100 registered adult patients (aged 18+) with a mobile number on record via SMS containing the survey link. Set soft quotas across the four required age bands (18-44, 45-64, 65-74, 75+) and send targeted SMS reminders to underrepresented age bands (especially 65-74 and 75+) to ensure balanced subgroup sizes. Note on precision vs. sample size: under the ~900-complete constraint (~225 completes per age band), the overall margin of error at 95% confidence is +/-3.3 percentage points, but each individual age band will have a margin of error of roughly +/-6.0 to +/-6.5 percentage points (even after finite population correction for smaller strata like 75+). Achieving +/-5 percentage points at 95% confidence within every age band requires ~340 to 384 completes per band (~1,400 to 1,500 total completes), so the practice should either expand total completes to ~1,450 or accept ~+/-6.5% band precision at n=900. Note on mode bias: an SMS-to-online survey will systematically under-represent patients without smartphones/internet access, patients with severe visual/cognitive disabilities, over-75s, and patients with limited English—the very groups central to the decision. To mitigate this, provide an optional assisted phone or paper completion route at reception/via outbound calls for a sample of 75+ and limited-English patients if feasible, and weight final overall results back to the practice's true adult age distribution from the patient register.