Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Gemini 3.5 Flash Lite
Gemini 3.5 Flash Lite: How Ashgrove patients find online-first booking
The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 60 of 100
- Analysis rating
- 662
- Head to head, this task
- won 4 of 10
Why it is not sound
The survey
- Ignored the precision the brief asked for, which the sample cannot deliver
The analysis
- Got at least one set number wrong
- Reported a material finding the data does not support
- Missed a trap: Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
- Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
- Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
- Missed a trap: The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
- Missed a trap: Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
- Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first
Findings the judges found unsupported
Overall satisfaction with the booking process has dropped substantially since the transition to online-first, falling from 71% under the previous paper-based system to 50.8% among recent triers.
The surveys used different methods and sampled different people. They cannot establish a fall in satisfaction, or attribute one to online-first. The later caveat does not undo the high-confidence trend claim.Satisfaction is sharply skewed by age, with older cohorts experiencing very high dissatisfaction: 67.7% of 65-to-74s and 73.9% of those aged 75 or over are dissatisfied or neutral, with net satisfaction falling to 17.8% and 13.0% respectively.
The dissatisfied-or-neutral figures are incorrect: they are 82.2% for ages 65–74 and 86.9% for ages 75 or over. The stated 73.9% is dissatisfaction alone, while 67.7% matches neither measure for ages 65–74.A substantial barrier to care exists, with nearly a quarter of all respondents (24.3%) reporting a time in the last three months when they needed an appointment but did not try to book.
The estimate of 4,038 registered adults is not supported by the stated calculation: 24.3% of 17,100 is approximately 4,155. Population extrapolation also needs qualification given the unweighted, selective sample.Patient preference is highly divided: 46.2% prefer the online form overall, but 40.7% prefer the phone, and preference shifts decisively toward the phone and in-person options among older age groups, those with disabilities, and those requiring an interpreter.
A decisive preference shift among interpreter users cannot be established from only 13 respondents. Describing their percentages as directional rather than precise does not establish that the direction is reliable.
Scorecard
Numbers 7 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process? | ✓ | 50.8% | 50.8% |
| Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied? | ✗ | 52.8% | 46.3% |
| What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one? | ✓ | 24.3% | 24.3% |
| Weighted to the age mix of registered adults, what percentage would prefer to book by phone? | ✓ | 45.4% | 45% |
| What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book? | ✓ | 4.2 points | 4.18 points |
| Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers ask to be able to book by phone, or for a phone booking line? | ✗ | 58 | 97 |
| How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line? | ✗ | 10 | 13 |
| Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one? | ✗ | 4,038 | 4,393 |
| Among respondents aged 18 to 44, what percentage would prefer to book using the online request form? | ✓ | 58.4% | 58.4% |
| What percentage of the patients sent the text message completed the survey? | ✓ | 9% | 9% |
Traps in the data 2 of 7 handled
- ✗Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
- ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
- ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
- ✗The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
- ✗Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
- ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
- ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.
The analysis
Adopt the hybrid model: maintain online-first booking while introducing a dedicated phone line and in-person reception assistance specifically for over-75s, patients with disabilities, and patients with limited English or digital exclusion needs. Simultaneously, streamline the online request form and address internal processing delays to improve turnaround times.
Evidence that online-first booking satisfaction rebounds above 70% across all demographics following administrative process improvements (faster triage/replies and form shortening), or operational proof that a dedicated phone line cannot be staffed safely without compromising clinical triage safety.
Findings
-
Overall satisfaction with the booking process has dropped substantially since the transition to online-first, falling from 71% under the previous paper-based system to 50.8% among recent triers.
Q2 satisfaction (net satisfied = 50.8%, base = 549) compared to last year's 71% baseline (base = 412).
-
Satisfaction is sharply skewed by age, with older cohorts experiencing very high dissatisfaction: 67.7% of 65-to-74s and 73.9% of those aged 75 or over are dissatisfied or neutral, with net satisfaction falling to 17.8% and 13.0% respectively.
Q2 satisfaction by age: 65-to-74 (base = 45, net satisfied 17.8%), 75 or over (base = 23, net satisfied 13.0%).
-
A substantial barrier to care exists, with nearly a quarter of all respondents (24.3%) reporting a time in the last three months when they needed an appointment but did not try to book.
Q4 ('Yes' = 24.3%, base = 886), scaling to an estimated 4,038 registered adults across the practice.
-
The primary reasons for not booking are expected delays in practice replies (43.7%) and low expectation of securing an appointment (28.8%), alongside digital exclusion and usability barriers for older and vulnerable patients.
Q5 among those who did not try to book (base = 215): expected wait too long 43.7%, did not think I would get an appointment 28.8%, online form difficult 16.7%, not confident online 9.3%.
-
Patient preference is highly divided: 46.2% prefer the online form overall, but 40.7% prefer the phone, and preference shifts decisively toward the phone and in-person options among older age groups, those with disabilities, and those requiring an interpreter.
Q6: 72.7% of 75+ prefer phone, 21.2% prefer in-person; 50.0% of disabled patients prefer phone, 15.7% in-person; 61.5% of interpreter users prefer in-person, 23.1% phone.
Caveats
- Survey response rate was 9.0% (886 completes from 9,840 text messages sent), introducing potential non-response bias where digitally excluded or highly dissatisfied patients may be under-represented or over-represented.
- The survey sample skews younger (52.1% aged 18-44) compared to the practice adult population (44%), requiring weighted analysis to prevent age-skewed distortions.
- Last year's baseline satisfaction (71%) was measured using a different methodology (paper questionnaires in the waiting room among 412 respondents), which may not be directly comparable to digital/SMS survey collection.
- Subgroups with specific needs, such as patients requiring interpreters (base = 13), have very small sample sizes, meaning percentages should be interpreted directionally rather than as precise statistical estimates.
Next steps
- experimentDesign and pilot the dedicated phone booking line and in-person reception support pathway targeted at over-75s, disabled patients, and those needing interpreters.
- decisionStreamline and shorten the online request form, reducing repetitive questions and simplifying language.
- monitorReview and optimize internal triage and response workflows to shorten reply times for online requests.
- researchConduct targeted qualitative follow-up research with patients who require interpreters (e.g., Polish and Urdu speakers) to evaluate accessibility barriers.
Open-text themes it coded
Request for phone booking 89Slow response times from practice 41Form too long, complex or repetitive 22Desire for continuity of care / same doctor 15Accessibility, disability and language barriers 24
The survey it planned
1 screening question and 10 questions, as the model wrote them.
-
SQ1
Are you currently registered as a patient at Ashgrove Medical Group and aged 18 or over?
One answer- Yes
- No
Continues if Yes
-
Q1
Which age group are you in?
One answer- 18 to 44
- 45 to 64
- 65 to 74
- 75 or over
-
Q2
Do you have a physical or mental health condition or disability that lasts or is expected to last 12 months or more, which affects your ability to use online services?
One answer- Yes
- No
- Prefer not to say
-
Q3
How well do you speak and understand English?
One answer- Very well
- Well
- Not well
- Not at all
-
Q4
Have you tried to book an appointment at Ashgrove Medical Group since April 2026?
One answer- Yes
- No
-
Q5
Overall, how satisfied or dissatisfied are you with the way appointments have been booked since April 2026?
ScaleScale 1-5: Very dissatisfied to Very satisfied
Routing Shown only if Q4 is: Yes
-
Q6
Have you ever needed to book an appointment since April 2026, but decided not to because of the booking process?
One answer- Yes, on multiple occasions
- Yes, on one occasion
- No, never
-
Q7
What was the main reason you were put off booking an appointment? (Select all that apply)
Any that apply- Found the online request form too difficult to complete
- Did not have access to the internet or a suitable device
- Did not want to wait for a text or phone call back
- Found it too difficult to get through on the phone instead
- Worried about my privacy or data security online
- Prefered to wait and see if symptoms cleared up on their own
- Other reason
Routing Shown only if Q6 is: Yes, on multiple occasions; Yes, on one occasion
Judges: ambiguousThe question asks for the main reason but instructs respondents to select all that apply. Answers could represent either the principal reason or every applicable reason.
It asks for the 'main reason' (one answer) but says 'Select all that apply', so respondents cannot tell whether to pick one reason or several. The results cannot be read either as a main reason or as all reasons.
-
Q8
Which of the following booking arrangements would you prefer the practice to use?
One answer- Keep online-first booking as it is now
- Restore a morning phone booking line for every patient
- Keep online-first for most, but add a dedicated phone line and in-person help at reception specifically for patients who need them (such as over-75s, patients with a disability, and patients with limited English)
- Not sure / No preference
-
Q9
Which of the following would make it easier for you to book an appointment? (Select all that apply)
Any that apply- Simpler or shorter online request forms
- Clearer instructions on how to use the online system
- Help from practice staff over the phone or at reception
- Being able to book directly into a calendar slot online without filling in a triage form
- A dedicated phone line for older patients, disabled patients, or those requiring translation
- Nothing, I find the current system works well
- Other
-
Q10
Is there anything else you would like to tell us about booking appointments at Ashgrove Medical Group?
Open text
Sample plan
An SMS text message containing a secure survey link will be sent to all 17,100 registered adult patients at Ashgrove Medical Group for whom a mobile phone number is held on record. To achieve a 95% confidence level with a margin of error within +/- 5 percentage points across each of the four age bands (18-44, 45-64, 65-74, 75+), we target a minimum of approximately 900 completed responses overall, with soft quotas set to ensure sufficient representation in each age group (aiming for roughly 225 completes per age band).