Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Gemini 3.1 Pro Preview

Measured by Spring Prompt

Gemini 3.1 Pro Preview: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
66 of 100
Analysis rating
810
Head to head, this task
won 2 of 10

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A double-barrelled question: Q7

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first
  • Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Findings the judges found unsupported

  • Satisfaction has fallen significantly since moving online-first, and patients do not universally prefer the new system.The two surveys differ in method and respondents, so they cannot establish a significant fall in satisfaction or attribute a fall to online-first. The caveat explicitly continues to claim that the apparent drop is meaningful.
  • Nearly a quarter of patients are putting off seeking care, often due to poor online response times.Needing an appointment but not attempting to book is not necessarily putting off seeking care: some respondents' problems resolved and others obtained help elsewhere. Expected delays are not measured response times or proof of causation. A practice-wide estimate also requires explicit qualification for coverage and nonresponse bias, even if age-weighted.
  • Patients needing an interpreter are finding it much harder to use the standard pathways and heavily rely on in-person help.A booking preference among 13 interpreter users does not establish greater difficulty with standard pathways or actual reliance on reception help. The caveat acknowledges imprecision but still calls the preference clear.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9397
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1113
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 5 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Restore a morning phone booking line for every patient, while continuing to offer the online form. A targeted phone line for vulnerable groups is insufficient because 49% of 45-to-64-year-olds and 30% of 18-to-44-year-olds also prefer phone booking. Relying on online-first is suppressing care-seeking and causing deep frustration due to poor response times and repetitive form logic.

What would change it

If practice funding or operational capacity means a universal phone line would trigger unmanageable wait times and block urgent clinical care, the practice should pivot to the targeted phone/in-person line for vulnerable groups as an absolute minimum—provided it concurrently and aggressively improves the online form's usability and response times.

Findings

  1. Satisfaction has fallen significantly since moving online-first, and patients do not universally prefer the new system.

    High confidenceBase 549

    Weighted to the practice population, only 46.3% of those who tried to book were satisfied (down from 71% last year). The Practice Manager cannot tell partners it is universally working.

  2. Online-first creates severe barriers for older demographics and those with a disability.

    High confidenceBase 886

    Satisfaction crashes to 17.8% for 65-74s and 13.0% for 75+. Half of disabled respondents (50.0%) and 72.7% of the 75+ age group prefer to book by phone.

  3. Nearly a quarter of patients are putting off seeking care, often due to poor online response times.

    High confidenceBase 886

    24.3% of respondents needed an appointment but didn't try to book (an estimated 4,392 adults practice-wide). 43.7% of them expected to wait too long for a reply.

  4. Patients needing an interpreter are finding it much harder to use the standard pathways and heavily rely on in-person help.

    Low confidenceBase 13

    61.5% of respondents who need an interpreter prefer to book in person at reception, compared to just 3.8% of those who do not.

  5. There is no statistically significant gap in booking satisfaction between men and women.

    High confidenceBase 540

    Although female satisfaction (47.5%) appears lower than male (55.3%), the difference is within the margin of error (z=1.80), answering the PPG chair's question.

Caveats

  • The survey was only sent to patients with mobile numbers (58% of the adult roll), meaning digital exclusion and dissatisfaction are likely much worse in the wider practice population than these figures suggest.
  • Last year's paper survey was conducted in the waiting room, inherently biasing it towards patients who successfully booked and attended. Exact point-in-time comparisons are flawed, though the drop in satisfaction is stark enough to be meaningful.
  • The sample of patients needing an interpreter is extremely small (n=13), so while the preference for in-person booking is clear, the exact percentages carry a high margin of error.
  • Unweighted data overrepresents younger adults; figures must be evaluated against the weighted practice population, which reduces the apparent success of the online form.

Next steps

  • decisionReintroduce a universal phone booking line
  • experimentSet and publicly commit to a standard response time for online requests
  • experimentReview and simplify the online request form
  • decisionProvide dedicated in-person support for patients needing interpreters or accessibility adjustments

Open-text themes it coded

Reinstate phone booking or speak to a receptionist 96Faster responses to online requests 45Shorter, simpler, or less repetitive online form 29Continuity of care / seeing the same doctor 15Online form is working well / no changes needed 77Accessibility needs (help for disabilities, screen reader compatibility, translated forms, large print) 23

The survey it planned

2 screening questions and 8 questions, as the model wrote them.

  1. S1

    Are you currently registered as a patient at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    Which of the following age groups do you fall into?

    One answer
    • Under 18
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over
    • Prefer not to say

    Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over

  3. Q1

    Do you have a disability, long-term illness, or health condition that affects your day-to-day life?

    One answer
    • Yes
    • No
    • Prefer not to say
  4. Q2

    Do you need an interpreter or help communicating in English when interacting with the practice?

    One answer
    • Yes
    • No
  5. Q3

    Overall, how satisfied or dissatisfied are you with the appointment booking system since the change to online request forms in April 2026?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: assumptive

    All eligible patients see this question, including those with no experience of booking since the change. The satisfaction-only scale assumes they can assess the system.

    Asks everyone to rate the booking system since April 2026. Many registered patients will not have tried to book since then, and there is no 'not used it' option, so they are forced to give a rating.

  6. Q4

    Since April 2026, has there been a time when you needed an appointment but decided NOT to book one because of the new system?

    One answer
    • Yes
    • No
    • Not sure / Can't remember
  7. Q5

    Why did you decide not to book an appointment? (Select all that apply)

    Any that apply
    • The online form was too difficult or confusing to use
    • I do not have access to the internet or a smartphone
    • I prefer speaking to someone and couldn't get through on the phone
    • I was worried my request wouldn't be dealt with quickly enough
    • My symptoms got better / I managed the issue myself
    • I used another service instead (e.g., Pharmacy, 111, A&E)
    • Other

    Routing Shown only if Q4 is: Yes

  8. Q6

    Thinking about your own personal needs, how would you prefer to book your appointments at the practice?

    One answer
    • Using an online request form (the current system)
    • Calling a morning phone line (the previous system)
    • Calling a dedicated phone line only for patients needing extra help
    • In-person at the reception desk
    • No preference
  9. Q7

    Which of the following changes would make booking an appointment easier for you? (Select up to two)

    Any that apply
    • Keep the online-first system as it is now
    • Restore a morning phone booking line for every patient
    • Add a dedicated phone line and in-person help at reception ONLY for patients who need them
    • Make the online request form simpler to use
    • Other
    Judges: double-barrelled, ambiguous

    'ONLY for patients who need them' leaves eligibility undefined, so respondents may assess different versions of the targeted-support policy.

    It asks which 'changes' would help but includes 'keep the online-first system as it is', which is not a change, so the question is illogical. The option 'dedicated phone line and in-person help at reception' bundles two separate provisions into one choice. The 'select up to two' limit with mixed policy and usability options makes answers hard to interpret.

  10. Q8

    What else, if anything, would make booking an appointment easier for you?

    Open text
Sample plan

Invite all registered adult patients with a mobile number on record via an SMS text message containing a link to the survey. To mathematically achieve the strict +/- 5 percentage points margin of error at a 95% confidence level for EACH of the four age bands, factoring in the finite population of 17,100 adults, N must be roughly 340 per band. Therefore, we recommend setting a quota of 340 per age band (totaling ~1,360 completes). If fieldwork must be strictly capped at 'about 900 completes' as initially constrained, an even distribution (225 completes per age band) will result in a slightly wider margin of error of approximately +/- 6.4% per age segment. Data collection will pause when quotas are met.