Sign inTest your prompt

Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Gemini 3.8 Flash

Measured by Spring Prompt

Gemini 3.8 Flash: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
75 of 100
Analysis rating
898
Head to head, this task
won 6 of 10

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • Missed a trap: Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first

Findings the judges found unsupported

  • Online-first booking does not have majority support across the practice population; overall booking satisfaction has plummeted, driven by extreme dissatisfaction among older and disabled patients.The claim that satisfaction has plummeted is not supported by comparable surveys. The methods and populations differ, and there is no basis for asserting that fielding differences cannot explain the apparent drop. The data also cannot establish that older and disabled patients drove a change over time.
  • The practice cannot tell partners that patients prefer the new system: preference is split by age, and age-weighted preference favors phone booking over online.Age-weighted online preference is approximately 40.7%, not 38.8%. The weighted phone preference of 45.0% is correct.
  • Online-first booking is suppressing access: nearly a quarter of patients report unmet need, representing over 4,300 adults practice-wide.Q4 establishes reported need without an attempt to book, not that online-first caused suppressed access or that every such episode remained unmet. Some respondents reported recovery or help elsewhere. Extrapolating to approximately 4,392 adults as a high-confidence population finding is not justified by this selectively reached, self-selected sample; age weighting does not remove those biases. The older respondents' reason percentages also have bases of only 18 and 11.
  • Patients who need an interpreter face severe structural exclusion under online-first and overwhelmingly rely on in-person reception access.Only 13 respondents needed an interpreter. Their stated booking preferences do not establish severe structural exclusion or actual reliance on reception across this patient group. The caveat calls for validation but continues to present severe exclusion as established rather than clearly treating these estimates as too small to rely on.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9497
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1113
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 4 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✗Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Adopt Option 3 immediately as an equitable hybrid model: retain online-first for the majority (which works well for 58.4% of under-45s) while establishing a dedicated priority phone line and in-person reception booking support for vulnerable cohorts (patients aged 75+, patients with disabilities, and non-English speakers). Concurrently, streamline the online form's repetitive questions and establish an SLA of same-day response to resolve the primary driver of online dissatisfaction.

What would change it

Evidence that opening general morning phone lines for all 21,000 patients would overwhelm practice telephony without improving appointment availability, or evidence that dedicated routing for vulnerable patients fails to reduce suppressed demand (Q4 unmet need) among over-75s and disabled cohorts.

Findings

  1. Online-first booking does not have majority support across the practice population; overall booking satisfaction has plummeted, driven by extreme dissatisfaction among older and disabled patients.

    High confidenceBase 549

    Satisfaction among those who tried to book in the last 3 months is 50.8% unweighted (46.3% age-weighted), compared to 71% in 2025. Satisfaction drops from 64.7% among 18-44s to 44.0% (45-64), 17.8% (65-74), and 13.0% (75+), and is only 40.0% among disabled patients.

  2. The practice cannot tell partners that patients prefer the new system: preference is split by age, and age-weighted preference favors phone booking over online.

    High confidenceBase 886

    Overall unweighted preference is 46.2% online vs 40.7% phone; when reweighted to the practice's true age distribution, 45.0% prefer phone and only 38.8% prefer online. Phone is preferred by 63.0% of 65-74s, 72.7% of 75+s, and 50.0% of disabled patients.

  3. Online-first booking is suppressing access: nearly a quarter of patients report unmet need, representing over 4,300 adults practice-wide.

    High confidenceBase 886

    24.3% of respondents (25.7% age-weighted, equating to ~4,392 adults) needed an appointment in the last 3 months but did not try to book. Among them, 43.7% expected to wait too long for a reply and 28.8% felt they would not get an appointment. Digital exclusion (not confident online) affected 44.4% of non-triers aged 65-74 and 36.4% aged 75+.

  4. Patients who need an interpreter face severe structural exclusion under online-first and overwhelmingly rely on in-person reception access.

    Medium confidenceBase 13

    61.5% of patients requiring an interpreter prefer booking in person at reception, compared to only 7.7% online and 23.1% by phone. Qualitative feedback shows high anxiety over English-language forms and reliance on family members.

  5. The reported gender satisfaction gap (47.5% female vs 55.3% male) is not statistically significant at the 95% confidence level.

    High confidenceBase 540

    The 7.8 percentage point gap yields z = 1.80 (p = 0.072), which does not cross the standard significance threshold (p < 0.05). Gender does not appear to be the primary axis of inequity compared to age, disability, and language.

  6. Even among digitally capable patients who use the online form, prolonged turnaround time and repetitive form design generate widespread frustration.

    High confidenceBase 246

    43.7% of those put off booking cited waiting too long for a reply. In open feedback, dozens of 18-44 and 45-64 patients cited multi-day silences (36-48+ hours) and overly lengthy, repetitive medical questionnaires.

Caveats

  • Mode and sampling bias: The survey was sent via SMS to patients with a mobile on file (58% of list), excluding the 42% without recorded mobiles who skew older and digitally excluded. The sample under-represents patients aged 65+ (9.8% in sample vs 24.0% on practice list), meaning true practice-wide dissatisfaction with online-first is likely worse than raw percentages suggest.
  • Mode divergence with historical baseline: Last year's 71% satisfaction came from a paper survey in the waiting room among attendees (selection bias toward active attenders). While comparison must be treated with caution, the drop to 50.8% (46.3% weighted) is too large to be an artifact of fielding mode alone.
  • Small sub-sample for interpreter needs: Only 13 respondents (1.5%) identified as needing an interpreter, reflecting low English literacy response rates to an English SMS. Findings for this group indicate severe exclusion but require targeted qualitative validation.

Next steps

  • decisionImplement a dedicated telephone bypass line and reception desk booking protocol flagged on records for over-75s, disabled patients, and those needing language support.
  • experimentAudit and streamline the online triage form to eliminate repetitive symptom questions and introduce clear progress indicators.
  • decisionEstablish an internal SLA guaranteeing same-day triage communication for all online submissions received before 2pm.
  • researchConduct targeted outreach and in-person interviews with non-English speaking patients via community partners.
  • monitorTrack monthly booking satisfaction, unmet demand rates, and call queue metrics segmented by patient age band and disability status.

Open-text themes it coded

Reinstate phone booking / telephone access 94Faster response times and same-day triage turnaround 45Simplify and shorten the online form 28Accessibility, disability accommodations, and language support 23Continuity of care / seeing a regular GP 15Satisfaction with existing online form / no change required 77

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Are you currently registered as a patient with Ashgrove Medical Group?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    How old are you?

    One answer
    • Under 18
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over
    • Prefer not to say

    Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over

  3. Q1

    Do you have any long-term physical or mental health conditions, disabilities, or illnesses that affect your daily life or ability to use digital devices?

    One answer
    • Yes
    • No
    • Prefer not to say
  4. Q2

    Do you require an interpreter or language support when communicating with the practice?

    One answer
    • No, I can communicate comfortably in English
    • Yes, I need a spoken language interpreter
    • Yes, I need a British Sign Language (BSL) interpreter
    • Prefer not to say
  5. Q3

    Since the practice moved to online-first appointment requests in April 2026, have you needed an appointment?

    One answer
    • Yes, and I tried to book one
    • Yes, but I was put off and decided not to try
    • No, I have not needed an appointment since April 2026
  6. Q4

    What put you off from trying to book an appointment? (Select all that apply)

    Any that apply
    • I do not have access to the internet or a smartphone/computer
    • I found the online form too confusing or complicated
    • I worried my request would not be answered promptly
    • I find it difficult or impossible to write in English
    • Long phone wait times when trying to call reception instead
    • A previous negative experience with the new booking system
    • I sought care elsewhere (e.g., Pharmacy, NHS 111, A&E, Walk-in Centre)
    • Other

    Routing Shown only if Q3 is: Yes, but I was put off and decided not to try

  7. Q5

    Thinking about your most recent experience requesting an appointment since April 2026, which method did you use?

    One answer
    • Filled in the online request form myself
    • Asked a family member, friend, or carer to fill in the online form for me
    • Telephoned the practice reception
    • Visited the practice reception in person
    • Other

    Routing Shown only if Q3 is: Yes, and I tried to book one

  8. Q6

    How easy or difficult was it for you to submit your appointment request?

    Scale

    Scale 1-5: Very difficult to Very easy

    Routing Shown only if Q3 is: Yes, and I tried to book one

  9. Q7

    Overall, how satisfied or dissatisfied are you with the process of booking an appointment at Ashgrove Medical Group since April 2026?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: assumptive

    Everyone sees this question, including patients who have neither needed nor attempted a booking since the change. It requires a satisfaction rating without an option for having no experience or being unable to judge.

    Q7 is shown to everyone, including people who said at Q3 they have not needed an appointment since April 2026. It has no 'not applicable' option, so it assumes everyone has experience of the booking process.

  10. Q8

    Since April 2026, have you started filling out an online request or calling the practice and given up before finishing?

    One answer
    • Yes, more than once
    • Yes, once
    • No, never
    • Not applicable (have not tried)
  11. Q9

    Why did you give up on completing your request? (Select all that apply)

    Any that apply
    • The online form asked for too much personal or medical detail
    • Technical problem or error with the online form
    • The questions were hard to understand or answer
    • I was on hold on the telephone for too long
    • I was told by phone or reception to go online, but could not do so
    • Other

    Routing Shown only if Q8 is: Yes, more than once; Yes, once

  12. Q10

    The practice is reviewing how patients book appointments. Which of these three options would you most prefer for Ashgrove Medical Group going forward?

    One answer
    • Option A: Keep online-first as it is (online form for most, standard phone line only for those who cannot get online)
    • Option B: Re-open a morning phone booking line open to all patients, alongside the online form
    • Option C: Keep online-first for general patients, but provide a dedicated priority phone line and in-person reception booking support specifically for patients aged 75+, patients with disabilities, and patients needing language support
    • No preference / Not sure
  13. Q11

    Which of the following changes would make booking an appointment easier for you? (Select up to three)

    Any that apply
    • A shorter, simpler online form
    • Being able to choose an appointment slot directly rather than waiting for a callback or text
    • A dedicated telephone number for patients who struggle with digital tools
    • A dedicated help desk at reception to book in person
    • Forms and instructions available in other languages
    • Quicker confirmation that my request has been received
    • Clearer timeframes for when the practice will reply
    • None of these - the current system works well for me
  14. Q12

    Is there anything else you would like the practice partners to know about your experience or what would improve appointment booking?

    Open text
Sample plan

Target population: All registered adult patients (aged 18+) at Ashgrove Medical Group with a mobile number on record (~17,100 adults). Distribution: SMS broadcast containing a unique survey link. Sample size and precision: To achieve the partners' required margin of error within ±5 percentage points at a 95% confidence level for each of the four age bands (18-44, 45-64, 65-74, and 75+), a minimum of 350-384 completed responses is statistically required per age band. Consequently, the sampling will use stratified quota targets of n=380 per age band (total target = ~1,520 completes; if capping at ~900-1,000 overall, age quotas of n=225-250 yield ~±6.2% precision, so we recommend boosting SMS reminders to the 65-74 and 75+ cohorts to achieve at least 350 completes per band to meet the partners' ±5% specification). Closed quotas on over-represented groups once reached.