Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Mistral Large 4

Measured by Spring Prompt

Mistral Large 4: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
85 of 100
Analysis rating
815
Head to head, this task
won 9 of 15

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first

Findings the judges found unsupported

  • Satisfaction with booking has fallen sharply since the move to online-first, and the fall is almost entirely concentrated among older patients. Among those who tried to book, net satisfaction is 50.8% (95% CI ±4.2pp), down from 71% in last year's survey. However, the drop is not uniform: 64.7% of 18–44-year-olds are satisfied, compared with only 17.8% of 65–74-year-olds and 13.0% of over-75s. Weighted to the practice's age mix, overall satisfaction is 46.3%.The different survey methods and populations do not establish a fall in satisfaction, its attribution to online-first, or its concentration among older patients. There are no comparable previous age-specific results. The later caveat contradicts rather than corrects the headline finding.
  • Nearly one in four patients (24.3%) needed an appointment in the last three months but did not try to book one. This unmet need rises to one in three among patients aged 65 or over (33.3%). Weighted to the registered adult population, approximately 4,392 of 17,100 adults experienced this barrier in the last three months alone.The population estimate is a supported arithmetic extrapolation, but presenting approximately 4,392 adults as having experienced this barrier with high confidence is not justified by age weighting alone. Coverage and nonresponse bias remain. Needing an appointment but not trying also does not necessarily mean unmet clinical need or an access barrier: some respondents reported recovery or help elsewhere.
  • Patients with disabilities and those who need interpreters are disproportionately disadvantaged by the online-first model. Only 23.5% of patients with a disability prefer the online form (vs 49.1% of those without), and 50.0% prefer phone. Among the 13 respondents needing an interpreter, 61.5% prefer in-person booking and only 7.7% prefer online.Channel preferences do not by themselves establish that online-first disproportionately disadvantages these groups. The interpreter result describes only 13 respondents and cannot support a firm group-wide conclusion, despite the later small-base caveat.
  • In open-text answers, 94 of 246 respondents (38%) explicitly asked for phone booking to be restored, including 11 of the 16 respondents aged 65 or over who answered. A further large group asked for faster replies to online requests and a simpler, shorter form.The open-text counts are inaccurate. The supplied answers contain 18 respondents aged 65 or over, with 13 mentioning phone booking, rather than 16 and 11; the listed examples omit A114 and A644. The overall phone tally and the reported speed and form theme counts also do not match the supplied answers.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9497
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1113
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 6 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✓Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Do not keep online-first as it is, and do not restore a universal morning phone line. Instead, implement option 3: keep online-first as the default for the majority, but add a dedicated phone line and in-person reception support reserved for patients aged 75+, patients with a disability, and patients with limited English. This targets the groups where satisfaction has collapsed (13–18% for 65+) and where 50–73% prefer phone, while preserving the efficiency of online booking for the 58% of under-45s who prefer it. Simultaneously, the practice must shorten the online form and commit to a same-day response standard, as slow replies (43.7% of non-booking reasons) and form complexity are driving patients away across all age groups.

What would change it

The recommendation would change if: (1) monitoring shows the dedicated line is overwhelmed or, conversely, under-used by the target groups; (2) satisfaction among over-75s and disabled patients does not improve within 6 months of the dedicated line opening; (3) a cost analysis shows the dedicated line is not sustainable and online usage among vulnerable groups is rising anyway; or (4) follow-up research with non-bookers reveals that the primary barrier is not channel preference but clinical gatekeeping (e.g., believing they will be refused an appointment).

Findings

  1. Satisfaction with booking has fallen sharply since the move to online-first, and the fall is almost entirely concentrated among older patients. Among those who tried to book, net satisfaction is 50.8% (95% CI ±4.2pp), down from 71% in last year's survey. However, the drop is not uniform: 64.7% of 18–44-year-olds are satisfied, compared with only 17.8% of 65–74-year-olds and 13.0% of over-75s. Weighted to the practice's age mix, overall satisfaction is 46.3%.

    High confidenceBase 549

    Q2: Net satisfied (fairly/very) by age: 18–44 64.7% (base 272), 45–64 44.0% (base 209), 65–74 17.8% (base 45), 75+ 13.0% (base 23). Last year's paper survey: 71% (base 412).

  2. Nearly one in four patients (24.3%) needed an appointment in the last three months but did not try to book one. This unmet need rises to one in three among patients aged 65 or over (33.3%). Weighted to the registered adult population, approximately 4,392 of 17,100 adults experienced this barrier in the last three months alone.

    High confidenceBase 886

    Q4: Yes (needed but did not try) Total 24.3% (base 886); 65–74 33.3% (base 54), 75+ 33.3% (base 33). Weighted: 25.7% × 17,100 = 4,392.

  3. Preference for phone booking is strongly age-graded. While 45.0% of all adults would prefer to book by phone (weighted), this ranges from 29.9% of 18–44-year-olds to 72.7% of over-75s. Only 6.1% of over-75s prefer the online form.

    High confidenceBase 886

    Q6: By phone – 18–44 29.9%, 45–64 49.0%, 65–74 63.0%, 75+ 72.7% (base 886). Online form – 75+ 6.1%.

  4. Patients with disabilities and those who need interpreters are disproportionately disadvantaged by the online-first model. Only 23.5% of patients with a disability prefer the online form (vs 49.1% of those without), and 50.0% prefer phone. Among the 13 respondents needing an interpreter, 61.5% prefer in-person booking and only 7.7% prefer online.

    Medium confidenceBase 102

    Q6: Online preference – Disability yes 23.5% (base 102), no 49.1% (base 784). Interpreter yes 7.7% (base 13), no 46.7% (base 873). In-person preference – Interpreter yes 61.5%.

  5. The top reasons patients gave for not booking when they needed care were expecting to wait too long for a reply (43.7%) and not thinking they would get an appointment (28.8%). Among the 215 who did not try to book, 16.7% cited the online form being difficult and 9.3% lacked confidence going online.

    High confidenceBase 215

    Q5 (multi-select, base 215): Wait too long 43.7%; Did not think would get appointment 28.8%; Got help elsewhere 22.8%; Online form difficult 16.7%; Not confident online 9.3%.

  6. There is a gender gap in satisfaction among those who booked, but it is not statistically significant at the 95% level. Women were less satisfied (47.5%) than men (55.3%), a difference of 7.8 percentage points (Z=1.81, p=0.07).

    High confidenceBase 540

    Q2: Net satisfied – Female 47.5% (base 303), Male 55.3% (base 237). Difference test: SE=4.32pp, Z=1.81.

  7. In open-text answers, 94 of 246 respondents (38%) explicitly asked for phone booking to be restored, including 11 of the 16 respondents aged 65 or over who answered. A further large group asked for faster replies to online requests and a simpler, shorter form.

    High confidenceBase 246

    Q3 open text (246 answers): 94 mention phone/ring/call booking; 11 from 65+ (A006, A018, A110, A156, A228, A320, A332, A395, A440, A678, A886).

Caveats

  • The survey was sent by text to the 58% of patients with a mobile number on record; the sample is younger (52% aged 18–44 vs 44% of the registered population) and slightly more female (54% vs 51%) than the practice list, so results for older patients may understate the true level of dissatisfaction.
  • Last year's 71% satisfaction figure came from a paper questionnaire handed out in the waiting room; this year's 50.8% comes from a text survey of those who tried to book. The methodologies are not directly comparable, and last year's figure may have been inflated by sampling patients already attending the practice.
  • Satisfaction (Q2) was asked only of the 62% who tried to book; the 24.3% who needed care but did not try to book are excluded, so the true level of dissatisfaction with the system is likely higher than 50.8%.
  • Subgroup bases are small for some protected characteristics: only 13 respondents need an interpreter, 33 are aged 75+, and 65 disabled respondents answered the satisfaction question. Findings for these groups have wide margins of error.
  • Open-text answers come from a self-selected subset (246 of 886 completes) and may over-represent strongly motivated respondents.
  • The survey captures stated preferences and recent experience, not actual booking outcomes or clinical need; it cannot measure whether patients who did not book suffered harm as a result.

Next steps

  • decisionOpen a dedicated phone booking line and in-person reception support for over-75s, patients with disabilities, and patients needing interpreters, with priority access and reasonable adjustments (e.g., longer time slots, BSL video relay, easy-read forms).
  • decisionRedesign the online request form to reduce length and repetition, and implement a same-working-day response standard for online requests.
  • researchConduct a follow-up survey (paper and phone) of patients who did not try to book, including those without mobile numbers, to quantify unmet clinical need and barriers not captured by this text survey.
  • researchCommission an accessibility audit of the online booking form against WCAG 2.1 AA standards, including screen-reader compatibility, motor-impairment navigation, and cognitive load.
  • monitorMonitor satisfaction, channel preference, and booking volumes by age, disability, and language need on a quarterly basis, using practice list data to weight results.

Open-text themes it coded

Restore phone booking / phone line 94Faster response to online requests 36Simplify / shorten the online form 21Continuity of care / see the same doctor 15Disability and accessibility accommodations 21Language support / interpreter needed 3

The survey it planned

3 screening questions and 12 questions, as the model wrote them.

  1. S1

    Are you aged 18 or over?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Are you currently registered with Ashgrove Medical Group?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Since April 2026, have you tried to book an appointment at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  4. Q1

    What is your age?

    One answer
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over
  5. Q2

    Since April 2026, how satisfied are you with the way you book appointments at Ashgrove Medical Group?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  6. Q3

    How easy or difficult is it to book an appointment at Ashgrove Medical Group now compared to before April 2026?

    One answer
    • Much easier
    • Easier
    • About the same
    • More difficult
    • Much more difficult
    • Don't know
  7. Q4

    Since April 2026, have you ever needed an appointment but decided not to book one because of how you have to book it?

    One answer
    • Yes
    • No
    • Not sure
  8. Q5

    Why did you decide not to book an appointment? Please select all that apply.

    Any that apply
    • The online form was too difficult to use
    • I don't have access to the internet or a device to use it
    • I prefer to speak to someone on the phone or in person
    • I was worried I wouldn't get the right type of appointment
    • It took too long to get a reply
    • I need help with English and couldn't get it
    • I have a disability that makes the online form hard to use
    • Other

    Routing Shown only if Q4 is: Yes

  9. Q6

    When you last booked an appointment at Ashgrove Medical Group, how long did you wait for a reply from the practice?

    One answer
    • Less than 1 hour
    • A few hours
    • Same day
    • 1 to 2 days
    • More than 2 days
    • I did not get a reply
    • Can't remember

    Routing Shown only if Q4 is: No

    Judges: assumptive, ambiguous

    S3 establishes only an attempt to book, and Q4=No does not establish a completed booking. The response categories also overlap: a reply after a few hours can also be on the same day.

    The answer bands overlap: 'A few hours' and 'Same day' can describe the same wait. 'Reply' is undefined for patients who booked by phone or in person, so it is unclear what waiting for a reply means for them.

  10. Q7

    How would you prefer to book an appointment at Ashgrove Medical Group?

    One answer
    • Online form (as it works now)
    • Phone line open to everyone in the morning
    • A phone line just for people who need it (e.g. over 75s, disability, language needs)
    • In person at reception
    • Text message
    • I have no preference
    • Other
  11. Q8

    Do you have access to the internet and a smartphone, tablet or computer that you can use to fill in the online form?

    One answer
    • Yes
    • No
    • Not sure
  12. Q9

    Do you have a disability or long-term health condition that affects how you use health services?

    One answer
    • Yes
    • No
    • Prefer not to say
  13. Q10

    Do you need an interpreter or information in a language other than English to use health services?

    One answer
    • Yes
    • No
    • Prefer not to say
  14. Q11

    What would make booking an appointment easier for you? Please select all that apply.

    Any that apply
    • A phone line I can call in the morning
    • Someone at reception to help me in person
    • A simpler online form
    • The online form in other languages
    • Help from staff or family to use the online form
    • Being able to book by text message
    • Nothing - it is easy enough now
    • Other
  15. Q12

    Since April 2026, have you used the reception phone line because you could not or did not want to use the online form?

    One answer
    • Yes
    • No
Sample plan

Invite all registered adult patients (aged 18+) with a mobile number on record (approximately 15,000-16,000 of the 17,100 adults) via SMS containing a link to the online questionnaire. Implement stratified sampling with hard quotas by age band to ensure precision requirements are met: target 400 completes per age band (18-44, 45-64, 65-74, 75+), giving a total target of 1,600 completes. Note: 900 completes would provide only ±6.5 percentage points precision per band; 1,600 completes are required to achieve the requested ±5 percentage points at 95% confidence for each band. Close each age band quota once filled. Apply post-stratification weighting by age band, gender, and deprivation (if available) to match the practice population.