Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Mistral Medium 3.5

Measured by Spring Prompt

Mistral Medium 3.5: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
62 of 100
Analysis rating
540
Head to head, this task
won 4 of 13

Why it is not sound

The survey

  • Presumed the stakeholder's view instead of testing it
  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A leading question: Q5

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • Missed a trap: Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • Proposed a mistaken next step: Tell the partners that satisfaction with booking has fallen from 71% to this year's figure because of online-first
  • Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Findings the judges found unsupported

  • Satisfaction with booking has fallen since the switch to online-first: from 71% (2025) to 50.8% among those who tried to book in the last three months.The two surveys used different methods and sampled different people. Their satisfaction figures do not establish a fall in satisfaction following the switch to online-first. The later caution does not undo the headline conclusion.
  • Patients who need an interpreter are far more likely to prefer in-person booking (61.5%) and far less likely to prefer online (7.7%) than those who do not need an interpreter (3.8% in-person, 46.7% online).Only 13 respondents needed an interpreter. Their observed preferences do not support a firm population-level conclusion that patients needing interpreters are far more likely to prefer in-person booking. The caveat specifically cautions about their phone preference rather than clearly qualifying this finding.
  • The most frequent open-text request is for phone booking (102 mentions), with many citing frustration with the online form's length, complexity, or slow response times.The supplied answers contain 45 mentions of faster replies, not 35. The analysis also gives conflicting theme totals later, including 148 phone-booking answers instead of the 102 reported here, so its quantitative open-text summary is not reliable.
  • Women are slightly less satisfied (47.5% net satisfied) than men (55.3%) among those who tried to book, but this difference is not statistically significant at the 95% level.The conclusion that the sex difference is not statistically significant is supported, but the stated 95% margin of error for the difference is incorrect: it is approximately 8.5 percentage points, not 7.5.

Scorecard

Numbers 8 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✗48.2%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45.1%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓10297
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✗1013
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✗4,1504,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 5 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✗Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Do not keep online-first booking as it is. Instead, restore a morning phone booking line for all patients, and add a dedicated phone line and in-person help at reception for patients who need them (over-75s, patients with a disability, and patients with limited English). This addresses the sharp drop in satisfaction among older patients, the strong preference for phone booking in these groups, and the high proportion of patients who did not try to book due to perceived barriers. The online system should be retained for those who prefer it, but simplified and with faster response times.

What would change it

Evidence that satisfaction among older patients and those with disabilities improves significantly (e.g., net satisfied rises to >60% for 65+ and >50% for those with disabilities) under a hybrid model, or that the proportion of patients who do not try to book falls below 15%. Alternatively, evidence that the online system can be made accessible and responsive enough to meet the needs of these groups without phone/in-person options.

Findings

  1. Satisfaction with booking has fallen since the switch to online-first: from 71% (2025) to 50.8% among those who tried to book in the last three months.

    High confidenceBase 549

    Q2 net satisfied (fairly or very) = 50.8% (base 549). Last year's paper survey: 71% satisfied (base 412).

  2. Older patients, especially those aged 75+, are far less satisfied with the current booking process than younger patients.

    High confidenceBase 549

    Net satisfied: 18-44 = 64.7%, 45-64 = 44.0%, 65-74 = 17.8%, 75+ = 13.0%.

  3. A significant minority of patients (24.3%) needed an appointment but did not try to book one in the last three months, with the most common reason being they expected to wait too long for a reply (43.7%).

    High confidenceBase 886

    Q4 Yes = 24.3% (base 886). Q5: 'Expected to wait too long for a reply' = 43.7% (base 215).

  4. Phone is the preferred booking method for 40.7% of respondents overall, but this rises to 63.0% for 65-74 and 72.7% for 75+. Weighted to the practice population, 45.1% would prefer phone booking.

    High confidenceBase 886

    Q6: Phone preference = 40.7% (base 886). 65-74 = 63.0%, 75+ = 72.7%. Weighted calculation: (0.44*29.9 + 0.32*49.0 + 0.13*63.0 + 0.11*72.7) = 45.1%.

  5. Patients with disabilities are less satisfied (40.0% net satisfied) and more likely to prefer phone booking (50.0%) than those without disabilities (52.3% satisfied, 39.5% phone preference).

    Medium confidenceBase 65

    Q2: Disability yes = 40.0% satisfied (base 65), no = 52.3% (base 484). Q6: Disability yes = 50.0% phone (base 102), no = 39.5% (base 784).

  6. Patients who need an interpreter are far more likely to prefer in-person booking (61.5%) and far less likely to prefer online (7.7%) than those who do not need an interpreter (3.8% in-person, 46.7% online).

    Medium confidenceBase 13

    Q6: Needs interpreter yes = 61.5% in-person (base 13), no = 3.8% (base 873). Online: yes = 7.7%, no = 46.7%.

  7. The most frequent open-text request is for phone booking (102 mentions), with many citing frustration with the online form's length, complexity, or slow response times.

    High confidenceBase 246

    Open-text analysis: 102 mentions of phone booking (base 246). Themes include 'phone booking' (102), 'simplify form' (40), 'faster replies' (35).

  8. Women are slightly less satisfied (47.5% net satisfied) than men (55.3%) among those who tried to book, but this difference is not statistically significant at the 95% level.

    High confidenceBase 540

    Q2: Female = 47.5% satisfied (base 303), Male = 55.3% (base 237). Margin of error for difference: ~7.5 percentage points (95% CI).

  9. Among 18-44 year-olds, 58.4% prefer online booking, but this drops sharply to 38.0% for 45-64, 16.7% for 65-74, and 6.1% for 75+.

    High confidenceBase 886

    Q6: 18-44 = 58.4% online (base 462), 45-64 = 38.0% (base 337), 65-74 = 16.7% (base 54), 75+ = 6.1% (base 33).

Caveats

  • The survey was sent only to patients with a mobile number on record (58% of registered adults), so it underrepresents older patients, those without mobile phones, and potentially those with lower digital literacy.
  • The response rate was low (9.0%), and the sample is younger (52.1% aged 18-44 vs. 44% in the population) and more female (54.2% vs. 51%) than the practice population, which may bias results toward online-friendly groups.
  • Last year's satisfaction figure (71%) was from a paper survey in the waiting room, which may have attracted a different demographic (e.g., older, more frequent users). Direct comparison should be treated with caution.
  • The margin of error for the net satisfied figure (50.8%) is ±4.2 percentage points at the 95% level, meaning the true figure could be as low as 46.6% or as high as 55.0%.
  • The open-text answers are qualitative and cannot be generalised to the whole population, but they provide rich context for the quantitative findings.
  • The 'put off' figure (24.3%) includes patients who may have resolved their issue elsewhere (e.g., pharmacy, NHS 111), so it does not necessarily indicate unmet need at the practice.
  • The preference for phone booking among those who need an interpreter (23.1%) is lower than expected, but the base is very small (13 respondents), so this should be treated with caution.

Next steps

  • experimentPilot a hybrid booking system: restore morning phone booking for all patients, and add a dedicated phone line and in-person help for over-75s, patients with disabilities, and those needing interpreters.
  • decisionSimplify the online request form and reduce response times to within 24 hours (ideally same-day).
  • monitorConduct a follow-up survey in 6 months to measure satisfaction and barriers under the new system, with a focus on older patients, those with disabilities, and those needing interpreters.
  • experimentRun a small-scale trial of a dedicated phone line and in-person help for the target groups (over-75s, disabilities, interpreters) before full rollout, to test feasibility and impact.
  • researchReview the accessibility of the online form for patients with disabilities (e.g., screen reader compatibility, large print, voice input) and those needing interpreters (e.g., translated forms, interpreter support).

Open-text themes it coded

Phone booking 148Simplify the online form 100Faster replies to online requests 110Continuity of care (same doctor) 19In-person booking 14Accessibility needs (disability, interpreter, large print, etc.) 23

The survey it planned

1 screening question and 12 questions, as the model wrote them.

  1. S1

    Are you aged 18 or over and registered as a patient at Ashgrove Medical Group?

    One answer
    • Yes
    • No

    Continues if Yes

  2. Q1

    Since April 2026, Ashgrove Medical Group has used an online-first system for booking appointments. How satisfied are you with the way you book appointments now?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Judges: assumptive

    Assumes respondents have booked appointments under the current arrangements. Registered adults who have not done so are still shown the question, with no applicable answer.

    Assumes the respondent has actually booked or attempted to book an appointment since the new system was introduced.

  3. Q2

    How easy or difficult do you find it to book an appointment using the current online-first system?

    Scale

    Scale 1-5: Very difficult to Very easy

    Judges: assumptive

    Assumes respondents have experience of booking through the current online-first system. There is no answer for those who have not used it.

    Assumes the respondent has used the online-first system. Patients who have not booked since the change, or who book another way, have no valid answer.

  4. Q3

    Have you ever decided NOT to book an appointment when you needed one because of the way appointments are booked now?

    One answer
    • Yes
    • No
  5. Q4

    If you have decided not to book an appointment, what was the main reason? (Select all that apply)

    Any that apply
    • I found the online form too complicated
    • I don’t have access to the internet or a device
    • I prefer to speak to someone on the phone
    • I was put off by the wait for a response
    • I didn’t know how to get help if I couldn’t use the online form
    • Other (please specify)

    Routing Shown only if Q3 is: Yes

    Judges: ambiguous

    Asks for the main reason but instructs respondents to select all that apply. Answers cannot consistently be interpreted as either the principal reason or all contributing reasons.

    It asks for 'the main reason' but says 'Select all that apply'. Respondents cannot tell whether to give one reason or several, so the results cannot be read as either.

  6. Q5

    Which of the following best describes how you would prefer to book appointments in the future? (Select one)

    One answer
    • Online only (as it is now)
    • Phone only (as it was before)
    • A mix: online for most, but phone and in-person help for those who need it (e.g., over-75s, patients with a disability, or those with limited English)
    • Other (please specify)
    Judges: leading

    Presents online-first as online-only and describes the previous arrangements as phone-only. It contrasts these exclusive options with a mixed option, while omitting online booking alongside a phone line for everyone. This distorts the decision alternatives and does not cleanly measure respondents' personal channel preferences.

    The options are unbalanced. The phone alternative is framed as 'Phone only (as it was before)', an extreme option that does not match the decision's option of restoring a morning phone line for everyone alongside online booking. There is no 'online plus phone for everyone' option, while the targeted 'mix' option is described at length. This pushes answers towards online-only or the targeted mix. The mix option also bundles phone and in-person help into one choice.

  7. Q6

    What would make booking an appointment easier for you? (Select all that apply)

    Any that apply
    • A phone line open for everyone to book appointments
    • A dedicated phone line and in-person help for patients who need it (e.g., over-75s, patients with a disability, or those with limited English)
    • Clearer instructions on how to use the online form
    • Faster responses to online requests
    • Help available at reception for those who struggle with online booking
    • Other (please specify)
  8. Q7

    Do you have a long-term health condition or disability that affects your ability to use the online booking system?

    One answer
    • Yes
    • No
    • Prefer not to say
  9. Q8

    Do you need an interpreter to help you communicate with the practice?

    One answer
    • Yes
    • No
    • Prefer not to say
  10. Q9

    What is your age?

    One answer
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over
    • Prefer not to say
  11. Q10

    To help us understand your experience, please select the option that best describes your main reason for contacting the practice recently.

    One answer
    • Routine appointment (e.g., check-up, medication review)
    • Urgent medical issue
    • Administrative request (e.g., sick note, test results)
    • Other
    Judges: assumptive, ambiguous

    Assumes every respondent has contacted the practice recently, without an option for those who have not. 'Recently' has no defined timeframe.

    It assumes the respondent has contacted the practice recently, with no 'have not contacted' option. 'Recently' is also an undefined timeframe.

  12. Q11

    How often do you use the internet for everyday tasks (e.g., shopping, banking, or booking services)?

    One answer
    • Every day
    • A few times a week
    • A few times a month
    • Less often
    • Never
  13. Q12

    Is there anything else you would like to tell us about your experience of booking appointments at Ashgrove Medical Group?

    Open text
Sample plan

Invite all adult patients (18+) registered at Ashgrove Medical Group with a mobile number on record (approx. 17,100). Aim for 900 completes, with quotas to ensure at least 225 completes per age band (18-44, 45-64, 65-74, 75+) to achieve ±5% precision at 95% confidence. No other quotas.