Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Claude Haiku 5.5

Measured by Spring Prompt

Claude Haiku 5.5: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
88 of 100
Analysis rating
1,412
Head to head, this task
won 6 of 9

Why it is not sound

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Findings the judges found unsupported

  • Among patients who tried to book, satisfaction is 50.8% overall, and it falls steeply with age. Online-first works for under-45s but not for over-65s. Satisfaction for patients aged 65 and over is 13 to 18%, and it is not a small-sample blip.The age gradient in satisfaction is supported, but it does not establish that online-first works for under-45s and does not work for older patients. Q2 measures satisfaction with booking, not the causal effect of online-first. The categorical dismissal of small-sample uncertainty also overstates evidence from older-group bases of 45 and 23.
  • A substantial share of patients need an appointment but do not try to book. Weighted to the adult age mix, this is roughly 4,400 registered adults in three months, so the practice should treat it as unmet demand, not as a satisfaction issue.The weighted population projection is a supported calculation, but treating all reported occasions of needing an appointment without trying to book as unmet demand is not supported. Some respondents said the problem went away or they obtained help elsewhere, and unmet clinical need was not measured.
  • The online request form is seen as too long, repetitive or confusing by a sizeable group, and this complaint is spread across ages and includes people with accessibility needs. Shortening it is a low-cost change that the decision-makers should make whichever channel option they choose.The complaints support simplifying the form, but the claim that doing so is low-cost is unsupported: no implementation-cost or technical-feasibility evidence was supplied.
  • Phone demand is not confined to the three groups named in the options. Almost half of 45-64s prefer phone, and phone is the most-requested change in open text from every age band, including under-45s. Restricting phone to over-75s, disabled patients and patients with limited English would leave a large group without an option they want.It concludes that restricting phone access would leave a large group without an option they want. That rests on stated preference and on unweighted, self-selected, analyst-coded open-text counts. It sets aside that weighted preference is split and that under-45s mostly prefer online. It is used to justify restoring a phone line for everyone before the patients the survey did not reach have been heard.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9797
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1213
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3914,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 7 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✓Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✓Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Do not keep online-first as it is. Do not restrict phone access to the three named groups either. Recommended course: (1) keep the online request form, but cut its length and fix turnaround for online requests now, with a same-working-day acknowledgement and reply target; (2) restore a morning phone booking line that is open to all patients, because about 45% of adults and most older and disabled patients want it and restricting it would exclude a large group; (3) alongside that, offer a dedicated phone line and in-person assisted booking at reception for over-75s, patients with a disability and patients with limited English, and check that the assistance can be requested by people who cannot easily use a phone. Stage the phone line as a time-limited test so its capacity can be measured against wait times. If the practice cannot resource a phone line for all, the minimum acceptable option is the targeted service (option 3), but the survey shows that it would leave many 45 to 64 year olds and under-45s without a channel they want.

What would change it

Reception data showing that a morning phone line for all would overwhelm capacity, so that waits for phone calls exceed the online reply time it replaces. Evidence that online turnaround can be cut to a few hours and a shorter form raises satisfaction among patients aged 65 and over to near the level of under-45s, which would reduce the case for a phone line for all. A weighted re-survey that includes patients without a mobile and interpreter-need subgroup data that shows a different pattern from the one reported here.

Findings

  1. Practice manager's question: online-first is not clearly preferred across the registered population. Weighted to the age mix of adults, 45.0% would prefer to book by phone against 40.7% online. Unweighted, online is slightly ahead (46.2% vs 40.7%), but that lead comes from the 18 to 44 group, who are 44% of adults. Online-first cannot be presented to the partners as the patient preference.

    Medium confidenceBase 886

    Q6 by age: online 58.4% (18-44), 38.0% (45-64), 16.7% (65-74), 6.1% (75+); phone 29.9%, 49.0%, 63.0%, 72.7%. Weighting by registered-adult shares (44/32/13/11) gives phone 45.0% and online 40.7%. Stated future preference, not observed behaviour.

  2. Older patients and patients with a disability strongly prefer booking by phone or in person. The gap is large and is the clearest finding in the survey.

    High confidenceBase 102

    Q6: 65-74 phone 63.0%, in person 14.8%, online 16.7%; 75 or over phone 72.7%, in person 21.2%, online 6.1%. Disability: yes, phone 50.0%, online 23.5%, in person 15.7%; no, phone 39.5%, online 49.1%. Age-band bases are 54 and 33, so these two bands are indicative; the 75+ result has a wide margin.

  3. Among patients who tried to book, satisfaction is 50.8% overall, and it falls steeply with age. Online-first works for under-45s but not for over-65s. Satisfaction for patients aged 65 and over is 13 to 18%, and it is not a small-sample blip.

    High confidenceBase 549

    Q2 net satisfied: 18-44 64.7% (n=272); 45-64 44.0% (n=209); 65-74 17.8% (n=45); 75+ 13.0% (n=23). Overall 50.8% (n=549, 95% margin +/-4.2 points, simple random sample assumed). Weighted to the adult age mix: 46.3%.

  4. GP partner's question: we cannot tell from this survey whether satisfaction has fallen since going online. The 71% from last year and the 50.8% now come from different modes, populations and denominators, so no change can be attributed to online-first.

    Low confidenceBase 549

    Last year: paper questionnaire handed out in the waiting room in October 2025, 412 returned, 71% satisfied (fairly or very). This year: text-invited online survey of adults with a mobile on record, 549 people who tried to book, 50.8% satisfied. Last year's sample was people physically in the building, and the denominator is unknown. The gap of about 20 points is large enough to take seriously, but it is confounded.

  5. Slow replies to online requests are the most fixable barrier. Waiting too long for a reply is the most common reason people who needed an appointment did not book, and many open-text comments describe waits of one to five days. This is an operational problem, separate from the choice of channel.

    Medium confidenceBase 215

    Q5: 'I expected to wait too long for a reply' 43.7% of those who did not book (base 215; the multi-select columns sum above 100%). Open text: 45 of 246 comments ask for faster replies, often citing waits of 24 to 120 hours. Q5 is self-reported expectation, not measured turnaround.

  6. A substantial share of patients need an appointment but do not try to book. Weighted to the adult age mix, this is roughly 4,400 registered adults in three months, so the practice should treat it as unmet demand, not as a satisfaction issue.

    Medium confidenceBase 886

    Q4: 24.3% of 886 said yes (about 215 people; 95% margin +/-2.8 points). Among those who tried to book, 17.1%; among those who did not, 35.1%. Weighted by age: 25.7%, or about 4,391 of 17,100 adults. Weighting uses the age-specific Q4 results; the 75+ band (n=33) is noisy.

  7. The online request form is seen as too long, repetitive or confusing by a sizeable group, and this complaint is spread across ages and includes people with accessibility needs. Shortening it is a low-cost change that the decision-makers should make whichever channel option they choose.

    Medium confidenceBase 246

    37 of 246 open-text comments criticise the form's length, repetition or wording, or its accessibility, and several describe giving up part-way. Q5 'The online form is difficult to use': 16.7% overall, 27.8% of 65-74s and 27.3% of 75+ (small bases). Open-text coding is by the analyst, and comments are self-selected.

  8. The PPG chair's question on women: we cannot conclude that women find booking harder. Women report 7.8 points lower satisfaction than men, but this is not statistically significant at the 95% level, and age is not controlled for.

    Low confidenceBase 540

    Q2 satisfied: female 47.5% (n=303), male 55.3% (n=237). Difference 7.8 points; standard error about 4.3 points; z of about 1.8, p of about 0.07. Age composition of the female and male subgroups is not reported, so age and sex cannot be separated.

  9. The PPG chair's question on disability: satisfaction is lower among patients with a disability, and their preference for phone and in-person help is much stronger. The satisfaction gap is not statistically significant at this sample size.

    Low confidenceBase 549

    Q2 satisfied: disability yes 40.0% (n=65), no 52.3% (n=484); difference 12.3 points; z of about 1.9, p of about 0.06. Q6 is clearer: disability yes online 23.5% vs no 49.1%.

  10. The PPG chair's question on interpreter need cannot be answered. Only 13 respondents said they need an interpreter, and the satisfaction table is not broken down by interpreter need. The two open-text comments from people who need one (A112, A756) both describe form difficulties and language barriers.

    Low confidenceBase 13

    Q10: 1.5% yes (13 respondents). Q6 for those 13: in person 61.5%, phone 23.1%, online 7.7%. Q2 has no interpreter column. Open text: A112 asks for the form in Polish; A756 asks for an Urdu speaker. Treat the 13-person result as directional only.

  11. Phone demand is not confined to the three groups named in the options. Almost half of 45-64s prefer phone, and phone is the most-requested change in open text from every age band, including under-45s. Restricting phone to over-75s, disabled patients and patients with limited English would leave a large group without an option they want.

    Medium confidenceBase 337

    Q6: 45-64 phone 49.0% (n=337), vs 29.9% for 18-44. Open text: 97 of 246 comments (39%) ask for phone booking or a phone line, including 85 from patients under 65, and only 12 of the 17 comments from patients aged 65 or over ask for phone. Coding is by the analyst; comments are self-selected and not weighted.

  12. The sample probably over-represents digitally confident patients, so the satisfaction and online-preference figures are probably too favourable to online-first. Patients without a mobile number on record, who are likely to be older, were not invited at all.

    High confidenceBase 886

    Only 58% of adults (9,840 of 17,100) have a mobile on record. Patients aged 65 and over are 24% of adults but 9.8% of respondents (6.1% + 3.7%). Response rate: 886 completes from 9,840 texts (9.0%); 1,046 started. Weighting by age corrects only part of this: it cannot correct for people with no mobile or who did not respond.

Caveats

  • The sample is a text invitation to the 58% of adults with a mobile on record. Patients without a mobile, probably older, were excluded, and patients aged 65 and over are under-represented (9.8% of respondents vs 24% of adults). Age weighting corrects only part of this.
  • Response was 886 completes from 9,840 texts (9.0%). Non-response bias is unknown; people with strong views, either way, may be more likely to respond.
  • Q2 satisfaction is asked only of the 549 people who tried to book in the last three months, so it excludes the 215 who needed an appointment and did not try. Satisfaction is therefore the satisfaction of people who did book.
  • Last year's 71% came from a paper questionnaire in the waiting room, with 412 returns and a different denominator. It is not a valid baseline for this year's online survey.
  • Margins of error assume a simple random sample. They are unweighted and ignore design effects. Subgroup bases are small: 65-74 n=54, 75 or over n=33, disability n=65 in Q2, interpreter need n=13. Results for these groups are indicative only.
  • Preference (Q6) is stated, not observed. Q5 reasons are self-reported and cover only the most recent occasion. Q2 is recall of the last booking attempt.
  • Open-text counts are the analyst's coding of 246 self-selected comments. Themes overlap, and counts are of mentions, not weighted people. Borderline items were coded as phone requests only where the respondent asked to phone or speak to someone on the phone.
  • The survey does not measure outcomes: whether people who could not book were harmed, went to A&E, or were seen elsewhere (though Q5 shows 22.8% got help elsewhere). It also has no cost or reception-capacity data, so the cost of restoring a phone line cannot be weighed here.
  • The interpreter-need subgroup (n=13) cannot support any conclusion about difficulty booking. Q2 satisfaction is not broken down by interpreter need, and the sex comparison is not adjusted for age.

Next steps

  • decisionDecide to change online-first. Approve a package: shorter online form, a same-working-day reply target for online requests, and a morning phone line for booking open to all patients, with assisted booking at reception for the three named groups.
  • monitorFix online turnaround now and measure it. Set up a daily report of time from online request to first reply, and a same-working-day target.
  • experimentRun a time-limited pilot of the morning phone line for all patients, with logged call volumes, abandonment rates and wait times, and assisted in-person booking for the three named groups.
  • researchCommission a re-survey by mixed mode: paper or telephone for patients without a mobile number, with weighting, and a boost sample of over-75s, patients with a disability and patients with limited English.
  • researchRun short interviews with 15 to 20 patients from the three named groups and with patients with limited English, including those who did not book.
  • researchRepeat the satisfaction question using the same method as the October 2025 paper survey (or run both modes in parallel) before drawing any conclusion about a change since going online.

Open-text themes it coded

Wants to book by phone, or a phone booking line 97Online form too long, repetitive, confusing or hard to use 37Slow replies to online requests 45Wants a regular or known GP (continuity) 15Accessibility or support needs (disability, sight, hearing, learning disability, language, dexterity, cognition) 23Happy with online booking; no change needed 46No suggestion, don't know, or rarely uses the practice 31

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    Are you aged 18 or over?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Are you registered with a GP at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • I am not sure

    Continues if Yes

  3. Q1

    What is your age?

    One answer
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over
  4. Q2

    Since April 2026, which of these best describes you?

    One answer
    • I have asked Ashgrove for an appointment
    • I needed an appointment but did not ask for one
    • I have not needed an appointment
  5. Q3

    Do any of these apply to you? Select all that apply.

    Any that apply
    • I have a disability or long-term health condition that affects how I use a phone or the internet
    • I need an interpreter, or I prefer to communicate in a language other than English
    • I have difficulty reading or understanding written English
    • I have sight or hearing loss
    • None of these
    • Prefer not to say
  6. Q4

    Overall, how satisfied or dissatisfied are you with the way you have been able to ask for an appointment at Ashgrove since April 2026?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

    Routing Shown only if Q2 is: I have asked Ashgrove for an appointment

  7. Q5

    Thinking about your most recent request for an appointment, what, if anything, went wrong? Select all that apply.

    Any that apply
    • I could not complete the online form
    • I did not get a reply
    • The reply came too late for my needs
    • I could not get through by phone
    • The appointment offered did not suit me
    • I did not understand the reply
    • Nothing went wrong
    • Other

    Routing Shown only if Q2 is: I have asked Ashgrove for an appointment

  8. Q6

    You said you needed an appointment but did not ask for one. What were the reasons? Select all that apply.

    Any that apply
    • The online form looked too difficult to fill in
    • I did not want to use the online form
    • I could not get through by phone
    • I did not know I could ask for an appointment online
    • I did not think the problem was serious enough
    • I got help elsewhere, for example from a pharmacy, NHS 111 or a hospital
    • I thought I would not get an appointment
    • Other

    Routing Shown only if Q2 is: I needed an appointment but did not ask for one

  9. Q7

    How comfortable are you filling in an online form on a phone or computer?

    Scale

    Scale 1-5: Very uncomfortable to Very comfortable

  10. Q8

    If you needed a GP appointment, how would you most like to ask for one? Put these in order, with your first choice at the top.

    Ranking
    • Online request form
    • Text message to the practice
    • Phone reception at a set time each morning
    • Phone reception at any time
    • Go in person to reception
  11. Q9

    If you needed help to ask for an appointment, who would you want to help you? Select all that apply.

    Any that apply
    • A family member or friend
    • A member of reception staff by phone
    • A member of reception staff in person
    • An interpreter
    • A text message service
    • Other

    Routing Shown only if Q3 is: I have a disability or long-term health condition that affects how I use a phone or the internet; I need an interpreter, or I prefer to communicate in a language other than English

  12. Q10

    Which of these would make it easier for you to ask for an appointment? Select all that apply.

    Any that apply
    • A dedicated morning phone line to reception for everyone
    • A phone line and in-person help at reception for people who need it
    • Help from a family member or friend to use the online form
    • Longer opening hours
    • A clearer idea of how long a reply will take
    • Booking information in more languages
    • Keeping online booking as it is
    • Other
  13. Q11

    When the practice replies about an appointment request, how would you prefer to be contacted?

    One answer
    • By text message
    • By phone call
    • Either is fine
  14. Q12

    Is there anything else you would like to tell the practice about asking for appointments?

    Open text
Sample plan

Invite all adult patients (18 or over) registered at Ashgrove Medical Group who have a mobile number on record (approximately 17,100 adults minus those without a mobile; confirm the number from practice records) by SMS with a link to the online questionnaire. Target 900 completes. Quotas by age band, closing each band when it is full: 18 to 44 = 225, 45 to 64 = 225, 65 to 74 = 225, 75 or over = 225. Precision: at 50 percent, 225 completes gives plus or minus 6.5 percentage points at 95 percent confidence per band, not the plus or minus 5 requested. Plus or minus 5 in every band needs about 384 completes per band (about 1,540 in total), which exceeds the 900 available. The partners should choose between accepting plus or minus 6.5 in every band or prioritising plus or minus 5 for the two oldest bands (about 384 each, leaving about 130 for the two younger bands at plus or minus 8.5 or wider). Coverage limits: the survey is mobile-only and online-only, so patients without a mobile or without internet access, who are most likely to be older, disabled or needing an interpreter and most affected by the complaints, will be under-represented. Interpreter-need and disability subgroups will be small (expect fewer than 100 interpreter-need completes); report their results as indicative with counts shown. Offer the survey in the main languages used by registered patients if the practice can provide translations, and tell non-English-speaking patients that they can ask reception for help completing it. Send a reminder after 4 days to non-responders, with extra reminders to the 75 or over and 65 to 74 bands if their quotas are not filling. Weight results to the age and sex profile of registered adults from practice records.