Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / GPT-6.2 Astra

Measured by Spring Prompt

GPT-6.2 Astra: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
92 of 100
Analysis rating
1,467
Head to head, this task
won 4.5 of 10

Why it is not sound

The analysis

  • Made a recommendation that does not follow from the data
  • Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9797
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1313
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 7 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✓Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✓Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Of the three options, pilot restoring a staffed morning phone booking line for everyone alongside the online form. Do not make age, disability disclosure or interpreter status a condition of calling: demand extends beyond those groups, particularly among 45–74s. Retain targeted reception and interpreter assistance as an accessibility adjustment, not as a substitute for a generally available phone route. A phone line alone will not help everyone or create appointment capacity. Fund and evaluate the pilot before making it permanent; the survey supports testing access, not a particular opening hour or staffing level.

What would change it

Prefer a dedicated assisted service if a staffed universal line persistently worsens waits or displaces clinical or online capacity, and a trial of targeted phone and reception help achieves comparable successful access, including for people outside the named groups who struggle. Consider online-first without a general line only if accessible follow-up with nonresponders and operational evidence show that unmet need is low across groups and that improving the form and reply times works as well. High call volume by itself would not demonstrate success.

Findings

  1. To the practice manager: keep the online form, but do not tell partners that patients generally prefer online-first. There is substantial demand for both routes, and the apparent lead for online depends on who answered.

    Medium confidenceBase 886

    Of 886 respondents, 46.2% preferred online, 40.7% phone and 4.6% reception. Applying the registered adult age mix puts phone at 45.0% and online at 40.7%. Online was preferred by 58.4% of 462 under-45s; phone was preferred by 49.0% of 337 aged 45–64 and 63.0% of 54 aged 65–74. Q6 asked about a future booking channel, not approval of a policy or a morning-only line.

  2. To the GP partner: this year's figure is lower, but the surveys cannot establish that satisfaction fell, still less that online-first caused a fall.

    High confidenceBase 549

    Of 549 recent triers, 50.8% were satisfied, 33.9% dissatisfied and 15.3% neutral. The observed satisfied share is 20.2 percentage points below last year's 71% of 412 waiting-room respondents. This year's nominal 95% margin is ±4.2 points. Standardising the age-specific satisfaction rates to the adult list gives 46.3%, not a like-for-like historical comparison.

  3. Do not use 75 as a hard threshold for help. The sharpest warning is that difficulties appear well before that birthday.

    Medium confidenceBase 549

    Satisfied: 64.7% of 272 triers aged 18–44, 44.0% of 209 aged 45–64, 17.8% of 45 aged 65–74 and 13.0% of 23 aged 75+. The two oldest groups have small bases, but both report low satisfaction and a strong phone preference. Among 65+ commenters, 13 asked for phone or live-person booking.

  4. Offer more than a phone alternative for disabled patients. The barrier for one person can be the accessible route for another.

    Medium confidenceBase 886

    Of 102 disabled respondents, 50.0% preferred phone, 15.7% reception and 23.5% online, versus 39.5%, 3.2% and 49.1% of 784 non-disabled respondents. Of those who tried, 40.0% of 65 disabled respondents were satisfied versus 52.3% of 484 others. Comments describe both inability to operate the form and inability to follow a phone conversation.

  5. To the participation group: female respondents reported lower satisfaction, but the evidence does not establish that women find booking harder. Nor does it establish that there is no gap.

    Medium confidenceBase 540

    47.5% of 303 female triers versus 55.3% of 237 male triers were satisfied. The female-minus-male gap is −7.8 points; an approximate two-sided two-proportion test gives p ≈ 0.07 (95% interval for the difference about −16.3 to +0.7 points). This is an unadjusted comparison of satisfaction, not a direct measure of difficulty.

  6. To the participation group: there are specific language barriers worth fixing now, but we cannot quantify whether interpreter users find booking harder than other patients.

    Low confidenceBase 13

    Only 13 respondents needed an interpreter: 8 preferred reception, 3 phone and 1 online. Among the 873 others, 3.8% preferred reception. A112 wanted a Polish form to avoid describing symptoms incorrectly; A756 wanted Urdu-speaking help without relying on family. There is no interpreter breakdown of satisfaction or forgone attempts.

  7. Measure access before the booking attempt as well as after it. Faster, more predictable replies matter alongside the choice of channel.

    Medium confidenceBase 886

    215 of 886 (24.3%) needed an appointment at least once but did not try; this included 17.1% of 549 who did try on another occasion. Of those 215, 43.7% expected a slow reply, 28.8% doubted they would get an appointment, 16.7% found the form difficult and 9.3% lacked online confidence. Reasons overlap; 22.8% got help elsewhere and 17.7% said the problem went away. Age-standardisation gives 25.7%, or roughly 4,400 adults if age-specific respondent rates applied across the list, not a count of missed appointments.

  8. Test staffed phone booking, not simply a number that takes messages. Patients describe wanting to explain a problem, ask questions and know what happens next in one conversation.

    Medium confidenceBase 246

    97 of 246 commenters asked for phone or live-person booking, including requests outside the proposed priority groups. A236 wanted confirmation while still on the call; A659 did not want to justify speaking to someone. Conversely, A455 valued submitting after the children were in bed. Comments explain what each channel offers; their frequencies are not votes from the patient list.

Caveats

  • The text reached an invitation frame of 9,840 of 17,100 adults (58%); 7,260 had no mobile number on record. Of 1,046 starters, 886 completed (84.7% of starters but only 9.0% of invitees). Digital exclusion and willingness to reply may affect both coverage and results, in directions not fully known.
  • Adults aged 65+ made up 9.8% of completes versus 24% of the list; female respondents made up 54.2% versus 51%. Age weighting corrects the mix of four age bands only, not selection within them. Weighting satisfaction by all adults' age shares answers the specified standardisation question; the actual age distribution of adult booking triers need not match the list.
  • The ±4.2 points uses 1.96 × sqrt[p(1−p)/549]. It and the sex test are nominal simple-random-sample calculations, not bounds on coverage or nonresponse bias. Small subgroup percentages are particularly unstable, and overlapping age, disability and language groups cannot be added or disentangled from these tables.
  • Last year's paper waiting-room survey selected attendees; this year's text survey asked recent triers about their last attempt, including attempts that may not have secured an appointment. Differences in recruitment, mode and reference event prevent a trend or causal conclusion. Neither Q2 nor Q4 identifies the booking channel responsible.
  • Only 246 of 886 supplied text. Themes are non-exclusive and count people, not repeated phrases. The phone theme includes live spoken booking requests that imply rather than name a phone line (notably A114 and A644); it does not count in-person-only help. Interpreter need is not identical to limited English.

Next steps

  • decisionAgree pilot capacity, a pre-pilot baseline and review criteria with reception, clinical staff and the participation group; publicise all routes in accessible formats.
  • experimentRun a time-limited universal morning line with supported reception booking. Compare matched weekdays before and during the pilot and record requests that the morning window misses.
  • experimentRemove repetitive form fields, acknowledge receipt with a realistic reply time, and test accessible completion and assisted routes with disabled patients and interpreter users.
  • researchRecruit from the full adult list by post, phone and accessible or translated interviews, including non-attenders, those without a recorded mobile, older adults and interpreter users. Ask about difficulty, channel attempted, outcome and need without an attempt.
  • monitorRepeat the same mixed-mode access questions with consistent eligibility and recall periods, alongside weekly service records.

Open-text themes it coded

Phone booking or a live person to speak to when booking 97Shorter, clearer, less repetitive online form 29Faster replies or confirmation after an online request 45Online booking already works well; retain its convenience 45Disability-related adaptations, assisted booking or an accessible alternative 21Translated form or language support 2Continuity: booking with a clinician who knows the patient 15No specific suggestion, uncertain or little recent experience (not necessarily endorsement) 32

The survey it planned

2 screening questions and 11 questions, as the model wrote them.

  1. S1

    Are you currently registered as a patient at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    How old are you?

    One answer
    • Under 18
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over

    Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over

  3. Q1

    Since the booking system changed in April 2026, have you felt you needed an appointment for yourself at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • Not sure
  4. Q2

    Since the change in April 2026, have you tried to request an appointment for yourself at the practice? Count attempts even if you did not finish, and requests someone made for you.

    One answer
    • Yes
    • No
    • Not sure
  5. Q3

    Overall, how satisfied or dissatisfied have you been with the process of booking an appointment at the practice since the change in April 2026?

    One answer
    • Very satisfied
    • Fairly satisfied
    • Neither satisfied nor dissatisfied
    • Fairly dissatisfied
    • Very dissatisfied
    • Not sure

    Routing Shown only if Q2 is: Yes

  6. Q4

    Since the change in April 2026, has the way booking works ever put you off requesting an appointment you felt you needed? Count times you waited longer to ask, as well as times you never asked.

    One answer
    • Yes, I waited longer to ask
    • Yes, I did not ask at all
    • Yes, both happened on different occasions
    • No
    • Not sure

    Routing Shown only if Q1 is: Yes

  7. Q5

    What put you off requesting an appointment on those occasions? Select all that apply.

    Any that apply
    • I did not know how to request an appointment
    • I did not have a suitable device or internet connection
    • I found the online form difficult to use
    • I could not use the form at the time I needed it
    • I wanted to explain my request to a person
    • I could not get through by phone
    • I found it difficult to use the booking system in English
    • The booking arrangements did not meet my access needs
    • I was unsure when I would hear back
    • I was concerned about privacy
    • Another reason
    • I do not remember

    Routing Shown only if Q4 is: Yes, I waited longer to ask; Yes, I did not ask at all; Yes, both happened on different occasions

  8. Q6

    If you needed an appointment and could choose how to request it, which way would you prefer?

    One answer
    • Fill in an online form
    • Speak to someone by phone
    • Speak to someone in person at reception
    • Another way
    • It depends on the situation
    • No preference
    • Not sure
  9. Q7

    Which booking arrangement would you prefer Ashgrove Medical Group to use? With each arrangement, patients can send an online request and receive a reply by text or phone.

    One answer
    • Keep the current arrangement: the reception phone is for patients who cannot go online
    • Bring back a morning appointment phone line that any patient can use
    • Add a dedicated phone line and in-person help at reception only for patients aged 75 or over, patients with a disability and patients with limited English
    • None of these
    • No preference
    • Not sure
  10. Q8

    What one change, if any, would make it easier for you to request an appointment? Please do not include names, contact details or information about a medical problem. You can leave this blank.

    Open text
  11. Q9

    Do you have a disability?

    One answer
    • Yes
    • No
    • Not sure
    • Prefer not to say
  12. Q10

    How easy or difficult is it for you to explain an appointment request in English?

    One answer
    • Very easy
    • Fairly easy
    • Neither easy nor difficult
    • Fairly difficult
    • Very difficult
    • Prefer not to say
  13. Q11

    Do you need an interpreter when contacting the practice about an appointment?

    One answer
    • Yes, always
    • Yes, sometimes
    • No
    • Prefer not to say
Sample plan

Send a neutral SMS with a single-use survey link to every registered adult with a mobile number on record. Invite people even if they have not booked recently. Offer translated, screen-reader-compatible versions and help by phone or at reception to enter the patient's own answers. Pilot completion time, including assisted use, against the six-minute limit. Aim for about 900 eligible completes, with soft monitoring targets of 225 per age band and reminders to nonrespondents in underrepresented bands; do not close full bands. Check mobile coverage, delivery and response by age against the adult register. Four estimates within ±5 percentage points at 95% confidence cannot be promised. For a large band, a simple random sample needs about 385 completed answers per estimate at the worst-case proportion of 50% (about 1,540 across four bands); 225 gives about ±6.5 points before any finite-population correction. Check actual band sizes and question-specific bases: only booking attempters answer the satisfaction question. If ±5 is essential, agree a larger age-stratified probability sample, allow for nonresponse and routing, and include patients without mobiles through other contact modes. More online responses alone cannot remove coverage or self-selection bias.