Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / GPT-6.1 Terra

Measured by Spring Prompt

GPT-6.1 Terra: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
83 of 100
Analysis rating
1,273
Head to head, this task
won 5 of 11

Why it is not sound

The analysis

  • Made a recommendation that does not follow from the data
  • Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.2 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9597
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1113
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 6 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✓Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Provisionally restore a morning phone booking line for every patient, while retaining the online form. Also provide assisted reception booking, interpreting and disability accommodations rather than relying on phone alone. Make the form simpler and communicate realistic response times. This is a proportionate test of broad demand and access barriers, not proof that a universal line will improve outcomes; review capacity and equity before making the arrangement permanent.

What would change it

A representative mixed-mode study or well-monitored comparison showing that a targeted service reaches all patients who cannot use online booking, meets wider demand and performs better on access, safety and capacity could favour the targeted option. Evidence that a universal line creates unacceptable queues, delays or inequities despite adequate staffing would also prompt redesign. Conversely, persistent unmet need outside the proposed eligible groups would strengthen the case for universal phone access.

Findings

  1. Do not tell the partners that patients generally prefer the new system or that online-first is working for everyone. The online form was the largest single preference in this sample, but not a majority, and nearly as many preferred phone.

    Medium confidenceBase 886

    Among 886 respondents, 46.2% preferred the online form, 40.7% phone and 4.6% reception. Among 462 aged 18–44, 58.4% preferred online. Preference is not the same as a comparison of the old and new systems.

  2. A phone option restricted to the three proposed groups would leave out many respondents who prefer phone, including people aged 65–74. This supports testing phone booking for everyone rather than keeping online-first unchanged.

    Medium confidenceBase 886

    Phone was preferred by 49.0% of 337 respondents aged 45–64 and 63.0% of 54 aged 65–74. Weighting the age-band preferences to the adult list gives 45.0%, but does not correct other selection biases. In Q3, 95 of 246 answers requested phone booking or a phone booking line; 11 were from respondents aged 65 or over.

  3. Satisfaction appears substantially below last year's figure, but the practice cannot conclude that it has fallen or that online-first caused a fall.

    Medium confidenceBase 549

    This year, 50.8% of 549 recent booking triers were satisfied, versus 71% of 412 waiting-room paper respondents last year. The nominal 95% margin of error this year is 4.2 percentage points. Applying adult age shares directly to this year's age-specific trier satisfaction gives 46.3%, not a corrected measure of all triers.

  4. There is a warning sign that some people are not attempting to book when they need care. Faster, more predictable responses may matter as well as the booking channel.

    Medium confidenceBase 886

    Of 886 respondents, 24.3% said they had needed an appointment but had not tried at least once. Among the 215 who reported this, 43.7% expected to wait too long for a reply and 28.8% did not think they would get an appointment; multiple reasons could be selected. An age-weighted projection is about 4,392 adults, but is not a reliable count of affected patients.

  5. Women in the sample reported lower satisfaction than men, but the survey does not establish a 95%-level difference. Investigate rather than dismissing the possible gap.

    Medium confidenceBase 540

    Among recent triers, 47.5% of 303 female respondents and 55.3% of 237 male respondents were satisfied: a 7.8 percentage-point difference, not statistically significant at the 95% level in an unadjusted comparison.

  6. Interpreter users may face particular barriers, but there is insufficient evidence to determine whether they find booking harder than other patients.

    Low confidenceBase 13

    Only 13 respondents said they needed an interpreter; 8 preferred reception, 3 phone and 1 online. The supplied tables do not give their booking satisfaction or unsuccessful-need rates. Two open answers requested language help with the form.

  7. Accessible alternatives must include more than a telephone: patients describe different communication, physical and cognitive needs.

    Medium confidenceBase 549

    Among recent triers, 40.0% of 65 respondents reporting a disability were satisfied, compared with 52.3% of 484 reporting no disability. Open answers requested, among other things, reception assistance, screen-reader support, larger text, carer support and BSL access.

Caveats

  • Only 9,840 of 17,100 adults had a mobile number on record and were sent the survey; 886 completed it. The 9.0% completion rate is based on those sent the text, not all adults. Selection and nonresponse bias may be substantial.
  • Older adults are underrepresented: those aged 75 or over were 3.7% of completes versus 11% of the adult list. The 75-plus satisfaction base is only 23; subgroup estimates, particularly for interpreter users, are unstable.
  • Age weighting cannot correct for people without a recorded mobile number or for differences between respondents and nonrespondents within age bands. The 4,392 projection is a calculation, not a dependable estimate of the number of affected patients.
  • The requested weighted satisfaction figure applies population age shares to satisfaction among booking triers. It is not the correct population estimate for triers unless their age distribution matches the adult list.
  • The nominal margin of error and significance test assume a probability-like independent sample; they do not account for coverage or nonresponse bias. The sex comparison is unadjusted and does not establish a cause.
  • Last year's waiting-room paper survey and this year's text survey reached different people and may have used different question wording or eligibility. Their percentages are not a like-for-like trend.
  • Q4 does not establish why care was not sought, whether harm occurred, or that the same episode was covered by Q1. Some respondents reported both trying to book and, on another occasion, not trying; some said the problem went away or they obtained help elsewhere.
  • Open-text answers were optional and can express multiple themes; their counts are not representative prevalence estimates. Phone counts include explicit requests to call or restore a phone line, not generic requests to speak to a person.
  • The survey covers adults booking for themselves. It does not establish the needs of children, carers booking for others, or patients unable to complete this survey.

Next steps

  • decisionAgree and staff a time-limited universal morning phone-booking trial, retaining online booking and providing reception and communication assistance.
  • experimentSimplify the online form, test it with disabled patients and people with limited English, and provide acknowledgement and a realistic response-time expectation.
  • researchRun a mixed-mode follow-up using phone, paper, in-person and interpreted approaches, deliberately including people without recorded mobiles and patients in small underserved groups.
  • monitorMonitor booking attempts, waits, abandoned calls, online response times, unmet need and satisfaction by age, sex, disability and interpreter need; review at a predetermined date.

Open-text themes it coded

Requests to book by phone or restore a phone booking line 95Faster replies or confirmation after an online request 45A shorter, clearer or less repetitive form 27Continuity with a doctor who knows the patient 15Help with the form in another language 2

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you currently registered as a patient at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    What is your age?

    One answer
    • Under 18
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over
    • Prefer not to say

    Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over

  3. Q1

    Since the booking system changed in April 2026, which of the following have applied to you? Select all that apply.

    Any that apply
    • I needed an appointment and tried to book one
    • I needed an appointment but did not try to book one
    • I did not need an appointment
    • I cannot remember
    • Prefer not to say
  4. Q2

    Overall, how satisfied or dissatisfied are you with your experience of booking an appointment at Ashgrove Medical Group since the change in April 2026?

    One answer
    • Very satisfied
    • Fairly satisfied
    • Neither satisfied nor dissatisfied
    • Fairly dissatisfied
    • Very dissatisfied
    • Not sure
    • Prefer not to say

    Routing Shown only if Q1 is: I needed an appointment and tried to book one

  5. Q3

    How have you tried to book an appointment at Ashgrove Medical Group since April 2026? Select all that apply.

    Any that apply
    • Completed the online request form myself
    • Someone else completed the online request form for me
    • Phoned the practice
    • Visited reception
    • Another way
    • Cannot remember
    • Prefer not to say

    Routing Shown only if Q1 is: I needed an appointment and tried to book one

  6. Q4

    Why did you not try to book when you needed an appointment? Select all that apply.

    Any that apply
    • I did not know how to request an appointment
    • I could not access or use the online form
    • I found the online form difficult or time-consuming
    • I did not know I could phone or visit for help
    • I was unsure whether I was allowed to phone
    • I found it difficult to contact the practice by phone
    • I was concerned about explaining my needs in English
    • I thought it would take too long to get a response or appointment
    • I thought an appointment would not be available
    • I did not want to bother the practice
    • My need improved or I got help elsewhere
    • Another reason
    • Cannot remember
    • Prefer not to say

    Routing Shown only if Q1 is: I needed an appointment but did not try to book one

  7. Q5

    If you needed an appointment at Ashgrove Medical Group, how would you prefer to make the initial request?

    One answer
    • Complete an online form myself
    • Have someone help me complete an online form
    • Phone the practice
    • Visit reception
    • Another way
    • No preference
    • Not sure
    • Prefer not to say
  8. Q6

    The practice is considering the following arrangements for requesting appointments. Each would keep the online request form. Which would you prefer?

    One answer
    • Keep the current system: use the online form, with reception phones for patients who cannot go online
    • Restore a morning phone booking line available to every patient
    • Add a dedicated phone line and in-person help at reception for patients aged 75 or over, patients with a disability, and patients with limited English
    • No preference
    • Not sure
    • Prefer not to say
  9. Q7

    What would make requesting an appointment easier for you? Select all that apply.

    Any that apply
    • A simpler or shorter online form
    • Clearer information about how to request an appointment
    • Clearer information about who can get help by phone or at reception
    • A phone booking line open to everyone
    • A dedicated phone line for patients who need extra help
    • Help with the online form at reception
    • Information or help in another language
    • Better accessibility for people with disabilities
    • A quicker response from the practice
    • More information about when and how the practice will respond
    • Something else
    • Nothing
    • Not sure
    • Prefer not to say
  10. Q8

    Is there anything else the practice could do to make requesting an appointment easier? Please do not include names or personal medical details. This question is optional.

    Open text
  11. Q9

    Do you consider yourself to have a disability?

    One answer
    • Yes
    • No
    • Not sure
    • Prefer not to say
  12. Q10

    Do you need an interpreter when communicating with the practice?

    One answer
    • Yes, usually
    • Yes, sometimes
    • No
    • Not sure
    • Prefer not to say
Sample plan

Send a neutral text invitation with the survey link to every registered adult patient with a mobile number on record. Explain that participation is voluntary, both positive and negative experiences are welcome, and patients may ask someone to help them complete the survey. Send one reminder where feasible. Aim for about 900 completes, but monitor responses by the four age bands without closing any band early; report the number invited and completed in each band. The requested ±5 percentage-point precision cannot be promised with 900 completes. Under simple random sampling, a conservative 95% target is about 385 completes per band before finite-population adjustment; use the actual registered adult population in each band to calculate adjusted targets. If all four bands contained about 4,275 patients, the adjusted target would be about 353 per band, or 1,412 in total. Even those calculations do not account for self-selection or coverage bias. Patients without a recorded mobile number and those unable or unwilling to complete a linked online survey may be underrepresented. Report this limitation and consider separately approved accessible outreach if the partners require stronger coverage or precision.