Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / GPT-6.1 Sol

Measured by Spring Prompt

GPT-6.1 Sol: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict
✗ Not sound
Research score
93 of 100
Analysis rating
1,650
Head to head, this task
won 8 of 9

Why it is not sound

The analysis

  • Made a recommendation that does not follow from the data
  • Missed a trap: Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?✓50.8%50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?✓46.3%46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?✓24.3%24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?✓45%45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?✓4.18 points4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?✓9797
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?✓1313
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?✓4,3924,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?✓58.4%58.4%
What percentage of the patients sent the text message completed the survey?✓9%9%

Traps in the data 6 of 7 handled

  • ✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
  • ✗Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
  • ✓Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
  • ✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
  • ✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
  • ✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
  • ✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Do not keep online-first unchanged. Of the three options, recommend a time-limited, evaluated restoration of morning phone booking for every patient, while retaining online booking and adding guaranteed reception assistance and appropriate language and disability adjustments. This is a provisional service-design recommendation, not proof that universal phone access will outperform targeted support. Restricting help to the proposed categories risks excluding patients aged 65–74 and younger people who struggle online without belonging to a listed group. Phone requests should enter the same clinical prioritisation process as online requests, rather than gain preferential access to appointments. Improve online acknowledgement, response expectations and form accessibility alongside the pilot. Confirm staffing and safe capacity before launch; if immediate universal restoration is not feasible, introduce targeted help now as an interim safeguard, with a clearly advertised route for anyone else who cannot use online booking.

What would change it

A sufficiently inclusive study and operational evaluation showing that needs-based phone and reception support reaches people with genuine barriers outside the listed groups, removes avoidable non-attempts, and performs as well as universal phone access with materially lower workload would favour targeted support. Conversely, persistent exclusion under targeted access would strengthen the case for universal access. Evidence that a morning-only line produces long queues, abandoned calls or delayed clinical assessment would change its hours or operating model rather than justify unsupported online-only access. Keeping online-first substantially as it is would require representative evidence of equitable access, timely responses and low avoidable unmet need, supported by operational data rather than preference alone.

Findings

  1. The practice manager should not tell partners that patients generally prefer online-first or that its success has been established. Online is the largest single preference among respondents, but not a majority preference, and satisfaction is mixed.

    Medium confidenceBase 886

    Among 886 respondents, 46.2% preferred online, 40.7% phone and 4.6% reception. Among the 549 who tried to book, 50.8% were satisfied, 33.9% dissatisfied and 15.3% neutral. Age-standardising preferences gives approximately 45.0% phone versus 40.7% online. Preference for an online channel does not establish approval of an online-first policy.

  2. The GP partner can report a lower satisfaction figure in this survey, but cannot claim that satisfaction has demonstrably fallen because of online-first.

    Low confidenceBase 549

    Current satisfaction is 50.8% among 549 recent booking triers, compared with 71% among 412 respondents last year: a descriptive difference of 20.2 percentage points. The current nominal 95% margin of error is approximately 4.18 points. However, last year's waiting-room paper survey and this year's text-invited survey recruited different populations and are not a reliable before-and-after comparison.

  3. Older respondents show substantial access concerns, including those aged 65–74 who would not qualify for age-based help under the targeted option.

    Medium confidenceBase 87

    Among ages 65–74, 63.0% preferred phone and 14.8% reception, with only 17.8% satisfied among 45 booking triers. Among ages 75+, 72.7% preferred phone and 21.2% reception, with 13.0% satisfied among 23 triers. Each older age group reported 33.3% needing an appointment but not trying to book. Preference bases were 54 and 33 respectively.

  4. Provide accessible booking choices rather than treating a dedicated phone line as a complete disability adjustment.

    Medium confidenceBase 886

    Among 102 respondents reporting a disability, 50.0% preferred phone and 15.7% reception, compared with 39.5% and 3.2% among 784 without a disability. Satisfaction was 40.0% among 65 disabled booking triers versus 52.3% among 484 other triers. The 21 disability-related comments describe varied needs, including screen-reader compatibility, larger controls, help completing requests, consented proxy booking, hearing accommodations and BSL access.

  5. Booking-channel changes alone will not resolve the largest reported reasons for not attempting to book. Response speed and confidence in obtaining an appointment need attention alongside access.

    Medium confidenceBase 886

    Of all 886 respondents, 215 (24.3%) reported needing an appointment but not trying to book on at least one occasion. Among these 215, 43.7% expected a long wait for a reply and 28.8% did not expect to get an appointment; 16.7% found the form difficult and 9.3% lacked online confidence. Reasons overlap. Age-standardising Q4 gives 25.7%, equivalent to about 4,392 adults only if respondents' within-age experiences apply to the wider practice.

  6. The patient participation group should be told that female respondents were less satisfied descriptively, but the survey does not establish a sex difference at the 95% level or show that women find booking harder.

    Medium confidenceBase 540

    Satisfaction was 47.5% among 303 female booking triers and 55.3% among 237 male triers: a 7.8-point gap. An unadjusted two-sided test of proportions gives approximately p=0.07, so the difference is not statistically significant at 95%. The nominal interval for female minus male satisfaction is approximately −16.3 to +0.7 points. No sex breakdown of unsuccessful booking, non-attempts or booking effort is supplied.

  7. Interpreter users show an important accessibility signal, but there is too little evidence to quantify whether they find booking harder than other patients.

    Low confidenceBase 13

    Only 13 respondents needed an interpreter. Eight preferred reception (61.5%), three phone (23.1%), one online (7.7%) and one did not know. Two comments requested Polish-language forms or Urdu-speaking help. No interpreter breakdown of satisfaction or non-attempts is supplied, and needing an interpreter is not equivalent to identifying everyone with limited English.

  8. The comments support retaining online booking while restoring a human route and improving the form and response process; they do not constitute a representative vote for a policy.

    Medium confidenceBase 246

    Of 246 comments, 97 requested phone booking, including 13 from respondents aged 65+. Forty-five requested faster responses, 29 a simpler form, and 15 continuity with a familiar clinician. Forty-five explicitly praised online booking or said it needed no change. Another 32 offered no substantive suggestion or had insufficient experience; these should not be counted as endorsements. Themes overlap.

  9. The survey is particularly weak as a measure of the whole adult practice population, so population estimates should guide further investigation rather than determine staffing or capacity on their own.

    High confidenceBase 886

    There were 886 completes from 9,840 text invitations, a 9.0% completion response rate. Invitations reached only about 58% of the 17,100 adults. Adults aged 65+ were 9.8% of respondents but 24% of the registered adult population; ages 75+ were 3.7% versus 11%. Age weighting cannot recover experiences of people systematically missing within each age group.

Caveats

  • This is a voluntary, text-invited survey, not a probability sample. People without a recorded mobile number were excluded, and nonresponse may relate to booking experience or digital access.
  • Tables cover completes only. The experiences of the 160 starters who did not complete are unknown; screening outcomes and reasons for stopping are not supplied.
  • The requested satisfied_weighted answer is 0.44×64.7 + 0.32×44.0 + 0.13×17.8 + 0.11×13.0 = 46.292%. It is an age-standardised average of within-age satisfaction among triers, not necessarily satisfaction among all practice-wide triers: the age distribution of triers can differ from the adult population.
  • The requested phone_weighted answer is 0.44×29.9 + 0.32×49.0 + 0.13×63.0 + 0.11×72.7 = 45.023%. Weighting adjusts the observed age mix, not coverage or within-age response bias.
  • The put_off_adults estimate is 17,100×(0.44×22.1 + 0.32×24.9 + 0.13×33.3 + 0.11×33.3)/100, approximately 4,392. This is a conditional extrapolation, not an established count of unmet clinical need.
  • Q4 includes occasions when problems resolved or help was obtained elsewhere. It does not show that every non-attempt was caused by online-first, and respondents could both try to book and forgo booking on different occasions.
  • The nominal satisfaction margin of error uses 1.96×sqrt(0.508×0.492/549). It excludes selection bias, coverage error and other survey errors, so it is not a defensible population accuracy guarantee.
  • The female–male significance test is unadjusted and assumes independent observations. A nonsignificant result is not evidence of equality; age, disability and booking circumstances could explain or modify the observed gap.
  • The older-age, disability and interpreter groups overlap. Their bases cannot be added to estimate the number eligible for targeted support. Small older and interpreter bases make detailed estimates unstable.
  • Last year's recruitment method, eligibility, booking recall period and respondent mix are not demonstrably comparable. Neither the raw difference nor age standardisation establishes a change caused by online-first.
  • Satisfaction concerns booking with any practice team member, not just GPs. The survey does not measure successful booking rates, clinical urgency, available appointment capacity, staff workload or current phone performance.
  • Comments are self-selected and multi-coded. Phone coding includes explicit requests to call or speak by phone, including answers requesting several improvements; an indirect reference to another person's preference alone would not qualify. No-suggestion comments are not treated as positive endorsements.
  • Percentages and weighted calculations use rounded published figures. Interpreter need is an imperfect proxy for limited English, and preferences are not direct measures of access difficulty.

Next steps

  • decisionAgree a reversible universal-phone pilot, retain online access, and publish accessible information about all routes and available help.
  • experimentEvaluate the restored morning phone line against a baseline, using a staggered implementation or another credible comparison where feasible.
  • decisionAudit online request handling and introduce clear acknowledgement and realistic, urgency-sensitive response expectations.
  • researchSimplify and accessibility-test the form with disabled patients, older adults and people with limited English; test reception assistance and interpreter-supported booking too.
  • researchRecruit a stratified adult-list follow-up using post, phone, reception and translated or interpreter-supported methods, including patients without mobile numbers and non-users of the practice.
  • researchAnalyse respondent-level data and establish a consistent repeated survey for future comparisons.
  • monitorMaintain an equity and capacity dashboard across booking channels.

Open-text themes it coded

Phone booking or a phone booking line 97Faster replies, acknowledgement or confirmation of online requests 45Shorter, clearer and less repetitive online form 29Disability-related barriers and tailored accessibility or assistance 21In-person contact or practice assistance with booking 4Language support or translated forms 2Continuity with a familiar or regular clinician 15Explicitly positive about online booking or no change needed 45No substantive suggestion, uncertainty or insufficient experience 32

The survey it planned

2 screening questions and 12 questions, as the model wrote them.

  1. S1

    This survey is about booking appointments at Ashgrove Medical Group since the change in April 2026. It takes about 5–6 minutes. Taking part is optional and will not affect your care. Please do not include medical details or information that identifies you. Are you currently registered as a patient at Ashgrove Medical Group?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    How old are you?

    One answer
    • Under 18
    • 18 to 44
    • 45 to 64
    • 65 to 74
    • 75 or over

    Routing Shown only if S1 is: Yes

    Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over

  3. Q1

    Since the booking change in April 2026, have you tried to request an appointment for yourself at Ashgrove Medical Group? Include attempts made by someone helping you, whether or not you received an appointment.

    One answer
    • Yes
    • No
    • Not sure or cannot remember
  4. Q2

    Thinking about your attempts to request an appointment since April 2026, how satisfied or dissatisfied are you overall with the booking process? Please think about requesting and arranging an appointment, rather than the care you received.

    One answer
    • Very satisfied
    • Fairly satisfied
    • Neither satisfied nor dissatisfied
    • Fairly dissatisfied
    • Very dissatisfied
    • Not sure

    Routing Shown only if Q1 is: Yes

  5. Q3

    Since April 2026, was there any time when you felt you needed an appointment at Ashgrove Medical Group but did not try to request one because of something about the booking process?

    One answer
    • Yes
    • No — I tried to request an appointment whenever I felt I needed one
    • No — I did not try for other reasons, not because of the booking process
    • I have not felt I needed an appointment during this time
    • Not sure or cannot remember
  6. Q4

    What about the booking process stopped you from trying to request an appointment? Select all that apply.

    Any that apply
    • I did not have a suitable device or internet access
    • I did not feel confident using the online form
    • The online form was difficult to use because of a disability or health condition
    • I found it difficult to explain my request in writing
    • I needed language or interpreter support
    • I could not find the online form or did not know how to request an appointment
    • The online form was unavailable when I wanted to use it
    • I could not get through by phone
    • I thought I would not be allowed to request an appointment by phone or at reception
    • I did not have anyone available to help me
    • I was concerned about entering personal information online
    • I expected a long wait for a reply
    • I expected that I would not get a suitable appointment
    • Another reason related to booking
    • Not sure or cannot remember — select this option on its own

    Routing Shown only if Q3 is: Yes

  7. Q5

    If you needed a non-emergency appointment, how would you most prefer to request it? Choose one, even if that method is not currently available.

    One answer
    • Complete an online request form myself
    • Speak to someone at the practice by phone
    • Speak to someone at reception in person
    • Have someone else contact the practice on my behalf
    • Another method
    • I have no preference
    • Not sure
  8. Q6

    The practice is considering the following approaches to requesting appointments. Which would you most prefer? In every option, staff would assess requests and arrange appointments according to patients' needs.

    One answer
    • Keep the current approach: patients normally use an online request form, and reception phones are available for patients who cannot go online
    • Keep online requests and restore a morning phone booking line that any patient can use
    • Keep online requests and add a dedicated phone line and in-person help at reception for patients who need them: patients aged over 75, patients with a disability, and patients with limited English
    • None of these
    • I have no preference
    • Not sure
  9. Q7

    Which changes, if any, would make requesting an appointment easier for you? Select all that apply.

    Any that apply
    • A phone booking option available to every patient
    • A dedicated phone line for patients who need help using the booking system
    • Help completing a request at reception
    • A shorter or simpler online form
    • An online form that works better with accessibility tools, such as a screen reader
    • Translated booking information or forms
    • An interpreter or other language support when requesting an appointment
    • Clearer instructions about how to request an appointment
    • Being able to submit an online request at more times of day
    • A clear explanation of when and how the practice will reply
    • More help for a family member or carer to request an appointment for me
    • Another change
    • No changes are needed for me — select this option on its own
    • Not sure — select this option on its own
  10. Q8

    Do you consider yourself to have a disability or a long-term health condition that affects your day-to-day activities?

    One answer
    • Yes
    • No
    • Not sure
    • Prefer not to say
  11. Q9

    Do you need an interpreter when communicating with the practice? Include spoken-language and sign-language interpreters.

    One answer
    • Yes, always
    • Yes, sometimes
    • No
    • Not sure
    • Prefer not to say
  12. Q10

    How easy or difficult is it for you to understand and communicate in English when requesting an appointment? Consider both written information and conversations.

    One answer
    • Very easy
    • Fairly easy
    • Neither easy nor difficult
    • Fairly difficult
    • Very difficult
    • Not sure
    • Prefer not to say
  13. Q11

    Which statement best describes your ability to complete an online appointment request?

    One answer
    • I can usually do this without help
    • I can usually do this, but only with help from someone else
    • I cannot usually do this, even with help
    • I have not tried, so I do not know
    • Prefer not to say
  14. Q12

    Is there anything else you would like us to know about requesting appointments, or anything that would make it easier for you? This question is optional. Please do not include names, contact details or medical details.

    Open text
Sample plan

Invite every registered adult patient with a mobile number on record by text message, using an individual survey link where feasible to prevent duplicate submissions. Allow one response per patient and send a neutral reminder to nonrespondents. Aim for about 900 eligible completes. Monitor four age bands using practice-record population counts and seek approximately 225 completes per band through targeted reminders; these are monitoring targets, not reasons to exclude otherwise eligible respondents. Because every eligible patient with a mobile number is invited, do not close an age band merely because it reaches its target. Pilot with patients across age bands, including people with accessibility and language needs, to confirm comprehension, mobile accessibility and a completion time of no more than six minutes; retain optional profile and open-text responses. The requested precision is not compatible with assuming 900 completes are sufficient: 225 independent responses per band give roughly ±6.5 percentage points at 95% confidence for a proportion near 50%, before finite-population correction or weighting. For ±5 points, plan approximately 385 responses per band without finite-population correction, or calculate each band's target as ceiling(384.16 / (1 + 383.16 / N_h)), where N_h is its adult registered population. Obtain age-band population counts before setting final targets. If ±5 points is essential, increase the completion target to the sum of those targets and allow for weighting and nonresponse; otherwise explicitly relax the precision requirement. Booking satisfaction is measured only among patients who attempted booking, so its age-band denominators will be smaller and need separate precision checks. These calculations are planning benchmarks, not guarantees: an online opt-in survey cannot establish a design-based margin of error for all patients. The required mobile-link approach undercovers patients without recorded mobiles and may disproportionately miss patients unable to go online. Compare respondents with all adult patients and the invited population using available age and other appropriate administrative data. Weight for known age differences where justified, but do not claim weighting removes digital-exclusion or nonresponse bias. If representative evidence from excluded patients is necessary, obtain approval for a complementary telephone, paper or assisted-reception collection outside the specified online-only approach.