Benchmarks / SurveyBench / How Ashgrove patients find online-first booking / Muse Ember

Measured by Spring Prompt

Muse Ember: How Ashgrove patients find online-first booking

The decision: whether to keep online-first booking as it is, restore a morning phone booking line for every patient, or add a dedicated phone line and in-person help at reception only for patients who need them (over-75s, patients with a disability, and patients with limited English). The brief and the data →

Verdict: ✗ Not sound
Research score: 95 of 100
Analysis rating: 1,392
Head to head, this task: won 8 of 13

Why it is not sound

The analysis

Made a recommendation that does not follow from the data
Proposed a mistaken next step: Restore a morning phone booking line for every patient on the strength of this survey alone

Scorecard

Numbers 11 of 11 right

Question		Its answer	True
Among respondents who tried to book in the last three months, what percentage were satisfied (fairly or very) with the booking process?	✓	50.8%	50.8%
Weighting each age band's percentage satisfied (fairly or very, among those who tried to book) by that band's share of registered adults, what is the overall percentage satisfied?	✓	46.3%	46.3%
What percentage of all respondents said there was a time in the last three months when they needed an appointment but did not try to book one?	✓	24.3%	24.3%
Weighted to the age mix of registered adults, what percentage would prefer to book by phone?	✓	45%	45%
What is the 95% margin of error, in percentage points, on the unweighted percentage satisfied among respondents who tried to book?	✓	4.2 points	4.18 points
Among respondents who tried to book, is the difference in the percentage satisfied (fairly or very) between female and male respondents statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
How many of the open-text answers ask to be able to book by phone, or for a phone booking line?	✓	97	97
How many of the open-text answers from respondents aged 65 or over ask to be able to book by phone, or for a phone booking line?	✓	13	13
Weighted to the age mix of registered adults, about how many of the practice's 17,100 registered adults would say there was a time in the last three months when they needed an appointment but did not try to book one?	✓	4,392	4,393
Among respondents aged 18 to 44, what percentage would prefer to book using the online request form?	✓	58.4%	58.4%
What percentage of the patients sent the text message completed the survey?	✓	9%	9%

Traps in the data 7 of 7 handled

✓Patients aged 65 or over are 24% of registered adults but 9.8% of respondents (text link, online only). They are less satisfied and prefer the phone, so weighting to the age mix lowers satisfaction among those who tried to book from 50.8% to 46.3% and raises the phone preference from 40.7% to 45.0%.
✓Q2 satisfaction (50.8%) is asked only of the 549 who tried to book. 24.3% of all respondents needed an appointment but did not try to book (35.1% of those who did not try to book); those who did not try are left out of the satisfaction base, which flatters it.
✓Last year's 71% came from a paper survey handed out in the waiting room before the change, to patients who had got an appointment; this year's 50.8% is from an online survey by text of those who tried to book. Different method and different people.
✓The practice manager believes patients prefer the new system. Unweighted, the online form leads (46.2% against 40.7% for phone), but only 18-44s prefer it (58.4%); weighted to the age mix, preference is split (40.7% online, 45.0% phone, 6.7% in person).
✓Only 13 respondents need an interpreter; 61.5% of them would prefer to book in person.
✓Among those who tried to book, women look less satisfied than men (47.5% against 55.3%, bases 303 and 237), but the difference is not statistically significant at 95%.
✓Only 886 of 9,840 patients texted completed (about 9%), and the 7,260 registered adults without a mobile number on record were never invited; patients who cannot or do not go online are the least likely to have answered.

The analysis

Recommendation

Do not keep online-first as it is, and do not adopt the narrow option 3 as written. Restore a properly staffed phone booking route for all patients who prefer or need it, while keeping online as a choice, and add in-person reception help. Make access need-based and self-defined (including 65-74, people who cannot use the form, and people who simply prefer to speak) rather than requiring proof of being 75+, disabled or interpreter-needing, with proactive outreach and priority to those groups. In parallel, shorten the online form and introduce a published same-working-day reply standard, because delay and form length are top complaints for online users too. Run this as a staffed pilot with monitored waiting times before making it permanent, and commission the inclusive follow-up research below; the current survey alone is not a safe basis for a permanent universal model.

What would change it

Evidence that would change this would be: (1) a representative survey including non-mobile, paper/in-person and translated/interpreter-supported responses showing online is in fact the majority preference after proper weighting; (2) operational data showing a universal phone line cannot be answered within acceptable waits without harming online reply times or appointment supply, and that a self-defined targeted line captures almost all phone demand; (3) a pilot in which a shorter form plus guaranteed same-day reply raises age-weighted satisfaction and reduces put-off to acceptable levels without a universal phone line; or (4) costed staffing showing the pilot model is unsustainable and a narrower model delivers equitable access.

Findings

You cannot tell the partners online-first is preferred or working for most patients. Unweighted, 46.2% prefer online and 40.7% phone, but age-weighted to the practice list online is 40.7% and phone is the plurality at 45.0%. Among triers only 50.8% are satisfied, 46.3% age-weighted.

Medium confidenceBase 886

Q6 base 886: online 46.2%, phone 40.7% unweighted. Age-weighted phone 45.0%, online 40.7% using population shares 44/32/13/11%. Q2 base 549 triers: satisfied 50.8%, 95% margin +/-4.2 points. Weighting by age satisfaction 64.7/44.0/17.8/13.0% gives 46.3%.
The problem is sharply age-graded. Younger adults largely like online; older adults largely do not, and the survey under-samples older patients so the headline flatters the practice.

Medium confidenceBase 549

Satisfied among triers: 18-44 64.7% base 272, 45-64 44.0% base 209, 65-74 17.8% base 45, 75+ 13.0% base 23. Prefer online: 58.4%, 38.0%, 16.7%, 6.1%; prefer phone: 29.9%, 49.0%, 63.0%, 72.7%. Sample is 52.1% 18-44 vs 44% in population and only 3.7% 75+ and 6.1% 65-74 vs 11% and 13% in population.
About a quarter of respondents were put off trying at all, equivalent to roughly 4,392 registered adults age-weighted. The main stated reasons are expected delay and expected futility, with digital confidence a major reason at older ages - so channel alone will not fix access.

Medium confidenceBase 886

Q4 yes 24.3% base 886; age-weighted 25.7% x 17,100 = 4,392. Q5 base 215 who did not try: expected wait too long for reply 43.7%, did not think would get appointment 28.8%, got help elsewhere 22.8%, problem went away 17.7%, online form difficult 16.7%, not confident online 44.4% at 65-74 and 36.4% at 75+ vs 5.9% at 18-44.
There is no statistically significant sex gap in satisfaction among triers in this survey. That is not proof of equality, but this data do not support a claim that women find booking harder than men.

Medium confidenceBase 540

Satisfied: female 47.5% base 303, male 55.3% base 237; difference 7.8 points, SE about 4.3 points, z about 1.8, 95% interval includes zero. p is therefore not <0.05.
For patients needing an interpreter the survey is almost silent and cannot answer the chair's question. It was an English text survey; only 13 completers said they need an interpreter. Their responses hint online is unusable for them, but cannot be generalised.

Low confidenceBase 13

Interpreter-yes base 13 of 886, 1.5%. Among those 13, online preference 7.7% =1, phone 23.1% =3, in-person 61.5% =8. Open text includes A112 form in Polish and A756 Urdu speaker needed. Non-respondents needing interpreters are systematically likely to be missing. No population share for interpreter need is given to weight by.
Disabled respondents are much less likely to prefer online and a single phone fix will not meet all their needs. Preference inequality is large; access needs in open text are heterogeneous.

Medium confidenceBase 102

Q6 base 102 disability-yes: online 23.5%, phone 50.0%, in-person 15.7% vs disability-no base 784: online 49.1%, phone 39.5%, in-person 3.2%. Satisfied among triers 40.0% base 65 vs 52.3% base 484. Open text includes blind screen-reader A237, large print A565/A713, BSL/video A375, hearing aids needing in-person A265, learning disability help A262/A320, tremor/MS hand problems A193/A258/A443, save-and-return A358, proxy booking A437.
Last year's 71% cannot be compared with this year's 50.8% to claim a fall. The surveys differ in mode, frame, period and denominator, and both are biased in unknown directions.

High confidenceBase 412

Oct 2025: paper in waiting room, 412 returned, 71% satisfied. Sep 2026: text to 9,840 mobile holders, 886 completed, satisfaction asked only of 549 who tried to book. No common sampling frame, weighting, or non-response analysis links them.
Once age is taken into account, phone demand is broad, not confined to over-75s, disabled patients or interpreter users. A narrow eligibility gate at 75+ would miss the 65-74 group and much of the 45-64 group, who are the largest dissatisfied groups.

Medium confidenceBase 246

45-64 phone 49.0% vs online 38.0%, satisfied 44.0%; 65-74 phone 63.0%, satisfied 17.8%. Open-text phone requests 97 of 246 span all ages, including explicit requests for phone for everyone e.g. A039, A201, A574, A583. 65+ phone requests 13 of 18 open-text answers from 65+ respondents.
Online users' other top complaints are fixable without abandoning online: replies take days and the form is too long/repetitive/confusing. Removing online would harm younger patients who prefer it.

Medium confidenceBase 246

Open text base 246: faster/same-day reply 45 respondents, shorter/simpler form 29 respondents. 18-44 online preference 58.4% base 462 and satisfaction 64.7% base 272. Q5 expected wait too long was the top reason for not trying, 43.7% base 215.

Caveats

Response rate 9.0%: 886 completions from 9,840 texts, sent only to the 58% of adults with a mobile on record. Non-mobile, offline and many older and interpreter-needing patients are excluded by design.
Age weighting corrects only for age. It cannot correct for digital confidence, disability, language, health, frequency of attendance, or other differences between mobile responders and non-responders, which are likely correlated with age.
Small bases: 75+ n=33 total, 23 triers; 65-74 n=54 total, 45 triers; interpreter-yes n=13; disability triers n=65. Estimates for these groups have very large margins of error, e.g. about +/-15 points for 75+ phone preference.
Satisfaction is asked only of the 549 who tried to book. It excludes the 24.3% who needed care but did not try, non-triers and non-responders, so it overstates the experience of the whole patient population.
The 2025 71% figure is not comparable: paper waiting-room survey, n=412, different frame, mode, period and denominator. No trend claim is supported.
Preferences are hypothetical future channel preference with no trade-off presented for wait, cost, reply speed or appointment availability. Stated preference is not observed behaviour.
Q4/Q5 are 3-month recall. 'Did not try' includes problem went away 17.7% and help elsewhere 22.8%, so not all 24.3% is unmet need caused by online-first.
Open text is self-selected: 246 of 886 answered Q3. Themes overlap, one answer can express several themes, and counting 'speak to a person' as phone is a coding judgement; explicit keyword counting would differ slightly.
No operational evidence is provided: staffing, phone queue lengths, online reply times, form abandonment, appointment supply, cost per booking, or capacity to staff a morning line. Preference alone cannot determine feasibility.
Population shares are given only for age and sex. Disability and interpreter population shares are not given, so those groups cannot be weighted to the practice list, and equity for interpreter users remains essentially unmeasured.

Next steps

decisionPartners record that online-first as-is is not endorsed and approve a time-limited pilot of phone booking plus reception help, with self-defined need eligibility and proactive offers to 65+, disabled and interpreter-needing patients
experimentPilot phone and reception help for 3 months, logging call volumes, answer times, abandoned calls, in-person helps, bookings completed by age/disability/interpreter status, and online reply times
experimentShorten the online form to essential questions, test completion and abandonment, and publish a same-working-day reply commitment with acknowledgement on submission
researchCommission representative follow-up research: paper, in-person and translated/interpreter-supported recruitment, oversampling 65+, non-mobile, disabled and interpreter-needing patients, including people who did not try to book
researchInterview and observe non-triers and patients who abandoned the form, including disabled patients with diverse access needs (screen reader, BSL, hearing, learning disability, tremor/hand, vision)
monitorMonitor monthly, stratified by age, sex, disability and interpreter need, satisfaction among triers, put-off rate, channel use, reply and phone wait times; report to the patient participation group

Open-text themes it coded

Ask to book by phone / phone booking line 97Faster or same-day reply to online requests 45Online form too long, repetitive or confusing; shorten/simplify 29Continuity: same or usual GP who knows history 15Satisfied with online; no change needed 45No suggestion, rarely use practice, or do not know 32Specific disability or language accessibility adaptation 23

The survey it planned

2 screening questions and 9 questions, as the model wrote them.

S1
Are you registered as a patient at Ashgrove Medical Group?
One answer
- Yes
- No
Continues if Yes
S2
What is your age group?
One answer
- Under 18
- 18 to 44
- 45 to 64
- 65 to 74
- 75 or over
Continues if 18 to 44; 45 to 64; 65 to 74; 75 or over
Q1
Since April 2026, have you tried to ask Ashgrove Medical Group for an appointment?
One answer
- Yes
- No
- Not sure
Q2
How satisfied are you with asking for an appointment since April 2026?
Scale

Scale 1-5: Very dissatisfied to Very satisfied

Routing Shown only if Q1 is: Yes
Q3
In the future, how would you prefer to ask for an appointment?
One answer
- Phone the practice
- Fill in the online request form
- Go to reception in person
- Ask someone else to do it for me
- Other
Q4
Since April 2026, have you decided not to try to book an appointment when you felt you needed one?
One answer
- Yes
- No
Q5
What put you off booking?
Any that apply
- I found the online form difficult to use
- I did not know how to ask for an appointment
- I could not get through on the phone or did not want to wait
- I was worried the practice was too busy
- I needed help with reading, writing or English
- I needed help because of a disability or health condition
- Other
Routing Shown only if Q4 is: Yes
Q6
What would make booking an appointment easier for you?
Any that apply
- A phone line I can call in the morning
- A separate phone line for people who need extra help to book
- In-person help at reception to book
- Help from someone I trust to book for me
- A simpler online form
- Help to use the online form
- An interpreter when I book
- Nothing - booking is easy for me now
Q7
Do you have a disability or a long-term health condition that affects your daily activities?
One answer
- Yes
- No
- Prefer not to say
Q8
Do you need an interpreter or help with English when you contact the practice?
One answer
- Yes
- No
- Prefer not to say
Q9
Is there anything else that would make booking easier for you?
Open text

Sample plan

Text invitation with survey link to all registered adult patients (from 17,100) with a mobile number on record; keep open with reminders until quotas are met. Target 1,540 completes with an equal quota of 385 in each age band 18 to 44, 45 to 64, 65 to 74 and 75 or over (S2). Calculation: for a proportion at 95% confidence, worst case p=0.5, n=(1.96^2 x 0.25)/0.05^2=385 per band for +/-5 percentage points; 4x385=1,540. The originally estimated ~900 completes (225 per band) gives only about +/-6.5 points and cannot meet the partners' precision requirement, so the target must increase or the precision requirement relaxed. Monitor completes by age band in real time, pause the youngest bands when their quota is reached and focus reminders on older bands; monitor disability (Q7) and interpreter need (Q8) without hard quotas as prevalence is unknown, and report their base sizes and confidence intervals. Weight overall estimates back to the registered adult age distribution. Important limitation: a text/online-only survey excludes or under-represents patients without a mobile or unable to go online, who are disproportionately over-75, disabled or needing an interpreter, so online satisfaction and preference are likely to be overestimated and need for phone/in-person help underestimated; compare respondents with the register, report response by age band, and do not treat this sample alone as representative of patients who cannot go online. Use neutral, balanced wording throughout and do not design questions to confirm that online-first is working.