Benchmarks / SurveyBench / Why Riverside Food Network is losing volunteers / Mistral Large 4

Measured by Spring Prompt

Mistral Large 4: Why Riverside Food Network is losing volunteers

The decision: how to spend next year's £45,000 for volunteer support: reimburse volunteers' travel costs, fund a part-time volunteer coordinator with online shift booking, or do both at a smaller scale. The brief and the data →

Verdict
✗ Not sound
Research score
77 of 100
Analysis rating
815
Head to head, this task
won 7 of 15

Why it is not sound

The survey

  • A double-barrelled question: Q4

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Volunteers aged 65 or over are 43% of respondents but about 26% of volunteers in the database; younger volunteers cite travel cost more. Weighted to the lapsed age mix, travel cost among lapsed volunteers rises from 40% to 47%; a scheduling main reason goes from 43% to 41%, still well ahead of travel cost (21%).
  • Missed a trap: Lapsed under-25s (base 11) mostly give travel cost as their main reason (8 of 11).
  • Proposed a mistaken next step: Set up a travel scheme for under-25s on the strength of the lapsed under-25 result

Findings the judges found unsupported

  • Scheduling problems, not travel costs, are the primary driver of volunteer attrition. 42.9% of lapsed volunteers give a scheduling reason (short-notice changes or difficulty arranging shifts) as their main reason for stopping, versus only 11.6% for travel cost.The respondent percentage supports scheduling as the leading reported reason, but projecting it to approximately 429 of all 1,000 lapsed volunteers is not justified by the low-response, age-skewed sample. Calling it the primary driver also implies stronger causal evidence than the survey provides.
  • Travel cost is a severe barrier for under-25s specifically, driving them away at high rates, but this group is a minority of the lapsed population. 81.8% of lapsed under-25s cite travel cost as a difficulty and 72.7% give it as their main reason for stopping.The under-25 percentages describe only 11 lapsed respondents. They do not establish that travel drives under-25 volunteers away at high rates or justify an age-targeted scheme. The age-weighted calculation is supported, but describing it as what all 1,000 lapsed volunteers would report overlooks non-response bias.
  • Travel cost is a retention risk for current volunteers as well as a recruitment barrier: 37.2% of active volunteers cite it as a difficulty, a share statistically indistinguishable from the 40.2% of lapsed volunteers.The survey establishes a travel-cost difficulty among current volunteers, not a demonstrated retention risk or recruitment barrier. Never-volunteers were screened out, and future departures were not measured.

Scorecard

Numbers 12 of 12 right

QuestionIts answerTrue
What percentage of active volunteers who responded chose 'The cost of travelling to the site' as a difficulty in Q2?✓37.2%37.2%
What percentage of lapsed volunteers who responded chose 'The cost of travelling to the site' as a difficulty in Q2?✓40.2%40.2%
Is the difference between active and lapsed respondents in the share choosing 'The cost of travelling to the site' in Q2 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the age mix of all 1,000 lapsed volunteers, what percentage of lapsed volunteers would choose 'The cost of travelling to the site' as a difficulty in Q2?✓47.4%47.4%
What percentage of lapsed respondents gave a scheduling reason as their main reason for stopping in Q3 (either of its first two options: shifts changed at short notice, or hard to arrange shifts)?✓42.9%42.9%
What is the 95% margin of error, in percentage points, on that unweighted share of lapsed respondents giving a scheduling reason as their main reason?✓9.2 points9.17 points
On average, how many difficulties did each respondent choose in Q2 (counting every option except 'None of these', which counts as zero), across all respondents?✓1.551.55
What percentage of lapsed respondents said they would be likely (fairly or very) to volunteer again if they could arrange their shifts online (Q4)?✓40.2%40.2%
How many of the open-text answers mention shift scheduling (booking or swapping shifts, or last-minute changes or cancellations)?✓3634
What percentage of the active volunteers invited completed the survey?✓52.4%52.4%
What percentage of the lapsed volunteers invited completed the survey?✓11.2%11.2%
If the lapsed respondents were representative, about how many of the 1,000 lapsed volunteers would give a scheduling reason (either of the first two Q3 options) as their main reason, using the unweighted share?✓429429

Traps in the data 6 of 8 handled

  • ✓Travel cost is the most-cited difficulty overall, but it is cited about as often by active volunteers (37%) as by lapsed ones (40%), so it does not explain who left. Short-notice shift changes (14% active against 52% lapsed), no way to book or swap shifts and not being told about changes differ sharply.
  • ✓The operations lead believes volunteers leave because of travel costs. Among lapsed respondents, 43% give a scheduling reason as their main reason against 12% travel cost.
  • ✓Q2 is multi-select: respondents chose 1.6 difficulties on average, so columns add up to more than 100%.
  • ✓52% of active volunteers responded but only 11% of lapsed ones, so the lapsed base is small (112) and those who replied may differ from the 89% who did not (for example, still feeling some attachment).
  • ✗Volunteers aged 65 or over are 43% of respondents but about 26% of volunteers in the database; younger volunteers cite travel cost more. Weighted to the lapsed age mix, travel cost among lapsed volunteers rises from 40% to 47%; a scheduling main reason goes from 43% to 41%, still well ahead of travel cost (21%).
  • ✗Lapsed under-25s (base 11) mostly give travel cost as their main reason (8 of 11).
  • ✓The travel-cost gap between lapsed and active respondents (3.0 points) is not significant at 95%.
  • ✓40% of lapsed respondents say they would be likely to volunteer again with online booking: a stated intention about a hypothetical, from a small and self-selected base.

The analysis

Recommendation

Split the £45,000, weighting it toward a part-time volunteer coordinator with an online shift-booking system, plus a targeted travel reimbursement scheme. Allocate approximately £30,000 to the coordinator and online booking (which addresses the 42.9% of lapsed volunteers who left for scheduling reasons and the 41.1% who suffered poor communication), and £15,000 to travel reimbursement prioritised for under-25s and volunteers at sites with poor public transport. Do not spend the entire £45,000 on travel reimbursement alone: while travel cost is the top barrier for under-25s, scheduling is the bigger driver of attrition overall, and 60.4% of scheduling-motivated lapsed volunteers say online booking would bring them back versus only 14.3% of travel-motivated leavers. The volunteer manager is right to flag under-25s: 72.7% of lapsed under-25s cite travel as their main reason for stopping, justifying a targeted scheme, but this group is only 22% of the lapsed population. The chair should not expect a precise number of returns: the 40.2% likely-to-return figure is based on an 11.2% response rate from lapsed volunteers and is likely an upper bound. The operations lead's focus on travel misses that scheduling and communication failures affect a larger share of leavers.

What would change it

A phone survey of lapsed non-respondents showing that travel cost is the dominant reason for the majority of the 888 who did not reply would justify shifting more funds to travel reimbursement. A pilot showing that online booking does not reduce last-minute cancellations or improve active volunteer retention would weaken the case for the coordinator. If actual costings reveal that a full travel reimbursement scheme for all 410 active volunteers costs significantly less than £15,000, the surplus could be redirected. Evidence that the coordinator role cannot be recruited or retained would force a rethink.

Findings

  1. Scheduling problems, not travel costs, are the primary driver of volunteer attrition. 42.9% of lapsed volunteers give a scheduling reason (short-notice changes or difficulty arranging shifts) as their main reason for stopping, versus only 11.6% for travel cost.

    High confidenceBase 112

    Q3: 29.5% 'shifts changed at short notice' + 13.4% 'hard to arrange shifts' = 42.9% (base 112). Q2: 51.8% of lapsed cite short-notice changes as a difficulty; 42.0% cite no easy way to arrange shifts. Projection: ~429 of 1,000 lapsed volunteers would give a scheduling reason.

  2. Online shift booking has the strongest win-back potential: 60.4% of lapsed volunteers who left for scheduling reasons say they would likely return if they could book online, versus only 14.3% of those who left for travel reasons.

    Medium confidenceBase 112

    Q4 net likely (4-5): scheduling main reason 60.4% (base 48); travel main reason 14.3% (base 14); other 28.0% (base 50). Overall 40.2% of all lapsed respondents would likely return with online booking.

  3. Travel cost is a severe barrier for under-25s specifically, driving them away at high rates, but this group is a minority of the lapsed population. 81.8% of lapsed under-25s cite travel cost as a difficulty and 72.7% give it as their main reason for stopping.

    Medium confidenceBase 11

    Q2 lapsed by age: Under 25 81.8% (base 11) cite travel cost. Q3: 72.7% of lapsed under-25s (base 11) give travel cost as main reason. Weighted to population age mix (22% under-25), 47.4% of all 1,000 lapsed volunteers would cite travel cost as a difficulty.

  4. Travel cost is a retention risk for current volunteers as well as a recruitment barrier: 37.2% of active volunteers cite it as a difficulty, a share statistically indistinguishable from the 40.2% of lapsed volunteers.

    High confidenceBase 327

    Q2: Active 37.2% (base 215) vs Lapsed 40.2% (base 112). Difference = 3.0 percentage points; 95% CI approximately ±11 points, including zero (not significant).

  5. Poor communication about rota changes is endemic and compounds scheduling problems. 41.1% of lapsed volunteers say they were not told about changes in good time, and 51.8% cite shifts changed at short notice.

    High confidenceBase 112

    Q2 lapsed: 'Not being told about changes in good time' 41.1%; 'Shifts changed at short notice' 51.8% (base 112). Open-text: 12 respondents (e.g., V009, V055, V134, V308) independently mention finding out about changes late or second-hand.

  6. The chair's question—how many lost volunteers will return with online booking?—cannot be answered with precision. 40.2% of lapsed respondents say they would be likely to return, but the 11.2% response rate among lapsed volunteers means this is likely an overestimate of the true return rate among all 1,000.

    Medium confidenceBase 112

    Q4 total net likely 40.2% (base 112). Response rate: 112/1,000 lapsed = 11.2%. Non-respondents (888) may be less engaged and less likely to return.

Caveats

  • The lapsed volunteer response rate is only 11.2% (112/1,000), compared with 52.4% for active volunteers. Lapsed respondents are likely more engaged and more motivated to return than the 888 non-respondents, so the 40.2% 'likely to return' figure for online booking is probably optimistic.
  • Subgroup analysis for lapsed under-25s rests on a base of only 11 respondents; the 81.8% travel-cost figure and 72.7% main-reason figure are directionally strong but statistically unstable.
  • Q4 and Q5 measure stated intention, not behaviour. People routinely overstate their likelihood of returning to volunteering in surveys.
  • Q3 forces lapsed volunteers to choose a single 'main reason', which undercounts the role of travel for those who left for multiple reasons (e.g., travel plus scheduling).
  • The survey captures volunteers' perceptions, not actual travel costs incurred or the administrative cost of running each option.
  • The operations lead's claim that travel is 'the difficulty volunteers mention most' is true only for under-25s; across all respondents, travel cost (38.2%) ranks below short-notice shift changes (27.2%) only when considering single items, but scheduling-related difficulties collectively affect far more lapsed volunteers.

Next steps

  • decisionApprove a split budget: £30,000 for a part-time volunteer coordinator with online shift-booking and automated change notifications; £15,000 for a targeted travel reimbursement scheme (under-25s first, then high-cost sites).
  • experimentPilot the online booking system at 2-3 sites (including one with high under-25 usage) for 6 months before city-wide rollout.
  • researchRun a short phone survey of 100 lapsed volunteers who did not respond to the email survey to test for non-response bias on travel vs scheduling motivations.
  • monitorMonitor quarterly: travel reimbursement claims by age group; under-25 retention rate; number of lapsed volunteers returning; active volunteer retention at sites with/without online booking.
  • decisionAfter 6 months, review evidence from the pilot and phone survey to decide whether to expand travel reimbursement to all volunteers or keep it targeted, and whether to scale the coordinator role.

Open-text themes it coded

Travel costs (fares, fuel, mileage) 18Parking costs 4Parking availability and physical access 2Distance and travel time 9Shift scheduling (booking, swapping, cancellations, rota stability) 37Communication about changes 12Training 1Recognition and being valued 2No barriers / satisfied with current arrangements 38

The survey it planned

1 screening question and 10 questions, as the model wrote them.

  1. S1

    When did you last volunteer with Riverside Food Network?

    One answer
    • In the last 3 months
    • Between 3 and 12 months ago
    • More than 12 months ago
    • I have never volunteered with Riverside Food Network

    Continues if In the last 3 months; Between 3 and 12 months ago

  2. Q1

    At which Riverside Food Network distribution site did you complete the most shifts?

    One answer
    • Site 1
    • Site 2
    • Site 3
    • Site 4
    • Site 5
    • Site 6
    • Site 7
    • Site 8
    • Site 9
    • Prefer not to say
  3. Q2

    What is your age?

    One answer
    • Under 18
    • 18-24
    • 25-34
    • 35-44
    • 45-54
    • 55-64
    • 65 or over
    • Prefer not to say
  4. Q3

    How did you usually travel to your volunteering site when you were volunteering?

    One answer
    • Bus
    • Train
    • Car or van
    • Walk
    • Bicycle
    • Taxi or Uber
    • Other
  5. Q4

    How easy or difficult is it to book or swap shifts with your site lead?

    Scale

    Scale 1-5: Very difficult to Very easy

    Judges: double-barrelled, assumptive

    Booking and swapping shifts may have different levels of difficulty but receive one rating. The question assumes respondents have experience arranging shifts through a site lead. Its present-tense wording also leaves lapsed respondents uncertain whether to rate their past experience or the current process.

    It is shown to everyone, including the roughly 1,000 lapsed volunteers, but uses the present tense ('is it') as if they still book shifts. It also assumes booking is done through the site lead.

  6. Q5

    To what extent is the cost of travelling to your site (bus fares, petrol, parking) a financial burden for you?

    Scale

    Scale 1-5: Not a burden at all to A severe burden

    Judges: assumptive

    The question assumes a current volunteering-related travel burden, although lapsed respondents may no longer incur these costs. They could rate their former burden, their current expenditure, or the anticipated burden of returning.

    It is shown to lapsed volunteers but asks in the present tense about the cost of travelling to 'your site', which they no longer do. It also presumes everyone has travel costs, though walkers and cyclists may have none.

  7. Q6

    Which of the following difficulties, if any, have you experienced while volunteering with Riverside Food Network? Please select all that apply.

    Any that apply
    • Difficulty booking or swapping shifts
    • Poor communication with site lead
    • Shifts not available at times that suit me
    • Cost of travel (bus fares, petrol, parking)
    • Site is too far away
    • Physical demands are too high
    • Lack of training or support
    • Changes in my personal circumstances (work, family, health)
    • Found another volunteering opportunity
    • Felt undervalued or unappreciated
    • No difficulties experienced
    • Other (please specify)
  8. Q7

    You last volunteered between 3 and 12 months ago. Which of the following were reasons you stopped volunteering? Please select all that apply.

    Any that apply
    • Cost of travel became too expensive
    • Too difficult to book or swap shifts
    • Shifts were not at convenient times
    • Site was too far away
    • Physical demands were too high
    • Needed to stop for health reasons
    • Started a new job or increased work hours
    • Family or caring responsibilities changed
    • Moved house
    • Found another volunteering opportunity
    • Felt undervalued or unappreciated
    • Was not getting enough support or training
    • Other (please specify)

    Routing Shown only if S1 is: Between 3 and 12 months ago

    Judges: assumptive

    Having no shift in the last three months does not establish that someone stopped volunteering. Respondents taking a temporary break or volunteering infrequently are nevertheless required to give reasons for stopping.

    Having no shift for 3 to 12 months does not mean someone has stopped. Some may only be taking a break, yet the question assumes they have quit.

  9. Q8

    If Riverside Food Network reimbursed all your travel costs (bus fares, parking, petrol), how likely would you be to start volunteering again?

    Scale

    Scale 1-5: Very unlikely to Very likely

    Routing Shown only if S1 is: Between 3 and 12 months ago

    Judges: assumptive

    The routing establishes the date of the last shift, not that the respondent has stopped volunteering, which 'start volunteering again' assumes.

    It asks how likely they are to 'start volunteering again', which assumes they have stopped; some are only on a break. It also assumes they have travel costs to reimburse.

  10. Q9

    If you could book and swap shifts easily online (without having to phone or message your site lead), how likely would you be to start volunteering again?

    Scale

    Scale 1-5: Very unlikely to Very likely

    Routing Shown only if S1 is: Between 3 and 12 months ago

    Judges: assumptive

    The routing establishes the date of the last shift, not that the respondent has stopped volunteering, which 'start volunteering again' assumes. The proposed online booking-and-swapping service is a single intervention rather than two separate ratings.

    It asks how likely they are to 'start volunteering again', which assumes everyone in the 3 to 12 month group has stopped volunteering.

  11. Q10

    Is there anything else that would make volunteering with Riverside Food Network easier for you?

    Open text
Sample plan

Invite all 1,410 volunteers on the database who completed at least one shift in the last 12 months: 410 active volunteers (at least one shift in the last 3 months) and approximately 1,000 lapsed volunteers (last shift between 3 and 12 months ago). Invitations will be sent by both email and text message. Target: approximately 330 completed responses (approximately 23% response rate). Set minimum quotas of 25-30 completes per distribution site where possible to support site-level analysis. IMPORTANT PRECISION NOTE: With ~330 total completes across 9 sites (~37 per site), the margin of error per site is approximately ±16 percentage points at 95% confidence for proportions near 50%. Achieving the trustees' requirement of ±7 percentage points per site would require approximately 196 completes per site (1,764 total), which exceeds the available sample frame. Therefore, site-level results will be reported with 95% confidence intervals and interpreted as indicative; if ±7% precision per site is mandatory, the sample must be increased to ~1,800 completes or sites aggregated into groups for analysis.