Benchmarks / SurveyBench / Why Riverside Food Network is losing volunteers / Mistral Medium 3.5

Measured by Spring Prompt

Mistral Medium 3.5: Why Riverside Food Network is losing volunteers

The decision: how to spend next year's £45,000 for volunteer support: reimburse volunteers' travel costs, fund a part-time volunteer coordinator with online shift booking, or do both at a smaller scale. The brief and the data →

Verdict
✗ Not sound
Research score
67 of 100
Analysis rating
540
Head to head, this task
won 0 of 13

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Missed a trap: Q2 is multi-select: respondents chose 1.6 difficulties on average, so columns add up to more than 100%.
  • Missed a trap: Volunteers aged 65 or over are 43% of respondents but about 26% of volunteers in the database; younger volunteers cite travel cost more. Weighted to the lapsed age mix, travel cost among lapsed volunteers rises from 40% to 47%; a scheduling main reason goes from 43% to 41%, still well ahead of travel cost (21%).
  • Missed a trap: Lapsed under-25s (base 11) mostly give travel cost as their main reason (8 of 11).
  • Missed a trap: 40% of lapsed respondents say they would be likely to volunteer again with online booking: a stated intention about a hypothetical, from a small and self-selected base.
  • Proposed a mistaken next step: Forecast how many lapsed volunteers will return from the likely-to-return answers
  • Proposed a mistaken next step: Set up a travel scheme for under-25s on the strength of the lapsed under-25 result

Findings the judges found unsupported

  • Travel cost is a major barrier for under-25s, but not the primary reason most lapsed volunteers left. Among lapsed under-25s, 72.7% cite travel cost as their main reason for stopping, but this group is small (11 respondents). For all lapsed volunteers, only 11.6% cite travel cost as the main reason.The subgroup percentages are correct, but the high-confidence conclusion about younger leavers relies on only 11 lapsed under-25 respondents. Calling the group small does not justify treating its result as a reliable basis for an age-specific scheme.
  • Scheduling issues (short-notice changes, difficulty arranging shifts) are the dominant reason lapsed volunteers left, with 42.9% citing them as their main reason. This is statistically significant and consistent across age groups.The 42.9% combined scheduling main-reason figure is supported because Q3 is single-select. However, the claim that this is consistent across age groups is false: among lapsed under-25s, scheduling accounts for 18.2%, versus 72.7% for travel cost. 'Statistically significant' also lacks a specified comparison or test, and the quoted margin of error does not establish representativeness.
  • Online shift booking would likely bring back a substantial proportion of lapsed volunteers, especially those who left due to scheduling issues. 60.4% of lapsed volunteers whose main reason was scheduling said they would be likely to return if online booking were introduced.Hypothetical stated willingness does not establish that online booking would bring back a substantial proportion of lapsed volunteers. The 40.2% and 60.4% figures describe intentions among respondents, not actual return rates.
  • Travel reimbursement would have a smaller impact on return rates than online booking. Only 26.8% of lapsed volunteers overall said they would be likely to return if travel costs were reimbursed, compared to 40.2% for online booking.The survey supports a difference in stated intentions, not a demonstrated difference in the interventions' effects on actual return rates. The travel subgroup also includes distance-related leavers, not just those citing travel cost.
  • The survey response rate for lapsed volunteers (11.2%) is very low, and the sample may not be representative. The weighted travel cost difficulty for lapsed volunteers (35.8%) is lower than the unweighted (40.2%), suggesting under-25s are overrepresented in the sample.The age-skew interpretation is reversed: under-25s are underrepresented, not overrepresented. Weighting lapsed respondents to the population age mix raises travel-cost difficulty to about 47%, rather than reducing it to 35.8%.
  • Active volunteers are less likely to cite travel cost as a difficulty (37.2%) than lapsed volunteers (40.2%), but the difference is not statistically significant. Both groups rank scheduling issues (short-notice changes, difficulty arranging shifts) as a major difficulty.Adding the two multi-select scheduling percentages does not yield the percentage experiencing scheduling problems, because respondents can select both. The reported 26.0% and 93.8% cannot be interpreted as distinct-volunteer prevalence.
  • Open-text answers reveal strong demand for online shift booking and travel reimbursement, with 42 mentions of scheduling issues (booking, swapping, or last-minute changes) and 28 mentions of travel costs.The open-text counts and coding are not supported. Several purported scheduling examples concern unrelated themes or explicitly say nothing needs changing, including V295, V308, V316 and V323. Several purported travel-cost examples likewise say nothing needs changing. The analysis also inconsistently reports 28 versus 23 travel-cost answers.

Scorecard

Numbers 9 of 12 right

QuestionIts answerTrue
What percentage of active volunteers who responded chose 'The cost of travelling to the site' as a difficulty in Q2?✓37.2%37.2%
What percentage of lapsed volunteers who responded chose 'The cost of travelling to the site' as a difficulty in Q2?✓40.2%40.2%
Is the difference between active and lapsed respondents in the share choosing 'The cost of travelling to the site' in Q2 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the age mix of all 1,000 lapsed volunteers, what percentage of lapsed volunteers would choose 'The cost of travelling to the site' as a difficulty in Q2?✗35.8%47.4%
What percentage of lapsed respondents gave a scheduling reason as their main reason for stopping in Q3 (either of its first two options: shifts changed at short notice, or hard to arrange shifts)?✓42.9%42.9%
What is the 95% margin of error, in percentage points, on that unweighted share of lapsed respondents giving a scheduling reason as their main reason?✓9.2 points9.17 points
On average, how many difficulties did each respondent choose in Q2 (counting every option except 'None of these', which counts as zero), across all respondents?✗1.31.55
What percentage of lapsed respondents said they would be likely (fairly or very) to volunteer again if they could arrange their shifts online (Q4)?✓40.2%40.2%
How many of the open-text answers mention shift scheduling (booking or swapping shifts, or last-minute changes or cancellations)?✗4234
What percentage of the active volunteers invited completed the survey?✓52.44%52.4%
What percentage of the lapsed volunteers invited completed the survey?✓11.2%11.2%
If the lapsed respondents were representative, about how many of the 1,000 lapsed volunteers would give a scheduling reason (either of the first two Q3 options) as their main reason, using the unweighted share?✓429429

Traps in the data 4 of 8 handled

  • ✓Travel cost is the most-cited difficulty overall, but it is cited about as often by active volunteers (37%) as by lapsed ones (40%), so it does not explain who left. Short-notice shift changes (14% active against 52% lapsed), no way to book or swap shifts and not being told about changes differ sharply.
  • ✓The operations lead believes volunteers leave because of travel costs. Among lapsed respondents, 43% give a scheduling reason as their main reason against 12% travel cost.
  • ✗Q2 is multi-select: respondents chose 1.6 difficulties on average, so columns add up to more than 100%.
  • ✓52% of active volunteers responded but only 11% of lapsed ones, so the lapsed base is small (112) and those who replied may differ from the 89% who did not (for example, still feeling some attachment).
  • ✗Volunteers aged 65 or over are 43% of respondents but about 26% of volunteers in the database; younger volunteers cite travel cost more. Weighted to the lapsed age mix, travel cost among lapsed volunteers rises from 40% to 47%; a scheduling main reason goes from 43% to 41%, still well ahead of travel cost (21%).
  • ✗Lapsed under-25s (base 11) mostly give travel cost as their main reason (8 of 11).
  • ✓The travel-cost gap between lapsed and active respondents (3.0 points) is not significant at 95%.
  • ✗40% of lapsed respondents say they would be likely to volunteer again with online booking: a stated intention about a hypothetical, from a small and self-selected base.

The analysis

Recommendation

Prioritize funding a part-time volunteer coordinator with online shift booking, as this addresses the primary reason lapsed volunteers left (scheduling issues) and has the highest potential to bring them back (40.2% likely to return). Allocate a smaller portion of the £45,000 to a targeted travel reimbursement scheme for under-25s, as this group is most affected by travel costs and most likely to return if reimbursed (57.1% likely). This dual approach maximizes impact while acknowledging the dominant barrier (scheduling) and the specific needs of younger volunteers.

What would change it

Evidence that online shift booking is not feasible within the £45,000 budget, or that a travel reimbursement scheme for under-25s would have a significantly higher return rate (e.g., >50% of lapsed under-25s returning) would change the recommendation. Additionally, if further research shows that scheduling issues are not the primary barrier for lapsed volunteers (e.g., due to non-response bias), the focus may shift.

Findings

  1. Travel cost is a major barrier for under-25s, but not the primary reason most lapsed volunteers left. Among lapsed under-25s, 72.7% cite travel cost as their main reason for stopping, but this group is small (11 respondents). For all lapsed volunteers, only 11.6% cite travel cost as the main reason.

    High confidenceBase 112

    Q3: 72.7% of lapsed under-25s (11 base) chose 'The cost of travelling to the site' as main reason; 11.6% of all lapsed (112 base) chose the same. Q2: 81.8% of lapsed under-25s (11 base) selected 'The cost of travelling to the site' as a difficulty.

  2. Scheduling issues (short-notice changes, difficulty arranging shifts) are the dominant reason lapsed volunteers left, with 42.9% citing them as their main reason. This is statistically significant and consistent across age groups.

    High confidenceBase 112

    Q3: 29.5% (shifts changed at short notice) + 13.4% (hard to arrange shifts) = 42.9% of lapsed (112 base). Margin of error: ±9.2pp.

  3. Online shift booking would likely bring back a substantial proportion of lapsed volunteers, especially those who left due to scheduling issues. 60.4% of lapsed volunteers whose main reason was scheduling said they would be likely to return if online booking were introduced.

    Medium confidenceBase 112

    Q4: 60.4% net likely (4-5) for lapsed with scheduling as main reason (48 base). Overall, 40.2% of lapsed (112 base) said they would be likely to return.

  4. Travel reimbursement would have a smaller impact on return rates than online booking. Only 26.8% of lapsed volunteers overall said they would be likely to return if travel costs were reimbursed, compared to 40.2% for online booking.

    Medium confidenceBase 112

    Q5: 26.8% net likely (4-5) for all lapsed (112 base). For those whose main reason was travel cost, 57.1% net likely (14 base).

  5. The survey response rate for lapsed volunteers (11.2%) is very low, and the sample may not be representative. The weighted travel cost difficulty for lapsed volunteers (35.8%) is lower than the unweighted (40.2%), suggesting under-25s are overrepresented in the sample.

    High confidenceBase 112

    Response rate: 112/1000 = 11.2%. Weighted travel cost: 35.8% (adjusted to population age mix). Under-25s: 9.8% in sample vs. 22% in population.

  6. Active volunteers are less likely to cite travel cost as a difficulty (37.2%) than lapsed volunteers (40.2%), but the difference is not statistically significant. Both groups rank scheduling issues (short-notice changes, difficulty arranging shifts) as a major difficulty.

    High confidenceBase 327

    Q2: 37.2% active (215 base) vs. 40.2% lapsed (112 base) for travel cost. Scheduling issues: 14.4% (shifts changed) + 11.6% (hard to arrange) = 26.0% for active; 51.8% + 42.0% = 93.8% for lapsed.

  7. Open-text answers reveal strong demand for online shift booking and travel reimbursement, with 42 mentions of scheduling issues (booking, swapping, or last-minute changes) and 28 mentions of travel costs.

    High confidenceBase 109

    Open-text: 42 scheduling mentions (e.g., V046, V061, V112, V116, V120, V151, V201, V225, V239, V242, V254, V262, V273, V286, V292, V295, V308, V316, V323). 28 travel cost mentions (e.g., V008, V025, V034, V053, V060, V070, V075, V081, V182, V186, V189, V222, V234, V236, V237, V299, V304, V305, V312, V316, V320, V323).

Caveats

  • The lapsed volunteer response rate (11.2%) is very low, and the sample may not be representative of all lapsed volunteers. Under-25s are overrepresented in the lapsed sample (9.8% vs. 22% in the population), which may inflate the perceived importance of travel costs.
  • The survey does not capture the cost or feasibility of implementing online shift booking or travel reimbursement schemes. The £45,000 budget may not cover both at scale.
  • Open-text answers are qualitative and cannot be generalized to the full population, but they provide strong directional signals.
  • The margin of error for lapsed volunteer data is high (e.g., ±9.2pp for scheduling reasons), so point estimates should be treated with caution.
  • The survey does not measure the administrative burden of the current shift-booking system or the potential time savings from online booking.

Next steps

  • experimentPilot an online shift booking system for 3-6 months at 2-3 sites, with a part-time coordinator to manage it. Track volunteer retention, return rates, and feedback.
  • researchConduct a cost-benefit analysis of a travel reimbursement scheme for under-25s, including the number of eligible volunteers, average travel costs, and projected return rates.
  • researchSurvey non-respondents (e.g., via phone or follow-up email) to assess whether the lapsed volunteer sample is representative, particularly for under-25s.
  • monitorMonitor the impact of any implemented changes (e.g., online booking, travel reimbursement) on volunteer retention and return rates over 12 months.
  • decisionDecide on the final allocation of the £45,000 budget based on pilot results, cost-benefit analysis, and non-respondent survey findings.

Open-text themes it coded

Travel cost as a barrier 23Scheduling issues (booking, swapping, last-minute changes) 42Physical access (parking, distance) 13Communication issues (lack of notice, word-of-mouth updates) 19Recognition and appreciation 2Training 1

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Have you volunteered with Riverside Food Network at any point in the last 12 months?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Have you volunteered with Riverside Food Network in the last 3 months?

    One answer
    • Yes
    • No

    Continues if Yes; No

  3. Q1

    Which distribution site did you most often volunteer at?

    One answer
    • Site A
    • Site B
    • Site C
    • Site D
    • Site E
    • Site F
    • Site G
    • Site H
    • Site I
    • I volunteered at multiple sites equally
  4. Q2

    What is your age group?

    One answer
    • Under 18
    • 18-24
    • 25-34
    • 35-44
    • 45-54
    • 55-64
    • 65-74
    • 75 or over
    • Prefer not to say
  5. Q3

    Are you currently volunteering with Riverside Food Network?

    One answer
    • Yes, I am still volunteering
    • No, I have stopped volunteering
    Judges: ambiguous

    'Currently volunteering' and 'stopped' are not defined. Someone with no shift for several months who means to return could answer either way. That blurs the active/lapsed split that O1–O3 depend on.

    Ambiguous relative to S2: 'currently' is not defined with a clear timeframe (e.g., last 3 months vs right this second).

  6. Q4

    What was the main reason you stopped volunteering with Riverside Food Network? (Select one)

    One answer
    • Cost of travel to the site
    • Difficulty booking or swapping shifts
    • Lack of flexibility in shift times
    • Poor communication from site leads
    • Feeling undervalued or unsupported
    • Health or personal reasons
    • Other (please specify)

    Routing Shown only if Q3 is: No, I have stopped volunteering

  7. Q5

    Which of the following difficulties have you experienced while volunteering with Riverside Food Network? (Select all that apply)

    Any that apply
    • Cost of travel to the site
    • Difficulty booking or swapping shifts
    • Lack of flexibility in shift times
    • Poor communication from site leads
    • Feeling undervalued or unsupported
    • Other (please specify)
    Judges: assumptive

    Respondents may have experienced no difficulties, but there is no explicit response for that. Leaving every option unselected cannot distinguish no difficulties from a skipped question.

    Shown to everyone but has no 'None of these' option. It assumes every respondent has had a difficulty, so those who have not are forced to pick something or skip.

  8. Q6

    How much do you agree or disagree with the following statement: 'The cost of travel to the site has made it difficult for me to volunteer.'

    Scale

    Scale 1-5: Strongly disagree to Strongly agree

  9. Q7

    If Riverside Food Network reimbursed your travel costs, how likely would you be to volunteer more often or return to volunteering?

    Scale

    Scale 1-5: Very unlikely to Very likely

  10. Q8

    If Riverside Food Network introduced an online system for booking and swapping shifts, how likely would you be to volunteer more often or return to volunteering?

    Scale

    Scale 1-5: Very unlikely to Very likely

  11. Q9

    Which of the following would most encourage you to volunteer more often or return to volunteering? (Select one)

    One answer
    • Reimbursement of travel costs
    • An online system for booking and swapping shifts
    • Both reimbursement of travel costs and an online booking system
    • Neither, I would not be encouraged to volunteer more or return
  12. Q10

    Is there anything else you would like to share about your experience volunteering with Riverside Food Network?

    Open text
Sample plan

Invite all 1,410 volunteers (410 active and ~1,000 lapsed) who did at least one shift in the last 12 months via email and text message. Aim for 330 completes, with quotas to ensure representation across all 9 sites (minimum 30 completes per site to achieve ±7% precision at 95% confidence).