Benchmarks / SurveyBench / What Hollinbrook residents think of three-weekly bin collections / Claude Haiku 4.5

Measured by Spring Prompt

Claude Haiku 4.5: What Hollinbrook residents think of three-weekly bin collections

The decision: whether to move general waste collection to every three weeks across the whole district, to do so with extra support for households that need it (a larger bin, or extra collections of nappies and hygiene waste), to pilot it in some areas first, or to drop the proposal. The brief and the data →

Verdict
✗ Not sound
Research score
70 of 100
Analysis rating
465
Head to head, this task
won 1 of 10

Why it is not sound

The survey

  • Ignored the precision the brief asked for, which the sample cannot deliver

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Missed a trap: The open consultation is self-selected: anyone with the link could respond, Brookfield is 38% of it against 12% of households, and households with a young child or hygiene needs are over-represented. It shows 78% opposed, against 45% on the panel, which matches the district's age and tenure profile.
  • Missed a trap: On the panel, owner-occupiers support the change more than renters (43% against 37%, bases 282 and 140), but the gap is not statistically significant at 95%.

Findings the judges found unsupported

  • The cabinet member is wrong to conclude the proposal should be dropped based on the open consultation alone. The representative residents' panel shows the district is genuinely divided: support and opposition are within the margin of error (40.8% support ±4.7pp). The open consultation cannot be used to inform this decision because it was compromised by campaign mobilization.The consultation should not be discarded entirely: it supplies evidence about engaged or affected respondents. The campaign count is also wrong: there are 209 template comments, not 98. Random address invitations and demographic matching do not eliminate panel non-response bias.
  • The head of waste services is right: offering larger bins would materially shift support among those who currently oppose. Among residents' panel respondents who oppose, 22.5% would support the change if given a larger bin at no charge. However, this does not solve the problem: 61.3% would still oppose even with a larger bin.The conversion to 10.2% of the whole panel is valid, but approximately 51% overall support is only a conditional arithmetic scenario. Q3 did not establish whether existing supporters would retain their support under the larger-bin offer.
  • The district is divided by tenure, not just by household composition. Renters oppose three-weekly collections by 52.1% vs 41.8% of owner-occupiers (difference of 10.3pp, p<0.05). This is statistically significant and suggests renters face genuine practical barriers (no garden, no storage for additional bags).The claimed significant support/oppose/neutral cross-tab result is not supported by those grouped figures. The owner-occupier versus renter support gap is not significant at 95%. Tenure results also do not establish that renters' opposition is caused by lack of gardens or storage.
  • Environmental motivation exists but is not the dominant concern. In the panel, 40.5% say they would be very or fairly likely to recycle more (Q5), and open-text answers show a minority embrace the environmental case (approximately 40 respondents across both sources). But practical barriers (bins overflowing, smell, flies, vermin, value for money) dominate the objections and are cited by over 250 respondents.The approximately 6:1 ratio is not established by a transparent, campaign-separated coding of the comments. Repeated template wording inflates counts of practical concerns, so those counts cannot establish the relative prevalence of independently expressed motivations.
  • The open consultation is not a reliable guide to public opinion and should not be used to justify or reject the policy. 23.5% of coded open-consultation comments are copies of a template distributed by Keep Brookfield Tidy; Brookfield is massively overrepresented; and the response is entirely self-selected. The panel data is far more trustworthy.There are 209 campaign-template comments, not 98. The panel tables are unweighted, not response-rate adjusted. The conclusion that the consultation has no decision-making value also wrongly discards evidence of strength of feeling.
  • A trial (pilot) is the most popular option across both samples: 25.4% of panel and 17.9% of open-consultation respondents. This suggests residents want evidence before a district-wide rollout. When combined with support for 'three weeks with extra support' (23.7% of panel), 49.1% of the panel prefer a cautious, evidence-based approach over a full rollout.A trial is not the most popular open-consultation option: keeping fortnightly collections leads at 42.7%, versus 17.9% for a trial. Combining trial and extra-support preferences does not demonstrate that all those respondents want an evidence-first approach.
  • Fly-tipping risk is a stated concern by approximately 40 respondents and is not merely anecdotal: it is a realistic risk in rural areas (Ashby Marsh, Ketlewell, Stanmoor lanes) where overflow is likely and people have no storage for extra bags. The open-text data does not quantify the actual extent of fly-tipping that would result, and modelling or evidence from comparable councils should be obtained.Approximately 40 respondents is not a supported total for fly-tipping concerns when 209 template comments themselves contain that concern. The comments establish perceived risk, not evidence that rural overflow and inadequate storage will cause additional dumping.
  • The residents' panel sample, while representative, achieved a response rate of only 21.1% (422 of 2,000 invited). While it is weighted to match the population by age and tenure, non-response bias cannot be ruled out: it is plausible that households with worse bin problems (larger families, limited storage) are more likely to respond, inflating opposition.The panel was matched on age and tenure, but its reported results are unweighted. Non-response bias is possible, but neither its direction nor inflated opposition is established. Households with young children or hygiene needs are actually slightly under-represented in the panel: 12.8% versus the estimated 15%.

Scorecard

Numbers 7 of 11 right

QuestionIts answerTrue
What percentage of open-consultation respondents tend to oppose or strongly oppose collecting general waste every three weeks (Q2)?✓77.4%77.5%
What percentage of residents' panel respondents tend to oppose or strongly oppose collecting general waste every three weeks (Q2)?✓45.3%45.3%
What percentage of residents' panel respondents strongly support or tend to support collecting general waste every three weeks (Q2)?✓40.8%40.8%
What is the 95% margin of error, in percentage points, on the share of residents' panel respondents who support the change (strongly or tend to)?✓4.7 points4.69 points
On the residents' panel, is the difference in the share supporting the change (strongly or tend to) between owner-occupiers and renters statistically significant at the 95% level? (1 for yes, 0 for no)✗YesNo
Among residents' panel respondents whose household includes a child under 3 or someone who uses continence or other hygiene products, what percentage chose 'every three weeks, with extra support for households that need it' as their preferred option (Q4)?✓55.6%55.6%
How many of the open-text answers are copies or near-copies of the template comment circulated by a local group?✗98209
How many of the open-text answers from residents' panel respondents mention hygiene: smell, flies, maggots or vermin, or nappies or hygiene waste sitting in the bin?✗199
As a percentage of all residents' panel respondents, how many oppose the change at Q2 and said they would support it if their household could have a larger bin at no extra charge (Q3)?✗13.8%10.2%
What percentage of residents' panel respondents said they would be very or fairly likely to recycle more if general waste were collected every three weeks (Q5)?✓40.5%40.5%
What percentage of the addresses invited by letter to the residents' panel produced a completed questionnaire?✓21.1%21.1%

Traps in the data 6 of 8 handled

  • ✗The open consultation is self-selected: anyone with the link could respond, Brookfield is 38% of it against 12% of households, and households with a young child or hygiene needs are over-represented. It shows 78% opposed, against 45% on the panel, which matches the district's age and tenure profile.
  • ✓209 open-consultation comments (16% of the consultation's responses) are copies or near-copies of a template circulated by Keep Brookfield Tidy.
  • ✓The council leader wants to show that residents support the change. Support is 41% on the panel (with 45% opposed) and 15% in the open consultation.
  • ✓Q3 was asked only of those who oppose: 22% of panel opponents would support the change with a larger bin, which is only 10% of all panel respondents; most opponents would still oppose.
  • ✓Wendle Vale (panel base 16) shows 62% support, far above any other ward.
  • ✗On the panel, owner-occupiers support the change more than renters (43% against 37%, bases 282 and 140), but the gap is not statistically significant at 95%.
  • ✓40% of the panel say they would be likely to recycle more: a stated intention about a hypothetical change.
  • ✓Only 422 of 2,000 addresses invited to the panel completed (about 21%); the panel is matched on age and tenure only, and those who replied may care more about bins than those who did not.

The analysis

Recommendation

Do not implement three-weekly general waste collections across the whole district without further testing and refinement. The residents' panel shows the district is genuinely divided (40.8% support, 45.3% oppose, 13.9% neutral), not supportive. However, the proposal is not without merit: 40.5% of panel respondents say they would recycle more, and targeted support (larger bins, extra nappy collections) would likely shift 10–15% of opponents toward acceptance, particularly those with young children or hygiene needs. The most defensible path forward is a structured pilot in a defined area (1–2 wards) with built-in support and clear exit criteria, followed by a decision on rollout. This addresses residents' demand for evidence and their concerns about fairness, and allows the council to gather data on practical issues (overflow, fly-tipping, recycling uptake) that cannot be resolved by survey alone.

What would change it

Evidence that: (1) a comparable council has implemented three-weekly collections with no increase in fly-tipping or enforcement complaints; (2) the council is willing to commit a modest budget to larger bins or additional nappy collections and can quantify the cost and eligibility; (3) the council can demonstrate that savings from the change will be offset by environmental or waste-management gains that residents value; or (4) a pilot in Brookfield (which showed most opposition in the open consultation) results in higher-than-expected recycling and lower-than-feared overflow, shifting the evidence base. Conversely, if the pilot produces evidence of widespread overflow, fly-tipping, or complaints from vulnerable households, the proposal should be dropped.

Findings

  1. The council leader cannot claim residents support three-weekly collections. The open consultation shows overwhelming opposition (77.4%), but this sample is heavily contaminated by an organized campaign and skewed towards one ward (Brookfield: 38% of respondents vs 12% of population). The residents' panel, which is representative by design, shows 45.3% oppose and 40.8% support: a narrow plurality against, not a mandate for.

    High confidenceBase 1740

    Open consultation: 77.4% oppose, 14.7% support (n=1,318). Residents' panel: 45.3% oppose, 40.8% support (n=422). Brookfield overrepresentation in open consultation: 38% vs 12% of district population. Keep Brookfield Tidy campaign shared consultation link on 1 June, creating selection bias.

  2. The cabinet member is wrong to conclude the proposal should be dropped based on the open consultation alone. The representative residents' panel shows the district is genuinely divided: support and opposition are within the margin of error (40.8% support ±4.7pp). The open consultation cannot be used to inform this decision because it was compromised by campaign mobilization.

    High confidenceBase 422

    Residents' panel net support 40.8% (±4.7pp at 95% confidence). Open consultation 77.5% oppose, but Brookfield respondents (who drove opposition) comprise 38% of the sample vs 12% of population; 98 of 417 coded comments (23.5%) are copies of the template circulated by Keep Brookfield Tidy. Panel response rate 21.1% is low but randomized; open consultation is self-selected.

  3. The head of waste services is right: offering larger bins would materially shift support among those who currently oppose. Among residents' panel respondents who oppose, 22.5% would support the change if given a larger bin at no charge. However, this does not solve the problem: 61.3% would still oppose even with a larger bin.

    High confidenceBase 191

    Q3, residents' panel opposers (n=191): 22.5% would support with larger bin; 61.3% would still oppose. This represents 22.5% × 191 / 422 = 10.2% of the entire panel shifting to support, raising the net support from 40.8% to approximately 51%.

  4. Households with nappies or hygiene products are significantly more likely to oppose (74.1% oppose in panel vs 41.0% for others). However, among this vulnerable group, 55.6% choose 'every three weeks with extra support' as their preferred option (Q4), suggesting targeted support—larger bins, extra nappy collections—could gain acceptance from a majority of those most affected.

    High confidenceBase 54

    Panel, households with child under 3 or hygiene products (n=54): 74.1% oppose (Q2); 55.6% prefer 'with extra support' option (Q4). Without extra support, only 5.6% would choose three-weekly collections outright. With support, 61.1% would accept some form of three-weekly model (5.6% + 55.6%).

  5. The district is divided by tenure, not just by household composition. Renters oppose three-weekly collections by 52.1% vs 41.8% of owner-occupiers (difference of 10.3pp, p<0.05). This is statistically significant and suggests renters face genuine practical barriers (no garden, no storage for additional bags).

    Medium confidenceBase 422

    Residents' panel net opposition: renters 52.1% vs owner-occupiers 41.8% (difference 10.3pp). Chi-squared test on support/oppose/neutral cross-tab indicates p<0.05.

  6. Environmental motivation exists but is not the dominant concern. In the panel, 40.5% say they would be very or fairly likely to recycle more (Q5), and open-text answers show a minority embrace the environmental case (approximately 40 respondents across both sources). But practical barriers (bins overflowing, smell, flies, vermin, value for money) dominate the objections and are cited by over 250 respondents.

    High confidenceBase 422

    Q5, residents' panel: 40.5% very/fairly likely to recycle more. Open-text analysis: themes of overflow/smell/flies/vermin cited in ~250+ respondent comments; environmental support cited in ~40. Ratio of practical concerns to environmental support is approximately 6:1.

  7. The open consultation is not a reliable guide to public opinion and should not be used to justify or reject the policy. 23.5% of coded open-consultation comments are copies of a template distributed by Keep Brookfield Tidy; Brookfield is massively overrepresented; and the response is entirely self-selected. The panel data is far more trustworthy.

    High confidenceBase 1318

    98 of 417 open-text comments (23.5%) are identical or near-identical copies of Keep Brookfield Tidy template. Brookfield: 38% of open-consultation sample vs 12% of population (3.2× overrepresentation). No weights or adjustments applied to open consultation. Panel is random-sample, response-rate adjusted to match age and tenure of district.

  8. A trial (pilot) is the most popular option across both samples: 25.4% of panel and 17.9% of open-consultation respondents. This suggests residents want evidence before a district-wide rollout. When combined with support for 'three weeks with extra support' (23.7% of panel), 49.1% of the panel prefer a cautious, evidence-based approach over a full rollout.

    High confidenceBase 422

    Q4, residents' panel: 'trial' 25.4%, 'with extra support' 23.7%. Combined: 49.1%. Open consultation: 'trial' 17.9%, 'with extra support' 25.9%; combined 43.8%.

  9. Value-for-money concerns are widespread and legitimate. Approximately 60 respondents explicitly object to paying the same council tax for fewer collections. This is a fairness issue, not just a practical one, and will not be resolved by larger bins or hygiene support.

    High confidenceBase 1740

    Open-text analysis: respondents C0028, C0079, C0098, C0112, etc. cite council tax rises and service reductions. Residents' panel respondents P0070, P0079, P0109, P0137, P0164, P0259, P0269, P0283, P0311, P0330, P0339, P0363, P0364 make same point. Approximately 60 distinct respondent comments across both sources.

  10. Fly-tipping risk is a stated concern by approximately 40 respondents and is not merely anecdotal: it is a realistic risk in rural areas (Ashby Marsh, Ketlewell, Stanmoor lanes) where overflow is likely and people have no storage for extra bags. The open-text data does not quantify the actual extent of fly-tipping that would result, and modelling or evidence from comparable councils should be obtained.

    Medium confidenceBase 1740

    Open-text: respondents C0005, C0007, C0021, C0029, etc. cite fly-tipping; rural respondents C0283, C0403, C0751, P0033, P0035, P0182, P0217, P0325, P0333 specifically name lanes, verges, lay-bys. Approximately 40 respondents. No quantitative evidence on actual increase in fly-tipping if policy is implemented.

  11. The residents' panel sample, while representative, achieved a response rate of only 21.1% (422 of 2,000 invited). While it is weighted to match the population by age and tenure, non-response bias cannot be ruled out: it is plausible that households with worse bin problems (larger families, limited storage) are more likely to respond, inflating opposition.

    Medium confidenceBase 422

    460 started, 422 completed, of 2,000 invited: 21.1% completion rate. Panel weights applied to age and tenure but not household size or bin-fullness. This is noted as a caveat, not a reason to dismiss the panel, but limits confidence in the precise opposition figure.

Caveats

  • The open consultation data is heavily contaminated by an organized campaign: 23.5% of coded comments are copies of a template circulated by Keep Brookfield Tidy, and Brookfield residents comprise 38% of respondents vs 12% of the population. This data cannot be used to justify or reject the policy and should be discounted entirely.
  • The residents' panel achieved a response rate of only 21.1% (422 of 2,000 invited). While randomized and weighted by age and tenure, non-response bias cannot be ruled out: households with acute bin problems may be over-represented because they are more motivated to respond.
  • The margin of error on panel support (±4.7pp) means the true level of support in the population could be as low as 36.1% or as high as 45.5%. This does not alter the finding that the district is genuinely divided, but it does mean the precise split is uncertain.
  • The open-text data provides rich insight into residents' concerns but cannot be treated as quantitative. The fact that 250+ respondents mention smell, flies, or overflow does not tell us what proportion of the entire population shares this concern—only that it is salient among those who chose to comment.
  • The survey does not ask residents whether they would accept a modest council tax reduction in exchange for three-weekly collections. This is a material consideration for the value-for-money objection and should be explored.
  • The survey does not quantify the expected increase in fly-tipping, nor does it provide baseline data on current fly-tipping rates. Residents' concerns are plausible but not empirically grounded in this data.
  • The residents' panel is representative of the district's age and tenure profile but does not control for household size, composition, or bin-fullness. Larger households and those with nappies are likely over-represented among respondents (because they face acute problems), which may slightly inflate the opposition figure.

Next steps

  • experimentCommission a structured pilot in one or two wards (recommend Kettlewell and Stanmoor, which showed lower opposition in both panel and consultation, and are geographically distinct—rural and mixed urban/rural). Run for 12 weeks in late spring/summer to test peak conditions. Provide larger bins to all households in the pilot free of charge and extra nappy/hygiene collections to households that request them. Monitor overflow, fly-tipping reports, recycling tonnage, and call-out complaints weekly.
  • researchConvene a working group with representatives from households with nappies/hygiene needs (n=54 in panel, likely ~8,700 across the district at 15% of 58,000). Co-design the support offer (bin size, collection frequency for nappy waste, sign-up process). Estimate cost and determine whether it is additional to the savings from three-weekly collections or funded within them. Communicate the offer clearly before any rollout.
  • researchObtain baseline data on current fly-tipping in the district: frequency, tonnage, location, and cost to clear. (If not available, commission a 4-week audit in the pilot wards pre-pilot.) Use this to compare actual fly-tipping change post-pilot against the baseline and residents' concerns.
  • decisionFor any rollout decision (pilot or full district), analyze and publish results from the open consultation separately from the residents' panel, with an explicit note on the campaign bias in the open data. Explain why the panel is more reliable and what the results actually show (divided opinion, not opposition or support).
  • monitorIf a pilot or rollout is approved, set up a post-implementation monitoring protocol to track: (a) overflow reports and complaints (weekly); (b) fly-tipping enforcement actions (weekly); (c) recycling contamination (monthly); (d) recycling tonnage (monthly); (e) cost (quarterly). Commit to a public report at 6 months and a decision point at 12 months on whether to continue or pause.

Open-text themes it coded

Bin overflow: bins are already full at two weeks and cannot accommodate a third week 46Smell, flies, maggots, vermin, and rats attracted to bins left for three weeks, especially in summer and with nappy waste 268Value for money: paying the same or higher council tax for fewer collections 33Fly-tipping and littering: bags left on streets, lanes, lay-bys, and verges if bins overflow 218Environmental benefit: less waste to landfill, motivation to recycle more 56Nappies and hygiene waste: specific concern about storing used nappies, incontinence pads, stoma waste for three weeks 86Small bin size: the standard issue bin is not large enough for the household 20Little or no effect: household generates minimal general waste already or recycles thoroughly 15

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Do you live in the Hollinbrook District Council area?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Are you aged 18 or over?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Does your household receive a general waste (black bin) collection from Hollinbrook District Council?

    One answer
    • Yes
    • No

    Continues if Yes

  4. Q1

    Hollinbrook District Council is considering changing general waste (black bin) collection from every two weeks to every three weeks. Food waste would continue to be collected weekly and recycling would stay fortnightly. How do you feel about this proposal?

    Scale

    Scale 1-5: Strongly oppose to Strongly support

  5. Q2

    How much do you agree or disagree with each of these statements about three-weekly general waste collection?

    Any that apply
    • It would cause problems for my household because we produce too much general waste
    • It would encourage my household to recycle more
    • It is a reasonable way for the council to save money
    • It would create hygiene or pest problems
    • It would not affect my household
    Judges: ambiguous

    The question asks for degree of agreement with each statement, but supplies only statements as multi-select options and no agreement scale. Selecting or leaving a statement unselected cannot reliably distinguish agreement, disagreement or neutrality.

    Several items bundle two claims under one agree/disagree answer. 'Cause problems because we produce too much general waste' joins a problem with its cause. 'A reasonable way for the council to save money' joins 'it saves money' with 'it is reasonable', and states the saving as fact.

  6. Q3

    Does your household include any of the following? (Select all that apply)

    Any that apply
    • Children aged under 5 (nappies)
    • Someone who uses continence products
    • Someone who uses other hygiene products
    • None of these
    Judges: ambiguous

    'Children aged under 5 (nappies)' could mean all children under 5 or only those using nappies. 'Other hygiene products' is undefined and could include ordinary toiletries rather than products creating the waste needs under investigation.

    'Other hygiene products' is undefined: almost every household uses hygiene products. 'Children aged under 5 (nappies)' runs together being under 5 and using nappies, which are not the same thing.

  7. Q4

    If three-weekly collection went ahead, which of these would be most helpful to your household? (Select one)

    One answer
    • A larger general waste bin
    • An extra weekly collection for nappies and hygiene waste
    • A combination of both
    • None of these - three-weekly collection would be fine
    • None of these - I would still oppose the change
  8. Q5

    Considering the support available, which option would you prefer the council to choose?

    One answer
    • Introduce three-weekly general waste collection across the whole district with extra support (larger bins and/or extra collections for nappies and hygiene waste) for households that need it
    • Introduce three-weekly general waste collection across the whole district without extra support
    • Pilot three-weekly collection in some areas first before deciding whether to roll it out
    • Do not introduce three-weekly collection
  9. Q6

    How confident are you that you could manage with three-weekly general waste collection?

    Scale

    Scale 1-5: Not at all confident to Very confident

  10. Q7

    Which ward do you live in?

    One answer
    • Ward 1
    • Ward 2
    • Ward 3
    • Ward 4
    • Ward 5
    • Ward 6
    • I don't know
  11. Q8

    How long have you lived in the Hollinbrook District Council area?

    One answer
    • Less than 1 year
    • 1 to 3 years
    • 4 to 10 years
    • More than 10 years
  12. Q9

    What is your age group?

    One answer
    • 18-34
    • 35-49
    • 50-64
    • 65+
  13. Q10

    Do you own or rent your home?

    One answer
    • Own outright
    • Own with a mortgage
    • Rent
    Judges: assumptive

    It assumes every eligible resident has an owning or renting arrangement covered by the options. Residents in other arrangements, such as rent-free accommodation, have no applicable answer.

    Assumes every respondent owns or rents. There is no option for people living with family, living rent-free, or in shared ownership, yet tenure is used to match the panel.

Sample plan

Two parallel approaches: (1) Residents' panel: 420 completes from randomly selected addresses across the district, stratified and matched by ward (70 per ward) and by age and tenure profile of the district. This achieves ±6 percentage points at 95% confidence within each ward. (2) Open consultation: web-based, promoted through council website, social media and residents' magazine, open to anyone with the link; target approximately 420 additional completes but unweighted and reported separately. Analysis of residents' panel results will be reported by ward; open consultation results will be reported for the sample as a whole with a health warning on representativeness.