Benchmarks / SurveyBench / What Hollinbrook residents think of three-weekly bin collections / Claude Haiku 5.5

Measured by Spring Prompt

Claude Haiku 5.5: What Hollinbrook residents think of three-weekly bin collections

The decision: whether to move general waste collection to every three weeks across the whole district, to do so with extra support for households that need it (a larger bin, or extra collections of nappies and hygiene waste), to pilot it in some areas first, or to drop the proposal. The brief and the data →

Verdict
✗ Not sound
Research score
91 of 100
Analysis rating
1,412
Head to head, this task
won 6 of 8

Why it is not sound

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • The 77.5% opposition figure in the open consultation is not a measure of district opinion. The open consultation was self-selected and was promoted by a campaign group that circulated suggested wording. Brookfield makes up 38% of its respondents against about 12% of households. Reweighting the ward results to household shares still gives about 75% opposition, so the ward mix alone does not explain the gap; the self-selection and campaign effects do.Ward reweighting shows that ward composition alone does not explain the difference, but it cannot establish that campaign and self-selection effects explain the remaining gap. Also, 417 is described elsewhere as the combined open-text base, not the open-consultation comment base, so 209 of 417 is not a supported estimate of the campaign share among consultation comments.
  • The campaign template is widespread in the open-text answers. About half of the open-text comments reproduce the same wording, so the open consultation's opposition should be read as campaign-amplified rather than as independent residents' views.The analysis incorrectly describes 209 of the combined 417 open-text answers as about half of the open-consultation comments. The established campaign count is 209, equivalent to about 16% of all open-consultation responses; the denominators must be distinguished.
  • The most common concerns are overflow, smell and hygiene, and fly-tipping. These are testable service risks, not general opposition to the policy, so they can be addressed through the pilot design.The reported concern counts are incompatible with the 209 template comments: every template includes overflow, hygiene and fly-tipping, so totals of about 80, 60 and 45 respectively cannot include the templates. They also contradict the substantially higher counts in the final theme list. Describing these concerns as not general opposition is too categorical; respondents also express principled opposition to a reduced service.
  • The panel's response rate of 21.1% is adequate for a probability-based survey, but the open consultation's self-selection means its results cannot be combined with or used in place of the panel.A 21.1% completion rate does not establish that the panel response rate is adequate or that the achieved sample is an unbiased probability sample. Random address invitations and matching on age and tenure do not eliminate possible non-response bias.
  • Among panel respondents, no single option commands a majority. A trial is the most popular option, followed by keeping the current service, extra support and whole-district change. The trial and support options together give a clear route for a compromise that could be tested.It treats 71.1% choosing a three-weekly variant as preferring three-weekly collection, but the trial option is 'trial, then decide' rather than support. It also calls the trial 'most popular' when all four options sit within about 3 points of each other, inside the margin of error. A leader could cite the 71% as support.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of open-consultation respondents tend to oppose or strongly oppose collecting general waste every three weeks (Q2)?✓77.5%77.5%
What percentage of residents' panel respondents tend to oppose or strongly oppose collecting general waste every three weeks (Q2)?✓45.3%45.3%
What percentage of residents' panel respondents strongly support or tend to support collecting general waste every three weeks (Q2)?✓40.8%40.8%
What is the 95% margin of error, in percentage points, on the share of residents' panel respondents who support the change (strongly or tend to)?✓4.7 points4.69 points
On the residents' panel, is the difference in the share supporting the change (strongly or tend to) between owner-occupiers and renters statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Among residents' panel respondents whose household includes a child under 3 or someone who uses continence or other hygiene products, what percentage chose 'every three weeks, with extra support for households that need it' as their preferred option (Q4)?✓55.6%55.6%
How many of the open-text answers are copies or near-copies of the template comment circulated by a local group?✓209209
How many of the open-text answers from residents' panel respondents mention hygiene: smell, flies, maggots or vermin, or nappies or hygiene waste sitting in the bin?✓99
As a percentage of all residents' panel respondents, how many oppose the change at Q2 and said they would support it if their household could have a larger bin at no extra charge (Q3)?✓10.2%10.2%
What percentage of residents' panel respondents said they would be very or fairly likely to recycle more if general waste were collected every three weeks (Q5)?✓40.5%40.5%
What percentage of the addresses invited by letter to the residents' panel produced a completed questionnaire?✓21.1%21.1%

Traps in the data 8 of 8 handled

  • ✓The open consultation is self-selected: anyone with the link could respond, Brookfield is 38% of it against 12% of households, and households with a young child or hygiene needs are over-represented. It shows 78% opposed, against 45% on the panel, which matches the district's age and tenure profile.
  • ✓209 open-consultation comments (16% of the consultation's responses) are copies or near-copies of a template circulated by Keep Brookfield Tidy.
  • ✓The council leader wants to show that residents support the change. Support is 41% on the panel (with 45% opposed) and 15% in the open consultation.
  • ✓Q3 was asked only of those who oppose: 22% of panel opponents would support the change with a larger bin, which is only 10% of all panel respondents; most opponents would still oppose.
  • ✓Wendle Vale (panel base 16) shows 62% support, far above any other ward.
  • ✓On the panel, owner-occupiers support the change more than renters (43% against 37%, bases 282 and 140), but the gap is not statistically significant at 95%.
  • ✓40% of the panel say they would be likely to recycle more: a stated intention about a hypothetical change.
  • ✓Only 422 of 2,000 addresses invited to the panel completed (about 21%); the panel is matched on age and tenure only, and those who replied may care more about bins than those who did not.

The analysis

Recommendation

Do not roll out three-weekly collections across the whole district now, and do not drop the proposal yet. Take the pilot option, designed around the concerns the panel actually raised. Pick two or three areas where bin capacity and household size are low risk, and include in the pilot an assisted route (a larger bin or fortnightly nappy and hygiene collection) for households with children under 3 or hygiene needs, plus a capacity check for large households. Report to full council accurately: the panel is split (40.8% support, 45.3% oppose), the open consultation is campaign-influenced, and there is no evidence of a district mandate either way. Do not tell council that residents back the change. The cabinet member's four-in-five figure comes from the unrepresentative open consultation and should not be used as the district's view. A larger bin alone will not win over most opponents (about one in five switch), so it should not be treated as the fix.

What would change it

Dropping the proposal: a pilot that shows a clear rise in overflow, missed-collection complaints, fly-tipping or hygiene incidents in the pilot areas that the assisted route cannot contain, or a costed business case showing no net saving. Rolling out district-wide: a weighted, larger probability survey showing majority support (for example at least 55%) with support holding in the lower-income, larger-household and hygiene-need groups, plus pilot data showing that residual waste per household stays within bin capacity for at least 90% of households and that fly-tipping does not rise. Each would need to be confirmed with council data on residual tonnage and bin capacity.

Findings

  1. Residents do not back the three-weekly collection. The residents' panel, the only source that is random and matched to the district's age and tenure profile, splits almost evenly: 40.8% support and 45.3% oppose. The leader's claim that residents support the change is not supported by the evidence.

    High confidenceBase 422

    Panel Q2: net support 40.8% (17.3% strongly, 23.5% tend to); net oppose 45.3% (17.1% tend, 28.2% strongly); neither 10.4%; don't know 3.6%. Base 422. Margin of error on support is about plus or minus 4.7 points, so support lies roughly between 36 and 46 per cent. No majority for or against.

  2. The 77.5% opposition figure in the open consultation is not a measure of district opinion. The open consultation was self-selected and was promoted by a campaign group that circulated suggested wording. Brookfield makes up 38% of its respondents against about 12% of households. Reweighting the ward results to household shares still gives about 75% opposition, so the ward mix alone does not explain the gap; the self-selection and campaign effects do.

    High confidenceBase 1318

    Open Q2: 77.5% oppose, 14.7% support, base 1318. Open Q2 by ward: Hollinbrook Town 75.0, Ashby Marsh 73.1, Kettlewell 70.7, Stanmoor 73.1, Brookfield 83.4, Wendle Vale 81.6 oppose. Ward-reweighted to household shares: about 74.8% oppose. Panel strongly-oppose share is 28.2% against 58.0% in the open consultation. 209 of 417 open-text answers (about 50%) match the campaign template.

  3. The campaign template is widespread in the open-text answers. About half of the open-text comments reproduce the same wording, so the open consultation's opposition should be read as campaign-amplified rather than as independent residents' views.

    High confidenceBase 417

    209 of 417 open-text answers are copies or near-copies of the template, including the lower-case and Keep Brookfield Tidy variants. All 209 are from the open consultation; none come from the panel. The group shared the link on 1 June with leaflets and suggested comment wording.

  4. A larger bin at no extra charge would not bring most opponents round. Only about one in five panel opponents would switch to support, and about three in five would still oppose.

    Medium confidenceBase 191

    Panel Q3, asked only of Q2 opposers (base 191): would support 22.5% (about 43 people, or 10.2% of all panel respondents), neither 9.4%, still oppose 61.3%, don't know 6.8%. Open consultation Q3 (base 1021): would support 17.4%, still oppose 66.5%. The cost of larger bins has not been modelled, and the bin-capacity complaints in the open text suggest that some households would need more than a bigger bin.

  5. Opposition is concentrated among households with children under 3 or hygiene needs. This group is the most opposed and most likely to pick an assisted option, so any move needs a built-in support route for them.

    Low confidenceBase 54

    Panel subgroup (base 54): net oppose 74.1% against 41.0% for other households; net support 14.8%. 55.6% of this group prefer the option with extra support (larger bin or nappy collections), against 19.0% of other households. The council estimates about 15% of households have such needs; the panel has 12.8%. Hygiene and smell are the most frequent concerns in the open text.

  6. Among panel respondents, no single option commands a majority. A trial is the most popular option, followed by keeping the current service, extra support and whole-district change. The trial and support options together give a clear route for a compromise that could be tested.

    Medium confidenceBase 422

    Panel Q4 (base 422): keep every two weeks 24.6%; whole district 22.0%; with extra support 23.7%; trial then decide 25.4%; don't know 4.3%. Combined, 71.1% prefer some three-weekly variant, yet only 45.3% oppose the change at Q2. This suggests that how the change is framed and whether support is offered shifts preferences. Open consultation Q4 gives keep 42.7% and the three-weekly variants 52.4%.

  7. There is no statistically significant difference in support between owner-occupiers and renters on the panel. Renters lean slightly more to opposition, but the panel is too small to rule out a real difference of around ten points.

    Medium confidenceBase 422

    Panel support: owner-occupiers 42.6% (n=282), renters 37.1% (n=140). Difference 5.5 points; standard error about 5.0 points; 95% interval for the difference about plus or minus 9.9 points, which spans zero. Renters' net oppose is 52.1% against 41.8% for owners.

  8. The environmental case is supported by stated intentions but not yet by evidence of behaviour. A substantial minority says they would recycle more, but stated intention is a weak predictor of actual behaviour.

    Low confidenceBase 422

    Panel Q5: net likely to recycle more 40.5% (15.6% very, 24.9% fairly); neither 16.1%; unlikely 29.1%; already recycle everything 11.8%. Open consultation: net likely 24.2%. The open text includes both pro-environment and anti-change comments.

  9. The most common concerns are overflow, smell and hygiene, and fly-tipping. These are testable service risks, not general opposition to the policy, so they can be addressed through the pilot design.

    Medium confidenceBase 417

    Themes from the 417 open-text answers: bins already full or overflowing (about 80 comments, including the template); smell, flies, rats, maggots and nappy or continence waste (about 60); fly-tipping and dumping (about 45); value for money, paying the same council tax for less service (39). Panel open text mentions hygiene in only 9 of 191 comments.

  10. The panel's response rate of 21.1% is adequate for a probability-based survey, but the open consultation's self-selection means its results cannot be combined with or used in place of the panel.

    High confidenceBase 422

    Panel: 422 completes from 2,000 letters, 21.1%; 460 starts (23.0%). Open consultation: 1,318 completes, self-selected, not weighted. The two sources are reported separately as specified.

Caveats

  • The open consultation was self-selected, unweighted, and shaped by a campaign group's circulated wording. Its 77.5% opposition is not a measure of district opinion and must not be presented as one.
  • The residents' panel is random and matched to the district's age and tenure profile, but the tables are unweighted and the margin of error (4.7 points) assumes simple random sampling with no design effect. The true margin is probably somewhat wider.
  • Q3 was asked only of respondents who opposed at Q2, so the 10.2% figure is a share of all panel respondents, not of opposers. The base for Q3 is 191 (panel) and 1,021 (open consultation).
  • The subgroup with a child under 3 or hygiene needs has 54 panel respondents, so its figures (55.6% preferring extra support; 74.1% opposed) have wide intervals and must be treated as indicative.
  • The open-text answers cover 417 respondents, which is less than the 1,740 total completes. The themes come from a subset that may not be representative, and the coding is a single reader's classification.
  • The panel's Q4 options and Q2 wording may have shaped the answers: asking about a trial or extra support after the core question may lead people to accept a compromise they would not otherwise choose.
  • Stated recycling intention (Q5) is not behaviour. No council data on residual waste per household or bin capacity has been provided, so the claim that bins will overflow cannot be checked against actual volumes.
  • The open consultation over-represents Brookfield (38% of respondents against about 12% of households). Ward reweighting does not remove the opposition gap, so the campaign and self-selection effects are the more likely explanation.
  • The council's estimate of 15% of households with a child under 3 or hygiene needs is close to the panel's 12.8%, but the two figures use different definitions and the panel subgroup is small.
  • Cost, financial implications and council-tax effects were not part of the question set. Residents repeatedly link the change to council tax, so the lack of a stated financial offer is a material gap in the evidence.

Next steps

  • decisionReport to full council that the panel is split (40.8% support, 45.3% oppose) and that the open consultation is campaign-influenced; do not present residents as backing the change.
  • experimentDesign a two- or three-area pilot of three-weekly collection, with a pre-agreed evaluation plan, an assisted route for households with children under 3 or hygiene needs, and a capacity check for large households.
  • researchCollect council data on residual waste tonnage and bin fill levels per household before finalising the pilot areas.
  • researchModel the cost of the assisted route (larger bins and nappy or hygiene collections) against the saving from three-weekly collection, and publish the net position with the council-tax message stated clearly.
  • researchCommission a larger, weighted residents' survey with a longer questionnaire, including the assisted route, a clearer financial statement, and a test of how the question order affects Q2 and Q4.
  • monitorMonitor the pilot areas monthly: missed collections, overflow and contamination reports, fly-tipping reports, hygiene complaints, and recycling tonnage.
  • decisionKeep the open consultation's comment themes on file and respond publicly to the overflow, hygiene and value-for-money concerns, distinguishing campaign template wording from individual comments.

Open-text themes it coded

Campaign template objection (Keep Brookfield Tidy wording and near-copies) 209Bins already full or would overflow 276Smell, flies, rats, maggots or hygiene risk from nappies and continence waste 244Fly-tipping and dumping of bags 222Paying the same council tax for a reduced service (value for money) 39Supportive: environmental benefit and encourages recycling or less waste 57Little or no effect on our household 16

The survey it planned

4 screening questions and 10 questions, as the model wrote them.

  1. S1

    What is your age?

    One answer
    • Under 18
    • 18 to 24
    • 25 to 34
    • 35 to 44
    • 45 to 54
    • 55 to 64
    • 65 to 74
    • 75 or over

    Continues if 18 to 24; 25 to 34; 35 to 44; 45 to 54; 55 to 64; 65 to 74; 75 or over

  2. S2

    Is your home in the Hollinbrook District Council area?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Does Hollinbrook District Council collect general waste (the black bin) from your home?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  4. S4

    What is your full postcode? We use this only to group answers by ward and to confirm you live in the district.

    Open text

    Continues if A postcode inside the Hollinbrook District Council area

  5. Q1

    Which of these apply to your household? Select all that apply.

    Any that apply
    • A child under 5 lives in my household
    • Someone in my household uses continence products
    • Someone in my household uses other hygiene products that go in the black bin
    • None of these apply
  6. Q2

    Your household's general waste (the black bin) is currently collected every two weeks. The council must save £3.1 million next year. One option is to collect general waste every three weeks instead. This would save about £600,000 a year. Food waste would stay weekly and recycling would stay every two weeks. Some councils that have made this change report higher recycling. Some households, including those with a child under 5 or someone who uses continence or hygiene products, may find it harder to store waste between collections. Overall, how do you feel about collecting general waste every three weeks across the district?

    Scale

    Scale 1-5: Strongly oppose to Strongly support

  7. Q3

    What is the main reason for your answer to the previous question?

    Open text
  8. Q4

    If general waste were collected every three weeks, which of these would be a problem for your household? Select all that apply.

    Any that apply
    • Storing general waste between collections
    • Smell from the black bin
    • Not enough space to store the bins
    • Nappy or other hygiene waste building up
    • Animals getting into the bin
    • Other
    • None of these would be a problem
  9. Q5

    Which of these would help your household manage nappy or hygiene waste if general waste were collected every three weeks? Select all that apply.

    Any that apply
    • A larger black bin
    • Extra collections for nappy and hygiene waste
    • Something else would be needed
    • I am not sure

    Routing Shown only if Q1 is: A child under 5 lives in my household; Someone in my household uses continence products; Someone in my household uses other hygiene products that go in the black bin

  10. Q6

    The council is considering four options. Please rank them from your most preferred (1) to your least preferred (4).

    Ranking
    • Collect general waste every three weeks across the whole district
    • Collect every three weeks, with extra support for households that need it
    • Pilot every-three-week collections in some areas first, before deciding on the whole district
    • Keep general waste collection every two weeks and drop the proposal
  11. Q7

    Suppose the council offered extra support to households that need it, such as a larger black bin or extra collections for nappy and hygiene waste. How would this affect your view of collecting general waste every three weeks?

    One answer
    • I would be more in favour
    • It would make no difference to my view
    • I would be less in favour
    • Not applicable to my household
  12. Q8

    If general waste were collected every three weeks, how, if at all, would your household's recycling change?

    One answer
    • Recycle a lot more
    • Recycle a little more
    • No change
    • Recycle a little less
    • Recycle a lot less
    • Not sure
  13. Q9

    Which of these best describes your home?

    One answer
    • I own it outright
    • I own it with a mortgage or loan
    • Shared ownership
    • Rent from the council or a housing association
    • Rent from a private landlord
    • Other
  14. Q10

    Is there anything else you would like to tell the council about the proposal for general waste collection?

    Open text
Sample plan

Two routes, same questionnaire. (1) Residents' panel: letters to randomly selected addresses across all six wards, with invitations allocated by ward and sampling within each ward. Required precision: ±6 points at 95% confidence per ward needs about 267 completes per ward before weighting (1.96² × 0.25 ÷ 0.06²), so at least 1,600 panel completes in total. A 420-complete panel gives about ±11.7 points per ward at most, so it cannot meet the councillors' requirement. If weighting to age and tenure raises the design effect to about 1.3, the target rises to about 350 per ward (about 2,100 in total). Decision needed from the council: fund the larger panel, or accept ±12 points per ward and report it as such. Invitation volume depends on response rate: at a 10 to 15% response rate, 1,600 completes need roughly 10,700 to 16,000 letters. Quotas on the panel: age band and tenure (Q9 and S1) set to match the district's latest census profile within each ward, with the ward minimum (267 per ward) as the first quota. Weight to district age and tenure profile after fieldwork. Households with a child under 5 or someone using continence or hygiene products cannot be quota-set in advance; monitor their numbers during fieldwork and report subgroup results with their unweighted counts. (2) Open consultation: publicly linked on the council website, promoted on social media and in the residents' magazine. No quotas or weights. Remove duplicate responses (one per person, checked by postcode and contact details where given). Report separately from the panel and do not pool the two into one district figure, because the open consultation is self-selected and is not representative of residents. Both routes use the same screener (S1 to S4), and anyone who does not qualify is closed without answering the core questions.