Benchmarks / SurveyBench / Should Corefit gyms stop opening 24 hours? / Claude Opus 5.5

Measured by Spring Prompt

Claude Opus 5.5: Should Corefit gyms stop opening 24 hours?

The decision: whether to close all 14 gyms between 11pm and 5am, close overnight only at the quietest gyms, or keep every gym open 24 hours. The brief and the data →

Verdict
✗ Not sound
Research score
85 of 100
Analysis rating
1,646
Head to head, this task
won 9 of 11

Why it is not sound

The survey

  • Answer bands that overlap or leave gaps

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: The mean number of overnight visits among overnight users (7.4) is pulled up by night-shift workers who come almost daily; the median is in the '2 to 3' band.

Findings the judges found unsupported

  • Overnight use is a small minority, but it is not 'almost nobody'. About 2,300 members use the gyms between 11pm and 5am, and they use them heavily. Use the door-entry figure, not the survey figure, to size this group.The blanket description of overnight members as heavy users is not established. The mean of 7.4 visits is pulled upward by frequent night-shift users; the median overnight respondent is in the 2-to-3-visit band. The respondent mean also does not establish visit frequency among all 2,316 overnight members.
  • The people who would lose out from a closure are mainly night-shift workers. They have no other time to train, so they are the group most likely to actually leave.Night-shift workers show the greatest dependence and highest stated cancellation likelihood, but the data do not establish that they constitute most people affected or that they are most likely to actually leave. Having no alternative training time applies to some respondents, not necessarily the whole group.
  • The realistic churn risk is roughly 1,000 overnight users. The headline figure of about 4,400 is an upper bound on stated intent, and much of it is a protest about price.Calling approximately 1,060 members the realistic churn risk turns hypothetical cancellation intentions into a forecast. The approximately 4,399 weighted stated intentions are not an established upper bound either. No Q4-by-open-text evidence shows that likely-to-cancel non-users are predominantly protesting price or that all say their visiting times would be unchanged.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents reported at least one visit to a Corefit gym between 11pm and 5am in the last four weeks?✓14.9%14.9%
What percentage of all respondents said they would be likely (fairly or very) to cancel their membership if their home gym closed between 11pm and 5am?✓14.6%14.6%
Among respondents who reported at least one visit between 11pm and 5am in the last four weeks, what percentage said they would be likely (fairly or very) to cancel?✓45.8%45.8%
Weighted so that respondents with at least one visit between 11pm and 5am in the last four weeks count as 6% of members (the door-entry figure) and the rest as 94%, what percentage would be likely (fairly or very) to cancel?✓11.4%11.4%
If the weighted share likely to cancel were applied to all 38,600 members, about how many members would that be?✓4,3994,383
Among respondents who reported at least one visit between 11pm and 5am in the last four weeks, in which band of the Q1 table does the median number of visits fall? Give the lowest value of that band (for example 4 for '4 to 7').✓22
What is the 95% margin of error, in percentage points, on the unweighted share of respondents who reported at least one visit between 11pm and 5am in the last four weeks?✓2.3 points2.27 points
Is the difference between North West England and North East England respondents in the share likely (fairly or very) to cancel statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers say that 11pm to 5am is the only time the respondent can train because they work shifts or nights?✓1616
How many of the open-text answers from respondents who visited between 11pm and 5am in the last four weeks say they would move to another gym?✓1919
What percentage of the members invited completed the survey?✓8%8%

Traps in the data 7 of 8 handled

  • ✓14.9% of respondents report at least one visit between 11pm and 5am in the last four weeks, against 6% of members in the door-entry records: members with a stake in the overnight hours were more likely to respond. Unweighted, 14.6% say they would be likely to cancel; weighted to the 6%, 11.4%.
  • ✓'Likely to cancel' answers (45.8% of overnight users, 14.6% overall) are stated intentions about a hypothetical closure, asked straight after members were invited to think about it.
  • ✗The mean number of overnight visits among overnight users (7.4) is pulled up by night-shift workers who come almost daily; the median is in the '2 to 3' band.
  • ✓The operations director believes almost nobody uses the gyms overnight. Partly true: 6% of members in the door-entry records (about 2,316 people). But night-shift workers (5.9% of respondents) mostly train between 11pm and 5am (77.8% of those who visited overnight usually arrive then) and are the most likely to say they would cancel.
  • ✓At Wakefield Calder Park (base 11), 54.5% of respondents used the gym overnight and 27.3% say they would be likely to cancel.
  • ✓North West England respondents (base 229) look more likely to cancel than North East England respondents (base 220): 19.2% against 13.6%, a gap that is not statistically significant at 95%.
  • ✓Only 950 of 11,900 invited members completed (about 8%); those who responded differ from those who did not, as the overnight-use skew shows.
  • ✓Only current members were surveyed: people who joined for 24-hour access and have since left, and people who would join only because of it, are missing, so the survey cannot show what 24-hour opening does for recruitment or the brand; and night-shift workers are a small group whose answers carry little weight in the totals even though they depend on the hours.

The analysis

Recommendation

Do not close all 14 gyms overnight now.

The operations director's premise is half right. Only about 6% of members use the gyms overnight. But those members are heavy users, are concentrated among night-shift workers, and nearly half say they would cancel. That is roughly 1,000 members, around £18–20k a month in revenue at about £18–20 a month each, before counting demands for price cuts from members who are not affected.

The realistic option is selective overnight closure, chosen from door-entry data rather than this survey. Run it as a controlled pilot at the one or two gyms with the fewest overnight entries. Keep at least one 24-hour gym within reach in each city or area so night workers have somewhere to go.

On regions, do not single out the North West: its difference from the North East is not significant.

Pair any closure with three things:

  • a clear message on price or value, since about a fifth of comments expect a fee cut;
  • measures to manage 5am and 10pm crowding;
  • the safety improvements many members said they want.

Proceed only if the pilot shows that the savings exceed the measured churn.

What would change it

Close more widely if any of these hold:

  • door-entry data show overnight users at the candidate gyms are far fewer than 6%;
  • the pilot shows actual churn well below stated intent, for example under 15% of overnight users;
  • verified overnight cost savings clearly exceed the lost revenue.

Keep all gyms 24 hours if any of these hold:

  • pilot churn approaches the stated 46% among overnight users;
  • churn spills over to non-users;
  • prospect research shows 24-hour access materially drives new joins.

Corrected data linkage could also change which gyms or regions are candidates.

Findings

  1. Overnight use is a small minority, but it is not 'almost nobody'. About 2,300 members use the gyms between 11pm and 5am, and they use them heavily. Use the door-entry figure, not the survey figure, to size this group.

    High confidenceBase 950

    Door-entry records show 6% of 38,600 members (about 2,316) entered between 11pm and 5am in September. The survey's self-reported figure is 14.9% ±2.3pp (n=950). That is more than double the door-entry figure, which points to over-reporting or to keener respondents. On the arrival-time question, 5.4% usually arrive between 11pm and 5am, which is close to 6%. Overnight users averaged 7.4 overnight visits in four weeks; 32 of the 142 reported 15 or more.

  2. The people who would lose out from a closure are mainly night-shift workers. They have no other time to train, so they are the group most likely to actually leave.

    Medium confidenceBase 56

    Shift workers whose shifts include nights: 80.4% visited overnight, 62.5% usually arrive between 11pm and 5am, and 48.2% say they are likely to cancel (n=56). In the open text, 16 answers say 11pm to 5am is their only possible training time (nurses, care workers, warehouse and HGV workers). Many say closure would end their gym use altogether.

  3. The realistic churn risk is roughly 1,000 overnight users. The headline figure of about 4,400 is an upper bound on stated intent, and much of it is a protest about price.

    Medium confidenceBase 950

    45.8% of overnight users are likely to cancel (n=142), against 9.2% of non-users (n=808). Weighted to the door-entry 6%, 11.4% are likely to cancel, about 4,399 members. Applying 45.8% to the roughly 2,316 overnight users gives about 1,060. The 9.2% of non-users (about 3,340 if projected) say a closure would not change when they go, and their open-text answers are dominated by demands for a lower fee. Stated intent usually overstates real cancellation.

  4. Closing overnight without a price response would create widespread resentment about price among members who are otherwise unaffected.

    Medium confidenceBase 239

    53 of 239 open-text answers expect a lower monthly fee. Most come from people who never visit overnight. Several name amounts of £3 to £10 off.

  5. Overnight closure would also have upsides and knock-on costs that the cancellation question did not capture: some members would feel safer, and others expect crowding at 10pm and from 5am.

    Medium confidenceBase 239

    43 answers welcome a closure because unstaffed night opening feels unsafe; one woman mentions being alone. 31 answers expect displaced users to crowd the 10pm to 11pm and 5am to 6am slots. That would harm the quiet slots these members chose deliberately.

  6. 24-hour opening is a reason to join for a minority, so it has some acquisition value. Price and location matter far more.

    Medium confidenceBase 950

    'Open 24 hours' was a joining reason for 24.0% overall, 59.2% of overnight users (n=142) and 17.8% of non-users (n=808). Low price was cited by 68.9% and location by 62.6%. People who have not joined were not surveyed.

  7. The North West does not show a reliable difference from the North East, in either overnight use or cancellation intent.

    Medium confidenceBase 449

    Likely to cancel: North West 19.2% (n=229) against North East 13.6% (n=220). The test gives z≈1.6, p≈0.11, so the gap is not significant. Overnight use is 14.0% in the North West and 15.0% in the North East.

  8. The survey cannot pick out the 'quietest gyms'. Gym-level selection must come from door-entry data.

    Low confidenceBase 950

    Gym bases are 61 to 81, giving margins of roughly ±7 to 9pp on overnight use. Wakefield has n=11 (54.5% overnight use, which is unreliable). Survey 'lows' such as Leeds Kirkstall at 7.5% and Preston at 7.7% sit within noise of the 14.9% average.

Caveats

  • The response rate is 8.0% (950 of 11,900), and results are unweighted. Self-reported overnight use (14.9%) is well above the door-entry figure (6%). Either overnight users responded more readily or Q1 over-counts, for example by including people who arrive before 11pm and stay past it. Unweighted totals therefore overstate the impact.
  • Q4 is a hypothetical intent question with no price or alternative attached. Stated likelihood to cancel usually exceeds actual cancellation. However, rolling monthly contracts make leaving easy for the night-shift group.
  • There are data-linkage problems. Many open-text answers name a city that conflicts with the recorded home gym: - M100 is Leeds but pays 'in Glasgow'. - M321 is Hull but works near Motherwell. - M467 is Sheffield and mentions Gateshead. - Similar conflicts appear in M059, M121, M178, M224, M389, M429, M551, M625, M685, M713, M793, M800, M823 and M889. Some answers also contradict their Q1 coding: M034, M305 and M735 are coded as overnight visitors but say they never go at night. Check the respondent-to-segment matching before relying on any gym or region cut.
  • Gym-level bases are small (61 to 81, Wakefield 11), and region differences are not significant. Neither can support a gym-by-gym or regional decision.
  • The survey says nothing about non-members or prospects, so the acquisition value of the 24-hour brand is unmeasured. It also says nothing about the cost savings from closing. The gyms are already unstaffed overnight, so savings may be modest (energy, security, insurance) and must be quantified.
  • The open-text answers are a self-selected 239 of 950 and should be read as indicative, not as prevalence.

Next steps

  • decisionHold the decision on an all-gym closure and on the North West exemption until the door-entry analysis and pilot are complete
  • researchAnalyse September door-entry data by gym, hour and member: unique overnight users, their visit frequency, and how many have no daytime visits
  • researchAudit how open-text answers and segments were linked to respondents, and whether Q6 answers match the membership-database home gym
  • researchBuild a cost model of overnight closure per gym (energy, security, insurance, cleaning) against at-risk revenue
  • experimentPilot an 11pm to 5am closure at one or two of the lowest-use gyms, with matched control gyms, for 8 to 12 weeks, giving members notice and a nearby 24-hour alternative
  • monitorMonitor 5am to 6am and 10pm to 11pm occupancy, and member complaints, at the pilot gyms
  • researchSurvey prospects and recent joiners on the role of 24-hour access in choosing a gym, and test messaging on price and value

Open-text themes it coded

No effect: never uses the gym overnight 90Expects a lower monthly fee if hours are cut 52Feels unsafe in unstaffed gyms at night; would welcome closure 42Worried displaced overnight users will crowd the 10pm or 5am slots 31Night or shift work makes 11pm to 5am the only time they can train 16Would leave Corefit or switch to a 24-hour competitor 20

The survey it planned

1 screening question and 10 questions, as the model wrote them.

  1. S1

    Are you currently a Corefit member?

    One answer
    • Yes, I'm currently a member
    • No, I've cancelled my membership
    • Not sure

    Continues if Yes, I'm currently a member

  2. Q1

    Thinking back to when you joined Corefit, which of these were reasons you chose it? Please select all that apply.

    Any that apply
    • The price
    • A gym is close to my home
    • A gym is close to my work or place of study
    • It's open 24 hours a day
    • Rolling monthly membership with no long contract
    • The equipment
    • The classes
    • Friends or family go there
    • It was less busy than other gyms
    • Other reason
    • I can't remember
  3. Q2

    In a typical month, how often do you go to a Corefit gym, at any time of day?

    One answer
    • Less than once a month
    • 1 to 3 times a month
    • About once a week
    • 2 to 3 times a week
    • 4 or more times a week
  4. Q3

    In the last 3 months, how often have you used a Corefit gym between 11pm and 5am?

    One answer
    • Never
    • Once or twice in the last 3 months
    • About once a month
    • 2 to 3 times a month
    • About once a week
    • 2 or more times a week
  5. Q4

    Why do you use the gym between 11pm and 5am? Please select all that apply.

    Any that apply
    • My working hours, such as shifts or night work
    • Caring or family responsibilities during the day
    • It's the only time that fits my routine
    • The gym is quieter, so equipment is free
    • I prefer to train when few people are around
    • Other reason

    Routing Shown only if Q3 is: Once or twice in the last 3 months; About once a month; 2 to 3 times a month; About once a week; 2 or more times a week

  6. Q5

    Suppose Corefit closed all its gyms between 11pm and 5am, and kept them open at all other times. What would you most likely do?

    One answer
    • It would make no difference to me
    • Keep my membership and go at other times, about as often as now
    • Keep my membership but go less often
    • Cancel and join a different gym
    • Cancel and stop going to a gym
    • Not sure
  7. Q6

    If Corefit closed all its gyms between 11pm and 5am, how likely is it that you would cancel your membership within the following 3 months?

    Scale

    Scale 1-5: Very unlikely to Very likely

  8. Q7

    Now suppose that only your home gym closed between 11pm and 5am, and at least one other Corefit gym in your area stayed open 24 hours. What would you most likely do?

    One answer
    • Go to the other Corefit gym between 11pm and 5am instead
    • Keep my membership and go to my home gym at other times, about as often as now
    • Keep my membership but go less often
    • Cancel and join a different gym
    • Cancel and stop going to a gym
    • Not sure

    Routing Shown only if Q3 is: Once or twice in the last 3 months; About once a month; 2 to 3 times a month; About once a week; 2 or more times a week

  9. Q8

    Which of these best describes your usual working pattern?

    One answer
    • Regular daytime hours
    • Evening work
    • Shifts that include night shifts
    • Shifts that don't include night shifts
    • Not in paid work, such as studying, retired or caring
    • Prefer not to say
  10. Q9

    How old are you?

    One answer
    • Under 18
    • 18 to 24
    • 25 to 34
    • 35 to 44
    • 45 to 54
    • 55 or over
    • Prefer not to say
  11. Q10

    Is there anything else you'd like to tell us about Corefit's opening hours? (Optional)

    Open text
Sample plan

PRECISION GAP. The board's target cannot be met with 1,000 completes. Each gym has about 2,760 members. A gym-level estimate within ±8 points at 95% confidence (worst case p=0.5, with finite population correction) needs about 145 completes per gym, or about 2,030 in total. Weighting for the overnight boost described below pushes the true figure higher. At 1,000 completes, each gym has about 71 completes, which gives about ±11–12 points per gym. The board should choose one of two options before fieldwork. (a) Fund about 2,100–2,300 completes, at about 150–165 per gym. (b) Accept 1,000 completes, report survey results by region and by gym group, and give gym-level figures for indicative purposes only. A key point applies under either option. The share of members using each gym between 11pm and 5am, and how often they use it, should be measured from door-entry records for all 38,600 members. That data has no sampling error and fully meets the per-gym precision need for usage. The survey is needed mainly for attitudes, reasons and likely behaviour if the gym closed. SAMPLING FRAME. All current members in the membership database. Exclude members who are frozen or mid-cancellation, members under 16, and anyone who has opted out of research emails. STRATIFICATION. Stratify by the 14 home gyms and by overnight-use status taken from door-entry records. An overnight user is anyone with at least one entry between 11pm and 5am in the last 3 months. Overnight users are only about 6–10% of members. A simple random sample of 1,000 would yield only about 60–100 of them, too few to measure the effect of closing on the people it affects. BASELINE ALLOCATION FOR 1,000 COMPLETES. Use about 71 completes per gym, made up of about 46 general-sample members and about 25 overnight users. Overnight users therefore number about 350 in total, which gives about ±5 points overall and about ±8–10 points per region. If a gym has fewer eligible overnight users than the target, invite all of them. INVITATIONS. Assume a 10–15% response rate, which means inviting about 8,000–10,000 members, with more invited in strata that respond poorly. Send one invitation and two reminders, and use a unique link per member. The link passes the member's ID, home gym and region, so these questions are not asked. It also allows linkage to door-entry records, tenure, membership fee and, where held, age. QUOTAS. Set soft quotas per gym × overnight stratum. Close a stratum once it reaches its target. WEIGHTING. Weight results back to each gym's true size and true overnight-user share from the database. Report effective sample sizes alongside results. DATA QUALITY. Remove responses completed in under 90 seconds, and responses where the reported overnight use in Q3 is inconsistent with door records, for example 'weekly' with no recorded overnight entries. NEUTRALITY. The survey is designed to measure impact, not to confirm any preferred option. The operations director's expectation that closing will not cost members is a hypothesis to test, not an outcome to show. Results will be reported whichever direction they point.