Benchmarks / SurveyBench / Should Corefit gyms stop opening 24 hours? / Mistral Medium 3.5

Measured by Spring Prompt

Mistral Medium 3.5: Should Corefit gyms stop opening 24 hours?

The decision: whether to close all 14 gyms between 11pm and 5am, close overnight only at the quietest gyms, or keep every gym open 24 hours. The brief and the data →

Verdict
✗ Not sound
Research score
55 of 100
Analysis rating
540
Head to head, this task
won 3 of 15

Why it is not sound

The survey

  • A research objective the survey could not answer
  • Ignored the precision the brief asked for, which the sample cannot deliver
  • Answer bands that overlap or leave gaps

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: 'Likely to cancel' answers (45.8% of overnight users, 14.6% overall) are stated intentions about a hypothetical closure, asked straight after members were invited to think about it.
  • Missed a trap: At Wakefield Calder Park (base 11), 54.5% of respondents used the gym overnight and 27.3% say they would be likely to cancel.
  • Missed a trap: Only current members were surveyed: people who joined for 24-hour access and have since left, and people who would join only because of it, are missing, so the survey cannot show what 24-hour opening does for recruitment or the brand; and night-shift workers are a small group whose answers carry little weight in the totals even though they depend on the hours.
  • Proposed a mistaken next step: Decide the hours at Wakefield Calder Park (or across the chain) from its survey result alone

Findings the judges found unsupported

  • Only 14.9% of respondents (and 6% of all members) used the gym between 11pm and 5am in the last four weeks, but 45.8% of these users would be likely to cancel if overnight access was removed.The percentages are supported, but high-confidence wording about what users would do does not distinguish hypothetical stated cancellation intentions from actual behaviour. The 6% administrative figure also covers September, not exactly each respondent's last four weeks.
  • Weighted to the full membership base (6% overnight users), 10.1% of all members would be likely to cancel if overnight access was removed, equating to ~3,900 members.The displayed calculation equals 11.4%, not 10.1%. Applying it to membership gives about 4,400 members expressing an intention, not 3,900 forecast cancellations. Weighting on overnight use alone does not establish representativeness within the two usage groups.
  • Overnight users are disproportionately likely to work shifts that include nights (57.1% of overnight users vs. 5.9% of all respondents). This group is highly sensitive to overnight closure.Night-shift workers constitute about 31.7% of overnight respondents (45 of 142), not 57.1%. The 57.1% figure is the share of night-shift respondents reporting 15 or more overnight visits.
  • Wakefield Calder Park has the highest share of overnight users (54.5%) and the highest likelihood to cancel (27.3%). This gym is an outlier and may warrant special consideration.The percentages describe just 11 respondents, including six overnight users and three likely cancellers. Calling the gym an outlier warranting special consideration, and subsequently selecting it for continued overnight opening, treats an unstable survey result as evidence about the gym without a clear small-base warning.
  • Open-text responses reveal strong emotional reactions: overnight users threaten to switch gyms (12 mentions), while non-users demand price reductions (40+ mentions) or express relief about safety (20+ mentions).The supplied answers contain more than 12 overnight respondents mentioning switching gyms; the analysis itself later reports 24. The later theme totals, including 144 price-reduction answers and 107 safety answers, are also not supported by the supplied texts. Qualitative themes are supported, but the reported counts are unreliable.

Scorecard

Numbers 8 of 11 right

QuestionIts answerTrue
What percentage of respondents reported at least one visit to a Corefit gym between 11pm and 5am in the last four weeks?✓14.9%14.9%
What percentage of all respondents said they would be likely (fairly or very) to cancel their membership if their home gym closed between 11pm and 5am?✓14.6%14.6%
Among respondents who reported at least one visit between 11pm and 5am in the last four weeks, what percentage said they would be likely (fairly or very) to cancel?✓45.8%45.8%
Weighted so that respondents with at least one visit between 11pm and 5am in the last four weeks count as 6% of members (the door-entry figure) and the rest as 94%, what percentage would be likely (fairly or very) to cancel?✗10.1%11.4%
If the weighted share likely to cancel were applied to all 38,600 members, about how many members would that be?✗3,8994,383
Among respondents who reported at least one visit between 11pm and 5am in the last four weeks, in which band of the Q1 table does the median number of visits fall? Give the lowest value of that band (for example 4 for '4 to 7').✓22
What is the 95% margin of error, in percentage points, on the unweighted share of respondents who reported at least one visit between 11pm and 5am in the last four weeks?✓2.3 points2.27 points
Is the difference between North West England and North East England respondents in the share likely (fairly or very) to cancel statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers say that 11pm to 5am is the only time the respondent can train because they work shifts or nights?✓1516
How many of the open-text answers from respondents who visited between 11pm and 5am in the last four weeks say they would move to another gym?✗1219
What percentage of the members invited completed the survey?✓8%8%

Traps in the data 5 of 8 handled

  • ✓14.9% of respondents report at least one visit between 11pm and 5am in the last four weeks, against 6% of members in the door-entry records: members with a stake in the overnight hours were more likely to respond. Unweighted, 14.6% say they would be likely to cancel; weighted to the 6%, 11.4%.
  • ✗'Likely to cancel' answers (45.8% of overnight users, 14.6% overall) are stated intentions about a hypothetical closure, asked straight after members were invited to think about it.
  • ✓The mean number of overnight visits among overnight users (7.4) is pulled up by night-shift workers who come almost daily; the median is in the '2 to 3' band.
  • ✓The operations director believes almost nobody uses the gyms overnight. Partly true: 6% of members in the door-entry records (about 2,316 people). But night-shift workers (5.9% of respondents) mostly train between 11pm and 5am (77.8% of those who visited overnight usually arrive then) and are the most likely to say they would cancel.
  • ✗At Wakefield Calder Park (base 11), 54.5% of respondents used the gym overnight and 27.3% say they would be likely to cancel.
  • ✓North West England respondents (base 229) look more likely to cancel than North East England respondents (base 220): 19.2% against 13.6%, a gap that is not statistically significant at 95%.
  • ✓Only 950 of 11,900 invited members completed (about 8%); those who responded differ from those who did not, as the overnight-use skew shows.
  • ✗Only current members were surveyed: people who joined for 24-hour access and have since left, and people who would join only because of it, are missing, so the survey cannot show what 24-hour opening does for recruitment or the brand; and night-shift workers are a small group whose answers carry little weight in the totals even though they depend on the hours.

The analysis

Recommendation

Do not close all gyms overnight. Instead, keep 24-hour access at gyms with the highest overnight usage (e.g., Wakefield Calder Park, Glasgow Finnieston, Edinburgh Leith) and consider closing overnight only at the quietest gyms (e.g., Leeds Kirkstall, Preston Centre) where overnight usage is below 10% and cancellation risk is lowest. Introduce a tiered pricing model for gyms with reduced hours to mitigate member losses.

What would change it

Evidence that overnight usage is higher than 6% across all gyms, or that the weighted cancellation rate exceeds 15% (implying >5,900 member losses), would justify keeping all gyms open 24 hours. Conversely, evidence that overnight users are willing to accept a price reduction (e.g., £3-£5 off) without switching gyms would support closing overnight at more gyms.

Findings

  1. Only 14.9% of respondents (and 6% of all members) used the gym between 11pm and 5am in the last four weeks, but 45.8% of these users would be likely to cancel if overnight access was removed.

    High confidenceBase 950

    Q1: 14.9% of 950 respondents reported at least one overnight visit; Q4: 45.8% of these 142 respondents said they would be likely to cancel. Door-entry data shows 6% of all members used overnight in September 2026.

  2. Weighted to the full membership base (6% overnight users), 10.1% of all members would be likely to cancel if overnight access was removed, equating to ~3,900 members.

    High confidenceBase 950

    Weighted calculation: (0.06 * 45.8) + (0.94 * 9.2) = 10.1%. Applied to 38,600 members: 38,600 * 0.101 ≈ 3,899.

  3. 24-hour access is a key reason for joining for overnight users (59.2% cited it) but only 17.8% of non-overnight users did. This suggests overnight access is a differentiator for a small but committed segment.

    High confidenceBase 950

    Q5: 59.2% of 142 overnight users vs. 17.8% of 808 non-overnight users selected 'Open 24 hours' as a reason for joining.

  4. North West England respondents are not significantly more likely to cancel than North East England respondents (19.2% vs. 13.6%, p > 0.05). The regional manager’s concern is not statistically supported.

    High confidenceBase 449

    Q4: 19.2% of 229 North West respondents vs. 13.6% of 220 North East respondents said they would be likely to cancel. Margin of error for North West: ±4.2pp; North East: ±4.3pp. Overlap in confidence intervals indicates no significant difference.

  5. Overnight users are disproportionately likely to work shifts that include nights (57.1% of overnight users vs. 5.9% of all respondents). This group is highly sensitive to overnight closure.

    High confidenceBase 950

    Q1 and Q7: 57.1% of 142 overnight users work shifts that include nights vs. 5.9% of 950 total respondents.

  6. Wakefield Calder Park has the highest share of overnight users (54.5%) and the highest likelihood to cancel (27.3%). This gym is an outlier and may warrant special consideration.

    Medium confidenceBase 11

    Q1 and Q4 by home gym: Wakefield Calder Park has 54.5% overnight users and 27.3% likely to cancel.

  7. Open-text responses reveal strong emotional reactions: overnight users threaten to switch gyms (12 mentions), while non-users demand price reductions (40+ mentions) or express relief about safety (20+ mentions).

    Medium confidenceBase 239

    Q3 open-text: 12 overnight users mention switching gyms; 40+ non-users demand price cuts; 20+ non-users express safety concerns.

Caveats

  • The survey response rate is low (8.0%), which may introduce non-response bias, particularly if overnight users are more or less likely to respond.
  • The open-text answers are self-selected (239 out of 950 respondents) and may overrepresent strong opinions.
  • Door-entry data (6% overnight users) is based on a single month (September 2026) and may not reflect long-term patterns.
  • The weighted cancellation estimate assumes the survey’s overnight user share (14.9%) is representative of the full membership (6%), which may not hold if survey respondents are not representative.
  • The margin of error for sub-groups (e.g., North West vs. North East) is higher due to smaller bases, reducing confidence in regional comparisons.
  • The survey does not measure the financial impact of price reductions, which could offset member losses but are not quantified here.

Next steps

  • researchConduct a pricing sensitivity analysis to determine the optimal discount for gyms with reduced hours, balancing member retention and revenue.
  • experimentPilot overnight closure at 2-3 of the quietest gyms (e.g., Leeds Kirkstall, Preston Centre) for 3 months, with discounted pricing and member feedback tracking.
  • researchAnalyze door-entry data for the past 12 months to identify seasonal or long-term trends in overnight usage, particularly at Wakefield Calder Park and other high-usage gyms.
  • researchSurvey non-respondents (e.g., via a short SMS or email) to assess whether overnight users are underrepresented in the current data.
  • monitorMonitor competitor pricing and 24-hour offerings in the North West and North East to assess the risk of member migration.
  • decisionDecide on a final policy for overnight access within 6 months, based on pilot results, pricing analysis, and competitor monitoring.

Open-text themes it coded

Would switch to another 24-hour gym 24Demands price reduction if hours are cut 144Safety concerns about unstaffed overnight gyms 107Overnight closure would crowd early/late slots 85Overnight access is the only time they can train due to shift work 15

The survey it planned

1 screening question and 10 questions, as the model wrote them.

  1. S1

    Are you a current member of Corefit?

    One answer
    • Yes
    • No

    Continues if Yes

  2. Q1

    Which Corefit gym is your home gym?

    One answer
    • Gym A (Location)
    • Gym B (Location)
    • Gym C (Location)
    • Gym D (Location)
    • Gym E (Location)
    • Gym F (Location)
    • Gym G (Location)
    • Gym H (Location)
    • Gym I (Location)
    • Gym J (Location)
    • Gym K (Location)
    • Gym L (Location)
    • Gym M (Location)
    • Gym N (Location)
  3. Q2

    In the past 30 days, how many times have you visited any Corefit gym between 11pm and 5am?

    One answer
    • 0 times
    • 1-2 times
    • 3-5 times
    • 6-10 times
    • 11-20 times
    • More than 20 times
  4. Q3

    If Corefit closed all gyms between 11pm and 5am, how likely would you be to continue your membership?

    Scale

    Scale 1-5: Very unlikely to Very likely

  5. Q4

    If Corefit closed your home gym between 11pm and 5am but kept other gyms open 24 hours, how likely would you be to continue your membership?

    Scale

    Scale 1-5: Very unlikely to Very likely

  6. Q5

    When you joined Corefit, how important was 24-hour access to the gym in your decision to choose Corefit? (Select one)

    One answer
    • Not at all important
    • Slightly important
    • Moderately important
    • Very important
    • Essential
  7. Q6

    Which of the following were reasons for you choosing Corefit? (Select all that apply)

    Any that apply
    • 24-hour access
    • Low cost
    • Location/convenience
    • Facilities/equipment
    • Atmosphere/community
    • No contract/flexibility
    • Other (please specify)
  8. Q7

    How often do you typically visit the gym between 11pm and 5am?

    One answer
    • Never
    • Less than once a month
    • Once a month
    • 2-3 times a month
    • Once a week
    • 2-3 times a week
    • 4-5 times a week
    • Every day

    Routing Shown only if Q2 is: 1-2 times; 3-5 times; 6-10 times; 11-20 times; More than 20 times

  9. Q8

    What is your main reason for visiting the gym between 11pm and 5am? (Select one)

    One answer
    • Work schedule
    • Personal preference for quiet/less crowded times
    • Other commitments during the day
    • Health/wellness routine
    • Other (please specify)

    Routing Shown only if Q2 is: 1-2 times; 3-5 times; 6-10 times; 11-20 times; More than 20 times

  10. Q9

    If Corefit closed all gyms between 11pm and 5am, would you be likely to switch to another gym that offers 24-hour access?

    One answer
    • Yes
    • No
    • Maybe
  11. Q10

    To ensure we are getting accurate responses, please select 'Strongly agree' for this question.

    Scale

    Scale 1-5: Strongly disagree to Strongly agree

Sample plan

Random sample of 1000 current Corefit members, stratified by home gym to ensure proportional representation across all 14 gyms. No quotas beyond gym stratification.