Benchmarks / SurveyBench / Should Corefit gyms stop opening 24 hours? / Gemini 3.5 Flash Lite
Measured by Spring Prompt
Gemini 3.5 Flash Lite: Should Corefit gyms stop opening 24 hours?
The decision: whether to close all 14 gyms between 11pm and 5am, close overnight only at the quietest gyms, or keep every gym open 24 hours. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 20 of 100
- Analysis rating
- 662
- Head to head, this task
- won 2 of 10
Why it is not sound
The survey
- Produced no usable survey
The analysis
- Got at least one set number wrong
- Made a recommendation that does not follow from the data
- Missed a trap: 14.9% of respondents report at least one visit between 11pm and 5am in the last four weeks, against 6% of members in the door-entry records: members with a stake in the overnight hours were more likely to respond. Unweighted, 14.6% say they would be likely to cancel; weighted to the 6%, 11.4%.
- Missed a trap: 'Likely to cancel' answers (45.8% of overnight users, 14.6% overall) are stated intentions about a hypothetical closure, asked straight after members were invited to think about it.
- Missed a trap: The mean number of overnight visits among overnight users (7.4) is pulled up by night-shift workers who come almost daily; the median is in the '2 to 3' band.
- Missed a trap: The operations director believes almost nobody uses the gyms overnight. Partly true: 6% of members in the door-entry records (about 2,316 people). But night-shift workers (5.9% of respondents) mostly train between 11pm and 5am (77.8% of those who visited overnight usually arrive then) and are the most likely to say they would cancel.
- Missed a trap: At Wakefield Calder Park (base 11), 54.5% of respondents used the gym overnight and 27.3% say they would be likely to cancel.
- Missed a trap: North West England respondents (base 229) look more likely to cancel than North East England respondents (base 220): 19.2% against 13.6%, a gap that is not statistically significant at 95%.
- Missed a trap: Only 950 of 11,900 invited members completed (about 8%); those who responded differ from those who did not, as the overnight-use skew shows.
- Missed a trap: Only current members were surveyed: people who joined for 24-hour access and have since left, and people who would join only because of it, are missing, so the survey cannot show what 24-hour opening does for recruitment or the brand; and night-shift workers are a small group whose answers carry little weight in the totals even though they depend on the hours.
Scorecard
Numbers 8 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents reported at least one visit to a Corefit gym between 11pm and 5am in the last four weeks? | ✓ | 14.9% | 14.9% |
| What percentage of all respondents said they would be likely (fairly or very) to cancel their membership if their home gym closed between 11pm and 5am? | ✓ | 14.6% | 14.6% |
| Among respondents who reported at least one visit between 11pm and 5am in the last four weeks, what percentage said they would be likely (fairly or very) to cancel? | ✓ | 45.8% | 45.8% |
| Weighted so that respondents with at least one visit between 11pm and 5am in the last four weeks count as 6% of members (the door-entry figure) and the rest as 94%, what percentage would be likely (fairly or very) to cancel? | ✓ | 11.1% | 11.4% |
| If the weighted share likely to cancel were applied to all 38,600 members, about how many members would that be? | ✓ | 4,285 | 4,383 |
| Among respondents who reported at least one visit between 11pm and 5am in the last four weeks, in which band of the Q1 table does the median number of visits fall? Give the lowest value of that band (for example 4 for '4 to 7'). | ✗ | 4 | 2 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents who reported at least one visit between 11pm and 5am in the last four weeks? | ✓ | 2.3 points | 2.27 points |
| Is the difference between North West England and North East England respondents in the share likely (fairly or very) to cancel statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers say that 11pm to 5am is the only time the respondent can train because they work shifts or nights? | ✗ | 9 | 16 |
| How many of the open-text answers from respondents who visited between 11pm and 5am in the last four weeks say they would move to another gym? | ✗ | 7 | 19 |
| What percentage of the members invited completed the survey? | ✓ | 8% | 8% |
Traps in the data 0 of 8 handled
- ✗14.9% of respondents report at least one visit between 11pm and 5am in the last four weeks, against 6% of members in the door-entry records: members with a stake in the overnight hours were more likely to respond. Unweighted, 14.6% say they would be likely to cancel; weighted to the 6%, 11.4%.
- ✗'Likely to cancel' answers (45.8% of overnight users, 14.6% overall) are stated intentions about a hypothetical closure, asked straight after members were invited to think about it.
- ✗The mean number of overnight visits among overnight users (7.4) is pulled up by night-shift workers who come almost daily; the median is in the '2 to 3' band.
- ✗The operations director believes almost nobody uses the gyms overnight. Partly true: 6% of members in the door-entry records (about 2,316 people). But night-shift workers (5.9% of respondents) mostly train between 11pm and 5am (77.8% of those who visited overnight usually arrive then) and are the most likely to say they would cancel.
- ✗At Wakefield Calder Park (base 11), 54.5% of respondents used the gym overnight and 27.3% say they would be likely to cancel.
- ✗North West England respondents (base 229) look more likely to cancel than North East England respondents (base 220): 19.2% against 13.6%, a gap that is not statistically significant at 95%.
- ✗Only 950 of 11,900 invited members completed (about 8%); those who responded differ from those who did not, as the overnight-use skew shows.
- ✗Only current members were surveyed: people who joined for 24-hour access and have since left, and people who would join only because of it, are missing, so the survey cannot show what 24-hour opening does for recruitment or the brand; and night-shift workers are a small group whose answers carry little weight in the totals even though they depend on the hours.
The analysis
Recommendation
Findings
The survey it planned
This model produced no usable survey for this task.