How many test cases do I need?
A dozen cases tell you less than they seem to. Enter your own numbers to see how sure they can make you. It's the same maths the Prompt test uses for its verdicts.
How sure is one pass rate?
The range is a 95% Wilson interval: given these cases, the pass rate you would see over many more like them very likely sits inside it. It stays honest at the edges, so 12 of 12 doesn't mean "always".
| Cases | 5 | 12 | 24 | 50 | 100 |
|---|---|---|---|---|---|
| Likely pass rate | 57–100% | 76–100% | 86–100% | 93–100% | 96–100% |
Is one model really better than another?
Only the cases where one model passed and the other failed say which is better. Cases they both passed, or both failed, say nothing about the difference. On those cases we use an exact sign test: how likely a split at least this lopsided would be if the two were equally good. Below 1 in 20 (p under 0.05) we call it clear. With fewer than 6 cases nothing can be clear: even a 5–0 split comes up by chance 1 time in 16.
Questions
So how many cases do I need?
It depends on the gap you need to see. A model that fails half of 12 cases can be told apart from one that passes all 12: that's a 6–0 split, which chance would give less than 1 time in 20. Two models a few points apart can take hundreds of cases. Start with 12 to 24 cases that look like your real inputs, run them, and let the second calculator tell you whether to add more.
Why not just compare the two pass rates?
Because most of each rate comes from cases both models got right or both got wrong. Eleven of twelve against nine of twelve sounds like a gap, but it could be two cases where one did better, and that happens by chance often.
Does a clear difference mean the better model is right for me?
Only on cases like these, and only as far as the checks are fair. A clear result on easy cases says nothing about hard ones. Make the cases look like your real inputs, including the awkward ones, and check that the checks match what you'd judge by hand: see how often our judge agrees with people.
What if a test is judged by an AI?
Then a pass is the judge's verdict, and the judge can be wrong. The maths here treats every verdict as right, so the real uncertainty is a little wider.
Run the cases, then read the verdict.
The Prompt test runs your cases on several models, checks each answer, and tells you whether the gap between them is real.