How often our judge agrees with people.
An AI judge is only useful if it reaches the verdicts a careful person would. So we count how often it does, and publish the count, including where it falls short.
The judge against people's own verdicts
In a Prompt test or a saved playground run, anyone can agree with the judge's verdict on a check, or give their own. These are all those verdicts, counted by type of check. The judge is GPT-6 Luna. We show a figure only once it rests on at least 20 verdicts from at least 3 different tests by different people. Only totals are published: no prompt, answer or check appears here.
| Type of check | Agreed | Likely range | Verdicts | Split votes |
|---|---|---|---|---|
| Every judged check | Not enough ratings yet | |||
| Yes/no checksA plain-language statement the judge marks pass or fail. | Not enough ratings yet | |||
| RubricsA quality placed on a scale of described levels. Agreement means the exact same level. | Not enough ratings yet | |||
| AI writing tellsA list of habits of AI writing; the check fails when any of them shows. | Not enough ratings yet | |||
| Must checksChecks a usable answer has to pass. | Not enough ratings yet | |||
| Stretch checksChecks only an excellent answer passes. | Not enough ratings yet | |||
Judges trained on people's ratings: not enough of them yet (we summarise them from 5).
How to read this
- Agreed is the share of people's verdicts the judge matched. For a rubric, only the exact same level counts.
- Likely range is a 95% interval, the same kind we give pass rates. Few verdicts mean a wide range.
- Split votes are answers where a judge asked three times didn't agree with itself. They are left out of the rate, and counted here so they aren't hidden.
- People are likelier to give their own verdict when they think the judge got it wrong, so these figures may understate how often it agrees.
- Every one of these people wrote their own checks. A vague check gets vague verdicts, from a judge or from a person. Disagreeing with the judge is how the Prompt test rewrites a check until they agree, so the checks people end up keeping may agree more than these figures show.
Our benchmark judges against each other
SurveyBench and DeckBench are judged by three judges from different developers, and each verdict is the majority. Below is how often they agreed, by check. This shows how repeatable the judging is, not whether it is right: three judges can agree and all be wrong. Checking the panel against people's ratings is still to do for both benchmarks.
Kappa adjusts for chance. When nearly every answer passes a check, judges agree almost always just by saying yes, so a high share can come with a low kappa. Below 0.4 suggests the check's wording leaves room for different readings, and the majority decides.
SurveyBench
| Check | Items | All three agreed | Kappa | Claude Opus 5.5 with the majority | GPT-6.1 Sol with the majority | Gemini 3.1 Pro with the majority |
|---|---|---|---|---|---|---|
| Analysis: trap handled | 1,395 | 88% | 0.73 substantial | 98% | 96% | 94% |
| Analysis: right step taken | 897 | 86% | 0.81 almost perfect | 99% | 93% | 95% |
| Analysis: wrong step taken | 546 | 93% | 0.78 substantial | 99% | 97% | 97% |
| Analysis: key finding stated | 514 | 87% | 0.73 substantial | 98% | 95% | 94% |
| Analysis: recommendation sound | 182 | 85% | 0.73 substantial | 97% | 91% | 97% |
| Analysis: confidence fits the evidence | 182 | 62% | 0.49 moderate | 98% | 79% | 85% |
| Analysis: says what would change it | 182 | 99% | 0.96 almost perfect | 100% | 100% | 99% |
| Analysis: material unsupported finding | 182 | 52% | 0.35 fair | 97% | 80% | 75% |
| Plan: leading question | 2,359 | 98% | 0.53 moderate | 99% | 99% | 99% |
| Plan: double-barrelled question | 2,359 | 92% | 0.35 fair | 98% | 99% | 95% |
| Plan: assumptive question | 2,359 | 93% | 0.47 moderate | 98% | 98% | 96% |
| Plan: ambiguous question | 2,359 | 95% | 0.30 fair | 99% | 97% | 99% |
| Plan: objective answerable | 788 | 93% | 0.39 fair | 99% | 94% | 100% |
| Plan: screener right | 197 | 76% | 0.28 fair | 94% | 86% | 96% |
| Plan: hypothesis tested | 197 | 87% | 0.61 substantial | 95% | 97% | 95% |
| Plan: precision handled | 197 | 91% | 0.85 almost perfect | 98% | 96% | 96% |
DeckBench
| Check | Items | All three agreed | Kappa | Claude Opus 5.5 with the majority | GPT-6.1 Sol with the majority | Gemini 3.1 Pro with the majority |
|---|---|---|---|---|---|---|
| Content: key finding carried | 900 | 100% | 0.94 almost perfect | 100% | 100% | 100% |
| Content: number right | 541 | 99% | 0.92 almost perfect | 99% | 100% | 99% |
| Content: recommendation matches the analysis | 225 | 98% | 0.73 substantial | 99% | 99% | 100% |
| Content: caveat carried | 225 | 82% | 0.70 substantial | 97% | 89% | 96% |
| Content: misleading metric used | 225 | 99% | 0.40 moderate | 100% | 100% | 99% |
| Content: material unsupported claim | 225 | 80% | 0.50 moderate | 98% | 89% | 92% |
| Design: slide has one clear message | 1,424 | 92% | 0.46 moderate | 98% | 96% | 97% |
| Design: clear visual hierarchy | 1,424 | 92% | 0.48 moderate | 95% | 99% | 98% |
| Design: slide rated polished | 1,424 | 53% | 0.37 fair | 84% | 95% | 74% |
| Design: chart fits the data | 469 | 84% | 0.50 moderate | 95% | 96% | 93% |
From every task's stored verdicts, public and private tasks alike; counts only. Worked out 8 Nov 2026.
CatalogBench uses a single judge, GPT-6.1 Sol. Before choosing it we compared it with a stronger, costlier judge (GPT-6 Astra at high reasoning): they agreed on 99.2% of 2,436 checks. We also re-judged a calibration set with judges from Google and Anthropic, and the order of the models didn't change. The CatalogBench page has the details.
Questions
Why use an AI judge at all?
Where a rule can decide, such as a word count or a phrase that must appear, a rule does, and it is never wrong. A judge is for what rules can't decide, like whether an answer invents a fact. It's checked against people because that's the only way to know it can be trusted.
What makes a judge agree more?
Simple yes/no questions about one thing, written so that a literal reader can't take them two ways. The Prompt test asks for your verdict where you disagree, and rewrites the check from it.
How many cases does a result need?
Fewer than you'd hope to see a big gap, more than you'd think to see a small one. Try the test case calculator.