No Benchmark Ranked This Month's AI Models the Way Reviewers Did. Ours Included.

The day after Gemini 4 Neon came out, Theo, who runs his own coding test suite, posted this: "Still waiting on numbers but honestly? I don't care. Astra showed me that my benchmarks suck. Neon is super pleasant to work with, it unblocks itself, makes good decisions, writes good code, and writes english SO well." (Theo, X, 19 November)
That same day our own run put Neon first of 35 models on the factual brief of CatalogBench, our product-listing benchmark. A reviewer and a benchmark agreed. We wanted to know whether that happens often, so we checked every model launched in the past month.
It doesn't. No leaderboard ranked the month's models the way reviewers did, and ours are not an exception. On direct "A beat B" calls the picture is better: agent benchmarks agreed with 83% to 88% of reviewer votes, the Artificial Analysis index with 80%, and most of our own business-task benchmarks with 63% to 71%. Reviewer by reviewer, each one's taste lines up best with a different benchmark, and none of those matches is stronger than chance would produce.
Top of the leaderboards, bottom with reviewers
A model's vibe score is the average of those −2 to +2 scores. We average within each reviewer first, so someone who posted ten times about one model counts once, and we rank only models that at least three reviewers used.

We measured how closely each benchmark's order follows the reviewers' with a rank correlation: +1 means the same order, 0 means no relation. Across the benchmarks the values ran from −0.55 to +0.17: no detectable relationship. Ours sit in the lower half of that range, with the Spring Prompt Score at −0.35 and DeckBench at −0.55. Three things explain most of it.
GPT-6.2 tops the charts and the complaints. Astra and Sol score 62 and 57 on the Artificial Analysis index and come 2nd and 3rd on our Score. With reviewers they are last. The most-viewed take we collected was Matt Shumer's on the pair: "I had a terrible experience testing them. They were essentially unusable." (X, 17 November) Every's team on Astra: "This is not a great model. Across everyone on our team who tried it, no one loved it." (YouTube, 17 November)
Theo's video shows the split inside one person. It opens with "In my own personal benches, 62 Astra shows dominance across the board". Half an hour in: "I do not like 62 Astra. I would not choose this over Opus 55 for any of my work as a dev." (YouTube, 18 November; later in the same video) His tests and his working day disagreed. (Our own figures are in the GPT-6.2 review.)
Cheap models are judged against their price. Simon Willison on Muse Ember: "The Ember pelicans are comparable in quality to those I got from Muse Wonder, which is 40x the price of Muse Ember." (blog, 10 November) Matthew Berman called DeepSeek Flash 1017 "potentially the best overall model on the planet right now" (YouTube, 28 October). Both models are in the reviewers' top three and score 36 and 33 on the Artificial Analysis index, against 65 for Claude Fable 5.2. A capability leaderboard doesn't ask whether a model is good for what it costs. Reviewers often do.
The top of the ranking is thin. Neon leads on four reviewers and one day of posts. Claude Fable 5.2 and GPT-6.2 Astra rest on 21 and 17 reviewers.
| Model | Vibe score | Reviewers |
|---|---|---|
| Gemini 4 Neon | +1.38 | 4 |
| Muse Ember | +1.25 | 4 |
| DeepSeek V4.1 Flash 1017 | +1.00 | 3 |
| Claude Fable 5.2 | +0.88 | 21 |
| Gemini 4 Carbon | +0.85 | 14 |
| GPT-6.1 Terra | +0.73 | 5 |
| Gemini 4 Hydrogen | +0.15 | 6 |
| Kimi K3.1 Pro | +0.08 | 7 |
| Qwen4-Max | +0.04 | 3 |
| GPT-6.2 Astra | −0.01 | 17 |
| GPT-6.2 Sol | −0.06 | 9 |
On "A beat B", agent benchmarks agree most
A narrower test: when a reviewer says from their own use that model A beat model B, does the benchmark also put A above B? We found 149 such statements and counted each reviewer once per pair of models.

Agent benchmarks come out on top because they test the work these reviewers do: 67 of the 250 quotes were about coding or agents, and another 59 were front-end, 3D and game demos. The fit is not automatic, though. LiveBench's coding category agreed with only 33% of votes.
The bars are not a league table. Four match-ups carry most of the votes: Fable 5.2 over GPT-6.2 Astra (8 reviewers to 1), Fable over Gemini 4 Carbon (7 to 0), Carbon over Astra (3 to 1) and Fable over GPT-6.2 Sol (3 to 0). Any benchmark that puts Fable first and Astra below Carbon scores well almost by default.
Seven reviewers, seven different months
The vibe score is an average, and the people behind it don't agree. Gemini 4 Carbon was the top or joint-top model of the month for Bijan Bowen and Ethan Mollick. Bijan: "I'm extremely impressed with this model" (YouTube, 10 November). For Theo it was near the bottom: "This model is slow and it sucks to use cuz it's slow." (YouTube, 11 November) They test different things: Bijan builds 3D scenes from one prompt, Theo runs coding agents all day.

Taken one at a time, the seven who rated at least five models look like this.
- Theo rewards speed and feel. Neon is his top model; Carbon and Hydrogen are at the bottom. His is the one ranking where our factual CatalogBench is among the closest (+0.48 across 8 models).
- Ethan Mollick sits closest to the big leaderboards: +0.60 against Vals, Terminal-Bench 4.1 and Arena Agent, and the highest of the seven against the Artificial Analysis index (+0.30). On GPT-6.2: "Fable and opus and carbon all have more ambition, seem to have better anticipation of what your needs are, and generally have better taste on research or innovation topics" (X, 17 November).
- Simon Willison runs one test many times. Ten of his 12 takes are pelican drawings, and his ranking is closest to DesignArena (+0.63 across 5 models).
- Bijan Bowen covered the most models, ten. Closest: LiveBench coding (+0.69).
- ThePrimeagen runs against nearly every leaderboard, and says so: "I know I am a bit of a contrarian... I like sol 6.2" (X, 19 November).
- Matthew Berman rated Fable 5.2 and DeepSeek Flash 1017 high and the rest level. AICodeKing scores his own coding suite in a narrow band. Neither lines up strongly with any benchmark.

It is tempting to read that list as "Willison is a DesignArena person". The data doesn't support it. With five to ten models per reviewer and about 25 benchmarks tried against each, a high best match turns up by luck. When we shuffled each reviewer's scores at random across the same models, the best match was typically +0.63 to +0.76, and 30% to 88% of shuffles matched or beat the real one. The profiles describe each reviewer's taste this month. They don't show that anyone follows a benchmark.

Where our own benchmarks sit
Mid-table on head-to-heads, lower half on the monthly ranking. The furthest from reviewers is the sales brief of CatalogBench, which asks for persuasive product copy. GPT-6.2 Sol and Astra are publish-ready on it 80% and 75% of the time, against 66% for Fable and 31% for Carbon, close to the reverse of the month's most common reviewer calls. It agreed with 40% of votes.
The two are measuring different work. Our benchmarks cover product listings, slide decks, survey analysis and ad budgets. Of the 250 hands-on quotes, 20 were about writing and 3 about research, so the reviews hold almost nothing for a business-task benchmark to agree or disagree with. For a specific job, a benchmark built on that job is still the more reliable signal. A review tells you what a model was like to use that week, for the things that reviewer builds.
The Neon match that started this is one data point. Neon is first on our factual listings and first with reviewers, but 8th of the month's models on our Score, largely because its survey analyses often ran out of our 32,000-token output budget at its default medium thinking (details in the review). Third-party leaderboards place it high as well: 3rd of the month's models on Artificial Analysis, 2nd on Vals. Across all 11 models, factual CatalogBench against the vibe score is −0.04.
What a month of takes is made of
The sample has a shape. 93 of the 250 quotes came from people who had the model before launch. Takes cluster within hours and cite each other, the GPT-6.2 launch most of all. Many of these reviewers use Claude daily, so Opus 5.5 is the usual yardstick. Opinions move: Theo's early posts on Fable 5.2 complained of mistakes, and by 16 November he was praising it. Mistral Large 4 and MiniMax M3.1 Flash got no hands-on take at all from this panel.
Limits
One month, 11 rankable models, four of them on only three or four reviewers. Neon and GPT-6.2 had one or two days of takes when we collected; Fable had ten. With so few models, one or two can move a correlation a lot: using the second rater's scores alone shifts some benchmarks by 0.3 to 0.5. That is too little to rank benchmarks against each other, and we make no claim of statistical significance. Effort settings also differ: our runs use provider defaults, Artificial Analysis and Arena mostly the top setting.
We will run it again when Neon and GPT-6.2 have two weeks of reactions.
Your prompt is not on any of these charts
Neither a leaderboard nor a reviewer has run your prompt. You can test it across models on your own cases and see which answers pass.
Method
- Reviewers (27; Every's team counts as one): Theo, Bijan Bowen, Peter Yang, Matthew Berman, Every, Simon Willison, Matt Shumer, AICodeKing, Ethan Mollick, Prompt Engineering, ThePrimeagen, Flavio Adamo, Wes Roth, Zvi Mowshowitz, Chetaslua, Pankaj Kumar, Sam Witteveen, Pietro Schirano, Paweł Huryn, Claire Vo, WorldofAI, McKay Wrigley, Fireship, AI Explained, The Neuron, Riley Brown and Deedy Das. None works at an AI lab today; Pietro Schirano previously worked at Anthropic, and Deedy Das invests in Anthropic and Inception. Several had early access. Ties and early access are recorded per quote in the data file.
- Quotes: verbatim. Video quotes come from auto-captions and keep their spellings ("62 Astra" is GPT-6.2 Astra, "Opus 55" is Claude Opus 5.5). Reactions to benchmark charts, hearsay and announcements are left out.
- Scoring: both raters were Claude Opus 5.5 (
claude-opus-5-5). The first scored each quote while collecting it. The second saw only the words, with no first-rater scores and no benchmark figures. 79% of scores were identical and none flipped between positive and negative (rank correlation 0.93). The vibe score uses the mean of both. - Benchmarks: ours are from our 19 November run, with CatalogBench as its publish-ready rate. Third-party figures are as published on 18 and 19 November; three Artificial Analysis values are second-hand (GPT-6.2 Astra via press, GPT-6.1 Terra from our reading of its page on 5 November, MiniMax M3.1 Flash via an aggregator). Terminal-Bench scores an agent and a model together.
- Statistics: Spearman rank correlation across the models each benchmark shares with the vibe ranking. DeepSeek V4.1 Flash 1017 came out three days before the window and is included as a borderline case.
- Data: every quote with its link, date and both ratings, plus the correlation and head-to-head tables: 2026-11-benchmarks-vs-vibes.json. Today's benchmark figures are on the leaderboard.
Quick answers
Which AI benchmark is most accurate?
It depends on the job. In our comparison for 20 October to 19 November 2026, no benchmark ranked 11 new models the way 27 well-known reviewers did. On direct 'A beat B' calls, agent benchmarks (LiveBench long agentic, Terminal-Bench 4.1, Arena Agent) agreed with 83% to 88% of reviewer votes, because reviewers mostly test coding agents. For a specific task, a benchmark built on that task is the better guide.
Do AI benchmarks match real-world use?
Partly. Across one month of launches, rank correlations between 15 benchmark measures and reviewers' average verdicts ran from −0.55 to +0.17, which is no detectable relationship. On head-to-head calls the agreement was much higher, 80% or more for agent benchmarks and the Artificial Analysis index. Reviewers and benchmarks often measure different work.
What is vibe checking an AI model?
Using a new model on your own tasks for a few hours or days and reporting how it felt: speed, readability, whether it finishes the job, whether it is good for its price. It is first-hand and fast, but it reflects one person's work. In our sample the seven most prolific reviewers disagreed with each other as much as with any benchmark.
Why did reviewers dislike GPT-6.2 when it scores so well on benchmarks?
GPT-6.2 Astra and Sol score 62 and 57 on the Artificial Analysis index and place 2nd and 3rd of the month's models on our Spring Prompt Score, yet came last with reviewers. The complaints were about use, not capability: hard to steer, stops early, hard to read. Not everyone agreed: ThePrimeagen rated Sol highly. The takes cover only the first one or two days after launch.
Should I pick a model from benchmarks or from reviews?
Use a benchmark that measures your kind of task to make a shortlist, read reviewers for what a model is like to work with, then run your own prompt on your own cases. Neither a leaderboard nor a reviewer has tested your prompt.
Model pages: Gemini 4 Neon · GPT-6.2 Astra · GPT-6.2 Sol · Claude Fable 5.2 · Gemini 4 Carbon · Gemini 4 Hydrogen · Muse Ember · DeepSeek-V4.1-Flash-1017 · Kimi K3.1 Pro · Qwen4-Max · GPT-6.1 Terra