Research

No Benchmark Ranked This Month's AI Models the Way Reviewers Did. Ours Included.

Headline 'No leaderboard ranked the month the way reviewers did' beside a chart of 11 new AI models ranked three ways. GPT-6.2 Astra and Sol sit near the top of the Artificial Analysis index and the Spring Prompt Score and at the bottom with reviewers. Gemini 4 Neon is first with reviewers.
250 hands-on takes from 27 reviewers on 11 models released 20 October to 19 November 2026, against 28 benchmark measures.

The day after Gemini 4 Neon came out, Theo, who runs his own coding test suite, posted this: "Still waiting on numbers but honestly? I don't care. Astra showed me that my benchmarks suck. Neon is super pleasant to work with, it unblocks itself, makes good decisions, writes good code, and writes english SO well." (Theo, X, 19 November)

That same day our own run put Neon first of 35 models on the factual brief of CatalogBench, our product-listing benchmark. A reviewer and a benchmark agreed. We wanted to know whether that happens often, so we checked every model launched in the past month.

It doesn't. No leaderboard ranked the month's models the way reviewers did, and ours are not an exception. On direct "A beat B" calls the picture is better: agent benchmarks agreed with 83% to 88% of reviewer votes, the Artificial Analysis index with 80%, and most of our own business-task benchmarks with 63% to 71%. Reviewer by reviewer, each one's taste lines up best with a different benchmark, and none of those matches is stronger than chance would produce.

Data note. Models released 20 October to 19 November 2026. We kept 250 hands-on quotes from 27 reviewers on YouTube, blogs, newsletters and X, among them Theo, Simon Willison, Matt Shumer, Every and Ethan Mollick. Each quote was scored from −2 (strongly negative) to +2 (strongly positive) by two raters, both Claude Opus 5.5, the second blind to the first. We compared the result with 28 benchmark measures, ours and third parties'. One month and 11 models is a small sample. Method and data.

Top of the leaderboards, bottom with reviewers

A model's vibe score is the average of those −2 to +2 scores. We average within each reviewer first, so someone who posted ten times about one model counts once, and we rank only models that at least three reviewers used.

Three columns ranking 11 new models: Artificial Analysis index, reviewers, and Spring Prompt Score. GPT-6.2 Astra is 2nd on both leaderboards and 10th with reviewers. GPT-6.2 Sol is 5th and 3rd on the leaderboards and 11th with reviewers. Gemini 4 Neon is 3rd on Artificial Analysis, 1st with reviewers and 8th on the Spring Prompt Score. The lines between columns cross heavily.
Rank of each model on the Artificial Analysis Intelligence Index, on the reviewers' vibe score and on the Spring Prompt Score. 250 hands-on takes from 27 reviewers, 20 October to 19 November 2026; models with at least three reviewers.

We measured how closely each benchmark's order follows the reviewers' with a rank correlation: +1 means the same order, 0 means no relation. Across the benchmarks the values ran from −0.55 to +0.17: no detectable relationship. Ours sit in the lower half of that range, with the Spring Prompt Score at −0.35 and DeckBench at −0.55. Three things explain most of it.

GPT-6.2 tops the charts and the complaints. Astra and Sol score 62 and 57 on the Artificial Analysis index and come 2nd and 3rd on our Score. With reviewers they are last. The most-viewed take we collected was Matt Shumer's on the pair: "I had a terrible experience testing them. They were essentially unusable." (X, 17 November) Every's team on Astra: "This is not a great model. Across everyone on our team who tried it, no one loved it." (YouTube, 17 November)

Theo's video shows the split inside one person. It opens with "In my own personal benches, 62 Astra shows dominance across the board". Half an hour in: "I do not like 62 Astra. I would not choose this over Opus 55 for any of my work as a dev." (YouTube, 18 November; later in the same video) His tests and his working day disagreed. (Our own figures are in the GPT-6.2 review.)

Cheap models are judged against their price. Simon Willison on Muse Ember: "The Ember pelicans are comparable in quality to those I got from Muse Wonder, which is 40x the price of Muse Ember." (blog, 10 November) Matthew Berman called DeepSeek Flash 1017 "potentially the best overall model on the planet right now" (YouTube, 28 October). Both models are in the reviewers' top three and score 36 and 33 on the Artificial Analysis index, against 65 for Claude Fable 5.2. A capability leaderboard doesn't ask whether a model is good for what it costs. Reviewers often do.

The top of the ranking is thin. Neon leads on four reviewers and one day of posts. Claude Fable 5.2 and GPT-6.2 Astra rest on 21 and 17 reviewers.

Reviewers' vibe score, −2 to +2, hands-on takes only, 20 October to 19 November 2026.
ModelVibe scoreReviewers
Gemini 4 Neon+1.384
Muse Ember+1.254
DeepSeek V4.1 Flash 1017+1.003
Claude Fable 5.2+0.8821
Gemini 4 Carbon+0.8514
GPT-6.1 Terra+0.735
Gemini 4 Hydrogen+0.156
Kimi K3.1 Pro+0.087
Qwen4-Max+0.043
GPT-6.2 Astra−0.0117
GPT-6.2 Sol−0.069

On "A beat B", agent benchmarks agree most

A narrower test: when a reviewer says from their own use that model A beat model B, does the benchmark also put A above B? We found 149 such statements and counted each reviewer once per pair of models.

Horizontal bars showing the share of reviewer head-to-head votes each benchmark agrees with. LiveBench long agentic 88%, Terminal-Bench 4.1 86%, Arena Agent 83%, LiveBench design 82%, Artificial Analysis index 80%, Vals Index 80%, BulletBench Blitz 77%, DesignArena 72%, ROASBench 71%, Spring Prompt leaderboard 71%, Spring Prompt Score 69%, CatalogBench factual 69%, SurveyBench 66%, Arena WebDev 66%, DeckBench 63%, CatalogBench sales 40%, LiveBench coding 33%, LiveBench reasoning 12%. A dashed line marks 50%.
Share of reviewer "A beat B" votes each benchmark ranks the same way. One vote per reviewer per pair, 29 to 35 votes per benchmark. Spring Prompt's benchmarks in green; the dashed line is a coin flip.

Agent benchmarks come out on top because they test the work these reviewers do: 67 of the 250 quotes were about coding or agents, and another 59 were front-end, 3D and game demos. The fit is not automatic, though. LiveBench's coding category agreed with only 33% of votes.

The bars are not a league table. Four match-ups carry most of the votes: Fable 5.2 over GPT-6.2 Astra (8 reviewers to 1), Fable over Gemini 4 Carbon (7 to 0), Carbon over Astra (3 to 1) and Fable over GPT-6.2 Sol (3 to 0). Any benchmark that puts Fable first and Astra below Carbon scores well almost by default.

Seven reviewers, seven different months

The vibe score is an average, and the people behind it don't agree. Gemini 4 Carbon was the top or joint-top model of the month for Bijan Bowen and Ethan Mollick. Bijan: "I'm extremely impressed with this model" (YouTube, 10 November). For Theo it was near the bottom: "This model is slow and it sucks to use cuz it's slow." (YouTube, 11 November) They test different things: Bijan builds 3D scenes from one prompt, Theo runs coding agents all day.

Seven rows, one per reviewer, each a strip from −2 to +2 with a dot per model coloured by lab. Bijan Bowen: Carbon, Hy4 and Muse Ember highest, Kimi K3.1 Pro lowest, closest benchmark LiveBench coding. Theo: Neon highest, Hydrogen lowest, closest CatalogBench factual. Simon Willison: Kimi K3.1 Pro highest, GPT-6.2 and Hydrogen lowest, closest DesignArena. AICodeKing: Carbon and GPT-6.2 Sol highest, Astra lowest, closest DesignArena. Matthew Berman: Fable 5.2 highest, closest DeckBench. ThePrimeagen: GPT-6.2 Sol and Muse Ember highest, Fable 5.2 lowest, closest CatalogBench sales. Ethan Mollick: Carbon highest, GPT-6.2 Sol lowest, closest agent benchmarks.
Where each reviewer placed the month's models, by lab, and the benchmark their ranking follows most closely. The seven reviewers who rated at least five of the month's models.

Taken one at a time, the seven who rated at least five models look like this.

  • Theo rewards speed and feel. Neon is his top model; Carbon and Hydrogen are at the bottom. His is the one ranking where our factual CatalogBench is among the closest (+0.48 across 8 models).
  • Ethan Mollick sits closest to the big leaderboards: +0.60 against Vals, Terminal-Bench 4.1 and Arena Agent, and the highest of the seven against the Artificial Analysis index (+0.30). On GPT-6.2: "Fable and opus and carbon all have more ambition, seem to have better anticipation of what your needs are, and generally have better taste on research or innovation topics" (X, 17 November).
  • Simon Willison runs one test many times. Ten of his 12 takes are pelican drawings, and his ranking is closest to DesignArena (+0.63 across 5 models).
  • Bijan Bowen covered the most models, ten. Closest: LiveBench coding (+0.69).
  • ThePrimeagen runs against nearly every leaderboard, and says so: "I know I am a bit of a contrarian... I like sol 6.2" (X, 19 November).
  • Matthew Berman rated Fable 5.2 and DeepSeek Flash 1017 high and the rest level. AICodeKing scores his own coding suite in a narrow band. Neither lines up strongly with any benchmark.
Scatter chart with seven circles, one per reviewer. The horizontal axis is each reviewer's average correlation with the big general leaderboards, the vertical axis their average correlation with Spring Prompt's benchmarks, both from −0.6 to +0.6. All seven sit near the middle. Theo and Matthew Berman are slightly above zero on Spring Prompt's benchmarks, Ethan Mollick is furthest right on the big leaderboards, ThePrimeagen furthest left. No one is in the top-right area labelled 'Agrees with both'.
Each reviewer's average rank correlation with the big general leaderboards (Artificial Analysis, Vals, Arena Agent, LiveBench) against their average with Spring Prompt's six measures. Circle size is the number of models rated, five to ten.

It is tempting to read that list as "Willison is a DesignArena person". The data doesn't support it. With five to ten models per reviewer and about 25 benchmarks tried against each, a high best match turns up by luck. When we shuffled each reviewer's scores at random across the same models, the best match was typically +0.63 to +0.76, and 30% to 88% of shuffles matched or beat the real one. The profiles describe each reviewer's taste this month. They don't show that anyone follows a benchmark.

Grid of rank correlations with seven reviewers as rows and 16 benchmarks as columns, grouped as agents and coding, design, general indexes and Spring Prompt. Green cells mean the same order, amber the opposite. Each reviewer's closest match is ringed: Bijan Bowen LiveBench coding +0.69, Theo CatalogBench factual +0.48, Simon Willison DesignArena +0.63, AICodeKing DesignArena +0.39, Matthew Berman DeckBench +0.39, ThePrimeagen CatalogBench sales +0.36, Ethan Mollick +0.60 on four benchmarks. No column is green all the way down.
Rank correlation between each reviewer's scores and each benchmark, across the models both cover. No benchmark is green for every reviewer, and no reviewer is green across the board.

Where our own benchmarks sit

Mid-table on head-to-heads, lower half on the monthly ranking. The furthest from reviewers is the sales brief of CatalogBench, which asks for persuasive product copy. GPT-6.2 Sol and Astra are publish-ready on it 80% and 75% of the time, against 66% for Fable and 31% for Carbon, close to the reverse of the month's most common reviewer calls. It agreed with 40% of votes.

The two are measuring different work. Our benchmarks cover product listings, slide decks, survey analysis and ad budgets. Of the 250 hands-on quotes, 20 were about writing and 3 about research, so the reviews hold almost nothing for a business-task benchmark to agree or disagree with. For a specific job, a benchmark built on that job is still the more reliable signal. A review tells you what a model was like to use that week, for the things that reviewer builds.

The Neon match that started this is one data point. Neon is first on our factual listings and first with reviewers, but 8th of the month's models on our Score, largely because its survey analyses often ran out of our 32,000-token output budget at its default medium thinking (details in the review). Third-party leaderboards place it high as well: 3rd of the month's models on Artificial Analysis, 2nd on Vals. Across all 11 models, factual CatalogBench against the vibe score is −0.04.

What a month of takes is made of

The sample has a shape. 93 of the 250 quotes came from people who had the model before launch. Takes cluster within hours and cite each other, the GPT-6.2 launch most of all. Many of these reviewers use Claude daily, so Opus 5.5 is the usual yardstick. Opinions move: Theo's early posts on Fable 5.2 complained of mistakes, and by 16 November he was praising it. Mistral Large 4 and MiniMax M3.1 Flash got no hands-on take at all from this panel.

Limits

One month, 11 rankable models, four of them on only three or four reviewers. Neon and GPT-6.2 had one or two days of takes when we collected; Fable had ten. With so few models, one or two can move a correlation a lot: using the second rater's scores alone shifts some benchmarks by 0.3 to 0.5. That is too little to rank benchmarks against each other, and we make no claim of statistical significance. Effort settings also differ: our runs use provider defaults, Artificial Analysis and Arena mostly the top setting.

We will run it again when Neon and GPT-6.2 have two weeks of reactions.

Your prompt is not on any of these charts

Neither a leaderboard nor a reviewer has run your prompt. You can test it across models on your own cases and see which answers pass.

Method

  • Reviewers (27; Every's team counts as one): Theo, Bijan Bowen, Peter Yang, Matthew Berman, Every, Simon Willison, Matt Shumer, AICodeKing, Ethan Mollick, Prompt Engineering, ThePrimeagen, Flavio Adamo, Wes Roth, Zvi Mowshowitz, Chetaslua, Pankaj Kumar, Sam Witteveen, Pietro Schirano, Paweł Huryn, Claire Vo, WorldofAI, McKay Wrigley, Fireship, AI Explained, The Neuron, Riley Brown and Deedy Das. None works at an AI lab today; Pietro Schirano previously worked at Anthropic, and Deedy Das invests in Anthropic and Inception. Several had early access. Ties and early access are recorded per quote in the data file.
  • Quotes: verbatim. Video quotes come from auto-captions and keep their spellings ("62 Astra" is GPT-6.2 Astra, "Opus 55" is Claude Opus 5.5). Reactions to benchmark charts, hearsay and announcements are left out.
  • Scoring: both raters were Claude Opus 5.5 (claude-opus-5-5). The first scored each quote while collecting it. The second saw only the words, with no first-rater scores and no benchmark figures. 79% of scores were identical and none flipped between positive and negative (rank correlation 0.93). The vibe score uses the mean of both.
  • Benchmarks: ours are from our 19 November run, with CatalogBench as its publish-ready rate. Third-party figures are as published on 18 and 19 November; three Artificial Analysis values are second-hand (GPT-6.2 Astra via press, GPT-6.1 Terra from our reading of its page on 5 November, MiniMax M3.1 Flash via an aggregator). Terminal-Bench scores an agent and a model together.
  • Statistics: Spearman rank correlation across the models each benchmark shares with the vibe ranking. DeepSeek V4.1 Flash 1017 came out three days before the window and is included as a borderline case.
  • Data: every quote with its link, date and both ratings, plus the correlation and head-to-head tables: 2026-11-benchmarks-vs-vibes.json. Today's benchmark figures are on the leaderboard.

Quick answers

Which AI benchmark is most accurate?

It depends on the job. In our comparison for 20 October to 19 November 2026, no benchmark ranked 11 new models the way 27 well-known reviewers did. On direct 'A beat B' calls, agent benchmarks (LiveBench long agentic, Terminal-Bench 4.1, Arena Agent) agreed with 83% to 88% of reviewer votes, because reviewers mostly test coding agents. For a specific task, a benchmark built on that task is the better guide.

Do AI benchmarks match real-world use?

Partly. Across one month of launches, rank correlations between 15 benchmark measures and reviewers' average verdicts ran from −0.55 to +0.17, which is no detectable relationship. On head-to-head calls the agreement was much higher, 80% or more for agent benchmarks and the Artificial Analysis index. Reviewers and benchmarks often measure different work.

What is vibe checking an AI model?

Using a new model on your own tasks for a few hours or days and reporting how it felt: speed, readability, whether it finishes the job, whether it is good for its price. It is first-hand and fast, but it reflects one person's work. In our sample the seven most prolific reviewers disagreed with each other as much as with any benchmark.

Why did reviewers dislike GPT-6.2 when it scores so well on benchmarks?

GPT-6.2 Astra and Sol score 62 and 57 on the Artificial Analysis index and place 2nd and 3rd of the month's models on our Spring Prompt Score, yet came last with reviewers. The complaints were about use, not capability: hard to steer, stops early, hard to read. Not everyone agreed: ThePrimeagen rated Sol highly. The takes cover only the first one or two days after launch.

Should I pick a model from benchmarks or from reviews?

Use a benchmark that measures your kind of task to make a shortlist, read reviewers for what a model is like to work with, then run your own prompt on your own cases. Neither a leaderboard nor a reviewer has tested your prompt.

Model pages: Gemini 4 Neon · GPT-6.2 Astra · GPT-6.2 Sol · Claude Fable 5.2 · Gemini 4 Carbon · Gemini 4 Hydrogen · Muse Ember · DeepSeek-V4.1-Flash-1017 · Kimi K3.1 Pro · Qwen4-Max · GPT-6.1 Terra

More research
All posts →