Ellis Crosby
Posts by Ellis 51
-
Guides · · 9 min readNo benchmark ranked this month's AI models the way reviewers did. Ours included.
We scored 250 hands-on takes from 27 well-known AI reviewers on a month of model launches and set them against 28 benchmark measures. No leaderboard put the models in the reviewers' order. On direct 'A beat B' calls, agent benchmarks agreed with reviewers most, and our own sat mid-table.
-
Model launches · · 8 min readGemini 4 neon review: The most reliable product listings we've measured
Google's Gemini 4 Neon wrote the most reliable factual product listings of 35 models on our benchmarks, 78.6% publish-ready in every run, ahead of every OpenAI and Anthropic model. It sits between Hydrogen and Carbon on speed for listings, and was weaker on sales copy, slide decks and survey analysis.
-
Guides · · 6 min readBraintrust alternatives: who should stay, and where a prompt test fits
Braintrust is a hosted platform for tracing, evals and datasets, with a free Starter plan and a $249 a month Pro plan. What it does better than a browser prompt test, who should stay, and how to move a dataset across.
-
Model launches · · 10 min readGPT-6.2 review: Sol writes the better listings and decks, Astra the better analysis, at four times the price
We ran OpenAI's GPT-6.2 Sol and GPT-6.2 Astra through our five business benchmarks. Sol wrote the most reliable product listings we have measured and better decks than Astra; Astra was ahead on survey analysis and ad budgets, at four times the token price.
-
Guides · · 6 min readLangfuse alternatives: who should stay, and where a prompt test fits
Langfuse joined ClickHouse in January 2026 and its core is still MIT-licensed and self-hostable. What it does better than a browser prompt test, who should stay, and how to bring a Langfuse dataset into a Prompt test.
-
Guides · · 6 min readLangSmith alternatives: who should stay, and where a prompt test fits
LangSmith is LangChain's hosted platform for tracing and evaluating LLM apps, priced per seat and per trace, with self-hosting only on Enterprise. What it does better, who should stay, and how to bring a LangSmith dataset into a Prompt test.
-
Guides · · 6 min readPromptfoo alternatives: when to stay, and when a browser test fits
Promptfoo agreed to be acquired by OpenAI in March 2026 and is still MIT-licensed. What it does well, who should stay with it, and where a browser-based Prompt test fits instead or alongside.
-
Model launches · · 3 min readGemini 4 hydrogen got faster: We reran our speed chess test
Three days after launch, Gemini 4 Hydrogen answers our speed chess moves 19 to 35% faster than it did on 13 November. It no longer loses every 10-second game on time, and its 60-second score rose from 557 to 889, 4th of 71 configurations.
-
Model launches · · 9 min readQwen4-max review: Top of our slide deck ratings, weak on product listings
Alibaba's Qwen4-Max rated first of 32 models on our slide deck benchmark and fifth on survey analysis, at $1.50/$4.50 per million tokens. It was weak on product listings and lost almost every game of speed chess on time.
-
Model launches · · 8 min readGemini 4 carbon review: Slower than argon on our business tests
Carbon improved our survey and ad-budget scores, but reliable product listings fell behind Argon. It also took longer and cost more per listing, deck and survey task.
-
Model launches · · 9 min readClaude Fable 5.2 review: First on our survey and ad budget tests, and one of the dearest models to run
Claude Fable 5.2 came first on our survey analysis and ad budget benchmarks and third on product listings and slide decks, a large step up from Fable 5.1. It also cost about ten times as much per task as GPT-6.1 Sol.
-
Model launches · · 9 min readMuse ember review: GPT-6 luna's price, better at analysis, and no Gemini Flash against a clock
Meta's Muse Ember costs the same per token as GPT-6 Luna and Claude Haiku 5.5. On our business benchmarks it landed beside Luna, clearly ahead on survey analysis and behind on sales copy and decks, and it cost up to four times as much per job because it reasons at maximum effort by default.
-
Guides · · 5 min readHelicone alternatives: where things stand and how to keep your data
Mintlify announced it was acquiring Helicone in March 2026, and Helicone moved to maintenance mode. Its site now says it is no longer part of Mintlify. What we know, how to export your logs, and your options.
-
Guides · · 5 min readHumanloop alternatives: where to test prompts after the shutdown
Humanloop's platform closed on 8 September 2025, after its team announced it was joining Anthropic. What happened, what its export contains, and where to run your prompt tests now.
-
Guides · · 5 min readOpenAI Evals is shutting down: your options before 30 November
OpenAI's Evals platform shuts on 30 November 2026. What's closing, how to keep your data, and the routes from here, including importing your evals into a browser tool.
-
Guides · · 9 min readWe switched a support-reply prompt from Claude Sonnet 4.5 to Sonnet 5.5
Claude Sonnet 4.5 retires on 30 November. We ran one shop's customer-email prompt on it and on Sonnet 5.5, its named replacement, across 12 emails. Every reply is below, with what changed in length, tone, promises and cost.
-
Model launches · · 8 min readGPT-6.1 terra review: Close to the top of our business tests at half the price of Sol
GPT-6.1 Terra is 2nd on our provisional leaderboard at $1/$5. It set the best ad budget score we have measured and did solid business work, but GPT-6.1 Sol scored higher on decks, surveys and factual listings, and its rank rests on our own benchmarks alone.
-
Model launches · · 5 min readKimi K3.1 Pro first look: The best open-weight model on our board, and very slow
Kimi K3.1 Pro is the top open-weight model on our provisional leaderboard, 12th overall, with a strong ad budget score. It is also very slow, and on 10 of 12 survey tasks it thought until it ran out of room.
-
Guides · · 6 min readBest AI for product descriptions: Accuracy, cost and real examples
Compare AI models for product descriptions using publish-ready listings, unsupported claims and measured generation cost, with real outputs from CatalogBench.
-
Guides · · 6 min readHow to test AI hallucinations in business content
A practical method for testing unsupported AI claims, using actual product listings, business slides and survey findings from our public benchmarks.
-
Model launches · · 5 min readMercury 3 first look: Very fast with thinking off, too slow with it on
We ran Inception's Mercury 3 diffusion model through our benchmarks on launch night. With thinking off it played speed chess at 0.4 seconds a move without losing on time; with thinking on it lost on time at every clock, and its business results were near the bottom of our tables.
-
Model launches · · 8 min readDeepSeek-V4.1-Flash-1017 review: Better at analysis, much worse at sticking to the product data
We ran DeepSeek's 17 October Flash snapshot through six benchmarks two days after launch. Its surveys and slide decks improved a lot, but its product listings were reliably publishable half as often, because it added three times as many claims the data didn't support.
-
Model launches · · 6 min readMiniMax M3.1 Flash first look: Careful with product data, unreliable on long jobs
We ran MiniMax M3.1 Flash through six benchmarks on launch day, on OpenRouter's free route. It wrote some of the most reliable product listings of any cheap model, then ran out of room thinking on half our survey tasks.
-
Benchmarks · · 4 min readAI Checkpoint: Argon matches Opus, until it sells
Week ending 16 October: Gemini 4 Argon opens up, Muse Wonder arrives, Muse Spark Open ships open weights, how long models take per task, and a three-sentence fix for models that call real news fake.
-
Model launches · · 8 min readGemini 4 argon review: Opus-level product data at a quarter of the cost, until you ask it to sell
We ran Gemini 4 Argon through six benchmarks on launch day. It matched Claude Opus 5.5 on product data at a quarter of the cost, then added five times as many unsupported claims when asked to sell.
-
Model launches · · 6 min readClaude Opus 5.5 vs Muse wonder full: Eight crafts, and Opus still wins most
We asked Claude Opus 5.5, Muse Wonder and Muse Wonder Full to make eight crafts in a web page, from a thrown bowl to an ice swan. Full closed part of the gap that standard Wonder left, with an agent and second chances. Opus still made the more convincing object in most of them.
-
Model launches · · 7 min readMuse wonder review: Good analysis, work that still needs checking
We tested standard Muse Wonder on six evaluations. Survey analysis was strong; product copy, slide layout and decisions against a clock still needed work. Full is outside this API comparison.
-
Guides · · 9 min readWe told 11 AI models what happened in 2026. Most of them thought we were making it up.
We texted 11 models real 2026 news, from the Knicks title to Allbirds becoming an AI company. Given only today's date, most hedged, one called nearly all of it fake, and one claimed to have searched sources it had no way to reach.
-
Guides · · 9 min readThe anatomy skill for Claude Code: A practical guide, tested on everyday objects and seven models
Anatomy is a Claude Code skill that explains an idea by building it as an interactive isometric machine. How to install it, how to prompt it, what worked and what didn't, with 24 everyday objects you can play with, from a toilet flush to a lift's safety brake, and the same prompt run through seven AI models.
-
Benchmarks · · 4 min readAI Checkpoint: Sol’s new fast lane
Week ending 9 October: GPT-6.1 Sol Ultrafast, Haiku 5.5, Mistral Large 4, new AI interfaces, SurveyBench and a practical lesson in image tiling.
-
Guides · · 7 min readWe raced AI models to find one small thing in a huge picture. Cutting it up more than doubled what they found.
We timed LLMs and decision models finding one small target in 79 full-size scenes, from Where's Waldo pages to our own puzzles. Gemini 3.5 Flash Lite found the most; a decision model that can only answer yes or no found a target for a tenth of a cent.
-
Model launches · · 8 min readClaude Haiku 5.5 is far better than Haiku 4.5 at a tenth of the price. GPT-6 Luna still beats it at business work
We ran Claude Haiku 5.5 through our business benchmarks and 18 playground tests on launch day. It is a big step up from Haiku 4.5 and very cheap, but GPT-6 Luna, at the same price, beat it on listings, decks and ad planning.
-
Model launches · · 7 min readMistral Large 4 is fast, cheap and Europe's best model. On business work, it still makes things up.
We ran Mistral Large 4 through our four business benchmarks on launch day. It is quick, inexpensive and careful with structured data, but it added unsupported claims to most product listings and finished last on slide decks.
-
Model launches · · 5 min readGemini 3.6 Flash wins the minute. Gemini 3.5 Flash-Lite wins the second.
Google's two new fast models split BulletBench's hardest latency tests: Gemini 3.6 Flash topped the 60-second board, while Gemini 3.5 Flash-Lite nearly won Lightning without losing a single game on time.
-
Benchmarks · · 4 min readAI is now a better football pundit than the pundit. It still lost to one line of code.
We had six frontier AI models predict all 32 knockout matches of the 2026 World Cup, blind. They out-called a top BBC pundit. Then a one-line rule, always back the higher-ranked team, beat every single one of them on…
-
Benchmarks · · 3 min readSix models, one mind: What the World Cup revealed about how AIs actually think
Across 34 World Cup fixtures, six frontier AI models from five different labs picked the same winner 31 times. Then we gave them memory of the tournament and they changed zero of 96 picks. The herd is real, and it has…
-
Benchmarks · · 3 min readThe AI that won our World Cup benchmark didn't predict best. It read the rules.
GLM-5.2 won World Cup Bench by two points while spending the fewest reasoning tokens and posting an ordinary hit rate. It found the one free lunch in our scoring rules and ate it 34 times. A story about Goodhart's law…
-
Guides · · 5 min readHow to continue a Claude Code conversation in Codex after hitting your usage limit
Ran out of Claude Code allowance halfway through a task? Here is how to move the conversation into Codex and continue without losing the decisions behind the code.
-
Benchmarks · · 5 min readGPT-5.6 Luna was meant to be fast. It still lost to the clock.
Luna arrived as the fastest GPT-5.6. Across 96 BulletBench games, it lost 71 on time. The result exposes the model the market still has not built: genuinely smart, sub-second and predictable.
-
Benchmarks · · 7 min readBulletBench: We made 23 AI models play speed chess, and the clock was the judge
Every leaderboard measures how smart a model is. None measure how smart it is per second. So we built one: AI models play speed chess on a real clock, and slow genius loses on time.
-
Benchmarks · · 3 min readSonnet 5 gave the sharpest marketing analysis we've ever tested. It also lost money every single time.
A ROASBench case study in why fluent reasoning isn't the same as good decisions, and why for this model more thinking made it worse.
-
Benchmarks · · 4 min readClaude vs Gemini vs GPT in a 12-month marketing simulation
ROAS Bench is one of the clearest examples of where frontier models diverge in practice: not on prose quality, but on economic compounding.
-
Benchmarks · · 6 min readHow we built ROASBench to feel like real growth work
ROASBench was built from operator experience, not benchmark theatre. We designed it to feel like real performance marketing: stateful, constrained, path-dependent, and economically unforgiving.
-
Guides · · 10 min readLiteLLM alternatives for 2026
Looking for a LiteLLM alternative? First decide whether you need a Python library for switching LLM providers or an AI gateway for routing, fallbacks and control. Here are the options for each.
-
Benchmarks · · 4 min readWhy most LLMs still can't run growth
ROAS Bench shows that growth is not a copywriting problem. It is a compounding systems problem, and most models still break the system faster than they improve it.
-
Benchmarks · · 3 min readThe AI marketing benchmark that punishes plausible-sounding strategy
Most LLMs can sound like a competent growth lead for one turn. ROAS Bench is interesting because it makes them live with the consequences for twelve months.
-
Benchmarks · · 8 min readGemini Embedding 2 just launched - so we benchmarked it
Google launched gemini-embedding-2-preview on 10 March 2026 as its first multimodal embedding model, with one shared embedding space for text, images, video, audio, and PDFs. Google specifically positions it for…
-
Benchmarks · · 4 min readThe great AI gifting showdown: Which model should you trust for Christmas shopping?
It’s that time of year again. You’re out and about, the clock is ticking, and you still haven't found the perfect gift for your partner, your roommate, or that difficult-to-shop-for in-law. Naturally, many of us are…
-
Model launches · · 3 min readGoogle Gemini 3 review: The benchmarks actually match the hype
So, on Tuesday Google launched Gemini 3. The hype was massive leading up to this, and honestly? It is justified. It is really, really good. Trying to explain how good is difficult without getting bogged down in…
-
Model launches · · 4 min readGPT-5.1 first look: Smarter, warmer… but not a breakthrough
2025’s flagship model season kicked off on 12 November with the unexpected arrival of GPT-5.1, with OpenAI getting their release out before Gemini 3. While we’re still waiting for API access (and therefore can’t run proper…
-
Guides · · 2 min readHow to prepare for Gemini 3 + GPT 5.1
Here we go again: new-flagship season. Google’s Gemini 3 has been peeking through A/B tests in AI Studio and docs watchers have noticed model lifecycle shuffles, while OpenAI is lining up a GPT-5.1 family (base…