Benchmarks / BulletBench
When every second of thinking time comes off the clock, which models are fast enough to still make good decisions?
Last updated 8 Oct 2026
8 results above zero · % of games, lower is better. Choose a model to highlight it.Clear highlight
4 models had none: Claude Haiku 4.5, Gemini 3.5 Flash-Lite, Mistral Large 4, Mistral Medium 3.5.
Each configuration's median time per move at 10 seconds plus 1 a move. Right of the line, a model thinks longer than the increment it earns and the clock drains.
Named: the six bestOther models (hover for names)
| # | Model | Games lost on time, Blitz 3+2 % of games, lower is better | Lightning 10+1 ladder Elo | Bullet 60s ladder Elo | Blitz 3+2 ladder Elo | Move time seconds | Cost per game US dollars | Slowest 10% of moves seconds, 90th percentile | Games at this clock |
|---|---|---|---|---|---|---|---|---|---|
| 1≈ | Claude Haiku 4.5Anthropic |
0.0%
|
33 | 445 | 491 | 1.3 s | $0.0318 | 1.9 s | 8 |
| 1≈ | Gemini 3.5 Flash-LiteGoogle |
0.0%
|
892 | 780 | 707 | 0.8 s | $0.0087 | – | – |
| 1≈ | Mistral Large 4Mistral AI |
0.0%
|
249 | – | 555 | 0.8 s | $0.0082 | 2.1 s | 8 |
| 1≈ | Mistral Medium 3.5Mistral AI |
0.0%
|
529 | 385 | 555 | 0.5 s | $0.0261 | 0.8 s | 8 |
| 5 | Gemini 3.8 FlashGoogle |
12.5%
|
– | 529 | 1,126 | 4.0 s | $0.0795 | 7.9 s | 8 |
| 6 | Claude Fable 5.1Anthropic |
37.5%
|
– | – | 358 | 5.4 s | $0.38 | 18.2 s | 8 |
| 6 | Claude Haiku 5.5Anthropic |
37.5%
|
– | – | 557 | 6.2 s | $0.0164 | – | – |
| 6 | Claude Sonnet 5.5Anthropic |
37.5%
|
– | – | 557 | 8.1 s | $0.12 | 14.0 s | 8 |
| 6 | GPT-6 AstraOpenAI |
37.5%
|
– | 197 | 839 | 5.6 s | $0.28 | 14.0 s | 8 |
| 10 | Gemini 3.1 Pro PreviewGoogle |
50.0%
|
– | – | 683 | 6.2 s | $0.21 | 16.2 s | 8 |
| 10 | GPT-6 LunaOpenAI |
50.0%
|
– | – | 557 | 7.0 s | $0.0072 | 20.2 s | 8 |
| 12 | Claude Opus 5.5Anthropic |
62.5%
|
– | – | 557 | 7.5 s | $0.18 | 13.6 s | 8 |
Swipe the table sideways for more columns.
Ranks follow the score as shown, so equal numbers share a rank. Each model runs at its provider's default reasoning setting and, where the provider offers a faster one, again at its fastest setting, listed separately. Blitz 3+2 was run at default settings only. Equal ratings are not a glitch: the ladder's ratings come in steps (see Honest caveats below).
These configurations lost their first four games on time, so they were stopped there and are not rated for that clock. Each shows its median time per move and the time that 1 move in 10 took or exceeded: a median under the increment still loses on time when the slow tail is long. Being too slow is the result, not a fault.
Decision models answer a typed question instead of writing text. Each move is one question whose options are the legal moves, so they cannot play an illegal move, and they are not told their clock. They play the same clock, ladder and engine and are in the table above; here they are ranked among themselves.
No decision model was rated on this measure.
Too slow for the clock. These lost their first four games on time and are not rated for that clock.
Each model's clock after every move of one game. A fast model earns back more than it spends; a slow one runs out within a few moves, whatever the position.
The same at 60 seconds for the whole game, no increment. A 40-move game leaves 1.5 seconds a move.
Named: the six bestOther models (hover for names)
Share of games lost on time, and moves with no legal reply, by clock.
| Model | On time, Lightning | On time, Bullet | Invalid moves, Lightning | Invalid moves, Bullet |
|---|---|---|---|---|
| Gemini 3.5 Flash-LiteGoogle | 0.0% | 0.0% | 0.0% | 0.0% |
| Gemini 3.1 Flash-Lite (minimal reasoning)Google | 0.0% | 8.3% | 0.2% | 0.0% |
| Gemini 3.5 Flash (minimal reasoning)Google | 33.3% | 25.0% | 0.6% | 0.5% |
| Mercury DecideInception | 0.0% | 0.0% | 0.0% | 0.0% |
| d1Liquid AI | 0.0% | 0.0% | 0.0% | 0.0% |
| Seed-2.0-Mini (minimal reasoning)ByteDance | 0.0% | 0.0% | 0.8% | 0.4% |
| Mistral Small 4 (no reasoning)Mistral AI | 0.0% | 25.0% | 1.0% | 0.6% |
| GPT-5.4 mini (no reasoning)OpenAI | 0.0% | 50.0% | 0.0% | 0.0% |
| Inkling Small (no reasoning)Thinking Machines | 0.0% | 0.0% | 5.3% | 7.2% |
| Jev 1.13TypeSafe | 0.0% | 0.0% | 0.0% | 0.0% |
| Clef FlashCloudflare | 0.0% | 0.0% | 0.0% | 0.0% |
| ClefCloudflare | 0.0% | 0.0% | 0.0% | 0.0% |
| Mistral Medium 3.5 (no reasoning)Mistral AI | 0.0% | 0.0% | 0.0% | 0.4% |
| Mistral Medium 3.5Mistral AI | 0.0% | 8.3% | 0.3% | 0.8% |
| GPT-6 Luna DecisionsOpenAI | 0.0% | 0.0% | 0.0% | 0.0% |
| Decider V1 27BPerplexity | 0.0% | 8.3% | 0.0% | 0.0% |
| Mistral Large 4Mistral AI | 58.3% | – | 1.8% | – |
| Gemini 3.8 Flash (low reasoning)Google | 83.3% | 50.0% | 0.0% | 0.0% |
| GPT-5.4 nano (no reasoning)OpenAI | 75.0% | 66.7% | 4.0% | 3.1% |
| Claude Haiku 4.5Anthropic | 83.3% | 25.0% | 0.0% | 0.3% |
| Claude Sonnet 5.5 (low reasoning)Anthropic | 91.7% | – | 0.0% | – |
| Gemini 3.6 Flash (minimal reasoning)Google | 91.7% | 41.7% | 0.0% | 0.3% |
| Kev 4BJared Palmer | 66.7% | – | 0.0% | – |
What one Lightning game cost at the run date's prices, on a log scale. Up and to the left is better; the dashed line joins the strongest model at each cost.
Named: the six best and the best for the moneyOther models (hover for names)Best score at each cost
Cheapest first: each model here beats every cheaper one on score.
The model gets the position, the moves so far, its remaining clock and the legal moves.
It answers with one move. The whole API round trip, including any hidden reasoning, comes off its clock.
Stockfish replies instantly at the model's current ladder level.
Win and the next game is a level up; lose and it is a level down; draw and it stays. The results give a rating with a 95% interval.
Many products need an answer in about a second: routing, triage, autocomplete, live agents. BulletBench asks which models can still think usefully at that speed. Chess gives a hard, objective score, and the clock makes slow thinking a real cost instead of a free extra.
A dozen games or fewer per clock (the table gives each configuration's count) give wide intervals, so neighbouring ranks are often within each other's ranges. Ratings also come in steps: the engine plays at fixed levels and the rating is fitted from which levels a model beat, drew or lost to, so models with the same results against the same levels get exactly the same rating. Latency depends on the provider's servers on the day. The ladder's Elo anchors come from v1's Stockfish; this edition runs Stockfish 19, so ratings are not comparable with v1's.