⏱️ Lectura: 11 min
Codex Astra, in its highest-effort configuration (xhigh), played 18 full matches of StarCraft: Brood War against other language models and didn’t lose a single one. It’s the most decisive result from Brood War Bench, a new benchmark that measures how well the latest AI agents play a real-time strategy video game.
📑 En este artículo
The experiment pitted nine model families (including Claude Fable, Claude Opus 5, Grok 4.6, and several Codex variants) against each other in full matches of Blizzard’s 1998 classic. The result is clear: even the model that won everything still plays at a beginner level.
TL;DR
- Codex Astra (xhigh) won all 18 of its matches in Brood War Bench: 100% win rate at $10.54 per game.
- Claude Fable finished third with 15 wins and 3 losses, the only AI that built out a full tech tree.
- Grok 4.6 (xhigh) generated 11,138 reasoning tokens in G043 without producing a single combat unit.
- Just 0.3 actions per minute: that’s how little Claude Haiku played, with 0 wins in 16 matches.
- The benchmark has AI agents play on a playable version of the 1998 StarCraft: Brood War.
- No model beat beginner level: a human using a photon rush strategy would beat all of them.
- Codex’s economy and army subagents don’t coordinate with each other: they attack unit by unit.
- In match G009, Codex 5.6 Terra evacuated its last Command Center and held on for six more minutes after losing its base.
What happened in Brood War Bench
The project started from a simple idea. Its creator built a version of Brood War that can only be played through AI agents, to test it with friends who had barely touched StarCraft a couple of times. They did surprisingly well, and when he asked why, the answer was that they’d barely done anything: they asked their agent to attack, and it built a small army and carried out the entire attack on its own. That anecdote sparked the project’s central question: how far can an AI agent get playing Brood War completely on its own?
Brood War Bench answers with a leaderboard that pits different models and reasoning effort levels (low, medium, xhigh) against each other in full matches, tracking wins, losses, actions per minute (APM), and cost in dollars per match. Codex Astra on xhigh finished first with 18 wins and 0 losses (100.0% win rate) at a cost of $10.54 per match. Behind it came Codex Astra on medium (16-2, 88.9%) and Claude Fable (15-3, 83.3%), the model that came closest to playing normally.
Context and history
StarCraft: Brood War is Blizzard’s 1998 expansion to the original StarCraft, and it remains, nearly three decades later, one of the most demanding real-time strategy games out there: it requires managing economy, production, and combat simultaneously, often with hundreds of actions per minute in the human competitive scene. It’s no coincidence that the AI research community has used it as a benchmark for years, from academic tournaments with hand-coded bots to reinforcement learning systems trained for StarCraft II.
Brood War Bench takes a different approach: instead of training a model specifically for the game, it uses general-purpose language models (the same ones used for coding or answering questions) and connects them to the match as agents that observe the game state and issue commands. There’s no prior training on Brood War: each model has to reason out its strategy from scratch, match after match.
Technical details and performance
The table below summarizes the top tier of the full leaderboard, ranked by wins:
| Model (effort) | Record | Win rate | Cost/match | APM |
|---|---|---|---|---|
| Codex Astra / xhigh | 18-0 | 100.0% | $10.54 | 12.6 |
| Codex Astra / medium | 16-2 | 88.9% | $15.11 | 17.2 |
| Claude Fable | 15-3 | 83.3% | $12.24 | 12.6 |
| Codex Astra / low | 14-4 | 77.8% | $21.07 | 25.7 |
| Codex 5.6 Sol / medium | 13-5 | 72.2% | $5.12 | 10.1 |
⚠️ Heads up: more reasoning effort doesn’t always mean cheaper or better performance. Codex Astra on low costs $21.07 per match (more expensive than medium, at $15.11) and wins less often (77.8% versus 88.9%): thinking slower doesn’t make up for thinking worse.
The gameplay patterns reveal why. Codex found more success in disruption than in macroeconomics: in matches playing Protoss, it would send a Probe across the map to attack workers or buildings, something that worked because rival agents spent dozens of seconds deciding what to do about that single unit instead of continuing to play. The same model family, on the other hand, was much weaker at sustaining production: it delayed teching up, trickled one or two basic units toward defended bases, and threw its workers into a last-ditch defense.
Part of the problem is architectural. Codex tended to create separate subagents to handle economy, army production, and army control, which barely communicated with each other. The production subagent would send every new unit straight into the attack without waiting for the army to reach critical mass, one of the most typical mistakes a beginner player makes.
The following diagram illustrates the coordination problem observed in Codex’s subagents:
flowchart TD
A["Codex main agent"] --> B["Economy subagent"]
A --> C["Production subagent"]
A --> D["Army control subagent"]
C --> E["New unit ready"]
E --> F["Immediate attack, unit by unit"]
D -.-> F
subgraph "No communication between subagents"
C
D
endGrok 4.6 showed the opposite problem: it spent the match thinking instead of acting. In match G043, the xhigh run generated 11,138 reasoning tokens but issued only six batches of commands in 43 minutes, and never got around to producing a single combat unit. In G003, Grok on xhigh built three Marines and never crossed the map to the enemy base. In G002, Grok on medium made two Zealots and also failed to cross. These don’t look like bad strategies. They look like failures to sustain the observe-and-act loop.
Claude Fable was, according to the report itself, the model worth rooting for: instead of stopping at the first available unit, it tried to build up its economy and climb the tech tree. In match G007 it reached a Lair, a Spire, and produced Mutalisks, and won. In G027 it added a Robotics Facility, a Citadel of Adun, an Observatory, and a Templar Archives before winning. Ambition didn’t always pay off: in G036, Fable reached a Factory and an Academy, but Claude Opus 5 crushed it before it could capitalize.
The aggregate numbers at the 15-minute mark confirm the difference in approach. Codex Astra averaged 14.7 living workers, 8.3 army units, and 6.2 structures; Claude Fable, 16.7 workers, 7.6 army units, and 7.4 structures; Grok 4.6, just 7.4 workers, 2 army units, and 4.5 structures. In completed tech research, Codex Astra reached 0.3 levels, Fable 0.2, and Grok stayed at 0.
How to try it
The full report, with interactive filters by model, effort, race, and match minute, is available at bw.swerdlow.dev/report. There you can compare up to five models at once and see the minute-by-minute evolution of workers, army, structures, and accumulated minerals.
If you want to experiment with the same idea Brood War Bench uses (a language model that observes a game state and returns commands), the basic pattern is easy to reproduce with any LLM API that supports tool use. Install the SDK on any operating system:
# Windows (PowerShell), macOS, and Linux: same command
pip install anthropic
That’s enough to put together a minimal loop that takes the game state as text and returns an action:
import anthropic
client = anthropic.Anthropic()
def turno_agente(estado_juego):
respuesta = client.messages.create(
model="claude-sonnet-5",
max_tokens=512,
tools=[{
"name": "ejecutar_comando",
"description": "Executes a game command (build, attack, move)",
"input_schema": {
"type": "object",
"properties": {
"accion": {"type": "string"},
"objetivo": {"type": "string"}
},
"required": ["accion"]
}
}],
messages=[{"role": "user", "content": estado_juego}]
)
return respuesta.content
That loop (read state, decide action, execute, repeat) is exactly what separated Codex Astra from Grok 4.6 in Brood War Bench: not the model’s abstract intelligence, but how many times per minute it completes the cycle without getting stuck reasoning.
💡 Tip: if you’re going to measure your own agent, track the ratio between reasoning tokens and commands issued per minute: it’s the metric that exposed Grok 4.6’s paralysis in Brood War Bench.
Impact and analysis
The benchmark’s most important result isn’t that Codex Astra won all its matches. It’s that none of the models, not even the winner, played above beginner level. According to the report itself, a human player executing a photon rush (an aggressive opening well-known in the StarCraft community) would beat any of the 19 systems evaluated in the table.
💭 Key point: the pattern repeats across almost every long-horizon agent benchmark: the bottleneck isn’t how much the model knows, but whether it can sustain the observe, decide, and act loop without getting stuck thinking.
That tempers any reading that AI already knows how to play strategy video games on its own. What Brood War Bench shows is more specific: general-purpose language models can execute a reasonable sequence of actions (build, produce, attack) but fail to sustain a plan over the 15 or 20 minutes a short match lasts, to coordinate multiple areas of attention at once, or to adapt when the opponent changes strategy.
Cost also matters for anyone looking to reproduce this kind of evaluation at scale. The $21.07 per match for Codex Astra on low mode, compared to the $0.16 for Codex 5.6 Luna on xhigh, shows a difference of more than 100x in spend per match between configurations of the same model family, without the more expensive one always winning. That’s the kind of data any team evaluating AI agents for long-horizon tasks should look at before assuming more effort is synonymous with better results.
The project’s own most honest limitation is sample size: each configuration played only between 16 and 18 matches, a low volume for drawing firm statistical conclusions about which model is rigorously better. It’s enough to spot broad patterns (Grok doesn’t complete the action loop, Codex finds cheese strategies, Fable actually tries to play) but not to precisely rank models that end up close to each other.
What’s next
The project’s creator said Brood War Bench is far from running out of runway: there’s a lot more agents can learn to do, and a lot more the benchmark itself can demand of them. The likely next iterations point toward requiring more complex strategies, concrete defenses against simple attacks, and longer matches where medium-term planning matters more than immediate reaction.
📖 Summary on Telegram: View summary
Try it yourself: open the interactive leaderboard at bw.swerdlow.dev/report and filter by your favorite model to see how its match played out minute by minute.
Frequently Asked Questions
What is Brood War Bench?
It’s a benchmark that has different language models, connected as agents, play full matches of StarCraft: Brood War, and publishes a leaderboard with wins, losses, APM, and cost per match.
Which model won the most matches?
Codex Astra in xhigh effort mode, with 18 wins and 0 losses (100.0% win rate) across the 18 matches it played.
Why did Grok 4.6 perform so poorly?
Because it spent too much time reasoning between actions. In match G043 it generated 11,138 reasoning tokens but issued only six batches of commands in 43 minutes, without producing a single combat unit.
Did any model play like an expert?
No. According to the report itself, none of the 19 systems evaluated rose above beginner level: a human executing an aggressive opening known as a photon rush would beat any of them.
What does APM mean in this context?
Actions per minute: how many commands (build, move, attack) the agent issues on average during the match. A very low APM, like Claude Haiku’s 0.3, indicates the agent barely acted at all.
How much does it cost to run a match on this benchmark?
It varies a lot depending on the model and effort level: from $0.16 per match (Codex 5.6 Luna on xhigh) to $21.07 (Codex Astra on low).
References
- Brood War Bench, full report: interactive leaderboard, progress charts, and the matches analyzed in this article.
- StarCraft: Brood War on Wikipedia: historical context on Blizzard’s 1998 expansion.
- Official Anthropic documentation: reference for the Claude models used in the benchmark.
- OpenAI’s official site: developer of the Codex model family evaluated in the report.
📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
0 Comments