Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.
Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?
I think those kinds of questions would be more interesting than the raw win/loss ratios.
Dan Gant (author of Purple Wave) has written about some of this.
I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.
I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.
none of these games really require a lot of experimental feedback, the goals are well stated and known; experiments may drive optimization if thats the question.
as far as the on rails comment : there's no good way to know the provenance of an LLM output, but the LLM made bots generally look like very small, modular, sometimes functionalized versions of the human equivalents; the difference being that the LLM is calling the funtions often rather than a conditional loop in a main().
a factorio bot I worked on when Astra was first getting previewed was essentially a directory of very small argument-receiving functional and self-supporting programs that had very limited scope, and an agents.md file that directed the LLM to where such capabilities existed to preempt a directory search every compaction/context-swap.
so you'd have things like "python walk.py N 40" for 40 units of northern travel. a watchdog that waited for HP to drop or movement to become inhibited, etc.
My very first experience with LLM driven bot development in the GPT3 days created huge unmaintainmable monoliths that kind of worked due to TDD but were essentially spaghetti. LLMs have been trending away from those paradigms into small self-encapsulated programs. probably because those are less abusive to ctx count and produce better results since they lose/compact less.
I would disagree that creating a StarCraft: BW bot doesn't require experimental feedback. The way a human author typically improves their bot is by having it play opponents and A/B test tactics and strategies.
There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).
> ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
Anthropic (Claude models): Claude Code
OpenAI (GPT models): Codex
Google (Gemini models): Antigravity CLI
Meta (Muse models): Muse Code
> The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.
One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed
- logic analysis
- understanding of the game platform and how use of C may influence it
- considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
Also: given that the human written bots are almost certainly in the training data, the winning rate against human-written bot seems to be not well-calibrated. You would expect at least they are 50% or better, given that they can "memorize" the best human-written bot.
Worth digging into the protocol and see what went wrong.
Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
> Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
This is somewhat interesting to me as I’ve recently come to a different conclusion. I’ve been populating a ModernUO server with bots, and have found Opus 5.5 to be substantially better at understanding and building it. I actually started with 6.1 Sol and 6 Astra but pulled them after a week of little progress, had Opus 5.5 take over and it rebuilt everything properly in a night and has carried it forward quite a way since.
I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.
This kind of makes me think about making better bots for Counter Strike based on human top level players, which i think its a very undervalued category of software. You can even forget about about CS and name any FPS and you'll be hard pressed to find a game with seriously good bots.
I think you are thinking about insta-killing machine bots set to detect a head through a wall and shoot it. I am talking about properly trained bots that while playing pretty good are also making mistakes and taking actions that may or maybe not suit the team strategy. A bot with the randomness of a human.
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
I assume lots of bot code is part of the training data? So it's basically a version of how will they can improve upon those solutions public out there?
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
I think we want to move beyond code based bots asking the LLMs to try to train a custom neural net, in a limited time + compute, to play StarCraft BW is probably the most promising direction
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
If you are talking about take a screen shot, analyze move the window a bit, take another screenshot, move the window, repeat until they have seen enough of the map to make a decision, then decide what to click on, issue order.... this could absolutely be done. It's very similar to how LLM's have driven a car on a safe track.
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
Even the most powerful frontier models on SOTA hardware are not fast enough for starcraft, which generally involves multiple informed decisions per second at peak moments.
You can train a specialized model for this, but a frontier model is ill-fitting.
There is currently an AI bot playing on the ladder in Korea and essentially it will nearly always win because it can micro every unit individually across the entire map. If a normal human can max out at hundreds of actions per minute, the bot can do thousands. Broodwar is so insanely difficult mechanically that there is no way for humans to compete. I imagine in a few months, maybe a year, it will be completely unbeatable.
Maybe because Protosses have been doing bad lately and need some help improving :) (won only 3 out of 22 ASL championships in the past 10 years)
On a more serious note: all three races play quite differently so it makes sense to focus on just one. And protoss is mechanically the easiest and most straightforward race: fewer but stronger units, a single worker can be used to build multiple buildings at once, versatile basic static defence
Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.
[1] https://rolandgao.com/blog/gobench/
Evidence for g in LLMs: https://en.wikipedia.org/wiki/G_factor_(psychometrics)
Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?
I think those kinds of questions would be more interesting than the raw win/loss ratios.
Dan Gant (author of Purple Wave) has written about some of this.
I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.
I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.
https://github.com/llmskirmish/starskirmish-hillclimb
none of these games really require a lot of experimental feedback, the goals are well stated and known; experiments may drive optimization if thats the question.
as far as the on rails comment : there's no good way to know the provenance of an LLM output, but the LLM made bots generally look like very small, modular, sometimes functionalized versions of the human equivalents; the difference being that the LLM is calling the funtions often rather than a conditional loop in a main().
a factorio bot I worked on when Astra was first getting previewed was essentially a directory of very small argument-receiving functional and self-supporting programs that had very limited scope, and an agents.md file that directed the LLM to where such capabilities existed to preempt a directory search every compaction/context-swap.
so you'd have things like "python walk.py N 40" for 40 units of northern travel. a watchdog that waited for HP to drop or movement to become inhibited, etc.
My very first experience with LLM driven bot development in the GPT3 days created huge unmaintainmable monoliths that kind of worked due to TDD but were essentially spaghetti. LLMs have been trending away from those paradigms into small self-encapsulated programs. probably because those are less abusive to ctx count and produce better results since they lose/compact less.
I would disagree that creating a StarCraft: BW bot doesn't require experimental feedback. The way a human author typically improves their bot is by having it play opponents and A/B test tactics and strategies.
There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).
> ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
> The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
3. thinking levels
Benchmark vs intelligence which considers more?
I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
Also: given that the human written bots are almost certainly in the training data, the winning rate against human-written bot seems to be not well-calibrated. You would expect at least they are 50% or better, given that they can "memorize" the best human-written bot.
Worth digging into the protocol and see what went wrong.
Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
If Astra is leading the pack...Considering it's supposed to be one of the slowest model.
> Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
> I am just not sure what it measures.
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
Yes, but I guess my objection is that if it attempts to measure everything then we can't really tell what it measures in what proportions.
Yep, that’s a feature of end to end measures, like company revenues, vehicle fuel efficiency, or human life expectancy.
The customer may not care what the internal contributions are to such a measure. Instead it represents the customers’ overall values in some way.
Ok. I think you convinced me based on those. I withdraw the objection.
general "agenticness"
So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly - https://broodbench.com/ and https://news.ycombinator.com/item?id=49766966
This is somewhat interesting to me as I’ve recently come to a different conclusion. I’ve been populating a ModernUO server with bots, and have found Opus 5.5 to be substantially better at understanding and building it. I actually started with 6.1 Sol and 6 Astra but pulled them after a week of little progress, had Opus 5.5 take over and it rebuilt everything properly in a night and has carried it forward quite a way since.
They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.
I did a live stream where Astra played Pluto. Astra went 0 - 1000 https://x.com/kaimcpheeters/status/2106782151622504911?s=20
I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.
In any game that requires reflexes and muscle based twitch accuracy and so on, usually the hard part is making the ai lose.
Can't that be fixed just be setting levels of reaction speeds?
This kind of makes me think about making better bots for Counter Strike based on human top level players, which i think its a very undervalued category of software. You can even forget about about CS and name any FPS and you'll be hard pressed to find a game with seriously good bots.
The bots are bad not because it's hard to write a good FPS bot, but because it's not fun to play against ones that are great.
I think you are thinking about insta-killing machine bots set to detect a head through a wall and shoot it. I am talking about properly trained bots that while playing pretty good are also making mistakes and taking actions that may or maybe not suit the team strategy. A bot with the randomness of a human.
The video shows GLM but it's not in any graphs. Are there more complete charts anywhere else? Ones that include the 45 LLMs?
GLM 5.3 is captured in the v0.1 write up here: https://starskirmish.com/bench/v0-1/
This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
I did a 48 hour live stream - Astra cleared in 43 hours.
Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
Would love to see an LLM written bot play Red Alert 2/Red Alert.
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
I assume lots of bot code is part of the training data? So it's basically a version of how will they can improve upon those solutions public out there?
Does anyone know if there was a project having LLMs play Cities Skylines and see how well it manages to design and build a city?
If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net (SC2)
Oh my gosh! It feels so cool trying this that is played by LLMs. I am sooo eager to try argon 4 and build many robotic projects.
Not played by LLMs. The LLMs write the bots that play the game. They don't play directly. Read the article!
It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
The site overall looks nice too. It’s fun not to see slop for a ui for this kind of project.
What if we let the agents play directly instead of writing bots?
I think we want to move beyond code based bots asking the LLMs to try to train a custom neural net, in a limited time + compute, to play StarCraft BW is probably the most promising direction
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
https://broodbench.com/
TLDR they suck and many default to doing probe rushes.
If you are talking about take a screen shot, analyze move the window a bit, take another screenshot, move the window, repeat until they have seen enough of the map to make a decision, then decide what to click on, issue order.... this could absolutely be done. It's very similar to how LLM's have driven a car on a safe track.
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
Even the most powerful frontier models on SOTA hardware are not fast enough for starcraft, which generally involves multiple informed decisions per second at peak moments.
You can train a specialized model for this, but a frontier model is ill-fitting.
How do the best humans do against the LLMs?
There is currently an AI bot playing on the ladder in Korea and essentially it will nearly always win because it can micro every unit individually across the entire map. If a normal human can max out at hundreds of actions per minute, the bot can do thousands. Broodwar is so insanely difficult mechanically that there is no way for humans to compete. I imagine in a few months, maybe a year, it will be completely unbeatable.
Any thoughts on building this for Starcraft 2?
why protoss?
Maybe because Protosses have been doing bad lately and need some help improving :) (won only 3 out of 22 ASL championships in the past 10 years)
On a more serious note: all three races play quite differently so it makes sense to focus on just one. And protoss is mechanically the easiest and most straightforward race: fewer but stronger units, a single worker can be used to build multiple buildings at once, versatile basic static defence
Protoss has the most mature human written SC BW bot scene right now. If you look on BASIL ladder much of the top 10 are Protoss.
Protoss vs Protoss only also helps downscope the problem given the time constraints
for AIur
My wife for hire
toss imba