Author here. I’ve been running this experiment for a couple of years whenever a new model comes out, and this is the first time the results have been good enough to share.
I provide an LLM with a natural language specification (e.g., "an HTTP server on port 8125"), and a bash loop executes what the model generates; the actual error encountered is then fed back to the model until the test passes. There are three levels: C with gcc, assembly with as + ld, and raw bytes (where the model writes the ELF file directly, with nothing between it and the binary other than `xxd -r`).
The raw bytes level isn't always reliable: in 5 runs using Claude Sonnet 5.5 on x86-64, "Hello, World" succeeded in 4 out of 5 attempts, and the HTTP server in 2 out of 5. The C and assembly levels succeeded in all 5 attempts. The system also executes binaries generated by the model, so please check what the code does before running it.
So could Astra, it uses pythons but just for assembling the final exe and the bytes are in the script. It makes simple stuff like a calculator and you can even asked to change/fix anything not working correctly. also I've tested astra can write pixels to a tiff to get an image directly, without using any other tools
it's still vastly underestimated how much these models can do and what would happen once these domains are scaled up
Author here. I’ve been running this experiment for a couple of years whenever a new model comes out, and this is the first time the results have been good enough to share.
I provide an LLM with a natural language specification (e.g., "an HTTP server on port 8125"), and a bash loop executes what the model generates; the actual error encountered is then fed back to the model until the test passes. There are three levels: C with gcc, assembly with as + ld, and raw bytes (where the model writes the ELF file directly, with nothing between it and the binary other than `xxd -r`).
The raw bytes level isn't always reliable: in 5 runs using Claude Sonnet 5.5 on x86-64, "Hello, World" succeeded in 4 out of 5 attempts, and the HTTP server in 2 out of 5. The C and assembly levels succeeded in all 5 attempts. The system also executes binaries generated by the model, so please check what the code does before running it.
I’d be happy to answer any questions
So could Astra, it uses pythons but just for assembling the final exe and the bytes are in the script. It makes simple stuff like a calculator and you can even asked to change/fix anything not working correctly. also I've tested astra can write pixels to a tiff to get an image directly, without using any other tools
it's still vastly underestimated how much these models can do and what would happen once these domains are scaled up