This is just going to correlate directly with the number of character build guides/etc that exist and are parsed. Pinning this to some kind of inherent intellect seems extremely naive.
It would have been interesting to have asked top STS players to review the most popular guides for each character and then to compare ones they determine to be 'strong' or 'misleading' against model performance.
I recognize that, generally, the circles I'm a part of tend to pay the most attention to Coding-focused evals and benchmarks as frontier models improve. Biggest jumps I'm seeing with Astra and 6/6.1 Sol, and their Anthropic counterparts to a smaller degree, seems to be in spatial reasoning and playing games.
Looks like Argon wasn't covered here, but I'm getting the sense that Argon made similar leaps as Astra did. I would certainly be curious how Argon would have done here- it did quite well on RuneBench (https://maxbittker.github.io/runebench/).
This is just going to correlate directly with the number of character build guides/etc that exist and are parsed. Pinning this to some kind of inherent intellect seems extremely naive.
It would have been interesting to have asked top STS players to review the most popular guides for each character and then to compare ones they determine to be 'strong' or 'misleading' against model performance.
Just reporting the data outright seems lazy.
I recognize that, generally, the circles I'm a part of tend to pay the most attention to Coding-focused evals and benchmarks as frontier models improve. Biggest jumps I'm seeing with Astra and 6/6.1 Sol, and their Anthropic counterparts to a smaller degree, seems to be in spatial reasoning and playing games.
Looks like Argon wasn't covered here, but I'm getting the sense that Argon made similar leaps as Astra did. I would certainly be curious how Argon would have done here- it did quite well on RuneBench (https://maxbittker.github.io/runebench/).