what's interesting some of the models do better at various task types. I wonder if you built a simple decision tree and tried a new out of sample test how well it would score.
Good question, so I just tried it. On each test I picked the best model using half the questions, then scored that pick on the other half (repeated 300 times):
Best single model (GPT-6 Astra): 87.8% right
Best model per task type: 86.9%
So routing doesn’t beat just using the top model. With not enough questions per test, many of the gaps between top models are noise. We would need more questions to test routing properly.
Interestingly, routing seems to pick cheaper models (Gemini 4 Argon, GPT-6.1 Sol), so it costs about half as much for about one point less.
what's interesting some of the models do better at various task types. I wonder if you built a simple decision tree and tried a new out of sample test how well it would score.
Good question, so I just tried it. On each test I picked the best model using half the questions, then scored that pick on the other half (repeated 300 times):
Best single model (GPT-6 Astra): 87.8% right
Best model per task type: 86.9%
So routing doesn’t beat just using the top model. With not enough questions per test, many of the gaps between top models are noise. We would need more questions to test routing properly. Interestingly, routing seems to pick cheaper models (Gemini 4 Argon, GPT-6.1 Sol), so it costs about half as much for about one point less.
[dead]