Good point! I used hugging face as a fixed baseline for the hill climb so I could measure what each change bought! Also at some point I switched to AlphaEvolve style work to see how can i do that and then I recorded the process.
This was a personal project just wanted to see how I can do speed ups and track the process of it! It was more of a project to learn how to systematically do speedup rather than the final results!
Nice but it would be also great to include strata and llamacpp to the benchmark. Otherwise it’s 11x speedup over the baseline no one uses
Good point! I used hugging face as a fixed baseline for the hill climb so I could measure what each change bought! Also at some point I switched to AlphaEvolve style work to see how can i do that and then I recorded the process.
[dead]
That's cool, but what are you doing that you want to run qwen3.6-35b-a3b, over qwen3.8-flash.next, which runs great on a 4090
This was a personal project just wanted to see how I can do speed ups and track the process of it! It was more of a project to learn how to systematically do speedup rather than the final results!
Good enough reason :)
Do you think the engine could handle Q38FN?
That would have to be more heavily quantized right? It's a 128bn vs 35bn? How much does that affect it?
You can run flash-next on a 4090 at a 4bit quantization, which is by most accounts close enough.
It looks like OP is starting with the bf16 unquantized qwen3.6-35b-a3b, but I couldn't tell if they are quantizing it down from there or not.
I kept the 35B fixed for the hill climb to see how much runtime changes each change.