I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.
AMD 7900 XTX with Vulkan here as well, wasn't faster on my test either. Might be much different on Nvidia though.
I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.
4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.
4090 isn't that much higher than the XTX (I also have the XTX), it's 1008GB/s (4090) vs 960GB/s for the XTX's.
The 5090 destroys both at 1792GB/s.
It's not really one thing with the nvidia cards best I can tell it's that they compounded incremental gains from software drivers, card kernels and optimization from been the primary choice (plus first mover advantage).
I didn't buy the XTX for AI purely gaming but it's a capable enough local card for running Qwen et al.
Absolute treasure of a website with these graphs, thank you for sharing this. Huge help for me to find a faster model (smaller quantization) for my VRAM.
Not really. The largest IQ4_XS quant here is still worth it because Bonsai doesn't offer larger quants. They could beat it if they made a quaternary variant though, I don't know why they're stopping at ternary.
I wonder the same thing for Bonsai 2. ByteShape offers 5 models from IQ2_XXS-2.56bpw (8.8GB), IQ3_XXS-2.88bpw (9.9GB), IQ3_XS-3.01bpw (10.4GB), IQ3_S-3.23bpw (11.0GB) to IQ4_XS-3.84bpw (13.1GB). Their benchmarks show gradual improvement with size and users can pick one to fit theirs need. Bonsai-2-27B now is about 8.6GB. It might be good to have a quaternary version around 10-11GB to fit a computer with 16-24GB RAM.
Seems like a lot of them do, I only compare them within the same repo because there doesn't seem to be a very standard way of saying all the possible combinations/rearrangements.
bpw is the way to compare. huggingface has standard tags that must be used so it forces anyone releasing models to choose a tag that doesn't necessarily equal the actual bpw.
I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.
From my own test. It's not faster than the unsloth model.
Disclarer: I'm unsing Vulkan on an AMD GC.
AMD 7900 XTX with Vulkan here as well, wasn't faster on my test either. Might be much different on Nvidia though.
I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.
4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.
4090 isn't that much higher than the XTX (I also have the XTX), it's 1008GB/s (4090) vs 960GB/s for the XTX's.
The 5090 destroys both at 1792GB/s.
It's not really one thing with the nvidia cards best I can tell it's that they compounded incremental gains from software drivers, card kernels and optimization from been the primary choice (plus first mover advantage).
I didn't buy the XTX for AI purely gaming but it's a capable enough local card for running Qwen et al.
4090 vs 5090 performance difference is largely GDDR6 vs GDDR7, I think
Surprised its not meaningfully slower.
Vulkan and ROCm paths are missing a few optimized versions of the quants they're using.
I feel weird that I like your typos, because clearly AI did not write your post. Typos have become downright charming and nostalgic for me.
Absolute treasure of a website with these graphs, thank you for sharing this. Huge help for me to find a faster model (smaller quantization) for my VRAM.
Aged like milk
https://news.ycombinator.com/item?id=49746618
prismml's title is very misleading. in their own paper the model is at 75% of coding scores.
In the comments it reads like bonsei falls apart on longer running tasks.
Not really. The largest IQ4_XS quant here is still worth it because Bonsai doesn't offer larger quants. They could beat it if they made a quaternary variant though, I don't know why they're stopping at ternary.
I wonder the same thing for Bonsai 2. ByteShape offers 5 models from IQ2_XXS-2.56bpw (8.8GB), IQ3_XXS-2.88bpw (9.9GB), IQ3_XS-3.01bpw (10.4GB), IQ3_S-3.23bpw (11.0GB) to IQ4_XS-3.84bpw (13.1GB). Their benchmarks show gradual improvement with size and users can pick one to fit theirs need. Bonsai-2-27B now is about 8.6GB. It might be good to have a quaternary version around 10-11GB to fit a computer with 16-24GB RAM.
What's GPU-5?
The fattest quantization. They show all of them here: https://huggingface.co/byteshape/Qwen3.8-27B-GGUF
Since they’re not using a stable number of bits per token, they use their own naming convention.
> they use their own naming convention.
Seems like a lot of them do, I only compare them within the same repo because there doesn't seem to be a very standard way of saying all the possible combinations/rearrangements.
bpw is the way to compare. huggingface has standard tags that must be used so it forces anyone releasing models to choose a tag that doesn't necessarily equal the actual bpw.