At the same time, game studios can't publish a game that runs steady at their target frame-rate on ps5 to save their lives (except for indie titles). I work in game dev: it's doable and makes no sense to me that tiny teams are optimizing their games better than $100M projects
The rumor is that the 16 GB 5080 will be replaced by a 24 GB version.
No more 32 GB consumer cards, the next step up is a "Pro" card with an extra digit on its MSRP.
I'm reminded of the "beginning of the end" for 3dfx where they kept doubling-down on basic scaling without fundamental improvements, and were wiped out by more efficient cards from upstart little manufacturers like... NVIDIA.
I'm going to rub my crystal ball and bet that a Chinese vendor will start making consumer cards with 256 GB of High Bandwidth Flash (HBF) and a smaller DRAM frame buffer, maybe just 8 GB or so. Flash memory would work fine for either texture storage or for AI weights, so such a card would sell like hotcakes!
> I'm reminded of the "beginning of the end" for 3dfx
The big difference now is that the fundamental shift in component pricing is making it hard for competitors to bring something with a significantly better bang for your buck factor. I strongly doubt we'd see these moves made by NVIDIA if component prices were "normal", I mean even PCBs are affected[1].
The use-case would be inference, not training. So you'd write the weights there when you load the model, and then not touch them until you replace the model with something else. Remember, flash experiences wear from writes, not reads.
Honestly, it seems like a perfectly plausible solution: With 20us latency, you'd be able to load 50k layers per second. You can get as much bandwidth as you need by putting more dies in parallel.
Qwen3.8-flash-next has 48x sparse-MoE layers, and you don't know which experts need to be loaded for each layer until the prior layer has finished. So that requires 48x 20us delays per forward pass. That admits more than 1000 forward passes per second.
With typical MTP acceptance rates, you'd be looking at maybe over 2k tok/sec as the theoretical upper bound. A big flash memory on the GPU would be great for n-gram tables as well.
Not my area of expertise at all, but I thought flash was durable for read-only use. Put your weights there, leave them there, and use other memory for the KV cache.
At the same time, game studios can't publish a game that runs steady at their target frame-rate on ps5 to save their lives (except for indie titles). I work in game dev: it's doable and makes no sense to me that tiny teams are optimizing their games better than $100M projects
The rumor is that the 16 GB 5080 will be replaced by a 24 GB version.
No more 32 GB consumer cards, the next step up is a "Pro" card with an extra digit on its MSRP.
I'm reminded of the "beginning of the end" for 3dfx where they kept doubling-down on basic scaling without fundamental improvements, and were wiped out by more efficient cards from upstart little manufacturers like... NVIDIA.
I'm going to rub my crystal ball and bet that a Chinese vendor will start making consumer cards with 256 GB of High Bandwidth Flash (HBF) and a smaller DRAM frame buffer, maybe just 8 GB or so. Flash memory would work fine for either texture storage or for AI weights, so such a card would sell like hotcakes!
> I'm reminded of the "beginning of the end" for 3dfx
The big difference now is that the fundamental shift in component pricing is making it hard for competitors to bring something with a significantly better bang for your buck factor. I strongly doubt we'd see these moves made by NVIDIA if component prices were "normal", I mean even PCBs are affected[1].
[1]: https://www.tomshardware.com/pc-components/ai-demand-trigger...
I'd be a bit worried about the durability of anything that was described as "flash", but I wonder if something analogous to Optane could work.
Intel could have had an interesting play there since they have GPUs and they have the tech for a product between flash and DRAM performance.
The use-case would be inference, not training. So you'd write the weights there when you load the model, and then not touch them until you replace the model with something else. Remember, flash experiences wear from writes, not reads.
Honestly, it seems like a perfectly plausible solution: With 20us latency, you'd be able to load 50k layers per second. You can get as much bandwidth as you need by putting more dies in parallel.
Qwen3.8-flash-next has 48x sparse-MoE layers, and you don't know which experts need to be loaded for each layer until the prior layer has finished. So that requires 48x 20us delays per forward pass. That admits more than 1000 forward passes per second.
With typical MTP acceptance rates, you'd be looking at maybe over 2k tok/sec as the theoretical upper bound. A big flash memory on the GPU would be great for n-gram tables as well.
Not my area of expertise at all, but I thought flash was durable for read-only use. Put your weights there, leave them there, and use other memory for the KV cache.
I guess we'll have to learn to do more with less!!
Related:
RTX 5090 Street Prices Top $9k as Nvidia GPU Costs Climb
https://news.ycombinator.com/item?id=50363861