While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI.
Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the present ROI crisis for LLM-based AI.
If you are into small local models I highly recommend vibe thinker. It's a model trained specifically for reasoning. Basically a problem solver. When compared with other models, on math problems benchmarks, it's closer to models hundred times its size than ten times its size which it beats comfortably.
It supports long contexts on limited VRAM and is blazing fast.
This seems really interesting - I was curious about this line from the website.
“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”
How does soup auto tune the hyper parameters and make some of these more complex training decisions?
The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.
Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.
Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.
Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.
Small open weight local models are the future.
While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI.
Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the present ROI crisis for LLM-based AI.
If you are into small local models I highly recommend vibe thinker. It's a model trained specifically for reasoning. Basically a problem solver. When compared with other models, on math problems benchmarks, it's closer to models hundred times its size than ten times its size which it beats comfortably.
It supports long contexts on limited VRAM and is blazing fast.
https://github.com/WeiboAI/VibeThinker
I run a fine-tuned 4B for AML compliance at community banks — the ROI math is exactly this
How much data do you need to fine tune a model?
This seems really interesting - I was curious about this line from the website.
“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”
How does soup auto tune the hyper parameters and make some of these more complex training decisions?
Looks cool, I'll try this when I get home.
I have a couple of comments about https://trysoup.dev
> Get Started for Free
Does this mean that this will not be free at some point?
The website is difficult to read (gray on black doesn't work well for me).
Why is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
Author here.
The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.
Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.
Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.
Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.
Measurement records, including the ones I threw away: https://github.com/MakazhanAlpamys/Soup/tree/main/benchmarks
Write-up: https://doi.org/10.5281/zenodo.21771064
Happy to answer anything about the scheduler or the correctness protocol.