This seems to be a wrapper around llama.cpp with several tuned parameter settings. Most of the speedup is gained by enabling speculative decoding [1]. No amazing breakthroughs, just good tuning!
I have never understood how it is that "Ollama" manages to stay some sort of de-facto standard local hosting solution when all it does is wrap llama.cpp poorly. It's not as if llama.cpp is hard to use - it offers pre-compiled single file binaries! It implements every extant endpoint format! It has a web interface! What more could you ask for? Yet everything from agentic frameworks to vibe coded chat apps offer configurations for a variety of proprietary cloud providers, plus "Ollama". Never "llama.cpp". Why?
This seems to be a wrapper around llama.cpp with several tuned parameter settings. Most of the speedup is gained by enabling speculative decoding [1]. No amazing breakthroughs, just good tuning!
[1] https://en.wikipedia.org/wiki/Speculative_decoding
Speculative decoding was on in all of my tests, the gain came from moving the round's inputs off host memory.
I have never understood how it is that "Ollama" manages to stay some sort of de-facto standard local hosting solution when all it does is wrap llama.cpp poorly. It's not as if llama.cpp is hard to use - it offers pre-compiled single file binaries! It implements every extant endpoint format! It has a web interface! What more could you ask for? Yet everything from agentic frameworks to vibe coded chat apps offer configurations for a variety of proprietary cloud providers, plus "Ollama". Never "llama.cpp". Why?
agreed, if not for this project I've been making I would at least be using base llama.cpp
Personally I think it comes down to simplicity, but there's no reason for it's performance drops compared to llama.cpp while it's a wrapper of it.