20 points | by unofficialmerve 13 hours ago
2 comments
Well, also GH Release:
- https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0
- https://news.ycombinator.com/item?id=50481715
Recently I see quite a few changes, related to KV caches.
Sounds like earlier the advice was: for coding, quantizing KV cache is bad, keep it at full F16/BF16.
Sounds like now, 8 bit KV cache is ok for coding, but the question is which one? I see f8_e4m3, q8_0. Would be great to get more info about that.
Well, also GH Release:
- https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0
- https://news.ycombinator.com/item?id=50481715
Recently I see quite a few changes, related to KV caches.
Sounds like earlier the advice was: for coding, quantizing KV cache is bad, keep it at full F16/BF16.
Sounds like now, 8 bit KV cache is ok for coding, but the question is which one? I see f8_e4m3, q8_0. Would be great to get more info about that.