4 points | by incognito124 9 hours ago
2 comments
One improvement the author could've made is to switch from llama.cpp to a more optimised inference engine like Strata. 60 tok/s is on the lower end for an A6000, with Strata I'd expect upwards of 90 tok/s.
I'm the author: thanks, I'll look into it, I didn't yet heae about it!
One improvement the author could've made is to switch from llama.cpp to a more optimised inference engine like Strata. 60 tok/s is on the lower end for an A6000, with Strata I'd expect upwards of 90 tok/s.
I'm the author: thanks, I'll look into it, I didn't yet heae about it!