I've been using Andrej Karpathy's nanochat [0] framework to chew on training tiny ternary (1.58 bit, or 'bitchat') models on my GPU.
So far I've managed to get 126 million param at full bf16, post-training quantization [1], and quantization aware training (QAT) [2].
Our QAT training produced a 126mm param model at 62MB that essentially reached performance parity with the bf16 reference model in very early training. Note, we aren't making chat bots here, or even instruction tuned bots, these are trained on just 500 million tokens, and their various params are checked for parity.
Next up is a replication of '"Two-timescale threshold flips (TTF)" [3], which is what I'm most excited for. Doing this we can avoid having full bf16 weights during training, which is appealing for my very limited VRAM budget.
My ultimate goal is a 1B param ternary model trained from scratch.
I apologize if I've gotten any terminology wrong, or misrepresented research in my explanations. I'm not a mathematician, or a ML researcher, I'm just poking around to see what is possible.
Hello!
I've been using Andrej Karpathy's nanochat [0] framework to chew on training tiny ternary (1.58 bit, or 'bitchat') models on my GPU.
So far I've managed to get 126 million param at full bf16, post-training quantization [1], and quantization aware training (QAT) [2].
Our QAT training produced a 126mm param model at 62MB that essentially reached performance parity with the bf16 reference model in very early training. Note, we aren't making chat bots here, or even instruction tuned bots, these are trained on just 500 million tokens, and their various params are checked for parity.
Next up is a replication of '"Two-timescale threshold flips (TTF)" [3], which is what I'm most excited for. Doing this we can avoid having full bf16 weights during training, which is appealing for my very limited VRAM budget.
My ultimate goal is a 1B param ternary model trained from scratch.
I apologize if I've gotten any terminology wrong, or misrepresented research in my explanations. I'm not a mathematician, or a ML researcher, I'm just poking around to see what is possible.
--- REFS
[0] https://github.com/karpathy/nanochat
[1] https://huggingface.co/papers/2606.26650 and similar (CAT-Q, PT2-LLM, etc)
[2] https://www.ibm.com/think/topics/quantization-aware-training
[3] https://github.com/oguzeldereli/ttf-ternary