I kind of assumed the model would process the text 'directly', from what I understand, wouldn't this be biasing the input based on how you tokenise as it's lossy?
I assume this tradeoff is purely for speed/compression. Or am I missing what's going on here?
I kind of assumed the model would process the text 'directly', from what I understand, wouldn't this be biasing the input based on how you tokenise as it's lossy?
I assume this tradeoff is purely for speed/compression. Or am I missing what's going on here?
GitHub: https://github.com/marcelroed/gigatoken
They even added a section ”AI Use Disclosure”, beautiful!