Interesting. Glad to see this done, though, wonder how often large models run into these numerical bugs? They note in the paper that it's uncommon, but, I wonder if it'd affect anything still (though, I doubt it. If it was run into in practice, they would have probably shown an improvement on a larger problem than the small functions they were learning on):
> Though the extreme margins and near-zero student–teacher gaps probed by our experiments are uncommon in standard large-scale training, our fixed functions are easy drop-in replacements that can be implemented without meaningful slowdown, and may prove useful whenever models make high-confidence predictions or distill from close teachers.
Interesting. Glad to see this done, though, wonder how often large models run into these numerical bugs? They note in the paper that it's uncommon, but, I wonder if it'd affect anything still (though, I doubt it. If it was run into in practice, they would have probably shown an improvement on a larger problem than the small functions they were learning on):
> Though the extreme margins and near-zero student–teacher gaps probed by our experiments are uncommon in standard large-scale training, our fixed functions are easy drop-in replacements that can be implemented without meaningful slowdown, and may prove useful whenever models make high-confidence predictions or distill from close teachers.