I found the disagreement striking. Kaiser argues Transformers still win unless someone shows a better scaling curve while the other researchers argue the field is overfitting to current hardware and missing better architectures.
There was a back-and-forth on scaling, hardware constraints, continual learning and latent reasoning.
Very Interesting panel, Kaiser is one of the only scientists in the field with no social accounts, no public presence and who appears in the media maybe twice a year. That’s exactly why the full talk matters.
I had the same reaction when I came to know. IMO, the panel is interesting cause Kaiser wasn’t especially dismissive of the Post-Transformer side, in his rebuttal he explicitly said he was “very sympathetic” to their arguments.
He also more or less conceded Adrian’s framing that we still haven’t had a real “PageRank moment for intelligence” yet even while defending Transformers as the strongest thing that currently works and scales on the current hardware.
One of the sharpest lines in the whole debate is probably Llion’s version of the local-minimum argument: Kaiser may be right up until the day a real breakthrough arrives and then wrong forever.
I found the disagreement striking. Kaiser argues Transformers still win unless someone shows a better scaling curve while the other researchers argue the field is overfitting to current hardware and missing better architectures.
There was a back-and-forth on scaling, hardware constraints, continual learning and latent reasoning.
[dead]
Very Interesting panel, Kaiser is one of the only scientists in the field with no social accounts, no public presence and who appears in the media maybe twice a year. That’s exactly why the full talk matters.
I had the same reaction when I came to know. IMO, the panel is interesting cause Kaiser wasn’t especially dismissive of the Post-Transformer side, in his rebuttal he explicitly said he was “very sympathetic” to their arguments.
He also more or less conceded Adrian’s framing that we still haven’t had a real “PageRank moment for intelligence” yet even while defending Transformers as the strongest thing that currently works and scales on the current hardware.
One of the sharpest lines in the whole debate is probably Llion’s version of the local-minimum argument: Kaiser may be right up until the day a real breakthrough arrives and then wrong forever.