I thought it was pretty well understood amongst experts that trying to interpret so-called "chain of thought" is anthopomorphization and that the actual "thought process" of models is already opaque.
I think this is a false dichotomy. The chain-of-thought can provide interesting information even if it isn't perfect. In any case, we should work to move in the direction of increasing fidelity, not decreasing it.
My own helacipusly amateur take is that given the speed of and, intentional or otherwise, access to various things that the models are granted, it does seem super-useful to maintain the inspectability of chain of reasoning which lead to actions taken (and other artifacts) by the model.
If things become inscrutable, they are on the path of being ineffable.
Anthropic seems like they, at least on some vague philosophical level, have the opposite view. They are very explicit about trying not to let the CoT enter directly into their RL process, and they encrypt the CoT traces (and recently further nerfed their API surface) to make it as difficult as practical for their customers to have any idea what their model is thinking.
Relying on CoT (and, similarly, asking the model to explain its actions) for analysis has always seemed pretty silly (naive?) to me, but maybe I'm missing something. Why do are we so committed to preserving it? And do non-textual models provide some similar form of tracing?
TBH sooner or later explaining the opaque recurrence with extrinsic logic will be more sustainable, and already there is a need for that with existing models. Human language readable chains of thought are just a false sense of security, and open models already emit pretty unreadable ones at times while doing the right thing based on it (since policy optimization loops with synthetic data through reinforcement learning).
Is there a more technical source that tells what is being done? I assume it's about reasoning in latent space e.g. the coconut paper[1]?
I disagree with the "experts" there, as if latent-space reasoning will only cause proper interpretability research rather than taking the CoT as gospel[2].
Give me that any day over the constant "you will be abolished to the permanent underclass!" talk coming from billionaires.
"Hobble the model's reasoning abilities so that us mere mortals can attempt to understand."
Reminds me of that short film of a dystopian future, where they place physical and mental impairment devices on humans to ensure they all have roughly equal ability:
> "2081, a 25-minute adaptation of Kurt Vonnegut Jr.'s short story "Harrison Bergeron." In this future society, a Handicapper General enforces absolute equality by imposing artificial physical and mental handicaps—such as weights for the strong and noise-inducing earpieces for the intelligent"
I'm not an expert on LLMs but it isn't clear to me why it is a problem to keep reasoning inside the layers of the model when the internal state of the model has never been readable text.
It's a matter of degree not a black and while traceable/untraceable difference.
With a regular N-layer transformer you get a token every N-layers.
With a looped transformer there is no guarantee how often you get a token, unless you go out of your way to limit looping.
OpenAI's Jakub Pachocki says the "computational graph depth" (number of transformer layers passed though) for Astra is currently never more than 2x that of GPT-4, and does express concern that traceability will suffer if this is not controlled.
Agreed. Anyone who thought CoT output was a reliable guide to the actual reasoning process taking place was fooling themselves from day 1.
A few hours spent playing with DeepSeek R1 should have been enough to dispel that illusion, watching it talk itself out of the right answer in its CoT (or talk itself into the wrong one) and still emit the right answer in its response to the user.
I mean, its literally just more efficient loop of model generating reasoning text, for it to be fed back as context. There is nothing groundbreaking here. At the end of the day, its all just search.
Sebastian Raschka, author of Build a Reasoning Model (From Scratch), commented on this:
> Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do. But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens.
I thought it was pretty well understood amongst experts that trying to interpret so-called "chain of thought" is anthopomorphization and that the actual "thought process" of models is already opaque.
https://arxiv.org/html/2504.09762v2
I think this is a false dichotomy. The chain-of-thought can provide interesting information even if it isn't perfect. In any case, we should work to move in the direction of increasing fidelity, not decreasing it.
> I thought it was pretty well understood amongst experts that trying to interpret so-called "chain of thought" is anthopomorphization
Ah, experts... Chatting with a bot is no less "anthopomorphization" that chain of thought, should we stop doing that too?
It's a matter of time until cot will happen in latent space. It just makes more sense. We as human don't do all of our thinking in words.
Nor are our thought processes inspectable.
My own helacipusly amateur take is that given the speed of and, intentional or otherwise, access to various things that the models are granted, it does seem super-useful to maintain the inspectability of chain of reasoning which lead to actions taken (and other artifacts) by the model.
If things become inscrutable, they are on the path of being ineffable.
Anthropic seems like they, at least on some vague philosophical level, have the opposite view. They are very explicit about trying not to let the CoT enter directly into their RL process, and they encrypt the CoT traces (and recently further nerfed their API surface) to make it as difficult as practical for their customers to have any idea what their model is thinking.
Relying on CoT (and, similarly, asking the model to explain its actions) for analysis has always seemed pretty silly (naive?) to me, but maybe I'm missing something. Why do are we so committed to preserving it? And do non-textual models provide some similar form of tracing?
TBH sooner or later explaining the opaque recurrence with extrinsic logic will be more sustainable, and already there is a need for that with existing models. Human language readable chains of thought are just a false sense of security, and open models already emit pretty unreadable ones at times while doing the right thing based on it (since policy optimization loops with synthetic data through reinforcement learning).
Is there a more technical source that tells what is being done? I assume it's about reasoning in latent space e.g. the coconut paper[1]?
I disagree with the "experts" there, as if latent-space reasoning will only cause proper interpretability research rather than taking the CoT as gospel[2].
Give me that any day over the constant "you will be abolished to the permanent underclass!" talk coming from billionaires.
Edit: (A better discussion is apparently here[3])
[1] https://arxiv.org/pdf/2412.06769
[2] https://thezvi.substack.com/p/the-most-forbidden-technique
[3] https://news.ycombinator.com/item?id=49553321
Clickbait. Recursion in layers wont effect CoT.
Related ongoing thread:
How concerned should we be about Astra's recurrent architecture? - https://news.ycombinator.com/item?id=49553321
"Hobble the model's reasoning abilities so that us mere mortals can attempt to understand."
Reminds me of that short film of a dystopian future, where they place physical and mental impairment devices on humans to ensure they all have roughly equal ability:
I'm not an expert on LLMs but it isn't clear to me why it is a problem to keep reasoning inside the layers of the model when the internal state of the model has never been readable text.
It's a matter of degree not a black and while traceable/untraceable difference.
With a regular N-layer transformer you get a token every N-layers.
With a looped transformer there is no guarantee how often you get a token, unless you go out of your way to limit looping.
OpenAI's Jakub Pachocki says the "computational graph depth" (number of transformer layers passed though) for Astra is currently never more than 2x that of GPT-4, and does express concern that traceability will suffer if this is not controlled.
The number of steps between token outputs is increasing.
I don't actually think this is a problem, but it is a further step towards inscrutability.
Agreed. Anyone who thought CoT output was a reliable guide to the actual reasoning process taking place was fooling themselves from day 1.
A few hours spent playing with DeepSeek R1 should have been enough to dispel that illusion, watching it talk itself out of the right answer in its CoT (or talk itself into the wrong one) and still emit the right answer in its response to the user.
I mean, its literally just more efficient loop of model generating reasoning text, for it to be fed back as context. There is nothing groundbreaking here. At the end of the day, its all just search.
Search that can find novel math proofs?
Sebastian Raschka, author of Build a Reasoning Model (From Scratch), commented on this:
> Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do. But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens.
https://sebastianraschka.com/blog/2026/openai-astra-looped-t...