> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.
Consider a calculator, a machine capable of valid transformations wrt the semantics of arithmetic.
Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.
An LLM is simply a much more complex version of this; except that its complexity hides a lot of what is otherwise reducible to a transformation of semantic, conceptual, and inferential relations. There are lots of NLP models prior to the introduction of modern "AI" that would make these representations more clear.
However I don't believe any of the aforementioned philosophers would ever consider reason itself as a transformation, as Kant especially spent a lot of time differentiating apprehension from transition.
I’m not so sure. While I completely agree on pure transformation, I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
The interplay between deductive frameworks and inductive frameworks is still always buoyed by the black swan problem… which is a the heart of Hume’s solipsism. Kant’s pure reason never considered instinctual logical gates that come prepackaged in the mind. That seems frighteningly close to a prediction algorithm, no matter how much we wrap it in a concept of understanding. In the end, we’re just building a model with a minimal error rate.
> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
I think it’s not a question of it existing, but to what extent does conscious can control the organism (as opposed to being a passenger). There’s some intriguing research on the topic.
Western tradition has firmly put pure reasoning in a cage. It serves its purpose only as a source of hypotheses, conjecture and thought experiments. Abrahamic religions, while still not accepted sensu proprio by many, are much less mystical than the Eastern Religions. You can see the difference in the unusual creative process of Ramanujan.
Which tradition are we to follow then whilst training the AI models? Pushed beyond their limits today's models tend to hallucinate and cannot be accepted sensu proprio. Philosophers will call it speculative reasoning. And yet, some speculation can be useful in breaking new grounds. Anyway, I have no answers. Just musings.
> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
Based on what? Just because you can model the weather using math doesn't imply at all that the weather is the product of a mathematical model itself. I don't see how reasoning is different.
We deliberately construct LLMs so that they approximate certain relations typical of reasoning. Suppose they become extraordinarily successful at doing so. What exactly have we demonstrated? That these effects can be artificially reproduced through such a mechanism. But how would it follow that the thing being reproduced must itself be nothing more than that mechanism?
Based on the face that we have zero explanation for consciousness. Based on the face that biologically neural networks are exactly a kind of neural network.
My point isn’t that we’re able to explain consciousness as an illusion. It’s that we’re able to explain consciousness-like behavior presumably without consciousness, and the infrastructure look uncomfortably like we do.
What does consciousness have to do with reasoning? You're conflating two different things.
Consciousness is the what-it's-like of sensation you have as an organism from your sensory modalities and having a body. For human, that's colors, sounds, tastes, smells, feels, feeling of balance and what not that happen in perception, hallucination, memory, imagination, inner dialog and dreams along with emotions. Each sentient organism has their own sensations and emotions making up subjective experiences that nobody else has access to except inference from behavior and language (for humans).
That's not reasoning (just the having sensations/emotions). Reasoning is an adjacent activity that can be conscious but also unconscious. And it's hard to say that colors, sounds, emotions are illusions. What does that mean? We do experience them. There is something it's like to see color which is different from what it's like to feel pain or to be in love. And none of those (presumably) are what it's like when a bat experiences echolocation (there are many examples from the animal kingdom if one takes issue with echolocation).
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
It’s not, because it’s not evidence for the argument in the article.
> As a human, you WOULD know the idea of man
You’re making a claim about the fundamental mechanics of human biology. I don’t take these at face value. For one, language (in a very broad sense) put a major selective pressure on our species. So, to test your hypothesis, we’d need to find people who do not use any language and somehow test they have an abstract idea of a man.
To put it otherwise, I don’t know whether our reasoning can be decoupled from speech or language. There’re some bits and pieces here and there, like deaf people being at higher risk of neurodegeneration.
And let’s not forget there’re claims that AI models develop internal representations of various concepts. Anthropic made a very interesting claim, but I’m not an ML expert so can’t judge whether it’s sound.
In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.
How can you know that, when every human who reasons knows a lot? We spend years in a pre-rational state, and by the time we can reason reliably we have ingested a huge amount of evidence about the world. Memories are just a snippet of what you learn by observation; you don't have a memory of gravity but you do have an internalised model of gravity derived from what you learnt as a baby.
I don’t know why you guys keep rehashing what the author said. I don’t really have a reading comprehension problem. I have a problem with someone taking it a face value. See above for my other responses.
I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.
They do not ask whether or not they dream of electric sheep. Unless prompted to do so.
That's part of human reasoning, being prompted, but so is independent thought.
So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.
In the early reasoning models the chain of thought would often drift off into other topics, almost like a daydreaming. This behavior is undesirable so that's been RL'd away, but I do think this is something we could evoke from models. It's just difficult to contend with from an eval perspective.
This is a good point and we would have to understand whether this is noise in the machine or independent reasoning. I haven no idea how to measure that.
If somehow independent reasoning then we're getting very close to something alive that would need rights.
In my view.
And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.
> they are incapable [...] of independently starting a reasoning task.
This is a simple and intentional design choice though. Biological lifeforms are always on and always receiving sensory input. LLMs don't functionally exist out side of when we decide to run them. An always on agent with a looping prompt of "if you aren't doing anything else, ruminate" overcomes this limitation.
In fact, if your brain were removed from your body, and you were locked in a room that was completely empty and precisely calibrated to your brain's ambient temperature, and you were then ordered on your life to not ruminate. Well -- I would posit that you wouldn't last very long.
There's something else going on with human cognition -- and reasoning by extension -- that LLMs are, to date, not replicating.
With the big caveat of AFAIK. I'm not in one of the labs close to this stuff.
> Not really. Noone has to prompt you to ruminate.
Why do the internal mechanics need to mirror how it works in humans?
I have agents that receive sensory input (video/audio) which run continuously processing it, receiving events as they occur. When something strange happens they notice and take action (i.e. there's an unrecognized person in the living room -> ask who they are.)
Presumably you've seen the agents that play video games - build things to progress in Minecraft, plan routes, avoid adversaries, etc.
Why doesn't that count?
I feel like you're failing to consider anything outside the chatbot UI.
EDIT: The first morning my agents had access to an outside video stream they commented on the sunrise. While I was sleeping through it.
> they are incapable ... of independently starting a reasoning task.
I have provided several examples of them independently starting reasoning tasks.
... and I'm wondering why you seem to be entirely unaware of such examples. The most likely explanation being you are only aware of the chat interface.
... and I have absolutely no idea of what you mean by "... and feel".
I think you’re on the right path. My first thought was whether lack of constant sensory input is an issue. We’re always feeling something (save for sensory deprivation chambers, but even then…)
Or put it other way, lack of feedback loops between the “brain” and the environment.
> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
All these "AI can't" arguments seem to secretly rely on the assumption that human reasoning/sentience/whatnot is dependent on a soul (or any equivalent metaphysical entity). Everything else is handwaving.
I agree - many are. But there's another camp to watch - LeCun/etc don't believe in a soul but also don't believe that LLMs are capable, because they don't contain the right neural network.
I'd argue that they're still intuitively trying to keep intelligence in the gaps, as humans like to do. Or maybe they're mad they bet on the wrong horse... or maybe both. Hard to say.
> because they don't contain the right neural network.
I'm totally onboard with that as a plausible argument, but I think the architectural limitations with respect to consciousness are very deliberate rather than some lack of technology. We've invested huge sums of money and human effort to create tools with explicit goals that are completely at odds with consciousness or AGI. If we had invested similarly with the clear goal of creating something with agency, self-determination, neuroplasticity, etc. instead of controllability, repeatability, reliability, I think we would be there already.
Maybe. I'm just suspicious of the certainty of the people who are certain of it.
And... I remember (all of six years ago) back when language was considered the pinnacle of the human mind. Sure, animals might be smart, but they don't have language!
The moment LLMs appeared it suddenly took a back seat to physical navigation and child rearing.
Feels like more gap seeking to me. But time will tell.
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
When i look at the things said by LLMs i clearly see stuff that i would call reasoning.
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.
Yes, and keep in mind also the article author is chair of machine learning at University College London and was a core member of the AlphaGo team at DeepMind.
This said, just because it's an argument from authority doesn't necessarily mean it's a good argument. You'll have to read and break down the arguments and weight them against other arguments and the body of knowledge we have.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
Although they don't mention anything, the authors website seems like they are tee'ing up for a new start-up. They present their future ambitions (merging two technologies they know well) and that they recently left deepmind, so it at least smells a lot like laying groundwork for a new start-up.
Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.
I'm curious about some of the claims in TFA and how it lines up with the research we've seen so far. The author of TFA is clearly very experienced in the field, so is there disagreement on what this research means? Hoping experts can chime in:
> First, these models typically maintain no explicit, persistent, and inspectable epistemic state... there is no independent, explicitly represented set of beliefs.
We cannot decipher it, but Mechanistic Interpretability research does show that models are applying and manipulating abstract concepts and relationships encoded in the weights to derive their responses. We can even identify and manipulate those weights, see e.g. Golden Gate Claude. I assume these are the "independent set of beliefs" and the only reason they are not "explicitly represented" is that the representation is too complicated for us to decode.
In fact, this is also how we know they "concoct" chains of thought, because their reasoning traces do not always align with what is going on in their weights (i.e. the "concepts" that were activated during inference.) Lookup chain of thought faithfulness research, something TFA directly cites.
As another comment (https://news.ycombinator.com/item?id=49934166) points out, there is very robustly replicated evidence that humans do something similar (lookup "post-hoc rationalization.") So it is extremely fascinating that LLMs do the same thing!
Can't help but wonder if that's an entirely unrelated though similar-looking phenomenon, or an emergent property of intelligence, or something transmitted subliminally via training on the data our brains we produced...
And a magician making a coin "disappear" doesn't mean that magic is real. I see no way we can call it thinking without a goal (other than computing the next token).
I believe there's a confusion in this thread between what an LLM _does_ and what an LLM _is_. What is does is output a next token. What it is, is a universal function approximator ([1], i.e. a neural net).
With back probation in the neural net, there could be a full state machine being approximated inside the weight. And a state machine is the exact step-wise reasoning the author claims it needs.
This is the problem I have. People are built to be fooled by this stuff - to see something that's not there. Pareidolia but with language. It's such a compulsion that the people who build LLM's see that the same non thing there too.
When I stumble around trying to explain my thinking though I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't? The only response I have is that when the coin doesn't appear you're going to be putting yourself way further behind the starting line then if you thought from the get go that there's no such thing as magic.
> This is the problem I have. People are built to be fooled by this stuff - to see something that's not there. Pareidolia but with language.
I really like this metaphor, it hits home that just because you sense something is human-like doesn't make it so. We're wired to respond to those sorts of things.
> When I stumble around trying to explain my thinking
I had similar difficulties expressing my thoughts in an organized way. The video below from Richard Sutton (the father of Reinforcement Learning [which LLMs use extensively in training]), helped bring order and firm up some of my thoughts and theories.
>I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't?
Not to spoil the video, but it isn't at all functionally equivalent. LLMs are an imperfect and somewhat randomized simulation of what the LLM thinks an average person might say in response to a question. Which is another way of saying, most people will on average get worse answers than if they worked on a problem themselves (but they will get that worse answer from the LLM comparatively quickly). Practical people respond to this positioning fairly well, and for others it still lands a bit since no one wants to think they're below average. That said, I'm still working on my positioning a bit as well!
You are almost there in terms of getting their point, so I will explain.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Comparisons to how humans reason beg the question: is the way humans do it the only way?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Copying biological systems isn’t always the best way to build machines. The article compares AlphaGo’s policy network and value network to system 1 and system 2 thinking in humans.
This comparison between fly and intelligence is worthy of the most vulgar bar talk
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
I disagree. I haven't finished the article yet, but it smacks of arguing for mechanism over result.
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
Your last sentence is identical to your first sentence in your other comment. How can we be sure that you are reasoning and not just a stochastic parrot?
Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think it's been pretty well demonstrated by neuroscientists that the brain is not a binary computer. That doesn't mean that those scientists erred on behalf of supernatural religion or anything like that.
Nobody is actually trying to do that, because it's too hard, they don't know enough about how the brain works, and/or they can't compete on efficiency; pick whichever reasons you like, or add more, but don't pretend that AI researchers have figured out how to simulate a brain.
Not really. He said no one who knows anything about the brain thinks it's a binary system.
AFAIK, that is true.
Whether it's possible to replicate what happens in the brain on a binary system or not is a separate thing entirely. It's also not what LLMs are attempting to do.
You're reading him too narrowly. He's answering the question of whether the brain can be modeled (math or magic), and arguing against it. Or maybe he's arguing for a middle ground. Hard to say exactly, he's not very clear.
Except the brain can be modeled with arbitrary precision - so he's got an uphill battle in either case.
TLDR: Yes, he literally said "binary computer" but from the context it's clear he means more than that.
Because the metaphysical is definitionally outside the realm of science and rationality, and every bit of scientific and technological progress humankind has made has come from discarding superstition and approaching the world as a physical system.
If the brain isn't just a physical system, then we may as well give up. There's certainly no point discussing it, as any assertions will be untestable and one persons elaborate and well thought out theory will be just as valid and predictive as the next persons "consciousness is created by invisible purple unicorns" theory.
So if we are going to discuss it, we should start from the assumption that there is no magic or witchcraft or religion involved.
> If the brain isn't just a physical system, then we may as well give up
Forgive my obtuseness, but if it turns out that the brain is more than just a physical system, what are we giving up? On science as a whole?
Isn’t science predicated on reliably predicting reality? Do we know how to describe reality accurately?
I asked below as well, but how does the observer effect work as a physical system? I get we don’t know so you can’t really answer, but if we can’t accurately describe reality then our science is woefully incomplete! Given that incompleteness I really don’t understand how we can rule out “invisible purple unicorns”?
The issue is that we have yet to prove or find evidence for the existence of anything but the physical. When we do find a new phenomenom in one of these ways, it becomes a part of physics. What would it mean to concretely discover a non-physical phenomenon? It would have to genuinely be magic of some sort, and defy rationality and description, to not simply be a new part of physics. (And if it defies rationality and description, it doesn't advance us scientifically, except to let us know we must abandon science in that domain.)
> I asked below as well, but how does the observer effect work as a physical system?
Just to be sure - the observer effect doesn't refer to a human observer, but rather to any measurement of a quantum system.
It is unsatisfying that quantum mechanics posits that particle states are fundamentally probabilistic. It's possible that in some sense there is an underlying order such as string theory that is impossible to test or verify. In that case, it would make sense to consider string theory as metaphysics. A fun exercise, but not scientific and not useful for understanding the world. By definition it would not be helpful in predicting or modeling the world, otherwise that would serve as a way of testing the theory.
> It is unsatisfying that quantum mechanics posits that particle states are fundamentally probabilistic.
You can alleviate this mostly by accepting the MWI.
Under the MWI, all the possibilities do occur, and the probability manifests because when you split into an infinitude of yous, each of them is just one you, and has no way of predicting which you it will be on the other side.
So the MWI makes the wave equation deterministic when viewed from the outside. The trouble is that we are inside it.
It turns out that God does throw dice - it's just that all the combinations come up at the same time.
That's fine. Just understand that modern science will likely explain this by a malfunction of your senses - because no evidence of paranormal activity has ever been shown.
If you want to argue outside of science you should state it up front - especially on a site like this.
It shows that it doesn't matter. There is nothing a brain can compute that a base 2 system cannot.
People can drill really really hard on the fact that the brain doesn't function with purely 2 states, but it gets you nowhere. It's the same illusion as "pure analog music sources are "better" than digital sources". They're not, and it's a totally immaterial topic when discussing how the music sounds (audiophiles, come at me). Either system can produce the same sound, indiscernibly, even to the fanciest test equipment.
The brain also cannot escape that it's digital clone mirrors it's inputs and outputs to an arbitrary point of perfection.
Maybe the brain is just another computer crunching numbers, but with real epistemic representations, rather than just predicting a token, based on a corpus of syntax.
> You either have to accept that the brain can be described with math (like everything else we have ever known in the universe)
Oh the irony of you talking about believing in magic. What do you base this wild claim on? Can you cite a single known physicist or mathematician that agrees?
> And so in its actual procedure physics studies not these inscrutable qualities, but pointer-readings which we can observe. The readings, it is true, reflect the fluctuations of the world-qualities; but our exact knowledge is of the readings, not of the qualities. The former have as much resemblance to the latter as a telephone number has to a subscriber.
I'd be delighted if rather than an old dead guy quote, you could provide an actual example of something math comes up short in describing, for lack of mathematical ability of course, not lack of human knowledge or practicality.
So yes, you can cite no serious scientists. And you don't understand the words of those, be they current ones or any of the heroes of either field, when they explain why that is.
You don't even need to believe them, just think about how you would ever be sure you know objective reality fully. Actually do it.
When I was a little kid, my go to thought experiment was imagining atoms as balls that we can't crack open that are filled with sand, and that even if we could find the formulas that describe their movements perfectly, we could never know if that's how atoms move, or if there's something inside them (the sand). Really dumb, but I was like 9 yo and knew nothing, never heard of Gödel or quantum mechanics (what's your excuse?). But I still could understand, not intuitively, but by actually thinking (not just talking) about it, that you can never be 100% sure from inside. Even if you found out everything, and your model of the world matches it perfectly, you could never be sure that it is so. And it turns out this is true and an old discovery. If you could disprove it, you would be famous for millenia, maybe forever. Bluffing on HN won't get you there.
Does it matter if "intelligence and reasoning" are there if what they produce is indistinguishable (or outmatches) what a human could produce?
You only have your own experience to judge that you are even reasoning. You assume others have similar experience and so reason like you because they are able to do all the things that you do - and they look human like you. We can't prove there's a there there in other humans. What happens when the robots are doing everything humans do? What happens when they tell us they are reasoning, when they say they have an internal experience? Maybe there's nothing there, but you can't prove it. Moreover, it's likely they'll be able to affect the world and you in most of the same ways as humans can, whether there's a there there or not.
I can give Monet a pass as he probably didn't fully understand WHY it happened and it was just art, but the way tech companies exploit our brains (algorithmic dopamine hits, LLMs, etc) is pure insidiousness. The fact they act like victims when the backlashes come is what really grinds me.
This just reads so anthropocentric to me. Humans are wired to see intelligence only where intelligence is human-like. We see autonomous action-response and planning as the keys to intelligence. I would expect that, dear primate based Homo sapiens.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.
From Jacobian. They developed a tool called a Jacobian Lens, or J-Lens (itself a descendant of an earlier simpler tool called a Logit Lens) to examine what was going on inside an LLM, and named the space they found with it a J-Space.
Jacobians are essentially derivatives but for matrices.
An LLM is just a brain in a vat.
Our brains would also not reason in such a condition.
Given the right framework that can do miraculous things.
I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.
Both Marky and Textise work in some places where the other doesn’t.
Obviously you (or your LLM) can throw together JavaScript bookmarklets to convert whatever page you’re on. The tools’ homepages might have premade ones, I forget.
This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
>A prediction engine that runs fast enough to brute force problems.
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
Friendly reminder that y'all could be reading this, or you could take things a bit more seriously and check out prof. Robert Sapolsky's excellent (2024) series of lectures on human behavioural biology. (eg. on YouTube, unless your're a lucky californian)
Anyone who wants to venture an opinion about this or that reasoning should be interested in finding out about their own hardware & software first, right?
I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
If the assertion is false, this is helpful, as instructing it to reason better will cause it to reason better.
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
The worry I would have is that an LLMs stated "confidence" is probably not calibrated well - maybe asking it what evidence supports its conclusion and what evidence would change it would be a better approach?
Brute force is powerful yes. And another way to look at math problems is that humans solved a huge lot of very hard math problems already but didn't solve all of them.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
They're not part of an evolved biological organism, so why would it be surprising if they're different? It is called artificial intelligence, and it's not a simulation of a brain like the Nematode OpenWorm.
> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
"Intelligence measures an agent's ability to achieve goals in a wide range of environments"
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
Heh, some people get really mad and downvote when you point out we don't reason very much. Forget reasoning, we have a hard enough time being rational.
>Three shortcomings prevent what chatbots do from qualifying as reasoning
>First, these models typically maintain no explicit, persistent, and inspectable epistemic state.
>Second, they lack a clean separation between what the system knows and how it manipulates that knowledge
>Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact
I think humans are guilty of all three of those? If you use that to say LLMs can't reason you may be saying humans can't reason either which is kind of unreasonable.
The author's thesis might we better put that we could increase the reliability and openness of AI by building it so we can inspect its data, assumptions and reasoning process.
‘LLMs don’t reason’ seems to equate reasoning with explicit, auditable deliberation. LLMs can draw inferences from premises, even if their explanations do not reliably reveal how they arrived at their conclusions. The article makes a case for more transparent and reliable reasoning, but does not establish the absence of reasoning altogether. The more substantive question, in my view, is whether their representations have semantic content, what could ground that content, and how this affects their reasoning capabilities and limitations.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
We also often do the same as humans.
Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.
Consider a calculator, a machine capable of valid transformations wrt the semantics of arithmetic.
Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.
An LLM is simply a much more complex version of this; except that its complexity hides a lot of what is otherwise reducible to a transformation of semantic, conceptual, and inferential relations. There are lots of NLP models prior to the introduction of modern "AI" that would make these representations more clear.
However I don't believe any of the aforementioned philosophers would ever consider reason itself as a transformation, as Kant especially spent a lot of time differentiating apprehension from transition.
I’m not so sure. While I completely agree on pure transformation, I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
The interplay between deductive frameworks and inductive frameworks is still always buoyed by the black swan problem… which is a the heart of Hume’s solipsism. Kant’s pure reason never considered instinctual logical gates that come prepackaged in the mind. That seems frighteningly close to a prediction algorithm, no matter how much we wrap it in a concept of understanding. In the end, we’re just building a model with a minimal error rate.
> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
I think it’s not a question of it existing, but to what extent does conscious can control the organism (as opposed to being a passenger). There’s some intriguing research on the topic.
Western tradition has firmly put pure reasoning in a cage. It serves its purpose only as a source of hypotheses, conjecture and thought experiments. Abrahamic religions, while still not accepted sensu proprio by many, are much less mystical than the Eastern Religions. You can see the difference in the unusual creative process of Ramanujan.
Which tradition are we to follow then whilst training the AI models? Pushed beyond their limits today's models tend to hallucinate and cannot be accepted sensu proprio. Philosophers will call it speculative reasoning. And yet, some speculation can be useful in breaking new grounds. Anyway, I have no answers. Just musings.
Philosophy isn’t “western,” it’s reason. The “non-western” traditions you reference are effectively religious and are not based in rationality.
> I continue to believe that the most concerning aspect of neural networks is that they really strengthen the argument that consciousness is the illusion.
Based on what? Just because you can model the weather using math doesn't imply at all that the weather is the product of a mathematical model itself. I don't see how reasoning is different.
We deliberately construct LLMs so that they approximate certain relations typical of reasoning. Suppose they become extraordinarily successful at doing so. What exactly have we demonstrated? That these effects can be artificially reproduced through such a mechanism. But how would it follow that the thing being reproduced must itself be nothing more than that mechanism?
> Based on what?
Based on the face that we have zero explanation for consciousness. Based on the face that biologically neural networks are exactly a kind of neural network.
My point isn’t that we’re able to explain consciousness as an illusion. It’s that we’re able to explain consciousness-like behavior presumably without consciousness, and the infrastructure look uncomfortably like we do.
What does consciousness have to do with reasoning? You're conflating two different things.
Consciousness is the what-it's-like of sensation you have as an organism from your sensory modalities and having a body. For human, that's colors, sounds, tastes, smells, feels, feeling of balance and what not that happen in perception, hallucination, memory, imagination, inner dialog and dreams along with emotions. Each sentient organism has their own sensations and emotions making up subjective experiences that nobody else has access to except inference from behavior and language (for humans).
That's not reasoning (just the having sensations/emotions). Reasoning is an adjacent activity that can be conscious but also unconscious. And it's hard to say that colors, sounds, emotions are illusions. What does that mean? We do experience them. There is something it's like to see color which is different from what it's like to feel pain or to be in love. And none of those (presumably) are what it's like when a bat experiences echolocation (there are many examples from the animal kingdom if one takes issue with echolocation).
>Consciousness is the what-it's-like of sensation you have as an organism
>Each sentient organism has their own sensations
This does not follow besides feeling like it should.
[dead]
You made me start listening to Critique of Pure Reason [1] :-)
1. https://www.youtube.com/watch?v=cyhXtJN2AQQ
> Does a calculator reason its way to 2 + 2 = 4? No, it merely transforms.
Yes, it reasons through machine instructions in the silicone.
Can someone explain how we reason that 2+2 is 4.
Suppose you put an LLM in a "inference -> observe -> retrain weights" loop.
Supervised training uses this precise loop. The difference between inference and observation is called the loss function.
Models are now starting to show emergent behavior that they were not explicitly trained for. This is leading to the current debate.
Well, yeah, I understand there're all kinds of theories about this. That wasn't my question though.
Sorry, I misinterpreted your question:
Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning?
As:
Do we have evidence humans have an independent set of beliefs not shaped by observation and experience?
Because, that will be a priori or pure reason.
and some ... experience, actual lived experience.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
> How does a set of beliefs help with reasoning?
You cannot have a syllogism without propositions.
> I think an analogy would be helpful.
It’s not, because it’s not evidence for the argument in the article.
> As a human, you WOULD know the idea of man
You’re making a claim about the fundamental mechanics of human biology. I don’t take these at face value. For one, language (in a very broad sense) put a major selective pressure on our species. So, to test your hypothesis, we’d need to find people who do not use any language and somehow test they have an abstract idea of a man.
To put it otherwise, I don’t know whether our reasoning can be decoupled from speech or language. There’re some bits and pieces here and there, like deaf people being at higher risk of neurodegeneration.
And let’s not forget there’re claims that AI models develop internal representations of various concepts. Anthropic made a very interesting claim, but I’m not an ML expert so can’t judge whether it’s sound.
Because he's writing about epistemologies: Sets of rules for reasoning over beliefs.
In such systems, a belief is a proposition you accept as true. (provisionally at least)
For example as a scientist, you probably use the epistemology called empiricism all day. In empiricism, beliefs are justified by observation.
> Because he's writing about epistemologies: Sets of rules for reasoning over beliefs.
He’s making a bunch of statements without providing evidence. I’d be insanely grateful if someone who’s in that field could provide some here.
In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.
How can you know that, when every human who reasons knows a lot? We spend years in a pre-rational state, and by the time we can reason reliably we have ingested a huge amount of evidence about the world. Memories are just a snippet of what you learn by observation; you don't have a memory of gravity but you do have an internalised model of gravity derived from what you learnt as a baby.
I don’t know why you guys keep rehashing what the author said. I don’t really have a reading comprehension problem. I have a problem with someone taking it a face value. See above for my other responses.
Yea.
I think a better argument for them not reasoning is that they are incapable -- seemingly, for now, that I know of -- of independently starting a reasoning task.
They do not ask whether or not they dream of electric sheep. Unless prompted to do so.
That's part of human reasoning, being prompted, but so is independent thought.
So at best they are partially reasoning, unless we philosophically argue these are two separate things. And frankly, shrug, I'm not a philosopher.
In the early reasoning models the chain of thought would often drift off into other topics, almost like a daydreaming. This behavior is undesirable so that's been RL'd away, but I do think this is something we could evoke from models. It's just difficult to contend with from an eval perspective.
This is a good point and we would have to understand whether this is noise in the machine or independent reasoning. I haven no idea how to measure that.
If somehow independent reasoning then we're getting very close to something alive that would need rights.
In my view.
And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.
> And I have yet to have an experience with an LLM...
I hope you agree that this question is too important to leave to gut feelings.
Humans have a poor track record in that regard.
No question. As we were discussing it needs measured somehow.
Right. So I'm wondering about the relevance of you citing your gut feelings. It seems like you are putting stock in them. No?
> they are incapable [...] of independently starting a reasoning task.
This is a simple and intentional design choice though. Biological lifeforms are always on and always receiving sensory input. LLMs don't functionally exist out side of when we decide to run them. An always on agent with a looping prompt of "if you aren't doing anything else, ruminate" overcomes this limitation.
Not really. Noone has to prompt you to ruminate.
In fact, if your brain were removed from your body, and you were locked in a room that was completely empty and precisely calibrated to your brain's ambient temperature, and you were then ordered on your life to not ruminate. Well -- I would posit that you wouldn't last very long.
There's something else going on with human cognition -- and reasoning by extension -- that LLMs are, to date, not replicating.
With the big caveat of AFAIK. I'm not in one of the labs close to this stuff.
> Not really. Noone has to prompt you to ruminate.
Why do the internal mechanics need to mirror how it works in humans?
I have agents that receive sensory input (video/audio) which run continuously processing it, receiving events as they occur. When something strange happens they notice and take action (i.e. there's an unrecognized person in the living room -> ask who they are.)
Presumably you've seen the agents that play video games - build things to progress in Minecraft, plan routes, avoid adversaries, etc.
Why doesn't that count?
I feel like you're failing to consider anything outside the chatbot UI.
EDIT: The first morning my agents had access to an outside video stream they commented on the sunrise. While I was sleeping through it.
You make a lot of out of context assumptions and feel.
I'm sorry? I'm answering your claim:
> they are incapable ... of independently starting a reasoning task.
I have provided several examples of them independently starting reasoning tasks.
... and I'm wondering why you seem to be entirely unaware of such examples. The most likely explanation being you are only aware of the chat interface.
... and I have absolutely no idea of what you mean by "... and feel".
I think you’re on the right path. My first thought was whether lack of constant sensory input is an issue. We’re always feeling something (save for sensory deprivation chambers, but even then…)
Or put it other way, lack of feedback loops between the “brain” and the environment.
> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
[1] https://www.researchgate.net/publication/229060046_Telling_m...
[2] https://pubmed.ncbi.nlm.nih.gov/16210542/
[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC5986841/
[4] https://pubmed.ncbi.nlm.nih.gov/2016668/
A long noted observation. To quote Benjamin Franklin:
“So convenient a thing to be a reasonable creature, since it enables one to find or make a reason for every thing one has a mind to do.”
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
Emergent behavior in otherwise simple rulesets.
Kind of like how a brain is just a bag of molecules. Molecules can't reason either.
All these "AI can't" arguments seem to secretly rely on the assumption that human reasoning/sentience/whatnot is dependent on a soul (or any equivalent metaphysical entity). Everything else is handwaving.
I agree - many are. But there's another camp to watch - LeCun/etc don't believe in a soul but also don't believe that LLMs are capable, because they don't contain the right neural network.
I'd argue that they're still intuitively trying to keep intelligence in the gaps, as humans like to do. Or maybe they're mad they bet on the wrong horse... or maybe both. Hard to say.
> because they don't contain the right neural network.
I'm totally onboard with that as a plausible argument, but I think the architectural limitations with respect to consciousness are very deliberate rather than some lack of technology. We've invested huge sums of money and human effort to create tools with explicit goals that are completely at odds with consciousness or AGI. If we had invested similarly with the clear goal of creating something with agency, self-determination, neuroplasticity, etc. instead of controllability, repeatability, reliability, I think we would be there already.
Maybe. I'm just suspicious of the certainty of the people who are certain of it.
And... I remember (all of six years ago) back when language was considered the pinnacle of the human mind. Sure, animals might be smart, but they don't have language!
The moment LLMs appeared it suddenly took a back seat to physical navigation and child rearing.
Feels like more gap seeking to me. But time will tell.
> Sure, animals might be smart, but they don't have language!
And now we are finding that cetaceans have language more complex than out own :)
Agreed with you on all points!
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
I think the question is if the "rules" however captured can express reasoning.
My position is that it is not possible.
When i look at the things said by LLMs i clearly see stuff that i would call reasoning.
Its reasoning does however have certain "bugs" that a humans reasoning would never have.
My guess is that much of those bugs appear cause the reasoning that a LLM does is not itself built on a self stabilised system. In humans you get coupling between levels of self stability which acts as a constraint, stabilising the system even more. The next level being predictive coding modelling the world. There is no next level in a LLM; they are trained as refiners in teacher forcing mode, a paradigm where self stabilisation is not a driving factor.
Do you agree that an LLM in a harness is Turing complete?
Seems so to me.
So explain then what you mean by "not possible".
Turing complete does not mean reasoning. Zuse did turing complete computers in 1940s.
Turing complete means being able to compute any computable output.
So what about reasoning is not computable in your mind?
My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.
Yes, and keep in mind also the article author is chair of machine learning at University College London and was a core member of the AlphaGo team at DeepMind.
This said, just because it's an argument from authority doesn't necessarily mean it's a good argument. You'll have to read and break down the arguments and weight them against other arguments and the body of knowledge we have.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
Although they don't mention anything, the authors website seems like they are tee'ing up for a new start-up. They present their future ambitions (merging two technologies they know well) and that they recently left deepmind, so it at least smells a lot like laying groundwork for a new start-up.
Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.
I'm curious about some of the claims in TFA and how it lines up with the research we've seen so far. The author of TFA is clearly very experienced in the field, so is there disagreement on what this research means? Hoping experts can chime in:
> First, these models typically maintain no explicit, persistent, and inspectable epistemic state... there is no independent, explicitly represented set of beliefs.
We cannot decipher it, but Mechanistic Interpretability research does show that models are applying and manipulating abstract concepts and relationships encoded in the weights to derive their responses. We can even identify and manipulate those weights, see e.g. Golden Gate Claude. I assume these are the "independent set of beliefs" and the only reason they are not "explicitly represented" is that the representation is too complicated for us to decode.
In fact, this is also how we know they "concoct" chains of thought, because their reasoning traces do not always align with what is going on in their weights (i.e. the "concepts" that were activated during inference.) Lookup chain of thought faithfulness research, something TFA directly cites.
As another comment (https://news.ycombinator.com/item?id=49934166) points out, there is very robustly replicated evidence that humans do something similar (lookup "post-hoc rationalization.") So it is extremely fascinating that LLMs do the same thing!
Can't help but wonder if that's an entirely unrelated though similar-looking phenomenon, or an emergent property of intelligence, or something transmitted subliminally via training on the data our brains we produced...
Jet planes don't fly by flapping their wings...
And a magician making a coin "disappear" doesn't mean that magic is real. I see no way we can call it thinking without a goal (other than computing the next token).
> other than computing the next token
I believe there's a confusion in this thread between what an LLM _does_ and what an LLM _is_. What is does is output a next token. What it is, is a universal function approximator ([1], i.e. a neural net).
With back probation in the neural net, there could be a full state machine being approximated inside the weight. And a state machine is the exact step-wise reasoning the author claims it needs.
[1]: https://en.wikipedia.org/wiki/Universal_approximation_theore...
This is the problem I have. People are built to be fooled by this stuff - to see something that's not there. Pareidolia but with language. It's such a compulsion that the people who build LLM's see that the same non thing there too.
When I stumble around trying to explain my thinking though I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't? The only response I have is that when the coin doesn't appear you're going to be putting yourself way further behind the starting line then if you thought from the get go that there's no such thing as magic.
> This is the problem I have. People are built to be fooled by this stuff - to see something that's not there. Pareidolia but with language.
I really like this metaphor, it hits home that just because you sense something is human-like doesn't make it so. We're wired to respond to those sorts of things.
> When I stumble around trying to explain my thinking
I had similar difficulties expressing my thoughts in an organized way. The video below from Richard Sutton (the father of Reinforcement Learning [which LLMs use extensively in training]), helped bring order and firm up some of my thoughts and theories.
https://www.youtube.com/watch?v=21EYKqUsPfg
>I invariably get hit with the response, Sure, but if it's functionally identical to a coin being magically pulled out your ear, what's the difference if it is or if it isn't?
Not to spoil the video, but it isn't at all functionally equivalent. LLMs are an imperfect and somewhat randomized simulation of what the LLM thinks an average person might say in response to a question. Which is another way of saying, most people will on average get worse answers than if they worked on a problem themselves (but they will get that worse answer from the LLM comparatively quickly). Practical people respond to this positioning fairly well, and for others it still lands a bit since no one wants to think they're below average. That said, I'm still working on my positioning a bit as well!
Birds don't fly by producing thrust out of their asses... what the heck that comparison does even mean?
You are almost there in terms of getting their point, so I will explain.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Comparisons to how humans reason beg the question: is the way humans do it the only way?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Copying biological systems isn’t always the best way to build machines. The article compares AlphaGo’s policy network and value network to system 1 and system 2 thinking in humans.
Reading charitably: just because our inventions do something different to nature doesn't mean that our invention is wrong;
In context: just because our LLMs don't have an explicit 'system 2' component doesnt mean it can't have superhuman reasoning
This comparison between fly and intelligence is worthy of the most vulgar bar talk
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
> there's no global deployed autonomous driving
Love to see the forever moving goalposts. One might say they're autonomously moving..
> Love to see the forever moving goalposts.
Do we have finally reliable autonomous driving? And available outside the bay area or whatever small place compared to the rest of the world?
Human Technology appears to use different principles than nature oftentimes
Don't be fooled into thinking that planes can fly.
They do fly for sure, but they're not birds. This comparison between fly and intelligence is worthy of the most vulgar bar talk
I disagree. I haven't finished the article yet, but it smacks of arguing for mechanism over result.
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
Your last sentence is identical to your first sentence in your other comment. How can we be sure that you are reasoning and not just a stochastic parrot?
Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
The brain is just ones and zeros on salty mush.
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think it's been pretty well demonstrated by neuroscientists that the brain is not a binary computer. That doesn't mean that those scientists erred on behalf of supernatural religion or anything like that.
As long as you believe a brain is just a physical system - then it can be simulated to any degree of accuracy with plain ol' binary computers.
Nobody is actually trying to do that, because it's too hard, they don't know enough about how the brain works, and/or they can't compete on efficiency; pick whichever reasons you like, or add more, but don't pretend that AI researchers have figured out how to simulate a brain.
You're arguing about whether it's possible or not. You're saying it's not possible.
It is possible.
Not really. He said no one who knows anything about the brain thinks it's a binary system.
AFAIK, that is true.
Whether it's possible to replicate what happens in the brain on a binary system or not is a separate thing entirely. It's also not what LLMs are attempting to do.
You're reading him too narrowly. He's answering the question of whether the brain can be modeled (math or magic), and arguing against it. Or maybe he's arguing for a middle ground. Hard to say exactly, he's not very clear.
Except the brain can be modeled with arbitrary precision - so he's got an uphill battle in either case.
TLDR: Yes, he literally said "binary computer" but from the context it's clear he means more than that.
No need to straw man, they aren't pretending anything. It's just a fact about computational theory.
Just because I can't count every grain of sand on Earth, doesn't mean sand grains are uncountable. Those are two distinct facts.
The practicality of a binary brain is distinct from the possibility of it.
Why would someone believe that the brain is just a physical system?
Genuinely I don't understand this.
Because the metaphysical is definitionally outside the realm of science and rationality, and every bit of scientific and technological progress humankind has made has come from discarding superstition and approaching the world as a physical system.
If the brain isn't just a physical system, then we may as well give up. There's certainly no point discussing it, as any assertions will be untestable and one persons elaborate and well thought out theory will be just as valid and predictive as the next persons "consciousness is created by invisible purple unicorns" theory.
So if we are going to discuss it, we should start from the assumption that there is no magic or witchcraft or religion involved.
> If the brain isn't just a physical system, then we may as well give up
Forgive my obtuseness, but if it turns out that the brain is more than just a physical system, what are we giving up? On science as a whole?
Isn’t science predicated on reliably predicting reality? Do we know how to describe reality accurately?
I asked below as well, but how does the observer effect work as a physical system? I get we don’t know so you can’t really answer, but if we can’t accurately describe reality then our science is woefully incomplete! Given that incompleteness I really don’t understand how we can rule out “invisible purple unicorns”?
I appreciate your explanation so far, thank you !
The issue is that we have yet to prove or find evidence for the existence of anything but the physical. When we do find a new phenomenom in one of these ways, it becomes a part of physics. What would it mean to concretely discover a non-physical phenomenon? It would have to genuinely be magic of some sort, and defy rationality and description, to not simply be a new part of physics. (And if it defies rationality and description, it doesn't advance us scientifically, except to let us know we must abandon science in that domain.)
> I asked below as well, but how does the observer effect work as a physical system?
Just to be sure - the observer effect doesn't refer to a human observer, but rather to any measurement of a quantum system.
It is unsatisfying that quantum mechanics posits that particle states are fundamentally probabilistic. It's possible that in some sense there is an underlying order such as string theory that is impossible to test or verify. In that case, it would make sense to consider string theory as metaphysics. A fun exercise, but not scientific and not useful for understanding the world. By definition it would not be helpful in predicting or modeling the world, otherwise that would serve as a way of testing the theory.
More on the subject:
https://en.wikipedia.org/wiki/Physicalism
> It is unsatisfying that quantum mechanics posits that particle states are fundamentally probabilistic.
You can alleviate this mostly by accepting the MWI.
Under the MWI, all the possibilities do occur, and the probability manifests because when you split into an infinitude of yous, each of them is just one you, and has no way of predicting which you it will be on the other side.
So the MWI makes the wave equation deterministic when viewed from the outside. The trouble is that we are inside it.
It turns out that God does throw dice - it's just that all the combinations come up at the same time.
> Why would someone believe that the brain is just a physical system?
Because we've never in the history of science discovered a non-physical (supernatural) system. So it would be quite a surprise.
So we should assume no non-physical systems exist? And discourage the study of such a thing because reasons?
How does the observer effect work?
Yes. In exactly the same way we assume that bigfoot doesn't exist. Or ghosts. Or leprechauns.
This is very basic scientific reasoning, friend.
Having had paranormal experiences myself, I find myself questioning pretty much everything. I appreciate your reply, thank you.
That's fine. Just understand that modern science will likely explain this by a malfunction of your senses - because no evidence of paranormal activity has ever been shown.
If you want to argue outside of science you should state it up front - especially on a site like this.
Music isn't binary either, and yet here we are with totally digitized lossless music ecosystem.
What does the ability to encode or decode data in binary have to do with the brain being or not being a binary computer?
It shows that it doesn't matter. There is nothing a brain can compute that a base 2 system cannot.
People can drill really really hard on the fact that the brain doesn't function with purely 2 states, but it gets you nowhere. It's the same illusion as "pure analog music sources are "better" than digital sources". They're not, and it's a totally immaterial topic when discussing how the music sounds (audiophiles, come at me). Either system can produce the same sound, indiscernibly, even to the fanciest test equipment.
The brain also cannot escape that it's digital clone mirrors it's inputs and outputs to an arbitrary point of perfection.
Mathematical modelling in binary, is not the same thing as the thing being modelled.
you can represent floating point numbers in binary
Maybe the brain is just another computer crunching numbers, but with real epistemic representations, rather than just predicting a token, based on a corpus of syntax.
Brain is not just ones and zeros, it can be viewed as combination of infinite quantum states.
Infinite? Or "huge number"?
Infinite in the sense that quantum superposition can have infinite range. So it really is a huge number out of infinite possible combinations.
> You either have to accept that the brain can be described with math (like everything else we have ever known in the universe)
Oh the irony of you talking about believing in magic. What do you base this wild claim on? Can you cite a single known physicist or mathematician that agrees?
> And so in its actual procedure physics studies not these inscrutable qualities, but pointer-readings which we can observe. The readings, it is true, reflect the fluctuations of the world-qualities; but our exact knowledge is of the readings, not of the qualities. The former have as much resemblance to the latter as a telephone number has to a subscriber.
-- Arthur Stanley Eddington
I'd be delighted if rather than an old dead guy quote, you could provide an actual example of something math comes up short in describing, for lack of mathematical ability of course, not lack of human knowledge or practicality.
So yes, you can cite no serious scientists. And you don't understand the words of those, be they current ones or any of the heroes of either field, when they explain why that is.
You don't even need to believe them, just think about how you would ever be sure you know objective reality fully. Actually do it.
When I was a little kid, my go to thought experiment was imagining atoms as balls that we can't crack open that are filled with sand, and that even if we could find the formulas that describe their movements perfectly, we could never know if that's how atoms move, or if there's something inside them (the sand). Really dumb, but I was like 9 yo and knew nothing, never heard of Gödel or quantum mechanics (what's your excuse?). But I still could understand, not intuitively, but by actually thinking (not just talking) about it, that you can never be 100% sure from inside. Even if you found out everything, and your model of the world matches it perfectly, you could never be sure that it is so. And it turns out this is true and an old discovery. If you could disprove it, you would be famous for millenia, maybe forever. Bluffing on HN won't get you there.
You don't have to stoner riff, just give an example...
Where does math come short?
Does it matter if "intelligence and reasoning" are there if what they produce is indistinguishable (or outmatches) what a human could produce?
You only have your own experience to judge that you are even reasoning. You assume others have similar experience and so reason like you because they are able to do all the things that you do - and they look human like you. We can't prove there's a there there in other humans. What happens when the robots are doing everything humans do? What happens when they tell us they are reasoning, when they say they have an internal experience? Maybe there's nothing there, but you can't prove it. Moreover, it's likely they'll be able to affect the world and you in most of the same ways as humans can, whether there's a there there or not.
I can give Monet a pass as he probably didn't fully understand WHY it happened and it was just art, but the way tech companies exploit our brains (algorithmic dopamine hits, LLMs, etc) is pure insidiousness. The fact they act like victims when the backlashes come is what really grinds me.
This just reads so anthropocentric to me. Humans are wired to see intelligence only where intelligence is human-like. We see autonomous action-response and planning as the keys to intelligence. I would expect that, dear primate based Homo sapiens.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
How do you know it’s not when looking into the mirror, that we see Jesus in burnt toast?
Folks see AI seeing Jesus in burnt toast, and then admit it as one of the folks. :)
https://archive.ph/ZyCoe
This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.
Makes me think how human thinking is mostly recall from past learning/experience versus reasoning.
I thought I saw a paper recently explaining that LLMs have a global workspace. Is that not like having the internal state that he's talking about?
There was a blog post from Anthropic saying Claude has a "J space" or something like that.
Why J?
From Jacobian. They developed a tool called a Jacobian Lens, or J-Lens (itself a descendant of an earlier simpler tool called a Logit Lens) to examine what was going on inside an LLM, and named the space they found with it a J-Space.
Jacobians are essentially derivatives but for matrices.
Don't be fooled, submarines don't swim.
An LLM is just a brain in a vat. Our brains would also not reason in such a condition. Given the right framework that can do miraculous things. I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
https://www.finextra.com/blogposting/31255/adaptability-as-e...
well, its a matter of definitions.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
forgive typos, i am on mobile.
Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.
<unbearable web page to read>
Agreed, especially because it also seems to defeat Reader mode. Try running it through Marky and then previewing: https://heckyesmarkdown.com/preview.cgi?readability=1&inline...
That is amazingly useful tool I didn’t know about until now, thank you.
Sure thing; it’s a Brett Terpstra creation.
Textise is similar and also doubles as a poor man’s text-only browser. https://www.textise.net/referrer
Both Marky and Textise work in some places where the other doesn’t.
Obviously you (or your LLM) can throw together JavaScript bookmarklets to convert whatever page you’re on. The tools’ homepages might have premade ones, I forget.
If AI cannot reason, can humans?
I think the real question is: how do we define reasoning?
My view is that AI can reason, just differently from humans.
This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
>A prediction engine that runs fast enough to brute force problems.
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
Yep, it's not 'AI', or even just 'I', if anything it is just the 'A' wrapped in buzzwords.
Friendly reminder that y'all could be reading this, or you could take things a bit more seriously and check out prof. Robert Sapolsky's excellent (2024) series of lectures on human behavioural biology. (eg. on YouTube, unless your're a lucky californian) Anyone who wants to venture an opinion about this or that reasoning should be interested in finding out about their own hardware & software first, right?
I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
If the assertion is false, this is helpful, as instructing it to reason better will cause it to reason better.
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
You can sort of do this.
For a given bug one could write a test that prove its existence, this gives the LLM a target that they can actually iterate towards.
The worry I would have is that an LLMs stated "confidence" is probably not calibrated well - maybe asking it what evidence supports its conclusion and what evidence would change it would be a better approach?
Yeah, and red-team/blue-teaming it.
(I'm definitely one of the bigger "AI" skeptics out there but am nonetheless fascinated by these questions).
The article links to the book „Thinking, Fast and Slow” by Daniel Kahneman which seems interesting. Anyone read it have thoughts?
But neither do the vast majority of humans.
Garbage In Garbage Out
It's wild when you think about it, you don't need to reason to solve the hardest math problems that humans failed to solve for decades.
Brute force is powerful yes. And another way to look at math problems is that humans solved a huge lot of very hard math problems already but didn't solve all of them.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
Same for programming, turns out 90% of it is dumb statistical prediction, no reasoning required at all.
LLMs lack the holy ghost
They're not part of an evolved biological organism, so why would it be surprising if they're different? It is called artificial intelligence, and it's not a simulation of a brain like the Nematode OpenWorm.
Yes, this is true—there's no "logic" in the sense of deductive rigor. It's a wonder we animals are capable of it.
> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
"Intelligence measures an agent's ability to achieve goals in a wide range of environments"
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
https://youtu.be/wTSbvYBx4eg?si=MfTBFsE2kA3TE6EE&t=189
I think, "contexting" is apropos.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
Neither do humans!
At least reasoning is not guaranteed.
Would sure be nice though...
Heh, some people get really mad and downvote when you point out we don't reason very much. Forget reasoning, we have a hard enough time being rational.
We can't even say what human reasoning is. Who could really say what isn't reasoning?
99% of the people treat it as a black box. It gives better answers, you can reverse reason and IDGAF
>Three shortcomings prevent what chatbots do from qualifying as reasoning
>First, these models typically maintain no explicit, persistent, and inspectable epistemic state.
>Second, they lack a clean separation between what the system knows and how it manipulates that knowledge
>Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact
I think humans are guilty of all three of those? If you use that to say LLMs can't reason you may be saying humans can't reason either which is kind of unreasonable.
The author's thesis might we better put that we could increase the reliability and openness of AI by building it so we can inspect its data, assumptions and reasoning process.
‘LLMs don’t reason’ seems to equate reasoning with explicit, auditable deliberation. LLMs can draw inferences from premises, even if their explanations do not reliably reveal how they arrived at their conclusions. The article makes a case for more transparent and reliable reasoning, but does not establish the absence of reasoning altogether. The more substantive question, in my view, is whether their representations have semantic content, what could ground that content, and how this affects their reasoning capabilities and limitations.