> Like everyone, I’m tremendously impressed by the lifelikeness of these systems compared to the history of earlier AI. But the disposition of some philosophers to take seriously the question whether these things are conscious or might soon become conscious is just ridiculous. They are assuming that the physical substrate is irrelevant. They all think that the material conditions for consciousness can be given in functionalist terms.
> (...)
> Functionalism is substrate-independent. It’s more than likely that the truth is one which links physical processes and systems of certain kinds to mental phenomena. Complicated data centers have absolutely nothing in common with the physical character of a nervous system. We have strong reasons to believe that while there may be consciousness elsewhere in the universe, it would require the appearance of some kind of organism with something analogous to a nervous system.
Why should we assume our substrate is the "right" substrate for truly conscious beings?
Perhaps we have fake consciousness too (just like programs running on silicon), and some advanced alien being could look at our biological neurons stuff and immediately recognize our fakeness. Perhaps those aliens could proceed to assert that their own substrate is the only one that could possibly be conscious - all the other substrates (like ours) being fake.
We couldn't possibly know, because we don't know what consciousness is.
> Why should we assume our substrate is the "right" substrate for truly conscious beings?
No one assumes that it is the only right substrate. But if you don't believe we are a right substrate, are you waiving your own consciousness? Seems like you are, which does make me question how coherently you can argue about this.
> We couldn't possibly know, because we don't know what consciousness is.
We don't know it's deepest mechanisms. That also holds true for the universe. Doesn't mean we can't see the difference in dynamics.
Simply put, consciousness cannot be turned off and on. No definition of consciousness holds up to a system that can turn off, then be turned on 100 years later to carry on generating text as it was.
Evolutionarily, such a thing would be quite idiotic and maladaptive. It's not feasible to be an "agent" in the world when you are, by default, off, and require a prompt somewhere upstream to come awake.
Biology, on the other hand, is exquisitely timed. Each cell (including each neuron) is a self-sustained oscillator, with oscillatory rhythms that impact behavior, learning and memory. These clocks are endogenous, meaning they continue as long as long as the organism is alive, even without external periodic signals. LLMs have no such endogenous self-sustained osciallation operating continuously.
Go back to before the LLM's are conscious hype started. Find me one definition or example of consciousness that can be switched off. And no, neither sleep nor anesthesia count. They're different states of consciousness, but the brain and body are not "off".
Nor will any definition of consciousness pre-LLMs even countenance the idea that a frozen set of weights is conscious. The very idea of consciousness without continual learning is utter stupidity. All the sadder that it's spread so wide and has already cost us trillions.
Unless you believe the molecular state of the body cannot be in theory ever recorded and resynthesized, how is it not possible in this universe to turn humans or other animals on and off?
Obviously its impossible right now. But let me get this clear are you saying its impossible as in say putting a lander on Pluto (impossible today, no theoretical roadblocks to be able to do it in future) or physically impossible. After all (morality aside) one can simply slice off molecules from the body using eg a similar method as atomic microscopy. You could argue that the information might be hard to read, or there might be a proof that its impossible to read without altering it appreciably (quite possible), but we would need such a propf first, thats the only possible argument you can give. Obviously its aburd to say the molecules int the body are different from the same molecules synthesized elsewhere.
The amount of precision depends on how sensitive the body and brain is, eg we don't need to go to qm to design or copy a car or motor.
I'm saying it is physically impossible. And it's not some vague theory. It's pretty darn well established.
> one can simply slice off molecules from the body using eg a similar method as atomic microscopy.
This sentence doesn't make sense. Atomic force microscopy (which I assume is what you're referring to) doesn't itself slice any molecules from the body. It can be used to study molecules or cells ripped from the body, sure. In the process, the molecular state of the body you wanted to copy changes, so even if you reconstitute the bit you got out and put under the microscope (itself a big if, since no known microscope can give you exact quantity and type of every molecule in your sample), that bit is already in a different state, so you're not reconstiting the state of that segment of the body, let alone the entire body.
A human body has about 36 trillion human cells. About an equal number of bacterial, archeal and unicellular eukaryotic cells (bacteria dominate this cluster). Together, they have about an octillion molecules. 99% of that is water, but the dynamics of that water make the Naveir stokes equation feel like the two-times multiplication table in comparison.
The rest is considerably harder. Proteins, for example, aren't floating around bearing the frozen crystal structures AlphaFold can so quickly solve. Instead, dissolved in all that water, protein molecules cycle through all kinds of unstable confirmational states, sometimes for very brief instances, which are nonetheless functionally critical, and part of the state description of any cell. But wait, in the seconds it took you to read the previous two sentences, a significant chunk of those proteins have gone through multiple confirmational changes. And somehow you're going to find and read the state of every protein in a picosecond? There's about a 100 sextillion of those in your body.
So which moment are you planning to copy? And how? Any read attempt is going to be destructive, especially to these fragile confirmational states. Freeze the cell and those states shift. Break up the cell to measure it, and those states shift. Use light to image it, and you can image a subset of proteins but either need to modify them, or kill and fix the cell first, and in either case, sufficient light to illuminate every part of the body in an instant to be read by whatever detector you design still won't get you what you want, because guess what, light changes the proteins too, and enough will destroy them.
Then consider lipids, metabolites, small peptides, and small RNAs (which also do all kinds of confirmational hijinks second to second)... None of these can be fully measured body wide for you to clone the body's state.
It's just not even within the realm of plausibility to do what you're suggesting. Even a simulation that's accurate down to atomic resolution, if we someday invent computers that can do it, will drift from the actual system really really rapidly.
The body isn't a car or a motor. It doesn't have their simple regular structure. And literally tiny molecular changes have huge impact on overall organism state. Thinking about the body like it's some engineered machine is probably why you fail to see that even though it is made of entirely boring matter that's pretty common in the universe, what separates it from non-life is the depth and complexity of it's dynamics. And no, no LLM comes remotely close to the complexity of a bacterium let alone a human.
Yes I meant a probr that reads a molecule then breaks off its bonds to check the further inside the object. As I said before I am quite open to any proof along the lines of its impossible to measure a particular state without appreciably changing it due to sensitvity arguments. It might be true but unless we have a proof I can't say either way.
Completely engineered systems can be unpredictable without fully solving it eg chaos, I don't have any complaints about that as a line of defense.
Yes some kind of mathematical proof that any measurement of this system by the time it had taken place has changed the system appreciably in a way that can't be inferred back or accounted for.
> We have strong reasons to believe that while there may be consciousness elsewhere in the universe, it would require the appearance of some kind of organism with something analogous to a nervous system.
At least he gets to the point quickly. But I'd like to know what those "strong reasons" are.
> Perhaps we have fake consciousness too (just like programs running on silicon), and some advanced alien being could look at our biological neurons stuff and immediately recognize our fakeness.
"That’s ridiculous. How can meat make a machine? You’re asking me to believe in sentient meat."
> We couldn't possibly know, because we don't know what consciousness is.
Great minds discuss ideas, average minds discuss events, small minds discuss people, philosphers discuss "consciousness".
Ok I'm on the total opposite of this guy, I am almost tempted to write a short letter telling him why Thomas Nagel, famous American philosopher, is wrong on this.
Basically his opinion is derivative of stochastic parrot theory. He admits as much in the interview. (Since stochastic parrot theory is wrong, I could stop right here, and so should he if he had a truly sufficient argument in the first place...)
He is committing the mistake as Scott Aaronson says, justa-ism. It's "just a language model" therefore it cannot possibly really be intelligent/alive/conscious/goalpostmoving. Followed by some tricky rhetorical move appealing to human exceptionalism.
The fact is that philosophers would do well to take computational complexity theory, the one involving actual math homework. All Nagel's woo-woo talk about a substrate and a need for an "analogous nervous system" shows yet again how CS as a nascent field, and theoretical CS in particular, has not fully been internalized by other disciplines.
Nagel says a bat is conscious. But an LLM with trillion dimensional neuron parameters, obtained through stochastic gradient descent, cannot possibly conscious because it is on the wrong substrate? Really?
In fact it is the other way around. The true "substrate" is the neural network model, of which humans (and bats) are merely one example of. That is the correct way to understand what LLM computation is. The abstract LLM neural net is what Alan Turing himself envisioned as the alternative to classical computation.
Yeah, I wouldn't necessarily call LLMs conscious, but I'm reminded of an argument that I heard once that computers could never hold consciousness because they don't process things in parallel. There's a lot of incurious reasoning abound when it comes to commentary on this topic.
Its absurd in this day an age that "philosophers" talking about consciousness who don't address at least thermodynamics, computation theory and information theory should not just be immediately laughed out of the room.
If they cannot at least acknowledge that the universe is a Turing computer, all Turing computers have equal power, that information has an energy cost so some special immaterial dual soul will have to violate thermodynamics to generate thoughts and actions, their arguments should be dismissed.
I wouldn't assume the universe is a Turing machine. The holographic theory does suggest that it has limited state in any given space, but that doesn't limit or to finite state transitions, or to finite space.
We can't see enough of space to know whether it's infinite.
I just want someone to define "consciousness" before they start arguing who can and cannot have it.
If the definition is "the processes of a biological brain" then clearly a computer cannot have it. It's also not very useful.
Turing machines have no trouble being infinite. That the universe is a tm comes from the fact that as of yet there is no knownaw that enables hypercomputation.
>theoretical CS in particular, has not fully been internalized by other disciplines.
Whereas it appears theoretical computer Science sometimes forgets there are many routes to computing per its own fundamentals, and certainly ignores the fact that Neuroscience didn't stop the day McCulloch and Pitts first conceived of the artificial neuron.
> But an LLM with trillion dimensional neuron parameters, obtained through stochastic gradient descent, cannot possibly conscious because it is on the wrong substrate? Really?
Yes really. You seem to think the word "trillion" should awe us. It's hard to understand why. Care to tell us?
Critically, whether a trillion or a significantly larger number, LLMs parameters have characteristics that make talk of consciousness absurd. To whit: they are frozen, and between two forward passes, a million years can pass and the model wouldn't know.
> In fact it is the other way around. The true "substrate" is the neural network model, of which humans (and bats) are merely one example of. That is the correct way to understand what LLM computation is. The abstract LLM neural net is what Alan Turing himself envisioned as the alternative to classical computation.
Here's just one problem with this theory. All neurons are 24-h oscillators, and have shorter oscialltions nested under this. A fact neither Turin nor McCullough nor Pitts could know when they were trying to imagine how the brain works. They assumed a global clock, which is absolutely false in biology.
And this rhythmicity is evolutionarily older than the rise of neurons. Cyanobacteria have 24 hour rhythms (though using different components), and can anticipate seasonal changes by this system, and first evolved this 2.5 billion years ago.
The issue isn't just substrate. Some fundamental comoutstuonal features of cells, neurons, brains and bodies need to be thoroughly ignored to land on a claim that LLMs are the true substrate of which human brains are an expression.
And also... Have you heard of astrocytes? Roughly as many of those as neurons in the brain, and they're part of every synapse. If artificial neural nets are the formula for consciousness, bats and humans sure seem to spend a lot on these "useless" cells, no? Interestingly, astrocyte size and complexity goes up as you go from mouse to human, and implanting human astrocytes in mice enhances their intelligence. One heck of an odd result to explain if it's all LLM like dynamics in the brain.
> Like everyone, I’m tremendously impressed by the lifelikeness of these systems compared to the history of earlier AI. But the disposition of some philosophers to take seriously the question whether these things are conscious or might soon become conscious is just ridiculous. They are assuming that the physical substrate is irrelevant. They all think that the material conditions for consciousness can be given in functionalist terms.
> (...)
> Functionalism is substrate-independent. It’s more than likely that the truth is one which links physical processes and systems of certain kinds to mental phenomena. Complicated data centers have absolutely nothing in common with the physical character of a nervous system. We have strong reasons to believe that while there may be consciousness elsewhere in the universe, it would require the appearance of some kind of organism with something analogous to a nervous system.
Why should we assume our substrate is the "right" substrate for truly conscious beings?
Perhaps we have fake consciousness too (just like programs running on silicon), and some advanced alien being could look at our biological neurons stuff and immediately recognize our fakeness. Perhaps those aliens could proceed to assert that their own substrate is the only one that could possibly be conscious - all the other substrates (like ours) being fake.
We couldn't possibly know, because we don't know what consciousness is.
> Why should we assume our substrate is the "right" substrate for truly conscious beings?
No one assumes that it is the only right substrate. But if you don't believe we are a right substrate, are you waiving your own consciousness? Seems like you are, which does make me question how coherently you can argue about this.
> We couldn't possibly know, because we don't know what consciousness is.
We don't know it's deepest mechanisms. That also holds true for the universe. Doesn't mean we can't see the difference in dynamics.
Simply put, consciousness cannot be turned off and on. No definition of consciousness holds up to a system that can turn off, then be turned on 100 years later to carry on generating text as it was.
Evolutionarily, such a thing would be quite idiotic and maladaptive. It's not feasible to be an "agent" in the world when you are, by default, off, and require a prompt somewhere upstream to come awake.
Biology, on the other hand, is exquisitely timed. Each cell (including each neuron) is a self-sustained oscillator, with oscillatory rhythms that impact behavior, learning and memory. These clocks are endogenous, meaning they continue as long as long as the organism is alive, even without external periodic signals. LLMs have no such endogenous self-sustained osciallation operating continuously.
Go back to before the LLM's are conscious hype started. Find me one definition or example of consciousness that can be switched off. And no, neither sleep nor anesthesia count. They're different states of consciousness, but the brain and body are not "off".
Nor will any definition of consciousness pre-LLMs even countenance the idea that a frozen set of weights is conscious. The very idea of consciousness without continual learning is utter stupidity. All the sadder that it's spread so wide and has already cost us trillions.
Unless you believe the molecular state of the body cannot be in theory ever recorded and resynthesized, how is it not possible in this universe to turn humans or other animals on and off?
Huh?
"the molecular state of the body cannot be in theory ever recorded and resynthesized"
This is 100% true. This is physically impossible to do.
What’s a hypothetical mechanism to get a molecular accurate clone that also preserves thermal and temporal states?
Heck, what’s hypothetical mechanism to read The exact state of a body down to every molecule?
Obviously its impossible right now. But let me get this clear are you saying its impossible as in say putting a lander on Pluto (impossible today, no theoretical roadblocks to be able to do it in future) or physically impossible. After all (morality aside) one can simply slice off molecules from the body using eg a similar method as atomic microscopy. You could argue that the information might be hard to read, or there might be a proof that its impossible to read without altering it appreciably (quite possible), but we would need such a propf first, thats the only possible argument you can give. Obviously its aburd to say the molecules int the body are different from the same molecules synthesized elsewhere.
The amount of precision depends on how sensitive the body and brain is, eg we don't need to go to qm to design or copy a car or motor.
I'm saying it is physically impossible. And it's not some vague theory. It's pretty darn well established.
> one can simply slice off molecules from the body using eg a similar method as atomic microscopy.
This sentence doesn't make sense. Atomic force microscopy (which I assume is what you're referring to) doesn't itself slice any molecules from the body. It can be used to study molecules or cells ripped from the body, sure. In the process, the molecular state of the body you wanted to copy changes, so even if you reconstitute the bit you got out and put under the microscope (itself a big if, since no known microscope can give you exact quantity and type of every molecule in your sample), that bit is already in a different state, so you're not reconstiting the state of that segment of the body, let alone the entire body.
A human body has about 36 trillion human cells. About an equal number of bacterial, archeal and unicellular eukaryotic cells (bacteria dominate this cluster). Together, they have about an octillion molecules. 99% of that is water, but the dynamics of that water make the Naveir stokes equation feel like the two-times multiplication table in comparison.
The rest is considerably harder. Proteins, for example, aren't floating around bearing the frozen crystal structures AlphaFold can so quickly solve. Instead, dissolved in all that water, protein molecules cycle through all kinds of unstable confirmational states, sometimes for very brief instances, which are nonetheless functionally critical, and part of the state description of any cell. But wait, in the seconds it took you to read the previous two sentences, a significant chunk of those proteins have gone through multiple confirmational changes. And somehow you're going to find and read the state of every protein in a picosecond? There's about a 100 sextillion of those in your body.
So which moment are you planning to copy? And how? Any read attempt is going to be destructive, especially to these fragile confirmational states. Freeze the cell and those states shift. Break up the cell to measure it, and those states shift. Use light to image it, and you can image a subset of proteins but either need to modify them, or kill and fix the cell first, and in either case, sufficient light to illuminate every part of the body in an instant to be read by whatever detector you design still won't get you what you want, because guess what, light changes the proteins too, and enough will destroy them.
Then consider lipids, metabolites, small peptides, and small RNAs (which also do all kinds of confirmational hijinks second to second)... None of these can be fully measured body wide for you to clone the body's state.
It's just not even within the realm of plausibility to do what you're suggesting. Even a simulation that's accurate down to atomic resolution, if we someday invent computers that can do it, will drift from the actual system really really rapidly.
The body isn't a car or a motor. It doesn't have their simple regular structure. And literally tiny molecular changes have huge impact on overall organism state. Thinking about the body like it's some engineered machine is probably why you fail to see that even though it is made of entirely boring matter that's pretty common in the universe, what separates it from non-life is the depth and complexity of it's dynamics. And no, no LLM comes remotely close to the complexity of a bacterium let alone a human.
Yes I meant a probr that reads a molecule then breaks off its bonds to check the further inside the object. As I said before I am quite open to any proof along the lines of its impossible to measure a particular state without appreciably changing it due to sensitvity arguments. It might be true but unless we have a proof I can't say either way.
Completely engineered systems can be unpredictable without fully solving it eg chaos, I don't have any complaints about that as a line of defense.
Define "proof" in this context please. Do you mean a math proof?
Yes some kind of mathematical proof that any measurement of this system by the time it had taken place has changed the system appreciably in a way that can't be inferred back or accounted for.
> We have strong reasons to believe that while there may be consciousness elsewhere in the universe, it would require the appearance of some kind of organism with something analogous to a nervous system.
At least he gets to the point quickly. But I'd like to know what those "strong reasons" are.
> Perhaps we have fake consciousness too (just like programs running on silicon), and some advanced alien being could look at our biological neurons stuff and immediately recognize our fakeness.
"That’s ridiculous. How can meat make a machine? You’re asking me to believe in sentient meat."
> We couldn't possibly know, because we don't know what consciousness is.
Great minds discuss ideas, average minds discuss events, small minds discuss people, philosphers discuss "consciousness".
Please use the original title (A Famed Philosopher Takes on AI Consciousness)
> https://news.ycombinator.com/newsguidelines.html
Ok I'm on the total opposite of this guy, I am almost tempted to write a short letter telling him why Thomas Nagel, famous American philosopher, is wrong on this.
Basically his opinion is derivative of stochastic parrot theory. He admits as much in the interview. (Since stochastic parrot theory is wrong, I could stop right here, and so should he if he had a truly sufficient argument in the first place...)
He is committing the mistake as Scott Aaronson says, justa-ism. It's "just a language model" therefore it cannot possibly really be intelligent/alive/conscious/goalpostmoving. Followed by some tricky rhetorical move appealing to human exceptionalism.
The fact is that philosophers would do well to take computational complexity theory, the one involving actual math homework. All Nagel's woo-woo talk about a substrate and a need for an "analogous nervous system" shows yet again how CS as a nascent field, and theoretical CS in particular, has not fully been internalized by other disciplines.
Nagel says a bat is conscious. But an LLM with trillion dimensional neuron parameters, obtained through stochastic gradient descent, cannot possibly conscious because it is on the wrong substrate? Really?
In fact it is the other way around. The true "substrate" is the neural network model, of which humans (and bats) are merely one example of. That is the correct way to understand what LLM computation is. The abstract LLM neural net is what Alan Turing himself envisioned as the alternative to classical computation.
Yeah, I wouldn't necessarily call LLMs conscious, but I'm reminded of an argument that I heard once that computers could never hold consciousness because they don't process things in parallel. There's a lot of incurious reasoning abound when it comes to commentary on this topic.
Its absurd in this day an age that "philosophers" talking about consciousness who don't address at least thermodynamics, computation theory and information theory should not just be immediately laughed out of the room.
If they cannot at least acknowledge that the universe is a Turing computer, all Turing computers have equal power, that information has an energy cost so some special immaterial dual soul will have to violate thermodynamics to generate thoughts and actions, their arguments should be dismissed.
I wouldn't assume the universe is a Turing machine. The holographic theory does suggest that it has limited state in any given space, but that doesn't limit or to finite state transitions, or to finite space.
We can't see enough of space to know whether it's infinite.
I just want someone to define "consciousness" before they start arguing who can and cannot have it.
If the definition is "the processes of a biological brain" then clearly a computer cannot have it. It's also not very useful.
Turing machines have no trouble being infinite. That the universe is a tm comes from the fact that as of yet there is no knownaw that enables hypercomputation.
>theoretical CS in particular, has not fully been internalized by other disciplines.
Whereas it appears theoretical computer Science sometimes forgets there are many routes to computing per its own fundamentals, and certainly ignores the fact that Neuroscience didn't stop the day McCulloch and Pitts first conceived of the artificial neuron.
> But an LLM with trillion dimensional neuron parameters, obtained through stochastic gradient descent, cannot possibly conscious because it is on the wrong substrate? Really?
Yes really. You seem to think the word "trillion" should awe us. It's hard to understand why. Care to tell us?
Critically, whether a trillion or a significantly larger number, LLMs parameters have characteristics that make talk of consciousness absurd. To whit: they are frozen, and between two forward passes, a million years can pass and the model wouldn't know.
> In fact it is the other way around. The true "substrate" is the neural network model, of which humans (and bats) are merely one example of. That is the correct way to understand what LLM computation is. The abstract LLM neural net is what Alan Turing himself envisioned as the alternative to classical computation.
Here's just one problem with this theory. All neurons are 24-h oscillators, and have shorter oscialltions nested under this. A fact neither Turin nor McCullough nor Pitts could know when they were trying to imagine how the brain works. They assumed a global clock, which is absolutely false in biology.
And this rhythmicity is evolutionarily older than the rise of neurons. Cyanobacteria have 24 hour rhythms (though using different components), and can anticipate seasonal changes by this system, and first evolved this 2.5 billion years ago.
The issue isn't just substrate. Some fundamental comoutstuonal features of cells, neurons, brains and bodies need to be thoroughly ignored to land on a claim that LLMs are the true substrate of which human brains are an expression.
And also... Have you heard of astrocytes? Roughly as many of those as neurons in the brain, and they're part of every synapse. If artificial neural nets are the formula for consciousness, bats and humans sure seem to spend a lot on these "useless" cells, no? Interestingly, astrocyte size and complexity goes up as you go from mouse to human, and implanting human astrocytes in mice enhances their intelligence. One heck of an odd result to explain if it's all LLM like dynamics in the brain.