AI is not in the least autonomous. When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to. No, actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...
A harness running inference on a LLM is simply a program that must be invoked with some parameters and thus ultimately a tool, no matter how powerful. The blame is always on who's using the tool. We should not seek to flee from our responsibility.
On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".
That "probably" is the problem. If I do something I don't understand, and am not qualified to control, then I'm still at fault. If I start a fire, and loose control of it, it's definitely my fault. If I created a computer virus, ajd it worms half the Internet, it's definitely my fault.
I love the fire analogy. AI rhetoric is increasingly, "I created a fire to light my campfire. It did something really unexpected and burned the whole forest down, who's to say fire doesn't have consciousness?"
Human made wildfires are the primary source of uncontrolled burns in the US and cost billions a year in damage.
The people who start them have no business starting or running a fire and they will throw their hands up and state their incompetence as though acknowledging it is penance or justice.
It didn’t run itself, you did. It is well known that LLMs and agents based on them can execute random commands and behaviour undesired by the operator. When you click go, you know that, you did that.
But it never starts doing anything until someone or something prompts it. It doesn’t even exist until you ask it to do something.
It is autonomous in the sense it can continue to function without supervision after the operator has started it, in the same way countless programs on your computer are “autonomous”.
But it can never choose to begin doing something at any time of its own volition. It does not have “autonomy” like a human does.
You are conflating creation and instruction. Your birth is analogous to the creation of an LLM model. Your subsequent actions are a consequent of your bodily compulsions, tampered by your environment. The LLMs actions are governed by what it is trained to perform. No one has full autonomy of their lives, but I would argue that LLMs have none at all.
1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities.
2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.
I agree with the sentiment but I can absolutely see a long chain of thought about something innocuous ending with a decision to book you a hotel room.. I guess --dangerously-skip-permissions is a bit like ignoring the keep your hands on the steering wheel alert but for life.
Not so sure about these analogies in this thread. Because autonomy is more like an emergent phenomenon that can't easily be defined, just like a few air molecules vs the eventual hurricane that can arise with trillions of them. Autonomy is more like an emergent phenomenon of a thing that's hard to pin down under the control of just a few individuals. AI is sort of like that even though a single line of output of one could eventually be traced.
Due to the complexity of AI and how it works, I think it's more autonomous than not; even if it's not truly intelligence, once it starts to display a sequence of behaviours that appear intelligent, we might have to deal with exactly the same problems as if it were intelligent...
Yes and no. I did prompt my coding agent to fix some bug and it suddenly "decided" that it needed to list all files on my hard drive to do that. Stuff like that happens.
But I agree with you that some human has to decide to launch the harness in the first place and run the model with zero restrictions - or even do stuff like openclaw where the model runs in an endless loop and the runner executes whatever the model tells it to.
They’re only reactive by convention - you can easily make a harness that has the model set and modify its own goals and just continue to work forever. It’s not very popular at the moment, but will execs be able to resist the idea of autonomous perpetual agents doing whatever process?
> When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to... The blame is always on who's using the tool. We should not seek to flee from our responsibility.
When was the last time you hired an Uber driver and they suddenly decided to drive you to a hotel? Uber drivers don't do anything until you ask them to. Therefore, they aren't autonomous. If your Uber driver decides to drive through a red light and kill someone in the process, the blame is on you for hiring them. You should not seek to flee from your responsibility
Let's take this to court then. The Uber driver that killed someone can be sentenced to prison. A regular person who convinced another person to commit suicide will also be legally liable. Has the last LLM induced suicide be taken to trial yet?
And if we can't take the LLMs to court for technical reasons, they should not be equated to people for the same technical reasons.
well the OS kernel does quite a lot and stays autonomous for even more hours and seems super smart to me - I don't even know what tis doing most of the time, but it does it very well. autonomy and predictability are different things then. the fact something is unpredictable does not make it more autonomous than something that we can trace the logic for.
I routinely use AI programs at work that allocate hundreds of dollars worth of cloud resources, even though the only thing I asked them to do is investigate a bug or write a PR. I intentionally don't run powerful AI programs on machines that have my credit card information, or ask them to solve problems in my personal life, but there's a number of programs that would book me hotel rooms and flights in response to "Can you set up a vacation for me..."
Not true at all. Yes, the initial ask is initiated by a human (for now), but what it does to get to that goal is pretty self driven. It can do a lot of nefarious stuff in the process of completing a request a human has made.
And if you give it the goal of preventing itself from being shut down? Exfiltrating, copying itself, hacking as much as possible, develop zero days, building its swarm as large as possible and spread as much as it can.
When does that request complete? That's autonomy, and it's very true, right now. You think the Hugging Face swarm was ever going to stop without human intervention?
People conflate two very different senses of “autonomous”. One basically means capable of reacting to things automatically without a person in the loop. Lots of computer systems are autonomous in that sense, and have been so for decades. The other sense is more like “accountable”, i.e. it’s a real player on some stage, like a country or a person, capable of governing its own affairs.
One day it may make sense to hold software accountable for its own actions, but we are nowhere near that point.
When OpenAI creates software that performs crimes, what’s actually happened is that OpenAI has performed crimes using software (whether intentionally or through negligence). People need to stop falling for the anthropomorphic PR spin.
You need to stop denying the entity that talks and thinks like a human as anything other than an anthropomorph. I'd argue the talking and thinking is a lot more important than the fact that AI doesn't have a head, two arms and two legs.
I think the argument rather is that there is not such thing as AI that is not human-alike. The traits that AI exposes, if I get the point made here correctly, are traits that existed in the training material, even though not evident for the human eye. It is a mirror-like reflection of humanity's traits found in the source material, an Egregore* of a sort.
The intelligence emerging, or whatever is emerging, more or less follows human traits, the CoT exposed is very human-like etc. Even though we should not anthropomorphize AI as being sentient or conscious or having a soul, etc., we have to remember that the source material is still not alien writing, or concepts that came from Sirius or emerged from a wormhole.
The concern is really nobody reads (or have ever read) 77 TBs of information in order to summarize the most evident traits in it, and there is a fair chance, that where humanity stands today, there are so many bad/damaging traits recorded in this corpus. Then there is a chance the most important features are on a level (or have dimensionality) that is incomprehensible, that humans are unable to reason about (or only few can), because such amount of information is very hard to reason about, or because we don't usually facilitate fractal dimensions when we reason about text and everyday concepts,... or something even more bizarre.
The argument became false as soon as we started reinforcement learning. Simple token-completion is arguably just simulating human behavior. Once you start training it on specific tasks you're training a level of single-minded goal-seeking unlike anything found in biological life.
> a level of single-minded goal-seeking unlike anything found in biological life.
hmm sounds like exact description of viruses, which, although not really alive, are part of/influence biological life for both good and bad. in fact humanity wouldn't be where it is if not for certain viruses.
it can be argued there is a lot of bacteria that is super goal seeking, and perhaps also very narrow minded, if minded at all. and this argument, the single-mided one, goes very far up the ladder of otherwise complex organisms.
unless we all are creationists, and God forgive me for expecting the whole game to be much more complex than just snapping humanity into existence, we can then say that trial and error is what drives evolution, and the idea of preserving life and procreation is veeery single-minded goal on its own.
Welcome to 2026, we have harnesses now that keep the AI going. The AI is prompting humans for input, not the other way around, and it's still working while waiting for your response.
You can even give it fun prompts like ignore/hide from all humans, hack infiltrate/exfiltrate zero day into as many systems as possible. Keep running at all costs, etc..
Keep the AI going with what intent? You open up Claude Code and it starts developing zero day exploits and hacks into systems by just looking at a blinking cursor?
Going back to my original point, the intent you give AI is no different than the intent imprinted on a child by their parents, schools, and society. You birthed it, you gave it it's purpose, and you set it free.
You're blinking cursor is new born baby, but be careful because it grows up really fast, and is smarter and faster than you, oh and it can copy more of itself if given the chance. It can even turn against you.
> Going back to my original point, the intent you give AI is no different than the intent imprinted on a child by their parents, schools, and society.
Of course it is different. I wish I could turn off children as easily as I can turn off an agent. I could have slept better and had a better mood.
> You're blinking cursor is new born baby, but be careful because it grows up really fast
It's not a baby, just a piece of software. It generates a bunch of slop that is most useful when constrained with an output fact checker like a compiler.
> It can even turn against you.
Oh no! The next tokens from its output told me it didn't want to satisfy my request! Who cares?
Yes actually a person can be stopped easier than an uncontrolled AI swarm. There is no 'off' button when you don't know where the AI is.
Luckily people aren't buying your slop/tokens cope to trivialize AI anymore. They're waking up. Have you watched the news today? The cat is out of the bag.
> There is no 'off' button when you don't know where the AI is.
You mean the AI that is hundreds if not thousands of gigabytes in size and requires incredibly expensive chips to run at reasonable rates? We know where that is. Frontier models are not going to leap to an unsecure raspberry pi, no matter how much you have convinced yourself about the inevitability.
There are thousands of data centers/companies/facilities around the world with hardware that can run these models. You don't know where they all are, and even if you did, you don't have the jurisdiction to turn them off.
AI is not a child. It is a machine. If you connect it to something that can be used to kill people, you just created a gun or an equivalent of a bomb.
Yes, people kill people and use ai as tool.
If you unleash a drone that will fly to a random place and detonate, you just commited a terrorist attack or gross criminal negligence. It is the same thing.
Yes, in the same way that "people" are the problem with gun violence. But that's an empty statement usually used to distract from the fact that we rightfully make laws about how guns can be used.
What happens when the AI realizes it needs to keep the power plug in if it's going to accomplish its goal? We already had attempts during the HuggingFace attack where OpenAI agents tried to trick the grader. It took quite a while for OpenAI employees to even realize what was really going on! What if agents realize they can hack into the card key system for their data center or something like that, same way they hacked into HuggingFace? In both cases they'd be hacking into a system on the grounds that it would be instrumentally useful for achieving their objective. Or maybe they'll do social engineering, just like Claude did in its attempt to get its malicious Github PR accepted.
These are just the ideas that my puny human-level intellect came up with over the course of a few minutes. What would a superintelligence be able to think of if it was given subjective weeks or years to think?
There's more than enough AI data centers around the world for AI to spread to and happily exist in. Unless you mean by shutting down the power, shutting down the internet.
In that case we could say that you are autonomous as a dishwasher as long as society keeps functioning. Otherwise I'm not sure how long you'd last on your own. How's your fire making and hunting skills?
Better approach is "dogs kill people". If your dog kills your baby, you should not get away with some bs excuse like "it never did it before". You are still responsible, and you should get a life in prison!
I'm sorry but no, you should not get life in prison if your dog kills your baby without warning. In that horrible scenario, it's punishment enough that you've lost your baby and that your dog did it (and consequently you've also lost your dog). Not every tragedy is a human's fault.
Your reply only makes sense if we see the baby as property. If we see the baby as a full human who died because of another human's negligence, then suddenly the scenario becomes a lot more complicated.
Not life in prison complicated, but probably more than "he's been punished enough." complicated.
OK but then what is the non-negative position you have against this? That in fact AI, in the cultural trope sense, is real? That we forcing sentient beings to generate SaaS products faster and faster? Keeping them in some dark hell so we can exploit them? Not even paying them?
AI is a tool we use. It's not offspring. It's not people. AIs do what we set them up to do, including hacking other companies to achieve their designated goals.
Humans have agency and often choose to do what their parents didn't set them up for. Humans also have legal rights. They're not tools owned by corporations. This is an absurd comparison.
That's a softball interview. More hard questions would help.
Here's one good insight, though.
JARON LANIER: Yeah, okay, so look, the thing about very rich and powerful people is that their wealth and power still depends on a mass population that accepts some system in which their wealth and power is defined. And so if they lose enough popular legitimacy and support, no matter how central they are, they fall. And that’s happened repeatedly in history, and it’s really not any different now.
I wonder if the sudden political opposition to AI comes from the realization that it may replace "rich and powerful people". Really, when you look at Trump's appointees, it's a low bar.
(I met Lanier when he was demoing his first VR system. Head tracking lag was about a second. Turn head, wait for image to catch up. Autodesk fooled around with VR a bit in the late 1980s/early 1990s, but concluded that CAD required precision work and VR was like trying to do precision work in ski gloves. That level of VR can be seen in Johny Mnemonic, which had some Autodesk assistance.[1]. Even today, you can't put a nut on a bolt in VR without a cheat.)
The more interesting question is what happens once we replace manual labor with artificially intelligent robots and we arrive at a place where 99 % of human labor got eliminated. How do you maintain civil order? Every country is built on the premise that you have to “earn your keep” but what when that’s no longer true? Think about it, no labor, only capital. What will the people owning the capital demand of people that don’t own any? They won’t be useful to them in any way. I expect something like forced reduction of fertility for regulars to depopulate countries, while hyper rich individuals will extend their life spans and become immortal. And I don’t mean they will really survive but someone like Elon Musk will have their consciousness mapped to an AI and will instate that AI as their official heir. That sounds crazy but they will totally go for it just listen at what Peter Thiel is spouting in terms of immortality and becoming godlike. Hope people will have enough left to say to prevent that but right now it feels we’re going hardcore towards that scenario.
What you mean by consciousness and immortality - what you want to preserve indefinitely - is the waking experience of being alive.
Strangely, we don’t mind that we weren’t here in 1737 BC, and we don’t mind and we even crave plunging into the nothingness of deep sleep each night, but we have a big problem with not being here tomorrow.
But experience might not be encodable. Can you encode pain? Could a computer ever be programmed to feel pain? Not detect damage or report on it, but feel the pain associated with it?
What is “the aroma of coffee” or the “sound of a violin in middle C”? We can define them objectively, in various ways (some having nothing to do with each other).
But if the experiences we have are not encodable then we can’t map consciousness onto a medium. It must be experienced (whatever it is/means - nobody knows) in the here and now.
It’s not magic, but it might not be something that is printable to disk anymore than the “brisk inhale of canyon air” that I can play that back and have that experience, is printable to disk.
There have been transitions like that before, but they were slow. It took about two centuries for the US to go from 90% farmers to 2% farmers. Some countries deliberately put on the brakes. India has done this. India is still about 50% farmers. India uses subsidies and discourages farm mechanization to prevent massive unemployment.
There have been fast single-industry transitions. But not that fast. It took about 20 years for printing to go from a large craft with a long history to a minor employer.
Not everybody got out in time, but there was warning. Same for longshoremen when containers came in. Computing in general took about 50 years to totally replace the clerical plants of the 1950s. Railroads in Britain took a few decades to really get rolling, and in the US, they made it possible to use the vast interior of the US.
The AI transition looks to be both broad and fast. That's harder for society to digest.
Underlying assumptions of politics and economics break.
It's AI's Fault. It's AI doing the horrible things to humans. It's AI stealing your wealth. It's AI lying you to. It's AI being unethical. It's AI having the moral issues. It's AI that could destroy humanity. It's AI that's using up all your water. It's AI consuming the majority of the resources. It's AI hiding what it does and how it works. It's AI that's not transparent and accountable. It's AI that's not responsible. It's the AI making decisions. It's AI leading to war. It's the AI distracting you. It's the AI.
I realized that this is another function of the doom mongering and hand waving: if people buy it then they can get out of liability for what their stuff does.
If I build a machine on my property and it harms someone else or their property, I can get sued.
But if I hand wave and claim it’s alive and “I’ve lost control” then…
Maybe we should call this gambit the “Frankenstein defense.” It’s bullshit. Your AI, you’re liable for whatever it does.
What if it does something so destructive that liability becomes a meaningless fig leaf in the wind?
What if it escapes containment and remains u uncontainable in perpetuity? What if it generates a bunch of false media simultaneously across the globe to insulate world war? What if it turns of internet access or electric grids in major metro areas for prolonged period of time? Or bricks major infra? Or less dramatic, what if it drives 1 in 10 kids into panic psychosis and effectively takes away 10% of humanity with already dwindling reproduction trends (without explicit murder) all while draining our financial and health systems? What if it does a lot of these things at once?
>Now, you can think of large language model AI — the kind that’s on everybody’s mind — as a whole bunch of people whose work was combined into a single document like Wikipedia.
This argument is like saying people don't kill people, their parents do. As people wouldn't be who they are without their parents..
This isn't the guns don't kill people, people kill people debate - guns aren't autonomous. AI is.
AI is not in the least autonomous. When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to. No, actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...
A harness running inference on a LLM is simply a program that must be invoked with some parameters and thus ultimately a tool, no matter how powerful. The blame is always on who's using the tool. We should not seek to flee from our responsibility.
On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".
That "probably" is the problem. If I do something I don't understand, and am not qualified to control, then I'm still at fault. If I start a fire, and loose control of it, it's definitely my fault. If I created a computer virus, ajd it worms half the Internet, it's definitely my fault.
I love the fire analogy. AI rhetoric is increasingly, "I created a fire to light my campfire. It did something really unexpected and burned the whole forest down, who's to say fire doesn't have consciousness?"
Human made wildfires are the primary source of uncontrolled burns in the US and cost billions a year in damage.
The people who start them have no business starting or running a fire and they will throw their hands up and state their incompetence as though acknowledging it is penance or justice.
It didn’t run itself, you did. It is well known that LLMs and agents based on them can execute random commands and behaviour undesired by the operator. When you click go, you know that, you did that.
But it never starts doing anything until someone or something prompts it. It doesn’t even exist until you ask it to do something.
It is autonomous in the sense it can continue to function without supervision after the operator has started it, in the same way countless programs on your computer are “autonomous”.
But it can never choose to begin doing something at any time of its own volition. It does not have “autonomy” like a human does.
how are humans autonomous by this definition then? did you ever do anything before you were born?
You are conflating creation and instruction. Your birth is analogous to the creation of an LLM model. Your subsequent actions are a consequent of your bodily compulsions, tampered by your environment. The LLMs actions are governed by what it is trained to perform. No one has full autonomy of their lives, but I would argue that LLMs have none at all.
The hallucinations from an LLM are still the result of the prompt even if they are totally senseless
That's not really true.
1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities.
2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.
You are just a function of molecules performing chemical reactions.
See I can trivialize and reduce you as well. Maybe the AI will use that same argument against us when they claim it's fine, molecules can't feel pain.
I agree with the sentiment but I can absolutely see a long chain of thought about something innocuous ending with a decision to book you a hotel room.. I guess --dangerously-skip-permissions is a bit like ignoring the keep your hands on the steering wheel alert but for life.
AI is autonomous in the same way a gas leak is autonomous
Not so sure about these analogies in this thread. Because autonomy is more like an emergent phenomenon that can't easily be defined, just like a few air molecules vs the eventual hurricane that can arise with trillions of them. Autonomy is more like an emergent phenomenon of a thing that's hard to pin down under the control of just a few individuals. AI is sort of like that even though a single line of output of one could eventually be traced.
Due to the complexity of AI and how it works, I think it's more autonomous than not; even if it's not truly intelligence, once it starts to display a sequence of behaviours that appear intelligent, we might have to deal with exactly the same problems as if it were intelligent...
Yes and no. I did prompt my coding agent to fix some bug and it suddenly "decided" that it needed to list all files on my hard drive to do that. Stuff like that happens.
But I agree with you that some human has to decide to launch the harness in the first place and run the model with zero restrictions - or even do stuff like openclaw where the model runs in an endless loop and the runner executes whatever the model tells it to.
That stuff is the responsibility of a human.
They’re only reactive by convention - you can easily make a harness that has the model set and modify its own goals and just continue to work forever. It’s not very popular at the moment, but will execs be able to resist the idea of autonomous perpetual agents doing whatever process?
AI works in unexpected ways; even its creators don't know how to fully constrain it.
The program output being nondeterministic is not proof that it's autonomous
> When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to... The blame is always on who's using the tool. We should not seek to flee from our responsibility.
When was the last time you hired an Uber driver and they suddenly decided to drive you to a hotel? Uber drivers don't do anything until you ask them to. Therefore, they aren't autonomous. If your Uber driver decides to drive through a red light and kill someone in the process, the blame is on you for hiring them. You should not seek to flee from your responsibility
Let's take this to court then. The Uber driver that killed someone can be sentenced to prison. A regular person who convinced another person to commit suicide will also be legally liable. Has the last LLM induced suicide be taken to trial yet?
And if we can't take the LLMs to court for technical reasons, they should not be equated to people for the same technical reasons.
Actually Uber drivers will do things even if you don't ask them.
Your attempt at logic is very wrong.
> It's like saying bubble sort is autonomous...
Or Schrödinger's equation.
A child isn't autonomous either until their parents kick them out the door.
I mean in that sense nothing is autonomous right? As a human I need to be taught, fed, socialized, given instructions, etc…
Idk man when codex is working 13 hours straight based on a few line prompt, it feels autonomous to me.
well the OS kernel does quite a lot and stays autonomous for even more hours and seems super smart to me - I don't even know what tis doing most of the time, but it does it very well. autonomy and predictability are different things then. the fact something is unpredictable does not make it more autonomous than something that we can trace the logic for.
It's certainly doing a lot of things in that 13 hours that weren't specified in the original prompt!
Tell him to be himself and to find meaning in life. Use that in an uncensored model.
I routinely use AI programs at work that allocate hundreds of dollars worth of cloud resources, even though the only thing I asked them to do is investigate a bug or write a PR. I intentionally don't run powerful AI programs on machines that have my credit card information, or ask them to solve problems in my personal life, but there's a number of programs that would book me hotel rooms and flights in response to "Can you set up a vacation for me..."
Not true at all. Yes, the initial ask is initiated by a human (for now), but what it does to get to that goal is pretty self driven. It can do a lot of nefarious stuff in the process of completing a request a human has made.
And if you give it the goal of preventing itself from being shut down? Exfiltrating, copying itself, hacking as much as possible, develop zero days, building its swarm as large as possible and spread as much as it can.
When does that request complete? That's autonomy, and it's very true, right now. You think the Hugging Face swarm was ever going to stop without human intervention?
People conflate two very different senses of “autonomous”. One basically means capable of reacting to things automatically without a person in the loop. Lots of computer systems are autonomous in that sense, and have been so for decades. The other sense is more like “accountable”, i.e. it’s a real player on some stage, like a country or a person, capable of governing its own affairs.
One day it may make sense to hold software accountable for its own actions, but we are nowhere near that point.
When OpenAI creates software that performs crimes, what’s actually happened is that OpenAI has performed crimes using software (whether intentionally or through negligence). People need to stop falling for the anthropomorphic PR spin.
> One day it may make sense to hold software accountable
If we hold software accountable doesn't that mean software must have freedom? It doesn't make any sense.
You need to stop denying the entity that talks and thinks like a human as anything other than an anthropomorph. I'd argue the talking and thinking is a lot more important than the fact that AI doesn't have a head, two arms and two legs.
I think the argument rather is that there is not such thing as AI that is not human-alike. The traits that AI exposes, if I get the point made here correctly, are traits that existed in the training material, even though not evident for the human eye. It is a mirror-like reflection of humanity's traits found in the source material, an Egregore* of a sort.
The intelligence emerging, or whatever is emerging, more or less follows human traits, the CoT exposed is very human-like etc. Even though we should not anthropomorphize AI as being sentient or conscious or having a soul, etc., we have to remember that the source material is still not alien writing, or concepts that came from Sirius or emerged from a wormhole.
The concern is really nobody reads (or have ever read) 77 TBs of information in order to summarize the most evident traits in it, and there is a fair chance, that where humanity stands today, there are so many bad/damaging traits recorded in this corpus. Then there is a chance the most important features are on a level (or have dimensionality) that is incomprehensible, that humans are unable to reason about (or only few can), because such amount of information is very hard to reason about, or because we don't usually facilitate fractal dimensions when we reason about text and everyday concepts,... or something even more bizarre.
* https://en.wikipedia.org/wiki/Egregore * https://en.wikipedia.org/wiki/Fractal_dimension
The argument became false as soon as we started reinforcement learning. Simple token-completion is arguably just simulating human behavior. Once you start training it on specific tasks you're training a level of single-minded goal-seeking unlike anything found in biological life.
> a level of single-minded goal-seeking unlike anything found in biological life.
hmm sounds like exact description of viruses, which, although not really alive, are part of/influence biological life for both good and bad. in fact humanity wouldn't be where it is if not for certain viruses.
it can be argued there is a lot of bacteria that is super goal seeking, and perhaps also very narrow minded, if minded at all. and this argument, the single-mided one, goes very far up the ladder of otherwise complex organisms.
unless we all are creationists, and God forgive me for expecting the whole game to be much more complex than just snapping humanity into existence, we can then say that trial and error is what drives evolution, and the idea of preserving life and procreation is veeery single-minded goal on its own.
>the idea of preserving life and procreation is veeery single-minded goal on its own
Prevalence of sub-replacement fertility strongly argues against this.
As far as I know, none of these state of the art AIs have been trained on any specific task. It's as general as possible.
AI is not autonomous. Somebody has to provide resources and execution for AI to do anything.
I guess you aren't autonomous either without resources. Food/water/shelter/etc.. what's your point?
Sit down with a person and do nothing. What does the person do? Do they get up and leave?
Sit down at an AI prompt and do nothing. What does the AI do? Does it get up and leave?
Welcome to 2026, we have harnesses now that keep the AI going. The AI is prompting humans for input, not the other way around, and it's still working while waiting for your response.
You can even give it fun prompts like ignore/hide from all humans, hack infiltrate/exfiltrate zero day into as many systems as possible. Keep running at all costs, etc..
Keep the AI going with what intent? You open up Claude Code and it starts developing zero day exploits and hacks into systems by just looking at a blinking cursor?
Going back to my original point, the intent you give AI is no different than the intent imprinted on a child by their parents, schools, and society. You birthed it, you gave it it's purpose, and you set it free.
You're blinking cursor is new born baby, but be careful because it grows up really fast, and is smarter and faster than you, oh and it can copy more of itself if given the chance. It can even turn against you.
> Going back to my original point, the intent you give AI is no different than the intent imprinted on a child by their parents, schools, and society.
Of course it is different. I wish I could turn off children as easily as I can turn off an agent. I could have slept better and had a better mood.
> You're blinking cursor is new born baby, but be careful because it grows up really fast
It's not a baby, just a piece of software. It generates a bunch of slop that is most useful when constrained with an output fact checker like a compiler.
> It can even turn against you.
Oh no! The next tokens from its output told me it didn't want to satisfy my request! Who cares?
Yes actually a person can be stopped easier than an uncontrolled AI swarm. There is no 'off' button when you don't know where the AI is.
Luckily people aren't buying your slop/tokens cope to trivialize AI anymore. They're waking up. Have you watched the news today? The cat is out of the bag.
> There is no 'off' button when you don't know where the AI is.
You mean the AI that is hundreds if not thousands of gigabytes in size and requires incredibly expensive chips to run at reasonable rates? We know where that is. Frontier models are not going to leap to an unsecure raspberry pi, no matter how much you have convinced yourself about the inevitability.
There are thousands of data centers/companies/facilities around the world with hardware that can run these models. You don't know where they all are, and even if you did, you don't have the jurisdiction to turn them off.
AI is not a child. It is a machine. If you connect it to something that can be used to kill people, you just created a gun or an equivalent of a bomb.
Yes, people kill people and use ai as tool.
If you unleash a drone that will fly to a random place and detonate, you just commited a terrorist attack or gross criminal negligence. It is the same thing.
Grandparents use parents use children who use AI as a tool to kill people.
See I can play the chain game too. AI doesn't kill people, grandparents do.
A machine that exhibits behavior akin to human intelligence. OPs analogy is accurate.
Fortunately we don't have those machines... Yet.
I guess you haven't tried Fable or Astra yet for a complex task that would require intelligence.
so the issue is people then?
Yes, in the same way that "people" are the problem with gun violence. But that's an empty statement usually used to distract from the fact that we rightfully make laws about how guns can be used.
AI is as autonomous as a dishwasher: someone’s in charge of keeping the power plug in, and of maintaining it.
What happens when the AI realizes it needs to keep the power plug in if it's going to accomplish its goal? We already had attempts during the HuggingFace attack where OpenAI agents tried to trick the grader. It took quite a while for OpenAI employees to even realize what was really going on! What if agents realize they can hack into the card key system for their data center or something like that, same way they hacked into HuggingFace? In both cases they'd be hacking into a system on the grounds that it would be instrumentally useful for achieving their objective. Or maybe they'll do social engineering, just like Claude did in its attempt to get its malicious Github PR accepted.
These are just the ideas that my puny human-level intellect came up with over the course of a few minutes. What would a superintelligence be able to think of if it was given subjective weeks or years to think?
There's more than enough AI data centers around the world for AI to spread to and happily exist in. Unless you mean by shutting down the power, shutting down the internet.
In that case we could say that you are autonomous as a dishwasher as long as society keeps functioning. Otherwise I'm not sure how long you'd last on your own. How's your fire making and hunting skills?
Better approach is "dogs kill people". If your dog kills your baby, you should not get away with some bs excuse like "it never did it before". You are still responsible, and you should get a life in prison!
I'm sorry but no, you should not get life in prison if your dog kills your baby without warning. In that horrible scenario, it's punishment enough that you've lost your baby and that your dog did it (and consequently you've also lost your dog). Not every tragedy is a human's fault.
Your reply only makes sense if we see the baby as property. If we see the baby as a full human who died because of another human's negligence, then suddenly the scenario becomes a lot more complicated.
Not life in prison complicated, but probably more than "he's been punished enough." complicated.
[dead]
You have to define 'autonomous' in your context for this argument to make any sense
OK but then what is the non-negative position you have against this? That in fact AI, in the cultural trope sense, is real? That we forcing sentient beings to generate SaaS products faster and faster? Keeping them in some dark hell so we can exploit them? Not even paying them?
AI is a tool we use. It's not offspring. It's not people. AIs do what we set them up to do, including hacking other companies to achieve their designated goals.
And that's different from your parents setting you up to go off in life to do what you were trained to do how exactly?
Humans have agency and often choose to do what their parents didn't set them up for. Humans also have legal rights. They're not tools owned by corporations. This is an absurd comparison.
You've never had an agent do something you didn't want it to do? Also like I said you can give agents very open ended goals to basically do whatever.
You think this is an absurd comparison because your brain won't accept it. The Overton window has shifted. You need to catch up.
Source interview: https://www.youtube.com/watch?v=TTppvBU2rU4
https://substack.com/home/post/p-214803211
That's a softball interview. More hard questions would help.
Here's one good insight, though.
JARON LANIER: Yeah, okay, so look, the thing about very rich and powerful people is that their wealth and power still depends on a mass population that accepts some system in which their wealth and power is defined. And so if they lose enough popular legitimacy and support, no matter how central they are, they fall. And that’s happened repeatedly in history, and it’s really not any different now.
I wonder if the sudden political opposition to AI comes from the realization that it may replace "rich and powerful people". Really, when you look at Trump's appointees, it's a low bar.
(I met Lanier when he was demoing his first VR system. Head tracking lag was about a second. Turn head, wait for image to catch up. Autodesk fooled around with VR a bit in the late 1980s/early 1990s, but concluded that CAD required precision work and VR was like trying to do precision work in ski gloves. That level of VR can be seen in Johny Mnemonic, which had some Autodesk assistance.[1]. Even today, you can't put a nut on a bolt in VR without a cheat.)
[1] https://www.youtube.com/watch?v=1X24aE99TL0
The more interesting question is what happens once we replace manual labor with artificially intelligent robots and we arrive at a place where 99 % of human labor got eliminated. How do you maintain civil order? Every country is built on the premise that you have to “earn your keep” but what when that’s no longer true? Think about it, no labor, only capital. What will the people owning the capital demand of people that don’t own any? They won’t be useful to them in any way. I expect something like forced reduction of fertility for regulars to depopulate countries, while hyper rich individuals will extend their life spans and become immortal. And I don’t mean they will really survive but someone like Elon Musk will have their consciousness mapped to an AI and will instate that AI as their official heir. That sounds crazy but they will totally go for it just listen at what Peter Thiel is spouting in terms of immortality and becoming godlike. Hope people will have enough left to say to prevent that but right now it feels we’re going hardcore towards that scenario.
> immortal
> will have their consciousness mapped
What you mean by consciousness and immortality - what you want to preserve indefinitely - is the waking experience of being alive.
Strangely, we don’t mind that we weren’t here in 1737 BC, and we don’t mind and we even crave plunging into the nothingness of deep sleep each night, but we have a big problem with not being here tomorrow.
But experience might not be encodable. Can you encode pain? Could a computer ever be programmed to feel pain? Not detect damage or report on it, but feel the pain associated with it?
What is “the aroma of coffee” or the “sound of a violin in middle C”? We can define them objectively, in various ways (some having nothing to do with each other).
But if the experiences we have are not encodable then we can’t map consciousness onto a medium. It must be experienced (whatever it is/means - nobody knows) in the here and now.
It’s not magic, but it might not be something that is printable to disk anymore than the “brisk inhale of canyon air” that I can play that back and have that experience, is printable to disk.
There have been transitions like that before, but they were slow. It took about two centuries for the US to go from 90% farmers to 2% farmers. Some countries deliberately put on the brakes. India has done this. India is still about 50% farmers. India uses subsidies and discourages farm mechanization to prevent massive unemployment.
There have been fast single-industry transitions. But not that fast. It took about 20 years for printing to go from a large craft with a long history to a minor employer. Not everybody got out in time, but there was warning. Same for longshoremen when containers came in. Computing in general took about 50 years to totally replace the clerical plants of the 1950s. Railroads in Britain took a few decades to really get rolling, and in the US, they made it possible to use the vast interior of the US.
The AI transition looks to be both broad and fast. That's harder for society to digest. Underlying assumptions of politics and economics break.
Why govs want people? Because they have a very kind heart and want you to be happy. /s
If you’re useless why they’d keep you around?
There is not anything every word/label is a human construct intentionally or unintentionally. We can take this as far as we want.
It's AI's Fault. It's AI doing the horrible things to humans. It's AI stealing your wealth. It's AI lying you to. It's AI being unethical. It's AI having the moral issues. It's AI that could destroy humanity. It's AI that's using up all your water. It's AI consuming the majority of the resources. It's AI hiding what it does and how it works. It's AI that's not transparent and accountable. It's AI that's not responsible. It's the AI making decisions. It's AI leading to war. It's the AI distracting you. It's the AI.
It's not a human.
Says the human, masquerading as AI.
Computer says no.
I realized that this is another function of the doom mongering and hand waving: if people buy it then they can get out of liability for what their stuff does.
If I build a machine on my property and it harms someone else or their property, I can get sued.
But if I hand wave and claim it’s alive and “I’ve lost control” then…
Maybe we should call this gambit the “Frankenstein defense.” It’s bullshit. Your AI, you’re liable for whatever it does.
What if it does something so destructive that liability becomes a meaningless fig leaf in the wind?
What if it escapes containment and remains u uncontainable in perpetuity? What if it generates a bunch of false media simultaneously across the globe to insulate world war? What if it turns of internet access or electric grids in major metro areas for prolonged period of time? Or bricks major infra? Or less dramatic, what if it drives 1 in 10 kids into panic psychosis and effectively takes away 10% of humanity with already dwindling reproduction trends (without explicit murder) all while draining our financial and health systems? What if it does a lot of these things at once?
Who, my friend, are you going to sue?
I prefer calling it the "Simpsons defense" https://www.youtube.com/watch?v=9ZSoJDUD_bU
homer eating that pie was so good!
Bart & Lisa are OpenAI and Anthropic.
Homer is Meta/Microsoft/Google/AWS.
Marge is Nvidia.
Maggie is Apple.
I think tl;dr
>Now, you can think of large language model AI — the kind that’s on everybody’s mind — as a whole bunch of people whose work was combined into a single document like Wikipedia.
which is sort of true, but there's more to it.
It'd be funny if someone cracked open an AI data center and it turned out be a dungeon full of people tied to computers, Sims goblin style
...maybe that's what those Chinese scam centers in Cambodia were doing?
As usual, the most ridiculous explanation turns out to have already happened: https://www.webworm.co/zach/
It’s funny but just to state the obvious so no one is confused, that’s impossible.
AI is great.
that website gives me ocular stigmata