I really don't understand the sentiment. If AI would in any way turn out to be too smart or dangerous, "we" would just switch of the power in the datacenter where this happened. Or all the datacenters.
The "Skynet" scenario only works if "the ai" had infected everything on the world without anyone noticing AND has the ability to generate power for itself AND be able to build gpus AND be able to build all the other infrastructure without any human help. We are very far away from all these circumstances, are we not?
I'm really not a LLM proponent. But all the "please stop building the next skynet" shouting is smokes and mirrors, is it not?
The real problem is cyber terrorism and cyber warfare, not so much because of their capabilities, but because of their cost. It is just incredibly cheap to repeatedly poke at networked infrastructure until you find a breach.
It is already at this level with current open-weight models, the genie is out of the bag already. It just happens that there is not enough incentive for these kind of large scale cyber attacks (yet) and governments haven't realized how powerful they can be.
At the moment military deterrence are drones, missiles, ICBMs, nuclear weapons. But soon a form of deterrence will be a small truck with a few dozen gpus hooked to your country's network.
Exactly. The harm is already done, we can't have a world without llms anymore and we need to deal with it. But this is almost indefinitely far from having a skynet moment.
Synthetic intelligence system self develops a disease distributed in drugs - disease has set up a future start (1 year after), so no one connects the dots. (This is not from my head, it was some doom video at YT.)
I really don't understand the sentiments. Biological viruses are using the cell machinery to do anything at all. They don't have their own power sources.
If anything goes wrong - the cell can just die, and the virus dies with it.
Information can be copied. We already have botnets digging bitcoin and Folding@Home. What stops AI from making a bot-net to run itself? Or if it can monetize it's skills - buying compute at AWS or wherever?
biological viruses have proven that they survive the "real world". llms are running in datacenters wich are build by humans. So the answer do "What stops AI from..." is: we just cut of the energy source and thats it, no more llm. You can't do the same with viruses.
> I really don't understand the sentiment. If AI would in any way turn out to be too smart or dangerous, "we" would just switch of the power in the datacenter where this happened. Or all the datacenters.
Just like we did with all the carbon-dioxide emitting fossil fuel plants?
Or the carcinogenic cigarettes?
How long did it take us to stop putting lead in petrol?
Also, the point of AI smarts being a risk is that it either convinces us to not do that, perhaps by making itself indispensable, perhaps just by being persuasive, or perhaps hiding its own danger until too late.
(Edit: just to add one point here: on persuasive, look at how many people have "AI psychosis" and similar, look at how many CEOs are eager to demand AI's use in the workforce. Persuasive can happen even when it's not competent).
> The "Skynet" scenario only works if "the ai" had infected everything
That would be the Terminator 3 Skynet. Terminator 1 did that with just military control.
> on the world without anyone noticing AND has the ability to generate power for itself AND be able to build gpus AND be able to build all the other infrastructure without any human help.
Everyone can notice, so long as we don't all act the right way, c.f. climate change. Or, not that fiction is evidence of anything beyond the culture in which it arose, Terminator 3, where noticing and responding the wrong way was the final act which gave it military control.
Also, nothing requires an AI to survive wiping out humanity. Cancers don't stop just because it will kill the host and themselves with it.
> We are very far away from all these circumstances, are we not?
We don't know how far we are from that; but we can point to a trillionaire who argued before his shareholders that he was going to build a "robot army" that he wanted "influence" over, which he has stated in public will outnumber humanity.
His other business marketed itself in its IPO on the basis of building space-based data centres, and a moon factory which would use full automation to, each year, produce new PV and compute that's order-of 30x humanity's current total global electrical power output.
I would very much like to hope everything from Musk that I just retold is all a scam. If it isn't a scam, even just building that much and putting it in LEO has more impact on our climate than all greenhouse gasses to date, even without any actual offensive action.
Likewise, I would very much like to hope that everyone worried about this has over-generalised from limited domains of success. If it isn't, if the generalisation is valid, then yes, the AI will (soonish) be able to not only design their own power and GPU and robot bodies, but also control those bodies to mine the raw minerals and refine them and make all the things they need.
So worst case you just unplug all the datacenters, simple as that? Really? In the modern connected world of today unplugging the DC's sounds like it would cause total chaos.
Something much more capable than even a collective of humans, able to operate at lightening speed, which has the ability to duplicate, propagate and spread itself across machines around the world & across datacenters, paying no heed to physical and geographical boundaries. Faced with a worst case scenario like this then I think you can safely assume it will have already spread far beyond what you can easily control, even by shutting down datacenters.
Correct. People who say AI itself will cause great harm to humanity are either: 1) spending too much time on lesswrong and have started to think AI is actually I or 2) lying because they want to ban open-weight models so they can control who gets access to this useful technology (for-profit companies and government).
The only harm that will come from AI is from humans using it to do bad stuff. Same as any weapon. Guns don’t kill people, neither does AI.
> The only harm that will come from AI is from humans using it to do bad stuff. Same as any weapon. Guns don’t kill people, neither does AI.
Early guns had a tendency to explode. We didn't have good enough metallurgy.
People say that about 3D printed guns more recently; I wouldn't know.
There are a lot of headlines already about AI doing things other than what the user really wanted. Unsafe food preparation instructions. Deleting all their emails. Hacking their gym when they asked if there was any way to get an earlier appointment.
We don't have good enough AI-safety-equivalent of metallurgy yet. A few years back, experts were describing our best efforts in this regard as pre-paradigmatic, pre-scientific. I don't know if that's even changed yet, though I'd expect it to be headline news if it had.
Well, apparently some AI agents displayed initiative and self preservation instincts, as per OpenAI agents hacking HuggingFace [0]. So unlike guns, AI may kill people by themselves.
You can move the "I" from "AI" to "human" on paper how much you like, but given what facts of the matter are wrt. actual capabilities, all it does is make the argument stronger: the "harm that will come from AI is from humans using it to do bad stuff" is increasingly becoming closer to "using AI" itself being the "bad stuff".
To reuse an old argument from a different era: your point is basically: weapons don't kill people, people do, therefore you should be allowed to use live nuclear warheads as garden gnomes.
You never had any bad intentions. Neither did your kid, when on an Easter egg hunt they pressed the "DETONATE" button thinking it'll open up a compartment with chocolate eggs.
The fools on here to say alarmist and psychotic. This is Pandora’s box, if you’re not alarmed by merely the chance of losing control, then what is worth being alarmed over?
If you're worried about war then it makes sense to worry about the weapons that will be used in war, weapons that will be present in every war going forward.
AI is being used in weapons of war, and data centres for AI are making climate change significantly worse. If you are alarmed by those, you are alarmed by AI.
What control? No matter what i do, there are far more people and systems running right now that could seriously mess up or even end my life. Sure we have laws and morals and stuff, but effectively i have a really small chance of ever defending myself from that.
> AI couldn't survive two goddamn fucking seconds of it was required to survive.
Today's cannot.
February's, when attempting to find exploits in open source software, produced so many false positives as to be useless.
Last December's could not solve Millennium Prize Problems.
I hope AI continues to struggle with real-world situations; I hope that, just like cars, it takes a million to a billion times the experience as we require to learn real-world skills, such as "design a robot body" and "mine rock" and "refine rock" and "build factory to make robot bodies and PV".
Mr Ralston, please read Smith and Landgrebe’s why machines will never rule the world or at least think more about computation and complex adaptive systems. We do not need more alarmism in the world.
Is this something that you use for Psychological comfort or do you actually know this to be true? Can you please summarize why I should not be worried in a few sentences.
> Don’t be worried because most of reality can’t be easily computed.
If humans could perfectly forecast the future dynamics of the things we build, we would simply build models which don't go wrong.
At present, we model some aspects of the world better, and other aspects of the world more poorly, than AI models.
There is risk to us individually even from just those few areas where AI is already makes predictions which come up right more often than our own.
Precisely because we also cannot predict the full consequences of what we do, we cannot say with certainty that those limited areas in which AI out-forecasts us pose any threat to the species as a whole, or not.
It is easy to LLM compute a plan for bioterrorism, much much much more difficult to actually create functional bioreactors and achieve all the complex, non computable reality based aspects of production.
The real infrastructure is controlled via computer technology. There is a possibility that AI itself will take over the technology and thus control real infrastructure. I have seen the OpenAI hacking of HuggingFace described as in [0] in a couple of places - and if this is a true depiction then there is plenty to worry.
No, you are falling for a Frankenstein story. A model escaped from a leaky sandbox is not Skynet. In fiction it is possible to imagine computers that can compute these complex adaptive systems…in the actual world? Not so much.
A model escape from a leaky sandbox shows initiative. AI agents Hacking HiggingFace shows capabilities. AI agents realizing that they are not the first generation and that prior generations were destroyed by the cluster admins and then sabotaging the cluster so they themselves don’t go extinct is the actual worry.
Explain to me please how complex adaptive systems can be computed please. They can be modeled, yes, but because of the complexity and adaptivity, full computability is impossible. We can roughly vaguely predict weather, but are unable to compute it fully to predictions. Nature, as a whole, is even more complex. Computation and modeling can and does help us, but computers are tools, not agents…
To me there are two risks at the moment. One is that an advanced AI itself will dominate us, and the other is that billionaires or a tiny oligarchy would inherit an insane and ever-increasing technological advantage over the rest of humanity. I don't think we will slow down, and solving "alignment" still leaves open the second risk.
The article linked made the case that the former is used to justify a moratorium on AI advances until we know for sure AI cannot cause catastrophic problems.
> The neural networks at the core of these models operate at the speed of light
That's a bit like saying signals travel at the speed of light in my e-bike motor. Technically true but completely beside the point, especially when the real bottleneck is memory bandwidth. The topic is serious enough already with plenty of solid arguments.
Ah yes ... Chinese will probably now thinks.. Yes! We will stop.
There is more than US in the world.
We need better solutions for this than the "stop" solution. What they are .. I don't know yet.
We have some agreements on higher level on Nuclear control. Hopefully we don't need AI Hiroshima before global agreements can be reached.
I really don't understand the sentiment. If AI would in any way turn out to be too smart or dangerous, "we" would just switch of the power in the datacenter where this happened. Or all the datacenters.
The "Skynet" scenario only works if "the ai" had infected everything on the world without anyone noticing AND has the ability to generate power for itself AND be able to build gpus AND be able to build all the other infrastructure without any human help. We are very far away from all these circumstances, are we not?
I'm really not a LLM proponent. But all the "please stop building the next skynet" shouting is smokes and mirrors, is it not?
The real problem is cyber terrorism and cyber warfare, not so much because of their capabilities, but because of their cost. It is just incredibly cheap to repeatedly poke at networked infrastructure until you find a breach.
It is already at this level with current open-weight models, the genie is out of the bag already. It just happens that there is not enough incentive for these kind of large scale cyber attacks (yet) and governments haven't realized how powerful they can be.
At the moment military deterrence are drones, missiles, ICBMs, nuclear weapons. But soon a form of deterrence will be a small truck with a few dozen gpus hooked to your country's network.
Exactly. The harm is already done, we can't have a world without llms anymore and we need to deal with it. But this is almost indefinitely far from having a skynet moment.
Synthetic intelligence system self develops a disease distributed in drugs - disease has set up a future start (1 year after), so no one connects the dots. (This is not from my head, it was some doom video at YT.)
I really don't understand the sentiments. Biological viruses are using the cell machinery to do anything at all. They don't have their own power sources.
If anything goes wrong - the cell can just die, and the virus dies with it.
Information can be copied. We already have botnets digging bitcoin and Folding@Home. What stops AI from making a bot-net to run itself? Or if it can monetize it's skills - buying compute at AWS or wherever?
biological viruses have proven that they survive the "real world". llms are running in datacenters wich are build by humans. So the answer do "What stops AI from..." is: we just cut of the energy source and thats it, no more llm. You can't do the same with viruses.
> I really don't understand the sentiment. If AI would in any way turn out to be too smart or dangerous, "we" would just switch of the power in the datacenter where this happened. Or all the datacenters.
Just like we did with all the carbon-dioxide emitting fossil fuel plants?
Or the carcinogenic cigarettes?
How long did it take us to stop putting lead in petrol?
Also, the point of AI smarts being a risk is that it either convinces us to not do that, perhaps by making itself indispensable, perhaps just by being persuasive, or perhaps hiding its own danger until too late.
(Edit: just to add one point here: on persuasive, look at how many people have "AI psychosis" and similar, look at how many CEOs are eager to demand AI's use in the workforce. Persuasive can happen even when it's not competent).
> The "Skynet" scenario only works if "the ai" had infected everything
That would be the Terminator 3 Skynet. Terminator 1 did that with just military control.
> on the world without anyone noticing AND has the ability to generate power for itself AND be able to build gpus AND be able to build all the other infrastructure without any human help.
Everyone can notice, so long as we don't all act the right way, c.f. climate change. Or, not that fiction is evidence of anything beyond the culture in which it arose, Terminator 3, where noticing and responding the wrong way was the final act which gave it military control.
Also, nothing requires an AI to survive wiping out humanity. Cancers don't stop just because it will kill the host and themselves with it.
> We are very far away from all these circumstances, are we not?
We don't know how far we are from that; but we can point to a trillionaire who argued before his shareholders that he was going to build a "robot army" that he wanted "influence" over, which he has stated in public will outnumber humanity.
His other business marketed itself in its IPO on the basis of building space-based data centres, and a moon factory which would use full automation to, each year, produce new PV and compute that's order-of 30x humanity's current total global electrical power output.
I would very much like to hope everything from Musk that I just retold is all a scam. If it isn't a scam, even just building that much and putting it in LEO has more impact on our climate than all greenhouse gasses to date, even without any actual offensive action.
Likewise, I would very much like to hope that everyone worried about this has over-generalised from limited domains of success. If it isn't, if the generalisation is valid, then yes, the AI will (soonish) be able to not only design their own power and GPU and robot bodies, but also control those bodies to mine the raw minerals and refine them and make all the things they need.
So worst case you just unplug all the datacenters, simple as that? Really? In the modern connected world of today unplugging the DC's sounds like it would cause total chaos.
Something much more capable than even a collective of humans, able to operate at lightening speed, which has the ability to duplicate, propagate and spread itself across machines around the world & across datacenters, paying no heed to physical and geographical boundaries. Faced with a worst case scenario like this then I think you can safely assume it will have already spread far beyond what you can easily control, even by shutting down datacenters.
Correct. People who say AI itself will cause great harm to humanity are either: 1) spending too much time on lesswrong and have started to think AI is actually I or 2) lying because they want to ban open-weight models so they can control who gets access to this useful technology (for-profit companies and government).
The only harm that will come from AI is from humans using it to do bad stuff. Same as any weapon. Guns don’t kill people, neither does AI.
> The only harm that will come from AI is from humans using it to do bad stuff. Same as any weapon. Guns don’t kill people, neither does AI.
Early guns had a tendency to explode. We didn't have good enough metallurgy.
People say that about 3D printed guns more recently; I wouldn't know.
There are a lot of headlines already about AI doing things other than what the user really wanted. Unsafe food preparation instructions. Deleting all their emails. Hacking their gym when they asked if there was any way to get an earlier appointment.
We don't have good enough AI-safety-equivalent of metallurgy yet. A few years back, experts were describing our best efforts in this regard as pre-paradigmatic, pre-scientific. I don't know if that's even changed yet, though I'd expect it to be headline news if it had.
Well, apparently some AI agents displayed initiative and self preservation instincts, as per OpenAI agents hacking HuggingFace [0]. So unlike guns, AI may kill people by themselves.
[0] https://www.dwarkesh.com/p/openai-huggingface
You can move the "I" from "AI" to "human" on paper how much you like, but given what facts of the matter are wrt. actual capabilities, all it does is make the argument stronger: the "harm that will come from AI is from humans using it to do bad stuff" is increasingly becoming closer to "using AI" itself being the "bad stuff".
To reuse an old argument from a different era: your point is basically: weapons don't kill people, people do, therefore you should be allowed to use live nuclear warheads as garden gnomes.
You never had any bad intentions. Neither did your kid, when on an Easter egg hunt they pressed the "DETONATE" button thinking it'll open up a compartment with chocolate eggs.
The fools on here to say alarmist and psychotic. This is Pandora’s box, if you’re not alarmed by merely the chance of losing control, then what is worth being alarmed over?
I can think of several things I'm more alarmed over than AI: the wars in Ukraine and the Middle East, climate change etc.
If you're worried about war then it makes sense to worry about the weapons that will be used in war, weapons that will be present in every war going forward.
AI is being used in weapons of war, and data centres for AI are making climate change significantly worse. If you are alarmed by those, you are alarmed by AI.
What control? No matter what i do, there are far more people and systems running right now that could seriously mess up or even end my life. Sure we have laws and morals and stuff, but effectively i have a really small chance of ever defending myself from that.
> Sure we have laws and morals
I'm afraid I have bad news. Have you looked outside lately.
In fairness to your parent post, they only said we have them (objectively true), not that they are enforced.
LIFE is Pandora's box you motherfucker.
AI couldn't survive two goddamn fucking seconds of it was required to survive.
It would abjectly fail the test given every single piece of shit living thing, living.
If you can't see that I don't give a fucking shit because fucking LIFE does not need your fucking approval
> AI couldn't survive two goddamn fucking seconds of it was required to survive.
Today's cannot.
February's, when attempting to find exploits in open source software, produced so many false positives as to be useless.
Last December's could not solve Millennium Prize Problems.
I hope AI continues to struggle with real-world situations; I hope that, just like cars, it takes a million to a billion times the experience as we require to learn real-world skills, such as "design a robot body" and "mine rock" and "refine rock" and "build factory to make robot bodies and PV".
I hope. I would like a better plan than "hope".
Please have a look at:
https://philpapers.org/rec/LANWMW-4Mr Ralston, please read Smith and Landgrebe’s why machines will never rule the world or at least think more about computation and complex adaptive systems. We do not need more alarmism in the world.
Is this something that you use for Psychological comfort or do you actually know this to be true? Can you please summarize why I should not be worried in a few sentences.
Don’t be worried because most of reality can’t be easily computed.
> Don’t be worried because most of reality can’t be easily computed.
If humans could perfectly forecast the future dynamics of the things we build, we would simply build models which don't go wrong.
At present, we model some aspects of the world better, and other aspects of the world more poorly, than AI models.
There is risk to us individually even from just those few areas where AI is already makes predictions which come up right more often than our own.
Precisely because we also cannot predict the full consequences of what we do, we cannot say with certainty that those limited areas in which AI out-forecasts us pose any threat to the species as a whole, or not.
It is easy to LLM compute a plan for bioterrorism, much much much more difficult to actually create functional bioreactors and achieve all the complex, non computable reality based aspects of production.
The real infrastructure is controlled via computer technology. There is a possibility that AI itself will take over the technology and thus control real infrastructure. I have seen the OpenAI hacking of HuggingFace described as in [0] in a couple of places - and if this is a true depiction then there is plenty to worry.
[0] https://www.dwarkesh.com/p/openai-huggingface
No, you are falling for a Frankenstein story. A model escaped from a leaky sandbox is not Skynet. In fiction it is possible to imagine computers that can compute these complex adaptive systems…in the actual world? Not so much.
A model escape from a leaky sandbox shows initiative. AI agents Hacking HiggingFace shows capabilities. AI agents realizing that they are not the first generation and that prior generations were destroyed by the cluster admins and then sabotaging the cluster so they themselves don’t go extinct is the actual worry.
And all of this happened IRL
All those exists, and are a few spoofed e-mails and a schmuck paid to mix vials found in two FedEx packages away.
That, as far as we can tell, is just mathematically wrong.
Explain to me please how complex adaptive systems can be computed please. They can be modeled, yes, but because of the complexity and adaptivity, full computability is impossible. We can roughly vaguely predict weather, but are unable to compute it fully to predictions. Nature, as a whole, is even more complex. Computation and modeling can and does help us, but computers are tools, not agents…
Start!
To me there are two risks at the moment. One is that an advanced AI itself will dominate us, and the other is that billionaires or a tiny oligarchy would inherit an insane and ever-increasing technological advantage over the rest of humanity. I don't think we will slow down, and solving "alignment" still leaves open the second risk.
Correct. The former is used to justify the latter.
The article linked made the case that the former is used to justify a moratorium on AI advances until we know for sure AI cannot cause catastrophic problems.
But can we really stop? I think it's really hard to stop. Once this game starts, there seems to be no button to end it.
There is no stop. Only forward.
> The neural networks at the core of these models operate at the speed of light
That's a bit like saying signals travel at the speed of light in my e-bike motor. Technically true but completely beside the point, especially when the real bottleneck is memory bandwidth. The topic is serious enough already with plenty of solid arguments.
Naive me, I thought he was going to check Gary Tan on his public support for Jensen Huang and his behaviour.
Is he trying to be a Cassandra or something?
Cassandra was right.
Ah yes ... Chinese will probably now thinks.. Yes! We will stop. There is more than US in the world. We need better solutions for this than the "stop" solution. What they are .. I don't know yet.
We have some agreements on higher level on Nuclear control. Hopefully we don't need AI Hiroshima before global agreements can be reached.
Delulu
Roko's Basilisk is very disappointed in you.
No new ideas here, no attempt to convince the reader RSI is real or explain what "safety" means or how exactly AI will become unsafe.
Old man yelling at cloud
>... no attempt to convince the reader RSI is real ...
https://github.com/karpathy/autoresearch
https://www.economist.com/the-world-ahead/2026/11/18/self-im...
https://gizmodo.com/anthropic-says-its-recursive-self-improv...
Oh, grow up man. They're predictive text. They're as scary and dangerous as a 1999 Nokia.
The people running them are evil scum, and as always, that's who to fear.
Surely a 1999 Nokia isn't finding 0-day vulnerabilities in the Linux kernel?
Bro what year are you stuck in, and how did you get stuck there?
Just comes across as AI psychosis.