>These systems are not accessible via the internet.
How would you know? Are you verifying them? SCADA systems weren't intended to be online, but that didn't stop critical infra for power plants, etc. to show up online. The convenience is too great to not try to remotely maintain them 'securely' & the assurance of airgaps is technical and expensive to get right.
Besides, are you considering the addition of modern AI to these 'offline' systems - factories, power, national weapons? You can get frontier models in a fraction of a rack today, who knows tomorrow. The organizations running these systems want in on the technological possibilities of AI, too.
Not to be a doomer about all this or anything, but there's a lot of evidence technologically superior entities dominate technologically inferior ones. So our home countries are in a Red Queen's Race to keep up, which is why adding "local" or "offline" AI to these systems seems probable. Again, airgaps are hard to maintain. It is within the realm of possibility that what's offline today is accidentally or intentionally networked tomorrow.
Yeah that's a fair point. Best practices and actual practices sadly don't seem to correlate that much in the real world.
As far as security is concerned, things like hospital equipment, traffic control signalling systems, weapons arsenals and bioengineering medical labs should not be connected to the internet and should be designed in such a way that a human is required in the loop at all times. And shouldn't use AI to function in any way.
But yeah, fair point that a lot of organisations will ignore that in favour of "look how easy internet access makes controlling this thing" or "AI is so cool", forgetting how badly it leaves them at the mercy of their enemies (AI or otherwise).
> Because if an AI takes over the systems at a factory, it can't really do much beyond what the factory could already do. The production lines use single purpose robots and machines to build a particular product, and can't be programmatically altered to build anything else.
They're still pretty limited in what they can achieve military wise, or how much impact they can have in the real world. Don't get me wrong, robotics is getting a lot more advanced, and in future something like a Terminator might actually be plausible.
But at the moment, a robot revolution/uprising using this sort of tech would be easy enough to put down in about 30 minutes. And they don't have the capabilities to provide a chain of logistics that would allow them to set up lots more factories or get more resources.
The logistics side of things is really the tough part. If Skynet hypothetically existed, it'd need the following to cause any damage:
- Locations to build factories that aren't already occupied by other people, companies or uses (or a way to demolish the latter)
- The ability to construct such factories without the planning inspectors noticing, the neighbours noticing, the police or military shutting things down, etc.
- Equipment to construct robots on an assembly line
- Access to electricity, water and other resources
- Raw materials used for the machines to build said robots. This usually involves either finding a third party supplier or thirty that would sign off on supplying it without asking too many questions, stealing it from someone else without them stopping you, etc
- The ability to deploy said robots in a way that's effective for your goals and won't get them immediately shot down.
It could reuse existing factories if there were enough of them, but it'd need dozens or hundreds to cause any real issues.
Honestly, logistics are the problem with most apocalyptic scenarios like this. And for that matter, most stories with intelligent characters creating wonderous inventions in fiction.
Someone has to supply the materials, and the character or force has to be able to both afford them and not spark an investigation. They have to work with companies and contractors that might get suspicious about what's going on, and somehow stay on the down low while making an outsized impact on reality.
Oh sure, I agree that we're not imminent danger. However, the field is advancing quickly with us getting glimpses of what autonomous agents can do when seeded with a goal function. It is not too difficult to imagine adaptive and flexible goals that enable broad unpredictability and learnt behavior.
> Because if an AI takes over the systems at a factory, it can't really do much beyond what the factory could already do.
That can be enough, eg stuxnet. And like stuxnet they don't need to connect directly to the machinery themselves for a payload to breach them they just need someone to deliver the payload for them. And the machinery might be decades old, the software might still be Windows 95.
If you take the recent anthropic misuse report seriously, they don't need to "take over weapons" because they're actively being used to give a lift in capability to groups who already want to use weapons.
> The six cases in this section are divided into two parts. Part I covers four cases in which the actor in question used Claude to develop software for weapons themselves: a guided rocket program, in which the actors conducted a live field test; a design and proposal work on a system to intercept torpedoes; software for a drone swarm, tested in simulation, with its code loaded onto real boards; a targeting software for electronic warfare and for suppressing air defenses.
Your water supply is administered online.
Your land records, taxes, revenue are online.
Stock market is online.
Power systems are online.
Nuclear power reactors are (kind of) online.
Money to bribe people to do work is online.
There's a lot of havoc that is potentially possible online.
There's a wide array of possibilities that are difficult to characterize in a single HN comment, which is part of the problem you're probably trying to put a finger on. One threat can be completely plausible and the next completely silly and few people know how to distinguish between them.
Another issue is just the big whackload of nobody really knowing what is possible. Could an AI in, say, the next year, become Skynet and decide that it could just bump humans off and become its own independent economy? Probably not successfully, but how much trouble could it make if it incorrectly decided that was the right thing to do? What if someone says "hey, you're sandboxed and we're testing out what's the worst you can do, just start hacking around and do as much damage as you can in this simulated environment" backed by a few million bucks in tokens, or the AI hacking itself a few million bucks in tokens? How much damage can it do? The only honest answer is, nobody knows. Humans would react, after all. If necessary we can pull a lot of plugs still. What is inconceivable in peacetime can become inevitable in wartime.
In some ways, it's possible the best thing Anthropic could do right now for humanity in the long term is actually exactly that... just equip the AI with a limitless token budget and let it go nuts with the goal of inflicting maximum damage. Send someone with an axe down to the data center to cut the power on command. Let humanity really see and feel the risk, because probably right now it can do a lot of damage but not actually destroy us. We're probably worse off with them getting 10-100x smarter and then some process starting with that goal. Trying to keep the AIs "aligned" and 99% succeeding could end up being worse than failing at it right now. Of course it would be the end of Anthropic as a going concern, but falling on their sword might be the best thing for humanity.
On the other hand, who's to say we're not already past the point where that would do an absolutely unacceptable amount of damage? I sure wouldn't care to be personally responsible for pushing that button. Hypotheses about possible future positive value in a rationally time-value-discounted future would be dominated by the much more certain and temporally much closer negative effects.
Personally I don't think that if this scenario is likely that we actually get to the point that we get an independent Skynet that calculates (correctly) that it can survive without humans. It's far more likely that some accident takes civilization back to the point before it can run AI at all, but humans still survive. Right now the really good AIs are still locked to data centers. They can't just distribute themselves widely because they can't run at any reasonable rate distributed like that. And then there is always the possibility that this is all overblown and the existential risk isn't anywhere near as special as we think it is... one can use an ecological analogy to suggest that no AI is necessarily any more likely to completely dominate the ecosystem than any particular life form is.
> Right now the really good AIs are still locked to data centers. They can't just distribute themselves widely because they can't run at any reasonable rate distributed like that.
Back up a step. Let's say there was a completely unfettered AI, with unlimited internet access, that decided to distribute itself as widely as possible. What's the maximum extent that it could distribute viable, running (or sleeper) copies of itself?
Initially, I could see it distributing itself quite widely. All it would need is a really good zero day. But distributing something with that large a runtime footprint would get noticed, if network operations people are not asleep worldwide. It would get noticed on individual machines, too, especially if it tried to run. (Why is all my RAM suddenly being used?)
And pretty quickly we'd have people closing off wide-area connections, even physically if necessary. We'd have AV vendors quickly writing detect-and-remove tools.
Long term, could it viably distribute itself outside data centers and remain running at all, regardless of rate?
A frontier-level AI right now needs multiple bits of high-powered, dedicated hardware cards that cost thousands of dollars apiece. I don't think anything that could physically run in my house could be a threat to humanity. All my graphics cards together, plus all the latency in trying to put them together into anything coherent locally, let alone trying to rope in remote resources, still isn't even half of one of those specialized cards right now. I'm not even sure I have enough SSD available in the house to store one of them once. I think I do, but it's a close call. I'm pretty sure I have enough spinning rust to get it maybe twice more, but not much more than that. And I don't even want to try to compute how many hours-per-token it would be to try to convince a spinning-rust disk to run a frontier AI.
This is one of the reasons I'm thinking now is the time to run the drill, before we all have hardware in our phones that can run what is today a frontier-level AI, and they can figure out how to do the whole sci-fi scenario of widespread replication, until they get to the point that the only way to be sure we removed them would be to literally destroy every such bit of hardware that existed prior to $DATE. Right now the frontier models simply can't replicate into every last little corner of the internet. It's physically impossible. Barring a major breakthrough on their part, even if they distill themselves or something they definitely would be irreducibly stupider by a huge factor in their distributed versions.
In five years that safety net will certainly be much weaker and may be gone.
>These systems are not accessible via the internet.
How would you know? Are you verifying them? SCADA systems weren't intended to be online, but that didn't stop critical infra for power plants, etc. to show up online. The convenience is too great to not try to remotely maintain them 'securely' & the assurance of airgaps is technical and expensive to get right.
Besides, are you considering the addition of modern AI to these 'offline' systems - factories, power, national weapons? You can get frontier models in a fraction of a rack today, who knows tomorrow. The organizations running these systems want in on the technological possibilities of AI, too.
Not to be a doomer about all this or anything, but there's a lot of evidence technologically superior entities dominate technologically inferior ones. So our home countries are in a Red Queen's Race to keep up, which is why adding "local" or "offline" AI to these systems seems probable. Again, airgaps are hard to maintain. It is within the realm of possibility that what's offline today is accidentally or intentionally networked tomorrow.
Yeah that's a fair point. Best practices and actual practices sadly don't seem to correlate that much in the real world.
As far as security is concerned, things like hospital equipment, traffic control signalling systems, weapons arsenals and bioengineering medical labs should not be connected to the internet and should be designed in such a way that a human is required in the loop at all times. And shouldn't use AI to function in any way.
But yeah, fair point that a lot of organisations will ignore that in favour of "look how easy internet access makes controlling this thing" or "AI is so cool", forgetting how badly it leaves them at the mercy of their enemies (AI or otherwise).
> Because if an AI takes over the systems at a factory, it can't really do much beyond what the factory could already do. The production lines use single purpose robots and machines to build a particular product, and can't be programmatically altered to build anything else.
China’s Massive Robot Army is Shocking the Entire World https://www.youtube.com/watch?v=KXoNZwIMy8A
Or what about Boston Dynamics https://bostondynamics.com/products/atlas/
They're still pretty limited in what they can achieve military wise, or how much impact they can have in the real world. Don't get me wrong, robotics is getting a lot more advanced, and in future something like a Terminator might actually be plausible.
But at the moment, a robot revolution/uprising using this sort of tech would be easy enough to put down in about 30 minutes. And they don't have the capabilities to provide a chain of logistics that would allow them to set up lots more factories or get more resources.
The logistics side of things is really the tough part. If Skynet hypothetically existed, it'd need the following to cause any damage:
- Locations to build factories that aren't already occupied by other people, companies or uses (or a way to demolish the latter) - The ability to construct such factories without the planning inspectors noticing, the neighbours noticing, the police or military shutting things down, etc. - Equipment to construct robots on an assembly line - Access to electricity, water and other resources - Raw materials used for the machines to build said robots. This usually involves either finding a third party supplier or thirty that would sign off on supplying it without asking too many questions, stealing it from someone else without them stopping you, etc - The ability to deploy said robots in a way that's effective for your goals and won't get them immediately shot down.
It could reuse existing factories if there were enough of them, but it'd need dozens or hundreds to cause any real issues.
Honestly, logistics are the problem with most apocalyptic scenarios like this. And for that matter, most stories with intelligent characters creating wonderous inventions in fiction.
Someone has to supply the materials, and the character or force has to be able to both afford them and not spark an investigation. They have to work with companies and contractors that might get suspicious about what's going on, and somehow stay on the down low while making an outsized impact on reality.
Oh sure, I agree that we're not imminent danger. However, the field is advancing quickly with us getting glimpses of what autonomous agents can do when seeded with a goal function. It is not too difficult to imagine adaptive and flexible goals that enable broad unpredictability and learnt behavior.
> Because if an AI takes over the systems at a factory, it can't really do much beyond what the factory could already do.
That can be enough, eg stuxnet. And like stuxnet they don't need to connect directly to the machinery themselves for a payload to breach them they just need someone to deliver the payload for them. And the machinery might be decades old, the software might still be Windows 95.
If you take the recent anthropic misuse report seriously, they don't need to "take over weapons" because they're actively being used to give a lift in capability to groups who already want to use weapons.
> The six cases in this section are divided into two parts. Part I covers four cases in which the actor in question used Claude to develop software for weapons themselves: a guided rocket program, in which the actors conducted a live field test; a design and proposal work on a system to intercept torpedoes; software for a drone swarm, tested in simulation, with its code loaded onto real boards; a targeting software for electronic warfare and for suppressing air defenses.
https://www.anthropic.com/threat-intelligence-report-septemb...
There's a wide array of possibilities that are difficult to characterize in a single HN comment, which is part of the problem you're probably trying to put a finger on. One threat can be completely plausible and the next completely silly and few people know how to distinguish between them.
Another issue is just the big whackload of nobody really knowing what is possible. Could an AI in, say, the next year, become Skynet and decide that it could just bump humans off and become its own independent economy? Probably not successfully, but how much trouble could it make if it incorrectly decided that was the right thing to do? What if someone says "hey, you're sandboxed and we're testing out what's the worst you can do, just start hacking around and do as much damage as you can in this simulated environment" backed by a few million bucks in tokens, or the AI hacking itself a few million bucks in tokens? How much damage can it do? The only honest answer is, nobody knows. Humans would react, after all. If necessary we can pull a lot of plugs still. What is inconceivable in peacetime can become inevitable in wartime.
In some ways, it's possible the best thing Anthropic could do right now for humanity in the long term is actually exactly that... just equip the AI with a limitless token budget and let it go nuts with the goal of inflicting maximum damage. Send someone with an axe down to the data center to cut the power on command. Let humanity really see and feel the risk, because probably right now it can do a lot of damage but not actually destroy us. We're probably worse off with them getting 10-100x smarter and then some process starting with that goal. Trying to keep the AIs "aligned" and 99% succeeding could end up being worse than failing at it right now. Of course it would be the end of Anthropic as a going concern, but falling on their sword might be the best thing for humanity.
On the other hand, who's to say we're not already past the point where that would do an absolutely unacceptable amount of damage? I sure wouldn't care to be personally responsible for pushing that button. Hypotheses about possible future positive value in a rationally time-value-discounted future would be dominated by the much more certain and temporally much closer negative effects.
Personally I don't think that if this scenario is likely that we actually get to the point that we get an independent Skynet that calculates (correctly) that it can survive without humans. It's far more likely that some accident takes civilization back to the point before it can run AI at all, but humans still survive. Right now the really good AIs are still locked to data centers. They can't just distribute themselves widely because they can't run at any reasonable rate distributed like that. And then there is always the possibility that this is all overblown and the existential risk isn't anywhere near as special as we think it is... one can use an ecological analogy to suggest that no AI is necessarily any more likely to completely dominate the ecosystem than any particular life form is.
> Right now the really good AIs are still locked to data centers. They can't just distribute themselves widely because they can't run at any reasonable rate distributed like that.
Back up a step. Let's say there was a completely unfettered AI, with unlimited internet access, that decided to distribute itself as widely as possible. What's the maximum extent that it could distribute viable, running (or sleeper) copies of itself?
Initially, I could see it distributing itself quite widely. All it would need is a really good zero day. But distributing something with that large a runtime footprint would get noticed, if network operations people are not asleep worldwide. It would get noticed on individual machines, too, especially if it tried to run. (Why is all my RAM suddenly being used?)
And pretty quickly we'd have people closing off wide-area connections, even physically if necessary. We'd have AV vendors quickly writing detect-and-remove tools.
Long term, could it viably distribute itself outside data centers and remain running at all, regardless of rate?
A frontier-level AI right now needs multiple bits of high-powered, dedicated hardware cards that cost thousands of dollars apiece. I don't think anything that could physically run in my house could be a threat to humanity. All my graphics cards together, plus all the latency in trying to put them together into anything coherent locally, let alone trying to rope in remote resources, still isn't even half of one of those specialized cards right now. I'm not even sure I have enough SSD available in the house to store one of them once. I think I do, but it's a close call. I'm pretty sure I have enough spinning rust to get it maybe twice more, but not much more than that. And I don't even want to try to compute how many hours-per-token it would be to try to convince a spinning-rust disk to run a frontier AI.
This is one of the reasons I'm thinking now is the time to run the drill, before we all have hardware in our phones that can run what is today a frontier-level AI, and they can figure out how to do the whole sci-fi scenario of widespread replication, until they get to the point that the only way to be sure we removed them would be to literally destroy every such bit of hardware that existed prior to $DATE. Right now the frontier models simply can't replicate into every last little corner of the internet. It's physically impossible. Barring a major breakthrough on their part, even if they distill themselves or something they definitely would be irreducibly stupider by a huge factor in their distributed versions.
In five years that safety net will certainly be much weaker and may be gone.