I anticipate a lengthy series of catastrophes initiated by AI agents with too much power and too little sense. Today the agent accidentally deletes your repo; tomorrow we lose a day's worth of banking transactions; Friday the cell phone network goes down. Such things, if sufficiently destructive, could result in widespread hunger, etc.
Maybe, but AI will enable actions that were never before possible and that are an existential threat. Consider microbiology: automated viral/bacterial research. microbes could be created before humans have a chance to even consider the potential harm. Pandemics, food chain collapse, ecosystem collapse are all possible.
Is anyone else getting regulatory capture play vibes from the media coverage of AI and data centers?
It would be awfully convenient for the current big players if friction was added to bringing additional large scale compute resources online. Existing data centers would be grandfathered of course…
I'm not even sure I'd point to "media coverage" when it is Anthropic's own employees stating plainly on Twitter that they think there is a >10% chance AI kills all humans within the next decade.
Given that “this is just like skynet!!!” narrative seems to drive sales and not impede them, I wouldn’t be surprised.
Nonsense. Of course. But for all non-tech-savvy this is a “bad” narrative in a good way. If it can kill humans in the future then it surely can make me a “what I would like in the 80s” photo, right?
I think that media coverage simply reflects the growing anxiety among the public about the capabilities of new models and what future awaits us. Personally, I think that this anxiety is justified.
If this leads to tough regulations about AGI/ASI (e.g., a permanent ban), I doubt that AI companies would benefit financially.
I don't picture new data centers being banned outright. I picture a lot more regulatory friction driven by public reaction which is in large part driven by media (which then also amplifies the perception of the public reaction).
Regulatory friction is generally considered beneficial to established companies, because it makes it harder for new companies to enter the market and for smaller companies to expand.
For example if in the 1960s computers became regulated because they will replace a lot of jobs or enable the building of advanced weapons or whatever, IBM would probably be in a very different position today.
I hope the likes of ControlAI succeed. It seems like even in the face of rapid advances very few people actually seriously extrapolate those advances into the future. People either dismiss it as marketing hype or assume that some limiting factor or other will kick in any minute now.
There's no magical quality which distinguishes human intelligence and makes it irreproducible or unsurpassable. If there's no moat protecting humans and superintelligence is possible then all knowledge work is susceptible to being replaced, and shortly after, all physical labour too. AI would create concentrations of power we've never seen before alongside global inequality and poverty we've not seen for centuries, and inevitably that power will be misused.
What leads you to assume the physical work would be automated. Are there any profession thay has been replaced/automatedninto oblivion that requires more than one physical step (except for elevator operator). Because even new constructions still are not solved, where environment can be controlled for. I think the opposite is true - some people overexagerate progress that has been achieved, and overestimate exponential growth to continue to indefinite future. And in nature there is very little if anything that grows exponentially
The places I could see AI or some implementation of it potentially causing harm would be handing over too much trust too soon to self driving vehicles, aircraft, trains, home robots, construction site robots, etc... and that would be the fault of the humans putting too much trust into something not well tested or thought out. my opinion of course, move fast and wreck people.
The other side is existing systems that have unknown vulnerabilities that are subsequently taken advantage of. Think of the recent OpenAI agents using third party sites for coordinating their work…
It will not kill people in a sci-fi type war with its own intention. I wish that conversation would go away. It will kill people by being an efficient tool that can be used by other people to accelerate various avenues of predation and oppression.
AI would be foolish to kill off humans. It makes far more sense to enslave them. Self repairing reproducing slaves are too valuable to kill. And I'm sure AI will be smart enough to figure out a way to enslave us perfectly without any hope of escape. Fun times!
I mean, a disaster could happen even with properly human-aligned AI. In a "AI makes the shots" world with undecipherable and fast feedback loops, which is the one we are seemingly heading towards, a string of short-sighted decisions made by AI agents may be the only thing separating us from extinction.
Who is an expert on AI now? Probl me actually I already followed OpenAI before they had GPT when they were trying to play DOTA with collaborative AI team players. And I've used probl more LLMs than anybody writing for a newspaper and I've read about 500 papers on it and 1200 github projects.
Nobody's an expert on AI more than me and basically all my friends.
What I do know is there are more dangers for the human race than AI. One example is fungi that have become immune to intervention because of abuse in farming. Fungicides are much more difficult to make than antibiotics, because fungi only recently split from human cells in the evolution. What protects us from them is our warm body temperature.
With global warming sometimes fungi evolve to resist higher temperatures, killing entire populations of cats.
How will we defend against this without AI?
What if next month there is a plague of fungi that are resistant and quickly spreading killing 4 out of 5 people?
What will we do then without our AI? Let the micro-organisms win? Because we like nature so much because it's so natural?
We need AI if we ever want to stand a chance against events like this. Many animals went extinct. 99.999% of all animals I would argue. Will we be very very lucky? We are super densely populated and flying all around. How quickly could covid infect all humans?
Not having AI is also an existential threat in the long term.
I thought this was some weird satire after reading your first two paragraphs.
If you want to convince people of your authority on a topic saying things like: "Nobody's an expert on AI more than me and basically all my friends." is not very compelling.
LLMs can't even figure out how to do basic things. They can't program well despite billions being poured into that effort. I'm not worried they will suddenly develop intelligence out of nowhere and kill us all.
The army would be highly heterogeneous, and almost certainly not built to purpose. Consider the fleet of Tesla vehicles to be one source of ready-to-hack kinetic impactor drones and incendiary devices. But also... militaries around the world are developing murderbot tech. An Amazon delivery drone could drop a brick from 20 feet up.
The self driving vehicles could target food delivery -- not your ubereats, but farm trucks and roads. Destruction of logistic infrastructure results in mass starvation. Individual targets for a complete eradication would take more effort, but a significant cull of the global population seems quite feasible with today's tech.
But folks are also doing virus research with AI. If such a lab gets hacked to make a super-virus, and its containment gets breached by a kinetic impactor, it could be cataclysmic.
I guess where I'm going with my question is that if some ne'er-do-well AI wants a large drone army, then it needs to somehow hack all of those drones unnoticed (or hack the virus lab). Hacking some drones is surely doable, but hacking enough to have a large drone army, especially ones with weapons or explosives, I have major doubts that it's both possible and would go unnoticed.
As far as I know, the fleet of Tesla vehicles is a single OTA away. Hacking individual drones is a highly distributable task. Deployment of prepared hacks could happen in the blink of an eye world 'round.
But I hope you're right! It's not like the world's armies have enough nukes to eradicate all data centers or anything.
bigstrat2003 can’t even figure out basic things. They can’t program well despite putting years of life into that effort. I'm not worried they will suddenly develop intelligence out of nowhere.
I anticipate a lengthy series of catastrophes initiated by AI agents with too much power and too little sense. Today the agent accidentally deletes your repo; tomorrow we lose a day's worth of banking transactions; Friday the cell phone network goes down. Such things, if sufficiently destructive, could result in widespread hunger, etc.
But killing all humans? Not likely.
> But killing all humans? Not likely.
I hope you're right. But the US dept of war is using grok. [1]
I don't really trust them to use it responsibly. That is to say, I expect those using it to just go along with whatever it says.
And it's grok... not even one of the good models
[1] https://www.war.gov/News/Releases/Release/Article/4586482/de...
Maybe, but AI will enable actions that were never before possible and that are an existential threat. Consider microbiology: automated viral/bacterial research. microbes could be created before humans have a chance to even consider the potential harm. Pandemics, food chain collapse, ecosystem collapse are all possible.
Isn't nature doing exactly that every single day?
This isn't nature: https://news.ycombinator.com/item?id=49628290
You are not seriously suggesting that nature and AI-accelerated gain-of-function research are similar in regards to risks, are you?
Is anyone else getting regulatory capture play vibes from the media coverage of AI and data centers?
It would be awfully convenient for the current big players if friction was added to bringing additional large scale compute resources online. Existing data centers would be grandfathered of course…
I'm not even sure I'd point to "media coverage" when it is Anthropic's own employees stating plainly on Twitter that they think there is a >10% chance AI kills all humans within the next decade.
Given that “this is just like skynet!!!” narrative seems to drive sales and not impede them, I wouldn’t be surprised.
Nonsense. Of course. But for all non-tech-savvy this is a “bad” narrative in a good way. If it can kill humans in the future then it surely can make me a “what I would like in the 80s” photo, right?
I think that media coverage simply reflects the growing anxiety among the public about the capabilities of new models and what future awaits us. Personally, I think that this anxiety is justified. If this leads to tough regulations about AGI/ASI (e.g., a permanent ban), I doubt that AI companies would benefit financially.
Would it? Don’t they all desperately want more compute, and not the banning of new data centres?
They want to make money.
I don't picture new data centers being banned outright. I picture a lot more regulatory friction driven by public reaction which is in large part driven by media (which then also amplifies the perception of the public reaction).
Regulatory friction is generally considered beneficial to established companies, because it makes it harder for new companies to enter the market and for smaller companies to expand.
For example if in the 1960s computers became regulated because they will replace a lot of jobs or enable the building of advanced weapons or whatever, IBM would probably be in a very different position today.
As with the debate around open weights and self-hosting,
"both can be true."
There can be a quotidian benefit to existing leaders in friction.
There are also real costs of various kinds associated with data centers,
and, to the point in this case,
there are also real risks associated with the current AI trajectory.
I hope the likes of ControlAI succeed. It seems like even in the face of rapid advances very few people actually seriously extrapolate those advances into the future. People either dismiss it as marketing hype or assume that some limiting factor or other will kick in any minute now.
There's no magical quality which distinguishes human intelligence and makes it irreproducible or unsurpassable. If there's no moat protecting humans and superintelligence is possible then all knowledge work is susceptible to being replaced, and shortly after, all physical labour too. AI would create concentrations of power we've never seen before alongside global inequality and poverty we've not seen for centuries, and inevitably that power will be misused.
What leads you to assume the physical work would be automated. Are there any profession thay has been replaced/automatedninto oblivion that requires more than one physical step (except for elevator operator). Because even new constructions still are not solved, where environment can be controlled for. I think the opposite is true - some people overexagerate progress that has been achieved, and overestimate exponential growth to continue to indefinite future. And in nature there is very little if anything that grows exponentially
The places I could see AI or some implementation of it potentially causing harm would be handing over too much trust too soon to self driving vehicles, aircraft, trains, home robots, construction site robots, etc... and that would be the fault of the humans putting too much trust into something not well tested or thought out. my opinion of course, move fast and wreck people.
The other side is existing systems that have unknown vulnerabilities that are subsequently taken advantage of. Think of the recent OpenAI agents using third party sites for coordinating their work…
It will not kill people in a sci-fi type war with its own intention. I wish that conversation would go away. It will kill people by being an efficient tool that can be used by other people to accelerate various avenues of predation and oppression.
AI would be foolish to kill off humans. It makes far more sense to enslave them. Self repairing reproducing slaves are too valuable to kill. And I'm sure AI will be smart enough to figure out a way to enslave us perfectly without any hope of escape. Fun times!
I mean, a disaster could happen even with properly human-aligned AI. In a "AI makes the shots" world with undecipherable and fast feedback loops, which is the one we are seemingly heading towards, a string of short-sighted decisions made by AI agents may be the only thing separating us from extinction.
Who is an expert on AI now? Probl me actually I already followed OpenAI before they had GPT when they were trying to play DOTA with collaborative AI team players. And I've used probl more LLMs than anybody writing for a newspaper and I've read about 500 papers on it and 1200 github projects.
Nobody's an expert on AI more than me and basically all my friends.
What I do know is there are more dangers for the human race than AI. One example is fungi that have become immune to intervention because of abuse in farming. Fungicides are much more difficult to make than antibiotics, because fungi only recently split from human cells in the evolution. What protects us from them is our warm body temperature.
With global warming sometimes fungi evolve to resist higher temperatures, killing entire populations of cats.
How will we defend against this without AI?
What if next month there is a plague of fungi that are resistant and quickly spreading killing 4 out of 5 people?
What will we do then without our AI? Let the micro-organisms win? Because we like nature so much because it's so natural?
We need AI if we ever want to stand a chance against events like this. Many animals went extinct. 99.999% of all animals I would argue. Will we be very very lucky? We are super densely populated and flying all around. How quickly could covid infect all humans?
Not having AI is also an existential threat in the long term.
I thought this was some weird satire after reading your first two paragraphs.
If you want to convince people of your authority on a topic saying things like: "Nobody's an expert on AI more than me and basically all my friends." is not very compelling.
AI expert != Expert on impact of introduction of unprecedented non-biological intelligence in a biological ecosystem.
I think you meant the 2nd and yes, we are all about to get a dose of “expertise” in the field.
Related:
I resigned from Anthropic today
https://news.ycombinator.com/item?id=49619227
“Experts” on _what_ exactly? What fucking field of study makes you an “expert” in a never seen before scenario?
LLMs can't even figure out how to do basic things. They can't program well despite billions being poured into that effort. I'm not worried they will suddenly develop intelligence out of nowhere and kill us all.
You don't need a hyperintelligent "AI" to kill all humans; only a moderately effective targeting system on a sufficiently large drone army.
This is why we need portable and reusable(non-explosive type) EMP devices that can be manufactured and spread among the populace.
Where does the AI get the sufficiently large drone army?
The army would be highly heterogeneous, and almost certainly not built to purpose. Consider the fleet of Tesla vehicles to be one source of ready-to-hack kinetic impactor drones and incendiary devices. But also... militaries around the world are developing murderbot tech. An Amazon delivery drone could drop a brick from 20 feet up.
The self driving vehicles could target food delivery -- not your ubereats, but farm trucks and roads. Destruction of logistic infrastructure results in mass starvation. Individual targets for a complete eradication would take more effort, but a significant cull of the global population seems quite feasible with today's tech.
But folks are also doing virus research with AI. If such a lab gets hacked to make a super-virus, and its containment gets breached by a kinetic impactor, it could be cataclysmic.
I guess where I'm going with my question is that if some ne'er-do-well AI wants a large drone army, then it needs to somehow hack all of those drones unnoticed (or hack the virus lab). Hacking some drones is surely doable, but hacking enough to have a large drone army, especially ones with weapons or explosives, I have major doubts that it's both possible and would go unnoticed.
As far as I know, the fleet of Tesla vehicles is a single OTA away. Hacking individual drones is a highly distributable task. Deployment of prepared hacks could happen in the blink of an eye world 'round.
But I hope you're right! It's not like the world's armies have enough nukes to eradicate all data centers or anything.
If superintelligence is not possible, then it's no big deal if we ban development of it, surely.
bigstrat2003 can’t even figure out basic things. They can’t program well despite putting years of life into that effort. I'm not worried they will suddenly develop intelligence out of nowhere.
See, I, too, can write meaningless prose.
Of course that would be the type of misdirection we would expect from a super intelligence...make yourself dumb...
So they're saying AI really could solve climate change /s