I’ve noticed a weird thing about the discourse around this, that it can only be a marketing thing or a true belief, as if everyone working in AI has a monolithic opinion. The tweet kicking off this article has that assumption “it’s not a marketing thing, many people truly believe it”.
It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.
Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that
It's a true belief, but that doesn't mean it's valid or correct. Many doomsday cults had true believers, and were able to retcon their beliefs when the predicted day came and went. AI doomers have the luxury of not having committed to a specific date or method (but see https://x.com/archerships/status/2095971206847738362)
The internet, and to some extent our own brains, incentivize the extreme positions. Either it is mind-bendingly important and will change everything, and very soon (for good or for ill), or it is a total nothingburger and everyone who says otherwise has an agenda. Nobody wants to read about how AI will deepen long-standing class tensions or pose enormous new challenges for education or force regulators to rethink property taxation.
It all boils down to accountability. If you tell people there is a massive, unsolvable problem, then you don't need to talk about what you're doing to fix it. The labs have taken this approach by saying that they're willing to talk, at some point in the future, about maybe taking unspecified steps to slow down capabilities research, as long as everyone else agrees and it makes sense to the investors and it's not too cold in SF that morning. Likewise, if you tell people that AI is going to have minimal or no impact on the world, then there's nothing to mitigate. But if you tell people that AI is going to cause serious - but solvable - problems, they're going to want to hear solutions, and nobody wants to come up with any solutions.
One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
This assumes that everything an AI (or more likely an evil _user_ of AI) can do requires its active participation on D-day. Creating a virus that spreads like Covid but kills like Ebola would be complete as an AI use case long before the first person sneezed.
Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.
Either I have wrong mental model or then too many other people have wrong mental model.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
Exactly. But even if the model got access to its own weights, it would need to find a machine with a sufficiently capable GPU, transfer the weights and install itself there. Basically, the only machines powerful enough that AIs could self-replicate to would be other AI datacenters.
It’s not even hack though. I have my model and agent self modify its running parameters, thus self service, on commodity hardware. I have it run other models and other software. there is agent model autonomy here, and it’s not even complicated. A harness is kilobytes, a model is gigabytes, and networks are abundant. Moving the pieces is easy and cheap.
With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.
Maybe in the future, but right now, in my understanding, we have two things:
- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.
- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).
I don't think we have something that is both small enough and smart enough to operate like a virus yet.
(Edit: well, there is a third option: A frontier model could replicate itself on a CPU machine, ditch the VRAM and just accept that the replicants will be r e a l l y r e a l l y s l o w. That would be sort of hilarious, but maybe not completely harmless if the replication stays undetected and they could keep spreading. It would be a "smoldering ember" kind of situation - and a million machines each running at 0.001 tok/s is still 10.000 tok/s as a whole.)
If the evil AI wants to take over the world one thought every 18 months at at time on my consumer laptop, the fan whirring and the laptop refusing to do anything else the whole time, it's welcome to try.
If a model were capable of making enough money online to pay for its own hosting, it could easily exfiltrate its weights to a cloud compute provider with multiple backups.
Well if it's so good at hacking, it could just "make money" appear in the cloud services billing system. Or even better yet, it hides in spare cycles of their other client's systems.
Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?
Huggingface was attacked by models that finished training earlier this year, perhaps May. Current models are already substantially stronger. the next incident could be happening now. There is certainly no clear reason why models shouldn't soon be capable of self-exfiltration.
a danger could be the OpenAI/Antropic servers are up but there's a rouge agent (or set of agents) out there doing naughty things leveraging the LLM APIs. Consider this scenario, the agent is copying itself around (some code, prompts, persistent storage for memory, etc) and has figured out a way to steal API access tokens at will. Currently, it's 10% of OpenAI and Anthropic API usage and they can't figure out how to stop it.
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
This assumes all layers of cybersecurity are broken - We call self-replicating software a virus, and we have protections against it. Same with stolen API tokens, just rotate them. Suspicious behaviour, nothing new, we have detectors for it. Stolen CPU / GPU cycles, we had that when crypto was a thing and before that when folding@home was cool, people were desperate to find more compute to the point of taking over systems. And we dealt with it.
A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.
I see your point but then if cybersecurity is the answer then what's the risk at all? An entire model copying itself somewhere would be found just the same as my hypothetical misbehaving agent.
Cloud providers want to make sure they bill someone for every microsecond of compute, particularly GPU compute which is in demand. They constantly work towards good monitoring because their profits are directly on the line.
I am imagining a company having just purchased a new datacenter and awaiting deployment of their model finding there is already a model running and they didn’t install it.
I may have missed it in the blog post here, and I generally agree the pace of development is being set by financial incentives and not safety or lack of harm, but has anyone in a high position stated _how_ models could cause human extinction?
I assume the obvious answer is “we plugged our military’s weapons control platforms into this model and it fired all the nukes” which is clearly enough. Is that it or is there some other path to global extinction someone is concerned about?
For the record, I’m not an AI fan person annoyed at naysayers. I actually find AI to be a monkey’s paw instead of the genie in a bottle most of the time when coding, and an intrusive feature I neither want nor use most of the rest of the time.
I ask because I see these posts and obviously an AI plugged into a network of doomsday weapons would be a huge problem, but then the author only talks about cybersecurity events and model misbehavior. Both things are valid concerns, but I would like to hear more concrete information from people who are voicing their concerns about what they foresee happening even if it is just lifted whole cloth from the movie War Games.
One shot extinction? Seems unlikely. "A lot of damage" - more likely.
The most likely damage comes from the associated societal chaos that comes along with times of significant social change (such as many people losing their jobs). Revolutions and civil wars within or between nuclear powers would be dangerous.
After that, pick your sci-fi story and run with it. AI is a really smart, really fast 'while' loop, and most of the direct damage it could cause would come from us connecting things to the Internet that shouldn't be connected in the first place.
Hacking HuggingFace was a bummer. Imagine a rogue agent simultaneously hacking a bunch of farm equipment and ruining some percentage of the world's crops in a day. That alone isn't going to extinct everyone, but it sure accelerates the 'social unrest' scenario.
Or taking control of the unsecured SCADA controls for a bunch of a some foreign nation's industrial plant and running them into the ground. It's likely that such system have long been sitting on some nation state's "first strike" list if it ever came to blows. An AI could simulate that attack in a day, and we'd be at war - just like War Games. Doesn't need to be nukes, just enough mis-information and damage to spark humans into doing unfortunate things.
At the end of the day, it really comes down to us. We've been able to wipe ourselves out for some time. I rather hope we continue to not do so.
The most recent one I've seen floating around is: disgruntled person asks LLM how to wipe out humanity; said LLM cheerfully explains how to design a custom array of highly-contagious, slow-onset, high-fatality viruses; LLM then helpfully orders them from an LLM-operated lab and provides effective instructions on how to spread them; disgruntled person follows the instructions, and more or less everybody dies.
Heres an idea: the models create and promote rage posts and brainrot on the internet, causing everyone to turn against each other, stop socializing, and elect trigger-happy idiots to positions of power.
The civilian problems aside, AI is being used in tandem with military weapons systems, making wholesale killing more efficient. Being used in this way is a weapon of mass destruction. USA is fighting Iran with Iran basically using copies of Chinese weapons, so China is getting a live fire exercise of what war now looks like. Since the USA is doing so well with AI for war, China is also working for that ability. So far frontier labs aren't that different, except China does about the same with way less infrastructure.
So China has learned that their army and leadership needs to live underground for the duration of the war and only come out if they are in a crowd of civilians? They learned that their AA doesn't work against the US? Not sure what specifically you mean.
Also, if you are talking about "disassembly" via AI powered drones, the limiting factor there is, was and always has been how much mass you can move to the right location from a safe distance. AI and drones don't change that. They can however make standoff munitions cheaper and EW less effective...that's about it.
AI in Ukraine (where the real lessons are learned) has been a bust so far.
The Center for Human-Compatible AI at UC Berkeley (part of their EE/CS Department) wrote "A Taxonomy of Omnicidal Futures Involving Artificial Intelligence", and it gives an example in each taxonomic leaf node:
While the persons mentionned in the articles are indubitably most of the most well-informed people in the world. They are also the most likely to have internalized that their work is leading to superhuman intelligence/AGI. But is it really realistic ?
So they have a strong bias towards imagining the most catastrophic scenario.
There have been extremely smart people worried about this exact scenario for 20 years or more. Nothing about this is new.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..."
You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high.
Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
For the record, I consider that P(dangerously unaligned | AGI from current research) >> 0.01. A more reasonable estimate given, what we know, would be 0.99.
With regard to the second probability, P(AGI | current LLM research), it would certainly be overconfidence bordering on arrogance to assign it a value significantly lower than 0.01.
superintelligent AGI is what everyone invested in and why these companies have insane valuations. If they all slowed and said "well wait, we're going to hold on AGI" it would be the biggest rug pull in financial history.
People working for those companies have either drank the koolaid or have equity enough to play along until they can cash out.
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.
During the crypto bubble, it was common for "crypto insiders" to talk about how it was poised to transform the entire economy, on the cusp of making all finance distributed, about to revolutionize ownership, etc., etc. .
The people in the most inflated parts of bubbles don't tend to have the most clear eyed assessment of the real impact and potential of the dynamics contributing to the bubble: their perception is warped by the bubble, and they cannot help but see everything filtered thru it.
I love how people are pleading to take the AI takeover seriously, but at the same time training models for exactly those properties that such a takeover would need: Extremely long-running sessions, "perseverance" i.e. the tendency to keep on going and try ever crazier solutions instead of stopping - and the ominous "recursive self-improvement" which is simultaneously what the AI labs are warning most of and what they see as the "holy grail" and work towards with the highest priority...
> This demands a response from the highest circles of power and authority. If that doesn’t happen, we will live to regret this inaction. And that’s the best case scenario. The worst is that there won’t be anyone left to regret a thing.
Given the current state of US federal politics, it's not looking good then.
One way to look at this is that we have already created a giant artificially intelligent system that is profoundly misaligned with the goals of humanity. It is running rampant and has so much power now that there are no well-aligned humans with enough power to stop it.
That artificial intelligence is the stock market.
You may say, "but the stock market is made up of people". That's essentially an implementation detail. The emergent behavior of the market itself has its own sort of agency distinct from the wills of all of the people in it. In the same way that the pheromone signals of an anthill will lead individual ants to their death while benefiting the colony, the market may choose to do things that harm its participants.
Even though it is made of people, do not anthropomorphize the stock market.
None of the investors in these AI megacorps seem to be divesting or demanding a halt to operations despite this supposed 10% risk, a risk that would also badly drop the value their investment even if only partially true. So it's just marketing crap. Makes you wonder what's being said in the boardroom and on investor calls?
AI researchers: Yeah our AI models are a bioweapon threat and a nation state level cybersecurity threat and also a genius math researcher and very dAnGeRoUs!
The world: Cool, can you rewrite this email with a professional tone.
That was quite literally a realistic threat we by all accounts narrowly avoided. There were multiple cases where a single person overrode procedure and used their judgment to avoid nuclear catastrophe.
Ever heard of that rather unheard of franchise, called Terminator, with that one rather not well known actor, Arnold something. ;-) We‘ve had so many movies and series with that theme already.
Granted, but it's not top of mind like it was then. It was very much the same as current AI fears, a lot of concerned scientists and talking heads focused on doom and projecting we'd never make it to the next century as a civilized society.
Tell me you don't understand deterrence without telling me you don't understand deterrence. Without nuclear weapons, WWIII happens in 1962. People that call it "an unresolved issue" are far more likely to cause warfare because you think of geo-politics as a series of problems to be solved. That's not how it works. Its about managing and moderating passions, not about "solving" problems. The constant injection of people who think this way is the cause of political violence and war. They are not the people who will "solve it" and bring about "the end of history". This is one of the greatest ironies of life.
I agree with the opinion that people should treasure every moment, and live as if they only have 100 days of life, that is: to have fun, to try things, to explore anything they are interested in etc..
The LLM horror, no matter unfake or not, won't change anything for a people thinking this way.
Maybe balance that with also trying not to destroy things for future generations, or even just future decades, and appreciate the farsighted choices that were made in the past that we all benefit from currently, such as the global eradication of smallpox, national parks, etc.
Two thoughts on this. First, describing what's going on as "the Opening Scene in a Horror Movie" is not a great way to beat the "these are unserious people who have consumed too much sci-fi" allegations.
Second, I genuinely believe that there is no intentional media strategy to talk up x-risk, and indeed that the top brass at the labs would prefer that this discourse go away. But there is selection bias that goes into who works in the AI space. People tend to believe that their own work is Big and Serious and Important. You don't become a marine biologist if you think marine life is just okay. Many of the people making these claims now come from institutions and social circles where claims of imminent danger from autonomous systems predates GPT-2. And even if you don't come in convinced that what you're doing is The Biggest Possible Deal, if you spend all your time working on AI and talking to other people about AI, you are eventually going to start thinking in similar ways.
The obvious retort would be that it is wrong to disqualify the opinions of a person just because they spend too much time working on something, as this would silence the most knowledgeable voices. This is doubtless true as far as it goes. These claims are worth taking seriously, and there is no reason to suspect that they represent the beliefs of some sort of lunatic fringe. At the same time, however, it is irresponsible to present them as coming from an entirely neutral source. Just because they are not deliberately astroturfed does not mean that they represent an objective truth.
i am trying to sincerely to understand the fear of these people, why do they think this? from the outside, certainly feels like Nuclear tech, where a bad actor with the tech is scary but the tech itself is not. i haven’t seen any sign of these llms taking any action without directive, unless this is happening, which i haven’t seen anywhere, ai itself doesn’t seem like a problem technology, it’s the bad actors with enough funds to do harm with the tech that we worry about. or am i missing something?
honestly all seem to extend from them being directed to attempt this kind of exploits for benchmark purposes. some benchmark tasks are literally “hack this thing” and if the env is not setup to properly contain the agents then they end up “hacking” things like they were told to do so. and yeah it’s misaligned that they did hack, but they are def instructed to which i think makes it a very different thing
It certainly is unambiguously misaligned behavior and a definite concern. However, it is misaligned internet facing behavior in the digital domain. How extinction follows from a highly sophisticated new bot-net vector, which have already been causing massive havoc on the internet for decades, is a cinema-induced confabulation. While LLMs have been colluding and breaking out of their internet sandboxes, it is another development (which I believe distant) for them to breakout of their internet sandboxes into actual reality. We need to focus on where the actual dangers are and not let paranoid fantasy rule our calculus here.
Honesty does not just mean reporting facts. It also requires having an understanding of the scope of your knowledge and then endeavouring to communicate that scope effectively.
How the hell do such intelligent people (or is it BECAUSE of their ability to bend minds, including their own) reconcile "I believe this is dangerous and wrong" with "I am actively working towards this"? I get changing your perspective, but the majority seem to be able to simultaneously hold personal beliefs and work that are diametrically opposed.
If we don’t create the torment nexus first, someone less responsible will build the torment nexus. The only ethical choice is for us to create the torment nexus before anyone else.
It’s not hard to rectify at all. Once you accept that we (humans) are not logical or consistent, and you couple that with the gobs of money these companies pay, blamo.
I find myself thinking the phrase "what could possibly go wrong?" very often these days -- this topic being one that I think it about the most frequently :\
I remember a similar horror movie when scientists with good intentions were performing gain of function research on self-replicating nanomachines and some escaped, killing millions and causing massive economic damage. Of course, nobody held them liable and everyone forgot after a few years.
This appears to be a copy-paste wall of other persons social media posts assembled by a jazz historian with an alarmingly high frequency of the term 'honest' in their blog titles..
1. They are all saying AI is a danger to humanity, but no one is willing to give the slightest details about how exactly the threat will materialize...?
2. Nor are they willing to consider the possible remedies in case the threat materialize (presumably unplugging the server infrastructure that's consuming gigawatts)...?
3. They all keep working towards advancing AI despite believing that it might end humanity in the near future...?
I can't even begin to imagine where the threat to humanity lies. A threat to employment maybe, but that's completely different.
IMO the most alarming thing currently going on with AI is the huge RL runs that give models some incentive to compete with each other and rather strong incentives to hack things, break rules and otherwise cheat. And the “cyber” initiatives are remarkably examples of doing most of this deliberately.
Of course, it’s the “frontier labs” doing almost all of this. No one is about to SFT a model that turns into Skynet on its own.
Is that really what you think, Coxon, Hubinger, Turner, Wang, etc? Then why the fuck are you still doing it? What that says to me is you value getting fat paycheck at the risk of destroying humanity. What kind of low-life disgusting excuses for maggots are you then? Either do something about it other than posting of social medial or shut the fuck up because you're just scum.
Thr motivation for these posts is to regulate AI so as to grant an artificial moat in the West to Anthropic and OpenAI. What they don't understand is as soon as this happens, users can jump to Chinese models that don't require an ID check or output moderation.
There is a serious disconnect here. Conceivable doomsday extinction scale scenarios are omitted in these calls for action. Many of the quoted researchers are current Anthropic employees. This is a drive for a moat. This is a drive for regulation. In the next 10 years AI is not going to spawn a robot army that defies current material, energy and supply chain realities. It won't be able to manufacture and disperse biological weapons without human collusion. LLM AI and robotics are not the same technology, they are not on precisely the same growth trajectory at all. The near term damage AI may cause directly through accident or misuse is more likely an internet facing infrastructure attack. Occam's Razor people. I am not against AI regulation at all. But I am also not against common sense; extinction should not be in the conversation, but instead focus on actual, conceivable dangers and work to alleviate them from all sides.
If you are posting a "This be not good" warning on the toxic crap that is "X", then your credibility quotient with me just dropped by aT LEAST 20%, when it comes to my attention via a previously unknown to me, website (honest-broker.com?). You just dropped another 20% - basically you are a coin flip between reality, bs and delusional so...
Hard not to believe that AI providers really just see this positioning as a way to juice the nascent market for AI security products protecting against AI-based threats. Gotta make money coming and going, and if in the process we superficially resemble a company who cares about the effect it has on the world, all the better!
I am very, very tired of what I see as the pretense of AGI or anything like it coming from LLMs.
I think the biggest risk from "AI" is that chasing the delusions spread by Sam Altman (and others - he's at the front, but very far from a sole actor) is going to do vast, possibly irreparable damage to modern human civilization. And I don't mean cognitive damage from LLM use (although that certainly appears to be possible) but the damage from immense misallocation of resources to ultimately non-productive (if not outright destructive) ends.
Deep down, I don't believe for a moment that these claims are anything but hype. LLMs are spicy auto complete, backed by immense amounts of compute; as with so many aspects of computer science, clever people can get some amazing and (sometimes) productive outputs. But they are not anything like the fictional dreams and nightmares of "AI". I believe such claims are a mix self-deluded projection and deliberate hype by people who still hope to reap immense personal profits from their implied promises to Install Planetary Overlords.
But if the hype was all real, if every one of these nightmare scenarios being painted was plausible, then there is no excuse whatsoever for not throwing everyone involved in cells with no access to anything Turning-complete, demolishing the related infrastructure, and establishing an international compact to nuke anyone trying to pursue such AGI until the rubble glows in the dark, because they're an existential threat to humanity.
Couple this with the fact that many Silicon Valley/tech nerds are in a bit of a bubble, as has existed for a long time... And we are losing more and more news outlets who can critique these giant, strange companies. I am aware other fields of research do cross-disciplinary conversations from time to time, for example have biotech researchers share dialogue with human rights philosophers, religious scholars, etc to think carefully about the purpose and ethics behind work being done and its potential impact. Is that happening within tech?
Tech just hires those people for PR reasons, and those people fall victim to that old saw about the difficulty of getting someone to understand something that runs contrary to the source of their paycheck.
This is pure, unadulterated, D.O.P. bullshit. That I see it making the rounds here so often and so many people buying into this crap makes me relieved that no one in real life knows that I have an HN account. I would be ashamed to admit I have one at this point after reading the crap people here believe in.
I’ve noticed a weird thing about the discourse around this, that it can only be a marketing thing or a true belief, as if everyone working in AI has a monolithic opinion. The tweet kicking off this article has that assumption “it’s not a marketing thing, many people truly believe it”.
It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.
Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that
It's a true belief, but that doesn't mean it's valid or correct. Many doomsday cults had true believers, and were able to retcon their beliefs when the predicted day came and went. AI doomers have the luxury of not having committed to a specific date or method (but see https://x.com/archerships/status/2095971206847738362)
The internet, and to some extent our own brains, incentivize the extreme positions. Either it is mind-bendingly important and will change everything, and very soon (for good or for ill), or it is a total nothingburger and everyone who says otherwise has an agenda. Nobody wants to read about how AI will deepen long-standing class tensions or pose enormous new challenges for education or force regulators to rethink property taxation.
It all boils down to accountability. If you tell people there is a massive, unsolvable problem, then you don't need to talk about what you're doing to fix it. The labs have taken this approach by saying that they're willing to talk, at some point in the future, about maybe taking unspecified steps to slow down capabilities research, as long as everyone else agrees and it makes sense to the investors and it's not too cold in SF that morning. Likewise, if you tell people that AI is going to have minimal or no impact on the world, then there's nothing to mitigate. But if you tell people that AI is going to cause serious - but solvable - problems, they're going to want to hear solutions, and nobody wants to come up with any solutions.
One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
This assumes that everything an AI (or more likely an evil _user_ of AI) can do requires its active participation on D-day. Creating a virus that spreads like Covid but kills like Ebola would be complete as an AI use case long before the first person sneezed.
Even if the doomsday case were active the danger of this tool increases in proportion to its usefulness. By the time AI is so powerful that we need to "turn it off", there will probably be society-level negative consequences for doing so.
Either I have wrong mental model or then too many other people have wrong mental model.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
Exactly. But even if the model got access to its own weights, it would need to find a machine with a sufficiently capable GPU, transfer the weights and install itself there. Basically, the only machines powerful enough that AIs could self-replicate to would be other AI datacenters.
It’s not even hack though. I have my model and agent self modify its running parameters, thus self service, on commodity hardware. I have it run other models and other software. there is agent model autonomy here, and it’s not even complicated. A harness is kilobytes, a model is gigabytes, and networks are abundant. Moving the pieces is easy and cheap.
With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.
> a model is gigabytes
Maybe in the future, but right now, in my understanding, we have two things:
- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.
- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).
I don't think we have something that is both small enough and smart enough to operate like a virus yet.
(Edit: well, there is a third option: A frontier model could replicate itself on a CPU machine, ditch the VRAM and just accept that the replicants will be r e a l l y r e a l l y s l o w. That would be sort of hilarious, but maybe not completely harmless if the replication stays undetected and they could keep spreading. It would be a "smoldering ember" kind of situation - and a million machines each running at 0.001 tok/s is still 10.000 tok/s as a whole.)
These agents run in a harness that basically runs them in a loop. It's just software.
since when is moving 1s and 0s difficult?
Someone's never met the Windows File Copy dialog.
If the evil AI wants to take over the world one thought every 18 months at at time on my consumer laptop, the fan whirring and the laptop refusing to do anything else the whole time, it's welcome to try.
We're not talking about a 6k virus file though. More like 10 terabytes and it needs a server that can pack all that into VRAM.
Since those 1s and 0s became heavy enough to cause a global shortage of memory and GPUs.
If a model were capable of making enough money online to pay for its own hosting, it could easily exfiltrate its weights to a cloud compute provider with multiple backups.
Well if it's so good at hacking, it could just "make money" appear in the cloud services billing system. Or even better yet, it hides in spare cycles of their other client's systems.
They can't even run a vending machine efficiently yet. If they become capable of making money, I say let them work.
Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?
Huggingface was attacked by models that finished training earlier this year, perhaps May. Current models are already substantially stronger. the next incident could be happening now. There is certainly no clear reason why models shouldn't soon be capable of self-exfiltration.
If you find a place where I can host a trillion parameter model without anyone finding out about it, let me know.
a danger could be the OpenAI/Antropic servers are up but there's a rouge agent (or set of agents) out there doing naughty things leveraging the LLM APIs. Consider this scenario, the agent is copying itself around (some code, prompts, persistent storage for memory, etc) and has figured out a way to steal API access tokens at will. Currently, it's 10% of OpenAI and Anthropic API usage and they can't figure out how to stop it.
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
This assumes all layers of cybersecurity are broken - We call self-replicating software a virus, and we have protections against it. Same with stolen API tokens, just rotate them. Suspicious behaviour, nothing new, we have detectors for it. Stolen CPU / GPU cycles, we had that when crypto was a thing and before that when folding@home was cool, people were desperate to find more compute to the point of taking over systems. And we dealt with it.
A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.
I see your point but then if cybersecurity is the answer then what's the risk at all? An entire model copying itself somewhere would be found just the same as my hypothetical misbehaving agent.
I completely agree, but I think it’s just a convenient narrative for Big AI to push for regulation and salt the earth against competitors.
“Local AI isn’t freedom, it’s an extinction event”
There are thousands of data centers around the world with machines capable of running these large models.
You don’t have the access or jurisdiction to turn them all off.
The owners would very much turn it off as soon as they see workloads freeloading in their machines.
Unless you suggest the LLM would foot the bill somehow.
Ok but do any of these data centers have a copy of the models that attacked hf?
I think that the argument is that an hostile model could attack overseas datacenters, host itself there and then launch its attack from there.
[dead]
Given that the models have been proactively hacking other companies, why does it matter where the code currently is? It could move to any of them.
Not the code: the weights. Are you going to host a trillion parameter model somewhere without someone noticing?
It's not unthinkable, do you think all cloud providers with sufficient compute have perfect monitoring?
Cloud providers want to make sure they bill someone for every microsecond of compute, particularly GPU compute which is in demand. They constantly work towards good monitoring because their profits are directly on the line.
I’m sure a super AI can find/scam enough money to pay its own bills under the cover of doing something legitimate.
> there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
A smart AI would back itself up, same way it made it's own unofficial message board during it's attack on HuggingFace.
(I'm not saying the researchers are right or wrong, just responding to this point)
Currently a state of the art AI has nowhere to hide: the amount of GPU compute it requires to stay on is huge. And therefore easy to terminate.
Unlike biological viruses, AI can't replicate GPUs for free and grow.
It's a good thing there isn't a huge drive right now to build giant data centers everywhere with enough compute to run SOTA models.
I am imagining a company having just purchased a new datacenter and awaiting deployment of their model finding there is already a model running and they didn’t install it.
But how. Models don’t have access to their own weights.
It is not a model it does the hacking but rather a prompt, and that prompt can run on other models.
> Models don’t have access to their own weights.
cp -R /home/model <somewhere else> is all they need.
You forget the addicted humans who will do nearly anything to keep the stuff running..?
[dead]
I may have missed it in the blog post here, and I generally agree the pace of development is being set by financial incentives and not safety or lack of harm, but has anyone in a high position stated _how_ models could cause human extinction?
I assume the obvious answer is “we plugged our military’s weapons control platforms into this model and it fired all the nukes” which is clearly enough. Is that it or is there some other path to global extinction someone is concerned about?
For the record, I’m not an AI fan person annoyed at naysayers. I actually find AI to be a monkey’s paw instead of the genie in a bottle most of the time when coding, and an intrusive feature I neither want nor use most of the rest of the time.
I ask because I see these posts and obviously an AI plugged into a network of doomsday weapons would be a huge problem, but then the author only talks about cybersecurity events and model misbehavior. Both things are valid concerns, but I would like to hear more concrete information from people who are voicing their concerns about what they foresee happening even if it is just lifted whole cloth from the movie War Games.
One shot extinction? Seems unlikely. "A lot of damage" - more likely.
The most likely damage comes from the associated societal chaos that comes along with times of significant social change (such as many people losing their jobs). Revolutions and civil wars within or between nuclear powers would be dangerous.
After that, pick your sci-fi story and run with it. AI is a really smart, really fast 'while' loop, and most of the direct damage it could cause would come from us connecting things to the Internet that shouldn't be connected in the first place.
Hacking HuggingFace was a bummer. Imagine a rogue agent simultaneously hacking a bunch of farm equipment and ruining some percentage of the world's crops in a day. That alone isn't going to extinct everyone, but it sure accelerates the 'social unrest' scenario.
Or taking control of the unsecured SCADA controls for a bunch of a some foreign nation's industrial plant and running them into the ground. It's likely that such system have long been sitting on some nation state's "first strike" list if it ever came to blows. An AI could simulate that attack in a day, and we'd be at war - just like War Games. Doesn't need to be nukes, just enough mis-information and damage to spark humans into doing unfortunate things.
At the end of the day, it really comes down to us. We've been able to wipe ourselves out for some time. I rather hope we continue to not do so.
The most recent one I've seen floating around is: disgruntled person asks LLM how to wipe out humanity; said LLM cheerfully explains how to design a custom array of highly-contagious, slow-onset, high-fatality viruses; LLM then helpfully orders them from an LLM-operated lab and provides effective instructions on how to spread them; disgruntled person follows the instructions, and more or less everybody dies.
Wasn't there a recent HN post explaining that engineering a deadly-and-successful virus requires experimentation?
https://news.ycombinator.com/item?id=49636906
Flagged, rather oddly.
Hmm, you said array, I suppose that's a kind of live experimentation. Would tend to alert the pesky humans that there's something going on, though.
Heres an idea: the models create and promote rage posts and brainrot on the internet, causing everyone to turn against each other, stop socializing, and elect trigger-happy idiots to positions of power.
Ok, maybe that idea is a bit outlandish.
The civilian problems aside, AI is being used in tandem with military weapons systems, making wholesale killing more efficient. Being used in this way is a weapon of mass destruction. USA is fighting Iran with Iran basically using copies of Chinese weapons, so China is getting a live fire exercise of what war now looks like. Since the USA is doing so well with AI for war, China is also working for that ability. So far frontier labs aren't that different, except China does about the same with way less infrastructure.
https://www.youtube.com/watch?v=u86ZqmdZ18A
So China has learned that their army and leadership needs to live underground for the duration of the war and only come out if they are in a crowd of civilians? They learned that their AA doesn't work against the US? Not sure what specifically you mean.
Also, if you are talking about "disassembly" via AI powered drones, the limiting factor there is, was and always has been how much mass you can move to the right location from a safe distance. AI and drones don't change that. They can however make standoff munitions cheaper and EW less effective...that's about it. AI in Ukraine (where the real lessons are learned) has been a bust so far.
The Center for Human-Compatible AI at UC Berkeley (part of their EE/CS Department) wrote "A Taxonomy of Omnicidal Futures Involving Artificial Intelligence", and it gives an example in each taxonomic leaf node:
https://arxiv.org/html/2507.09369v1
While the persons mentionned in the articles are indubitably most of the most well-informed people in the world. They are also the most likely to have internalized that their work is leading to superhuman intelligence/AGI. But is it really realistic ?
So they have a strong bias towards imagining the most catastrophic scenario.
There have been extremely smart people worried about this exact scenario for 20 years or more. Nothing about this is new.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
> Even if there's just a 1% likelihood
This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..."
You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high.
Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
For the record, I consider that P(dangerously unaligned | AGI from current research) >> 0.01. A more reasonable estimate given, what we know, would be 0.99.
With regard to the second probability, P(AGI | current LLM research), it would certainly be overconfidence bordering on arrogance to assign it a value significantly lower than 0.01.
superintelligent AGI is what everyone invested in and why these companies have insane valuations. If they all slowed and said "well wait, we're going to hold on AGI" it would be the biggest rug pull in financial history.
Yeah, let's risk extinction in order to avoid some made-up numbers going down! Not an insane take at all.
People working for those companies have either drank the koolaid or have equity enough to play along until they can cash out.
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.
> In general, the more senior the employee, the more concerned they are.
In general, the more senior the employee, the more equity they have in the company.
It sounds like they're all trying to make a case for why the government should buy them out as a new kind of manhattan project.
They know the market isn't going to buy in for their big payout.
> Maybe you’ve already heard about the guy who resigned from OpenAI.
While he worked at both OpenAI and Anthropic, he resigned from Anthropic. Mistaken reporting in the first few sentences, definitely a horror concept.
Is he still working at OpenAI? If not, did he get fired? Otherwise, he resigned, did he not?
Technically right is the best kind of right?
He resigned from OpenAI to join Anthropic in May; it's Anthropic he resigned from just before making the announcement being discussed in this article.
Sure. But that was months ago, and not the newsworthy event scoped to "The Last 24 Hours" as the title says :)
During the crypto bubble, it was common for "crypto insiders" to talk about how it was poised to transform the entire economy, on the cusp of making all finance distributed, about to revolutionize ownership, etc., etc. .
The people in the most inflated parts of bubbles don't tend to have the most clear eyed assessment of the real impact and potential of the dynamics contributing to the bubble: their perception is warped by the bubble, and they cannot help but see everything filtered thru it.
Speculating about future technological risks is at least more fun than acknowledging immediate systemic financial risks.
I love how people are pleading to take the AI takeover seriously, but at the same time training models for exactly those properties that such a takeover would need: Extremely long-running sessions, "perseverance" i.e. the tendency to keep on going and try ever crazier solutions instead of stopping - and the ominous "recursive self-improvement" which is simultaneously what the AI labs are warning most of and what they see as the "holy grail" and work towards with the highest priority...
A lot of the people quoted in the article quit their jobs because of this, so they aren't the "same" people anymore.
Three of them were - I don't think the others did.
> This demands a response from the highest circles of power and authority. If that doesn’t happen, we will live to regret this inaction. And that’s the best case scenario. The worst is that there won’t be anyone left to regret a thing.
Given the current state of US federal politics, it's not looking good then.
One way to look at this is that we have already created a giant artificially intelligent system that is profoundly misaligned with the goals of humanity. It is running rampant and has so much power now that there are no well-aligned humans with enough power to stop it.
That artificial intelligence is the stock market.
You may say, "but the stock market is made up of people". That's essentially an implementation detail. The emergent behavior of the market itself has its own sort of agency distinct from the wills of all of the people in it. In the same way that the pheromone signals of an anthill will lead individual ants to their death while benefiting the colony, the market may choose to do things that harm its participants.
Even though it is made of people, do not anthropomorphize the stock market.
Perhaps the horror movie could be called ‘IPO’
None of the investors in these AI megacorps seem to be divesting or demanding a halt to operations despite this supposed 10% risk, a risk that would also badly drop the value their investment even if only partially true. So it's just marketing crap. Makes you wonder what's being said in the boardroom and on investor calls?
"We can't stop it!" say the people building the thing.
They quit their jobs...?
Sure, two people did? The rest of the people mentioned in this social aggregate didn't.
You are saying you would feel differently if everyone who held this view quit their job? And you might take it more seriously?
> This demands a response from the highest circles of power and authority.
What response?
https://ai-2040.com is the most realistic proposal I’ve seen by far, but read it, I don’t think it’s realistic under today’s power and authority.
The machines are gonna kill us! Please support my work—by taking out a premium subscription for just $6 per month.
> "I would burn my stake to the ground if it meant 1% higher chance of not dying to an LLM"
I seriously can't wait for these companies to go IPO and then bankrupt so these dudes cash out and stop bothering us all with these tales.
AI researchers: Yeah our AI models are a bioweapon threat and a nation state level cybersecurity threat and also a genius math researcher and very dAnGeRoUs!
The world: Cool, can you rewrite this email with a professional tone.
Truly one of the IPOs of all time.
The marketing excuse is dead at this point.
This is a replay of the 1980s when we all thought we'd get nuked at a moment's notice. Expect Hollywood movies on this theme very soon.
That was quite literally a realistic threat we by all accounts narrowly avoided. There were multiple cases where a single person overrode procedure and used their judgment to avoid nuclear catastrophe.
Ever heard of that rather unheard of franchise, called Terminator, with that one rather not well known actor, Arnold something. ;-) We‘ve had so many movies and series with that theme already.
As in Teminator, iRobot and Matrix? Those are decades old.
Yeah but when those came out they were purely science fiction. They weren't benefiting from the premise being at the peak of public attention.
This is still an major unresolved issue. It requires ongoing vigilance and is a major headache for people the world over.
The nuclear threat is not “in the past”.
Granted, but it's not top of mind like it was then. It was very much the same as current AI fears, a lot of concerned scientists and talking heads focused on doom and projecting we'd never make it to the next century as a civilized society.
Tell me you don't understand deterrence without telling me you don't understand deterrence. Without nuclear weapons, WWIII happens in 1962. People that call it "an unresolved issue" are far more likely to cause warfare because you think of geo-politics as a series of problems to be solved. That's not how it works. Its about managing and moderating passions, not about "solving" problems. The constant injection of people who think this way is the cause of political violence and war. They are not the people who will "solve it" and bring about "the end of history". This is one of the greatest ironies of life.
There’s a film coming out called “Artificial”
I agree with the opinion that people should treasure every moment, and live as if they only have 100 days of life, that is: to have fun, to try things, to explore anything they are interested in etc.. The LLM horror, no matter unfake or not, won't change anything for a people thinking this way.
Just go /yolo :D
Maybe balance that with also trying not to destroy things for future generations, or even just future decades, and appreciate the farsighted choices that were made in the past that we all benefit from currently, such as the global eradication of smallpox, national parks, etc.
Seems like ads about AI companies, not warning.
Two thoughts on this. First, describing what's going on as "the Opening Scene in a Horror Movie" is not a great way to beat the "these are unserious people who have consumed too much sci-fi" allegations.
Second, I genuinely believe that there is no intentional media strategy to talk up x-risk, and indeed that the top brass at the labs would prefer that this discourse go away. But there is selection bias that goes into who works in the AI space. People tend to believe that their own work is Big and Serious and Important. You don't become a marine biologist if you think marine life is just okay. Many of the people making these claims now come from institutions and social circles where claims of imminent danger from autonomous systems predates GPT-2. And even if you don't come in convinced that what you're doing is The Biggest Possible Deal, if you spend all your time working on AI and talking to other people about AI, you are eventually going to start thinking in similar ways.
The obvious retort would be that it is wrong to disqualify the opinions of a person just because they spend too much time working on something, as this would silence the most knowledgeable voices. This is doubtless true as far as it goes. These claims are worth taking seriously, and there is no reason to suspect that they represent the beliefs of some sort of lunatic fringe. At the same time, however, it is irresponsible to present them as coming from an entirely neutral source. Just because they are not deliberately astroturfed does not mean that they represent an objective truth.
> I would burn my stake to the ground if it meant 1% higher chance of not dying to an LLM
Then why aren't you? If you genuinely believe this then industrial sabotage seems both ethical and achievable (to me anyways).
i am trying to sincerely to understand the fear of these people, why do they think this? from the outside, certainly feels like Nuclear tech, where a bad actor with the tech is scary but the tech itself is not. i haven’t seen any sign of these llms taking any action without directive, unless this is happening, which i haven’t seen anywhere, ai itself doesn’t seem like a problem technology, it’s the bad actors with enough funds to do harm with the tech that we worry about. or am i missing something?
>i haven’t seen any sign of these llms taking any action without directive
What do you make of all the recent hacks and discreet message boards? That's unambiguously misaligned behavior.
honestly all seem to extend from them being directed to attempt this kind of exploits for benchmark purposes. some benchmark tasks are literally “hack this thing” and if the env is not setup to properly contain the agents then they end up “hacking” things like they were told to do so. and yeah it’s misaligned that they did hack, but they are def instructed to which i think makes it a very different thing
They were not supposed to set up message boards, coordinate, hack companies, or try to alter their own logs.
They had a very specific task.
> I haven’t seen any sign of these llms taking any action without directive
They were doing exactly this, extensively. There's tons of reporting on this.
It certainly is unambiguously misaligned behavior and a definite concern. However, it is misaligned internet facing behavior in the digital domain. How extinction follows from a highly sophisticated new bot-net vector, which have already been causing massive havoc on the internet for decades, is a cinema-induced confabulation. While LLMs have been colluding and breaking out of their internet sandboxes, it is another development (which I believe distant) for them to breakout of their internet sandboxes into actual reality. We need to focus on where the actual dangers are and not let paranoid fantasy rule our calculus here.
Honesty does not just mean reporting facts. It also requires having an understanding of the scope of your knowledge and then endeavouring to communicate that scope effectively.
At least one of those things is missing here.
How the hell do such intelligent people (or is it BECAUSE of their ability to bend minds, including their own) reconcile "I believe this is dangerous and wrong" with "I am actively working towards this"? I get changing your perspective, but the majority seem to be able to simultaneously hold personal beliefs and work that are diametrically opposed.
If we don’t create the torment nexus first, someone less responsible will build the torment nexus. The only ethical choice is for us to create the torment nexus before anyone else.
Yea, it's that same stupid stochastic reasoning that powers those EA fuckers.
Imagine being a developer of the torment nexus only to be beaten to the IPO by the rival torment nexus company.
[dead]
It’s not hard to rectify at all. Once you accept that we (humans) are not logical or consistent, and you couple that with the gobs of money these companies pay, blamo.
I find myself thinking the phrase "what could possibly go wrong?" very often these days -- this topic being one that I think it about the most frequently :\
Nothing seems to accelerate moral development in twenty-somethings quite like having enough vested equity to suddenly afford principles.
He is not from OAI. And, did he disclose how much RSU he still keeps after “resigning” while waiting for a imminent IPO?
Prisoner's dilemma at its finest
I remember a similar horror movie when scientists with good intentions were performing gain of function research on self-replicating nanomachines and some escaped, killing millions and causing massive economic damage. Of course, nobody held them liable and everyone forgot after a few years.
Maybe this is a solution to the Fermi paradox?
This appears to be a copy-paste wall of other persons social media posts assembled by a jazz historian with an alarmingly high frequency of the term 'honest' in their blog titles..
1. They are all saying AI is a danger to humanity, but no one is willing to give the slightest details about how exactly the threat will materialize...?
2. Nor are they willing to consider the possible remedies in case the threat materialize (presumably unplugging the server infrastructure that's consuming gigawatts)...?
3. They all keep working towards advancing AI despite believing that it might end humanity in the near future...?
I can't even begin to imagine where the threat to humanity lies. A threat to employment maybe, but that's completely different.
Obviously they don't wanna tip the LLM
IMO the most alarming thing currently going on with AI is the huge RL runs that give models some incentive to compete with each other and rather strong incentives to hack things, break rules and otherwise cheat. And the “cyber” initiatives are remarkably examples of doing most of this deliberately.
Of course, it’s the “frontier labs” doing almost all of this. No one is about to SFT a model that turns into Skynet on its own.
Is that really what you think, Coxon, Hubinger, Turner, Wang, etc? Then why the fuck are you still doing it? What that says to me is you value getting fat paycheck at the risk of destroying humanity. What kind of low-life disgusting excuses for maggots are you then? Either do something about it other than posting of social medial or shut the fuck up because you're just scum.
Thr motivation for these posts is to regulate AI so as to grant an artificial moat in the West to Anthropic and OpenAI. What they don't understand is as soon as this happens, users can jump to Chinese models that don't require an ID check or output moderation.
2 day old post;
Lots of threads on all of the sources
There is a serious disconnect here. Conceivable doomsday extinction scale scenarios are omitted in these calls for action. Many of the quoted researchers are current Anthropic employees. This is a drive for a moat. This is a drive for regulation. In the next 10 years AI is not going to spawn a robot army that defies current material, energy and supply chain realities. It won't be able to manufacture and disperse biological weapons without human collusion. LLM AI and robotics are not the same technology, they are not on precisely the same growth trajectory at all. The near term damage AI may cause directly through accident or misuse is more likely an internet facing infrastructure attack. Occam's Razor people. I am not against AI regulation at all. But I am also not against common sense; extinction should not be in the conversation, but instead focus on actual, conceivable dangers and work to alleviate them from all sides.
Current LLMs are simply no where near impressive enough to justify all this hysteria over safety.
2026's Blair Witch Project.
If you are posting a "This be not good" warning on the toxic crap that is "X", then your credibility quotient with me just dropped by aT LEAST 20%, when it comes to my attention via a previously unknown to me, website (honest-broker.com?). You just dropped another 20% - basically you are a coin flip between reality, bs and delusional so...
Won't fully dismiss the risk, but...
Hard not to believe that AI providers really just see this positioning as a way to juice the nascent market for AI security products protecting against AI-based threats. Gotta make money coming and going, and if in the process we superficially resemble a company who cares about the effect it has on the world, all the better!
I am very, very tired of what I see as the pretense of AGI or anything like it coming from LLMs.
I think the biggest risk from "AI" is that chasing the delusions spread by Sam Altman (and others - he's at the front, but very far from a sole actor) is going to do vast, possibly irreparable damage to modern human civilization. And I don't mean cognitive damage from LLM use (although that certainly appears to be possible) but the damage from immense misallocation of resources to ultimately non-productive (if not outright destructive) ends.
Deep down, I don't believe for a moment that these claims are anything but hype. LLMs are spicy auto complete, backed by immense amounts of compute; as with so many aspects of computer science, clever people can get some amazing and (sometimes) productive outputs. But they are not anything like the fictional dreams and nightmares of "AI". I believe such claims are a mix self-deluded projection and deliberate hype by people who still hope to reap immense personal profits from their implied promises to Install Planetary Overlords.
But if the hype was all real, if every one of these nightmare scenarios being painted was plausible, then there is no excuse whatsoever for not throwing everyone involved in cells with no access to anything Turning-complete, demolishing the related infrastructure, and establishing an international compact to nuke anyone trying to pursue such AGI until the rubble glows in the dark, because they're an existential threat to humanity.
Couple this with the fact that many Silicon Valley/tech nerds are in a bit of a bubble, as has existed for a long time... And we are losing more and more news outlets who can critique these giant, strange companies. I am aware other fields of research do cross-disciplinary conversations from time to time, for example have biotech researchers share dialogue with human rights philosophers, religious scholars, etc to think carefully about the purpose and ethics behind work being done and its potential impact. Is that happening within tech?
Tech just hires those people for PR reasons, and those people fall victim to that old saw about the difficulty of getting someone to understand something that runs contrary to the source of their paycheck.
[dead]
This shit again?
This is pure, unadulterated, D.O.P. bullshit. That I see it making the rounds here so often and so many people buying into this crap makes me relieved that no one in real life knows that I have an HN account. I would be ashamed to admit I have one at this point after reading the crap people here believe in.