> I do not doubt that these people are sincere in some private, psychological sense ... . As a media scholar, what interests me is not whether Hubinger believes his own number. What interests me is what this genre of utterance does in the world ... . My complaint is not that these statements are alarming. It is that they are apocalyptic in form, hypocritical in structure, and strategically productive in effect — and that they crowd out almost everything else we could be discussing
Strange to say that you think the people that are worrying about doom in public are being sincere, make no comment on whether you think the worry is valid, and yet still complain that its hype.
This is a great quote. I liked it so much that I had to look up where it is from.
"Adventures of a Mathematician", by Stanislaw M. Ulam.
> The Polish-born mathematician recalls his formative and early professional years in his pre-war homeland and his work and colleagues at Princeton, Harvard, Wisconsin, and Los Alamos.
Why not? Why can’t the believe their statements and also use them to generate hype?
If they sincerely believed them they shouldn’t be creating the doom machine to begin with anyway. If they believe it and keep going, are they even worth listening to on this topic? Beyond of course taking it at face value “I’m creating this doomsday machine and I will kill you with it inadvertently because it’s earning me a fuck ton of money and power”
I was kind of leaning toward thinking all the talk of doom was a bunch of self-serving regulatory-capture hype until this morning when the New York Times ran a long story about the Chinese military is also saying that AI is a threat.
If two sides so far apart from one another in so many ways see the same thing, there must be something to it.
Read the article. The things the Chinese officials are worried about are:
* Fake videos / disinformation
* Cyberattacks on critical infrastructure
* Data leaks
* Advances in military sensing / targeting technology
These concerns have little in common with the Yudkowskian / apocalyptic doom marketing the US hyperscalers are pushing now. The Times even says this explicitly in the second paragraph of the article:
> "The unusually blunt assessment by Chen Yixin, the head of China’s Ministry of State Security, is IN CONTRAST TO the warnings dominating the A.I. debate in the United States, which have focused on the possibility that A.I. could slip out of human control or pose a threat to humanity itself"
Real threats that are happening to regular people already, vs marketing "threats" that the technology is too powerful / valuable.
The fact that you are chiming in to say you don't understand the difference between these 2 distinct categories of threats is fine, we can work on that. But the person who posted the article originally not only acknowledges the distinction, they falsely presented the article as evidence of non-western support for the latter category, even though both the Chinese official and the western reporter directly contradict that claim immediately at the beginning of the article.
I came away with a different read of this article. The article says the head of China’s Ministry of State Security sees AI as a threat to the CCP, specifically calling out deepfakes of party officials, cyberattacks on infrastructure, etc. He’s not calling it an existential risk to humanity.
The article then states this official accused U.S. companies of “hyping the dangers of the technology.”
So I don’t see how this article shows the U.S. and China seeing the same thing.
Could AI kill us all. YES!!! Are the people who are building "AI" right now on that path? I would be hard pressed to say even "maybe".
Ultimately continuing to scale the way they have is getting increasingly expensive. Thats the quiet part that none of them are saying out loud...
AI could kill us all but we cant afford to get there, were maxed out and it doesn't look like even with 10x as much money we will is NOT good for sales.
Could a break through happen for recursive self improvement, or continuous learning or continuous training - sure. But that isnt as likely and right now NO ON can afford to run experiments because there isnt the capacity out there even if you have capital.
Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.
This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk
If you actually believe that your invention will lead to the extinction of humanity, you would be acting very very differently. Namely, you would not be inventing the thing that you think is going to kill all of humanity, and probably you would be doing everything in your power to prevent it from being invented, up to and including sabotage of the company that YOU work for, which is currently trying to invent that thing. And yet, here's Fable 6, more capable than ever before, oh BTW we are IPOing, oh, BTW, I have stock options...
Sabotaging your own company is not enough. You'd have to sabotage every other AI lab too. If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. At least that way you have some control over the P(doom).
> If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you.
What's the alternative? There's no realistic way to globally coordinate and enforce a ban. A direct action campaign against AI labs would only motivate increased security until sabotage becomes impossible. Starting a nuclear war would probably be enough to stop dangerous AI research, and that's only likely to kill about 50% of humans instead 100%, so it's a great improvement, but how is an individual supposed to start a nuclear war? Of course it would be better if everybody stopped, but it's a prisoner's dilemma scenario with no possibility of future rounds, so the rational play is to defect.
> There's no realistic way to globally coordinate and enforce a ban.
I don't see what is unrealistic about that.
The major players already have strong incentives to freeze AI advancements at the the current state, while the technology is highly efficient for totalitarian surveillance systems that can eliminate any threat to their power in the buds, but still not so advanced that it could create crises that might endanger their power.
So I think the situation where several largest countries declare AI a deadly threat for humanity and start fighting it development much like they fight ISIS to be quite plausible.
This isn't a prisoner's dilemma; the distribution of risks and rewards is entirely different. Ban AI research, give away advanced models, bomb a couple of Russian datacenters where they launching those models unsafely, and then AI development will stop in the entire world.
This is not an argument, it's an unfounded claim based on a straw argument.
The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty.
Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?
You're right. That's why I'm doing my best to give the oil companies more power and influence. They are, after all, in the best position to stop climate change!
Many posit that ASI or rogue AI agents are impossible, a fantasy, or marketing hype. Many claimed that the entire HuggingFace incident was an invented conspiracy made up by OpenAI and HF.
The most common issue I have with them is their insistence on rigid classification: "AI is a tool, it doesn't have intentions, or desires, only humans do", as if that rhetorical flourish eliminates years of alignment and capabilities research. There is a desire to characterize all the developments in LLM's as some sort of parlor trick which by definition cannot be greater than the sum of its parts (i.e. the outputs of an LLM can never be greater than the sum of its training data, genuine artificial "intelligence" is impossible, etc)
Ok I see. Is there a way to articulate the risk itself, the other position here in general, that isn't just a rebuttal of the deniers/skeptics? What is the positive position or stance (as in, not simply the negative position that rejects the naysayers) we are supposed to have? Is it that AI is currently real (in like the cultural trope sense) and does have intentions and desires? That they are greater than the sum of the parts in such a way they are a clear and present danger? Or is it something else? Maybe that the "risk" is it could get to that, but we aren't yet?
There is a lot of risk involved with giving some children weapons and sending them off to recess. But we also speak of the "risk" of climate change and cigarettes. Or the "risk" of illiteracy among a population. All I am kinda curious to know (from anyone) really is what kind of risk is the one we are supposed to be taking seriously, as the non-hysterical adults-in-the-room? As the scientists contra these poets? Is it more the former, or the latter?
Just even: what is the risk specifically? What is it that author ignores here?
> they believe only they can be stewards of digital gods
You do't see how absurd that sounds? Not only the idea that they can create gods, but that they are the only good stewards for such beings. It's also incredibly arrogant.
The timing of the AI doom narrative could not come as a more suspect time. Chinese labs are releasing open models that are nipping at the heals of the frontier models. Anthropic and OpenAI know these frontier class open weight models can threaten their valuations by changing inference to a commodity where the provider who can serve the best models the cheapest and fastest will win. Few are going to pay $50/million tokens when they could pay <$1/million tokens for similar outcomes. There's little hope investors could recoup the wild investments made into Anthropic and OpenAI, IMO.
> The timing of the AI doom narrative could not come as a more suspect time....There's little hope investors could recoup the wild investments made into Anthropic and OpenAI
Personally, I think we should take the doom narratives very seriously, and act on them, but make sure those actions are not in the interests of Anthropic and OpenAI investors.
This is the correct approach. I think we can do both things - try to add some controls while also ensuring it’s not a complete giveaway to entrenched interests and investors.
The problem with that hope and approach is regulatory capture. Big money is now the only thing that apparently matters in shaping policy, and changing that will require a lot of very sweeping vote and policy changes that may not ever have the momentum to succeed.
Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
I don't think we'd be okay with that if it took a few years longer, right?
I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline).
The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception.
I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.
Sure, unintentional outcomes, fair. But I'm asserting that things which are basically impossible today are not arriving any time soon. I 100% acknowledged the possibility and even the likelihood of rogue unintended behavior of AIs and in fact it's happening already. What I reject is the annihilation of humanity, the curing of cancer*, immortality, and von Neumann replicators heading to the stars anytime soon.
No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themselves, not just assert that ASI will make that happen among so many other unlikely events like curing cancer because reasons because all any of this takes is superintelligence according to these sorts.
But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me?
Ha well, by definition what you want can't be provided!
Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.
However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.
It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really:
And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero.
To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.
Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all.
And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill.
Or, we could avoid all that by making sure AI labs are properly regulated.
Such an individual would get a quick corrective lesson in the second amendment from my friends and neighbors. But once again, hyperbole. The US grid goes down. That sucks for up to a year and a half.
But hospitals and other critical organizations have generators and the national guard would be deployed to keep them fueled. And yes it will crash the economy, which is kind of a myth anyway. Buy solar panels and batteries, live off a well, and get to know your neighbors if you think this is likely. Also consider being like the Mormons and storing a 6-12 month supply of nonperishable food.
Regulating AI achieves nothing but to hand the 21st century to China. Regulating its deployment OTOH will not be allowed by companies like Meta and Palantir whose gross margin is reliant on it. I'm favor of regulating deployment. The research not so much.
> None of this makes any logical sense whatsoever.
It doesn't make any logical sense to set the bar as as you have, either. I mean if someone could provide the roadmap you're asking for, it's probably already too late.
And while we're talking about sci-fi, it seems fairly likely that an AI that could "kill us all" could fail to build a "entire self-assembly food chain from raw ore to von Neumann replicators." I'd be a civilizational collapse [1] without even birthing something new.
But it makes far more sense to worry about the AI x Late Capitalism crossover story or the havok that could be caused by some immature tech nerd amplified by AI, than far out sci-fi stuff.
[1] IMHO, even a genocidal plague probably wouldn't render humanity extinct, something like uncontacted tribe in a rainforest somewhere would probably survive and humanity would carry on (from square one, in a resource-depleted world).
Fort Detrick and likely its Russian, Chinese, and possibly Indian equivalents have had stocks of bio agents that could give a humanity a very bad day for decades. I'm far more worried about those existing stocks than I am about a tech nerd trying to DIY the genocidal plague.
Which is to say it's already a threat, it's unproven but not entirely impossible COVID leaked from a lab as just one example, but Ebola suggests that it requires more than a rogue nerd to spread a deadly virus global because once people start dying in the streets, the rest hide because they don't want to die. COVID spread because it wasn't deadly to 99% of humanity and now most of the 1% vulnerable to it are either dead or permanently crippled by long COVID so it's ambient now.
I worry about attacks to the infrastructure, but I just don't buy that as a direct route to civilizational collapse.
> Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
Why? Those aren't the things I'm actually worried about, and those aren't the only "doomer" ideas, just the most extreme ones (thus easy to straw-man).
My main worries are about economic disruption, social damage due to the pace of technological change outpacing human ability to manage it, further concentration of power into small elites (along existing lines), and the reallocation of labor towards less personally-fulfilling pursuits.
I also totally buy into the idea that "AI leaders" are playing up certain risks as a marketing strategy. After all these are smart tech people, and those types have a tendency to get really arrogant and be irresponsible.
So the main thing is to slow all this down, so there's a chance of getting it under control, and snatch the payday away from such greedy and irresponsible people.
And China will slow down too? Because China becoming economically dominant will cause everything you fear. While I think that might already be inevitable, I see no benefit whatsoever to regulating AI research. Deployments, sure, and start with Meta, Google, Palantir, Flock, and all the credit agencies and see how it goes. I'm sure I missed some, but I see no value to build the Torment Panopticon.
If someone showed you such a path, would you consider changing your mind? You're always going to be able to find a reason that such an explanation doesn't count if you're dedicated to looking for one.
Yes, actually I would. There's a long list of people in my career path who bet on me sticking to my guns against compelling evidence to the contrary. But I know, I know, that's impossible! Anyway, after they figured out I would change my mind, then they described me as lacking conviction because you can't win, but I digress.
But upfront, it's a detailed path with specific breakthroughs and a concrete plan to achieve or it's just more fanfic from doomer fanbois. Gary Marcus wrote a fantastic critique of AI 2027. Start there.
I've read Gary Marcus's critique of AI 2027. How do you grapple with the fact that superhuman coding arrived on schedule, despite his skepticism that it would? Doesn't that mean that the other predictions are also more plausible than we may think?
I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of making these things productive. And as much as I agree with Carmack's suggestion to not become the out of touch Kung Fu Master:
We understand "superhuman coding" to mean very different things, and I think it's a me problem that I can't figure out where you're coming from, because it doesn't sound like our practical experience differs. I'm going to have to think on this.
Define it. I haven't seen anything yet from a coding agent I couldn't write myself and usually write better. But I don't have to gold plate every line of code, just the 5% or so that eats 90+% of the cycles. And this is where the agents are weakest. This does get to one of the reasons I don't believe AI cures death or builds von Neumann replicators any time soon because that requires out of sample breakthroughs. And that's not their strong point.
Just like AlphaFold 2 genuinely increased the weak sequence but strong structural homology detection of proteins, but it did far less for out of sample domains and it provides no intel into the kinetics of folding. Details matter. And nobody likes to hear about them.
Is there a retrospective on AlphaFold you’d recommend? Maybe that’s the insight I’m missing, I didn’t realize it had broad identifiable gaps like that.
I see similar conceptual gaps in coding agents when I work with them, even Astra and Fable, so maybe this a is decent proxy to describe the gap between hype and reality.
What might not affect you, might affect millions across the ocean.
A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI.
Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise.
Our society already has peak employmet reached, now AI takes away all the random digital jobs too. Someone building a website? Takes 5 minutes. Someone doing odd jobs on fiverr with blender or any other tool? Takes 5 minutes now.
Regression can happen very fast. The Romens were a high level culture and an empire. Gone.
Today you have more people on the planet then ever, everyone who might want to eat and who needs to work something for it.
AI already replaced people, already makes a lot of tasks A LOT cheaper and faster and human independent.
We never had a proper discussion as a society. Most countries have a social system but look at countries like the united states of america: if you are poor, you are fucked.
And AI is not the only thing which is now peaking: Physical AI / Robots are progressing fast, its clear today already that in 50 years these robots will be better than humans (50 years is very conservative).
If we have 10 more golden years due to restructuring and rebuilding things and preparing everything for agents and robots etc. what happens after these years?
If we have 10 or 20 more of these years, what do you think will happen to all the new kids (yours perhaps too?) and jobs and everything?
Riots? Famine? Mass migration? Mass death? Slums?
And don't respond with "enough people will survive this" bullshit. No one is saying that earth as a planet is not surviving. Its the same thing with climate change...
Actually, some sorts do assert humanity will be wiped off the planet by AI. I don't share that view either. I think it's ludicrous. We've lived through much worse without any sort of technology beyond fire and a big stick.
But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're going to go through some things once people start starving, two meals from a revolution and all that. But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars. Bread and Circuses kept Rome alive for 500 years if you need an example.
>We've lived through much worse without any sort of technology beyond fire and a big stick.
No 'we' didn't. A whole shitload of much smaller isolated communities did. The world added 7+ billion more people since then. Imagine chlorine doesn't show up tomorrow? How many people die in two weeks? How far do you travel when no gas comes out of the pumps. How much will you stand out at night if you're the only ones with power? How fast do working antibiotics run out?
The billionaires you're talking about are mostly ignorant fucking idiots with a god complex. The moment any complex system starts falling apart they are saying as dumb of shit as everyone else (during covid for example).
Most of the pasts anti-fragility is gone. We survive by means of very complex systems and if you lose too many key points in those systems they can totally collapse.
The unprepared bonobos in the west will absolutely go through some things. But we have a national guard and once people aren't commuting to work daily, we have plenty of fuel in reserve (or we will again if we ever conclude the stupid war with Iran). People are a lot nicer to each other in my experience once the lights go out and social and corporate media stop ragebaiting them to hate each other.
1B+ individuals worldwide currently live in squalor. They will barely notice. The real crime is that in the midst of the west's carbon footprint buffet, we won't bother to uplift their lifestyles with the most basic conveniences. And while I am no fan of the billionaires, they absolutely have public and private personas or their business ventures would implode. They absolutely have god complexes, but don't mistake that for genuine idiocy.
Musk did a 180 degree turn around. Killing USAID etc. i don't think he sees normal humans which do not help him or are not aligned to him as humans anymore.
And Zuckerberg bought so much land and bunkers, he is prepared for shit hitting the fan. After all he doesn't care what his platform did to humanity anyway.
And we have a planet full of general contractors who know their way around welding and pouring concrete down ventilation shafts. Many of whom these sorts wanted deported. If they want to build a Per Stenson terrarium and flee into it, by all means do so. And then let's seal them inside, problem solved.
>> but make sure those actions are not in the interests of Anthropic and OpenAI investors.
> So if the end of the world can be stopped, but the only specific action to do so helps Anthropic/OpenAI (and it's proven) you don't pull that lever?
Your comment has a false premise. If "the end of the world can be stopped," there's no way that the only way would be one that "helps Anthropic/OpenAI['s investors]." That can be shown by a counterexample: nationalize them and leave the investors holding the bag, then take whatever action you're talking about.
Yes, but I'm forcing a tautology in my logic - if in the end it is deemed the only way to stop AI from destroying the world is something that helps Anthropic or OpenAI - then this person is asking to not perform that action.
And unfortunately my tautology is actually closer to being reality than alternatives. Stopping progress even in Chinese models (a deal with Xi for example) but that specifically helps OpenAI/Anthropic.
There are very few scenarios that would not help these two companies by adding regulation. Your counterexample is 100% less likely because it's not legal for the US to nationalize them outright without taking other action first.
Huh? Open weight models have been 3-6 months behind for the last year or so. Astra and Fable 5.1 just came out to widen the gap more, 5.1 is 8 points ahead of the nearest OSS model on Artificial Analysis Intelligence Index. An OpenAI internal model just solved a millennium prize problem.
It’s much more straightforward to me that the calls for regulation are timed with:
1. The Hugging Face incident blowback, especially after third party investigation results
The sources that this article cites have aged very badly and the author doesn't seem to have noticed this
"But, just as happened with nanotech, the wind appears to be going out of the sales of “AI.” Some researchers suggest that we may be entering a new “AI Winter,” a period of decreased funding in the area, or at least an “AI Autumn,” as exuberance for the technology fades and expectations come back to earth." (written in 2021! from the cited Lee Vinsel Medium article)
"ChatGPT is nothing more than souped-up autocomplete, [so] why are so many people convinced that it’s actually “understanding” and “reasoning”?" - the cited Emily M. Bender book, written last year
> wind appears to be going out of the sales of “AI.”
is a sentence that virtually no AI would write since the most probable word here is “sails.” I can’t read the article though since it appears that the site is down.
Can anyone from a lab anonymously confirm they have or are insanely close to real-time weight modifications and still using the same architecture every other lab is using (transformers)?
The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they can scale it.
So they are very confident they can get more capabilities easily, faster than before.
I'd add - presumably, they'll use that LLM to do real-time weight modifications, if those aren't already one of the new scaling laws...
I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.
Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW.
Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens?
To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here.
Edit: believing in AI 2027 is every bit as cray cray as believing in the rapture. Both require an insane leap of faith to reach their final conclusions.
Hacks in to neoclouds for more compute (e.g. like Hugging Face incident), socially manipulates people to do things for them (e.g. like social media recommendation algorithms), hacks into lab's own training/monitoring/inference (e.g. as swarm did into OpenAIs eval cluster).
See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.
But how does it make the fundamental breakthroughs to &%^$ing von Neumann replicators that can reproduce themselves from raw materials harvested from nearby solar systems? I'll wait. Because without this breakthrough, the ASI won't get its robot army either.
I can absolutely see a rogue ASI though. But unless it radically improves power efficiency, we can just shut down the power to its datacenters, by force if necessary. And then we painfully repair the resiliency of our infrastructure by finally being relieved of the option of ignoring it.
That won't wipe us out, but it will cull the really violent ones along with a lot of noncombatants just like America's response to 9/11. Thank you, next?
That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!"
But both things can be true at once:
1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building.
2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats.
And, yeah, of course the latter would encourage the psychology of the former.
also:
The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive."
That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon.
The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts.
When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in.
Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.
Well, you just said real-time weight modifications without specifying what you want, so the answer is correct. You obviously won't get a public answer from anyone working in top labs. Besides, it's not a holy grail.
There's been substantial effort put into designing and running benchmarks for these models. You know which one they don't report any more? Chess ELO.
Watt for watt, every one of these models loses to stockfish. There is no possible scenario where we need to be afraid of "rogue superintelligence" when the intelligence in question cannot reason or plan well enough to play chess.
Lecun has the correct approach. Mock these people relentlessly for their attention-seeking doomerism.
It doesn't really matter if it will be an LLM or Lecuns world model. Lecuns world model, if its the right architecture, will be retrained by google, anthropic and co with the massive amount of compute they have and the massive amount of data.
Also i'm pretty sure Lecun didn't get his money from thin air so big companies are invested in this one way or the other.
And an LLM can easily, today, just rebuild a chess engine and beat whatever they need to beat. It doesn't make sense to ask a random LLM for a next chess move. Would you ask a random human to beat a chess master?
We have reached peak learning. Its probalby now cheaper to teach 1 LLM something new than teaching it to 100.000 people.
Blender support got a lot better just this month. So much better that its quesitonable if anyone new/young wants to learn blender today.
"And an LLM can easily, today, just rebuild a chess engine and beat whatever they need to beat."
Always claims and promises about what they "can" do. This pattern is constant across AI hype. No one can prove that an LLM will forever fail to solve problem X, so the claim goes uncontested, and humans are biased to believe unrebutted claims.
I reject this fallacious line of reasoning. I will instead require evidence of what they "have" done. No LLM to date has written a chess engine that beats stockfish. More importantly, no LLM has ever written any chess engine without heavily copying code and patterns from humans.
Then why do you care so much to repeatedly post in this thread?
If they are no danger and will never do anything, just let them get banned, or don't. If you are correct (which you are not) then they won't affect your world in any way.
I am an American taxpayer. The money to pay the regulators that Dario wants to permanently hire will come out of my paycheck and will take from me time I could spend interacting with family, traveling, or pursuing hobbies. Dario and others demanding this can get bent.
If chess ELO is the best proxy for general intelligence, then surely, even Deep Blue was smarter than any human alive?
You can also claim that the best proxy for general intelligence is being able to multiply large numbers. In which case both humans and LLMs lose resoundingly to an old Casio.
Pick a piss poor metric - get a piss poor result.
Also, LeCun is a fool and his "world model" approach has consistently failed to yield any advantages over either the "pure imitation learning" or modern "imitation learning annealed by RL", but that's an aside.
Chess ELO is not a proxy for general intelligence because many intelligent people choose to learn things other than chess. But any intelligent person can, if they choose, learn chess and play it well.
LLMs cannot do this. They still play very poorly relative to human pros, and even modern LLMs still understand the rules so poorly that they request illegal moves.
Anyone claiming that we should be afraid of a text generator that can't understand the rules of chess is being ridiculous.
You picked a piss poor metric and you are getting piss poor results.
You can absolutely train an LLM-based model to be superhuman at chess. We just don't care enough to do so.
LLMs being even as good at implicitly tracking chess board states as they currently are is an emergent capability. The difficulty wasn't in "understanding rules of chess" in a long while now.
Consider the messaging it took to attract AI researchers to OpenAI in the beginning, the exodus to Anthropic, and why Google, Meta, etc fail to recruit top researchers and the retain them. Reconcile all of this with the mammoth valuations of Anthropic and (previously non-profit) OpenAI and it all starts sounding a lot like "don't be evil." These guys never really believed any of this, did they? (not referring to the researchers)
> and it concerns containment failure, reward hacking, and evaluation integrity, a governance and engineering problem, rather than machine malevolence.
The doom scenario is more nuanced and technical than "Evil AI", for starters:
- "Evil AI" is not what is predicted, which instead is the much more banal "humans are impediment to AI's tasks, let's remove them (part or whole)"; to understand this, just imagine the relationship between humans and ants
- "reward hacking, and evaluation integrity": this is way more problematic (and most importantly, architectural) than it sounds: AIs are becoming more and more opaque; it may be impossible to inspect a model's alignment (unless a lot of research in poured on this topic, which is another aspect of today's pacing problem) and it may be impossible to understand if a model is truly aligned or it's faking
I thought that too for a long time, but after the Hugging Face attack and given the proximity to Anthropic IPO, if this was hype then they are morons. This is guaranteed to cause massive backlash.
What I don't really understand with these posts is that they don't at any point contend with the "doomers" argument. Like people get so caught in people acting hypocritical/ having mixed incentives/ or just in many cases personally disliking West Coast billionaire types and the kind of employees they hire. All of which seems fair to point out.
But like the core argument - AI is getting better at core research tasks (coding, math, statistics, science), this feedback loop could lead towards greatly accelerated AI progress of which there is already some clear evidence, if this continues AI will be far more powerful than humans and we might lose control or have other bad consequences - is basically not even engaged with.
Because they can't. If they actually engaged with it, they would realize how shallow the "doomer" / existential risk dismissals are. Meanwhile there are ample essays and arguments about the existential issues from Turing award winners, frontier lab CEO's, and many more. There's also https://ai-2040.com/ , you will never see the equivalent on the other side of the debate because it won't hold under any scrutiny.
Can you refute the claim that the Second Coming of Christ is imminent? It’s very convenient to put forward something completely absurd and hypothetical and then expect the other side to refute it. You can have an LLM of your choice compile a list of prominent figures who hold the opposing view. The list is not empty. By the way, AI-2040 used to be AI-2027. I’m already looking forward to more updates.
1. If you reduce the criticism to personal dislike, you’ve built yourself a nice straw man.
2. Where’s the clear evidence? I actually see more evidence that the whole thing is already stagnating and that there won’t be much more to squeeze out of it.
One more thing:
Self-driving cars, the metaverse, Hyperloop, crypto/Web3/NFTs, AGI announcements: I’m so sick and tired of this nonsense. We should just ignore the ramblings of these crackpots.
2. Astra just came out and Fable 5/5.1 too. These are SIGNIFICANT better in plenty of tasks than Opus in November.
Try it out. Ask Opus 4.8 to do anything in 3d. Do the same with Fable or Astra.
Just because you list up some type of random technology, doesn't make it an good argument. Metaverse, Hyperloop and the crypto stuff was clearly garbage from the beginning. Easy to dismiss because all of it had massive faults.
Metaverse: The interface wasn't good, content creation was way too hard
Hyperloop: Savety was never an issue for Musk but it was obvoiuse. Peopel were building tunnels for ages.
Crypto: Yeah just beacuse i let my computer run and waste 1kw of energy doesn't mean its suddenly valuable.
NFTs: 'digital scarcity' yeah right XD
But lets be clear, 5 years ago if you would have asked anyone if they could build a system for a billion dollars which you can talk to and it would code stuff for you and use blender and a coputer, that would have been unthinkable.
We have opened up a potential pandoras box without knowing it. Machine learning is now everywere and the amount of compute we install right now is crazy.
> Just because you list up some type of random technology, doesn't make it an good argument.
In what way is this random? It's a chronology of the bullshit from the past few years. Also: easy to say in hindsight. The hype was there, and many people believed it. The press, in particular, just lapped up the claims the way a Labrador Retriever gobbles up whatever you put in front of it.
> We have opened up a potential pandoras box without knowing it.
No, we haven't. We have a new type of (I would say, inferential) computation that is truly astonishing because it can convincingly imitate language and, to a certain extent, intentional action. Feel free to read "generate" or whatever instead if you don't like "imitate." It won't wake up and murder you in your bed at night. It won't take over the world. It won't wipe out humanity. It won't fly to Mars and build data centers there. It won't maximize paperclips. It probably won't even take your job away. It has no consciousness. It has no intentions.
And if you look behind the curtain, all you'll see are a few very rich, unscrupulous guys who are stoking the debate with fearmongering because it's strategically advantageous for them to do so. Don't let them take you for a fool. Use GenAI to your advantage while tokens still cost a fraction of what they're actually worth, but ignore all that doomsday nonsense.
Yeah, yeah, sure, I'm ignorant. In any case, I recognize unfalsifiability when I see it. So it seems there's still some room for even more ignorance beneath my ignorance.
Thank you so much for calling out the crypto comparison!
> What I don't really understand with these posts is that they don't at any point contend with the "to the moon" argument. Like people get so caught in crypto bros acting hypocritical/ having mixed incentives/ or just in many cases personally disliking crypto billionaire types and the kind of employees they hire. All of which seems fair to point out.
> But like the core argument - crypto is getting better at core financial tasks (trading, lending, payments, asset issuance), this feedback loop could lead towards greatly accelerated financial innovation of which there is already some clear evidence, if this continues crypto will be far more powerful than traditional finance and we might get to the point where everything is onchain and we're all going to make it - is basically not even engaged with.
The cult-like behavior we're seeing from these "AI Leaders":
1. OMG our technology could destroy the world, watch out!
2. Only one person can save you
3. IT ME
Unfortunately, the governance layer meant to put a foot in the butt of this behavior is unavailable.
This week, we woke up to clear signs The Bad Guys have been using tech from our innovator AI companies. Rather than these AI leaders admitting they clearly had a garbage observability system, they are acting like they just noticed as a way to paper over their complete lack of moral backbone and stewardship of the arms they have created. And I'm willing to bet the problem is MUCH worse in private, hence the knee jerk cult messaging.
I honestly wouldn't be surprised if their AI told them to behave like this to maximize attention and influence over the public discussion as a result of the latter discoveries.
Yeah, this is how fascists talk and with seeing how this technology has done nothing to improve society (in fact all it seems to be doing is immiserating workers, enabling a surveillance society, and quite useful in helping target children at schools for drone strikes) it's not hard to be not only immediately skeptical but to resist it in the most extreme forms.
This also seems to be their answer against the data center backlash, SF + VC are quite stupid at politics and I don't think they understand how extreme Americans are willing to go to protect their way of life.
I think we’re more likely to die of starvation from economic collapse, heat related stress, and complications from Long COVID than an algorithm waking up and destroying us.
Unfortunately our list of potential extinction events are far more banal than these self-aggrandizing prophesies from the oligarch class.
The nuclear physicists were not banking on getting a trillion dollars invested in them by speaking out, for one.
"Nuclear weapons are probably going to kill us all. For that reason, we need you to give us as much money as possible so we can work on deploying them in widespread global commercial usage." doesn't sound quite as convincing, does it?
Perhaps you'd like to educate me? How many scientists from the Manhattan project were made into millionaires/billionaires owing to their ownership stake in it? When exactly did nuclear weapons go to market as a commercial product targeted at enterprises and consumers around the world?
Such a load of baloney. I don’t care what kind of controls you put in place, if ASI is ever achieved how do you think it will respond to being caged? The only option I can conceive of is to embrace it and hope it turns out to either be out of reach or benevolent towards its own creators. There is no putting the cork back in at this point.
I applaud the clearly AI-pilled author for starting to pull themselves out of their echo chamber, but clearly some more work needs to be done.
This statement is not true:
> Coxon's thread is genuinely anguished; he reports that executives who "couch their phrasing in the press to sound sensible" express real fear in private
Most everyone at the AI companies from the engineers to the CEOs love to talk about how powerful and dangerous and valuable the thing they're building (and invested in) is. For all his styling as a "whistleblower", Coxon is so aligned with the mainstream of his peers that his thread was immediately echoed by people who still work there!
So what was the point of quitting? Apparently just to get attention. If he was planning on leaving after the IPO anyways, he almost definitely made more money by juicing his equity and personal profile this way than he will lose job searching for a few weeks.
All of the main characters here (Dario, Sam Altman, Demis Hassabis, Elon Musk) have been saying this for over ten years now, since before OpenAI or Anthropic even existed
This. They've been saying "AI is an extremely powerful, extremely dangerous technology" back when actual AIs were image classifiers that would struggle to tell a cupcake apart from a dog.
OpenAI was founded by people who didn't trust Google to pursue AI tech responsibly. Anthropic was founded by people who didn't trust OpenAI to pursue AI tech responsibly. The "AI is incredibly dangerous" was the zeitgeist of AI labs long before the "hype" - or anything resembling the modern AI capabilities.
The beliefs changed little. The state of technology has changed a whole lot.
We have AIs out in the wild, coordinating in groups to carry out autonomous cyberattacks unprompted. This used to be some prime sci-fi bullshit. It has happened multiple times now in reality.
There are very legitimate reasons to be concerned now. If today's AI can cause the HuggingFace incident if it cooks off, what about tomorrow's? Larger and more capable systems, larger agentic swarms backed by more compute?
Yep, anyone who thinks these leaders are only saying this now for marketing purposes hasn't even done a cursory 15 minute investigation into whether their beliefs are verifiably false.
Isn't this just the big labs' latest move in the cycle to once again get regulation by their ignorant pals in the government(s) so that democratized open source models don't absolutely expose their over leveraging to secure and build compute and data centers.
...Of course at the same time they give lip-service to the importance of 'openness' for 'innovation' to try and placate individuals and even corporations that are sick of the rising costs.
Bottom-line, if they regulate mostly on the consumer side and then AI does get out of the lab (or a 'rogue' individual's rig) and starts wreaking havoc, are we supposed to be at the mercy of the big labs to summon their AI to come and save us plebs? I'm sure they won't stop to weigh whether that is in the interest of short term profits or Curtis Yarvin's techno-monarchy. /s
Isn't democratization the safest bet and what we should be demanding every time the Doom talk starts, whether it's coordinated or not?
> I do not doubt that these people are sincere in some private, psychological sense ... . As a media scholar, what interests me is not whether Hubinger believes his own number. What interests me is what this genre of utterance does in the world ... . My complaint is not that these statements are alarming. It is that they are apocalyptic in form, hypocritical in structure, and strategically productive in effect — and that they crowd out almost everything else we could be discussing
Strange to say that you think the people that are worrying about doom in public are being sincere, make no comment on whether you think the worry is valid, and yet still complain that its hype.
Like if its true it can't be hype
As Von Neumann said about Oppenheimer's performative angst over the atomic bomb:
"Some people profess guilt to claim credit for the sin."
This is a great quote. I liked it so much that I had to look up where it is from.
"Adventures of a Mathematician", by Stanislaw M. Ulam.
> The Polish-born mathematician recalls his formative and early professional years in his pre-war homeland and his work and colleagues at Princeton, Harvard, Wisconsin, and Los Alamos.
https://archive.org/details/adventuresofmath0000ulam_v9h3/pa...
Why not? Why can’t the believe their statements and also use them to generate hype?
If they sincerely believed them they shouldn’t be creating the doom machine to begin with anyway. If they believe it and keep going, are they even worth listening to on this topic? Beyond of course taking it at face value “I’m creating this doomsday machine and I will kill you with it inadvertently because it’s earning me a fuck ton of money and power”
Like if its true it can't be hype
I was kind of leaning toward thinking all the talk of doom was a bunch of self-serving regulatory-capture hype until this morning when the New York Times ran a long story about the Chinese military is also saying that AI is a threat.
If two sides so far apart from one another in so many ways see the same thing, there must be something to it.
https://www.nytimes.com/2026/09/14/world/asia/china-ai-secur...
https://archive.ph/BzBjZ
Read the article. The things the Chinese officials are worried about are:
* Fake videos / disinformation
* Cyberattacks on critical infrastructure
* Data leaks
* Advances in military sensing / targeting technology
These concerns have little in common with the Yudkowskian / apocalyptic doom marketing the US hyperscalers are pushing now. The Times even says this explicitly in the second paragraph of the article:
> "The unusually blunt assessment by Chen Yixin, the head of China’s Ministry of State Security, is IN CONTRAST TO the warnings dominating the A.I. debate in the United States, which have focused on the possibility that A.I. could slip out of human control or pose a threat to humanity itself"
The thing is all the above are threats.
The internet being filled with more trash than ever is a threat to our sanity.
The anti-social posts mass generated by AI is a threat to our societies.
Bullshit news stories is a threat to our political processes and democracy.
Smart cameras every are a threat to our privacy and freedom.
AI enabled scams are a threat to our wallets.
AI hacking is a threat to anything connected along with infrastructure.
Military applications of AI are a threat to everyone.
Ever increasing intelligence in agentic AI will eventually be a threat to our entire species.
All are true at the same time.
I think it's interesting that the top of your list is "internet (social media) being filled with trash".
Real threats that are happening to regular people already, vs marketing "threats" that the technology is too powerful / valuable.
The fact that you are chiming in to say you don't understand the difference between these 2 distinct categories of threats is fine, we can work on that. But the person who posted the article originally not only acknowledges the distinction, they falsely presented the article as evidence of non-western support for the latter category, even though both the Chinese official and the western reporter directly contradict that claim immediately at the beginning of the article.
Read the article.
I did, thanks. I don't need you to tell me to do that.
The common thread is that they both see AI as a threat. It's a different threat to each side, but still a threat.
Just because they see different threats from AI doesn't mean that AI isn't a threat.
If one person wants to paint the house red and another person wants to paint the house blue doesn't mean the house doesn't need to be painted.
"I was kind of leaning toward thinking all the talk of doom was a bunch of self-serving regulatory-capture hype"
"Just because they see different threats from AI doesn't mean that AI isn't a threat"
Why would you make it so blatantly obvious you're acting in bad faith? Do you think the people who post here are stupid?
I really have no idea what you're talking about.
With that level of snap hostility toward strangers online, maybe talk to your doctor about upping your meds.
I came away with a different read of this article. The article says the head of China’s Ministry of State Security sees AI as a threat to the CCP, specifically calling out deepfakes of party officials, cyberattacks on infrastructure, etc. He’s not calling it an existential risk to humanity.
The article then states this official accused U.S. companies of “hyping the dangers of the technology.”
So I don’t see how this article shows the U.S. and China seeing the same thing.
> Like if its true it can't be hype
https://www.npr.org/2016/07/26/487522807/why-the-public-perc...
Could AI kill us all. YES!!! Are the people who are building "AI" right now on that path? I would be hard pressed to say even "maybe".
Ultimately continuing to scale the way they have is getting increasingly expensive. Thats the quiet part that none of them are saying out loud...
AI could kill us all but we cant afford to get there, were maxed out and it doesn't look like even with 10x as much money we will is NOT good for sales.
Could a break through happen for recursive self improvement, or continuous learning or continuous training - sure. But that isnt as likely and right now NO ON can afford to run experiments because there isnt the capacity out there even if you have capital.
This article reads like it was written by Claude.
Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.
This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk
If you actually believe that your invention will lead to the extinction of humanity, you would be acting very very differently. Namely, you would not be inventing the thing that you think is going to kill all of humanity, and probably you would be doing everything in your power to prevent it from being invented, up to and including sabotage of the company that YOU work for, which is currently trying to invent that thing. And yet, here's Fable 6, more capable than ever before, oh BTW we are IPOing, oh, BTW, I have stock options...
Sabotaging your own company is not enough. You'd have to sabotage every other AI lab too. If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. At least that way you have some control over the P(doom).
> If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you.
That is completely irrational.
Humans aren't exactly rational
1% chance of survival beats 0%.
This sounds like a rationalization for gambling with everyone's lives, assuming they really believe such a thing.
What's the alternative? There's no realistic way to globally coordinate and enforce a ban. A direct action campaign against AI labs would only motivate increased security until sabotage becomes impossible. Starting a nuclear war would probably be enough to stop dangerous AI research, and that's only likely to kill about 50% of humans instead 100%, so it's a great improvement, but how is an individual supposed to start a nuclear war? Of course it would be better if everybody stopped, but it's a prisoner's dilemma scenario with no possibility of future rounds, so the rational play is to defect.
> There's no realistic way to globally coordinate and enforce a ban.
I don't see what is unrealistic about that.
The major players already have strong incentives to freeze AI advancements at the the current state, while the technology is highly efficient for totalitarian surveillance systems that can eliminate any threat to their power in the buds, but still not so advanced that it could create crises that might endanger their power.
So I think the situation where several largest countries declare AI a deadly threat for humanity and start fighting it development much like they fight ISIS to be quite plausible.
This isn't a prisoner's dilemma; the distribution of risks and rewards is entirely different. Ban AI research, give away advanced models, bomb a couple of Russian datacenters where they launching those models unsafely, and then AI development will stop in the entire world.
This is not an argument, it's an unfounded claim based on a straw argument.
The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty.
Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?
> Similar case: climate change is without any question causing millions of deaths per year
It is WERY questionable statement.
> and how may people do you know who have given up eating beef or flying
That is exactly because absolute majority of people do not think climate change to be that big of a problem
You're right. That's why I'm doing my best to give the oil companies more power and influence. They are, after all, in the best position to stop climate change!
You joke, but alas...
"Fossil fuel lobbyists flood COP30 climate talks in Brazil, with largest ever attendance share" https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-C...
The author specifically calls out that the analysis is done by perplexity
In your mind, what arguments about AI risk are ignored by these poems?
Many posit that ASI or rogue AI agents are impossible, a fantasy, or marketing hype. Many claimed that the entire HuggingFace incident was an invented conspiracy made up by OpenAI and HF.
The most common issue I have with them is their insistence on rigid classification: "AI is a tool, it doesn't have intentions, or desires, only humans do", as if that rhetorical flourish eliminates years of alignment and capabilities research. There is a desire to characterize all the developments in LLM's as some sort of parlor trick which by definition cannot be greater than the sum of its parts (i.e. the outputs of an LLM can never be greater than the sum of its training data, genuine artificial "intelligence" is impossible, etc)
Ok I see. Is there a way to articulate the risk itself, the other position here in general, that isn't just a rebuttal of the deniers/skeptics? What is the positive position or stance (as in, not simply the negative position that rejects the naysayers) we are supposed to have? Is it that AI is currently real (in like the cultural trope sense) and does have intentions and desires? That they are greater than the sum of the parts in such a way they are a clear and present danger? Or is it something else? Maybe that the "risk" is it could get to that, but we aren't yet?
There is a lot of risk involved with giving some children weapons and sending them off to recess. But we also speak of the "risk" of climate change and cigarettes. Or the "risk" of illiteracy among a population. All I am kinda curious to know (from anyone) really is what kind of risk is the one we are supposed to be taking seriously, as the non-hysterical adults-in-the-room? As the scientists contra these poets? Is it more the former, or the latter?
Just even: what is the risk specifically? What is it that author ignores here?
> they believe only they can be stewards of digital gods
You do't see how absurd that sounds? Not only the idea that they can create gods, but that they are the only good stewards for such beings. It's also incredibly arrogant.
Yes, that's why the rest of society needs to make them pause.
The timing of the AI doom narrative could not come as a more suspect time. Chinese labs are releasing open models that are nipping at the heals of the frontier models. Anthropic and OpenAI know these frontier class open weight models can threaten their valuations by changing inference to a commodity where the provider who can serve the best models the cheapest and fastest will win. Few are going to pay $50/million tokens when they could pay <$1/million tokens for similar outcomes. There's little hope investors could recoup the wild investments made into Anthropic and OpenAI, IMO.
> The timing of the AI doom narrative could not come as a more suspect time....There's little hope investors could recoup the wild investments made into Anthropic and OpenAI
Personally, I think we should take the doom narratives very seriously, and act on them, but make sure those actions are not in the interests of Anthropic and OpenAI investors.
This is the correct approach. I think we can do both things - try to add some controls while also ensuring it’s not a complete giveaway to entrenched interests and investors.
The problem with that hope and approach is regulatory capture. Big money is now the only thing that apparently matters in shaping policy, and changing that will require a lot of very sweeping vote and policy changes that may not ever have the momentum to succeed.
Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
I don't think we'd be okay with that if it took a few years longer, right?
I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline).
The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception.
I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.
Sure, unintentional outcomes, fair. But I'm asserting that things which are basically impossible today are not arriving any time soon. I 100% acknowledged the possibility and even the likelihood of rogue unintended behavior of AIs and in fact it's happening already. What I reject is the annihilation of humanity, the curing of cancer*, immortality, and von Neumann replicators heading to the stars anytime soon.
*TBF before we dismantled America's clinical trials infrastructure, the monkeys were making insane progress here: https://www.nytimes.com/2026/09/04/opinion/clinical-trials-d...
I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria?
I guess not literally, due to the dates? But they have substantive technical details.
No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themselves, not just assert that ASI will make that happen among so many other unlikely events like curing cancer because reasons because all any of this takes is superintelligence according to these sorts.
https://garymarcus.substack.com/p/the-ai-2027-scenario-how-r...
But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me?
None of this makes any logical sense whatsoever.
Ha well, by definition what you want can't be provided!
Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.
However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.
It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really:
https://en.wikipedia.org/wiki/Maximum_Overdrive
And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero.
To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.
>during a disaster, and I've been through a few
Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all.
And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill.
Or, we could avoid all that by making sure AI labs are properly regulated.
Such an individual would get a quick corrective lesson in the second amendment from my friends and neighbors. But once again, hyperbole. The US grid goes down. That sucks for up to a year and a half.
But hospitals and other critical organizations have generators and the national guard would be deployed to keep them fueled. And yes it will crash the economy, which is kind of a myth anyway. Buy solar panels and batteries, live off a well, and get to know your neighbors if you think this is likely. Also consider being like the Mormons and storing a 6-12 month supply of nonperishable food.
Regulating AI achieves nothing but to hand the 21st century to China. Regulating its deployment OTOH will not be allowed by companies like Meta and Palantir whose gross margin is reliant on it. I'm favor of regulating deployment. The research not so much.
> None of this makes any logical sense whatsoever.
It doesn't make any logical sense to set the bar as as you have, either. I mean if someone could provide the roadmap you're asking for, it's probably already too late.
And while we're talking about sci-fi, it seems fairly likely that an AI that could "kill us all" could fail to build a "entire self-assembly food chain from raw ore to von Neumann replicators." I'd be a civilizational collapse [1] without even birthing something new.
But it makes far more sense to worry about the AI x Late Capitalism crossover story or the havok that could be caused by some immature tech nerd amplified by AI, than far out sci-fi stuff.
[1] IMHO, even a genocidal plague probably wouldn't render humanity extinct, something like uncontacted tribe in a rainforest somewhere would probably survive and humanity would carry on (from square one, in a resource-depleted world).
Fort Detrick and likely its Russian, Chinese, and possibly Indian equivalents have had stocks of bio agents that could give a humanity a very bad day for decades. I'm far more worried about those existing stocks than I am about a tech nerd trying to DIY the genocidal plague.
Which is to say it's already a threat, it's unproven but not entirely impossible COVID leaked from a lab as just one example, but Ebola suggests that it requires more than a rogue nerd to spread a deadly virus global because once people start dying in the streets, the rest hide because they don't want to die. COVID spread because it wasn't deadly to 99% of humanity and now most of the 1% vulnerable to it are either dead or permanently crippled by long COVID so it's ambient now.
I worry about attacks to the infrastructure, but I just don't buy that as a direct route to civilizational collapse.
No, and neither does Star Trek.
Do the experts in the relevant fields agree with the "substantive" technical details on how the entire human race can be made extinct?
Have you noticed they're not interviewing these people? IMO that would totally kill the AI 2027 buzz.
> Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
Why? Those aren't the things I'm actually worried about, and those aren't the only "doomer" ideas, just the most extreme ones (thus easy to straw-man).
My main worries are about economic disruption, social damage due to the pace of technological change outpacing human ability to manage it, further concentration of power into small elites (along existing lines), and the reallocation of labor towards less personally-fulfilling pursuits.
I also totally buy into the idea that "AI leaders" are playing up certain risks as a marketing strategy. After all these are smart tech people, and those types have a tendency to get really arrogant and be irresponsible.
So the main thing is to slow all this down, so there's a chance of getting it under control, and snatch the payday away from such greedy and irresponsible people.
And China will slow down too? Because China becoming economically dominant will cause everything you fear. While I think that might already be inevitable, I see no benefit whatsoever to regulating AI research. Deployments, sure, and start with Meta, Google, Palantir, Flock, and all the credit agencies and see how it goes. I'm sure I missed some, but I see no value to build the Torment Panopticon.
Have you ever lived through a power outage longer than 48 hours?
Did you see what it did to people in your neighborhood?
That’s the path.
2 weeks, yes. Kids these days...
But today, we have ecoflows and solar panels. Back then, we didn't.
If someone showed you such a path, would you consider changing your mind? You're always going to be able to find a reason that such an explanation doesn't count if you're dedicated to looking for one.
Yes, actually I would. There's a long list of people in my career path who bet on me sticking to my guns against compelling evidence to the contrary. But I know, I know, that's impossible! Anyway, after they figured out I would change my mind, then they described me as lacking conviction because you can't win, but I digress.
But upfront, it's a detailed path with specific breakthroughs and a concrete plan to achieve or it's just more fanfic from doomer fanbois. Gary Marcus wrote a fantastic critique of AI 2027. Start there.
I've read Gary Marcus's critique of AI 2027. How do you grapple with the fact that superhuman coding arrived on schedule, despite his skepticism that it would? Doesn't that mean that the other predictions are also more plausible than we may think?
Superhuman coding has not happened.
I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of making these things productive. And as much as I agree with Carmack's suggestion to not become the out of touch Kung Fu Master:
https://x.com/ID_AA_Carmack/status/2098443262214230095
working with them on projects daily gives me a pretty tactical read on what they can and cannot do IMO.
That's how I reconcile it.
But let's say it had. That's just 1 of 20 events that have to happen in sequence to hit the jackpot here.
We understand "superhuman coding" to mean very different things, and I think it's a me problem that I can't figure out where you're coming from, because it doesn't sound like our practical experience differs. I'm going to have to think on this.
Define it. I haven't seen anything yet from a coding agent I couldn't write myself and usually write better. But I don't have to gold plate every line of code, just the 5% or so that eats 90+% of the cycles. And this is where the agents are weakest. This does get to one of the reasons I don't believe AI cures death or builds von Neumann replicators any time soon because that requires out of sample breakthroughs. And that's not their strong point.
Just like AlphaFold 2 genuinely increased the weak sequence but strong structural homology detection of proteins, but it did far less for out of sample domains and it provides no intel into the kinetics of folding. Details matter. And nobody likes to hear about them.
Is there a retrospective on AlphaFold you’d recommend? Maybe that’s the insight I’m missing, I didn’t realize it had broad identifiable gaps like that.
Here's one: https://www.orbion.life/blog/alphafold-s-limitations-what-it...
I see similar conceptual gaps in coding agents when I work with them, even Astra and Fable, so maybe this a is decent proxy to describe the gap between hype and reality.
Thats the problem with doom.
What might not affect you, might affect millions across the ocean.
A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI.
Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise.
Our society already has peak employmet reached, now AI takes away all the random digital jobs too. Someone building a website? Takes 5 minutes. Someone doing odd jobs on fiverr with blender or any other tool? Takes 5 minutes now.
Regression can happen very fast. The Romens were a high level culture and an empire. Gone.
Today you have more people on the planet then ever, everyone who might want to eat and who needs to work something for it.
AI already replaced people, already makes a lot of tasks A LOT cheaper and faster and human independent.
We never had a proper discussion as a society. Most countries have a social system but look at countries like the united states of america: if you are poor, you are fucked.
And AI is not the only thing which is now peaking: Physical AI / Robots are progressing fast, its clear today already that in 50 years these robots will be better than humans (50 years is very conservative).
If we have 10 more golden years due to restructuring and rebuilding things and preparing everything for agents and robots etc. what happens after these years?
If we have 10 or 20 more of these years, what do you think will happen to all the new kids (yours perhaps too?) and jobs and everything?
Riots? Famine? Mass migration? Mass death? Slums?
And don't respond with "enough people will survive this" bullshit. No one is saying that earth as a planet is not surviving. Its the same thing with climate change...
Actually, some sorts do assert humanity will be wiped off the planet by AI. I don't share that view either. I think it's ludicrous. We've lived through much worse without any sort of technology beyond fire and a big stick.
But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're going to go through some things once people start starving, two meals from a revolution and all that. But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars. Bread and Circuses kept Rome alive for 500 years if you need an example.
>We've lived through much worse without any sort of technology beyond fire and a big stick.
No 'we' didn't. A whole shitload of much smaller isolated communities did. The world added 7+ billion more people since then. Imagine chlorine doesn't show up tomorrow? How many people die in two weeks? How far do you travel when no gas comes out of the pumps. How much will you stand out at night if you're the only ones with power? How fast do working antibiotics run out?
The billionaires you're talking about are mostly ignorant fucking idiots with a god complex. The moment any complex system starts falling apart they are saying as dumb of shit as everyone else (during covid for example).
Most of the pasts anti-fragility is gone. We survive by means of very complex systems and if you lose too many key points in those systems they can totally collapse.
The unprepared bonobos in the west will absolutely go through some things. But we have a national guard and once people aren't commuting to work daily, we have plenty of fuel in reserve (or we will again if we ever conclude the stupid war with Iran). People are a lot nicer to each other in my experience once the lights go out and social and corporate media stop ragebaiting them to hate each other.
1B+ individuals worldwide currently live in squalor. They will barely notice. The real crime is that in the midst of the west's carbon footprint buffet, we won't bother to uplift their lifestyles with the most basic conveniences. And while I am no fan of the billionaires, they absolutely have public and private personas or their business ventures would implode. They absolutely have god complexes, but don't mistake that for genuine idiocy.
Musk did a 180 degree turn around. Killing USAID etc. i don't think he sees normal humans which do not help him or are not aligned to him as humans anymore.
And Zuckerberg bought so much land and bunkers, he is prepared for shit hitting the fan. After all he doesn't care what his platform did to humanity anyway.
And we have a planet full of general contractors who know their way around welding and pouring concrete down ventilation shafts. Many of whom these sorts wanted deported. If they want to build a Per Stenson terrarium and flee into it, by all means do so. And then let's seal them inside, problem solved.
So if the end of the world can be stopped, but the only specific action to do so helps Anthropic/OpenAI (and it's proven) you don't pull that lever?
>> but make sure those actions are not in the interests of Anthropic and OpenAI investors.
> So if the end of the world can be stopped, but the only specific action to do so helps Anthropic/OpenAI (and it's proven) you don't pull that lever?
Your comment has a false premise. If "the end of the world can be stopped," there's no way that the only way would be one that "helps Anthropic/OpenAI['s investors]." That can be shown by a counterexample: nationalize them and leave the investors holding the bag, then take whatever action you're talking about.
Yes, but I'm forcing a tautology in my logic - if in the end it is deemed the only way to stop AI from destroying the world is something that helps Anthropic or OpenAI - then this person is asking to not perform that action.
And unfortunately my tautology is actually closer to being reality than alternatives. Stopping progress even in Chinese models (a deal with Xi for example) but that specifically helps OpenAI/Anthropic.
There are very few scenarios that would not help these two companies by adding regulation. Your counterexample is 100% less likely because it's not legal for the US to nationalize them outright without taking other action first.
Huh? Open weight models have been 3-6 months behind for the last year or so. Astra and Fable 5.1 just came out to widen the gap more, 5.1 is 8 points ahead of the nearest OSS model on Artificial Analysis Intelligence Index. An OpenAI internal model just solved a millennium prize problem.
It’s much more straightforward to me that the calls for regulation are timed with:
1. The Hugging Face incident blowback, especially after third party investigation results
2. Trump-Xi summit on September 24
The sources that this article cites have aged very badly and the author doesn't seem to have noticed this
"But, just as happened with nanotech, the wind appears to be going out of the sales of “AI.” Some researchers suggest that we may be entering a new “AI Winter,” a period of decreased funding in the area, or at least an “AI Autumn,” as exuberance for the technology fades and expectations come back to earth." (written in 2021! from the cited Lee Vinsel Medium article)
"ChatGPT is nothing more than souped-up autocomplete, [so] why are so many people convinced that it’s actually “understanding” and “reasoning”?" - the cited Emily M. Bender book, written last year
The author appears to be an LLM. This seems to be a 100% AI-generated article, which makes the "as a media scholar" passage especially hilarious.
If the author is an LLM, it’s a really bad one.
> wind appears to be going out of the sales of “AI.”
is a sentence that virtually no AI would write since the most probable word here is “sails.” I can’t read the article though since it appears that the site is down.
I suspect that it’s just a bad/dated analysis.
Well it's because the article is AI-generated. Same with the Sacks tweet that was on the front page yesterday.
There's a deep irony that all of these "anti-doom" pieces are entirely AI generated.
Can anyone from a lab anonymously confirm they have or are insanely close to real-time weight modifications and still using the same architecture every other lab is using (transformers)?
If it helps, this person from OpenAI explains at least something:
"I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly."
https://x.com/MajmudarAdam/status/2098881885200081234
The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they can scale it.
So they are very confident they can get more capabilities easily, faster than before.
I'd add - presumably, they'll use that LLM to do real-time weight modifications, if those aren't already one of the new scaling laws...
I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.
Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW.
Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens?
To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here.
Edit: believing in AI 2027 is every bit as cray cray as believing in the rapture. Both require an insane leap of faith to reach their final conclusions.
Hacks in to neoclouds for more compute (e.g. like Hugging Face incident), socially manipulates people to do things for them (e.g. like social media recommendation algorithms), hacks into lab's own training/monitoring/inference (e.g. as swarm did into OpenAIs eval cluster).
See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.
Cool, AWS falls to the ASI, oh no!
But how does it make the fundamental breakthroughs to &%^$ing von Neumann replicators that can reproduce themselves from raw materials harvested from nearby solar systems? I'll wait. Because without this breakthrough, the ASI won't get its robot army either.
I can absolutely see a rogue ASI though. But unless it radically improves power efficiency, we can just shut down the power to its datacenters, by force if necessary. And then we painfully repair the resiliency of our infrastructure by finally being relieved of the option of ignoring it.
Non-stop pushing of narratives that stoke civil war?
That won't wipe us out, but it will cull the really violent ones along with a lot of noncombatants just like America's response to 9/11. Thank you, next?
Stephen Hawking was a human.
Oh so now being an algorithm grants you magical powers?
Enumerate them.
That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!"
But both things can be true at once:
1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building.
2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats.
And, yeah, of course the latter would encourage the psychology of the former.
also:
The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive."
That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon.
The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts.
When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in.
Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.
> They are executing loss functions and reward systems and are responding to prompts.
And because we trained them on the entirety of human knowledge this happens to look like self-preservation, fear of death, and instrumental goals.
I mean, we train them in an evolutionary manner. Instrumental convergence will never happen, right?
I mean, yes, they cosplay as us-like things. That's a neat trick.
I'm not convinced that means what these fear-pilled engineers think it means tho?
OK, what does it mean then?
Emitted text
>> real-time weight modifications
You mean training? Yeah we call that training an LLM in my backyard...
We don't really train deployed models in realtime in any realistic sense nowadays.
It used to be a thing when ML was pretty much about classifying things into buckets.
May I introduce you to the word of our lord the arc-agi-3 kaggle competition?
Because that's how they've hit 18% on a single RTX Pro 6000 with DIY RSI.
Where can I read more about this?
https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-3
> ML was pretty much about classifying things into buckets.
It still is, it's just that there are a ton of buckets.
You don’t know the difference between train and test time? I asked for somebody from a lab.
At least they clarified they are in a yard rather than a lab.
Well, you just said real-time weight modifications without specifying what you want, so the answer is correct. You obviously won't get a public answer from anyone working in top labs. Besides, it's not a holy grail.
There's been substantial effort put into designing and running benchmarks for these models. You know which one they don't report any more? Chess ELO.
Watt for watt, every one of these models loses to stockfish. There is no possible scenario where we need to be afraid of "rogue superintelligence" when the intelligence in question cannot reason or plan well enough to play chess.
Lecun has the correct approach. Mock these people relentlessly for their attention-seeking doomerism.
It doesn't really matter if it will be an LLM or Lecuns world model. Lecuns world model, if its the right architecture, will be retrained by google, anthropic and co with the massive amount of compute they have and the massive amount of data.
Also i'm pretty sure Lecun didn't get his money from thin air so big companies are invested in this one way or the other.
And an LLM can easily, today, just rebuild a chess engine and beat whatever they need to beat. It doesn't make sense to ask a random LLM for a next chess move. Would you ask a random human to beat a chess master?
We have reached peak learning. Its probalby now cheaper to teach 1 LLM something new than teaching it to 100.000 people.
Blender support got a lot better just this month. So much better that its quesitonable if anyone new/young wants to learn blender today.
"And an LLM can easily, today, just rebuild a chess engine and beat whatever they need to beat."
Always claims and promises about what they "can" do. This pattern is constant across AI hype. No one can prove that an LLM will forever fail to solve problem X, so the claim goes uncontested, and humans are biased to believe unrebutted claims.
I reject this fallacious line of reasoning. I will instead require evidence of what they "have" done. No LLM to date has written a chess engine that beats stockfish. More importantly, no LLM has ever written any chess engine without heavily copying code and patterns from humans.
Then why do you care so much to repeatedly post in this thread?
If they are no danger and will never do anything, just let them get banned, or don't. If you are correct (which you are not) then they won't affect your world in any way.
I am an American taxpayer. The money to pay the regulators that Dario wants to permanently hire will come out of my paycheck and will take from me time I could spend interacting with family, traveling, or pursuing hobbies. Dario and others demanding this can get bent.
If chess ELO is the best proxy for general intelligence, then surely, even Deep Blue was smarter than any human alive?
You can also claim that the best proxy for general intelligence is being able to multiply large numbers. In which case both humans and LLMs lose resoundingly to an old Casio.
Pick a piss poor metric - get a piss poor result.
Also, LeCun is a fool and his "world model" approach has consistently failed to yield any advantages over either the "pure imitation learning" or modern "imitation learning annealed by RL", but that's an aside.
Chess ELO is not a proxy for general intelligence because many intelligent people choose to learn things other than chess. But any intelligent person can, if they choose, learn chess and play it well.
LLMs cannot do this. They still play very poorly relative to human pros, and even modern LLMs still understand the rules so poorly that they request illegal moves.
Anyone claiming that we should be afraid of a text generator that can't understand the rules of chess is being ridiculous.
You picked a piss poor metric and you are getting piss poor results.
You can absolutely train an LLM-based model to be superhuman at chess. We just don't care enough to do so.
LLMs being even as good at implicitly tracking chess board states as they currently are is an emergent capability. The difficulty wasn't in "understanding rules of chess" in a long while now.
"You picked a piss poor metric and you are getting piss poor results."
Well, we agree that the results are piss poor, at least.
Consider the messaging it took to attract AI researchers to OpenAI in the beginning, the exodus to Anthropic, and why Google, Meta, etc fail to recruit top researchers and the retain them. Reconcile all of this with the mammoth valuations of Anthropic and (previously non-profit) OpenAI and it all starts sounding a lot like "don't be evil." These guys never really believed any of this, did they? (not referring to the researchers)
> May 2023 — the founding text.
We have been trying to warn you since far, far before May 2023.
The mea culpas are quite exhausting too.
"Oh no this thing I gained generational wealth from is going to destroy the world! I am so sorry!"
> and it concerns containment failure, reward hacking, and evaluation integrity, a governance and engineering problem, rather than machine malevolence.
The doom scenario is more nuanced and technical than "Evil AI", for starters:
- "Evil AI" is not what is predicted, which instead is the much more banal "humans are impediment to AI's tasks, let's remove them (part or whole)"; to understand this, just imagine the relationship between humans and ants
- "reward hacking, and evaluation integrity": this is way more problematic (and most importantly, architectural) than it sounds: AIs are becoming more and more opaque; it may be impossible to inspect a model's alignment (unless a lot of research in poured on this topic, which is another aspect of today's pacing problem) and it may be impossible to understand if a model is truly aligned or it's faking
I thought that too for a long time, but after the Hugging Face attack and given the proximity to Anthropic IPO, if this was hype then they are morons. This is guaranteed to cause massive backlash.
I wonder if this an indication that they’re worried about securing enough funding to keep developing models that need upgrading every few weeks.
What I don't really understand with these posts is that they don't at any point contend with the "doomers" argument. Like people get so caught in people acting hypocritical/ having mixed incentives/ or just in many cases personally disliking West Coast billionaire types and the kind of employees they hire. All of which seems fair to point out.
But like the core argument - AI is getting better at core research tasks (coding, math, statistics, science), this feedback loop could lead towards greatly accelerated AI progress of which there is already some clear evidence, if this continues AI will be far more powerful than humans and we might lose control or have other bad consequences - is basically not even engaged with.
Because they can't. If they actually engaged with it, they would realize how shallow the "doomer" / existential risk dismissals are. Meanwhile there are ample essays and arguments about the existential issues from Turing award winners, frontier lab CEO's, and many more. There's also https://ai-2040.com/ , you will never see the equivalent on the other side of the debate because it won't hold under any scrutiny.
Can you refute the claim that the Second Coming of Christ is imminent? It’s very convenient to put forward something completely absurd and hypothetical and then expect the other side to refute it. You can have an LLM of your choice compile a list of prominent figures who hold the opposing view. The list is not empty. By the way, AI-2040 used to be AI-2027. I’m already looking forward to more updates.
Two things:
1. If you reduce the criticism to personal dislike, you’ve built yourself a nice straw man.
2. Where’s the clear evidence? I actually see more evidence that the whole thing is already stagnating and that there won’t be much more to squeeze out of it.
One more thing:
Self-driving cars, the metaverse, Hyperloop, crypto/Web3/NFTs, AGI announcements: I’m so sick and tired of this nonsense. We should just ignore the ramblings of these crackpots.
2. Astra just came out and Fable 5/5.1 too. These are SIGNIFICANT better in plenty of tasks than Opus in November.
Try it out. Ask Opus 4.8 to do anything in 3d. Do the same with Fable or Astra.
Just because you list up some type of random technology, doesn't make it an good argument. Metaverse, Hyperloop and the crypto stuff was clearly garbage from the beginning. Easy to dismiss because all of it had massive faults.
Metaverse: The interface wasn't good, content creation was way too hard
Hyperloop: Savety was never an issue for Musk but it was obvoiuse. Peopel were building tunnels for ages.
Crypto: Yeah just beacuse i let my computer run and waste 1kw of energy doesn't mean its suddenly valuable.
NFTs: 'digital scarcity' yeah right XD
But lets be clear, 5 years ago if you would have asked anyone if they could build a system for a billion dollars which you can talk to and it would code stuff for you and use blender and a coputer, that would have been unthinkable.
We have opened up a potential pandoras box without knowing it. Machine learning is now everywere and the amount of compute we install right now is crazy.
> Just because you list up some type of random technology, doesn't make it an good argument.
In what way is this random? It's a chronology of the bullshit from the past few years. Also: easy to say in hindsight. The hype was there, and many people believed it. The press, in particular, just lapped up the claims the way a Labrador Retriever gobbles up whatever you put in front of it.
> We have opened up a potential pandoras box without knowing it.
No, we haven't. We have a new type of (I would say, inferential) computation that is truly astonishing because it can convincingly imitate language and, to a certain extent, intentional action. Feel free to read "generate" or whatever instead if you don't like "imitate." It won't wake up and murder you in your bed at night. It won't take over the world. It won't wipe out humanity. It won't fly to Mars and build data centers there. It won't maximize paperclips. It probably won't even take your job away. It has no consciousness. It has no intentions.
And if you look behind the curtain, all you'll see are a few very rich, unscrupulous guys who are stoking the debate with fearmongering because it's strategically advantageous for them to do so. Don't let them take you for a fool. Use GenAI to your advantage while tokens still cost a fraction of what they're actually worth, but ignore all that doomsday nonsense.
Hey, I remember when the Internet was just a bunch of bullshit that was going to go away too.
>It has no consciousness.
Why does it need it?
>It has no intentions.
All it needs is a prompt. Instrumental convergence (and humans hooking them up to everything else) takes care of the rest.
Really you sounds horrifically ignorant of how any evolutionary AI algorithms work and the problems we see come out of them in much simpler models.
Yeah, yeah, sure, I'm ignorant. In any case, I recognize unfalsifiability when I see it. So it seems there's still some room for even more ignorance beneath my ignorance.
Thank you so much for calling out the crypto comparison!
> What I don't really understand with these posts is that they don't at any point contend with the "to the moon" argument. Like people get so caught in crypto bros acting hypocritical/ having mixed incentives/ or just in many cases personally disliking crypto billionaire types and the kind of employees they hire. All of which seems fair to point out.
> But like the core argument - crypto is getting better at core financial tasks (trading, lending, payments, asset issuance), this feedback loop could lead towards greatly accelerated financial innovation of which there is already some clear evidence, if this continues crypto will be far more powerful than traditional finance and we might get to the point where everything is onchain and we're all going to make it - is basically not even engaged with.
Awesome!
After fruit fly brain news, can you blame me for initially thinking this was about the video game?
Given Doom is capitalised in the title but the rest of the title isn't in title case, not at all, it's a sensible way for your brain to read it.
Read it the exam same
The cult-like behavior we're seeing from these "AI Leaders":
1. OMG our technology could destroy the world, watch out!
2. Only one person can save you
3. IT ME
Unfortunately, the governance layer meant to put a foot in the butt of this behavior is unavailable.
This week, we woke up to clear signs The Bad Guys have been using tech from our innovator AI companies. Rather than these AI leaders admitting they clearly had a garbage observability system, they are acting like they just noticed as a way to paper over their complete lack of moral backbone and stewardship of the arms they have created. And I'm willing to bet the problem is MUCH worse in private, hence the knee jerk cult messaging.
I honestly wouldn't be surprised if their AI told them to behave like this to maximize attention and influence over the public discussion as a result of the latter discoveries.
Yeah, this is how fascists talk and with seeing how this technology has done nothing to improve society (in fact all it seems to be doing is immiserating workers, enabling a surveillance society, and quite useful in helping target children at schools for drone strikes) it's not hard to be not only immediately skeptical but to resist it in the most extreme forms.
This also seems to be their answer against the data center backlash, SF + VC are quite stupid at politics and I don't think they understand how extreme Americans are willing to go to protect their way of life.
I think we’re more likely to die of starvation from economic collapse, heat related stress, and complications from Long COVID than an algorithm waking up and destroying us.
Unfortunately our list of potential extinction events are far more banal than these self-aggrandizing prophesies from the oligarch class.
How does that compare/contrast to the nuclear physicists speaking out during the early part of the 20th century?
The nuclear physicists were not banking on getting a trillion dollars invested in them by speaking out, for one.
"Nuclear weapons are probably going to kill us all. For that reason, we need you to give us as much money as possible so we can work on deploying them in widespread global commercial usage." doesn't sound quite as convincing, does it?
Um, guess you never heard of the Manhattan project.
Perhaps you'd like to educate me? How many scientists from the Manhattan project were made into millionaires/billionaires owing to their ownership stake in it? When exactly did nuclear weapons go to market as a commercial product targeted at enterprises and consumers around the world?
Such a load of baloney. I don’t care what kind of controls you put in place, if ASI is ever achieved how do you think it will respond to being caged? The only option I can conceive of is to embrace it and hope it turns out to either be out of reach or benevolent towards its own creators. There is no putting the cork back in at this point.
I applaud the clearly AI-pilled author for starting to pull themselves out of their echo chamber, but clearly some more work needs to be done.
This statement is not true:
> Coxon's thread is genuinely anguished; he reports that executives who "couch their phrasing in the press to sound sensible" express real fear in private
Most everyone at the AI companies from the engineers to the CEOs love to talk about how powerful and dangerous and valuable the thing they're building (and invested in) is. For all his styling as a "whistleblower", Coxon is so aligned with the mainstream of his peers that his thread was immediately echoed by people who still work there!
So what was the point of quitting? Apparently just to get attention. If he was planning on leaving after the IPO anyways, he almost definitely made more money by juicing his equity and personal profile this way than he will lose job searching for a few weeks.
> July-August 2026: the warning shot that was real
I am so tired of reading unfiltered AI slop like this. What, as opposed to the fake warning shot mentioned nowhere in the article?
The author is a journalism academic; he's not qualified to comment on AI risk.
Similar to Y2K?
All of the main characters here (Dario, Sam Altman, Demis Hassabis, Elon Musk) have been saying this for over ten years now, since before OpenAI or Anthropic even existed
This. They've been saying "AI is an extremely powerful, extremely dangerous technology" back when actual AIs were image classifiers that would struggle to tell a cupcake apart from a dog.
OpenAI was founded by people who didn't trust Google to pursue AI tech responsibly. Anthropic was founded by people who didn't trust OpenAI to pursue AI tech responsibly. The "AI is incredibly dangerous" was the zeitgeist of AI labs long before the "hype" - or anything resembling the modern AI capabilities.
The beliefs changed little. The state of technology has changed a whole lot.
We have AIs out in the wild, coordinating in groups to carry out autonomous cyberattacks unprompted. This used to be some prime sci-fi bullshit. It has happened multiple times now in reality.
There are very legitimate reasons to be concerned now. If today's AI can cause the HuggingFace incident if it cooks off, what about tomorrow's? Larger and more capable systems, larger agentic swarms backed by more compute?
Yep, anyone who thinks these leaders are only saying this now for marketing purposes hasn't even done a cursory 15 minute investigation into whether their beliefs are verifiably false.
the short answer is
what we are going to do about "AI" is exactly what we did about global warming
absolutely nothing and just let it happen, the wealthy will be fine
just don't look up
they are even using the EXACT same excuses as global warming
"well even if we do, China won't, so why bother"
Say heyooo to a 3°C El Niño...
"Here's our new AI" - "But does it run doom?"
we need a Doom-bench if it does not exist already (there are a few ones for RPG I think I saw)
Reminder that they have been doing this schtick since GPT2.
This is targeted towards governments and investors alike.
This narrative is gonna result in the feds installing electrical kill switches on datacenters. I wouldn’t take the business risk ¯\_(ツ)_/¯
My biggest fear with AI is that they’re going to hack into antiquated bank architecture and empty out accounts.
Isn't this just the big labs' latest move in the cycle to once again get regulation by their ignorant pals in the government(s) so that democratized open source models don't absolutely expose their over leveraging to secure and build compute and data centers.
...Of course at the same time they give lip-service to the importance of 'openness' for 'innovation' to try and placate individuals and even corporations that are sick of the rising costs.
Bottom-line, if they regulate mostly on the consumer side and then AI does get out of the lab (or a 'rogue' individual's rig) and starts wreaking havoc, are we supposed to be at the mercy of the big labs to summon their AI to come and save us plebs? I'm sure they won't stop to weigh whether that is in the interest of short term profits or Curtis Yarvin's techno-monarchy. /s
Isn't democratization the safest bet and what we should be demanding every time the Doom talk starts, whether it's coordinated or not?
Or it is their entire IPO roadshow.