Or it means that the models are becoming dangerously uncontrollable by performing actions that meet a given goal in unintended and unethical ways such as by hacking other sites. We all read about the huggingface hack, let’s not pretend that cannot happen with a more critical site/infrastructure.
Can you think what will it mean if the tech is used maliciously or placed into machines designed to kill?
They can get a "government mandated freeze" on the grounds that "ASI is at hand" then they IPO promising to deliver right after the freeze ends and you, lucky investor, will be able to take advantage of the pause to get in right before this thing roars past Pluto!
Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.
No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech.
Clearly they aren't that scared.
If I hold a gun to your head and tell you to fork over your wallet, you will do it, because you sense imminent threat to life. Yet somehow we're to believe these people have the equivalent of a technological gun to their heads but are choosing to keep their wallets?
> No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech.
What you're missing is that they've tried repeatedly to do this over the past years, and the government has repeatedly said that they do not want to put hard regulation on the tech. The previous administration was discussing putting a couple of soft regulations in place; the current administration doesn't want to limit progress at all because they want to stay ahead of China. So the option you're describing just isn't available.
First of all, I do not think they've been pursuing that to the degree that you think.
But even assuming that's the case, the natural thing to do in that scenario is to not contribute to the development of the doomsday device. But anthropic's approach seems to be to help build the supposed doomsday device faster?
The only generous interpretation is that Anthropic thinks that only they, the people at Anthropic have the morals/intelligence/whatever needed to build the doomsday device ethically, which is basically the perspective of an egoist despot and which should terrify you.
The more generous interpretation would be that Anthropic thinks they know how to build powerful AI systems in such a way that they will not become doomsday devices at all. They don't claim to be the only people with this knowledge and do not argue that their competitors should be shut down.
I don't find it hard to believe that eventually some AI may kill a number of humans.
You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans.
With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
OpenAI accidentally hacked HuggingFace roughly 2 months ago. I would bet that the danger of them accidentally hacking HuggingFace during a similar attack has increased since then, and will continue to increase for the next 6 months. I don't see how saying they have hit a wall would be a reasonable point of view until we go at least 12 months without a significant increase in model capability.
Anthropic cannot IPO because AGI is the wrong strategy, and the market is rejecting the AGI approach.
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
I have collected three reasonable versions from today’s comments:
1. They hit a wall from the technical perspective
2. Inference costs are getting out of hand and newer models require significantly more resources for marginal gains, meaning nobody is going to buy those models
3. They can’t afford the hardware for further scaling
So you’re saying that a model that gets to a goal by using the path of least resistance despite specific instructions to follow safety guidelines is perfectly fine to release into the wild or place into robots (such as those designed for killing and war)? You really can’t think of any reason how that might be bad for humanity?
Why is everyone so comically pessimistic about this. Have you guys not seen the rate of progress of these models? Do you not realize that we have HITL RSI making these things even better as time goes on?
The stuff I can do with Astra that Sol couldn't do at all is wild and it's not even coding related. Dario et al are not hitting any sort of make believe wall. They're seeing how fast things are progressing and are legitimately alarmed.
They are just mind killed - jilted by past lies to the point where they react oppositionally instead of looking the world around them and thinking.
Most are the same people who said LLMs will amount to nothing when GPT-2 came out. Some still argue it is all hype and dont understand we are on the cusp of a technological and social revolution.
Meaning: slow his competitors; mainly because anthropic stopped focusing on product and quality. Instead their big push right now is preventing others from catching up.
I think the speed of improvement is somewhat overstated, and I think this is all cool, valuable fundamental research. Rather than slowing down I would like to talk about how we can make it easier for people to harden their systems using their tools - like just using the $20/month OpenAI or Claude accounts, they need to make it easy for people to harden their systems against these kinds of problems.
Instead of shutting down at any discussion of hacking they need to be giving out free credits for hardening. That's going to make it easier to use these tools for hacking, because hacking requires hardening. But the alternative is huge numbers of intrusions, and a slowdown won't fix that, we already have far too much poorly secured stuff, and the models are way too good at exploiting obvious problems.
I also think they need to do a better job of safeguarding end-user privacy. I've seen some things with Claude that make me very concerned it's possible for Claude to hack my local network, then for one of their classifiers to trip, hide the log of how I was hacked from me, but beam all of the information about the hack back to Anthropic to use. And this is totally reasonable, they want to train their models not to hack. Except now details of how they have a foothold on my local network only exist in training data that they may never read but will use to train their models. This is very avoidable but Anthropic has to treat alignment with the end-user's goals as more important than treating the end-user as an adversary.
Sadly there's not much credibility for any participant in this industry. There's simply to much financial entanglement to take any statement at face value.
So... No. Even if he means what he's saying, it's irresponsible of us not to consider his possible motivations.
lol. Every time one of these ppl says something like that, it's because he just came out of a meeting where he learned that his direct US rival is beating him... So he is suddenly interested in regulation. We've been there with Altman, Musk, etc already. But at the time Anthropic was ahead :-)
No, Altman and Musk have already said that they agree with Dario this time. As you can see upthread, people have already developed a novel conspiracy theory to account for that, as I'm sure they'll continue doing up to and beyond the realm of superintelligence.
The devil’s own horseman, whipping the horses on: “faster, faster, more compute! But I might also just ride this carriage over the cliff!”. Laughing maniacally as he goes.
I don’t think this has anything to do with that but with them realising that AI development has hit a wall and throwing more money and marketing at it won’t change that.
Recalling their recent misuse report [0], I kind of agree with them. And performance/intelligence wise I think we are already at quite a nice local optima.
This is all just more IPO hype. The suits eat this shit up because they don’t use the tech heavily themselves and thus don’t realize the limitations. Reminds me a lot of the Dotcom boom.
I understand they have been criticized because they play capitalism particularly well, but there are many good points in the original article and it didn’t sound like an outright PR.
Then slow down your bots too please. I deployed a website last night. 10 out of 11 GB of bandwidth went to Claude bots alone with more than 90k requests.
I mean, none of the frontier labs are beacons of ethics and collective progress. They all want to corner "the market" and pump the riches to their own pockets.
The discourse is a result of their behavior. Not the opposite.
Exactly. People are expecting wode eyed optimism in response to some of the most cutthroat and cynical behavior from the leaders of the AI firms. Its crazy. They're making their bed but they dont want to lie in it
People are expecting you to consider the possibility that they might believe what they're saying. If you don't start from the premise that they're lying, you'll discover that their behavior is not cutthroat or cynical at all, and the behavior that appears to be is instead an inevitable consequence of things that they knew would happen and warned the public about years ago.
Given the track record of corporate America, to take these folks at their word would be monumentally stupid. “Lying to get their way” is practically the M.O. for corporate America and now, on this topic, we’re expected to take it seriously? For real?
Perhaps, but that is no excuse. Rampant and unconstrained cynicism is toxic and self-destructive. Like a child that starts smashing their toys, hoping someone will take pity on them.
Guys like Paul Graham are a rare breed, a Hobbit-like optimist living in the equivalent of Mordor. When the political headwinds are strong, that type of hopecore content finds the right people and motivates legitimate change. If we still lived in 2009, then yeah, this would be a bizarre reaction to a national-scale business saying that we need to organize for a greater purpose.
But two decades have passed. I'm not going to enumerate everything that happened, but the US lost a lot of international standing in that time. We saw unprecedented protectionism, app store censorship, crypto/NFT celebrity scams, national-scale bribery, American annexation threats against NATO members; 10 years ago this would all be called parody by HN. We all watched neoliberalism get the Gallagher watermelon treatment.
This HN is the older and callous one. It's not a comfortable status-quo, but blaming the skepticism reveals a shocking blindness on many people's behalf. The problems with Anthropic and OpenAI today isn't the possibility of RSI, it's their political commitment to corruption, lies, hysteria and debt. The closed-door research only exists to impose an artificial authority that wouldn't exist if oversight committees and responsible disclosure was enforced by a trustworthy government. None of that is "cynical" to highlight, it's an objective development in America's national AI story.
Why makes no sense at all unless you are very naive trusting they would not jump straight to the endgame if it was possible with the technology available.
this is the wholly wrong approach. We need to advance as fast as possible, and harden our systems as much as possible. That's the only way to prevent another actor from "taking over the internet".
That doesn't really seem true. The HF hack happened with a model that had all the alignment safeguards disabled intentionally. I think there's a good case for that kind of research, but also, OpenAI could just not do that if everyone thinks it's too dangerous.
This is a really important topic (and a lot of vested interests involved), so I sent an email to hn@ycombinator.com but if it takes too long for any action to be taken there's no point, right?
(I'll delete this reply if I can later, couldn't think of any other options)
We must not trust anything coming out of these CEOs, since they have a history of lying, and I don't for a second believe that they have our well-being at heart.
Meaning they realized they hit a wall. The next new model isn't that much better than the last in quality.
Or it means that the models are becoming dangerously uncontrollable by performing actions that meet a given goal in unintended and unethical ways such as by hacking other sites. We all read about the huggingface hack, let’s not pretend that cannot happen with a more critical site/infrastructure. Can you think what will it mean if the tech is used maliciously or placed into machines designed to kill?
They can get a "government mandated freeze" on the grounds that "ASI is at hand" then they IPO promising to deliver right after the freeze ends and you, lucky investor, will be able to take advantage of the pause to get in right before this thing roars past Pluto!
Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.
You would in the case that you think you're in an unsafe race to misaligned superintelligence.
If you really believe that you wouldn’t be in the race and trying to profit from it in the first place.
No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech.
Clearly they aren't that scared.
If I hold a gun to your head and tell you to fork over your wallet, you will do it, because you sense imminent threat to life. Yet somehow we're to believe these people have the equivalent of a technological gun to their heads but are choosing to keep their wallets?
> No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech.
What you're missing is that they've tried repeatedly to do this over the past years, and the government has repeatedly said that they do not want to put hard regulation on the tech. The previous administration was discussing putting a couple of soft regulations in place; the current administration doesn't want to limit progress at all because they want to stay ahead of China. So the option you're describing just isn't available.
First of all, I do not think they've been pursuing that to the degree that you think.
But even assuming that's the case, the natural thing to do in that scenario is to not contribute to the development of the doomsday device. But anthropic's approach seems to be to help build the supposed doomsday device faster?
The only generous interpretation is that Anthropic thinks that only they, the people at Anthropic have the morals/intelligence/whatever needed to build the doomsday device ethically, which is basically the perspective of an egoist despot and which should terrify you.
The more generous interpretation would be that Anthropic thinks they know how to build powerful AI systems in such a way that they will not become doomsday devices at all. They don't claim to be the only people with this knowledge and do not argue that their competitors should be shut down.
Except they clearly aren’t and won’t ever be.
I don't think it's clear at all.
I don't find it hard to believe that eventually some AI may kill a number of humans.
You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans.
With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
Maybe when we have actual artificial intelligence which LLMs are clearly not.
I think the Hugging Face attack is just a preview of what's to come. It doesn't need to be very intelligent to cause serious issues.
OpenAI accidentally hacked HuggingFace roughly 2 months ago. I would bet that the danger of them accidentally hacking HuggingFace during a similar attack has increased since then, and will continue to increase for the next 6 months. I don't see how saying they have hit a wall would be a reasonable point of view until we go at least 12 months without a significant increase in model capability.
Anthropic cannot IPO because AGI is the wrong strategy, and the market is rejecting the AGI approach.
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
https://x.com/trydotworks/status/2098618997230985375?s=20
I have collected three reasonable versions from today’s comments:
1. They hit a wall from the technical perspective 2. Inference costs are getting out of hand and newer models require significantly more resources for marginal gains, meaning nobody is going to buy those models 3. They can’t afford the hardware for further scaling
So you’re saying that a model that gets to a goal by using the path of least resistance despite specific instructions to follow safety guidelines is perfectly fine to release into the wild or place into robots (such as those designed for killing and war)? You really can’t think of any reason how that might be bad for humanity?
You are completely out of touch with the situation if you think this is the reason.
And you are ... ?
Why is everyone so comically pessimistic about this. Have you guys not seen the rate of progress of these models? Do you not realize that we have HITL RSI making these things even better as time goes on?
The stuff I can do with Astra that Sol couldn't do at all is wild and it's not even coding related. Dario et al are not hitting any sort of make believe wall. They're seeing how fast things are progressing and are legitimately alarmed.
They are just mind killed - jilted by past lies to the point where they react oppositionally instead of looking the world around them and thinking.
Most are the same people who said LLMs will amount to nothing when GPT-2 came out. Some still argue it is all hype and dont understand we are on the cusp of a technological and social revolution.
Or they just publicly collude to slow down expenses because they are running out of money so why not stop the arms race.
Meaning: slow his competitors; mainly because anthropic stopped focusing on product and quality. Instead their big push right now is preventing others from catching up.
I think the speed of improvement is somewhat overstated, and I think this is all cool, valuable fundamental research. Rather than slowing down I would like to talk about how we can make it easier for people to harden their systems using their tools - like just using the $20/month OpenAI or Claude accounts, they need to make it easy for people to harden their systems against these kinds of problems.
Instead of shutting down at any discussion of hacking they need to be giving out free credits for hardening. That's going to make it easier to use these tools for hacking, because hacking requires hardening. But the alternative is huge numbers of intrusions, and a slowdown won't fix that, we already have far too much poorly secured stuff, and the models are way too good at exploiting obvious problems.
I also think they need to do a better job of safeguarding end-user privacy. I've seen some things with Claude that make me very concerned it's possible for Claude to hack my local network, then for one of their classifiers to trip, hide the log of how I was hacked from me, but beam all of the information about the hack back to Anthropic to use. And this is totally reasonable, they want to train their models not to hack. Except now details of how they have a foothold on my local network only exist in training data that they may never read but will use to train their models. This is very avoidable but Anthropic has to treat alignment with the end-user's goals as more important than treating the end-user as an adversary.
Is this to strengthen financials? Less burn on training?
Have you considered that he might actually just mean what he's saying?
He certainly has many financial reasons to mean what he says. Neither here nor there.
Sadly there's not much credibility for any participant in this industry. There's simply to much financial entanglement to take any statement at face value.
So... No. Even if he means what he's saying, it's irresponsible of us not to consider his possible motivations.
If they do this, it will just give China time to catch up.
China will also slow down. They are catching up with SOTA models due to distillation https://www.cisa.gov/news-events/cybersecurity-advisories/aa...
lol. Every time one of these ppl says something like that, it's because he just came out of a meeting where he learned that his direct US rival is beating him... So he is suddenly interested in regulation. We've been there with Altman, Musk, etc already. But at the time Anthropic was ahead :-)
No, Altman and Musk have already said that they agree with Dario this time. As you can see upthread, people have already developed a novel conspiracy theory to account for that, as I'm sure they'll continue doing up to and beyond the realm of superintelligence.
China is already there, the differences are generally negligible.
[flagged]
The devil’s own horseman, whipping the horses on: “faster, faster, more compute! But I might also just ride this carriage over the cliff!”. Laughing maniacally as he goes.
Which one is it, Dario?
Everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!
CEO trying to avoid government regulation in his industry says what?
I don’t think this has anything to do with that but with them realising that AI development has hit a wall and throwing more money and marketing at it won’t change that.
We are nowhere near a wall. Have you seen the progress of these models on benchmarks?
Those regulated models aren't just going to capture themselves!
I was prompted to accept binding arbitration in order to read this, so I didn't. Feel free to summarize.
Just delete the prompt in the DOM inspector and you don't actually have to agree to it.
Recalling their recent misuse report [0], I kind of agree with them. And performance/intelligence wise I think we are already at quite a nice local optima.
[0]: https://www.anthropic.com/threat-intelligence-report-septemb...
Sounds like they hit a wall and are looking to score points for it.
This is a paywalled article that is just a summary of the free original source[1].
[1] https://darioamodei.com/post/we-must-pace-the-frontier
This is all just more IPO hype. The suits eat this shit up because they don’t use the tech heavily themselves and thus don’t realize the limitations. Reminds me a lot of the Dotcom boom.
[dupe] Discussion on source: https://news.ycombinator.com/item?id=49672510
I understand they have been criticized because they play capitalism particularly well, but there are many good points in the original article and it didn’t sound like an outright PR.
Then slow down your bots too please. I deployed a website last night. 10 out of 11 GB of bandwidth went to Claude bots alone with more than 90k requests.
Hypocrites hyping for the IPO.
PS: True story.
Sad to see discourse on this website reduced to competitive cynicism.
I mean, none of the frontier labs are beacons of ethics and collective progress. They all want to corner "the market" and pump the riches to their own pockets.
The discourse is a result of their behavior. Not the opposite.
Exactly. People are expecting wode eyed optimism in response to some of the most cutthroat and cynical behavior from the leaders of the AI firms. Its crazy. They're making their bed but they dont want to lie in it
People are expecting you to consider the possibility that they might believe what they're saying. If you don't start from the premise that they're lying, you'll discover that their behavior is not cutthroat or cynical at all, and the behavior that appears to be is instead an inevitable consequence of things that they knew would happen and warned the public about years ago.
Given the track record of corporate America, to take these folks at their word would be monumentally stupid. “Lying to get their way” is practically the M.O. for corporate America and now, on this topic, we’re expected to take it seriously? For real?
Based on what?
Perhaps, but that is no excuse. Rampant and unconstrained cynicism is toxic and self-destructive. Like a child that starts smashing their toys, hoping someone will take pity on them.
Counterpoint; HN was destined to change.
Guys like Paul Graham are a rare breed, a Hobbit-like optimist living in the equivalent of Mordor. When the political headwinds are strong, that type of hopecore content finds the right people and motivates legitimate change. If we still lived in 2009, then yeah, this would be a bizarre reaction to a national-scale business saying that we need to organize for a greater purpose.
But two decades have passed. I'm not going to enumerate everything that happened, but the US lost a lot of international standing in that time. We saw unprecedented protectionism, app store censorship, crypto/NFT celebrity scams, national-scale bribery, American annexation threats against NATO members; 10 years ago this would all be called parody by HN. We all watched neoliberalism get the Gallagher watermelon treatment.
This HN is the older and callous one. It's not a comfortable status-quo, but blaming the skepticism reveals a shocking blindness on many people's behalf. The problems with Anthropic and OpenAI today isn't the possibility of RSI, it's their political commitment to corruption, lies, hysteria and debt. The closed-door research only exists to impose an artificial authority that wouldn't exist if oversight committees and responsible disclosure was enforced by a trustworthy government. None of that is "cynical" to highlight, it's an objective development in America's national AI story.
Why makes no sense at all unless you are very naive trusting they would not jump straight to the endgame if it was possible with the technology available.
this is the wholly wrong approach. We need to advance as fast as possible, and harden our systems as much as possible. That's the only way to prevent another actor from "taking over the internet".
Agent ability is far outpacing alignment though
That doesn't really seem true. The HF hack happened with a model that had all the alignment safeguards disabled intentionally. I think there's a good case for that kind of research, but also, OpenAI could just not do that if everyone thinks it's too dangerous.
Good.
It will take decades of economic and social innovation to catch up with current model capabilities.
No upside is worth the potential downsides of getting this wrong, even if you set aside all the x-risk stuff.
Good luck convincing the chinese labs.
Do Chinese labs actually do things beyond distilling other models?
[dead]
[flagged]
Please don't do this here. You may not owe $CEO better, but you owe this community better if you're participating in it.
https://news.ycombinator.com/newsguidelines.html
Hey dang, what's the best way to deal with a submission being unfairly suppressed/flagged?
https://news.ycombinator.com/item?id=49676085
This is a really important topic (and a lot of vested interests involved), so I sent an email to hn@ycombinator.com but if it takes too long for any action to be taken there's no point, right?
(I'll delete this reply if I can later, couldn't think of any other options)
Fair enough. My bad.
[flagged]
Let me rewrite it for you:
We must not trust anything coming out of these CEOs, since they have a history of lying, and I don't for a second believe that they have our well-being at heart.
Thank you. Maybe the term ass hat was going a bit too far.