"AI Safety" is too broad to be a corporate plot, but there are some demands that people make in the name of AI safety that are obviously in the interest of frontier labs, like for example for NVidia to be banned from selling to China.
Really well put. I hope this article can finally put to bed this very frustrating dismissal.
Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb.
Why should we trust AI company leaders? Sam Altman was quoted as saying, "We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency." Does that sound like someone who considers safety a high priority?
There is a straight forward reason here. Almost everyone, dare I say the 99%, look around and see blatant corporate-government corruption. They see an oligarchy using their wealth and power to get more of the pie each day, while escaping consequences when the misdeed.
Why should we give deference to the newest corpo kids on the block? They go hang out with Trump at what remains of the White House. Do you really think they are different from the rest of the lot?
You too see the corruption across the corporations and government that spans party lines, yea?
What are we the people going to do to remedy the situation we find ourselves in? (because they are not going to)
You're literally taking the people whom you identify as being in a giant conspiracy and saying their not-actually-human product trained to mimic their own ideas of human behaviour is safe, while disregarding all the people saying "this is not safe, we should do something about that".
This is the opposite of having faith in humanity.
I have (some) faith in humanity, that's why my p(doom) is only 10%, rather than the 95% it would be if I simply looked at, say, Elon Musk making a bot which called itself "MechaHitler" then being defended by the US executive branch when EU nations are cross about Grok making CSAM (or how slow we're being at doing anything about greenhouse gasses despite knowing about the problem for over a century), and in disregarded all the positive efforts and progress towards non-bad outcomes.
10% is still really bad.
Thinking of near-death experiences, I'm reminded of a religious joke: Man caught in a flood, waits in the branches of a tree. A dinghy comes by, he turns them away saying "god will save me". Then a motorboat, he turns them away saying "god will save me, find others who need help". Then a helicopter, same. He dies, goes to heaven, and asks god "Why didn't you save me?", to which god replies "What are you talking about, I sent two boats and a helicopter".
You want faith in humanity? P(doom)ers want to make the number smaller. Help them make the number go down, rather than denying there's something to be fixed.
Those people are bat shit crazy, some random blog is not going to change my mind on that. There are Ai experts who disagree with them, who are not in the big lab friend circle / echo chamber. There are security experts who are not concerned so much that they are telling everyone humanity is doomed if we don't... give the labs complete control over who can use Ai and for what... do you want to speedrun 1984?
Sam Altman has a history, Silicon Valley has a history, the corporate elites and oligarchs have a history. Why should I believe this instance is free of that history?
Perhaps Dario is advocating for things that would amount to a ban on open weights out of fear, that does not change the effect, and I do not believe the more savvy investors do not see the benefit for that regulatory control. Open weights (real competition, not tech cartels) will undermine their financial projections. There is no way I'm going to believe the people who have been doing this for decades are not doing it here too, especially with the most corrupt administration this country has had in my life.
2. Opening with “these people are batshit” is a pretty undeniable sign that you have no coherent theory of mind for them.
3. 1984 wasn’t about regulation of superintelligence. Governments are the only ones who can say “who can use [fissile material], and for what” — has that created 1984? It started 30 years before that, so it should’ve created 1984 by 1984!
4. No one’s banning “open weights”, they’re banning superintelligence development without any of the protections we find to be obviously essential for far, far less dangerous things. Perhaps they would’ve been open to something more drastic, but post GLM-5.3 that’s kinda moot. The cat’s out of the bag.
5. Investors don’t control Anthropic, and their control over OpenAI is complex and highly unusual to say the least.
6. “Some capitalists have done X therefor all capitalists force all companies they invest in to do X” is just not sound logic.
7. Finally: that isn’t what regulatory capture looks like! It looks like revolving doors and packing boards and shady policies, not publicly campaigning for new statutes.
Scott is not an expert in AI or AI company politics. At best he can be described as a peculiar kind of cultural commentator or perhaps a political pundit.
2. I do not need a theory of mind to perceive (some of) the things they say are disconnected from reality. It also demeans our human abilities to handle these things. I'm not interested to live in a Nanny State.
3. it's about the ministry of truth, freedom of speech, Ai is closer to speech than nuclear science, which is a misleading analogy that relies on pre-supposing the Ai is super dangerous and inevitably wipe humans out.
4. there is nothing obviously essential other than humans should be staying in the loop more than they are
5. they do not need to control if they are in agreement, but that is besides the point
6. it's the same capitalists in many instances, and the patterns and trends are quite clear, especially with the biggest companies in the economy
7. Dario and Sam go to meet with the president, face to face, on a regular basis.
8. Sounds like you have too much attachment to these companies and their products to actually consider criticism in a fair light
9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
Many of your arguments rely on predeterminism, no one knows how this will play out.
I believe the best way forward is with more transparency and reproducibility, rather than gatekeepers from the oligarchy and government. We also need to put methods for proving real hummism, without sacrificing privacy, at the front of the queue.
> 9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
"Might" do.
Wanna risk it?
Even with the best will in the world, even with the best training, even with every advantage you could hope for, this is software, and the software is buggy.
We've already got loads of examples, including in the blog's comments, of it doing things other than what the user wants.
Whose speech is 3? The AI has to be a legal person in its own right if it is the AI's speech.
4. Good luck with that. The point of AI is automation and not being in the loop, which is why people were sleeping in their Tesla's while they drove as soon as it was possible, safe or not.
5. See blog
6. Name literally one company outside AI who can be used to justify the claim "safety is a plot". One.
Keeping AI in the test set just to emphasise the comparison: outside AI, I can think of literally zero examples where a company actively states their products are a danger to their customers, users, and the general public, while being sued for damages caused by their products, with no understanding of how to make them safe, as a sales pitch.
Even weapons manufacturers don't do that.
Conversely, look at NVIDIA: they don't like the safety discussion and would like everyone saying this is unsafe to stop talking.
And finally:
8. That's an incoherent position. Everyone concerned about AI safety is critical of these companies. Calling the companies' products "unsafe" is itself criticism.
I'm not against regulation, there are many places where it is needed. I agree that these things can be dangerous, but not existentially, nor today.
What I am against is oligarchy and government gatekeepers
Why is p(doom) >> p(hope)? Why is there a presupposition that bad is the guaranteed outcome?
Let's take a deep breath as a society and have a rational discussion based in fact instead of hyperbole and unprovable predictions.
8. I was alluding to cognitive biases, to the other commenter. I would ask the same of you, reflect on what cognitive biases might be at play in your own interpretations. Why are you taking the side of these companies and people with such fervor?
There is more than one kind of corporate plot though. The article defends against the kind where some people inside a cooperation have some idea to make more money with unsavory methods, in an imagined backroom I suppose, and then use corporate resources to pursue that plan to achieve their goals.
In this case, we have a bunch of people who believe something. Some of them get jobs in that field. They notice their belief could be beneficial to their company (and make them more money), and so using corporate resources to amplify this viewpoint is well justified spend to help them achieve their goals.
In my view, there is little difference in the outcome between the two and probably the line is very blurry... do people in the 2nd case sometimes say the quiet part outloud? Do people in the first often believe they are only doing what is right and rational without finances considered?
Often these posts come straight from the top of AI companies with specific policy goals in mind. What good will come from understanding Amodei / Altman's "true" motives anyway?
I am getting some downvotes, so let me clear up a little:
Amodei and Altman are both openly advocating for increased regulation, and I do question their motives, because both of them are running companies with a ton of debt, and facing some very real competition from open models.
What I think they want is for politicians to ensure their companies' survival via some form of regulatory capture; making it illegal to use other models; enforce their twisted version of copyright; etc. Lacking enough political power is existential to them since they run such high costs.
Are they out there amplifying the viewpoint that what they are doing is dangerous and must be regulated?
Yes.
Would they do that if it was actually bad for their business?
NO - the CEOs specifically would virtually not be allowed to ask for regulation if they thought it was bad for investors for them to do that.
So what we have is AI companies openly saying they are doing dangerous work and must be regulated; and I infer they want that to protect them and not us.
FWIW, I think AI will just about obsolete us all and that is bad for almost everyone, but I still think it is transparently driven by profit and corporate agendas when the top leaders of these companies encourage regulation - they do not expect it to work out badly for them.
1. Understanding motives is important. The fact that you’re questioning that should be a red flag in hindsight.
2. It seems your rebuttal to the article is “well maybe it’s a plot but just not one they’ve ever talked to each other about, but just happen to all secretly believe in the efficacy of”. Which, obviously: ???
3. We know who wrote this post, and it’s not from “the top of AI companies”. It’s from someone desperately trying to prevent the catastrophe that you’ll be up in arms about in 3-6 months.
I'm just laying reasons for thinking this does not mean there is no corporate plot to promote high p(doom) - it has no bearing on how much I agree with the people promoting the idea. I am providing a framework for why this post might fail to convince people... the narrative can still be reinforced via paid messaging which constitutes some kind of plot in my book.
"AI Safety" is too broad to be a corporate plot, but there are some demands that people make in the name of AI safety that are obviously in the interest of frontier labs, like for example for NVidia to be banned from selling to China.
Really well put. I hope this article can finally put to bed this very frustrating dismissal.
Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb.
Why should we trust AI company leaders? Sam Altman was quoted as saying, "We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency." Does that sound like someone who considers safety a high priority?
> I hope this article can finally put to bed this very frustrating dismissal.
Sadly, it won't.
I've made these points separately in HN threads to no apparent impact.
There is a straight forward reason here. Almost everyone, dare I say the 99%, look around and see blatant corporate-government corruption. They see an oligarchy using their wealth and power to get more of the pie each day, while escaping consequences when the misdeed.
Why should we give deference to the newest corpo kids on the block? They go hang out with Trump at what remains of the White House. Do you really think they are different from the rest of the lot?
You too see the corruption across the corporations and government that spans party lines, yea?
What are we the people going to do to remedy the situation we find ourselves in? (because they are not going to)
Take everything you've said at face value, and tell me that makes this not dangerous.
Also, if they're all in bed with Trump, why does he not repeat the "it is dangerous" talking point? Why did his admin blacklist Anthropic?
These people can all be corrupt evil cartoon villains, but that should make you see more risk, not less.
If you like, you can go through my comment history to find links to resources with more rational takes.
Otherwise, I will be publishing some things within the next few months that I'm working on with some other people
In the meantime, it can help to walk back from the edge and the doomers, have some faith in humanity.
I recently has a near death experience, and since, one of the things I'm changing is to see the glass half full
> have some faith in humanity.
You're literally taking the people whom you identify as being in a giant conspiracy and saying their not-actually-human product trained to mimic their own ideas of human behaviour is safe, while disregarding all the people saying "this is not safe, we should do something about that".
This is the opposite of having faith in humanity.
I have (some) faith in humanity, that's why my p(doom) is only 10%, rather than the 95% it would be if I simply looked at, say, Elon Musk making a bot which called itself "MechaHitler" then being defended by the US executive branch when EU nations are cross about Grok making CSAM (or how slow we're being at doing anything about greenhouse gasses despite knowing about the problem for over a century), and in disregarded all the positive efforts and progress towards non-bad outcomes.
10% is still really bad.
Thinking of near-death experiences, I'm reminded of a religious joke: Man caught in a flood, waits in the branches of a tree. A dinghy comes by, he turns them away saying "god will save me". Then a motorboat, he turns them away saying "god will save me, find others who need help". Then a helicopter, same. He dies, goes to heaven, and asks god "Why didn't you save me?", to which god replies "What are you talking about, I sent two boats and a helicopter".
You want faith in humanity? P(doom)ers want to make the number smaller. Help them make the number go down, rather than denying there's something to be fixed.
Those people are bat shit crazy, some random blog is not going to change my mind on that. There are Ai experts who disagree with them, who are not in the big lab friend circle / echo chamber. There are security experts who are not concerned so much that they are telling everyone humanity is doomed if we don't... give the labs complete control over who can use Ai and for what... do you want to speedrun 1984?
Sam Altman has a history, Silicon Valley has a history, the corporate elites and oligarchs have a history. Why should I believe this instance is free of that history?
Perhaps Dario is advocating for things that would amount to a ban on open weights out of fear, that does not change the effect, and I do not believe the more savvy investors do not see the benefit for that regulatory control. Open weights (real competition, not tech cartels) will undermine their financial projections. There is no way I'm going to believe the people who have been doing this for decades are not doing it here too, especially with the most corrupt administration this country has had in my life.
1. Not a “random blog”, not by a long shot.
2. Opening with “these people are batshit” is a pretty undeniable sign that you have no coherent theory of mind for them.
3. 1984 wasn’t about regulation of superintelligence. Governments are the only ones who can say “who can use [fissile material], and for what” — has that created 1984? It started 30 years before that, so it should’ve created 1984 by 1984!
4. No one’s banning “open weights”, they’re banning superintelligence development without any of the protections we find to be obviously essential for far, far less dangerous things. Perhaps they would’ve been open to something more drastic, but post GLM-5.3 that’s kinda moot. The cat’s out of the bag.
5. Investors don’t control Anthropic, and their control over OpenAI is complex and highly unusual to say the least.
6. “Some capitalists have done X therefor all capitalists force all companies they invest in to do X” is just not sound logic.
7. Finally: that isn’t what regulatory capture looks like! It looks like revolving doors and packing boards and shady policies, not publicly campaigning for new statutes.
> Not a “random blog”, not by a long shot.
Scott is not an expert in AI or AI company politics. At best he can be described as a peculiar kind of cultural commentator or perhaps a political pundit.
1. see peer comment
2. I do not need a theory of mind to perceive (some of) the things they say are disconnected from reality. It also demeans our human abilities to handle these things. I'm not interested to live in a Nanny State.
3. it's about the ministry of truth, freedom of speech, Ai is closer to speech than nuclear science, which is a misleading analogy that relies on pre-supposing the Ai is super dangerous and inevitably wipe humans out.
4. there is nothing obviously essential other than humans should be staying in the loop more than they are
5. they do not need to control if they are in agreement, but that is besides the point
6. it's the same capitalists in many instances, and the patterns and trends are quite clear, especially with the biggest companies in the economy
7. Dario and Sam go to meet with the president, face to face, on a regular basis.
8. Sounds like you have too much attachment to these companies and their products to actually consider criticism in a fair light
9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
Many of your arguments rely on predeterminism, no one knows how this will play out.
I believe the best way forward is with more transparency and reproducibility, rather than gatekeepers from the oligarchy and government. We also need to put methods for proving real hummism, without sacrificing privacy, at the front of the queue.
> 9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
"Might" do.
Wanna risk it?
Even with the best will in the world, even with the best training, even with every advantage you could hope for, this is software, and the software is buggy.
We've already got loads of examples, including in the blog's comments, of it doing things other than what the user wants.
Those examples also prove your reply to 2 false.
1. Spot the name: https://en.wikipedia.org/wiki/Template:Existential_risk_from...
Whose speech is 3? The AI has to be a legal person in its own right if it is the AI's speech.
4. Good luck with that. The point of AI is automation and not being in the loop, which is why people were sleeping in their Tesla's while they drove as soon as it was possible, safe or not.
5. See blog
6. Name literally one company outside AI who can be used to justify the claim "safety is a plot". One.
Keeping AI in the test set just to emphasise the comparison: outside AI, I can think of literally zero examples where a company actively states their products are a danger to their customers, users, and the general public, while being sued for damages caused by their products, with no understanding of how to make them safe, as a sales pitch.
Even weapons manufacturers don't do that.
Conversely, look at NVIDIA: they don't like the safety discussion and would like everyone saying this is unsafe to stop talking.
And finally:
8. That's an incoherent position. Everyone concerned about AI safety is critical of these companies. Calling the companies' products "unsafe" is itself criticism.
I'm not against regulation, there are many places where it is needed. I agree that these things can be dangerous, but not existentially, nor today.
What I am against is oligarchy and government gatekeepers
Why is p(doom) >> p(hope)? Why is there a presupposition that bad is the guaranteed outcome?
Let's take a deep breath as a society and have a rational discussion based in fact instead of hyperbole and unprovable predictions.
8. I was alluding to cognitive biases, to the other commenter. I would ask the same of you, reflect on what cognitive biases might be at play in your own interpretations. Why are you taking the side of these companies and people with such fervor?
There is more than one kind of corporate plot though. The article defends against the kind where some people inside a cooperation have some idea to make more money with unsavory methods, in an imagined backroom I suppose, and then use corporate resources to pursue that plan to achieve their goals.
In this case, we have a bunch of people who believe something. Some of them get jobs in that field. They notice their belief could be beneficial to their company (and make them more money), and so using corporate resources to amplify this viewpoint is well justified spend to help them achieve their goals.
In my view, there is little difference in the outcome between the two and probably the line is very blurry... do people in the 2nd case sometimes say the quiet part outloud? Do people in the first often believe they are only doing what is right and rational without finances considered?
Often these posts come straight from the top of AI companies with specific policy goals in mind. What good will come from understanding Amodei / Altman's "true" motives anyway?
I am getting some downvotes, so let me clear up a little:
Amodei and Altman are both openly advocating for increased regulation, and I do question their motives, because both of them are running companies with a ton of debt, and facing some very real competition from open models.
What I think they want is for politicians to ensure their companies' survival via some form of regulatory capture; making it illegal to use other models; enforce their twisted version of copyright; etc. Lacking enough political power is existential to them since they run such high costs.
Are they out there amplifying the viewpoint that what they are doing is dangerous and must be regulated? Yes.
Would they do that if it was actually bad for their business? NO - the CEOs specifically would virtually not be allowed to ask for regulation if they thought it was bad for investors for them to do that.
So what we have is AI companies openly saying they are doing dangerous work and must be regulated; and I infer they want that to protect them and not us.
FWIW, I think AI will just about obsolete us all and that is bad for almost everyone, but I still think it is transparently driven by profit and corporate agendas when the top leaders of these companies encourage regulation - they do not expect it to work out badly for them.
1. Understanding motives is important. The fact that you’re questioning that should be a red flag in hindsight.
2. It seems your rebuttal to the article is “well maybe it’s a plot but just not one they’ve ever talked to each other about, but just happen to all secretly believe in the efficacy of”. Which, obviously: ???
3. We know who wrote this post, and it’s not from “the top of AI companies”. It’s from someone desperately trying to prevent the catastrophe that you’ll be up in arms about in 3-6 months.
I'm just laying reasons for thinking this does not mean there is no corporate plot to promote high p(doom) - it has no bearing on how much I agree with the people promoting the idea. I am providing a framework for why this post might fail to convince people... the narrative can still be reinforced via paid messaging which constitutes some kind of plot in my book.