> Furthermore, export controls, for which Dario Amodei openly advocates, are already prompting China to build its own chips. China is, of course, behind the West, but not as much as you may think. Instead of adopting a Western technology stack and relying on our technology to build its AI models, China will increasingly rely on its own chips, reducing Western leverage.
China has been working hard to indigenize AI chip making since well before the October 2022 export controls. I don't think the controls had zero effect, but overall I think the effect is often very overstated, and anyway it's irreversible (there's no way China will stop its self-sufficiency efforts even if the AI chip ban is lifted; the CCP will always ensure there's a large enough domestic market carved out for every Huawei Ascend produced).
I used to think that export controls didn't work, until I realized that China has leapfrogged the West on every technology that wasn't subject to export controls.
To be brutally frank, this was always going to happen when Western countries outsourced their manufacturing.
And all of the rationales were self serving garbage, it was about juicing the next quarters numbers.
Fundamentally though, it's the removal of capital controls that has driven a lot of this behaviour, and it's amusing to me that reintroducing them is so far beyond the Overton window that it never even comes up.
> Anthropic already possesses the compliance, legal, and evaluation infrastructure needed to satisfy such a regime. A university lab, startup, or community does not, even if they somehow manage to train a sufficiently capable model.
All the actual proposals that Dario Amodei has endorsed with regard to safety testing and so on have exempted smaller models. I don't think university labs or startups would ever be hit by any such regulation.
Have exempted smaller models so far. Seems like they're trying to soft-launch then tighten the ratchet, and I'd be willing to accept that this is a cynical view if this hasn't been the modus operandi for a while now. See EA/Sony/etc. continually trying to find a way to get their foot in the "always online, you're just renting the game" door.
I think he probably didn’t go far enough to be honest, though I acknowledge that’s difficult to do in his position and with the amount of attacks being thrown at Anthropic at the moment.
At some point open weight models will need to be restricted in some manner, likely multiple manners. We have rules about everything, you need a license to drive a car, your car must be roadworthy, your car can’t have certain modifications, you can’t drive over certain speeds, you can’t drive in certain areas at all, you have to wear a seatbelt etc. That’s one analogy amongst many, with the point being that regulation is a general public good that really does make the world a safer and better place.
Nobody should care about smaller local models that don’t have significant capabilities enough to be dangerous, and people should have the right to run them locally, privately and do whatever they do with them.
I don’t think that an AI monopoly or duopoly should be allowed to exist either, and I think it is possible to reach a reasonable set of compromises on each of these points, if the will exists and people can be convinced to be reasonable with one another. I think the will may not exist, and that reason is lacking in our discussions of reasoning machines, but I would like to be wrong about that.
The only widely publicized AI orchestrated cyberattack in recent memory came from a closed-weight model, and the only widely publicized defense from said attack came from an open-weight Chinese model. What do you make of that?
> I don’t think that an AI monopoly or duopoly should be allowed to exist either
Monopolies have always been allowed to exist, even welcomed. People have gotten confused, but there is a big difference between a monopoly and an illegal monopoly. Deciding what should and shouldn’t be allowed to exist as determined by regulatory bodies and not consumers goes against democratic principles (though it is sometimes necessary, such as with drugs).
I don’t think I want to live in a world where police decides what brand of chips I can eat on the sole basis of the chips having gotten too popular. Do you?
I don’t make anything of it. That’s only been reported because it came from OpenAI and was a frontier model, incidents involving open models are probably happening but are not being widely reported, and those are happening with less capable models, so they aren’t successful enough yet to make headlines. I draw that inference from the amount of AI bot traffic which is now saturating the internet, AI driven attacks almost certainly have increased in like manner.
> Monopolies have always been allowed to exist
Antitrust law also exists, but is weaker than it should be. The customer doesn’t win from the existence of monopolies, they go against one of the basic premises of capitalism. Assuming you believe capitalism is a good thing which is its own conversation.
> I don’t think I want to live in a world where police decides what brand of chips I can eat on the sole basis of the chips having gotten too popular.
I really don’t know how you ended up with this as an example, did anyone say this, or anything close to it?
Same thing that makes any LLM dangerous. The intent and actions of the person using the LLM to do bad things. Any tool can be dangerous in the hands of the wrong person...
As the sibling comment says, user actions are the main driver, but also the fact that the weights can be modified. If a model is released with safeguards, they can be ablated out or the model can be fine tuned for nefarious purposes, and nobody can really stop that from happening.
That’s not to say that the majority of open weight models are being used in nefarious ways or are dangerous, but unfortunately despite (hopefully) most people being responsible, we have to guard against the small number of bad actors. Like with everything, a small number of people ruin it for everyone.
Nothing makes an LLM dangerous. An LLM remixes existing knowledge and outputs text. Any danger comes from what one subsequently does with the text.
For example, when a lab like OpenAI negligently wires up a harness to an LLM and executes arbitrary code in YOLO mode without common-sense isolation measures, the executed code can do bad things.
I tend to agree with this also. The idea that the impact of AI is being decided by private companies with no control either by governments or the public doesn’t sit well with me. You need the most regulation where most of the danger is, and most of the danger as you say is with the big AI companies.
But there are some moves towards regulating them now.
Fundamentally, we needed to decide, as a society, that should someone (anyone) have access to neural networks that can launch successful cyberattacks by discovering new zero-days and/or leveraging knowledge about recent zero-days and applying it in an adversarial way.
We de-facto settled on the answer being yes. These foundation models companies have access to such systems.
However, because the answer to the first question is yes, now we need to decide should everyone else be able to defend themselves with neural networks that are perhaps equally capable but are not sold by these foundation model companies as a product.
I think the answer to the second question has to be a yes, IMHO.
Yes it’s true. But public can’t also cry about safety but simultaneously cheer for open weight models. Whatever safety concerns the public has against ChatGPT and Claude are drastically higher in open weight models. What is the right thing to do then? Accept that safety was always a farce? I’m on that camp but mildly.
There's always a way to make your substantive points thoughtfully, though of course this takes more time and energy - sometimes a lot more - but if you'd please invest the effort to do that when commenting here, we'd appreciate it. Or, of course, it's always an option not to post.
It's good he is getting it out in the public, on record. He'll lie, conveniently ignore techno fascist astroturfing, selectively moderate on hot topic issues and then point to some wabisabi philosophy on both sideism.
The majority of top threads on hn on any given days are legit flame wars and llm accusation mud slinging.
But you cant say Dario Amodei is an effective altruist. Incredible.
> Furthermore, export controls, for which Dario Amodei openly advocates, are already prompting China to build its own chips. China is, of course, behind the West, but not as much as you may think. Instead of adopting a Western technology stack and relying on our technology to build its AI models, China will increasingly rely on its own chips, reducing Western leverage.
China has been working hard to indigenize AI chip making since well before the October 2022 export controls. I don't think the controls had zero effect, but overall I think the effect is often very overstated, and anyway it's irreversible (there's no way China will stop its self-sufficiency efforts even if the AI chip ban is lifted; the CCP will always ensure there's a large enough domestic market carved out for every Huawei Ascend produced).
Cf. https://newsletter.decisiontreeresearch.com/p/how-much-did-u...
I used to think that export controls didn't work, until I realized that China has leapfrogged the West on every technology that wasn't subject to export controls.
To be brutally frank, this was always going to happen when Western countries outsourced their manufacturing.
And all of the rationales were self serving garbage, it was about juicing the next quarters numbers.
Fundamentally though, it's the removal of capital controls that has driven a lot of this behaviour, and it's amusing to me that reintroducing them is so far beyond the Overton window that it never even comes up.
What did you expect. The West built factories in China and said “look, this is how we do it, now you do it”.
> Anthropic already possesses the compliance, legal, and evaluation infrastructure needed to satisfy such a regime. A university lab, startup, or community does not, even if they somehow manage to train a sufficiently capable model.
All the actual proposals that Dario Amodei has endorsed with regard to safety testing and so on have exempted smaller models. I don't think university labs or startups would ever be hit by any such regulation.
Have exempted smaller models so far. Seems like they're trying to soft-launch then tighten the ratchet, and I'd be willing to accept that this is a cynical view if this hasn't been the modus operandi for a while now. See EA/Sony/etc. continually trying to find a way to get their foot in the "always online, you're just renting the game" door.
has there ever been a CEO (or large shareholder) that ever said anything of substance against self interest?
it's a waste of time to take anything any of them say to heart, you should only read what they write to gauge what they are trying to signal.
Buffett and Munger, maybe.
I think he probably didn’t go far enough to be honest, though I acknowledge that’s difficult to do in his position and with the amount of attacks being thrown at Anthropic at the moment.
At some point open weight models will need to be restricted in some manner, likely multiple manners. We have rules about everything, you need a license to drive a car, your car must be roadworthy, your car can’t have certain modifications, you can’t drive over certain speeds, you can’t drive in certain areas at all, you have to wear a seatbelt etc. That’s one analogy amongst many, with the point being that regulation is a general public good that really does make the world a safer and better place.
Nobody should care about smaller local models that don’t have significant capabilities enough to be dangerous, and people should have the right to run them locally, privately and do whatever they do with them.
I don’t think that an AI monopoly or duopoly should be allowed to exist either, and I think it is possible to reach a reasonable set of compromises on each of these points, if the will exists and people can be convinced to be reasonable with one another. I think the will may not exist, and that reason is lacking in our discussions of reasoning machines, but I would like to be wrong about that.
The only widely publicized AI orchestrated cyberattack in recent memory came from a closed-weight model, and the only widely publicized defense from said attack came from an open-weight Chinese model. What do you make of that?
> I don’t think that an AI monopoly or duopoly should be allowed to exist either
Monopolies have always been allowed to exist, even welcomed. People have gotten confused, but there is a big difference between a monopoly and an illegal monopoly. Deciding what should and shouldn’t be allowed to exist as determined by regulatory bodies and not consumers goes against democratic principles (though it is sometimes necessary, such as with drugs).
I don’t think I want to live in a world where police decides what brand of chips I can eat on the sole basis of the chips having gotten too popular. Do you?
> What do you make of that?
I don’t make anything of it. That’s only been reported because it came from OpenAI and was a frontier model, incidents involving open models are probably happening but are not being widely reported, and those are happening with less capable models, so they aren’t successful enough yet to make headlines. I draw that inference from the amount of AI bot traffic which is now saturating the internet, AI driven attacks almost certainly have increased in like manner.
> Monopolies have always been allowed to exist
Antitrust law also exists, but is weaker than it should be. The customer doesn’t win from the existence of monopolies, they go against one of the basic premises of capitalism. Assuming you believe capitalism is a good thing which is its own conversation.
> I don’t think I want to live in a world where police decides what brand of chips I can eat on the sole basis of the chips having gotten too popular.
I really don’t know how you ended up with this as an example, did anyone say this, or anything close to it?
I'm mostly ignorant to anything regarding the safety of LLMs. What makes an open-weight LLM dangerous?
> What makes an open-weight LLM dangerous?
Same thing that makes any LLM dangerous. The intent and actions of the person using the LLM to do bad things. Any tool can be dangerous in the hands of the wrong person...
Computer keyboards are incredibly dangerous. You can even use them to type out the same text a dangerously capable LLM might write!
As the sibling comment says, user actions are the main driver, but also the fact that the weights can be modified. If a model is released with safeguards, they can be ablated out or the model can be fine tuned for nefarious purposes, and nobody can really stop that from happening.
That’s not to say that the majority of open weight models are being used in nefarious ways or are dangerous, but unfortunately despite (hopefully) most people being responsible, we have to guard against the small number of bad actors. Like with everything, a small number of people ruin it for everyone.
Nothing makes an LLM dangerous. An LLM remixes existing knowledge and outputs text. Any danger comes from what one subsequently does with the text.
For example, when a lab like OpenAI negligently wires up a harness to an LLM and executes arbitrary code in YOLO mode without common-sense isolation measures, the executed code can do bad things.
The first thing to restrict, long before open models, are large closed corporate LLMs.
After all, open models are not nearly as dangerous as rich companies.
I tend to agree with this also. The idea that the impact of AI is being decided by private companies with no control either by governments or the public doesn’t sit well with me. You need the most regulation where most of the danger is, and most of the danger as you say is with the big AI companies.
But there are some moves towards regulating them now.
[dead]
Fundamentally, we needed to decide, as a society, that should someone (anyone) have access to neural networks that can launch successful cyberattacks by discovering new zero-days and/or leveraging knowledge about recent zero-days and applying it in an adversarial way.
We de-facto settled on the answer being yes. These foundation models companies have access to such systems.
However, because the answer to the first question is yes, now we need to decide should everyone else be able to defend themselves with neural networks that are perhaps equally capable but are not sold by these foundation model companies as a product.
I think the answer to the second question has to be a yes, IMHO.
What makes me most sad is there are people at Anthropic, Dario included, who want to do good.
But thanks to financialization their incentives get in the way.
[dead]
Yes it’s true. But public can’t also cry about safety but simultaneously cheer for open weight models. Whatever safety concerns the public has against ChatGPT and Claude are drastically higher in open weight models. What is the right thing to do then? Accept that safety was always a farce? I’m on that camp but mildly.
Dario Amodei's stance on ________ is self-serving and short-sighted. It's the EA way.
Please don't cross into personal attack or post flamebait or take HN threads on generic tangents. We're trying for something different here.
https://news.ycombinator.com/newsguidelines.html
Criticizing EA is not a personal attack it's an observation of the outcomes associated with the ideology.
I'm sorry if it was flamebait.
The person in your comment was not EA.
There's always a way to make your substantive points thoughtfully, though of course this takes more time and energy - sometimes a lot more - but if you'd please invest the effort to do that when commenting here, we'd appreciate it. Or, of course, it's always an option not to post.
Claiming he's not an Effective Altruist is a stretch. He's been tightly coupled with that movement for nearly a decade.
It's good he is getting it out in the public, on record. He'll lie, conveniently ignore techno fascist astroturfing, selectively moderate on hot topic issues and then point to some wabisabi philosophy on both sideism.
The majority of top threads on hn on any given days are legit flame wars and llm accusation mud slinging.
But you cant say Dario Amodei is an effective altruist. Incredible.