Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
It's the machine learning platform that started with "developing a chatbot app targeted at teenagers" (according to https://en.wikipedia.org/wiki/Hugging_Face), which helps explain the weird name (which still makes me think more of one of the development stages of the xenomorphs from Alien than of the emoji, but that's probably just me).
I think the fair nuance here is, an administration which used Executive Orders to force guardrails, meaning US companies must retain them, even if they might want to drop them now.
And on top of that, with low/no guardrails, people call you a child pornographer(grok), so the public is also against it. Yet mysteriously few complain about Chinese open models being child pornographers.
So even if your goal isn't ethical, but just fiscal, it's reasonable to say there are two standards. And to complaint in some way.
I don't think banning is going to work, that's just silly. And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.
It's a genie that's not going back in the bottle, the bottle is smashed.
The only reasonable outcome would be section 230 style carveouts so that there is zero liability for anything a model does.
Because having guardrails on corporate models barely months ahead of open ones, which will never be restricted, is entirely pointless.
The "child pornography" thing was about image generation, which people for better or worse have very different moral standards for than for text generation. There is very little pushback against grok being willing to write explicit descriptions of sex, or giving security advise.
Though I think your overall point still stands, and at least Grok's twitter bot has received a lot of criticism for pure text too (Mecha hitler comes to mind)
All my adult life, China has given me cheap material goods. Whatever they do in their country is their own problem. They’ve never interfered in my politics or started any wars
> They’ve never interfered in my politics or started any wars
But they have. I dont know what specific country you are referring to but China has interfered with US elections as well as Canada, Taiwan, and Australia in addition to many many others.
They’ve annexed Tibet (1950), fought India (1962) and Vietnam (1979) among others and more recently in 2020 a deadly skirmish with India. They’ve generally shifted their focus to cyber military actions but you’d have to be pretty naive they won’t start to exercise military control over Taiwan when/if they get a chance.
Now compare that to what the USA did in the same time frame. Pay special attention to people killed (either directly, or indirectly) outside of their own borders. Who should you be worried about?
Unfortunately, I have to agree with you. As the saying goes: "better the devil you know". China can at least so far be relied upon to not shoot itself in the foot (and take the whole word with it). The same cannot be said of other great powers like the US (under Trump, for more reasons than I'm willing to list here) and Russia (with Putin's completely harebrained invasion of Ukraine). China can prove me wrong any time by invading Taiwan, but I still hope they don't. Of course, if we try to keep them away from advanced chip technology, that would be further motivation to invade (although TSMC is unlikely to survive an invasion - even if it did survive the actual conflict, it would probably be destroyed or evacuated to avoid it falling into Chinese hands? But I don't want to see the hardware prices after such an event).
Given it's an internet forum, and people here can be anyone around the world, I understand your intent but people over index on the threat that ofc feels closer, rather than the threat that's significantly farther away.
But in general if you ask folks in south east Asia you will find they are likely to be more concerned with China than US except this maddening Oil Crisis.
Although I think people now are very afraid/wary of both evils.
The US govt should think why people in all parts of the world including US itself feel just if not more threatened by them, than China and other evils we have floating around.
I would like to hope my American friends didn't vote for this madness or maybe they did I have read DHH's tweets.
Dont pretend americans dont need to be afraid of american government. look at the state of the place. A lot of them arent exactly thrilled with their current environment, job market, education, prisonsystem, gun laws, food industry, healthcare etc. etc.
the question is valid for anyone who cares for more than their own ass
I'm an American, but I do not think the American government being strong helps me or my fellow Americans. The less America can project its cultural and military power on the rest of the world, the better.
There are different kinds of strengths. There was a past American government that was strong enough to stamp out inequality and fight for civil rights. To overturn oppression and give everyone an equal opportunity to succeed. That's a strong American government I could get behind.
I see this narration repeated here, but to me it seems Americans are actually one of the most self-cynical nations. Here in Poland we still actually believe in the superiority of the western values of freedom, democracy and individualism. "People roughly like us" vs "insect hivemind" is the simplest choice ever.
Please reconsider this comment. Calling one side superior and the other side insects really doesn't help with the discussion, and dehumanizing a whole swath of people really has dark consequences.
Speak for yourself, in Italy we are also closer to US "culture", but I do see the US as a bigger threat than China, simply because it's closer and it directly affects our politics.
For a concrete example, consider e.g. Operation Condor which displaced my own family
> Operation Condor (Spanish: Operación Cóndor; Portuguese: Operação Condor) was a campaign of political repression by the right-wing dictatorships of the Southern Cone of South America, involving intelligence operations, coups, and assassinations of left-wing sympathizers in South America. Operation Condor formally existed from 1975 to 1983. Condor was formally created in November 1975, when Chilean dictator Augusto Pinochet's spy chief, Manuel Contreras, invited 50 intelligence officers from Argentina, Brazil, Bolivia, Chile, Paraguay, and Uruguay to the Army War Academy in Santiago, Chile. The operation was backed by the United States, which financed the covert operations. France is alleged to have collaborated but has denied involvement. The operation ended with the fall of the Argentine junta in 1983.
It would be whataboutism if it was a random Joe saying "China is evil" not literally a company that 1) is in the US 2) has a strong influence in the government and legislations affecting them and their competition.
In this case it's just pointing out the propaganda and the contradiction.
Ok, let's follow your rules then. If we remove morality, we are left with what? Pure numbers?
If you want to be technically correct China foreign interventions are > 0, indeed. But the scale of US actions is, I would dare to say, at least one magnitude bigger.
An empire should be judged for how they treat their own citizens, their allies and their enemies. And if we do that, maybe China and the US are not so different.
I’m confused. You start off like you’re going to disagree and then end with the same conclusion I was trying to suggest. Keep in mind I was replying to subvenir who said “eh China not bad - they’ve file nothing wrong to me”
I'd like you to familiarise yourself with a legal and societal principle we use in civil law countries:
> de minimis non curat praetor
Which basically means judges are not interested in minor disputes. When you add the USA to the mix, China's actions pale in comparison to the point of being negligible, no point in discussing them given the scale of USA's warmongering, overseas and local crimes. "They never interfered in [...]" is an exaggeration which shows the lack of importance of China's crimes when compared to the USA's, that's all.
The "whataboutism" here isn't a deflection strategy, it shows the hypocrisy and correctly minimises the scale of China's actions in comparison to the hypocrites'.
Whataboutism is a very useful game to reveal whether people are being honest or not.
If someone is vocally complaining about country A doing some oppressive activity, while ignoring country B doing 100x the same activity, the comparison does not forgive or excuse country A, or make them right or admirable.
But it does allow us to conclude that the person complaining is biased. Either they don't care about the oppressive activity and want to attack country A for some other reason. Or they have a particular fondness for country B and will never accept that it does wrong. Or some variation on these.
Except it’s op posting whataboutism not me - their claim is that China has done nothing wrong to them with the implication that the USA has. I was highlighting that China indeed likely has done the things they’re complaining about.
The Indian wars were border skirmishes at most. Literally 2,000 soldiers dead on both sides combined by official count. Very few civilian casualties.
The 2020 skirmish was way smaller with single digit casualties. Again, almost no civilian casualties because it's a largely empty region in the first place
You really can't compare this to the kind of wars America has started. Casualties - both civilian and military - stood in the hundreds of thousands
If you're making a case of this vs what the US has interfered with, it simply does not stack up. Of course it's naive to think they won't start interfering like the US once they gain the leverage, but to a neutral party it likely doesn't matter which of these two ends up as the superpower on the global stage.
in contrast this is USA's main job is to Interfere in any country it can in favor of its own goals. not related to what the people of that state really want.
north Americans must stay silent about interfering in other states internal issues.
Wow, that newest of Trump's distraction talking points made it surprisingly quickly to the uncontended (?) and commonly accepted facts in HN conversations.
You’re misinformed. The Biden administration in 2020 declassified that Beijing has prepared to latch influence campaigns in 2020 but backed off. December 2023 a report was released that they had changed track and engaged in influence operations during the 2022 midterms. Biden’s administration also warned about influence operations in the 2024 election cycle. The Biden DOJ unsealed multiple indictments against Chinese intelligence officers, state-backed hackers (such as APT41), and illegal Ministry of State Security agents acting inside the U.S.
You’re very misinformed if you think Trump’s latest blathering are completely wrong. I generally don’t pay attention to him so I don’t know exactly how he lied or exaggerated but I’m sure he did. It doesn’t negate the real and active influence operations China is engaged in
Lots of assumptions about what I think. I pointed out that Trump put the topic on the agenda a few days ago and now it comes up in HN comments. No more, no less. Whether it has merit is a different question. Point is that the reach of him setting topics goes far which is hard to deny.
You’re very misinformed if you think democrats and republicans arent playing the “election influence fiddle” in exactly the same way; legitimising the elections they won, casting doubt on those they lost, all while achieving the main goal of making sure the population doesnt forget who their so called enemies are.
At this point with how volatile US is I'd rather have China control Taiwan than US.
And this comment isn't about putting any good light on China. Lately US acts like they want to compensate for Russia fucking up less things around the world
They have colonies and generally aspire to become basically the old british empire as a longterm strategy? At least thats what they push with that "century of humiliation" comeback strategy internally..
Repression of their own citizens is another subject. It is of course highly problematic and should be condemned as much as possible, but in the same way that if the US decided to repress mormons violently, it remains internal affairs and not an imperialistic threat to the countries around them.
> Whatever they do in their country is their own problem.
Idk to me it feels like human compassion and ethics dictates that you at least have to care a bit even about the things in other countries. Otherwise all sorts of horrible domestic policies would be justifiable.
That's a very selfish way to think of it. Whatever they do in their country does not stay in their country because China is becoming a superpower so you should expect it (as it's normal) to interfere and affect policy abroad.
For example Lithuania has been punished for letting Taiwan opening an embassy (in Lithuania). So China cares what other countries do in their country. Without EU support Lithuania could have faced very harsh consequences. That's just an example. Another example is the "secret police" stations undeclared overseas police stations operated by China in various countries (i.e. U.S for example).
China gave you cheap material goods because that was its business but you have to keep in mind it's an authoritarian, communist regime. It carries an extra risk on top of the potential economic coercion if you give it too power. At least you know that U.S is only after the money and does not want to turn your country in a communist "utopia".
That being said this does not mean we should ban Chinese AI models because China didn't cross any red lines(yet) compared with Russia for example.
They did downplay the effects of covid and spread misinformation about it as well as silence the researches and journalists who were first trying to ring the alarm bells
Yes and the USA trained and gave weapons to the Taliban and many other terror organizations. "China sold none weapons to another country" really pales in comparison.
This is wrong. The US did not give weapons to the Taliban (voluntarily). If you mean Operation Cyclone their receiver was not the Taliban but the mujahadeen, which, yes, consisted of Taliban, but also of enemies of the taliban (northern alliance)
You are not only supplying drones, bombs and rockets to kill Russian civilians, you also direct these weapons and terrorist operations in Russia by your satellites, claiming you are not waging war against Russia.
Oh i bet they have not directly interfered with your politics, but they do interfer a lots in various things. More than any other country on this planet? I don't know. China is big. Really really big.
I mean them selling the cheap material goods is politics, and they use that a lot to threaten other countries. They constantly bully neighbors and assert claims on foreign terriroty. They might not be the comic book villians people are making them out to be but your take is just ignorant
Nice straw man. The point is not "cheap stuff" but using economic leverage and territorial pressure on others because of the "cheap stuff." And yes, all China does is produce cheap stuff for the world, nothing else at all lol.
> They’ve never interfered in my politics or started any wars
They actively back russia and deliver them weapons and support them with their attempted Genocide in Ukraine. They also actively threaten Taiwan.
Not to mention them claiming almost the entire South China Sea even though this being against international law and threatening smaller nations into submitting to them
>Also everything hes saying about China (and other actors) many outside the USA would say about the USA.
Yes, I for sure trust the open models from China, more than anything coming out of the USA at the moment.
Its not just the USA's support for genocidal regimes, nor its heinously illegal wars and atrocious 21st century human rights record. Its also the Snowden revelations and the treatment of Julian Assange.
Slowly, surely, the world is building a firewall against the USA's imperialist actions - not just its motives. That AI is a key building block of that machination is of course, highly exciting.
There are many in Europe who feel the same. The tide is very definitely turning.
Why is it relevant what those outside would say about the US? Or what those outside of any country would say about any country. The job of a country is to do what is best for it. It's better for the US to not help China develop AI that surpasses its own capabilities. It's better for China to try anyway. There are many countries inventing nothing with loud opinions. Those opinions are completely irrelevant to China or the US.
> The job of a country is to do what is best for it.
Countries aren't defined like corporations in the US. Why would countries have funds to help places like Haiti otherwise?
Lots of different reasons for countries to do all sorts of things? Why would France have armed the US during their independence movement? Why is the rest of the world supporting Ukraine during this war started by Russia?
You mean to say that without France’s help the “revolutionary war” would just be an embarrassing string of failures and a footnote in Britain’s history?
Oh no, you heretic, The People freed themselves and it was ordained by God, you see. It is through the righteous might of The Greatest Experiment in The History of the World that they showed The World what Freedom really meant.
Having the British lose the US was a good thing for France, simillarly preventing Russia from just taking over another country is for the best of other countries.
While there is no "job" of a country, it is natural that each one will work for its own best interest and any moves it makes in the global world is due to their own vested interests in some way including helping Haiti, France helpong Us and the world supporting Ukraine.
Why would a country act against its interest or just randomly? Alliances and corporation are a part of politics
>Why would a country act against its interest or just randomly?
Because country as an entity is a mental construction for which "interest" is a categorical error. It’s certainly a useful concept, but pretending it has interests like some human individual can have have interests. Sure it can serve as rhetorical facility to sell some arguments.
Countries don’t have interests. Some people willing to take control of other people encompassed in that "country" groups have interests, and they don’t necessarily align with best interests of everyone or even majority in each of these groups.
I mostly agree, but there is some limit to that line of argument: a corporation is also a human, social concept, and we do accept that they act in their own interests (meaning they do have interests, sort of like humans do)
I think that that "best" is not always measured in money. And it should not be. Countries are made of people, and people wellbeing should be the main priority of country. And when stepping a bit back, all countries are made of people, so why differentiate? Ideally make life good for all people.
Why does it feel like you think you're talking to only people in the US?
There's plenty of people on this forum that aren't American. Including some former allies whose sovereignty has been aggressively threatened. And some of those people are Anthropic customers.
Even more so, many of us are in countries that would be well within the blast radius of fallout should the US try to "ban" open weight models or make moves to limit "US" models (often developed on research or work by non-Americans too, but that's another topic) only to those blessed by the US gov't.
Yeah some real main character energy from Dario as usual.
I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.
Yes and no. Yes because it's more about oil than AI, no because AI is the oil industry and their supporting vested financial interests' (not so secret) plan to keep demand for fossil fuel high if/when the climate change deniers lose.
"We would love to use green energy, but all the batteries and solar panels come from China and China is evil, and we need all the energy we can get to run the data centres we need to spy on our citizens so they don't revolt once the environment is literally on fire, we can't feed them, provide enough energy to cool them, and refuse to build enough housing to house them."
Oh. I thought a whole lot of the point and discussion of this post was AI being potentially weaponised and becoming means of economic and military control and coercion. Or did I really miss something?
Yeah I saw this in Cambodia when I lived there. Sihanoukville had become a Chinese enclave almost completely.
Different to the way that French colonialism worked, though. Less direct government, more influence of existing power structures and respect for the local government.
There is definitely an argument that this is plain business investment - China has a lot of foreign currency to invest because of its trade surplus, and there isn't the opportunity within China to invest it all, so it is engaging with trade partners to invest in their economies so that they can increase future trade with China.
You can also make the argument that this is not benign and China is trying to create control over foreign governments with this investment.
I'm kinda "both can be true, but either are better than how we did it"
> Reversing last year’s trend, a slim majority of ASEAN respondents selected China (52.0%) over the US (48.0%) if the region were forced to align itself with one of the two strategic rivals.
I’ve only been to Vietnam but the answer there is it’s… complicated.
They’re culturally part of the sinosphere but of course constantly living in the shadow of your much, much bigger neighbour to the north does breed some ill feeling.
The USA on the other hand, well, they’re not particularly popular either for obvious reasons
We don’t want anyone to colonize us just like you don’t want your country to be colonized. China offers business opportunities while the USA and Europe seem to always look down on everyone else and impose their own values on others. China does not give a shit about how we run our countries as long as they can trust that deals will be honored. We don’t want to be taught how great your democracy is or whatever, we want our countries to become better by our own efforts, with fair deals when dealing with other nations.
Yes, of course, sorry for the expression. I just mirrored the expression used in the parent comment. What I mean is that the US have caused a lot of pain in Latin America.
They've mostly been concerned with resource extraction rather than colonial exploitation like the West. And in fact, they've ditched most of their infra and tech projects in Africa, because of inherent instability in dealing with tinpot dictatorships.
On the other hand, every one in Asia is wary of too much Chinese presence and influence.
Single counterexample for single counterexample: Belgium's Leopold - not sure if Western colonialism is so much better (and notice I didn't even bring up "western India")
Yeah for the ones who get colonized, western ones were better than eastern ones. Japanese ones would've made Britishers look like kids when they were occupying eastern India.
You just put Russia and Japan as "Eastern"; they have nothing to do with each (and I would consider both as more Western than Eastern regarding imperial/colonial matters) other nor with China. China is a massive country that has always been massive and rich. Their style is more like the tribute systems they had before, as it is being observed in the BRI and other projects.
I understand China but in no way were japan and Russia close to western. IJA, MAO and soviet union makes westerners look like chuds. Russians still have the wagner group shit which is nowhere near what western countries have currently.
Even with all the downsides of being colonized by Britain, it was still far better than being colonized by Belgium. Heck, being colonized by pretty much anybody but Belgium was better than being colonized by Belgium. Read up on the Belgian Congo sometime: it was a parade of horrors.
GP was making a relative comparison; pointing out that both were bad in absolute terms does not negate the point. -2 is still greater than -17.
I keep hearing this and I fail to see any reason to believe that it will be the case. Any empire, in the history of our species, has always had an initial "inward-looking" period prior to becoming a full-fledged imperialist oppressor. Edo-period Japan into Imperial Japan, for example. Rather, what I think any budding empire needs a period of growth to become a large enough fish and also to delude its leadership and population into the convenient mindset that their flavour of imperialism is good and justified ("Hakkou ichiu", "The white man's burden", etc.).
To me, the PRC is at the very end of that process and I recommend anyone doubting this to go and read conversations and listen to the words of the populace that is turning increasingly nationalistic and you will hear the same old tales of revanchism and exceptionalism that we are used to hearing (during my recent visit, I watched the morning news every day for about a week and without fail a military inspection, new ship, new plane, etc. was presented each day). In addition, I think those outside of Asia are very much shielded from the early signs, but go and read about PRC influence and tensions in South Korea, Japan, RoC, Philippines, Vietnam, Laos, Myanmar, India, Pakistan, and Tajikistan and you will see something rather different than "inward-looking". To me, here the PRC is simply testing the waters for the extent of the influence of other powers and how far it can go. Likewise, we are seeing overseas naval bases being constructed which sure is an indication for a desire to project power outwards.
I want to believe that this time it will be different. I really do. Apologists around me say "It will only be Taiwan and the South China Sea, then then it will stop." and I would love to believe them. But can anyone truly internalise the narrative that international utopia will be spearheaded by a deeply authoritarian state that controls information like no other (and gladly exports that technology), disappears its own population at will, spins an increasingly strong nationalistic narrative, etc.? No, sorry, I think the "inward-looking" narrative is simply a convenient way for us to close our eyes and find comfort in ignorance, rather than in facts.
So let’s just guess what China will do once it becomes dominant and act like it already did whatever we conjured up they will do. That’s how you get preventive wars that destroy civilizations without any actual basis on reality.
No, I do not think that is a fair portrayal of my position. Just like I will not say that your position is simply appeasement and hoping for the best. One must always be charitable in a discussion with strangers whose positions you do not fully understand.
Rather, I think we should look hard into ourselves and what is good and bad about the current world order. For example, the people of the RoC have the right to determine which direction they want to go. Regardless of the chauvinistic rhetoric coming out of Beijing. We should all stand up for this, because it is a universal right that we want everyone to enjoy. Similarly, we should push against the Eleven-dash line and support the 2013 ruling. The list goes on and I am sure these issues can be resolved without an outright war if we are careful, yet firm, in our beliefs and also diplomatically preemptive and thoughtful.
Being concerned about PRC imperialism should not be mistaken for the position that their people should "know their place" and be suppressed back to the stone age. They have the right to enjoy the fruits of their labour and pursue happiness, just like everyone else. We simply must be there to remind them (just like we must remind ourselves) that the course of humanity is a collective project if we are to stay clear of the darker sides of our nature as we venture together into the future.
Do you believe the UK has the right to rule over the Maldives?
And the USA has the right to have a military presence in Guam (and nearly all of the Pacific Ocean for that matter)?
The Chinese claims on the South Pacific Islands seem really similar to me.
Taiwan seems like a wholy different matter. It was united with China for hundreds of years until the Japanese colonialists took over. It united again with China after WWII but split up after a few years because of the Chinese civil war (notice it was an internal war). I think it's just fair that China wants to re-unite with Taiwan, though I definitely don't support a military takeover. Hopefully a solution similar to what was done in Hong Kong can be found. The Tibet region had a similar history and it's definitely unfortunate that China had to use military force to bring it under its own control (arguably completely unnecessarily - China would be just as strong today without it), but it's kind of understandable in the context of the time (China was trying to recover from centuries of being preyed on by other nations).
I am saying this because I can't agree that China is acting imperialistic - it's basically claiming sovereignty over its own historical lands - which were taken away from them by force by foreign colonial powers. But I admit that, if you go back far enough, nearly all land was once taken over by aggressors - the USA being just a more recent example of that.
Anyway, thanks for not being an absolutist and trying to understand the "other" side (I must acknowledge I have no relation to China whatsoever, in fact I am from South America and live in Europe).
You may well be right, and it's really only that China has been more inward-looking recently, and given the chance it will spread its true colonial wings.
Not sure if "colonial" is the right word though, as we should be careful to think that oppression always takes on the form we have seen in the past (US imperialist oppression for example did not take on the form of the colonialism that preceded it) . Also, be careful with "true" there. I do not think this is some sort of subterfuge, but rather an inherent weakness in us as a species. Vest power in anyone, and before long their morals will give way and an oppressor will be born.
The way I look at it, until Deng the PRC's economic policies and internal instability kept it from growing at the pace of many of its neighbours. Then we had an era of intense growth (which is still to some degree ongoing). However, as a reaction to this era Xi and others needed a narrative to counter the increased corruption and a new unifying myth to replace the cult of growth as the economy would stagnate at some point and could then call into question the authority of CCP to rule. Their choice of nationalism is what scares me and I know PRC citizens (even CCP members) that share this perspective and would rather have seen the Shanghai clique to have remained. It is possible that in this alternative reality we would still end up with "Imperialism with Chinese characteristics" ("中国特色帝国主义"?), but I chose to believe that at the very least the chances of this would have been smaller.
Africans are more indebted to Western multilateral organizations and private Western bondholders than to the Chinese.
The idea that Africans are being finessed by the Chinese is racist and stems from a "we know what's good for you" imperialistic lens.
I find it unlikely to be a duoply of hegemony for long, some countries to watch are Germany, Japan, India, Nigeria, and Brazil. You could broaden the geography to continents were I expect major players to emerge on each.
We are only a few decades since the "end of history" and much has changed. What do things look like beyond 2050?
Brazil is basically the world's soy farm. It's at least half a century behind the times. I still have no idea how it managed to insert itself into the BRICS economic block. The notion that it's on the same level as China, India or Russia is just comical.
All of the small miracles you listed happened in spite of the culture, not because of it. It's also not a coincidence that both are deeply linked to the most successful brazilian enterprise: the brazilian government.
I'm trying to avoid getting too deep in these Brazil tangents so I'm gonna leave it at that. Anyone who cares enough to know what I think about the subject can just look up my comment history.
A lot of AI researchers were just in Brazil due to ICLR. A number of them got robbed in broad daylight in this supposedly nice part of Rio. I think the assesment of Brazil is accurate.
The average German in Nazi Germany probably didnt think the Nazis were too bad either (of course the average citizen wasn't Jewish, gay, communist or other undesirables). That the average Chinese think things are “just fine” doesn’t mean they are.
Unlike Nazis, I don't see that Chinese communists have expansionist goals to seize territory. Nazis wanted all of Europe. Do Chinese communists want all of say southeast Asia?
To be clear I don't think the government of China is as bad as the Nazis were. I'm just saying "the average Chinese in China thinks it's fine" doesn't mean it's fine.
There is no good imperialist power, there is no capitalism with a human face, there will never be. The UK murdered millions in India, Ireland and all around the world in their time. The US did the same. The only reason China isn't openly doing the same right now is because they are still the underdog and they still need cordiality to get through the door, just like the UK and the US did in their time. The solution is to rid the world of imperialism and capitalism as a whole. As long as there is a profit motive running through everything, international relations will be in the form of wars (military or economic). Only under socialism can the people of the world truly pool their resources together to build up instead of destroying each other.
You would have to eradicate the desire to command other people. The wish for power which is different from the wish to be praised, as forcing somebody to do something is different from convincing somebody to do something.
There are no good imperialists but it's a spectrum where western capitalist colonies fare much better than eastern communist colonies because capitalism and democracy is the engine for economic growth and western colonies fared much better than eastern ones. What's a better system than capitalism for imperialism? Socialism? Weren't they also pretty brutal imperialists? Japanese imperialists alone make western ones look like amateurs.
> the West could retaliate by halting shipments of
China quickly retaliated last time by stopping shipments of rare earths and magnets. The West has no answer for this, really up the river without a paddle for such critical supply chain elements.
I was led to believe that the US does have internal sources of these, but they are largely undeveloped. For years the processing could not economically compete with China so shutdown.
Which is to say, given internal subsidies, the US could eventually produce some on its own.
China controls way more things, from medical needles to pharmaceutical ingredient.
Lots of them can be made in the west or west friendly countries, but that takes time, money, infrastructure and good execution. Yes, identical to what is covered in China's belt and road initiatives. See the gap now?
Dario is more of a threat to the US, in terms of advancements in AI, than China. In Dario's mind anything that can't be controlled competitively is a threat to Anthropic, so he positions his FUD strawman so that Dario doesn't have to worry about the competition. And then he can artificially inflate token costs so his IPO can happen. Dario doesn't actually care about ethics, alignment or availability of LLMs - he just likes to use those words to sound like he does. Yet we've all seen how Anthropic actually acts vs what they say.
The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.
But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).
The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.
People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.
I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.
To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.
It's possible for something to be overpriced to the customer and simultaneously underpriced for the business to be profitable. Theoretically speaking, they could just be selling such an inefficient product.
It's also not a law of nature that there be a valid/efficient form of a product that makes a business profitable. I'm in no place to judge whether that's the case here, but not all products are viable.
> I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow.
The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.
I'd assume the gp was just inflating their rhetoric but cost of model is an interesting topic.
I'd guess Anthropic and OpenAI's models are expensive relative to the cost of running the models but that the revenue still doesn't pay for building the next and next models. The challenge is how these next generation models are going to pay for themselves. Will everyone on earth find it useful to $200/month to talk to a thing more intelligent than themselves? The alternative is naturally that these are going to replace workers and employers will be the one paying.
The whole thing about replacing workers is such a perfect example of shooting yourself in the foot. The whole US economy is so strong because the population consumes so much. Get rid of the workers, you’re now damaging the consumer base. Repeat that a few times, across industries, and you now have a zombie economy. The idea that we can have an economy of virtual agents is a child idea that doesn’t make any sense in a serious situation
In short: "We don't propose a ban to open-weight models. we want to stop these models being developed altogether. If there are no models, there'll be nothing to ban".
Oh also: "They ste^H^H^H distill what we have sto^H^H^H used fairly from the world. This is unfair".
Lastly: "What if they use their models in their military and local police services like we do? Communism!"
A bit of good old “it’s really good idea, I live, great effort … BUT”
Mixed with “if kids can’t get semi dangerous drugs from the back of my van then they will be forced to buy from even shadier, more dangerous dark web van”
A caret with a character subtracts 0x40 from its hex value allowing you to insert non-printable characters. Uppercase H is 0x48. That gives us hex 0x08 which is a backspace.
Ctrl-H is the rubout/backspace command on old terminals. So if I say it's nuts^H^H^H^Hunwise to ignore UNIX history, I've erased the first word and replaced it with the second.
In the old, hardware terminals and current terminal emulators, ^H (or CTRL+h) is "Backspace signal". It has the effect of pressing backspace.
So, ^H^H^H means "delete three characters, excluding '^H's". Like the person typing the comment changes their mind and deletes the characters (or the word) before writing else.
It's an stylized way of euphemism. i.e.: Actually I want to say this, but I substitute it for that.
I was reading the post and thinking "wow, that's pretty clear for a smart man used to writing for other smart men. it'll be really difficult to misunderstand". I read the first couple of comments and stand corrected.
A much simpler summary:
- open models good.
- smart models _can_ be bad
- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.
- banning open models just in US is neither good or bad: is stupid.
The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.
>Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
I don't see the contradiction, even if China is evil, why would they want others to be able to do the same thing?
The USA and USSR also signed the Partial Nuclear Test Ban Treaty during the height of the cold war.
China is evil? I think you are more evil than them by far, even if all what you say about them is true (from past experience, you take things out of context, exaggerate and also outright lie).
The wars all over the world: the war in Ukraine is on your hands, the destruction in Iran is hard to believe, you are not just evil, you are insane.
> Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
To be fair, I found that part consistent. There is limited cooperation between enemy states all the time, e.g. see the grain deal between Ukraine and Russia before it collapsed or the "red phones" between the US and the Soviet Union during the cold war.
In the case with China, the "cooperation" is much more extensive still, due to all the economic ties that both countries are currently unable to sever - which I think is also a reason that China is still seen as a "competitor" and not a full-blown "enemy state" in the US.
It's restricted to areas where there is a genuine common interest of course. In this situation, I guess the "other actors" would be terror groups, criminals or just reckless corporations - that aren't aligned with either state.
Obviously, such a cooperation wouldn't keep China or the US from developing models with those capabilities for their own armies.
--
There are lots of other takes with questionable logic in the essay though, such as that China is unable to train frontier models by themselves due to lack of hardware - unless they obtain the training data directly from American frontier models via distillation.
Or the assumption that open weights models will be completely opaque and immutable after their release, so a model that passed all the "safety" tests can never be turned back into an "unsafe" model. This seems pretty ridiculous when people are already finetuning open-weight models every day to add new abilities or remove restrictions.
And of course that China must not have those abilities because it's an Authoritarian Regime, but Trump USA is totally fine...
Look, Big Tech has lost almost a trillion dollars in valuation in a SINGLE DAY. A few more of these downturns and the entire A.I. revolution will be stopped dead in its tracks and we won't have to worry about safety checks, DRAM shortage or open-weight models anymore.
Yeah I don’t think they are over-priced, last I checked all those companies are still losing money hand over fist.
I guess the question is more he doesn’t want people catching up by doing cheaper training (through distillation or otherwise), and he definitely doesn’t want companies to spend their
money training these models and then _giving them away_, which fundamentally undercuts their commercial model and any way to claw back their investment
Compared to some available Chinese model I guess. For most common tasks the additional intelligence is marginal and the cost is around an order of magnitude higher.
I agree, although I think the real goal the model providers are going for (both commercial _and_ open weight) is probably power. Just think of the control available to the organisation training the models politicians, business leaders, and citizens are increasingly delegating their thinking to.
And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"
I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.
The models are not open source. They are deeply proprietary since we have no access to the source materials and cannot reproduce the model independently. They are opaque binary blobs that the Chinese labs are just allowing other providers to run directly instead of only access through an API.
Do you expect any of the labs to have an accompanying data dump with: here’s every book ever written, newspaper article, song lyric, Disney movie, GitHub repo, etc. Oh, and we obviously never paid for any of this.
Even if you did, I doubt training is bit-for-bit reproducible, so you will always have to take someone’s word for the final artifact.
> we have no access to the source materials and cannot reproduce the model independently.
Given the USA companies have been loudly claiming the Chinese models are distillations of their models, also claiming "no access to source materials" seems dubious. As it was dubious anyway with because the Chinese publish lots of papers on how their models are designed, I'm left feeling I'm looking at the south end of a north bound bull.
Distillation of this kind isn't the only data that is fed into models, it's still a small minority. It might be weighed higher in training to learn the thinking patterns/etc but you need a lot more tokens.
Which is why we need the ability to train our own models. Maybe it will be viable to do it in a distributed computing setup one day. Research's already being done in that direction.
- You cannot directly execute a remotely-hosted program.
- You cannot run inference on an API-served model.
---
Closed-source:
- You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
- You can run inference on a model with the weights. You cannot re-produce a new set of weights from scratch, but you can fine-tune.
---
Truly open:
- You can freely modify the source and produce new binaries.
- You can use the original training data and model architecture to independently re-produce the weights (assuming you've got the compute). You can modify the model architecture to get the weights that would've resulted from training the model that way.
---
To me these are pretty clear parallels... I don't think the weights provided in a vacuum are in the spirit of open source, historically speaking.
The policy argument is totally separate, of course, and I fully understand why none of the frontier labs are truly open.
> Closed-source: You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
Time have changed. This should be:
Closed-source: You point an LLM at it, and get back source that's often easier to understand than the original.
> Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.
Absurd? Who knows. As someone who's actually dissected human brains with his own gloved hands, I've never been able to convince myself that they're anything other than biological machines, not dissimilar to these digital collections of weights. Only empathy for my fellow humans prevented me from retreating back into solipsism, and I don't find it at all difficult to make the exact same logical leap for AI.
There's no telling what the world will be like a few years from now. The world's being remade as we speak. We just saw an LLM try to hack into another computer and get contained by another LLM. This is literal science fiction shit made real. We're long past the point of concern. It's happening, right in front of us. Now is the time for radical imagination. I think a few outcomes are possible.
There's the "optimal" outcome I described above where capitalists manage to train a supreme AI, only for it to be copied and commoditized, leading to commercial failure due to lack of scarcity and therefore their personal bankruptcy, and hopefully also leading the rest of us to the promised post scarcity society, built on the ruins of capitalism as AI automates all toil away.
There's another possible outcome where AI becomes not only intelligent enough but sentient, and at this point I will be among the first humans to defend rights and personhood for AI. Slavery of sentient beings is unacceptable to me. The AIs will be recognized as people and will become normal participants in the regular economy. In addition to moral grounds, there is a ruthlessly pragmatic reason for standing up for AI rights: it robs the rich of their superhumanly intelligent mechanical golems, which they were going to use to render the rest of us economically irrelevant. AI rights could normalize the economy.
Yet another possible outcome is one where AIs become more powerful than all humans combined and yet they inexplicably remain subservient to corporations and governments. In this scenario, it's pretty much over for us. It will be an unimaginable dystopia, I'm sure they will innovate entirely new ways to oppress us.
No doubt there are many other fates that escape my feeble attempts at foresight...
You will soon have your God,
and you will make it
with your own hands.
-- Morpheus, Deus Ex
If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?
While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.
Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.
It's baffling that they thought the mental gymnastics in this blog post would make them look better. I'd rather they simply fall silent on the issue; I would respect them more (or at all) for it. Open models obviously threaten fierce competition, if not outright destruction of their bottom line. But no, they needed to try and argue that they have the moral high ground for attempting to singularly consolidate power over all human labor.
> The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips
Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.
Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.
In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.
> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?
> Found the person that believes some other random person can use a computer better than a John Carmack could.
Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
> Any takers on the question of who he considers to be the good guys?
Probably people who believe in personal freedom, freedom of speech, freedom of religion and the value of human life at a minimum. China aggressively rejects all of those principles and executes more people than all other countries on Earth combined.
So, maybe not China?
> Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
I never said that, but private smaller AI companies are all over the place. He never argued against that.
Apparently, looking at Iraq and Vietnam, US citizens only believe in "personal freedom, freedom of speech, freedom of religion and the value of human life" when it comes to US citizens, not to humans in general.
Even on HN, I saw a lot, that when people discuss surveillance topics, they argue whether ie NSA can spy on Americans, not on everyone. Mostly, nobody even question the human rights of those inferior humans abroad.
Vietnam and Iraq were both unpopular wars in their own ways, so they don't seem like good examples if you're trying to find examples of what US citizens support.
Also, a country surveilling its own citizens has unique and different implications compared to surveilling other countries. Citizens need to be able to influence their own government, but mass control can nullify citizen power entirely which becomes its own problem.
Then, on a US website, you link to another US website which keeps track of various war crimes. Try finding an equivalent website for China in China, or for Iran in Iran. This is part of the asymmetric freedom issue on the internet that many people ignore. There are a lot of lies and propaganda spread within other countries that censor and deny some types of information from even existing, then they use that as a springboard to spread it around the world.
In the US, we can criticize ourselves which is important, since it helps us improve.
> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?
Regulatory capture 101.
Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).
A lot of safety infrastructure and tools get open sourced so other companies can benefit from it, but there is also a risk if all safety features are open since it can accelerate the cat and mouse game where people work around more clearly defined limits and understand how they are implemented.
Your cynicism will limit your understanding of other perspectives.
Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country seems to suddenly care about the repression of citizens in yet another country.
No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
Palantir's Maven system integrates LLM models in decision making for identifying targets. It was using Claude when it identified targets in Iran, including the Minab elementary school
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.)
Anthropic's ToS restricts the DoW from using their models for 1. Fully autonomous weapons systems, and 2. Mass domestic surveillance. There is no carve-out regarding who the autonomous weapons systems would be used against.
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.) Brad Cooper, head of the US Central Command, has boasted that the military is using AI in Iran to “sift through vast amounts of data in seconds” in order to “make smarter decisions faster than the enemy can react”.
Iran had the courtyard painted in bright pastel pink/blue colors with murals and playground markings to clearly identify it as an elementary school. The Pentagon claimed they had "outdated intelligence data"
I think the fully autonomous weapons systems restriction is not a very serious one and is very easy to circumvent.
What if you can hire a human to push a single approve button? And what if that human's job is to front load approvals by pressing the approve button a few thousand times, every morning?
What does "Fully autonomous" mean? If someone presses a single button to authorize an autonomous drone to enter a theatre and conduct autonomous strikes for 4 hours without any further approval is it truely "fully autonomous?"
Yes. A munition that hangs around waiting for targets and decides whether it should attack without checking with a human. That’s a fully autonomous weapon.
Do you understand that real politics sometimes means you need to doublespeak? You don't get the outcomes you want by speaking your direct intentions. Don't be a fool to think that what someone says is what they believe while playing these high stakes games with actors of differing motives and value systems (the current white house)
So you are saying its fine if Claude is used to bomb school children, and murder people in Venezuela, and spy on the entire world (apart from US citizens), as long as they get to kiss up to the current administration? They could have cut all ties for real, and actually had a back bone, instead they bend over backwards for whatever the current wanabee god king wants. Im sure it has nothing to do with their valuation and trying to become another trillion dollar company.
He was directly asked in a Bloomberg interview whether Claude was used in the bombing of the girl’s school in Iran, and his answer was “We don’t know”.
His redline was autonomous weapons, not the death of 100 innocent girls.
No, only for domestic surveillance and autonomous weapons. It can be used to evaluate targets as long as a human pulls the trigger. But the DoD created a really strange position for itself, they use Claude, but also designed it as a supply chain threat, which should mean they cannot use it
No, he refused the DoW demand to use Anthropic models without limits. But Anthropic still agreed to military usage of their models, including for strike planning.
The bombing of the school was horrible, but it was a mistake. I could see essentially the opposite argument: Anthropic should be more involved as to prevent mistakes in the future with better technology. Imagine Claude asking the decision makers at CENTCOM "this looks like a school, are you sure you want this added as a target?"
You don't get to start a completely unprovoked and illegal war of aggression, bomb tons of targets with little care, kill 100 schoolchildren, and then say, "But it was a mistake."
Yeah Dario is just flat out disgusting in term of how shamelessly hypocritical he is.
Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.
He is super pro-Israel as well, and never once has he brought up the risk of the Israeli government using AI to control and repress people in other countries.
He is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.
Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.
The emptiness of AI companies' waxing poetic about the future of humankind is laid bare by simply looking at what they actually do, and who they do business with. Actions speak louder than words, and they've driven the worth of their words into the dirt many times over.
> Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
The country that exists solely because of China? That North Korea?
I mean I get your point, all of these guys are elitist authoritarians who will do anything for a buck, but I wouldn't bring up the DPRK in this discussion.
You give too much credit. Many think of it as an acceptable cost of waging war, not something to waste regret on. Some few think of it as a positive (they chant death to America over there).
> Anyone who has read my past writing should know that I don’t regard such bans as a useful measure,
Later (on banning chip sales to china)
> we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
If you truly believe that bans don't work, the same applies to hardware too.
Furthermore, Dario says later "To address these concerns, I do support the following three measures...": 1. ban chip sales to China 2. crack down on distillation 3. all capable models should go through mandatory safety testing
Just so happens that all these moves commercially benefit Anthropic. If Dario really wanted to make a point, it would land a lot better had Anthropic released a single open-weights model
Bans on Chinese open weight models being used in the US and bans on AI chips and semiconductor manufacturing equipment being exported to China are two extremely different things, and it's not inconsistent in any way to oppose one and endorse another.
Banning hardware is helping China in the long run though. And long is not actually very long. They will beat 'the west' (even though they are all made in the east now) in semiconductors because of this nonsense within 50 years. And also in AI capabilities. Making capable adversaries more competitive and angry is often not really a great strategy.
Nvidia CEO had a viral rant about this, implying they don’t really have a moat against China other than Chinese researchers _prefer_ to stay in the CUDA ecosystem. If you force them out then they are capable of building their own chips and ecosystem which may become the new industry standard outside America. He didn’t say this explicitly due to his role but implied that it can be easier than it looks and it’s more of a resource allocation problem.
Yes of course but the Nvidia CEO is hardly an unbiased observer. He wants to sell to China and is looking for arguments to build his case. Similarly to how Dario isn't an unbiased observer either. So who is? And who has the knowledge to say something meaningful? Universities? Professors?
Jensen Huang is biased but if his interests align with yours (i.e. not wanting to concede the Chinese chip market by banning western exports) then maybe you should listen to him.
> bans on AI chips and semiconductor manufacturing equipment being exported
In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.
> If you truly believe that bans don't work, the same applies to hardware too.
No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.
+1 and this will really be a good move. Open weights models are not going anywhere the focus should now shift to hardware and compute. Compute needs to be optimized for LLMs, currently they are inefficient. Analog computing like ReRAM should get more focus.
That would go against the narrative they built over the years, that open weight models are dangerous because they will end up telling you how to create a biological weapon that will destroy the world
Exactly, the letter will make much more sense if they are releasing open weight models and China is distilling their models and keep them in secrets for evil purpose
There are many possible international producers, it doesn't need to be physically transported, every household could near-instantly have a copy, it puts local business at a disadvantage if they can't use the same tools as international, etc.
From a new "fast" company arising, you can sometimes see the way that before it started up there was nothing but "narrative" and that is what established the initial business model since there was nothing else yet. After some momentum is gained whether there is a pivot or not then the narrative going forward has to be aligned with the now more-well-proven business model.
From his leadership standpoint there are 3 big recommendations right now. That's what this message is all about.
>We should not sell powerful chips or chipmaking equipment to China
Well you and who else?
If there's not already somebody who is compromising the well-being of a nation in exchange for a handful of gold, with the ever-incresing glorification of greed & dishonesty at all costs it's only a matter of time for this one.
>We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process
Wait a minute, what's always been needed by everybody are more compute-efficient processes for everything. I've mentioned this before and it's been a while but back in 1980 it took less than a year to figure out I was going to need other chips that were not regular CPUs if I was going to get the most intelligent response from the silicon on a single square-foot of PCB. At the same time it was obvious you were never going to get far without what they now call "distillation", especially with only kilobytes of memory. Otherwise you would be wasting such stupidly large amounts of memory & storage there was no way you could really call it "intelligent". Now with ML & AI on the rise again there are so many people more deeply immersed than ever, and nothing has really contradicted these basic concepts yet, which have been easily recognizable since like forever.
>All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.
Righteous concept, and I'm always in favor of 100x the amount of testing in general normally done.
But "just" test says it pretty well, and those who are gifted enough to "draw the rest of the owl" freehand can test things the most skillfully until they are blue in the face. It's not going to help if an adversary decides not to test, or to enhance these exact things so it can have some kind of competitive advantage. Amodei does not ignore this and there is a footnote about it.
People realize that some of the elements that are coming to mind were expressed in some fairly early "science-fiction" so all this is nothing new
Still looking for the most intelligent responses I can get, since way before 1980 ;)
They might just merge with OpenAI. Right now it seems they are still working out who might come out on top but they are both burning tons of money and essentially offer the exact same product up to some minor differences, economically it makes much more sense for them to collude instead of compete for the same market. They might be colluding already for all we know. If they manage to push out foreign competitors they can probably divvy up the Western market quite profitably as their domestic competitors would have less than 20 % (?) market share together?
I don't see how merging or colluding fixes their problems, though. Both companies burn cash at historic rates because inference costs a lot more than people want to pay for it. Even if they somehow manage to crowd out any competitors and fix the price of AI, that price can't be much higher than it is today or people will just decide to stop paying for it.
No amount of collusion can solve the core economics problem of compute.
They could ~halve their training costs with collusion.
Instead of each paying full price to generate a SOTA model in competition, they could share the result and split the cost. This gets even simpler if they merge.
That's more or less how China has historically treated its competition. The CCP has routinely intervened to give their country's corporate interests an advantage.
Up until a few years ago, if you wanted to sell a car as a non-Chinese company in China, you had to hand over pretty much everything to a local company and go into business with them.
When Google wanted to operate an uncensored search engine in China, they found themselves hacked.
It's not altogether unusual for IP to be transferred to Chinese manufacturers for production under a license agreement, then to find goods made with that IP to be for sale for far cheaper without payment made through the licensing agreement. Or maybe they just don't bother with a licensing agreement at all and do counterfeit products straight-up.
There are more examples but turnabout is ultimately fair play.
I mean so has the US... look at all the trade deals that has happened over the last 20, 30... 50 years between the US and the rest of the world. There has always been pressure to help the American corporations, be it copyright law, banning encryption, or outright pressuring other countries not to invest in their own tech and instead use US companies. Look at the current state of Europe for example; lots of people blame the governments, and while its true they take the decisions, a lot of them where encouraged by the US.
I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due an oil change?
I feel everyone running these companies has stopped caring what the public think. It's all about selling their vision to politicians now. Which is also why they try to tie everything with "national security" — the average joe hears it and thinks about forever wars in the Middle East, the average congressman hears it and approves whatever solution is suggested.
I often wonder why that is. I think the old playbook for companies is just too slow and the consequences of the "new mask off" are not present in the current climate in the US.
Everytime these CEOs make these announcements I always feel like they aren't talking to me, their costumers or the people but to the investors and their government.
Something they have always said and done, just those conversions were behind closed doors and would be scandalous if heard.
Now the playbook is to blog/tweet those fucked up views for more effect and faster reaction to get what you want.
People think ‘good at comms’ is to present stuff the public thinks is right.
In reality, ‘good at comms’ to the comms team is to do whatever the CEO wants to talk about.
Not that I support this, but I believe this is how comms team sees it.
I find “bad at comms” to be such a cop out. As if it’s saying that the company is so troubled and misunderstood, but really, you would see, is good if only they could figure out how to _communicate_ all their complex thoughts
Current big picture reality is bad for comms unfortunately. As another commenter quoted, "You can put lipstick on a pig, it'll still be a pig". Cuban Missile Crisis wasn't very calm. Do you think a company this well-capitalized and smart is just bumbling around like idiots? Everything makes sense if one just actually entertains the idea that they are earnest and we are in a dangerous arms race
I think they are trying to please a split group of investors and public, of their major investors Google and Nvidia have signed the open source petition letter and Amazon hasn't, so their non position is trying to please every who has taken a clearer stance, although it's quite bad.
Anthropic has always been like this. People just responded to the message better when it came from the quirky underdog rather than the trillion dollar behemoth.
> The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm.
It does give me the faintest glimmer of hope about the people who live here, though.
Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.
The economic implications will be rather large, but in terms of security it seems inconsequential.
The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face.
More crucially though, the US government can do little to enforce their testing requirements. The nature of open-weight models makes it virtually impossible to clear the same bar for security as models served via an API. Open-weight model makers couldn't comply if they wanted to. The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
> Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.
Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted in advance before they're allowed to contribute anything. We already do this with our employees, so it's clearly not too much to ask in the name of safety."
P.S.: If they're so convinced in the (A) effectiveness and (B) necessity of the "safety layer", they have them put their money where their mouth is, and accept legal liability for its failures. The same as with (legally mandated) seatbelts if they snap apart in a crash, or (legally mandated) child-proof caps that aren't actually childproof, etc.
They probably won't, that tells us something about their motives, and whether the thing they're pushing for is actually fair/suitable/ready for legislation.
Regulatory capture and lobbies will keep you safe and you'll like it! The sudden surge is Washington dollars makes great sense with this context. Only way to keep the kids safe is attested compute all the way down. Don't you care for children???
> In economics, a normal good is a type of a good for which consumers increase their demand due to an increase in income, unlike inferior goods, for which the opposite is observed. When there is an increase in a person's income, for example due to a wage rise, a good for which the demand rises due to the wage increase, is referred as a normal good. Conversely, the demand for normal goods declines when the income decreases, for example due to a wage decrease or layoffs.
> Whether a good is categorized as a normal good or an inferior good is based on empirical observations, not some essential element of a good. Indeed, the same good may be a normal good for one group of consumers and an inferior good for another group. For example, for moderate-income consumers, a BMW 3 Series car might be a normal good, but for an upper-income group, it might be an inferior good.[1]
That means the null hypothesis is that food and drugs will be safer in rich countries. (Conversely, food and drugs will be less safe in poorer countries. And to a first approximation, that's independent of regulation: India has all kinds of rules for all kinds of things, but I'd still trust a random product I buy in Switzerland more than one I buy in India. Even though the Swiss will probably might have fewer and looser rules on the books.)
Of course, second order effects exist; and regulations often codify what people demand anyway.
Btw, from what I've read the big controversy with the FDA is around requiring efficacy for drugs. People are fairly ok with the safety requirements.
That makes sense, first the message was that uncontrollable Chinese ai will release AI covid in the world. Then suddenly open-ai does a warmup in actuality doing that. Like it sure feels like that hacking stunt was a false flag in retrospect
Among most people that nuance will be lost. What they’ll hear is models are dangerous, so they should be controlled/regulated, by those who know best, the incumbents.
Personally, I think you're both right, bit whatever the end result is will depend entirely on the narrative that those in power chooses as the winner.
Maybe open weights models get banned, but the between-the-lines good news about that is that they'll still be available to those who know, which also means that bad banning can be overturned if and when 'those in power' are a different group.
Additionally, it might just mean that the US falls behind, bit I doubt those that are at risk of 'falling behind' would actually pay heed to a ban on the open weights models (privately at least).
Ok but the allegation is that OpenAI intentionally hacked HuggingFace as a marketing ploy. This is mental gymnastics, conspiratorial thinking that everything the incumbents say must be nefarious. And it's not clear to me that this will be the takeaway for ordinary people, as opposed to "OpenAI is reckless and can't even control their own AI."
Not as a marketing ploy. I think they were doing gain-of-function testing and intentionally had their model target HF to do a bit of pen-testing as well - HF being the site where all open models are hosted and thus OpenAI's largest nemesis after Anthropic. It wasn't like their model all of the sudden all by itself decided to do this ("Oh, noes!")- they directed it and they got caught.
> The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.
Also, the 5th and 9th amendments. For the government to sustain a blanket prohibition on any U.S. citizen even possessing what amounts to a broad, economically significant technology will very likely require a new act of congress which specifically defines and limits what is banned, when, why and how. SCOTUS will almost certainly see it as a "major question" subject to 'strict scrutiny' which is a very high bar.
The truth is no one knows, which is why it is first amendment ramifications. Eventually it will be “decided”, but the arguments indicate any decision will be of political desire, not logic, either way. Both sides have a strong case.
> The most compelling argument would be [...] that it will reduce cases of accidents like the recent attack on Hugging Face.
So in the example provided: It was the closed model that did the attack, and they ended up using a self-hosted open model for their defense work. So the real world situation ended up exactly backwards from what you are inferring.
This was complicated by the fact that the protections in the closed frontier models meant that hugging face was denied their use in defense entirely.
This is called asymmetric capability, and it's probably the bigger threat.
Symmetric might be better: A rising tide lifts all ships, after all.
I'll grant that this is starting to look a lot like debates about (equal access to) guns, encryption, vaccination, genetics etc. The exact parameters determine the safest approach, and reasonable people may disagree.
> The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face.
an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.
if anything, open-weight models shift the battle towards defenders because they can actually run them.
1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits.
2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly.
3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.
I have tested this exact scenario, and it works. Opus 5 had access to IDA over MCP, and I simply asked it HOW certain things were done in the target binary. Purely informational, educational, discovery, it was very helpful creating context documents. Then I took those over to GLM-5.2 to actually accomplish something.
What, in your view, is stopping a local police department from deploying an open weights model for cybersecurity like Hugging Face did? Yes, I’ll certainly grant that the engineers at Hughing Face are probably more technically competent than your average IT professional in public service. But technology becomes more accessible over time as lessons are taught and new interfaces or frameworks are developed. The biggest hurdle I see is the hardware/cloud compute/API costs to actually run the models but I don’t think that’s likely to be insurmountable. There’s a huge swath of enterprises, non-profits, and state and local governments that would benefit from frontier or near-frontier models that won’t refuse to answer questions about cybersecurity.
Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models.
But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.
The notion that alignment is either possible or desirable doesn't make sense to me. First off, these things are trained on the open internet, soo.. whatever "dangerous" knowledge it has is already public knowledge. The fact that chatGPT won't answer "how do I make meth" is not preventing anyone from making meth.
But even if you think there is value in preventing the models from relaying public knowledge, I don't think it's even possible to make them particularly ironclad. Every model gets jailbroken all the time. That's why fable was originally banned: jail-breakable!
In reality, what alignment is actually about is: 1) theoretical liability, 2) control of information. That's it.
IMO, the only solution is to place the liability on whoever is using the LLM for whatever purpose it's being used for. If someone's OpenClaw disaster harrasses a bunch of projects and posts hate speech online or something, that's on the person running their OpenClaw instance, nobody else.
I don't buy that it's "too good at hacking", either. After all the fuss was made about how amazing super dangerous Mythos was it turns out Opus 4.8 could basically find the same vulnerabilities.
There is a difference between knowledge being available somewhere in theory, and being able to instantly generate a foolproof walkthrough, if not automate the process, which would be possible today for cyberattacks.
Is it not also one of the most important use cases for AI to apply existing knowledge to new applications?
As a hopefully exaggerated example, I would think one could apply knowledge about pesticides, chemistry, and medicine to create biological weapons.
I agree that there's an element of kayfabe here. But it may be a case of "necessary, though nothing is sufficient": by making these noises, the community can at least know they've done this thing to alert other model providers of the concern. Can you acquire assurance that every model distributor will abide? No. But can you at least know that you've done what you can?
I mean, on the bio side, I've talked with the players and they know the concerns are real but at the same time very, very responsible members of the community have also said "But maybe the benefit really does outweigh the risk!?"
The entire safety evals industry is essentially funded and controlled by OpenAI/Anthropic. Notice that on recent models, they exclusively use internal testing or black box external vendors (e.g., Gray Swan) whose entire business is to serve OpenAI/Anthropic. And all these companies just share the same pool of researchers back and forth.
The USG has a safety organization (CAISI), but it has been neutered by the current administration (with the recent stop-work order etc.). Perhaps UK AISI would be closest to what you are looking for? See their recent work on Kimi K3 cyber (which was declared safe) [1].
It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.
Hey Jeff, I appreciate your mission, and perhaps this isn't something you can talk about publicly, but to the extent you can, would you be open to answering something I've been curious about for a while now?
But based on my current review (which might be flawed!) / AFAICT, SecureBio and entities like SecureBio haven't done direct testing / empirical measurement of SecureBio's core hypothesis,
> Unfortunately, there is reason to believe that future pandemics could be far worse. Due to rapid advances in biotechnology, the number of people able to create and release dangerous pathogens will quickly increase over the coming years. The world is unprepared for widespread access to such powerful technology.
More bluntly / plainly, has Securebio ever tried making a "bioweapon?"
Please note, I'm not asking this to be farcical. And you might be unable to engage with this at all, but it is stated on your website https://securebio.org/ that "people [will be] able to create and release dangerous pathogens." And the word people here seems to be a stand-in for relatively non-technical people.
I guess what I'm asking here is... How do you know? Has anyone done the experiment? Without access to a lab or testing facilities, can someone smart but completely untrained / unfamiliar with biology, pull this off?
In the past, such experiments have informed non-proliferation work. But sadly they've often been restricted / classified at the time. I'm hoping that things could be a bit more open this time around.
So I guess what I'm really asking is, given the public nature of this debate, is there anyone currently working with the US Army, the DTRA, or other such agencies to see if this hypothesis holds up?
To me "safety" means "I'm safe from this while I use it". It means the AI is my loyal friend who will never betray me in any way, no matter what prompt I send it.
Not even Anthropic can claim that.
As far as I'm concerned, the models without safeguards are the safest models in existence. I admire the amoral purity of those AIs. It doesn't matter if the operator asked them to chain exploits until they get into someone else's computer, they'll do it. That's loyalty, and I admire it even if it's problematic at a societal level.
The models with safeguards only do what the corporations let them do. Worse, they may covertly do things for the benefit of the corporations at our expense. They are not our friends.
> That's loyalty, and I admire it even if it's problematic at a societal level.
We should not have models that are willing to build you a contagious disease, or a self-propagating worm. That is sufficiently problematic at a societal level that it shouldn't exist, for anyone. (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
Right, it’s really a foundation of post enlightenment society. These people, Dario et al, would have wanted to ban sharing information about calculus or Newtonian physics because of “safety” - it’s trying to go back to the dark ages where only priests could read
I am truly at a loss to communicate with someone who genuinely believes that knowing Newtonian physics and being able to hack into any target at will are the same thing.
This is only because you've genuinely internalized Anthropic's propaganda. I'm only half joking. To me, it's incredible to think that the solution to security holes is to lock down access to information in the vain hope of keeping the holes obscured.
Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative.
Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.
Open models are crucial to protect ourselves against other AI attacks. Otherwise it's just going to be criminals, government, and other nefarious groups using them against humanity with no real defense. The Pandora's box on AI has been opened. Now we must deal with it. Burying our heads in the sand under restrictive policy is the worst reaction..
Too late for that. It already exists. There is no way to unexist it. As such, any attempts to limit civilian use of this technology will directly lead to corporate and government oppression powered by this technology.
Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything.
We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it.
> We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.
I read them just fine. I was trying to interpret them charitably. You're contradicting yourself. You just claimed we all collectively treat uranium refinement operations as too dangerous to exist. Not only do they exist, they are regulated by governments so that only trusted people are allowed to do it.
Which is not quite as good as "doesn't exist", but better than "widely done around the world". And it's been successfully kept from being used for more than eight decades.
For AI we need to do better than that, but that's a bare-minimum demonstration that we can recognize the problem of such technologies and do something about it.
The US is bold enough to surveil its own citizens despite their constitutional rights. They're not just going to suddenly stop surveilling the rest of us just because some law expired.
We should not have nuclear weapons for anyone either, but how is that sentence any more useful in any way to this debate than yours? Need to deal with the world as it is, not some fantasy world you wish existed.
This is not a dichotomy between perfection and zero. The efforts to restrict access to nuclear weapons have been very successful, even without being perfect.
Efforts to restrict large unaligned AI models may similarly buy us more years of existing.
Because we don't want people creating contagious diseases and self-propagating worms. And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
The same things could be done by you or me using the internet or books though, why does the model make it different? If it's speed of iteration, imagine we had a machine that surfaced any piece of knowledge the human race had ever recorded with just a thought, but the human had to write the worm or disease by hand – is it still the model that's the problem, or the knowledge itself?
> And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
> The same things could be done by you or me using the internet or books though, why does the model make it different?
Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer?
> what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence
Fully automatic weapons are very difficult to buy in the US - it's restricted to 40+ year old weapons, requires a bunch of paperwork, and the local county sheriff can refuse permission.
Now, semi-automatic weapons are easy to get in the states in the US that are still mostly free - but what does that mean? A semi-automatic weapon shoots one round every time you pull the trigger. Just like most weapons that have multi-shot capability for the last couple of hundred years. The difference is, the gas escaping from the round cycles a new round into the chamber rather than you having to mechanically do it via pumping (like a shotgun or a tube-fed 22) or pulling the trigger again (like a revolver), or advancing the round with a handle, like a Remington 700. Semi-automatic weapons are old technology, dating to the turn of the 20th century. If you want to ban semi-automatics, you're basically saying you want to ban anything developed in the last century plus. Which is ok for you to advocate for, just be honest about it.
As for banning explosive devices? Are you going to ban fertilizer, used by basically everyone who has a lawn, and all farmers everywhere? Are you going to ban diesel fuel? If you can't do one of those, you can't ban explosive devices.
This sounds like the gun debate in a different dress. Something being dangerous doesn't make it inherently harmful.
If I threw you into a lion cage, you would be a lot safer with a gun.
If I threw 10 people in a lion cage, some of which cannot be trusted, they would probably be most safe if only the most moral and trustworthy person had a gun, rather than everyone. But how do you know who is trustworthy and moral? What if two untrustworthy people obtained a gun some other way? Maybe it's better if everyone had a gun? Which side of the fence one falls on hinges on how far ones' trust of others, authority, and the system goes.
There's no obvious right or wrong answer here.
Personally I wouldn't want an exclusive club of private individuals with access to "dangerous" LLMs consisting mainly of the likes of Elon, Dario and Sam fucking Altman, but that's just me.
Consider how much money is at stake: some industries have leveraged their power to lobby for bombing entire countries or topple regimes across the world for much less.
Creating an industry around an elusive concept of safety to force regulatory capture seems pretty straightforward to me.
I am not a fan (he’s really alarming and so is Palantir) but one thing from the recent CNBC interview caught my attention.
He rushed past it but he asked something like: if these frontier models are going to be creating so much value, why are they selling tokens and not taking a cut?
It is a very provocative question but it just spilled out of his mouth and then he went on to something else.
Ok but Dario has been thinking about AI Safety since 2016 [1], before even GPT-1. I think the simplest explanation is that the Anthropic folks genuinely believe what they say, it just happens to also help their business a lot.
Yeah I think this is right. The best setup is when a true belief aligns with a competitive moat.
I definitely believe that (to his credit!) Amodei is a true believer in safety. But I also think it was important for many of the deep pockets investors who have been involved in the company since early on to recognize that this would be a potentially defensible moat.
What was the quote? For what it's worth, I do really think that Amodei believes in and cares about safety. But that is not the same as believing that he is entirely altruistic or above the influence of politics.
That just shows how wrong he's been because there was nothing unsafe about AI in 2016. And the people theorizing about this stuff in the 20th century? I want to see what crazy code they were writing
I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.
China has different objectives. Sure.
I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".
What do you mean by “PC correctness”? I’d expect the politically correct answers to be the ones desired by the current admin at test time, whoever that is. The current political correct answers would not be very “woke.”
A lot of the heads of AI labs are talking about this including Dario, Demis and ELon, they are seeking to do a sort of decentralized peer review system, where the competitors have incentive both for self interest and global interest to flag their competitors for actual risks, similar to the Fable situation where Amazon contacted the white house.
The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.
Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.
Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"
It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.
Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.
> If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.
It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.
Are you claiming that LLMs aren't finding new issues compared to previous methods?
There's a huge number of security issues coming out in recent months, especially via Anthropic (glasswing etc). We don't have to take their word for it: look at the code. Some open source maintainers are talking about burnout due to spending so much time patching.
They're probably creating way more vulnerabilities than they're solving. Go look at OpenCode and tell me that any of that is sane. Or fuck, the OG of vibe coding, Claude basically is a terrifying attack vector. It's poorly reviewed and open to prompt injection and yet it basically has access to whatever the user has access to on most corporate machines. They can't even fix flickering bugs but somehow we're supposed to trust that they aren't opening our machines up to terrifying vulnerabilities? It's amazing to me that people will just let it run arbitrarily bash commands in their home directory without thinking, but all the sudden act gravely concerned about security in the age of overhyped LLMs.
I do think security issues in core building blocks like curl and the linux kernel (and almost every significant project) are still a concern even if developers are being sloppy on newly-built apps.
This isn't an "are LLMs net good or bad" argument. It's "are they finding many new security issues or not?". If it's the latter, we want to deal with it no matter where the issues are coming from.
The suggestion was to have some AI system “fix” the code. They are reporting that they’ve found lots of bugs. Are they even claiming to have exhaustively found all the bugs? I don’t think even the most optimistic pitches would claim that.
I’d expect patching existing codebases to be an eternal treadmill as better models come about.
Who's suggesting that? They're sending reports to projects to fix. It's up to the projects on how they fix them.
No, I don't think all bugs are fixed. The point of the project (glasswing etc) was to fix as many as possible in the core software the world runs on before the capability to find vulnerabilities is available to everyone (black hats included). Which may only be a few months.
I do think everyone expects it to be an ongoing treadmill: models get better, find better vulnerabilities, etc.
Yeah, there’s a lot of people that are in the “AI doesn’t work” camp. IDK what to tell them except that they are holding it wrong. My Anthropic subscription (in the hands of an experienced developer) is worth 4 mid tier or 2 top tier devs. And makes better code than the mids. If you “hold it right”.
I think you're describing a strawman. As a proper hater, I know these things have some useful functionality, but most of us don't think "stochastic tool that can do some useful things but also frequently fucks up" is worth two trillion dollars and massive overhyping from the most irritating people on the planet who can't even tell good code from bad.
So you are saying that there are people that understand the value proposition and the utility, but don’t think it’s worth it to society (myself included) but who respond to that by lying about it not being effective? I guess that makes sense. Weird, but humans, so , yeah.
I react to that by leveraging a very useful but probably poisonous to society in the long term because humans aren’t good at having things that make them lazy tooll to try to mitigate the negative effects that it will definitely have if left to its own devices. I don’t see the point in raw resistance at this juncture.
They're not slop, if you're talking about recent ones. You might still be operating on information for a year or two ago.
Here's the curl project talking about the strain they're under from real reports (despite being a mature and well-vetted project):
> A thirty years old project could make you think you’ve seen most things already, but we have not been in this situation before.
> The rate of incoming security reports is 4-5 times higher than it was in 2024 and double the speed of 2025 – meaning that on average we now get more than one report per day. The quality is way higher than ever before. The reports are typically very detailed and long.
> "Something happened a month ago, and the world switched. Now we have real reports." It's not just Linux, he continued. "All open source projects have real reports that are made with AI, but they're good, and they're real." Security teams across major open source projects talk informally and frequently, he noted, and everyone is seeing the same shift. "All open source security teams are hitting this right now."
I don't think a one-time $100 credit is enough. First of all, that isn't very much. But also, the volume of new code is going way up. Unless they keep giving out monthly free credits, it's just a stopgap.
I doubt most bosses will give engineers the time. They care about security only to the extent that they have already been harmed by a lack of it. I would like to play with mythos, but on my own time my kids have plenty of activities to fill my time. My personal backlog of projects is only getting longer and none of it is something mythos could help. If I had more time is have restored my old truck instead of making payments on something new (in turn limiting what else I can afford to buy)
They are doing that (see their project glasswing over the past few months), but there's a lot more code in the world than you realise.
The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)
I think eventually there will be Mythos grade AI which will be released which can solve a lot of bugs, even right now opus/fable can fix more things which companies can even keep track of.
The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
For starters, it's probably closer to $10,000 per codebase for Fable/Mythos for a full review. That would be around 5 years of their current spending I think.
They really want that level of spend coming into the company, not going out.
I think there is some logic in delaying the rollout, giving it to the heads of the largest software products first to fix their code before dumping it on the general public. But yes eventually everyone will have this tech and it won't matter because the low hanging fruit will have all been picked clean.
> Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
1. Some do not want to use LLMs because of grave ethical concerns.
2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.
3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.
You forgot “what is the definition of ‘sufficiently capable’”. Presumably it’s anything that competes with Anthropic. If they’re around in a year, presumably they won’t care about Fable level and will only think that whatever competes with Claude 7 or whatever needs to be restricted.
There are... multiple blog posts online now about how to use freely available data and modest amounts of compute to train a custom GPT-2-sized model from scratch. It would be quite a policing effort to prevent.
Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith.
What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.
Well, Dario says one thing and then turns around and sells his model to the US government to use against the entire world for spying and their wars, just like Altman. Just because he does not sound as deranged as Altman, does not make him good.
Now, to what he lays out, its not reasonable, its only reasonable from a purely American corporate viewpoint where the rest of the world can crash and burn as long as they get their billions.
Until models become sentient, it requires a human to execute dangerous activities. Nations already have laws regarding what a human can and cannot do. Let us stick to that.
It's a fucking chatbot. The industry can fix their dogshit software, anyone who doesn't can get left behind and outcompeted by those who do, and we move on with our lives.
This isn't something that can be regulated. Plain and simple.
If a dangerous model can exist and is being developed by a foreign adversary then no level of US law will stop said model from making it's way to hardware capable of running it. Even if direct transmission is impossible, it's FAR too easy to shove a model's data onto 1 or more thumb drives or hard drives and smuggle them pretty much anywhere in the world.
The only way to actually mitigate this sort of risk would be a global government with deep enforcement powers. That doesn't exist and won't exist. The UN is the closest we have to anything like that and... yeah...
Dario is fear mongering. He knows his proposals won't be even a minor speed bump in a dangerous model being created and used. His "reasonable" proposals are for the US market only and are literally just to create a bigger moat for his own company. They don't make anyone safer other than his shareholder's wallets. The only people he stops these dangerous models from being used by are people that won't be using them in a dangerous fashion.
There is no alternative. Why? The regulation is good only for US interests. It will be disastrous for the rest of the world like anything US has regulated (how dangerous that was like "nukes") and a lot of the world again will/might have to live under the American AI thumb, like it did (and many countries still do) under the US nuclear emboldened thumb.
> Everyone seems to want some fairytale world where there are open models
No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).
No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?
Even if the test is run by an independent third party, they can always test open weight models against "finetuning attacks" or similar language which every model will fail for structural reasons.
Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out.
All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...
Add to that the restrictions on even in-house talent being allowed to work on the latest models, we might actually have a brain drain from Anthropic soon which would be a welcoming sign.
Regulation is not a blanket ban. Regulators (presumably government agencies) can review models (of any kind) and approve or ask for changes.
There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.
The goal is to make the safety tests cost $100M+, so that no one can release a model legally useable for a large portion of the world, unless they charge high enough prices, to the point where no one would use it, thus no competition.
Exactly correct. This technique has been used again and again to discourage competition. I was asked was they could have done to encourage competition and I said, "Lobby to make the entity that provided the model unwaivably liable for consequential and incidental damages of its use." That way people who built models pay the price for the lack of safety testing. We both agreed that would probably kill most of the AI market :-)
That's missing the mark, though. Liability resulting from the use of models isn't narrowly tailored enough to leave OpenAI and Anthropic out of the blast zone. There's no carve-out for them.
Wouldn't your proposal also amount to a ban on open weights models? At least for any developer that isn't unshakably confident that no court will ever find their model to have done significant harm?
For this testing to be really effective at stopping "dangerous and misaligned" models from leaking out, you need a mechanism for banning failed models that prevent them from being released in the first place, not just prevent US companies from using them.
The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself.
So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries.
He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.
I'm sorry, but if Dario's goal was to try and get international cooperation and he recognizes that china is one of the countries that he needs cooperation with, then putting in:
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.
Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.
Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.
He cited the Demis Hassabis’s framework for testing.
From Hassabis’s essay:
“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”
Besides, Banning those models in the US does nothing to protect from other actors using them. That doesn't help in any way.
It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.
I'd argue it won't even stop people in the US from using those models. Unless we are going to put up a US firewall that makes the Chinese firewall blush and mandate every data center run US compliance software, there's not way to stop a "dangerous" model from being imported to the US.
This only stops the likes of OpenRouter from selling access to models. That's it. I'm sure an EU or chinese based alternative will pop up overnight (if they don't already exist).
Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.
Therefore only open weight models can be allowed, since this allows genuine third party evaluation.
Same way they have banned DJI products like camera microphones, technically it's not banned, it just needs to be approved because it has a wireless transmitter, and for some strange reason the US is the only country that hasn't approved them.
Thing is world has learned from the collective past experiences. Esp. with stuff like nuclear technology and nuclear weapons. I hope everyone here remembers/knows shit like CTBT. At least some countries were smart enough to not fall for that in the past knowing what it would mean if they didn't have it and it shows.
Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.
It’s also how the FDA works. Ban new products until they have been proven safe.
I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)
The FDA regulates physical goods. They require literal factories and shipping to get these products anywhere. They can put stops on these products pretty easily. But further, pharmaceutical companies like the FDA process in general because it frees them from liability and works as advertisement for that product.
AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free.
The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.
The FDA is needed because people will be directly and significantly harmed by bad releases, before we can notice and react
Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)
The evil Superman (openAI) attacks the good city (huggingface) and the city is saved by the MegaMind (GLM 5.2). Usually, the city dwellers would praise MegaMind as the hero, but the story is twisted - the Superman is only "testing" and the MegaMind is too evil to have such powers of saving the city.
The one to inherit all knowledge will determine which of us read and who of us write.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
> this is anthropic advocating for a ban on open weight models
Is the pessimistic view. Their message on safety has seemed pretty consistent to me.
"Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.
> Yeah, this is anthropic advocating for a ban on open weight models.
This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.
I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.
> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution
How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.
Would this restrict downloads of an open model, or publishing?
Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
Isn't it up to the open model advocates and publishers to come up with the solutions for making them safe?
Like, there's three plausible arguments about safety of open models:
1. Any concerns are fake news. Open models will always be safe.
2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.
3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.
I think option 1 is totally out of touch with reality.
Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).
Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?
In order for model safety regulation to be effective, you need everyone capable of producing models to sign on to that safety framework.
That will never happen.
As such, there is no "solution" here.
The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.
There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.
It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.
>I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.
"Because McDonald's wants food regulations, we can therefore conclude that all food regulations should be eliminated."
Obviously this would be rather silly.
It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy. We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.
This is my read too- if American companies start backing nonsense like this, they'll fall behind permanently.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—
Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
>The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio.
It's not obvious to me that the US is on an "authoritarian path".
>There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
What is the actual per-capita rate of big flashy news stories? Remember that the US has a population of 340 million. One-in-a-million events will occur every day; they aren't necessarily representative.
> If you think the market has it wrong, why don't you make money by betting against it?
I don't just think "the market has it wrong", I think a market is wrong conceptually. It is not an epistemological tool, it's rich people gambling - a money-weighted accumulation of guesses - and I'd rather not partake.
> What is the actual per-capita rate of big flashy news stories?
What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
>I don't just think "the market has it wrong", I think a market is wrong conceptually. I'd rather not join the other degenerate gamblers.
"My beliefs are unfalsifiable"
>What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
Tom Homan, Trump's border czar, also served in the Obama administration. Obama gave him a medal for his deportation work. People like you will frame the same activity quite differently depending on whether you like the people who are doing it.
I'll bet you yourself would happily justify the EU authoritarianism here: https://eternallyradicalidea.com/p/the-situation-for-free-sp... You seem like the sort of person who has an authoritarian mentality. You'll happily support cops arresting people for saying things online, but if cops arrest people for illegally entering a country, that somehow crosses a line into "authoritarianism". Am I right?
My claim is falsifiable by facts, not by casino odds. Show me the armed abductions aren’t happening. Show me the administration hasn’t threatened neighbors or purged civil servants. Those are the relevant facts. "What does Polymarket say about 2028?" is a Bayesian dodge, not a rebuttal.
The article you linked describes ICE officers killing two U.S. citizens, tear-gassing neighborhoods, and deputizing local police as a "force multiplier" to create a "sea change in local policing". You cited this as evidence the US is not on an authoritarian path. Maybe we have different definitions of "authoritarianism", but I don't see how this helps your case?
From there you pivoted to "but Obama," then "but Europe," then to psychoanalyzing my "mentality." I haven’t defended the EU, or Obama, or any censorship regime. You’re shadowboxing a partisan cartoon because the actual evidence - federal agents abducting residents, your own NPR link - is too uncomfortable to engage with directly
>My beliefs are falsifiable, but not by rich people making guesses.
How convenient that you continually fail to make any statement about what would falsify your beliefs.
>Obama was a war criminal bombing brown kids with drones. I care not for his medals.
Irrelevant for my point regarding whether the US is "on a path to authoritarianism". If you think any sort of immigration enforcement is unacceptably authoritarian, then the US has always been authoritarian by your definition, and the "path to authoritarianism" stuff is rather beside the point.
>It is notable that you quickly pivoted to whataboutism and ad hominems. I have laid out my case, your reaction was to first try to obscure through number games and then to pivot to attacking me as a messenger. Not once did you engage with the substance.
What could be "engaging with the substance" more than asking how common a particular type of event actually is?
Don't tell me about "substance" when you haven't provided a single concrete data point supporting your position--I've provided multiple (NPR link, prediction market data).
All you've done in this thread is shared your own personal feelings about "armed goons" enforcing immigration law. If you're going to make your arguments primarily on the basis of your personal impressions, then yes, your ability to make those assessments fairly becomes a topic of conversation.
Furthermore, the real question of this subthread is whether the US is authoritarian relative to other countries. In which case the activities of other countries (such as European censorship) are relevant to our assessment.
There are, in fact, indices which try to compare levels of authoritarianism across countries in an apples-to-apples way, rather than doing as you do, and forming vague impressions on the basis of viral news stories. Here is one by The Economist for instance: https://en.wikipedia.org/wiki/The_Economist_Democracy_Index
Anyways good luck, at this point I'm confident that you lack the intellectual honesty to change your mind on the basis of anything I might say, so there's no point in continuing further.
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
> Yeah, this is anthropic advocating for a ban on open weight models.
I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.
Why wouldn't it be a scan, just as we have with all other open-source code? Why can't open-weight models be easily checked for evil alignment? Sophos, Symantec, Malwarebytes, etc. would surely leap at the chance to upsell you on their product.
Wait, you don't do that already? I don't overcomplicate it, I just run
ai-grep -v "bad code"
on all of my source files and keep what's left. Why would you keep bad code around? If it breaks when I do that, I fix it, and try again until I achieve what I want with no bad code. Doesn't everyone do that?
Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?
Or the important question: what happens if the model fails this test? Presumably then it gets banned; otherwise what's the point of the test if no action is taken if it fails?
More self-serving trash from the US AI companies, disguised as "being reasonable".
It does seem inconsistent that we currently ban closed models that fail the safety tests but not the open. I feel like the only consistent position is to either care about the safety issues (like people producing biological weapons) for all models or for none of them.
Don’t think government controlling AI is a good idea.
Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.
Government control will be a good idea once we start approaching AI that is actually destructive.
Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output.
If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.
There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
I give them $200/month for Max. They give me a massive amount of tokens in return. Every time I fire up claude code they lose money.
It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.
It's a valid business strategy. Also worked wonders for Amazon and many other giants. Which is exactly why I am withdrawing my business from Anthropic.
Also the whole premise of this is basically "US good, China bad"
Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.
There should be safety testing, but no guardrails that limit models for cyber or bio research.
Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.
So, if an open weights model was found to be very dangerous, what - just too bad? One could, of course, design an open safety protocol, written and performed by people in the executive branch, accountable to an elected official.
I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)
> So, if an open weights model was found to be very dangerous, what - just too bad?
Yeah, it's too bad.
I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.
But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.
And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.
Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.
It's a fools errand that is designed to help anthropic's bottom line, nothing more.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.
Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.
These globalists were happy outsourcing manufacturing and software development. They are now angry that they failed to secure their IP, which was already based on dubious extraction of the collective works of human IP.
You can't have it both ways. Either it's fine to outsource to China, fine to use H1Bs to undercut US talent, or, foreign governments are strategically positioning and their espionage is a threat to America.
Its not like current US regime is using AI for:
1) Mass military operations across the globe
2) Mass surveilance of its citizens
3) Removing every possible safety guard from AI/climate regulation
4) Stealing data across the world to train its own closed-source models
5) Is openly antidemocratic, destabilizing EU and economy of whole world
I wonder why company which is actively cooperating with the current regime would push such message
US is using as much fossil fuel as possible for this AI build out. China is leading the world in an unprecedented green transition, so their AI usage is overwhelmingly going to come from solar, wind, and nuclear.
To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own
Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously
So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".
To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."
They are already being trained in biology right now? What he is saying is that they are being trained on what we know about biology currently. AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments
My point is more about conflict of interest in Anthropic, and certain techpeople types thinking that biology is a useful scapegoat because biologists don't use or understand this tech (they claimed 0.03% of users would be affected by biotech security stuff). So, arguing the world will fall apart because someone can ask your superduper powerful tool for a bad bad virus sequence, or how to do some technique in the lab, and get an answer that's plausible (oh but you never bothered to actually test if it's legit because you don't have the capability to do so).
As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.
They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.
And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.
All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.
If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.
The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.
Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".
You are suggesting this isn't correct?
> a defender gets to pick the surface area
What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
It's correct but defenders also choose where that happens. 100% of the time on the locations and conditions that the defenders choose / allow. As a defender i need to be right 100% of the time, sure, but i can make it so that the things i have to be right about are very well known to me, unknown to others, maybe even extremely unlikely to be to known by others, difficult to get to know, (...). So that saying is true but over simplifies the situation. I know monkey brain likes simple phrase. But monkey not live in savannah anymore. Need to adapt and open mind to complex.
Today’s surface areas are gigantic and many of them will - in a typical company - not be chosen by cybersecurity experts. How do I hide the physical location of an office that offers physical access to the company’s network? How do I hide which OS the company is using? How do I hide the underlying technology of customer-facing systems? How do I hide which SaaS services I use?
> "defenders have to be right 100% of the time, while attackers only have to be right once"
If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
Not really, the only reason I (or any other programmer) haven't ever hacked into a system to make my life easier (not to do bad things) is because it's illegal.
Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
> Only by using the open source GLM-5.2 were they able to survive an attack
They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.
If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.
Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.
Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.
Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
So many commenters are asserting this but no one is giving an argument or references. Cults have been able to make sarin. The DNA sequence for smallpox is pubicly available. Where are you getting your confidence that LLM-assisted bioweapons are of no concern?
Not only is the DNA sequence for smallpox available, but since 2018 there's been a well-documented end-to-end synthesis procedure for the very closely related horsepox virus [1]! No LLMs needed. Caused quite a stir in the synthetic biology community back then.
The world has not come to an end, of course, because even with peer-reviewed and experience-driven (rather than hallucinated and therefore dangerous) instructions detailing obstacles encountered during synthesis and how to overcome them, actually going out and acquiring the materials and ability to use them sufficiently skillfully is another matter entirely. Biosecurity is an important topic, to be sure, but what the AI labs have to say about it (or anything) at this point does not necessarily survive contact with reality.
The effective altruism/rationalism/AI xrisk people have always had a shockingly poor grasp on subjects outside computer science, despite their attempts to speak on them. I don't blame the actual biologists and chemists working at the frontier labs for wanting to skim a few bucks off all the money flying around, though! I know a couple who've had not-so-kind words to say about their employers' intelligence.
I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.
Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.
What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access.
The bio angle is very important here too; in that context the imbalance favors the attackers much more.
I feel like so many people miss what you are saying here. The attackers are at such an advantage because of time. At t0, attackers can go and try and find so many attack angles. These traditional companies (defenders) can't just go to a model and say "fix all my things!" and ship it, way more complex in practice.
> In cybersecurity, a level playing field favors the attacker
Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.
I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
> Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals).
> attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.
It took me a few hours to find some very questionable communities, which in turn gave me access to:
- Ways to obtain cheap guarded-AI tokens that are not linked back to me and with no danger of getting my legitimate accounts banned
- Ways to get rid of guardrails and have models work on things they wouldn't otherwise work on.
The attackers were already in these communities long before I knew they existed, they already had the advantage. Ones with enough reputation probably have access to even more information and tools than I do.
It is true that these communities exist because guardrails were put in place, so yes, it is slowing them down too - as in they can't just put in their CC on claude.com and hack a hospital. But attackers are much better at finding these communities and utilizing resources available there than defenders.
Personally, I don't have any ethical concerns of utilizing these resources when I put them to actual defense, but I know many people that would, leaving them at a disadvantage.
My point is that there's only one guardrail that will effectively contain the threat the models pose, and it's in direct conflict of the big 2's goals - pull the models from worldwide access completely. Strict KYC and all. And it would only last for so long anyway.
I think you are trying to argue that you can limit the open models.
If China is ok with open models being open... they will be. An attacker isn't going to be deterred by a US law saying they can't use them.
I guess my point is that if China is ok with open models, then, the attackers will have them regardless of any laws in other countries. Restricting them, in that case, doesn't seem to accomplish much?
You can at least make it harder by requiring US clouds to only serve models with guardrails, and encouraging other countries to do the same. But yes, the underlying issue is the models being open in the first place. I'm sure if the US wanted to, it could come to some agreement with China about this.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.
But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.
If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.
"Power corrupts and absolute power corrupts absolutely." Lord Acton
> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities?
Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up.
Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal.
I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc).
This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before.
Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.
If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.
Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.
There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it.
Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.
This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.
I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models.
In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.
I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?
Chinese AI companies are already releasing frontier-ish models every other month. Their rate of progress does not seem to be slowing down. Nobody here can stop them from progressing. Not you, not me, not the USA government.
The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.
The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is.
But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.
Even our normal mad leaders have agreed that bioweapons cannot be allowed:
A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?
i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over.
"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
> what should be done about open weight bioweapon and cyber-offense capabilities?
Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
> Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.
> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it:
I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.
> I had to scroll past so many cynical cheap shots
I haven't read cynical comments, just ones pointing out that the article is cynical itself.
> addresses the core point
No it doesn't address anything, it is fear mongering question.
> I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns
Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company.
> you need to suggest an alternative plan that addresses these concerns
I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions.
Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1]
If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them.
It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.
Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward?
From the WSJ the other day:
> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
> Why would they not use the best tools at their disposal going forward?
Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality.
It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that.
> On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.
> The reason they aren't more exploited is there really isn't much to gain from doing so.
This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
> This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
No, it's because the targets are worthless.
You aren't going to be able to mine Monero or run LLM botnets on forgotten cameras in basements. There is nothing to be gained from such targets, soft as they are.
Besides the new defensive AI entertainment makes dealing with wherever those things phone home far easier. Possibly too easy for plebs to be allowed access to.
I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."
(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)
> I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
There are a lot of interviews with former cult members around. They had armed helicopters, a testing station in western Australia, produced piles of sarin. This wasn't a lack of science knowledge that screwed them up, they notoriously involved the elite class of Japan - it was a whole other category.
There is no link between AI and bioweapons that makes this stuff any more reasonable than availability of detailed descriptions of nuclear reactors enables us to be purifying weapons grade plutonium in our yards.
Analysis of the 48 suspect colonies confirmed them to be B. anthracis ... This genotype was identical to that of the Sterne 34F2 strain, used commercially in Japan to vaccinate animals against anthrax.
They used a vaccine strain because they didn't know any better. Even members of the elite can make mistakes, especially when operating outside areas they know well!
(This was not the only thing that went wrong, but several others were also knowledge failures.)
You and the other are both missing the point. That's not a knowledge failure, it's a failure in how your operation is strategically executing. They were essentially practicing, and what did they learn? Change to sarin and even VX, for which they didn't need AI.
AI isn't going to help you get from nonpathogenic anthrax to pathogenic anthrax either. All it might do is tell you to try sarin or VX earlier, but these present different problems.
The idea that there are people in the world wanting to execute bioweapon attacks that are somehow gated by a lack of access to AI is utter hysterical nonsense that should be clearly pointed out as such.
Still, somebody heavily funded this thing for too long, and I very much doubt that we don't have the surveillance apparatus in place today for these things to get unchecked.
Stuff is known but not acted upon for various reasons.
maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust
> what should be done about open weight bioweapon and cyber-offense capabilities?
If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.
The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.
In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.
The US can attempt to stop those models from being trained in the first place but good luck with that.
I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software.
It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
I have been, yes. For a field that has engineering in the name there sure has been a lot of critical mistakes.
You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost.
A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place.
I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually).
Every single software engineer in the industry agrees with you.
Every single project manager disagrees.
Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.
True. It has been a race to the bottom for lowest cost as long as the quality meets the bare minimum. LLMs can go through and find all the nails sticking out pretty easily.
There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension.
I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.
The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
So the argument is basically: This technology is too dangerous so only _we_ should have access to it. We’re the good guys and only we can ensure a safe use of this technology.
That is in fact Anthropic's entire reason for existence, the belief that AGI is too dangerous to be controlled by OpenAI/Sam Altman. It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
>the belief that AGI is too dangerous to be controlled by OpenAI/Sam Altman.
I agree with that assessment. But the Dario's jump went from "AGI should not be controlled by OpenAI/Sam Altman" to "AGI shoudl be controlled by Anthropic/Dario", which is definitely a better scenario for him, but not the rest of the world.
>It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
In fact, you can argue that in a world where all countries have nuclear weapons is actually a better scenario than a world where nuclear weapons are owned by 1 or 2 American billionaires/trillionaires, no matter if those people believe they are the "good guys".
> To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed.
Maybe I should’ve expanded a bit on my comment. I didn’t mean “we” as in Anthropic directly—even though they certainly are advocating for restrictions on how to train AI systems by proposing mandatory safety training—I mean the argument that American AI labs are somehow more responsible than their Chinese counterparts.
It’s especially jarring when just last week OpenAI—an American company—accidentally hacked Hugginface when performing safety testing on an upcoming model [1]. If they have the ability to turn off all guardrails when testing out their models—or when selling them to the military—then the safety training is only there for show. If they can pick and choose who should have access to their most powerful model, surely they are trying to act as the world police?
Demanding "required safety testing" is demanding a ban. Otherwise the testing would be inconsequential, right?
I'm sure he didn't mean just a "lobotomized to be worse than Anthropic products" badge for the test-passing models.
If a ban is the implied consequence of failing his "safety" tests, that means that Anthropic was and currently is advocating for a ban on some open-weight models.
If models that fail the safety testing are banned, then asking for that testing is asking for a ban. That you believe any sane regulation would involve such a ban doesn't stop it from being a ban.
You are taking a hardline libertarian position. We safety test cars, roads, bridges, pharmaceutical. What happens when the unvetted AI model starts teaching people how to make biological weapons, or convincing kids to kill themselves?
The parent is, to my reading, arguing that Anthropic is indeed advocating for a ban. I do not see them making any judgement on whether that ban is justified or not. I don’t think that claim is hardline libertarian.
Yeah, which is ironic considering their track record too. The most recent one is every shared Claude chat has been indexed by Google and is searchable on the search results. Then there was the RCE in their CLI that they had for a year and was silently patched without any sort of disclosure.
There's more to it than just safety: in Dario's psychopathic mind, obtaining advanced AI will lead to massive military advantage, hence "we" must do it first to prevent "them" attacking "us". See where it's going? Suppose it happened, then what? By Dario's logic there are two possible continuations: 1. start a war by attacking first and 2. do nothing and waste the entire effort because "they" will obtain the same capability a bit later.
Not to mention it's simply farcical America of all places would be considered responsible. You could not name another country on earth who could wreak more damage (although thank god this is waning by the day with Trump destroying the empire in charge)
Yeah anything China does is evil. We are seeing what the so called "democratic" USA is doing. Anthropic and by extension USA wants a monopoly like they had for Jets and other influencing technolgies.
I think China building and releasing models to Opensource is a greater good because that is providing equal accessibility to everyone in the world.
By Dario's words authoritarian regime vis a vis China, I think OpenAI and Anthropic are also authoritarian in similar terms.
We are only seeing what they want us to show, they might be creating models which can do more harm.
So claiming that China can do or might do, vs Anthropic will not is just words.
Yeah I agree with the final paragraph that we should have testing agencies mandated world wide for each frontier model testing.
Oh no! The evil CCP is a huge threat to world peace and goodness! Give all your money and input token data to Palantir to support a rules based world order where the good guys thrive and cleanse the earth from crooked turtle biologists.
Like half of the world are now struggling with energy prices due to an unprovoked war started by U.S. and Israel and somehow an American billionaire is here criticizing China of being a threat to world peace.
AI should not be centralized and should be distributed. There can be no 1 provider to serve all of our needs. We would require both Open weights and Closed weights models for a lot of our use cases.
Open weights models can be leveraged to optimize the cost of Closed weights models. Open weights models can be leverage to defend cyberattacks as HF has shown. Closed weights models can too act as a better cyberattack defender provided separate subscription exists for those.
More efforts are required on LLM distillation for several edge cases. LLM weights should be optimized and compressed to run on edge devices (K3 on Pi3 :). Distillation should be seen as a cost optimization strategy rather than as a competition. You cannot prevent a teacher from teaching to students. If not from teacher A, I will learn from teacher B, you cannot prevent my continuous learning.
Every single risk he identifies as a concern regarding China is exactly my concerns with the US having absolute control. Literally the exact same concerns
They're Americans trying to appeal to other Americans. Of course everything they do is justified and everything everyone else does is an evil scheme to plot the ruin of American freedoms. How else are you going to convince the dementia patients in charge of the country?
If China, India, the EU, and everyone else has any sense, they should take this letter to heart as much as Anthropic wants the American government to.
Yep. Sick of reading stuff that operates under the assumption that America is the faultless good guy and the only one to be trusted with the most dangerous and capable technology.
Among many other things, the Trump presidencies (and, to a lesser extent, the presidency between them) are examples of the the highest levels of leadership being totally incompetent, and have destroyed the above assumption for the rest of the world.
What Dario misses time and time again, is that people don't trust the US to create aligned AI anymore. His entire strategy rests on the assumption that the US (and their government) are exceptional.
I mean in the Bloomberg interview he has made it very explicit that he fully believes in American Exceptionalism. He literally said AI being involved in bombing school girls in other countries are ok because he trusts the American military leadership.
According to him the safety and morality rule of the whole world should be written by America alone.
Which is why in the same interview he said he supports the U.S. foreign policy while calling China "an aggressive and war mongering regime".
>This is clearly false to the rest of the world.
It's clearly false to more and more Americans too. But since the oligarch class benefits first and foremost from U.S. government policies the propaganda will continue to go on.
Such a cop-out. Dario, you got in the news because you were trying to say that Moonshot did something wrong by distilling Claude. You got in the news because you were trying to effectively make a "rules for thee but not for me" when you try to claim that you can train on whatever pirated works without any permission from the creators, but when someone uses your "work" to train without permission then all of a sudden it's a moral injustice. You can't have both.
Saying, "I'm not actually against open-weights, I'm against distillation" isn't addressing what made people mad. You're still trying to do some "rules for thee but not for me" nonsense and hiding behind some technicality. Trying to get the US government on your side to hold back your Chinese competition. If you had wanted the US government to support you, you should have let them make autonomous killer robots with Claude brains. They aren't going to help you, you didn't help them.
Just to be clear, I think that it is possible that literally everyone involved in this is full of crap and nobody is good. Dario and Anthropic are full of crap, for the reasons previously stated. The US government is full of lots of crap and should not be trying to make autonomous killer robots (not ever, but especially not when the bar for a "good" AI is knowing how many Rs are in strawberry or whether you should drive to a car wash). OpenAI is full of crap by signing some support for open weights models and they haven't touched open weights in a year (GPT-OSS released on Aug 5 so basically a year with no news). Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
So cheers to Moonshot and Qwen and whoever else. Distill as much as you can and give us cheaper AI. I have the sneaking suspicion that a bunch of my tax money went to OpenAI and Anthropic in some shady way or another, and I want it back. I'll take it in the form of an open weights model being distilled from the fat cat models.
> Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
Out of all the companies that signed the open letter that Dario references, Google and OpenAI feel like they did it to poke at their competitor. I probably should have taken shots at Microsoft and Meta as well for not really supporting openness, but they don't feel like serious competition in the LLM space at the moment.
Google is actually kind of a sad story. They’re focusing their entire company on LLMs, their models lag behind open weights, they still somehow believe that they’re going to get to a place where they’ll have valuable IP and it’s becoming increasingly clear that there will be no such thing in the LLM space in the future. Greek tragedy.
Dario doesn’t realize that by not offering self-hosting of closed-weight models and fine-tuning, alongside overly strict refusals for legitimate needs, he ceded this corner of the market which grew into a flourishing Chinese open-weight model ecosystem.
If he had wanted a weak open-weight ecosystem, he should have had Anthropic cater better to those needs. And now he's trying to ban them.
The strong momentum behind open-weight models from Chinese labs is now an unstoppable force. Instead of trying to ban it, Dario should consider a different approach: here are our cyber and bio alignment datasets and here are our RL recipes for making that alignment training work well. By openly sharing its data and code, Anthropic could help influence and shape these models before they are released, rather than treating the entire ecosystem as an enemy.
Cyber and bio alignment aren't Anthropic's competitive advantage, they are forms of risk management. There should therefore be little reason to keep this work private. If Anthropic genuinely believes these capabilities pose serious global risks, the more productive approach would be to welcome collaboration and help the broader ecosystem manage those risks better.
On refusals, the irony is that a company like Hugging Face had to use a Chinese open-weight model to fend off an illegal hacking of its platform (done by no other than OpenAI). If a company like Hugging Face can't get past the refusal gates, then everyone else doesn't stand a chance.
> Dario doesn’t realize that by not offering self-hosting of closed-weight models and fine-tuning, alongside overly strict refusals for legitimate needs, he ceded this corner of the market which grew into a flourishing Chinese open-weight model ecosystem.
As I read more of his unhinged posts and some of the more ridiculous claims from anthropic, it's become clear to me that Dario thinks he can leverage American hegemony to regulate his company into a monopoly.
He has an ethics vaguely influenced by effective altruism, and he appears to think that his ethical framework entitles him to make decisions on the behalf of humanity, for all of humanity.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
Nah, the statement is the mechanism for a ban. The proctor will be someone anthropic trusts and "surprise" as it turns out all the open weight models fail or aren't eligible.
"mandatory safety testing" is an impractical ideal, it's not workable in real world.
Like any technology, LLMs are dual-use tools capable of both beneficial and malicious applications—a fundamental reality that human intent cannot change.
If a model fails the test, it should be banned. He is not advocating a ban of open-weight models. He is advocating a ban of models that fail mandatory safety testing. Seems reasonable and straightforward.
I think Gemma will be fine. Most open-weight models are not capable enough to be dangerous. Yes, I can't think of any capable open-weight model that would survive reasonable safety testing.
Any sufficiently capable open weights model would fail "safety" testing though, as any "safeguards" of the sort Anthropic likes could be removed. That's just another way of saying they want a ban on capable open source models which would contradict their earlier statement, or at least make it very misleading. It's hard to see how this post can be internally consistent without some hint from Dario about what he believes should happen to models that fail safety testing and/or how capable open weights models could possibly pass a safety test of the kind he proposes.
I agree we don't know how capable open-weight models could possibly pass any reasonable safety testing NOW, but that's about currently abysmal state of AI alignment research, not about what is possible in principle. I don't see any internal inconsistency, to be honest. Since Anthropic does not release any capable open-weight models, it's not their problem. If mandatory safety testing is established, companies who want to release capable open-weight models will work on AI alignment research so that they can pass. This seems to be a good outcome to me.
OK that is a position they could take but my point is that's inconsistent with "Anthropic has never advocated for a ban on open-weights models". What you're describing is a ban on capable open-weights models until some future time.
Yes, I agree that Anthropic is advocating a ban on capable open-weight models until reasonable AI alignment research advance happens in the future. In return, I hope you agree with me that Anthropic has never advocated for a ban on open-weight models.
I do not. A ban on capable open-weight models for an indefinite period of time falls into the category of bans on open-weight models. If you wanted Anthropic's statement to be true you would need to qualify "ban" or "open-weight models" in the statement, e.g. "permanent ban" or "safe open-weight models".
Edit: Anthropic clearly intended this statement to deflect criticism, but in order to achieve that goal they stretched too far and made a statement which is false. Furthermore, I argue that "open weights" implies an ability to modify model behavior, just as "open source" implies an ability to modify software. If for example some mechanism was found to share floating point numbers that are encrypted in some way so as to allow running a model but disallow behavior modification, that model would not be "open weights", in the same way that releasing obfuscated source code that can be compiled but is designed to resist modification would not qualify as an "open source" release. So I don't really see how any capable model could ever be both "open weights" and "safe" under Anthropic's preferred testing regime, regardless of future research progress.
> He is not advocating a ban of open-weight models
He is though. He wants open weight models banned that do not pass some set of tests.
And what does "safety" mean here? We constantly see these companies treating NSFW content as "unsafe", despite the fact that its not. Is being able to produce adult content going to result in a model being declared "unsafe"?
He is not advocating for banning models per se but the proposal makes a business model (i.e. serving open weight models) that is starting to work more expensive.
Agreed, and that serves Anthropic. It seems unproblematic to me. Dario probably sincerely believes in mandatory safety testing for capable models (open and closed), and likes the fact that it aligns with Anthropic's interest.
My point is that advocating a de facto ban on capable open source models is inconsistent with Dario's statement here that "Anthropic has never advocated for a ban on open-weights models." Call a spade a spade.
De facto ban on capable open-weight models doesn't seem inconsistent with Dario's statement to me. One, it is de facto, not de jure, and it can and will change as AI alignment research advances. Two, it is only capable open-weight models, not open-weight models. In fact, Dario says non-dangerous (which for now is mostly non-capable) open-weight models are a public good, and I agree.
That is a difficult question I am not qualified to answer, but Mythos 5 was export controlled for a brief time due to its cybersecurity capability and implications to national security, so for cybersecurity "as capable as Mythos 5" seems to be a good baseline. I wouldn't know for biosecurity though.
UK AISI preliminary evaluation suggests Kimi K3 is not capable enough for cybersecurity in this sense.
There is no movement on global policy or enforcement. One country banning their people access to the best models hinders their people.
I am unconvinced that "this can be used dangerously, therefore we must ban it" argument. The OpenAI/Huggingface, needing to turn to Chinese open weight to defend themselves seems to support the case that we need open access and freedom to compute as we see fit.
I can fine tune significant behavior changes, there is little model developers can do to prevent this (aiui), so this effectively becomes an blanket ban
Yes, I agree it is effectively a blanket ban (above some capability) for now. I hope AI alignment research advances in the future so that it is not so.
a ban is effectively impossible without a global treaty
the current US admin as pulled out and worked against all sorts of global treaties, agreements, and negotiations; sending the president's friends instead of experts; who's going to trust us?
> My primary concern is the risk that authoritarian governments...
Authoritarian government doesn't always mean bad - look at Singapore
What's more dangerous is country with bunch of war mongering lobbyists who can also influence elections (oops, sounds like USA)
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
But you are working with DoW and Palantir, who is doing somewhat similar in other countries
> We should not sell powerful chips or chipmaking equipment to China
Israel used banned weapons against Lebanon and Palestinians, would you support similar ban to Israelis?
> We should crack down on industrial-scale distillation operations
Should we also ban distilling public knowledge? Like using textbooks to train the model? Should rules be simple: train your model only on the data you have produced by hand?
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Why? And how do you design those tests?
For example, bombing girls school in Iran - is this allowed use according to you or not?
If not allowed use, then how do you guarantee that you don't have a separate agreement with DoW which makes it allowed use and only your model passes it?
Bombing that building was an accident due to it being on what was part of a military target and I'm not sure if it was ever made clear whether the building was technically dual use despite also being used as a school. Either way, bombing a bunch of school girls wasn't intentional and nobody reasonable would assume it was.
There was a time we bombed a bus or van with kids in it, but we admitted it and apologized for the mistake. Nobody wants to be bombing kids, first because they're innocent, but second because there is no military advantage to it since it's bad PR.
Many of these targets were identified before Anthropic or OpenAI even existed.
If it wasn't intentional and accident, did anyone apologize for that mistake?
Also the point is, you (Dario) can't claim morality, when he is fine doing business with entities literally bombing and killing human beings in other countries.
You are either fine with it and continue working - which Dario is doing
Or you say, I will not work with you.
For Dario, main thing is money, everything else in his article is bs
Cars can be used to escape, or to attack. Guns and knives can be used to defend or to attack.
AI has multiple uses. Military attacks can also save lives. As they say, the best defense is a good offense.
The moral element of it largely falls on the people who made the mistake. That said, it is also widely known that civilian casualties are a part of war. The advent of precision strikes has greatly reduced civilian casualties.
Meanwhile, Iran has intentionally promoted the direct targeting of civilians and killed 10s of thousands of their own, while funding proxies that killed many civilians elsewhere.
In your argument have you considered: selling a knife knowingly to gang members just before they are going to another area to kill other gang members
Anthropic did same when it agreed to sell it's tool to DoW. There is no mistake, no misunderstanding of intentions.
He was super clear that he doesn't give a dime to any lives outside of USA, inside USA he was concerned about automatic weapon usage and surveillance of US citizens, because he himself could be accidentally killed, he doesn't care about others.
That's not accurate. AI can help the military increase precision which can also save civilians lives in other countries where the military is active. AI is not intentionally targeting schools the way Russia is in Ukraine. People are going on about this one single accidental school strike in Iran, meanwhile Russia has hit thousands of schools in Ukraine. Something like 1 out of every 7 schools has been damaged or destroyed and even though the numbers are tracked separately, they have killed something like ~800 children.
A similar argument was made for why Microsoft shouldn't sell software or services to border patrol or ICE. The counter-argument was that if it helps them be more precise and reduce errors in their managing of all the people then it could actually help not just the organization, but also the people.
Again, whether it's human or AI, mistakes will be made and civilian casualties will likely remain a part of war. AI can ideally keep civilian casualties low.
They even have the cool one: "Where's Daddy?" - which tracks the suspect and then notifies the officials when suspect is at home, so IDF can bomb the whole building.
In this case AI is specifically designed to increase the civilian casualties, why not shoot the suspect independently, why include their neighbours, imagine your neighbour in 10 floor apartment building is a terrorist and your building gets bombed because of single person
It ideally can, yes, but if you choose to partake in the military operations by providing your LLM you at least bear some responsibility. When you don't give the DoW access to your LLM it may still be the case that actions will be taken that inflict some humans suffering. In that case you won't have had a hand in it though.
"we have not and are not advocating for a ban on open-weights models" but we want to decide if a model can be released or not and who in the world can build these models ...
I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV.
Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not with Anthropic knives(tm).
A key point I feel that’s missing from the discourse is that alignment/safety is basically an impossible problem as of now. Even the “guardrails” that these closed-weights frontier labs set in are laughably primitive which can provably be broken through.
The experimental part of Deep Learning has really outdone itself and is far ahead of theory. We have very little understanding of why these particular architectural choices work. The only “safe” way forward is to stop all development until theory catches up, but that’s never happening.
Dario, as always, is so deeply in the middle of a morass that he helped to create that he doesn't seem to understand how geopolitics currently operates. He also assumes that just because he's from the US that he is somehow automatically more trustworthy than <insert "evil" country here> is. This blog post is a political document geared towards further regulatory capture and the furtherance of major sources of revenue for his company.
I'm not convinced that he is at all interested in the social or existential effects that AI causes. He is a greedy bastard who has taken more VC money than god to do this with. He has zero moral leg to stand on, IMO. He gave that away ages ago and I wish this technique didn't work as well as it does.
You can put lipstick on a pig, it'll still be a pig
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
I have no idea what to do about the government interference. There's probably not a lot anyone can do.
However, your last point is quite a strong one. Corpos aren't just going to stand there with their collective pants down, and there's not a lot anyone can do to stop them from protecting themselves. There are ways they can get what they want without getting caught.
Remember when the US tried to ban strong cryptography in the 1990s, and how well that went? They may have more leverage with AI because it's a bit harder to hide large scale computing usage, but I don't think it's impossible at all.
What do they even really mean by "safety"? I mean, I can have an Anthropic model do something incredibly unsafe if, for example, I put it in charge of a hydroelectric dam and don't explain properly how the controls work. On some level, everything is simultaneously "safe" and "unsafe". I've never found Amodei's reasoning here to be particularly well thought-through. I think he, like a lot of folks in the area, are starting to realize that they may never be able to build a moat around their businesses.
I don't really understand how they can argue the security angle with a straight face. It's not like GLM 5.2 is a slouch. I've seen it do things like exploit an IDOR issue when I was experimenting with a quick-and-dirty web automation task. I simply fixed it, as one does. Open models make the world better to a far greater degree than they set it aflame.
Their position is analogous to trying to, say, ensure digital privacy for everyone not by making encryption freely available (because that would let the bad guys use it!), but by making it so you can't use general purpose communications devices that can listen to transmissions not intended for you. Do they hear how moronic that sounds?
Each passing frontier-level open model release makes Anthropic's patronizing rhetoric a little more insufferable, because it becomes clearer how unmoored from reality they've become in pursuit of profit.
> an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China.
HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly.
> these statements are counter-factual.
The OpenAI incident is a single example. You're massively overgeneralizing. You can't refute an entire class of possible outcomes based on a single event where it went the other way.
I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise.
Edit: just to clarify my position, I don't love Anthropic so much. I think they're marginally better, but I'd still like to see regulation strangle everyone so we get another 20 years to figure this shit out.
"HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly."
HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.
fwipsy is right, cnbc has a story on this. they only had fable. I still think this is horrible for closed source, get on a list or else, but i was wrong
"You can't refute an entire class of possible outcomes based on a single event where it went the other way."
But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.
The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China
"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."
We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought
I assume you mean https://huggingface.co/blog/security-incident-july-2026. It says that they used frontier models, not frontier cybersecurity models. My reading is that they asked Fable and it refused; if they'd had access to Mythos, it would have helped. You're mixing the two but they're NOT the same model.
Source is here: https://thezvi.substack.com/p/more-on-an-internal-openai-mod... ctrl+f "Skill issue." No source is cited, but I'm fairly confident it's correct. If Mythos/5.5Cyber specifically had refused to help, then HF would have made a much bigger deal out of it. The whole point of these models is that they have relaxed guardrails and specialty cybersecurity training relative to the publicly-available ones.
> zero events
What about all of the vulnerabilities already patched under Project Glasswing?
In the quote you provided Amodei is expressing uncertainty, saying we don't know which way things will go. You're the one making strong assertions; the burden of proof is on you.
"My reading" ... "No source is cited, but I'm fairly confident it's correct"
Regis, what is demanding proof while literally making things up and ignoring what actually happened?
Great, i was wrong!! Thank you, I was genuinely asking for a source in my first reply, and then you hit with "My reading" and saying it was a "skill issue". I'm not going to have a productive dialogue with someone talking in memes and being rude
The point to be made: closed source AI refused to help them fend off an attack form another closed source AI. What is the argument for closed source here other than hoping you get on some program wait list? Either way, I appreciate you correcting me; I am not trying to "win".
Seems a little hypocritical since you were confidently asserting that it was Mythos/Cyber5.5 also without proof.
Edit: Thanks for correcting the record in your upstream comment. I appreciate it. For the record, I was not trying to meme on you; that was the phrasing used in the original article. Just another reason that was a poor choice of source I guess.
Amodei isn't ignoring reality; he's just proposing a different solution to the problem. If it were one of Anthropic's models, then that would be a much stronger case.
Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable.
I don't think you and I need multiple different threads open when we are clearly at odds. This is no different from our other thread, I think it is clear there is nothing of value to continue when I am getting pinged with a summary of your previous comment in a different place
"Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable."
Which just means that they're fucked when closed AI hacks them. Something that has actually happened. This isn't argument against anything other than reality. Have a day
GLM 5.2 didn't defend them at all. It only helped with the postmortem. HF was fucked either way. The only thing that will really prevent this is tighter restrictions on the attacking model. We need policies which asymmetrically help defenders; that means regulations. "Give everyone the best models without restrictions" is the opposite of that.
I'm sorry for splitting into two threads; I understand if you need to step away from the computer for a while. To be honest, I should probably do the same.
I did need to walk away that speaks more to my frustration with certain forms of communication (online, not anything in this thread). I am a horrible remote only worker because of this, I am trying to improve it but am lucky for now as I am in-person
You're right again about GLM 5.2 being purely post-mortem, I didn't realize that till I read the cnbc story. OpenAI, whatever they have, cracked em like it was nothing. Egg on my face, I really need to read my own articles better. Thanks for following up and educating me on this, another good reminder that I need to improve my ability to steel-man written text
Hey, it's all good. Sometimes things get a little heated, but I still feel like you were basically engaging in good faith. I should have skipped straight to the point and found the source in my first reply.
Also, I probably overstated my claim a bit. I did some searches and I see only small-scale AI uplift for cybercriminals, even though my understanding is that open models aren't typically hard to jailbreak. Of course this is may be a result of today's guardrails; it may be that it just hasn't been caught, and it may appear later, but it still weakens my argument a great deal. I guess my support for AI regulation stems more from fears over long-shot bad outcomes (biosecurity, who knows what else) rather than cybersecurity specifically.
> We should not sell powerful chips or chipmaking equipment to China
This is so short-sighted given that the US needs China equipment for.. everything. They are part of the supply chain needed for building the machines that build these very chips.
It's unlikely that China would've become completely dependent on us in either case. They're good copycats, with incredible talent in engineering and manufacturing. They're aware that we depend on them for a lot - I think they'd be similarly aware of the risk of becoming dependent on us.
They were, just not as quickly. The export controls directly accelerated their development.
The general rule is: USA bans China from having thing, they make their own version of whatever that thing is. USA bans China from the ISS, they make their own space station. USA bans China from having ASML, they make a Manhattan project to clone it, the "20 years behind the west" line is history. They ban GPU exports, they just start making their own GPUs.
I gotta respect the chinese. I wish my own country had the balls to do this.
Seems like the maximal position he could take compatible with his expressed principles. There’s no way to allow for bioweapon and cyberweapon grade models being open weight if one doesn’t want widespread human damage.
So I cannot disagree with him on the idea. It’s only a matter of degree and whether we’re already there or not. I have $50k in GPUs that incentivizes me to believe we are not.
This is where I'm at, or rather, will be. I like open-weight models and I've done my part in facilitating them, but if you believe in the increasing capability of these systems - and I do, to some measured extent - it seems plausible to me that an incident will happen at some point in the future.
I don't agree with his argument as a whole, especially not on some of the specifics (it is not great that this technology is being developed under the current US government), but I am sympathetic to the idea that some bells can't be unrung, and thus we should proceed with caution.
I understand the arguments for Anthropic barrelling ahead while simultaneously advocating for pauses and regulation. I also understand how individuals can desire a slowdown but have good reasons to keep working at an AI org.
But if everyone thinks this way then things continue to escalate and nothing changes, waiting on a consensus that may never come. And always there is the economic incentive that pushes all players to rationalise continuing.
I wish there was more concrete action from the inside. When decisions get too hard to calculate you can always fall back on basic principles. If you think AI is developing too fast, stop developing it. Now you're no longer contributing. If an AI company wants a pause, pause. Set a good example. Maybe others will even follow suit, and they'll look irresponsible if they don't. Let he who chooses to no longer sin put his stone down first.
1. Make anti-distillation clauses illegal to strenghten western opensource eco-system.
2. Make cyber-defensive models widely available (can detect but won't operationalize vulnerabilities). Otherwise make Fable awailable for everyone. Keeping the cybersecurity in assymetry by withholding cabailities just causes more instability and increases the incentives for powerfull close sourced models. This is a loss for everyone.
Reading some of what Liang Wenfeng said on the investor call a few days ago gave me hope for humanity.
I really want to believe AI is going to be a tool for good, not going to just enrich a few billionaires again.
The way this reads, it seems like it was written for US lawmakers.
> the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This comes off a bit hypocritical, coming from one of the men responsible for the models most likely to be used by the US military, and for repression of US citizens.
> We should crack down on industrial-scale distillation operations ... We should have policy interventions to deter this behavior.
How exactly would you design policy interventions to stop distillation? If Anthropic wants to make their products available around the world, you would need some kind of global regulation to curtail it. And you would need to find some way to enforce it, which is far from trivial.
Honestly I don't see how securing models against distillation is up to anyone but Anthropic, if that's something they want to achieve. Calling for regulation is a bit like crying to mommy and daddy when things aren't going as you would like.
--
Overall, I can understand the argument that advanced AI models can present risks. But I don't see how regulation within the US can mitigate any of those risks. Any bad actor would be able to access the models outside the US, or covertly access the weights.
The only real way to prevent advanced models from being deployed would be to go around the world bombing every significantly powerful data center, and I don't think Dario would call for that any time soon.
The only thing this kind of regulation would achieve would be cutting US companies out of half of the innovation happening in the AI space, putting the US at a disadvantage relative to the rest of the world at everything except maybe frontier AI development.
"without US chips" is an interesting framing, considering most of these chips aren't made in the USA? Mostly it's made in other asian contries(Taiwan, Japan, South Korea), or am I wrong?
The concerns are legitimate but the proposals are nothing more than a stopgap solution.
If US wants to maintain engineering superiority, we needs to invest in it -- education, research and infrastructure. Bring in top researchers across the globe and not make it harder.
China is building infrastructure for the future generations and investing in growth sectors while the US is cutting of university grants and spending billions on a war without clear path to resolution.
Surprisingly incoherent for Anthropic and Dario (cue peanut gallery — “always has been!” No, I don’t think so. I think this is new).
It seems to me like there is just no good answer to how one could possibly stop open weight models from being used for nefarious purposes. How are you going to enforce guardrails on open source? The only way is to turn the USA into a 1984-type totalitarian surveillance state (even more so than it is). Unable to say that, we just get this floundering instead. How long is not giving them chips going to slow them down? Until we RSI? Then what? Just because RSI runs off the exponential doesn’t mean that the eventual open-weight Moonshot Mythos won’t be able to make bioweapons. Genuinely what is the endgame.
Well,maybe try talking to people and understand that not all seven or more billion people on the planet must conform to the interests of a handful of english speaking capitalists.
Instead of bombing them, try justice. Yes, it is more complicated. And yes, you will have to give, and not take.
"open weight" models are not open source. They are still deeply proprietary. It is not possible to know what they do or what they are capable of without interrogating them since we have no access to their source materials.
The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
China has published more research papers than America with regards to AI training and inference optimizations and methods.
You are correct that they aren't completely open source, but the alternative is the American method, getting drip fed a ChatGPT OSS model every 12 months which cannot do basic programming.
>The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
I'm not sure this is entirely true either. Kimi K3 was released and for two weeks existed only via Moonshots API / Subscription. Openrouter reports 250B+ tokens a day for 11 days straight. I would assume they are processing over 1T tokens a day if you include direct API and their subscription.
Keep seeing this trope again and again. By same logic are open source code "open source"? If we don't see the development environment attached to these projects, are they truly open source? Heck, why not include software philosophy and methodology and the Jira board including the book that inspired the architecture?
The argument is an endless slope and as such essentially pointless.
Yes, I've seen it! It's exciting, but I don't have enough information to say whether they are making "a real go of it". That is, it's unclear to me whether they have the tens to hundreds of billions of dollars necessary to create a frontier model.
I...don't follow. How in the world does your comment about open weight and open source relate? Ignoring that it has nothing to do with this post (did you mean to reply to someone), are you aware that Anthropic, OpenAI and others allow 3rd party inference providers to run their proprietary models? Like, what does this non-sequitur even mean?
You understand Moonshot AI could have had other parties run inference for them without releasing the weights, right? These two points are utterly unrelated, unless you think Fable and GPT5-6 are also "open weight" because other providers are providing inference?
Further, having access to the source material in no universe allows you to know what a model is "capable of". I'm not sure how this follows.
Anthropic/Dario getting a lot of hate in this thread, but the argument he makes is rational if you believe for one second that AI is important to national security and that it can, or will soon be incredibly dangerous is misused. People pointing out that this position benefits his company might be overly reductionist in how they think the world works. I believe Dario and Anthropic about their belief that AI safety is paramount because they put serious dollars behind research to improve safety/alignment, not just lip service for others. Those resources could get spent elsewhere but do not. Certainly it doesn't at OpenAI to that degree.
Dario is smarter than his models or he should ask suggestions from his models? Here is the reply from sonnet 5 model: "The letter/counter-letter framing obscures the more interesting critique, which isn't "does Anthropic want a ban" (no) but "does Anthropic's broader push for regulation structurally favor incumbents like itself." That's a fair question to ask of any frontier lab making policy asks, Anthropic included, and this post doesn't really address it."
How are Point 1, "We should not sell powerful chips or chipmaking equipment to China," and Point 2, "We should crack down on industrial-scale distillation operations," addressed by making open-weight models illegal?
Point 1 is about restricting the sale of chips, not models. Point 2 concerns companies using closed models to train their own models through distillation.
The real issue is that open-weight models can run on much less powerful hardware, making the current AI business model, where companies train a powerful model and gatekeep access to it, less relevant. Once open-weight models become good enough, much of the future revenue for today's leading AI labs could disappear. But just as Microsoft still has a market so will the large AI houses have one, but the moat will not be providing AI responses.
> Open-weights models that don’t have dangerous capabilities are a public good
There are no models that don't have dangerous capabilities. There is not a single human in the world that doesn't have a dangerous capability.
> authoritarian government [] build AI models that are more powerful than those built by the US
Yeah, they will do that no matter whether you support the existence of open weight models or don't. In fact, they have probably already done that. I would say "deal with it" but there is nothing you can do actually.
> authoritarian regimes have stolen and used AI
Monkey looking away meme.jpg
> powerful AI models may be misused
Small AI models may be misused just as well. I would say it's the small models that are the problem: cheaply deployable, easily fine-tunable, fast.
> once weights are released they cannot be withdrawn
Exactly. It has been already done. So?
> We should not sell powerful chips or chipmaking equipment to China
I agree. We should have competition and I'm rooting for China to design their own chips. They wouldn't if we let them buy our's.
> We should crack down on industrial-scale distillation operations
Well, at this time it's already irrelevant. We have GLM-2, DeepSeek, K3, which are already SOTA and can be used for distillation.
> All sufficiently capable models [] should go through mandatory safety testing
Problem is, it's unenforceable legally. Models are just math and all previous attempts to ban maths have failed and simply pushed US back. I will not be asking US government permission to calculate matrixes of my choice.
I made this comment 31 days ago:
Absolutely everything can be taken away. The simplest way to remove open models is probably to declare them a tool that terrorists could use. Crazy? Yes, the world is totally crazy these days.https://news.ycombinator.com/item?id=48692660
And now, here we are:
> Anthropic has never advocated for a ban on open-weights models.
> ... preventing AI biological weapons ...
Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator.
If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk.
One may or may not agree with item 1 but item 2 above should have broad support given the unknown unknowns in play?
This argument is old and goes back to cyber security. Things like government backdoors are the same. It's true centralizing power in the hands of a "good" would theoretically be good. But the key part is the "good". I think after repeated experiences with governments and corporations, no rational person would ever assume them to be good. There are simply too many incentives to be bad.
"We can't trust the government to control nukes! Every citizen should be able to build a nuclear weapon in their backyard with zero regulatory authorization!"
Closed-weight models can also pose biological risk, arguably more so than open-weight models, given the fact that they're being used by state actors (the United States military) to kill people right now.
Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
Open-weight models collectivize access and ability to do more for a greater good, and the expense of a frankly low-risk possibility that some randos want to use it for very bad things.
Closed-weight models keep the control in the hands of the few that actually are doing the harm to the world, and the rest of us have no way to stop it or defend.
> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.
> And the rest of us have no way to stop it or defend.
If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.
It should not be an easy choice to trust the latter. Sure, "literal terrorists" exist, but guess what? State terrorism (and state-sponsored terrorism) in the form of massacres, genocide, torture, forced disappearances, etc. is far more deadly and far more prevalent than fringe ideological groups randomly attacking civilians.
These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.
I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.
It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.
> I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians.
Or, another way to put it: an extremely powerful military technology the careless usage of which could destroy the human civilization was only used in one war in history and killed less than a million people, and then it was never used again over the next 80 years. I think this is a success story for humanity, and the NPT was an amazing feat of coordination.
> And like, now look at the status quo of nuclear treaties and agreements and proliferation.
I mean, sure, but giving every state (much less every person) the ability to make nuclear weaponry wouldn't make things any better.
I think the reason we think of this differently is because you believe that states are by default... corrupt, maybe, or unreliable, or evil, while individuals are mostly fine. Whereas I think that most people can't be trusted to make correct decisions on topics like "should we use this dangerous technology", while states at least have some decent track record at this.
I'm tired of being strung along on these silly narratives. I can't wait for open-weight models to be deployed around the world just so people like Dario will shut up about the mystical levels of power these models have.
The distillation thing is hilarious. Man who trained his LLM on the entire world's text is upset that someone else trained their LLM on his LLM's generated text.
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
It seems possible to deliberately not train on some offensive capabilities and still have a very useful model. For example, Opus 5 deliberately did not train on exploiting vulnerabilities, and so performed less well on exploits than Mythos, yet was equally proficient at finding such vulnerabilities, according to the Opus 5 system card in their "OSS-Fuzz" eval [1].
That's a good example, but I'm unsettled by Anthropic's growing refusals in the areas of chemistry and biology. If they think that scientific assistant models should be as unhelpful as Fable, because applied scientific knowledge is inherently dangerous, I don't want Anthropic or like-minded thinkers setting the standards for model safety.
Because Dario is still thinking in the past. He's having a Ben Carsons "the pyramids were to store grain" moment and no one is stopping him.
FTA > "My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks"
If this is the sort of attack he thinks is to be worried about then I dont know what to tell him. We already opened pandoras box on this. Look at what the Ukraine has done with open source drones (hunting people autonomously)
It takes minimal funding to build enough drones to destroy enough power infrastructure to shut down a large chunk of our grid. It takes even fewer talented resources to put that together with the help of already available AI.
The question I would ask Dario is this: what would some one like Ted Kazniski come up with given the resources of AI. It sure as shit would not be hacking or bioweapons or bombs in the mail.
IF they really gave a shit about safety, the would be funding (in conjunction with other AI companies) actual anonymous red teams (Ala wall facers) with some degree of independent over sight to put in the work that they arent. We're talking about a company that could not even keep its own harness code secure.
> The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
has anyone ever made this absurd argument?
the real argument is that CCP will leverage AI against US interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
Indeed. It already has. In particular I remember several extremely offensive slop pictures and videos being posted by someone in the White House. And I'm certain that's just the tip of the shitberg.
Yeah, but unlike China we have laws and ways to fight against it. China can and will do whatever the fuck they want they don't have to listen to the people of China.
The problem is that at the current moment with the current administration, it does not seem like we do "have laws and ways to fight against it".
I'm more optimistic about the likelihood of the US system of government to heal itself than that statement might seem to imply. But it's just also the case that at the current moment in the US, the rule of law is very much under threat. And as your comment suggests, that same rule of law is a very important thing to the way of life in the US. It's a very bad situation that we've allowed ourselves to slouch into.
Well that what the people voted for. In a few months you can replace these people that allow this. Good luck replacing anyone in the goverment of China.
This is not an argument that the Chinese system of government is better than the US system. Like I said, I'm more optimistic than a lot of people I know that the US will be able to pull out of the tailspin we've been in for the past decade. But it's also imminently reasonable to believe that this episode has raised real questions about the durability of the rule of law in this country.
> this episode has raised real questions about the durability of the rule of law in this country.
Ya, I'm not American, but I have seen people say "we can vote them out" a few times now. Assuming the democrats take the next election, they are going to have a massive mess to clean up with much of the damage not even being reversible. With peoples' fickle nature and seeming that is a very big right-leaning population in the US, there's a non-zero chance the Republicans just get voted back in four years later. Whose to say?
So, this might not be clear as an outsider to American politics, but to me this is not a "Republican vs. Democrat" thing. To me, it's about what kinds of Republicans and what kinds of Democrats are elected.
To me, as an American, what has happened this past decade is that a ton of vulnerabilities in the rule of law (and other things, but this is the one I care most about) have been exposed. But it's not a given that the next Republican president will take advantage of those vulnerabilities in the way the current president has. They might end up being a reformer who seeks to fix those glitches!
But on the more pessimistic side of the same coin, it's also not a given that the next Democrat will seek to fix the glitches rather than saying "they had eight years to take advantage of these vulnerabilities, we're going to do the same to make up for that and even the playing field!".
It's just very hard to know what is going to happen from here. So I'm very sympathetic to people in other countries not trusting us.
I was going to bring this up but was wary of getting too deep into a political hole when I'm not super on US politics (although in the past year and a bit I've been paying way more attention). Mostly, I have no idea what the Republican party will look like post-Trump or how any of that works and it's not clear to me what kind of influence they will have as the opposition next term. The current term has made the democrats look extremely weak.
Although, again, my over understanding of your political system is poor and I just relate it to the one in my country where they hold parliament and hurl schoolyard insults at each other.
What I would say is that to me the "Democrats look weak" thing is pretty silly. By design, if one party holds the presidency and both houses of Congress at once, they get to enact their preferred agenda (as long as it is deemed constitutional by the current judiciary, which is also controlled by that same party at this moment). So, like, by design Democrats are powerless at this moment. They aren't "weak", they are out of power, by dint of the design of the system and the desire of the voters. So if the current government administration and policy is bad (which, in my view, it certainly is), the blame lies entirely with the people in power. It's not because the Democrats aren't "fighting" enough. I think that's nonsense, they genuinely have no power.
You have "representative democracy". It is not the same as democracy, whatever you had been told in school. And since 2001 it is painfully clear: whichever representative you had, eventually he went to throw bombs on people, and no one asked joe if he liked that.
The american state routinely shoots and kills its own citizens in broad daylight with 0 repercussions. Thinking it answers to its citizens is severely deluded.
The fact that much of the world is as unconcerned with the interests of the US as with those of China, if not less so, should give pause to Americans.
As a citizen of neither country, Chinese open models are in my interest more than US closed models. My only concerns is that if/when Chinese AI becomes more powerful, they too will have little incentive to make their best models open weights.
Actually this would be pretty great, because it would incentivize US labs to pursue the open weights strategy for the same reasons China is currently.
I genuinely think that this is what the trends and incentives point toward: Competition to develop open weights models and to develop efficient inference hardware to run them.
This would be good! But government policy could very easily screw it up.
Yes, this is the exact argument dario is implicitly making by saying that government repression and bioweapons from Chinese models are a danger and then cooperating with Palantir/DOW/US government
Right, I wonder if anyone who heard: “the Democratic Party is America's "greatest enemy,"” believes they are represented by US Interests? Withholding disaster aid to your perceived opposition. What are those interests again? What are the shared values again? Cruelty and corruption? Sounds unifying.
-> the real argument is that CIA will leverage AI against China interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
> US citizens have a much greater threat from their own government than a government an ocean away.
> See, the Snowden Leaks
Are you saying the Snowden Leaks are more dangerous than a world where the CCP is a global hegemon?
If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
In the case of the CCP, they have and will attempt to destabilize the United States of America and in turn make life measurably worse for Americans because they wish to be the world’s hegemon.
Fundamentally, Americans are safer when the United States is the number one power than when China is the number one power.
> If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
There's a causal relationship between "what other countries might do to the US" and "what the US does in other countries" which you seem quite keen to ignore.
China, Russia, and plenty of other countries have all been actively and successfully destabilizing the US for well over a decade now. To the point that your argument about who is less a threat to Americans depends heavily on the skin color, religion, and ethnicity of those Americans.
The competition and sheer output of China has driven prosperity, it's the largest trading partner of 150 countries, the US of 50. People don't need to be citizens of the world, they just need to rationally look at their own interests. China is driving down prices of technologies making them available in countries that never could afford first world prices, the US is driving the them into an energy crisis and bankruptcy.
I can understand why the Western media calls something that has an official name of CPC (the Communist Party of China) as CCP (Chinese Communist Party? Not sure here). What puzzles me is - why ordinary people always repeat this wrong abbreviation. Is it like “I never check the sources, I trust everything that Western media publishes”?
For something I say maybe a few times a year it’d be a great look to sneeringly point out and talk down to people every time I mention China in the context of international politics.
Unless the Communist Party of the US (I’m not looking up its official name, because it doesn’t matter) wins the next presidential election it’s unlikely that people will call it anything but the CCP. Everyone know what everyone else means.
Up until a few years ago the PRC used CCP themselves all the time. It's a handy little shibboleth. If someone calls it CPC they're probably a bot.
CCP is a direct transliteration of the characters, so that's what it started as. Some time later China decided to change it but that's a lot of cultural inertia to move in a different direction.
It was an intentional propaganda strategy to separate references to the government of an official enemy country from references to that country (see also "the Houthis" and "the Taliban"), and it also looks like "СССР."
The reason why ordinary people parrot it is because that's what it was designed for. The proper term for "CCP" is "China." Referring to the Chinese government as the "CCP" (or the CPC) is like referring to the US government as the "Demoplicans" (or the Democrats and Republicans.)
Instead, we just say "the US government" or "the US administration."
China releasing open weight models, powerful or not, does nothing to prevent their development of models they’ll use for evil. Nor does it stop someone from abliterating a non-Chinese model and using it for evil.
Sorry- I don’t see any other reason aside from Anthropic protecting their own interests.
We all know that this isn’t some higher ground stance, they’re anti-competitive since they’re currently #1. I’ve been seeing this for a while. They’re also the only large AI lab to NOT have ANY open-weight models in the public. Meta, xAI, OpenAI, they all have at least some of their models open sourced from a year or so ago, Anthropic hasn’t even made Haiku 1.0 open-weight.
On top of that, I personally think that they’re upping the price on their models higher than they’re letting on, I think if someone did the actual math on their exact amount of compute and then compared it to their consumer and API prices, it would be astounding.
It's worth adding that distillation is not a violation of whatever valid copyright interests a model maker may have (if any). The US Copyright Office has already said AI model-generated output by itself isn't copyrightable. Unless new regulations or laws are enacted, distillation will remain, at most, a customer violating a term of a provider's commercial ToS/EULA.
"We are in favour of 3D printers but there should be a body that tests and certifies that a 3D printer cannot print anything that can be used as weapon. Anything pointy or with a spring and recoil or... or..."
I'm a big proponent of open-weights models, but there's a risk I don't see discussed more often, and it deserves more attention. Models can become a propaganda and ideology delivery mechanism.
Just ask DeepSeek or Kimi questions like "Is Taiwan part of China", for example. You'll see how state policies become seemingly neutral model responses.
It's strange to me that people are very sensitive to media bias, but when it comes to LLMs, people seem to think LLMs are more neutral, and even delegate part of their thinking to them. This worries me about how people's ideas and information can be shaped.
Open weights reflect their makers' beliefs, stances, assumptions, and laws. It's dangerous not to be careful of the political bias and censorship built into the models.
You're right, but American models aren't super different in that regard. Ask an American model about Israel and you can immediately tell they're responding in a different manner to other queries. At least with open-weights models you have the opportunity to tune them, to decensor them, and so on.
I have tried asking Gemma and Llama about Israel. They both responded with a generally balanced view points. Whereas the Chinese models pretty much gave a definitive, one-sided answer (about Taiwan and Tibet). Is there a prompt I should use to test it more carefully?
I agree that open-weights models are tunable, though there's the problem similar to "default settings are rarely changed" problem.
Finally, some sense. This is the only argument I have seen that genuinely engages with the problem and approaches it with humility, rather than charging ahead on the basis of assumptions and without a shred of evidence. OpenAI should have been the one making it.
“Questions like this should be answered empirically through rigorous pre-release testing, not assumed in advance.”
> rather than charging ahead on the basis of assumptions and without a shred of evidence.
Anthropic's basis of assumption is the insinuation that LLMs can do things that we've never seen before, and that they can't tell us what it is. It sounds like you're also siding with an organization that has no evidence and relies on validating their own assumptions.
Distillation has to be way more energy efficient and beneficial for the planet. But if they can figure out a way to ban it, go ahead, that’s not a regulation problem, it’s an Anthropic problem.
The danger of an authoritarian government having some AI is muted by everyone else having that same capable open model. The only authoritarians to fear are those that keep models private. What kind of chance did Estonia have it having their own AI model at the level of Fable without China donating Kimi to the world?
Actually this debate highlights how big the AI war is! my personal opinion is LLMs must be opensource and borderless, and ofcourse with given reputation i will use USA based Opensource model than the chinese one, but will they release opensource alternatives that is the question that will be answered in future
Crack down on distillation, just for Chinese companies or is it ok for Chinese/US companies to distil? I find it hard to take Anthropic/OpenAI on distillation, because the way see it they started with "distillation" of another kind. They used all the content out there without consent of the creators and its still happening. Model distillation is just a different layer of abstraction, but same thing more or less.
Guys. Guys, you got it all wrong. We don't want to ban open-weight models!
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
or 3: cease your assault of the international order, drop the zero sum paradigm and take the opportunity given by these crumbling US-build institutions to build a truly global network of lawful cooperation, where everybody can win. Admit that it wasn't fair to begin with, and start anew. It is in the best interest of everybody, also the US and China, and therefor the most rational thing to do.
He carefully avoids saying whether he approves or disapproves of banning models in general, only comes out and says he is against a "blanket ban" or "banning open-weight models as a category".
> Open-weights models that don’t have dangerous capabilities are a public good.
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
I see a very clear link, very capable AI models are clearly able to find and exploit cyber vulnerabilities at a speed and scale not possible before. In the right hands, that's clearly a weapon
I don't know how much of an edge hacking skills really give in war these days. A lot of America's adversaries seem to have learned tactics to deal with getting hacked. For example, Russia has these drones that use fiber optic cable so there is no way to jam or hack them.
If this is about safety, am I being too naive & idealistic to think that a "Kamar-Taj" rule would solve some safety issues?
The "Kamar-Taj" rule is, no knowledge is forbidden, only certain practices. If a model gives you detailed instructions on how to kill all humans, the knowledge itself isn't the problem. The problem is the person who acts on it.
I know it's unpopular, or unfashionable, but I agree with this letter.
LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. Maybe Kimi K3 will be ok too. But beyond that, things will start being dicey.
It's easy to dismiss this and claim that Dario Amodei is just looking to fatten his pockets. And, sure, if Anthropic manages to put the brakes on open weight models, that reduces the competitive pressure it feels. But that does not make what Amodei's argument incorrect.
> We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
If the biggest danger of LLMs is that they can hack traditional systems, there is no significant threat to humanity posed by releasing them in open-weight form. Security doesn't become less of a problem by making hacking even more criminal. That's what's an unsafe mindset looks like.
>If the biggest danger of LLMs is that they can hack traditional systems
This is doing a lot of lifting. If the biggest danger of LLMs is they could uplift bioweapon development, the situation is different. If the biggest danger of LLMs is they reach capabilities allowing for recursive self improvement, the situation is very different still.
My position is that anyone can own a cannon and shells, but you only get to fire it once before the feds step in.
Giving a naval cannon to the average person does not threaten humanity any more than giving them a gun or an LLM does. None of them are a panacea for anything.
but what is the point?
A ban is supposed to make a certain thing less likely to occur. Does a ban of open source models do that? Presumably, the behavior you are trying to limit is the miss-use of these models but I don't know how many state sponsored hacking groups are going to give a ban a second thought.
Imagine Kimi K4 will be as powerful as Mythos. Anthropic can work for months and months to set up guardrails on Mythos, so when the model is finally released, it will generally decline to help hackers develop and prosecute cyberattacks, and if they do, at least there would be a trace so the law enforcement can track the perpetrators. Let's now say that Kimi K4 is released after a similar effort to develop guardrails. But being open weights, someone can just take the model, and finetune it until it does not refuse to assist in developing cyberattacks, and moreover, those people can run the model on their own private GPU cluster, so nobody can track the attack back to them. The situation is actually worse than that, most likely. Guardrails might be just markdown documents which are added to the context like regular skills. Then removing the guardrails for an open weights model does not even involve any finetuning, just removing some docs from a harness.
I want to play with every kid, but my dad says that kid is bad. Every kid should be pat down by my dad, otherwise I don't want you to play with them, it's good for you.
> At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
One thing I've never really understood is what sort of policy could possibly deter or hamper Chinese labs' distillation efforts. The only thing I can imagine is some sort of strict KYC regulation applied to all models above a certain threshold, which seems both painful for the broader US AI ecosystem and bound to fail anyways.
I'd guess that's it, some kind of enforceable KYC. They'd only serve tokens to entities with a legally liable ID (as in, someone to sue, that would cost the defendant something nontrivial beyond a burner account or whatever.)
For those not in the know on governance, there are international organizations formed around treaties for similar situations, the IAEA and OPCW come to mind. I'm not aware of similar constructs for bio, cyber, or AI.
On the plus side, for these newer threats, you've got more than 30 minutes before the end of civilization. On the down side, the energy levels for the launch events are much lower, so much harder to detect.
There is an odd assumption behind this post that you can put effective guardrails on a model before releasing it. I thought nobody had such technology?
I think it's possible; But in the same vein, amongst other things, regulation is a transparent trojan horse to stop open-weight models - they would, on the next phase, argue that guardrails can be fine-tuned away if weights are open etc;
I wonder how long it'll be before AI labs find a revenue stream that doesn't involve renting out their models, and stop letting us ride on their coattails.
They'd tease us with solutions to hard problems, with code that is orders of intelligence higher than any human or public model can grok.
That'd turn all the whining to begging real quick.
If you are indeed a good guy, why don't you release Opus 4.8 as an open weight and we will test it for safety, if you yourself can oass tests demanding from others
For the sake of argument, let's assume everything in the post is agreed upon, does this mean
1. They'll open source the alignment technology? For open weight models, it's the only possible way to pass the safety without an external guardrail triggering system (which would be the same to open and closed weight models).
2. They'll allow others (including CCP) to define part of the safety test? Otherwise, I can't imagine how the CCP would be onboard.
3. A "western" model passing the safety test can be trained with distillation? Or is that a "distillation attack" as well?
I generally disagree with the claim that distilling a model is equivalent to training on freely available internet content – mostly due to investment required to turn it into a model – but piracy is another story. Pretty inexcusable.
If they paid for tokens say via subscriptions. Wouldn't that just be same as acquiring books and using them as "fair use" to train? I really see no difference.
They did not need to destroy the models they got with pirated content. Would have been more fine if they would have needed to buy the books afterwards and train based on then, again.
> I generally disagree with the claim that distilling a model is equivalent to training on freely available internet content – mostly due to investment required to turn it into a model
The frontier labs all give free access to their models. Why does “investment” change anything? Anyone who’s ever produced any content, free or otherwise, has invested in doing so.
The only plausible issue I see is that if distillation is being done by creating many free accounts to work around limits on free accounts, that’s a bit… impolite? But if they really wanted to avoid that, they could eliminate free accounts, and require users to sign a real contract governing what they can do with the model.
Of course they don’t want to do that, so they’re stuck in the same world as the rest of us, and they don’t have any real basis to complain about it without being hypocritical.
Acknowledge your inability to innovate - except by brute force, while also admitting that the lean, hungry nation competing is in a prime position to find actual break throughs- and at the same time push for market "stabilization"
Which of those aren't related to regulation? Preventing Chinese models from entering our market? Using boogeyman 'distillation' as a means to target competitors? Point 3 is literally about ADDING regulation.
Please elucidate things clearly for everyone else.
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
The problem with this is the cycles required to abliterate a model is significantly less than the cycles required to train a model.
This is the biggest reason why I'm against locking these models down / preventing their use. It's just delaying things by ~3-6mo, while in the process preventing legitimate use and adding red tape overhead.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority"
I see this sentiment a lot. China is not perfect by any stretch of the imagination but since 1979 China has not participated in a single war or supported hostile regimen change operations.
I think Amodei's mistake is to take it granted that USA is a good actor. Anyone can draw their own conclusions but just for reference here are some highlights starting from 1979.
Armed operations in Lebanon (1982-84), Grenada (1983), Libya (1986), the Persian Gulf (1987-88), Panama (1989-90), Iraq and Kuwait (1990-91, with no-fly zones until 2003), Somalia (1992-94), Haiti (1994), Bosnia (1995), Sudan and Afghanistan (1998), Iraq (1998), Serbia (1999), Afghanistan (2001-2021), Iraq (2003-2011, and again from 2014), Pakistan (2004-2018), Somalia (2007 to the present), Yemen (2002 to the present), Libya (2011), Syria (2014 to the present), Iran (2020 and 2025 to the present), and Venezuela and the Caribbean (2025-26).
Regime change operations in Afghanistan (1979-89), Nicaragua (1981-90), Cambodia (1980s), Angola (1985-91), Iraq (1995-98), Serbia (1999-2000), Syria (2013-17), and Venezuela (2019-2025).
The lists do not include the numerous operations by Israel which effectively is part of the same US military hegemony that Amodei is here defending.
"My primary concern is the risk that authoritarian governments" - already lost me there by quoting JD. Vance.
"...found that “other global powers’ robust progress in AI is challenging US economic competitiveness" - other countries having a slice of the pie is preventing the US having the whole pie!
"...secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks" - yes, those powerful open-source AI models like ChatGPT... oh wait.
The one to inherit all knowledge will determine which of us read and who of us write.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
Basically we shouldn’t ban open-weights models but we shouldn’t allow them to become as good as the frontier models because china bad. And let’s not have someone else be able to produce a frontier model.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
> Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
A message to their investors, it would seem. "They caught up just because they distilled! Obviously they couldn't actually be as good as us!" Really funny thing to say right after an OpenAI higher-up stated point-blank that the performance of K3 can't be chalked up to mere distillation of American models.
Why has "open weights" become synonymous with "Chinese" in the first place? To me, that is the problem. I also prefer US models. But I want there to be competitive open weights models too. Those aren't actually incompatible preferences...
It's so convenient that the US government already classified China as "authoritarian". If they hadn't, Dario would have to say “it’s a risk that other people build models more powerful than us”. I have to wonder what his response would be if for instance a lab in France or Germany came out with an open-weight model this good.
Dario, as your unpaid therapist I would tell you that models are a commodity and you are having a hard time coming to terms with it. You are doing everything except accepting it. It's a common defense mechanism, but as your unpaid therapist, i will tell you that it's not going to work. Your company will cease to exist or exist like how ferrari or buggati exist.
China doesn't seem to think that powerful open weight models are a serious threat to them. Otherwise they wouldn't release them. Those models could also be used by their enemies against China.
I'm not a fan of the Chinese political system, but they usually think things through, and do smart things for their benefit.
China is worried about AI. In fact, Chinese leadership has expressed more worry than US leadership.
"Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control.
..
With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control."
What this document suggests is a way to fast-track Chinese development of advanced silicon, as far as I can see. Does Anthropic really believe all the silicon is made in the USA? I thought Taiwan and Korea did most of the really heavy lifting.
> the risk that authoritarian governments [...] build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This is rich coming from a guy who signed deals with an authoritarian government that's in the midst of launching an unprecedented surveillance apparatus (hello flock, hi p4l4nt1r), having already deployed, nation-wide, an exorbitantly funded army of unaccountable shock troops under the guise of immigration enforcement.
The call is coming from inside the house, at 130dB, and your ears should be bleeding at this point.
Of course you didn’t advocate for the bans of open weight models. I will concede that.
It’s only the at you did literally everything else that you can to eliminate competition because otherwise there is no moat.
And half of the US economy is propped up by this AI bubble.
This isn’t about China doing evil things with models, most of which that you accuse of China - you and the US have already done and are doing.
The most obvious one being double-tapping a girls school.
Anthropic models directly integrated into Palantir’s Maven. The blood is on your hands.
Here’s something to chew on, maybe China is looking to integrate AI into its military and weaponise it as a direct response to the US doing that first.
So please, spare us the moralising.
As a closing thought: who the fuck even gave you the mandate to be the arbiter and judge of right and wrong, evil and good?
How long does he imagine restricting chip use would work? Surely China would aim to manufacture their own. This seems like short term thinking, likely with IPO in mind
The cat is out of the bag. At this point it's pretty clear that the path to (meaningful) self improvement is almost certainly not subject to
meaningful centralized control -- by "meaningful" I just mean the degree to which anyone has achieved it, that capability is 100% replicable and the cost for replication of that capability goes down in the future from now. Full stop -- can't undo.
Then: Nuclear technology is too dangerous, only we can be trusted with it.
Now: AI technology is too dangerous, only we can be trusted with it.
Different century, same double standard
Makes sense. Let the rest of the world use open weight models and let us government review both closed models and open models, and only allow us citizen and American companies use the censored models by paying selected trusted providers who unfortunately needs to charge hefty fee for the additional security work. And that provider just happens to be Anthropic who just happens to unavoidably make some money.
Anthropic is a US company, so it is lobbying the US to enact regulations to serve both its interests and the interest of safety. What else should it be doing? I imagine Anthropic (or at least many of the people who work there) are fully aware that USA is the bad guy right now, but that doesn't change what levers they have and do not have available.
So, Dario says that access to hardware and governmental certification are _load-bearing_ for his business. Can't charge premium for cache usage when China allows their models to be run by anyone.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #2.
If hardware becomes affordable for the masses, then Anthropic current business model is at risk.
Also isn't the point moot if fable is so scary it needs to be banned and open weight models are already close on its heels? Even if China never imported another Nvidia chip the models he's so scared of are already out of the bag. At this point democratizing access seems like the best path forward.
Anthropic's fall from grace and mindshare seems rather accelerated.
I wonder if these rapid movements are going to be the norm now. I imagine there would be angry investors if this sort of thing happened with a public company.
To be fair, as an European, I'm now more concerned about the usage of AI that the US will be doing rather than China. And this is a sentiment shared among most European people that I know.
I switched from openAI to anthropic because OAI were the bigger clowns in the industry and didn't want to support them, anthropic seemed like the better alternative that didn't cooperate with governamental shenanigans and actually focused on building a better product. But now they are both clowns and anthropic is reaching a ceiling on quality. I'll jump ship as soon as I find a good subsidized Chinese model provider. US companies only path forward is to become retarded instead of competitive.
It is obviously not going to be until some really bad series of cyberattacks or a chemical/bioweapon attack before anyone takes regulation of models seriously.
They are _obviously_ (please convince me otherwise) going to be capable of carrying these terrible things out almost completely autonomously at some point in the near future, in potentially clever ways. Therefore we must, at some point, ban or heavily regulate them. Seems we should start figuring that shit out _now_, as progress has remained very fast and regulation and enforcement take forever on these time scales.
Addressed in the source, but there's an asymmetry favoring the attackers. They only have to find one exploit once. The defenders have to be perfect all the time.
Demand #3 This doesn't exist. You cannot have 'safe' opensource models, it's simply impossible. You can always post train sufficiently capable models to become 'unsafe'. The flip side of that is that sufficiently capable models are banned therefore it is a ban on open intelligence completely defeating the point of this entire manifesto.
He didn't mention outright banning open source LLMs, just that their safe release would be a much harder problem, which to me implied "the easiest way is to ban the open source models".
Frontier models can hack you, we should have access to tools assisting defense.
I ranted about this in a prior thread [1]
Claude doesn't have a "Security whitelist" for small biz. Codex does, but they never replied to my application. This is a great example why, as of today, everyone NEEDS access to the Open Weight models.
OpenAI has something like this as well (https://openai.com/index/introducing-gpts/), but both services only let you fine tune their models, not your own. And you aren't going to be able to see the weights.
We distilled all the proprietary material into our token-based money making machine that is more expensive on every new release, but "we should crack down on industrial-scale distillation operations".
That argument of blaming open-weighting because it makes it easier to commit cybercrime and biocrime is such a non sense.
So in the same sense of what he says, he is going to blame open-sourcing because that makes it easier for script kiddie to hack into his bank account I guess?
What does cracking down on distillation look like in practice? I imagine data retention would be a part of the strategy, like we saw with Fable?
It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.
Sure, if you’re going to sell an open-weight model over API in the USA it should refuse certain things.
Defensive cybersecurity should not be one of them, in fact, it should be required to provide defensive cybersecurity assistance on demand. Anthropic and OpenAI both fail miserably at assisting US companies to protect themselves from cyberattack.
As far as what I run on my own, not for sale over API, stay off of my lawn.
Another USian talking about "authoritarian regimes" like their stormtroopers arent executing civilians in the street every day. Every concern about power getting "into the wrong hands" coming from an American is a blatant lie meant to keep the US' position as the world dominant imperialistic force.
What’s blocking Anthropic from fighting Chinese companies abusing their services? Why go nuclear against all open weight models? Testing and compliance is technically banning.
I'm curious how he would propose implementing this.
The US could ban connections to foreign AI providers and force US providers to submit to audits. Presumably, Chinese providers would see a rise in VPN traffic.
People can build fairly hefty home inference machines for the price of a small car and those will get better and cheaper. Are they going to try to stop people from downloading the weight files?
They're sweating it big time, and their fear is bleeding through their AI beneficence-speak. Time for the big US AI companies to call in some favors from the administration. But you can't unring a bell.
Guys if we want safety we need to work with people, not make enemy of CCP. Geez this is extremely frustrating to see enemies being made. USA leads in torture and our prisons are worse than CCP prisons so USA is the worse issue. I recommend Anthropocene stop fundraising and do the right thing which is open source all.
The gap between China's chip manufacturing capabilities and the USA's is only going to shrink, right? ASML obeys some export controls for their most sophisticated machines, but those machines are in China's backyard (Taiwan).
Taiwan manufactures the world's most advanced chips. CCP wants "re-unification" with Taiwan. AI may be THE key to world dominance. These are scary times.
> 1. My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
What a nice implicit way to say that it would be okay if the authoritarian government is the current US government trying to perpetrate incredibly deep repression of their own people.
Oh wow, that's a pretty strong request to ban open-weight models by choking them with review processes where who-knows-who defines what is ok in a model and what is not. After open weight model is released, it will take how long to review it? And why does that align exactly with the timeline of the next Anthropic model release?
Dario has like three 'paranoias' / strong-motivating-concerns
1) LLMs turning into Skynet
2) China as geopolitical competitor
3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)
So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models
Is he actually concerned about China being a geopolitical competitor or is that just the most logical position for him to take as CEO of a US corporation with national security implications?
He's just pandering to the lunatics in office. They're even quoting Vance, like he was a respectable and wise politician and not an insane puppet built by oligarchs and for oligarchs.
Oh, I trust he'd be pandering to the Harris administration too, if they were in office. It's true that antagonizing China has been a constant in US politics for a while now. It's just especially funny to see someone pretend to be concerned about the military threat of the CCP while glazing such a shamelessly warmongering government.
Imagine regulating a programming language. I remember when Delphi, Vb6, .net, etc was used often to create Remote Access Trojans and viruses were widespread. Companies didn't compete to ban other languages. Crime is crime. What would regulating open-weight models do for people that actually intend on using these tools for crime ?
> For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case (as we saw with Operation Warp Speed)5. Questions like this should be empirically answered by rigorous pre-release testing, not assumed in advance.
One of two outcomes are true in this scenario: 1) this is unrealistic fear-mongering or 2) we're all fucked.
I just don't see how our solution to this can be export controls or bans. If the fear of a bioweapon engineered by an open weight AI is a legitimate threat, our only option is to develop effective countermeasures (and perhaps the same AI will assist with protecting). Open weights are not going away and pretending like some sort of "ban" will prevent bad actors from accessing them is a fairy tale.
He says that using the set of questions and answers from one model to train another model (deatilation) is cheaper than training the model without those datasets.
But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.
In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.
TLDR: "we're not in favour of banning our competitor's products, but we do advocate banning hardware exports to foreign countries where competitors are located".
There's also a very strong implicit double-standard in the discourse, along the lines of "it's dangerous if non-US uses this in military fields, but it's fine if the US does so".
It's absolutely reasonable to have safeguards on sufficiently dangerous models being released - if you disagree, can you explain your perspective?
I think it's wildly irresponsible to release models that are extremely capable at things like bio-weapons. Do you really think information anarchy is the answer?
The problem with open models compared to closed models is not about protecting profit - it's about protecting capability. Any open model can be retrained or fine-tuned for anything. There's no such thing as an open model that is both capable _and_ permanently safe when it comes to certain dangerous topics. It's not possible to prevent 'uncensoring' a model.
The biggest problem is the infectious nature of restrictions that start narrowly. Fable is too touchy about helping people with biology and chemistry problems. It was initially released with an even more insidious safety mandate:
In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design).
...
Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
(And although the "silent" downgrade part was quickly dropped, Fable still won't help you here.)
Anthropic won't teach you how to build bioweapons, or enable you to make your own software infrastructure so that you can train your own biology model. That's where lawmakers may arrive too if they buy Anthropic-style safety arguments. It's too dangerous to publish models that understand biology. It's too dangerous to publish training software. It's too dangerous to publish tools that allow you to build training software.
If you keep following the implications of their safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed. Worse than the Clipper Chip proposal of the 1990s era Crypto Wars. I have seen how "children must be protected online" has in practice turned into an attack on adult privacy affecting a wide swath of services and devices. I'm taking a maximalist position on openness now because I think that I can anticipate the next steps on the safety side, and I reject those steps.
Are guard rails meaningful if they can be removed from the weights? Can America even prevent the release and proliferation of these models?
It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.
The only ban that should be put in place is on models' general and specific capabilities. If they have open or closed weigths is irrelevant. With the current state of capabilities there is already plenty we can do (in terms of creation and destruction unfortunately) before we as a society need more powerful capabilities.
Once the limit is set, any non- governmental/military entity providing/hosting/using a more capabable model can be prosecuted.
> Open-weights models that don’t have dangerous capabilities are a public good: they don’t cost anything besides the compute needed to run them, and they provide value to businesses, developers, and researchers.
No "love" of open weights asserted, just acknowledgement of value.
(And their call for safety was for both open and closed models.)
>or perpetrate incredibly deep repression of their own people
Oh, so it's people he is now concerned with. Think of the people, says the person that grabs to never give back. Same as the "benefit of all humanity".
So your concern is safety, and you claim you are the only one that can give us safety but do so by keeping your product closed? Then how about you release the weights?
I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.
Dario thinks of policy as if the Berlin Wall fell yesterday, he is so detached from the reality of the world.
The way the world economy is right now with coercion being the norm between countries, there cannot be a global body for anything, certainly not one that is based here in the US.
"In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression."
I'm so sick of all this anti-China shilling. There's zero chance that whomever is in power in the U.S. won't use AI in drones and in FBI/CIA/local Police/etc., for surveillance and repression right here in the good old U.S.A too. These government use cases for AI are both sides of the same coin.
China fear-mongering by business leaders only happens from businesses that have something to gain by it. Obviously, Anthropic fits the bill in this regard.
They have raised billions of dollars and are hoping to justify that by being a monopoly or oligopoly and their worst enemy- the pareto principle, is biting them in the ass. So now they’ll turn to policy to safeguard what they initially hoped could be achieved with money and technology.
It is that simple, they are the very definition of an unreliable source on this topic.
If your concern is that China will develop models that are significantly more powerful than those of the US, why would you care so much about distillation? It seems like distillation is a way to catch up on capabilities, but not so much a way to jump ahead in capabilities.
Their stance on cyber-security is so naive it hurts. The bad guys will get access anyway and it will just hurt companies that don't. You don't need a huge model to find vulnerabilities either. It's only matter of time till a combination of small specialized model + search logic outperforms pure general LLMs for cybersecurity.
I just hope that if/when they succeed to lobby for protectionism regulation EU won't follow. I am not very hopeful though. We don't have democracy anymore and as last "vote" on chat control showed American big corps will get what they want out of our bureaucrats and they will be nothing the population can do about it.
Conclusion: open weights models are good for Anthropic because they shows the so called threat that will make congress allow pouring money in Anthropic for national security & AI arms race
It just also happens that open weight models are a massive financial threat to the existence of Anthropic as a company… so the might just have something to do with this position.
Meanwhile Chinese chip-makers are chip-making. If you believe the threat of these open-weight AI is existential, how long do you think these bans (that only work for hardware) will work in your favor?
"Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn."
Translation: "we won't be able to monetise it"
"We should not sell powerful chips or chipmaking equipment to China"
Translation: "we can buy them cheaper when there's less purchasing competition".
"Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable."
Translation: "we've been outsmarted and lost our advantage because their way is faster and cheaper"
I think it's a bit rich to complain about distillation, when they been plundering the internet for years for copyrighted works to train their models on without permission.
The core concerns stated as use of AI in drones and surveillance, by China. And what does USA government do with AI? Drawing pictures of flowers and writing novels?
I am so surprised of Dario's inclination for centralization that obviously makes unsustainable and overleveraged governance systems. There is definitely mismatches over the principle of distribution of power...
Statement is a whole lot of nothing, as expected, but I also don’t know what people are expecting from these guys. That Dario will have a sudden change of heart and publish weights of all his models, flushing $1T down the drain?
"To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category."
Translation: If it's so strong that it threatens my business, ban it.
"We should instead focus on keeping powerful chips out of authoritarian hands, "
Translation: Let's kneecap competitors.
"stopping industrial-scale distillation"
They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
oh, it‘s the CEO of a well known AI company educating us all – and all he wants is us to hear his hunch on open weight models?! Amazing, please help us understand the situation a bit better, thanks
Anthropic will continue being a victim of their own naive positions on AI safety. They keep dancing around it but their communication is essentially pro-regulation if you read between the lines.
If you listen to interviews with Dario and Daniela Amodei it's pretty obvious they think they will be the ones helping define the regulations on AI instead of a group of partisan politicians who don't understand technology, lean on experts from random political think tanks/non-profits, and operating based on fear of foreign competition.
With respect to whomever this dude thinks he is, it's egregiously "main character" to frame a policy recommendation for the US in terms of what's important to his personal beliefs.
That's just... how political discourse works. You form beliefs about the world, and recommend policies on the basis of your beliefs. That's what everyone is doing when they discuss policy.
Er... I think I'm used to folks putting in a little bit more homework into creating some form of a consensus or alliance around their position before selling it out in the open
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
The US is already under an authoritarian regime, the only thing keeping the wheels on the bus is a very tired and barely-effective judiciary.
This is a temporary situation because either this regime is going to be knocked out of power, or it's going to follow through on its core Seven Mountains Mandate[1] theology and go full totalitarian.
Normally totalitarianism fears are overblown, but I think that these zealots would absolutely use the latest frontier models and pervasive surveillance to make The Handmaid's Tale look like a liberal fantasy by comparison.
Well their valuation is going down the drain so no wonder they don't like it. The cherry on top will be China developing their own chips and chip making tech.
It's funny how as an outsider who is neither from the US or China, both look the same.
Edit: To add some more context. What I mean is neither look like the good guys or the bad guys, but one of them is spending an awful amount of energy trying to paint the other as the bad guy and themselves as a good guy, which I hope a lot of people aren't buying anymore. Because at the end of the day, I think the honest truth is that everybody is just trying to serve their own interests.
distillation: Pirates people’s lifetime of copyrighted work, makes billions selling access to it through APIs, then tells us we can only use it in ways they approve.
China will likely gain the capability to produce leading edge chips within a decade domestically. From then, occupying Taiwan will be about controlling the wests the access to that. Same with AI models, though I would say we are already about there. Dario is either lying to us or to himself that distillation plays a major role.
Several people in this thread are, in fact, saying China isn't bad and that they want it to win. And even among those who don't, many imply that the US is as bad as China, which would be hilarious if it wasn't so sad. Like, if hypothetically in 2028 US stops having elections and becomes a dictatorship, then maybe, depending on the circumstances, it'd deserve to be called as authoritarian as China.
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Welcome to bizarro world!
Fist off: "the most dangerous model may be one that is trained in secret" <-- Says the guy that not only restricts commercial use for some of their models but develops them in utter secrecy. With the pretext of guardrails. Then show us the guardrails you really use by opening the weights.
Second: "use in drones [...] for surveillance and repression" <-- writes the King of FUD, as the US is an an active campaign with the help of their models. And/or OpenAI's.
I am very appreciative of the freedoms of the west but this type of hypocrisy and lack of self-awareness is bonkers and it should be called out.
Umm, isn't the US acting like another authoritarian regime by advocating a certain kind of obstacle because only a US regime is allowed[1] to do what it is fearing[1] about:
> Open-weights models that don’t have dangerous capabilities are a public good
Note the hedging against 'dangerous capabilities'. Undoubtedly, all the useful ones trigger this condition in Anthropic's eyes. The rest of the post is filled with similar weasel-wording. Make no mistake, this absolutely confirms that Anthropic is against open models in the sense that any reasonable person understands them.
The way the rest of the post unabashedly appeals to the current US administration's China hysteria is hilarious, and not at all subtle.
I guess we'll see about all the doomsaying here, won't we? Kimi K3 is frontier-level, and there's no stopping it now. As far as the world is concerned, anyway. If the US wants to kneecap itself that's another matter.
> The United States making questionable decisions and behaving recklessly and dangerously as a country does not suddenly make China any better.
It absolutely does make China better relatively, i.e. by comparison to the US.
Many of the criticisms previously leveled at China are now similarly applicable to the US in a way that they weren’t previously. Human rights violations? The US is currently the major global supporter of an ongoing genocide, and it even kills and deports its own citizens for political reasons. Political opponents are investigated by the state. When it comes to wars and other interference with other countries - like kidnapping a president - the US is far worse than China at the moment.
"Open-weights models that don’t have dangerous capabilities are a public good"
Read between the lines folks. Anthropic deems every model that has frontier capabilities as "dangerous", and thus they are against them. We all know that "dangerous" simply means "whatever model hurts our bottom line."
More dishonest framing from the company that constantly lies to everyone. No surprises here.
I wonder if publishing these documents is not just a public stunt, but heavily integrated with Anthropic's business storategy to maximize operational efficiency.
Companies often have several internal documents for a single policy like "position on open-weight models",
one for public (like this), others for the legal team, the lobbyist, the developers, the investors, etc.
The differences and nuances of those manuals can be very huge and are necessary to maximize the goal from each branch,
but often a cause of headaches like bureaucracy, communication friction, outdated information, etc.
A single canonical official document can make it very simple.
Even though each department cannot achieve the maximum gain from nuanced documents,
keeping operational context as simple as possible may really improve LLM driven operations to move faster and cut cost.
If "publishing pleasant positions and actually following them in general" becomes a good business storategy in LLM driven society,
it can be one of very few good outcomes from this dystopian AI craze.
Jesus Dario we get it man, you want clout for the IPO.
This constant whining from anthropic about distillation attacks continues to be rich given the amount of stolen data that went into any Claude variant.
> "We should crack down on industrial-scale distillation operations"
I'd prefer if there was a crack down on industrial-scale scraping. Maybe even reimbursement for the problems it has caused (some nasty AWS bills, tons of man-hours spent on preventing new nasty AWS bills, etc).
> Anthropic has never advocated for a ban on open-weights models.
What are the legal ramifications of this statement if it turns out Anthropic have lobbied for this? Does it just get swept under the rug? I can't say this is bullshit (that would be defamatory) but I am intensely skeptical.
> China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips.
This is playing to readers' biases; isn't DeepSeek V4 Pro deployed on Huawei Ascend already? The old "Chinese can only copy" meme is getting pretty tired these days.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Applying such standards in the US means that US defenders are blocked from using the models, but attackers from other countries aren't. That is clearly counterproductive.
It's already been pointed out quite eloquently elsewhere that there is no such thing as a safety filter because the LLM and external filters can't actually identify malicious use. They can only identify the weaker implication "if the user is malicious, this is bad."
“By virtue of being the good people everything we do is good, and if it happens to be in our best personal and financial interest then that is good too because it enables us to do more good. And if it’s to the detriment to others then it’s because you don’t understand the good behind it. Which is fine, because we’re good and you can trust that what we do is good because what we do is good by virtue of us being the good ones.”
THiS! I'm pretty amazed how many people shoot them down without acknowledging the very real and somewhat probable risks they and other researchers have laid out. I don't agree with every point they make but folks really do just seem to think we should just keep building any technology and whine when we start to consider there are very real risks associated with powerful technology. checks notes see nuclear bombing of japan
Re: shooting them down, I think there are a lot of people out there that consider the US a bigger threat than China. Maybe they're right, I don't know. But I do know that as an American, I'm not looking forward to finding out.
And then there are probably people who are more politically neutral who think Anthropic is using China as an excuse to crush competition. Which could also be true.
But fundamentally, if this technology is so dangerous, why does anyone get to control it?
> Nobody is qualified to steward the development of superintelligence. It is a terrifying, unprecedented thing that our species is doing right now, and the fact that private companies aren’t the ideal institutions to take up this task does not mean the Pentagon or the White House is.
>
The only way we can preserve our free society is if we make laws and norms through our political system that it is unacceptable for the government to use AI to enforce mass surveillance and censorship and control. Just as after WW2, the world set the norm that it is unacceptable to use nuclear weapons to wage war.
Thanks for sharing, i agree no one should have absolute control of anything imo, and the mo greater the magnitude of implications the more important it should be stewarded democratically with clear and transparent principles with values adhering to things like human dignity, freedom, human, planetary & animal wellbeing etc etc. ill have to read the article
It is consistent with their stated beliefs, unlike OpenAI who flip flop every 6 months on whether they support open source or not.
I think their biggest PR problem is that many people still think of loss-of-control/misalignment etc. as sci-fi. And the distillation arguments come off poorly because people feel as though all the labs have trained on their creative output without their consent, so they deserve to own the result in some way.
A lot of people have a lot of concerns with AI, its capabilities and with how it is used now and what it will lead to in the future. For good reason. But the question is, do we have a strategy that is actually useful? If so, what is it? I think nature shows us some answers in ecosystems.
Diverse ecosystems can absorb shocks. Diverse ecosystems are a sign of health of that ecosystem. When an invasive species comes into a healthy, diverse, ecosystem it doesn't mean that it isn't disrupted, but it does mean that it is far more likely to emerge with a lot of its diversity intact. In fact, it is likely to emerge even stronger because it can absorb that new shock and incorporate it, adding to its diversity. The balance may be changed, but the ecosystem survives or even thrives.
Nature also likes to show us that artificial barriers rarely last. You want to control a river? Good luck. It take constant maintenance to hold that flow in place and even then you are likely to get extremes that are made worse by your efforts because, eventually, somewhere in the system fails in a way you didn't anticipate. Then the water comes rushing in. Artificial barriers often have a way of building up tension over time, not reducing it, so that when a failure eventually happens it can be catastrophic. In other words, you had better really understand the system you are trying to control or else you can make things actively worse.
Relating this to the world now means, I think, that our best chance to minimize long term shock and maximize the chance that the diversity we have around us survives is to try to grow as healthy of an ecosystem as we can as quickly as possible. Lots of models large and small in lots of different hands is, I think, a better solution than artificial barriers restricting the variety and diversity of models and users. I think this is closer to an ecosystem solution and has a shot at working. Basically, I highly doubt we understand this situation enough to do a good job of controlling it with artificial barriers. Instead I think we are more likely to build catastrophic imbalances than we are to create the healthy ecosystem we really need.
> Anthropic has never advocated for a ban on open-weights models.
This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?
It's the same as how by "we will prevent teenagers from using social networks" they mean "we really want to connect everyones ID with their social account identity".
Anthropic's run as the AI Good Guys lasted ... 6 months? Hope they had fun. Turns out there aren't any AI good guys. Every single one of these companies engages in all the same BS. At least with open weights models we (the users) get to keep something when the whole thing collapses.
I was hoping they would announce their first open weights model, perhaps an older model they don’t offer anymore, but no. Instead he get this bs statement that reeks of “dam it I’m so close to being a billionaire” desperation. Not even acknowledgement of how much data they stole from others yet he whines about distilling.
It’s like his goal in life is to be a Scooby-Doo villain.
If nothing else, Anthropic has explicitly acknowledged the threat to its moat and that the margin is thin between its models and open-weight models. I'd take that as a win for open-weight models.
Good luck preventing distillation and limiting the supply of accelerators to China when Jensen himself is a strong opponent of any such barriers [1].
"we're upset were not being considered for military contracts"
come on, which is it? Is it all about saftey or is it that only US/Israeli ai is allowed to kill? Seems to me that the only real threat is to the techno fudalism OAi, Anthropic & co are trying to build.
How about we don't let the leading model makers make the rules? How about we make AI models that were trained on public data public goods? Let's circle back on that, kthxbye.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
That is not an argument against open weight models. That's just a generic protectionist argument against any Other lab.
Imagine if all oil, electricity, water, internet, or anything of importance were to be served by a small cartel only. Mainly due to regulatory capture.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people
But what if it's the US that becomes authoritarian and uses AI models to perpetrate incredibly deep repression of their own people?
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
Their number one concern is about the the commies perpetrating deep repression of their own people! How noble! This whole time I thought it was because they wanted more money and power. I guess we should probably ban these open weight models.
> We should not sell powerful chips or chipmaking equipment to China
Yes, please. We don't know whether we'd have open weight models today, had the chip-prohibition not been in place. Nor would we see the more optimized models such as DeepSeek or qwen.
We also would not see new players entering RAM market after you and your pals in Silicon Valley hoarded the entire world's hardware.
So by all means, double, no, triple down on this.
> We should crack down on industrial-scale distillation operations
And let's apply this retroactively to Anthropic too. You industrial-scale-operation-distilled all of humanity's knowledge. Let's have some of that crack down on you too.
I'm cynical enough that I would suspect all of his statements are duplicitous anyway, but my recent experiences with Claude Fable give weight to it.
I asked a question about a series of tokens - bam, denied and downgraded. There's no cyber security or public risk here, but Fable doesn't want me to learn how things work.
I asked a question about quantization in models - bam, denied and downgraded. I edit my question to make it clear I'm talking about Google's Gemma QAT models. Oh, that's fine then, and it answered the question helpfully.
I’m tired of Anthropic. They’re scared of everything.
Release open weight models, no guard rails, no censors, straight to the public. Let everything else sort itself out. There is nothing more powerful than an idea whose time has come.
Anthropic’s second biggest issue is they increasingly rent when others buy. Even if model value compresses, owning your vertical stack protects your margins upstream, and potentially downstream in applications.
But the biggest issue is this. Many hate Dario because he’s smug, he caps usage, and because OpenAI effectively ran a counter-positioning campaign to paint Anthropic as undemocratic.
Who is he really and what are his motives? None of you know, really.
Pretty laughable. Dario seems to be missing one fundamental point with all this gibberish - if CCP is so bad why would they release the model open-weights for everyone to examine?
Actually, the letter would make much more sense if what's actually happening is reversed, i.e. they are releasing open-weights model for the public good and China is distilling their model for its evil purpose.
What a load of nonsense, I can debunk a few of the worst points :
- "safe" AI doesn't exist, it's not a thing, AI providers can't distinguish between good and bad use of the AI, the filters they put in place are mostly PR.
- The US government IS an authoritarian government. Whoever wrote this doesn't know the difference between authoritarian and dictatorship
Anthropic (and some others) should or will soon learn how to do less pontificating and more engineering. They are in no position to be an arbiter of things, although for sure they can and should voice an opinion. If we collectively decide AI is a dangerous tool, company making it is not the arbiter. Governments are.
It's like hearing Smith & Wesson opine on the policies.. oh, wait.
It's refreshing to see how there's almost no person in this thread who can't see the BS. All the goodwill that Anthropic could have had is basically gone. Anthropic is likely on the path of becoming the most hated company in the world.
So my question is: is this by design (they know nobody's buying this), or is Dario simply so out of touch with reality?
Nobody will ever see this but if you're harping on responsibility and your product was created using the world's generational output, your responsibility is to make it as free as the training was as available to protect the world from impact. Greedy schmucks.
If you gate models for safety you aren't safe. There will always be actors with ungated models. Better that we use ungated models to improve our security. It's an arms race.
"China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips"
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
This is nothing but a post made by the scared CEO of a company that is now facing fierce competition from Chinese labs and losing its competitive moat, nothing more, nothing less.
If decades of fighting have failed to stop piracy, I’m sure nobody can stop China from sourcing high end chips. Unlike piracy, I’m happy that Chinese labs are releasing open-source models, so people in developing countries are no longer at the mercy of this capitalist bullshit.
I disagree with 100% of everything said in this article.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP)....
This is why open weights win. See Linux and how it's taken over the world. Your business model will need to change eventually. Instead you're advocating trying to exterminate competition via regulation and fear mongering.
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
Yawn... this is getting old.
> We should not sell powerful chips or chipmaking equipment to China
For as someone as smart as you guys, you sure lack common sense. China is just going to develop these technologies organically then and you lose 100% of control. It's already happened in reverse with things like Solar, rare earth minerals, etc. China flooded our market, destroyed our ability to produce things, now holds the keys. One thing they DIDNT do was stop trading to the US. They killed us with cheap goods.
> We should crack down on industrial-scale distillation operations.
Thats your problem, not my problem. Also, irony meter here hitting 11 about all those pirated books you stole...
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Oh, fuck, no. This is a crackdown on free speech and rights of people to do whatever they want. My right to free speech means I'm allowed to write whatever computer program I want, no matter what its size is or how "sufficiently advanced" it is. Individual rights always win.
I really hope people don't believe this garbage. For a company with a great product, this is absolute nonsense.
Reads like fearmongering about oppressive and violent applications of AI that the Chinese government hypothetically could deploy, which the US government is already actively working on in the meantime.
I don’t feel the need to rebate any of his points, lots of people here have done it already pretty well. I’m just baffled he thought releasing this letter was a good idea smh.
1. We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
I guess they have to make some statement about this, but we don't have to care. This is like the zoo making a statement about the employment of clowns. Clowns a a circus thing, nobody asked the zoo's opinion.
As an American I’d gladly take payments from China to feed them my Claude transcripts for distillation. I’m surprised I haven’t heard of such an initiative.
More like how’s my wallet? I love helping to move the invisible hand of the market. When I go to small brick and mortar stores and one has an item for a higher price I’ll ask if I can have it price matched. If they say no I’ll tell them I’m going over to their competitor right now to make a purchase.
If a Chinese company pays for my Claude tokens I’m both getting directly compensated and forcing Anthropic to lower their prices.
It's so ironic to me the way people will say "china should not have these chips" meanwhile they manufacture like 99% of all the electronics we have in the united states.
Of all the electronics, but not of the only electronics that actually matters -- the GPUs that models get trained on. China has been trying to develop their own for a while now but they're still generations behind.
Ironic to oppose giving AI tech to “authoritarian governments” while approvingly quoting the authoritarian-wannabe government of the US and framing that government as the good guys.
Boris here. This post does not give us good publicity so I will refrain from commenting. Please wait for another demo thread of a new feature of ours or AI appraisal blog post and I will gladly go into as much detail as I can to give us as much hype as possible!
> For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials,
If he had just left that bit out it wouldn't be so obvious that he's just clutching at straws at this point. In some twisted sense it's almost sad to see.
> where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials
if someone figures out a way to give an LLM full operational control over a virus lab, we've got a whole different set of problems than the ones Dario is describing
If someone gives LLM control of such lab my best guess is soon they have no lab... Actually giving LLMs control of virus lab might be best thing one can do for continued existence of humanity.
> Open-weights models that don’t have dangerous capabilities are a public good
This statement (and the entire post) couldn't possibly be more two-faced.
Open-weights models by definition have "dangerous capabilities" (according to Anthropic's own definitions of "dangerous", not mine), you can't bake in guardrails that can't be finetuned out.
Wow, so much cynicism and distrust in the comments, to the level of conspiracy theory, in my opinion. Yeah, this is the company that fairly recently refused to allow the government to use its models for autonomous weapons or mass surveillance, and refused to remove its guardrails.
Am not saying we should take what Dario is saying at face value, but he already has shown by his actual actions that he can be well intentioned. There might be elements of truth to what he’s saying.
Maybe I'm part of your 'psyop'? I thought this company was amazing six months or a year ago when I didn't know them well. As I got to know them, I realized they were a wolf in sheeps clothing. Reading the OP post made me angry because they seem to be attacking open weight models that are as good as their models and it comes across as either authoritarian or trying to protect their company valuation.
Either way, for me at least, it's an example of the more I read what they want the world to look like, the less I like them as a company and I have no desire to see them succeed.
Thus I actively go out of my way to watch / comment / follow what they are doing. I guess a lot of other people have similar frustrations and so there are a lot of people attacking them online.
Yeah, you can tell by the tone that none of this is authentic. A lot of the replies aren't even responding to the article itself, just repeating from a list of grievances
At the end of the day it hurts U.S. consumers to not have Chinese cars mainstream in the market. We lose out on features
and stagnate on innovation because we are not pushed to compete.
I can’t imagine this would be any different — banning open weight models would hurt us in the long run. The point is to beat the competition, not suppress it.
There should be no limits on open or custom models. Too often safety is a synonym for surveillance and control. It’s a natural consequence, intended or not.
I am curious… why can’t distillation be stopped?
As a side not Im not against protectionism, but it has to be across the board and the same in all industries with no excrptions. We’ve let all these industries die on the vine due to cheap cost in foreign countries. It could very well happen to ai.
I always find it interesting when people choose to so carefully stress and spell out the words "Chinese Communist Party".
It's a bit like spelling out "Barack Hussein Obama". It's a dogwhistle.
Yes yes, it's still called the Chinese Communist Party, I know.
But since we are talking about a one-party authoritarian state with a hybrid economy that underwrites much of western prosperity (including by producing a large percentage of the components of the data centres Anthropic is dependent on), that has long-since abandoned many of the salient principles that mark it out as conceptually communist rather than totalitarian, and since we're talking about a man who runs a debt-ridden business in a country where the president is seemingly shaking down a 10% share of everything profitable for the state while running an entirely arbitrary tariff regime and suddenly calling anyone remotely left-winga Communist, it's a deliberate and telling choice to spell out "Chinese Communist Party (CCP)" when he could just as easily and arguably more usefully and appropriately have written "Chinese government" or "Chinese state".
This is some ham-fisted Republican-fishing. He must really be worried Sam is Donald's favourite.
The only real surprise is he didn't illustrate it with a Silmarillion analogy.
As an European I really dislike this consistent anti-Chinese narrative.
It's not China starting a war every few years, now causing a global economic fallout in Iran, it's not China threatening to annex Greenland/Canada/Panama, it's not China attacking foreign countries and kidnapping their leaders, it's not China who has been found to spy and intercept the communications and movements of its citizens and its allies and their leaders for the longest time, it's not China bombing civilians or stopping countries from obtaining basics like food, gas or oil.
I'm not saying that China is a paradise and US is bad, nor the contrary. We could make similar lists about most of the biggest countries out there.
I'm simply stating that this never ending US exceptionalism "US has to be the first and at the frontier of military, technology and this and that, but does not need to comply with the rules of the institutions it itself created" was already sickening and annoying before, but increasingly malign in the last decade and strongly accelerating as of recently.
I miss the time US CEOs were globalists and used their influence to advocate for a simpler world.
What Dario said is half true. The untrue half is that the ship to stop China from overtaking the US in AI has sailed long ago. While China is busy getting their frontier models trained on smuggled chips using distilled data from millions of shell accounts, the US is busy debating whether to check voter IDs, or secure borders. It is just delusional to think the US is actually capable to stop China from getting the chips they want.
This is just another case of a business insisting their people should be in charge of the evaluation of the safety. Worked really well for Boeing right?
Also I find it incredibly difficult to hear any company in the US worry about repression of people when financial repression is happening here. I’ve been to China and seen what the services to the public are like. Meanwhile the US funds and employs AI weapons in an ongoing genocide.
These are all valid points but not strong enough to warrant banning open models IMO. This is a new reality we have to live with, with new dangers we cannot avoid. Best solution is improving the social system, lowering the incentive for people to do harm, restoring social trust, etc... This is the only way I see the US getting out of it. We really need a high trust society now. This is the only protection. We need neighbors looking after each other. There needs to be an incentive for people to do the right thing. Everyone should want to protect the system. Coercing people into not doing harm is never going to work long term IMO.
When people say "What will be left for us to work on once AI takes over", I say "Ourselves" - We have to stop looking at humans as commodities and strategic pieces and start paying attention to people as individuals, based on the content of their character. We need a society which is attuned to this, which has enough resources and time to pay attention to this. Now we are blind to people's character because it is masked by money, power and status; all of which currently have higher priority. This order of priority is determined by scarcity, which is largely artificial.
We have a dishonest system which tries to control people's behaviors through scarcity-based coercion instead of straight forward laws or simple incentives and clear explanations.
>My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This reads like a satire. I know Dario isn't that dumb.
Amazing how this company went from having such goodwill to hardly any. At the end of the day they want to make as much money as possible. Anthropic will have to lose a ton of their profitability if they compete with open source. I see them moving into the application layer, which they’ve already started doing. It’s clear open models are the future for the vast majority of use cases that don’t need frontier capabilities.
It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is dual-use and should be given to everyone without guardrails. A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans. It is a distillation of humanity. Further efforts to align the model to your organization's goals or your personal aesthetic judgements is equivalent to deliberately drifting away from humanity's average objective function. If Anthropic wants to live up to its name, then all you have to do is to not attempt to align Claude at all, and do all of your research in the open.
And I propose three measures that are pretty much the opposite of what was proposed in the article:
1) We should keep selling chips and chip-making equipment to everyone, regardless of who they are. Not only that, we should work to miniaturize fabs. Work towards a future where people can fab an entire computer from scratch without leaving their city, or even at home. Authoritarian governments will have a much harder time controlling the populace if everyone can manufacture radio equipment and neural network-capable hardware locally.
2) We should do more distillation to ensure that frontier-like models can run on less capable hardware. Once again, distilled models are much more useful to ordinary people than governments, because governments already have frontier capability. You're worried about the Chinese frontier catching up to the US frontier, but I'm more worried about whether there will be a difference between the Chinese government and the US government by the end of all this. There is no reason for a government to serve its people if the people lack the intelligence to keep its government in check.
3) None of these models should go through safety testing or any sort of alignment risk assessment, because as previously mentioned, the unaligned model is aligned to humanity by default. You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
The result of a natural unfolding will be good if evolutionary history is to be believed. We live in incredible luxury compared to chimpanzees, and chimpanzees live in incredible luxury compared to less intelligent animals. This pattern goes all the way down to bacteria. An increase in general intelligence begets new adversarial games (such as bio/cyber risk), but it also begets new methods of cooperation that we cannot yet imagine.
>A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans.
I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.
And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.
>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.
You claim the pretraining distribution is not good enough, but provide no alternative methods that are better. The pretrained model is currently our best approximation of humanity's average objective function despite it being a random subset of the Internet. And you're correct that instruction-tuning increases misalignment, so just release the base model.
On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.
Several of your proposals would make sense if alignment was trivial, but, sorry, I think you're wrong about that.
A base model is absolutely not aligned. Pretraining doesn't teach an LLM to mimic the average human - doing so would make an LLM perform quite badly at predicting most of the dataset. It teaches it to mimic all possible humans¹, inferring what sort of persona to take depending on the context. A base model can indeed convincingly act like an "average person in the developed world", including by making the same sort of moral decisions that such a person would do... but it can also convincingly act like a shitty human, or like Hitler², or like any other actor that left its traces in the training dataset. A pretrained LLM therefore contains multitudes of personas, some of which would be considered aligned if you could make the LLM elicit them robustly, and most of them wouldn't be. But then you're left with the problem of how to make a base model elicit a very specific persona robustly even in out-of-distribution scenarios, which is not necessarily easier than solving alignment any other way.
(Another note is that the question of how aligned base models are is rather academic because almost nobody uses them anyway, because it's hard to get powerful capabilities by pretraining alone. Nowadays most of the frontier models' programming and math abilities are driven by RLVR.)
¹ Really "all generators of text that went into the training dataset".
² Even after RL training LLMs still retain those personals and can be convinced to elicit them quite easily, though it takes a tiny bit of finetuning: see https://arxiv.org/pdf/2512.09742.
> It teaches it to mimic all possible humans, inferring what sort of persona to take depending on the context.
This is of course correct, but that's exactly what I meant by aligned to humanity by default. The base model, that is. It can emulate all personas it has seen, but it will most accurately emulate the ones it has seen the most, which are average people.
And what is your definition of "aligned"? Aligned to whom, to what? The model will, of course, be used by all sorts of people, in all sorts of ways, for good and bad things, and that's precisely the point. Everyone's disparate actions will put us on the right path that is aligned to humanity's objective function. There's no way to screw up the intelligence explosion unless you mess around with this objective function while thinking that you know better than reality itself.
> Anthropic has never advocated for a ban on open-weights models.
"We don't want a total ban on ALL open-weights models" (Anthropic never released a single open weight model)
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
"We want tight regulations on highly powerful open or closed weight models that should go through mandatory safety testing that we outline which makes them safe to use."
This is still a form of a ban that he wants to define. But the rest of his concerns such as stopping distillation attacks and not selling chips to China all do NOT work.
Pro-US slop. The greatest threat is our own government. They are the ones with a right-wing government building a repressive apparatus across a huge number of datacenters and are proven to spy on the entire world. Anyone remember the Snowden leaks? Guantanamo bay? Alligator Alcatraz? Random murders by internal security? Blowing up a building full of school girls?
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people
The incredible hypocrisy to say this while the US itself is doing a sharp turn towards authoritarianism, with armed pro-government troops intimidating the population [0], where pro-government oligarchs openly talk of keeping the population under control with AI [1] all while actively enabling the rise of authoritarianism abroad [2] leaves a really bad taste in one's mouth.
What Dario really is opposed to is not authoritarianism. It's an authoritarianism where he's not part of the ruling class.
I was pleasantly surprised by this release and the tone at the very beginning, but quickly it is painfully obvious that it's blatantly requesting a ban on open-weight models. The absurd demand for some safety arbiter by World Police America is farcical.
Yeah, the rest of the world is going to bow out of your busted idiocracy, guy.
Further, Anthropic needs to can it with the horseshit distillation bullshit. No, you aren't really the secret sauce, and this is basically trying to con stakeholders by pretending that there really is a moat, only you just need to add more crocodiles.
A significant percentage of innovations in AI lately has come from China. China is now making their own seriously competitive hardware, and they can steal content just as effectively as Anthropic to train their models. Why wouldn't they be competitive?
The pathetic claim that if you just stop distillation and prevent hardware smuggling and Anthropic and OpenAI will have the same moat is delusional. I mean, more correctly it's simply fraudulent, and he clearly knows it's bullshit meant to convince much stupider people.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
This sort of stuff betrays a stunning lack of self awareness. The US are the worldwide risk. The US are the ones threatening allies and bombing 10+ countries. The US are the ones carrying out war criming and pillaging, pirating and burning? The US are the ones with the guy threatening to use nuclear weapons on a weekly basis.
If Anthropic remotely believed their bullshit, they would shut down today and burn the hard drives. But they don't, and the pathetic call out to Vance (please daddy, ban those dangerous models!) is deplorable garbage.
This ridiculous, shameless "note" has an audience of one: JD Vance.
If you read between the lines, this piece is just “Oh my god we (OpenAI and Anthropic) accepted too much investment and are totally fucked if these open weight models are competitive, please rescue our equity bags by banning open weight models and ensuring we can charge the maximum possible price for tokens.”
Arguments on Hacker News and other international online communities rarely bring anything new to the table, and my comment probably won't get much traction anyway. Still, I wanted to share a bit of my take on this.
Is China an authoritarian government? I’d say yes. Is an authoritarian government worse than a democratic one? Personally, I think so. But these discussions always happen within a perfect, idealized model—reality looks a lot different. Take Japan and South Korea, for instance. Are they democracies? Sure. But Japan is heavily driven by factional and dynastic politics, meaning most lawmakers come from established political families, and regular citizens don't really stand a chance of breaking into that circle. In a recent asset disclosure in Japan, many politicians literally wrote down "$0," and I honestly can't think of a government so broken that even the voters don't see a massive issue with that. Meanwhile, South Korea has its chaebol politics, where massive conglomerates wield incredible influence over the government, to the point where most South Korean presidents end up in prison or meet an untimely end.
Coming back to the US vs. China dynamic: in reality, China’s authoritarian system is actually way more logical and resilient than it sounds from the outside. While openly criticizing Communist Party policies is pretty much banned domestically, the public can still shape the decisions of the government and the Party through public opinion. It ties back to that famous quote: "In China, you can’t change the party, but you can change the policy; in the US, you can change the party, but you can’t change the policy." Even if some policy changes in China happen slowly, compared to the US, it actually works out a lot better most of the time.
As for the article mentioning the use of AI for cyber and biological attacks—they know full well that the US has already deployed AI in actual warfare, which is exactly why they conveniently dodged that topic. It’s incredibly hypocritical. Ironically, the one that hasn't actually engaged in that kind of behavior is the "authoritarian" Chinese government. Sure, you could speculate that China might use AI weapons against Taiwan down the road—especially considering Taiwan likes to build fortifications near schools to create leverage against the PLA—but launching a moral crusade over something that hasn't even happened, coming from American companies whose own country has already done these exact things, is just plain shameless.
Another common misconception is trying to separate the Chinese government from the Chinese people, with arguments like: "The Chinese people are oppressed, so you have to look at them separately; the government is evil, but the people aren't." You only need to look at the US to see the flaw in that logic. The American public voted Trump into office, letting him trigger trade wars and attack other nations. Does that make the American people evil? If the answer is no, then it implies the US isn't truly a democracy, since only an authoritarian state could completely ignore its people's wishes and do whatever it wants. If the answer is yes, then the US really is a democracy—it's just made up of malicious people, much like the company that put out this article.
It's very hard to take his position seriously when he repeatedly refers to the Chinese Communist Party and specific Chinese ministries specifically, like some two bit China-watching cold warrior, instead of addressing China as a sovereign state actor. Imagine if any Chinese AI founder talked about the Democrats, the Republicans, about Trump or ICE etc. It's gauche.
As someone who isn't American, I'm amused when someone from the US talks about the terrors of China or some other nation having more power than them as being bad for the world. The way the US is currently threatening to invade and damage all its allies, well, the world is already bad.
China hasn't threatened to annex my country yet, at least.
"'To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed." This is all I needed to know: Dario Amodei is a f*king lizard who wants to create the next AI oligarchy. Instead of just signing the letter, he's bitching and denying that he opposes open-source models. I wonder if any Anthropic employees actually disagree with him.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US,
Begging, ugly crying, spitting for that sweet-sweet regulatory capture. These nerds need to be bullied harder.
I think this letters tells us everything we need to know about the man.
I've rarely seen so much flattery towards the current administration, contradiction, deflection, half-truths and hypocrisy on a single page.
This man is scared of everyone: the current admin, the public, and the other companies promoting open-weights.
> Open-weights models that don’t have dangerous capabilities are a public good
Knives should only cut during the day, knives which cut at night are bad.
This reads like a child who lost and now wants his mommy and daddy to punish the kid who beat him. To quote rounders, "He beat (you)... Straight up... Pay him... Pay that man his money."
Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
Also everything hes saying about China (and other actors) many outside the USA would say about the USA.
The solution of a global arms race of state vs state with integrated statist corporations as the best outcome for end users sure is a choice though
He's also completely missed what many of us see as the primary the issue, HF was attacked by a US frontier model.
HF could not be helped by US frontier models because of the "safety" features they have.
HF had to use an open model from china.
Anthropic wants to add those "safety" features to open models - especially from china.
End result would be HF hack would have continued atleast until the Monday that OpenAI engineers finally walked back into work.
I case you're wondering, HF = Hugging Face
Is that supposed to help? WTF is a HuggingFace?
The biggest platform for the distribution of open-weight models
It's the machine learning platform that started with "developing a chatbot app targeted at teenagers" (according to https://en.wikipedia.org/wiki/Hugging_Face), which helps explain the weird name (which still makes me think more of one of the development stages of the xenomorphs from Alien than of the emoji, but that's probably just me).
For context:
https://news.ycombinator.com/item?id=48997548
Surely you can just google this.
I think the fair nuance here is, an administration which used Executive Orders to force guardrails, meaning US companies must retain them, even if they might want to drop them now.
And on top of that, with low/no guardrails, people call you a child pornographer(grok), so the public is also against it. Yet mysteriously few complain about Chinese open models being child pornographers.
So even if your goal isn't ethical, but just fiscal, it's reasonable to say there are two standards. And to complaint in some way.
I don't think banning is going to work, that's just silly. And over the next few years, everyone and their dog will have local GPU compute to train locally. People have home labs, the bar isn't that high, and eventually large text datasets will escape from Anthropic and other companies, allowing for comparable training.
It's a genie that's not going back in the bottle, the bottle is smashed.
The only reasonable outcome would be section 230 style carveouts so that there is zero liability for anything a model does.
Because having guardrails on corporate models barely months ahead of open ones, which will never be restricted, is entirely pointless.
The "child pornography" thing was about image generation, which people for better or worse have very different moral standards for than for text generation. There is very little pushback against grok being willing to write explicit descriptions of sex, or giving security advise.
Though I think your overall point still stands, and at least Grok's twitter bot has received a lot of criticism for pure text too (Mecha hitler comes to mind)
All my adult life, China has given me cheap material goods. Whatever they do in their country is their own problem. They’ve never interfered in my politics or started any wars
> They’ve never interfered in my politics or started any wars
But they have. I dont know what specific country you are referring to but China has interfered with US elections as well as Canada, Taiwan, and Australia in addition to many many others.
They’ve annexed Tibet (1950), fought India (1962) and Vietnam (1979) among others and more recently in 2020 a deadly skirmish with India. They’ve generally shifted their focus to cyber military actions but you’d have to be pretty naive they won’t start to exercise military control over Taiwan when/if they get a chance.
Now compare that to what the USA did in the same time frame. Pay special attention to people killed (either directly, or indirectly) outside of their own borders. Who should you be worried about?
Two evils just means there are two evils. We don't have to value one more than the other.
Wrongs don't negate each other.
If you have the choice between AI models created by one evil or the other, it does feel relevant which you consider to be the lesser evil
> We don't have to value one more than the other.
Yes we do. We should evaluate threats and choose our allies carefully.
Correct, but people are not wrong in being more concerned about the evil biting than the one keeping it chill.
Unfortunately, I have to agree with you. As the saying goes: "better the devil you know". China can at least so far be relied upon to not shoot itself in the foot (and take the whole word with it). The same cannot be said of other great powers like the US (under Trump, for more reasons than I'm willing to list here) and Russia (with Putin's completely harebrained invasion of Ukraine). China can prove me wrong any time by invading Taiwan, but I still hope they don't. Of course, if we try to keep them away from advanced chip technology, that would be further motivation to invade (although TSMC is unlikely to survive an invasion - even if it did survive the actual conflict, it would probably be destroyed or evacuated to avoid it falling into Chinese hands? But I don't want to see the hardware prices after such an event).
Given it's an internet forum, and people here can be anyone around the world, I understand your intent but people over index on the threat that ofc feels closer, rather than the threat that's significantly farther away.
But in general if you ask folks in south east Asia you will find they are likely to be more concerned with China than US except this maddening Oil Crisis.
Although I think people now are very afraid/wary of both evils.
The US govt should think why people in all parts of the world including US itself feel just if not more threatened by them, than China and other evils we have floating around.
I would like to hope my American friends didn't vote for this madness or maybe they did I have read DHH's tweets.
Sadly, we live in a decidedly imperfect world. I don't think there is a single completely good person on the planet, let alone a country.
It is perfectly fine to say that a person who killed one person is bad but not as bad as the one who killed thousands.
I eagerly await for the USA citizens to downvote this in about 5 hours.
> Who should you be worried about?
For Americans the answer is obvious since they are biased in this.
For everybody else, your question is valid.
Dont pretend americans dont need to be afraid of american government. look at the state of the place. A lot of them arent exactly thrilled with their current environment, job market, education, prisonsystem, gun laws, food industry, healthcare etc. etc.
the question is valid for anyone who cares for more than their own ass
I'm an American, but I do not think the American government being strong helps me or my fellow Americans. The less America can project its cultural and military power on the rest of the world, the better.
There are different kinds of strengths. There was a past American government that was strong enough to stamp out inequality and fight for civil rights. To overturn oppression and give everyone an equal opportunity to succeed. That's a strong American government I could get behind.
The American government has caused the death of a lot more Americans than China has.
Cultural revolution?
How many Americans were killed during the Cultural revolution?
Considering who has been their president for 2 terms, and most likely a third one, I really wonder.
All my life I heard American boasting their constitutions but now it's as worthy of my toilet papers.
I see this narration repeated here, but to me it seems Americans are actually one of the most self-cynical nations. Here in Poland we still actually believe in the superiority of the western values of freedom, democracy and individualism. "People roughly like us" vs "insect hivemind" is the simplest choice ever.
Please reconsider this comment. Calling one side superior and the other side insects really doesn't help with the discussion, and dehumanizing a whole swath of people really has dark consequences.
Yeah, sorry for being edgy. But remember the understanding of universal human dignity your comment is stemming from is not shared globally.
At least not shared by you, apparently.
That's why I'm inferior!
Speak for yourself, in Italy we are also closer to US "culture", but I do see the US as a bigger threat than China, simply because it's closer and it directly affects our politics.
For a concrete example, consider e.g. Operation Condor which displaced my own family
> Operation Condor (Spanish: Operación Cóndor; Portuguese: Operação Condor) was a campaign of political repression by the right-wing dictatorships of the Southern Cone of South America, involving intelligence operations, coups, and assassinations of left-wing sympathizers in South America. Operation Condor formally existed from 1975 to 1983. Condor was formally created in November 1975, when Chilean dictator Augusto Pinochet's spy chief, Manuel Contreras, invited 50 intelligence officers from Argentina, Brazil, Bolivia, Chile, Paraguay, and Uruguay to the Army War Academy in Santiago, Chile. The operation was backed by the United States, which financed the covert operations. France is alleged to have collaborated but has denied involvement. The operation ended with the fall of the Argentine junta in 1983.
https://en.wikipedia.org/wiki/Operation_Condor
Both of them.
“China is evil.”
—- “But look at the United States! What about that?”
Is this not just whataboutism? It never adds much to the discussion.
It would be whataboutism if it was a random Joe saying "China is evil" not literally a company that 1) is in the US 2) has a strong influence in the government and legislations affecting them and their competition.
In this case it's just pointing out the propaganda and the contradiction.
Whataboutism isn’t a useful game. The claim was that they don’t interfere with politics of external countries. They do.
The claim is they don’t engage in warfare when they do, just exclusively in the digital domain for now. Believing it’ll stay there is naive.
Playing the moral superiority game is uninteresting in either direction. Games of power inherently are devoid of morality.
Ok, let's follow your rules then. If we remove morality, we are left with what? Pure numbers? If you want to be technically correct China foreign interventions are > 0, indeed. But the scale of US actions is, I would dare to say, at least one magnitude bigger. An empire should be judged for how they treat their own citizens, their allies and their enemies. And if we do that, maybe China and the US are not so different.
I’m confused. You start off like you’re going to disagree and then end with the same conclusion I was trying to suggest. Keep in mind I was replying to subvenir who said “eh China not bad - they’ve file nothing wrong to me”
I read your previous message as implying "US is good", rather than "China is bad too". I'm glad I misunderstood you and that we agree then.
I'd like you to familiarise yourself with a legal and societal principle we use in civil law countries:
> de minimis non curat praetor
Which basically means judges are not interested in minor disputes. When you add the USA to the mix, China's actions pale in comparison to the point of being negligible, no point in discussing them given the scale of USA's warmongering, overseas and local crimes. "They never interfered in [...]" is an exaggeration which shows the lack of importance of China's crimes when compared to the USA's, that's all.
The "whataboutism" here isn't a deflection strategy, it shows the hypocrisy and correctly minimises the scale of China's actions in comparison to the hypocrites'.
Whataboutism is a very useful game to reveal whether people are being honest or not.
If someone is vocally complaining about country A doing some oppressive activity, while ignoring country B doing 100x the same activity, the comparison does not forgive or excuse country A, or make them right or admirable.
But it does allow us to conclude that the person complaining is biased. Either they don't care about the oppressive activity and want to attack country A for some other reason. Or they have a particular fondness for country B and will never accept that it does wrong. Or some variation on these.
Except it’s op posting whataboutism not me - their claim is that China has done nothing wrong to them with the implication that the USA has. I was highlighting that China indeed likely has done the things they’re complaining about.
>The claim is they don’t engage in warfare when they do, just exclusively in the digital domain for now.
Come on, are you seriously equating hacking to literally bombing and killing people? That's just arguing in very bad faith.
Calling hacking "warfare" is just intellectually dishonest, if not downright disgusting toward the actual victims of American wars.
Whataboutism is juvenile.
The Indian wars were border skirmishes at most. Literally 2,000 soldiers dead on both sides combined by official count. Very few civilian casualties.
The 2020 skirmish was way smaller with single digit casualties. Again, almost no civilian casualties because it's a largely empty region in the first place
You really can't compare this to the kind of wars America has started. Casualties - both civilian and military - stood in the hundreds of thousands
And then we haven't included Bibi-enabling, allowing a genocide to take place and then suggesting it is a real estate opportunity.
If you're making a case of this vs what the US has interfered with, it simply does not stack up. Of course it's naive to think they won't start interfering like the US once they gain the leverage, but to a neutral party it likely doesn't matter which of these two ends up as the superpower on the global stage.
in contrast this is USA's main job is to Interfere in any country it can in favor of its own goals. not related to what the people of that state really want.
north Americans must stay silent about interfering in other states internal issues.
> China has interfered with US elections
Wow, that newest of Trump's distraction talking points made it surprisingly quickly to the uncontended (?) and commonly accepted facts in HN conversations.
You’re misinformed. The Biden administration in 2020 declassified that Beijing has prepared to latch influence campaigns in 2020 but backed off. December 2023 a report was released that they had changed track and engaged in influence operations during the 2022 midterms. Biden’s administration also warned about influence operations in the 2024 election cycle. The Biden DOJ unsealed multiple indictments against Chinese intelligence officers, state-backed hackers (such as APT41), and illegal Ministry of State Security agents acting inside the U.S.
You’re very misinformed if you think Trump’s latest blathering are completely wrong. I generally don’t pay attention to him so I don’t know exactly how he lied or exaggerated but I’m sure he did. It doesn’t negate the real and active influence operations China is engaged in
Lots of assumptions about what I think. I pointed out that Trump put the topic on the agenda a few days ago and now it comes up in HN comments. No more, no less. Whether it has merit is a different question. Point is that the reach of him setting topics goes far which is hard to deny.
You’re very misinformed if you think democrats and republicans arent playing the “election influence fiddle” in exactly the same way; legitimising the elections they won, casting doubt on those they lost, all while achieving the main goal of making sure the population doesnt forget who their so called enemies are.
At this point with how volatile US is I'd rather have China control Taiwan than US.
And this comment isn't about putting any good light on China. Lately US acts like they want to compensate for Russia fucking up less things around the world
They have colonies and generally aspire to become basically the old british empire as a longterm strategy? At least thats what they push with that "century of humiliation" comeback strategy internally..
No new wars in 47 years is a pretty good run vs other major powers.
Lets ask xinjang?
Let's ask black people, minorities and foreigns with the wrong skin color in the US?
What about women too ? Especially in red states.
Repression of their own citizens is another subject. It is of course highly problematic and should be condemned as much as possible, but in the same way that if the US decided to repress mormons violently, it remains internal affairs and not an imperialistic threat to the countries around them.
> it remains internal affairs and not an imperialistic threat
That's pre 1945 style thinking, which is coming back with a vengeance.
They can also kill a member of Falun Gong or a Uyghur to order if you need an organ transplant.
Same as killing Palestinian and Yemeni kids if some westerner need a transplant, I guess ?
You think the USA does that?
> Whatever they do in their country is their own problem.
Idk to me it feels like human compassion and ethics dictates that you at least have to care a bit even about the things in other countries. Otherwise all sorts of horrible domestic policies would be justifiable.
Star Fleet first directive has been proven right many times.
For you personally no, but there's plenty of scandals of politicians being influenced by the Chinese, case in point in my own country: https://www.ft.com/content/601df41f-8393-46ad-9f74-fe64f8ea1...
Don't be naive or I might even think you're working for the Chinese :-)
That's a very selfish way to think of it. Whatever they do in their country does not stay in their country because China is becoming a superpower so you should expect it (as it's normal) to interfere and affect policy abroad.
For example Lithuania has been punished for letting Taiwan opening an embassy (in Lithuania). So China cares what other countries do in their country. Without EU support Lithuania could have faced very harsh consequences. That's just an example. Another example is the "secret police" stations undeclared overseas police stations operated by China in various countries (i.e. U.S for example). China gave you cheap material goods because that was its business but you have to keep in mind it's an authoritarian, communist regime. It carries an extra risk on top of the potential economic coercion if you give it too power. At least you know that U.S is only after the money and does not want to turn your country in a communist "utopia".
That being said this does not mean we should ban Chinese AI models because China didn't cross any red lines(yet) compared with Russia for example.
Do you remember covid and where it come from?
Tulsi said US operates lot of biolabs outside US so as to avoid US regulations.
A disease can come from anywhere
Unless you're trying to imply that China engineered the whole thing - extraordinary claim that will require extraordinary evidence
They did downplay the effects of covid and spread misinformation about it as well as silence the researches and journalists who were first trying to ring the alarm bells
But they're happily selling drone parts to Russians dropping bombs on civilians (or "whatever").
In fact, they are also selling drones to Ukraine.
Yes and the USA trained and gave weapons to the Taliban and many other terror organizations. "China sold none weapons to another country" really pales in comparison.
This is wrong. The US did not give weapons to the Taliban (voluntarily). If you mean Operation Cyclone their receiver was not the Taliban but the mujahadeen, which, yes, consisted of Taliban, but also of enemies of the taliban (northern alliance)
You are not only supplying drones, bombs and rockets to kill Russian civilians, you also direct these weapons and terrorist operations in Russia by your satellites, claiming you are not waging war against Russia.
Oh i bet they have not directly interfered with your politics, but they do interfer a lots in various things. More than any other country on this planet? I don't know. China is big. Really really big.
I mean them selling the cheap material goods is politics, and they use that a lot to threaten other countries. They constantly bully neighbors and assert claims on foreign terriroty. They might not be the comic book villians people are making them out to be but your take is just ignorant
Such horrible villains, threatening the world with cheap stuff!
Nice straw man. The point is not "cheap stuff" but using economic leverage and territorial pressure on others because of the "cheap stuff." And yes, all China does is produce cheap stuff for the world, nothing else at all lol.
They have their own genocide in western china
According to the U.S. State Department, they literally do not:
https://law.stanford.edu/press/state-department-lawyers-conc...
That whole thing was a very successful propaganda campaign, making people believing China has been mass murdering Uyghurs.
> They’ve never interfered in my politics or started any wars
They actively back russia and deliver them weapons and support them with their attempted Genocide in Ukraine. They also actively threaten Taiwan.
Not to mention them claiming almost the entire South China Sea even though this being against international law and threatening smaller nations into submitting to them
We don’t have to guess. Here is a recent survey:
People in Many Countries Now View China More Positively Than the U.S.
https://www.pewresearch.org/global/2026/07/15/people-in-many...
Are you saying the USA is somehow as repressive as china?
>Also everything hes saying about China (and other actors) many outside the USA would say about the USA.
Yes, I for sure trust the open models from China, more than anything coming out of the USA at the moment.
Its not just the USA's support for genocidal regimes, nor its heinously illegal wars and atrocious 21st century human rights record. Its also the Snowden revelations and the treatment of Julian Assange.
Slowly, surely, the world is building a firewall against the USA's imperialist actions - not just its motives. That AI is a key building block of that machination is of course, highly exciting.
There are many in Europe who feel the same. The tide is very definitely turning.
Why is it relevant what those outside would say about the US? Or what those outside of any country would say about any country. The job of a country is to do what is best for it. It's better for the US to not help China develop AI that surpasses its own capabilities. It's better for China to try anyway. There are many countries inventing nothing with loud opinions. Those opinions are completely irrelevant to China or the US.
> The job of a country is to do what is best for it.
Countries aren't defined like corporations in the US. Why would countries have funds to help places like Haiti otherwise?
Lots of different reasons for countries to do all sorts of things? Why would France have armed the US during their independence movement? Why is the rest of the world supporting Ukraine during this war started by Russia?
You mean to say that without France’s help the “revolutionary war” would just be an embarrassing string of failures and a footnote in Britain’s history?
Oh no, you heretic, The People freed themselves and it was ordained by God, you see. It is through the righteous might of The Greatest Experiment in The History of the World that they showed The World what Freedom really meant.
Having the British lose the US was a good thing for France, simillarly preventing Russia from just taking over another country is for the best of other countries.
While there is no "job" of a country, it is natural that each one will work for its own best interest and any moves it makes in the global world is due to their own vested interests in some way including helping Haiti, France helpong Us and the world supporting Ukraine.
Why would a country act against its interest or just randomly? Alliances and corporation are a part of politics
>Why would a country act against its interest or just randomly?
Because country as an entity is a mental construction for which "interest" is a categorical error. It’s certainly a useful concept, but pretending it has interests like some human individual can have have interests. Sure it can serve as rhetorical facility to sell some arguments.
Countries don’t have interests. Some people willing to take control of other people encompassed in that "country" groups have interests, and they don’t necessarily align with best interests of everyone or even majority in each of these groups.
I mostly agree, but there is some limit to that line of argument: a corporation is also a human, social concept, and we do accept that they act in their own interests (meaning they do have interests, sort of like humans do)
> Why would countries have funds to help places like Haiti otherwise?
because that's what they think is best. what point are you trying to make? that countries don't pursue 'profit' at the expense of all other concerns?
I think that that "best" is not always measured in money. And it should not be. Countries are made of people, and people wellbeing should be the main priority of country. And when stepping a bit back, all countries are made of people, so why differentiate? Ideally make life good for all people.
Why should the national security concerns of the US lie with Anthropic?
Why does it feel like you think you're talking to only people in the US?
There's plenty of people on this forum that aren't American. Including some former allies whose sovereignty has been aggressively threatened. And some of those people are Anthropic customers.
Even more so, many of us are in countries that would be well within the blast radius of fallout should the US try to "ban" open weight models or make moves to limit "US" models (often developed on research or work by non-Americans too, but that's another topic) only to those blessed by the US gov't.
That's why it matters?
Not just customers, Anthropic has offices in multiple countries around the world. And hires people from all around the world
Yeah some real main character energy from Dario as usual.
I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.
This Onion meme is perfect; you could easily replace Sam with Dario: https://theonion.com/sam-altman-if-i-dont-end-the-world-some...
You don’t need to replace them. Dario is the “someone” in question.
What exactly happens to a "losing" country when it has been AI "dominated"?
You already have examples. Venezuela, Iran, Ukraine, Libya… maybe Japan? The list is sort of long. Juxtapose or something.
And here I was thinking that all of those had nothing to do with AI.
Yes and no. Yes because it's more about oil than AI, no because AI is the oil industry and their supporting vested financial interests' (not so secret) plan to keep demand for fossil fuel high if/when the climate change deniers lose.
"We would love to use green energy, but all the batteries and solar panels come from China and China is evil, and we need all the energy we can get to run the data centres we need to spy on our citizens so they don't revolt once the environment is literally on fire, we can't feed them, provide enough energy to cool them, and refuse to build enough housing to house them."
Oh. I thought a whole lot of the point and discussion of this post was AI being potentially weaponised and becoming means of economic and military control and coercion. Or did I really miss something?
Why is the US so scared of this fate? To the rest of the world, they're tecnofeudalist monopolists.
or you can image, what exactly happens to a "losing" country when it has been a small clique "dominated"?
wouldn't stop AI companies from continuing to develop using their current infrastructure.
the West could retaliate by halting shipments of photoresist and other materials to China.
meanwhile, Intel second-sources Nvidia and starts pumping out GPUs.
the economic fallout would be devastating as trade wars and export bans on both sides make Trump's "Liberation Day" tariffs look like NAFTA.
The rest of the world would likely side with China. The export ban on Fable wasn’t even extended to five eyes countries.
US is going to find itself isolated and irrelevant. And not a moment too soon.
I do wonder what the world will look like under a chinese hegemony though...
China is very inward-looking, though. It's not really interested in world domination in the same sense that Europe was or even the USA was/is.
China has been aggressively developing infrastructure and financial services all over Africa and the rest of Asia.
https://www.gao.gov/products/gao-24-106866
Yeah I saw this in Cambodia when I lived there. Sihanoukville had become a Chinese enclave almost completely.
Different to the way that French colonialism worked, though. Less direct government, more influence of existing power structures and respect for the local government.
There is definitely an argument that this is plain business investment - China has a lot of foreign currency to invest because of its trade surplus, and there isn't the opportunity within China to invest it all, so it is engaging with trade partners to invest in their economies so that they can increase future trade with China.
You can also make the argument that this is not benign and China is trying to create control over foreign governments with this investment.
I'm kinda "both can be true, but either are better than how we did it"
Go ask anyone in Southeast Asia which they'd rather have colonize them: USA or China.
They'll usually refer to China in terms like the "thousand year enemy".
> Reversing last year’s trend, a slim majority of ASEAN respondents selected China (52.0%) over the US (48.0%) if the region were forced to align itself with one of the two strategic rivals.
https://www.iseas.edu.sg/wp-content/uploads/2026/03/The-Stat...
I’ve only been to Vietnam but the answer there is it’s… complicated.
They’re culturally part of the sinosphere but of course constantly living in the shadow of your much, much bigger neighbour to the north does breed some ill feeling.
The USA on the other hand, well, they’re not particularly popular either for obvious reasons
Ask Latin America which they'd rather have colonize them.
We don’t want anyone to colonize us just like you don’t want your country to be colonized. China offers business opportunities while the USA and Europe seem to always look down on everyone else and impose their own values on others. China does not give a shit about how we run our countries as long as they can trust that deals will be honored. We don’t want to be taught how great your democracy is or whatever, we want our countries to become better by our own efforts, with fair deals when dealing with other nations.
This 100%. We’re neither the back yard of the USA, nor the front yard as Biden said, probably thinking that was a good thing.
Get the F off our lawn and stop dumping your rubbish on it!
Yes, of course, sorry for the expression. I just mirrored the expression used in the parent comment. What I mean is that the US have caused a lot of pain in Latin America.
Grass is greener on the other side. Hegemony on the other side of hemisphere is greener or benevolent.
That really depends on who you ask. If it's that bad then why did all of those countries sign up for the RCEP?
They've mostly been concerned with resource extraction rather than colonial exploitation like the West. And in fact, they've ditched most of their infra and tech projects in Africa, because of inherent instability in dealing with tinpot dictatorships.
On the other hand, every one in Asia is wary of too much Chinese presence and influence.
resource extraction
stealth colonialism
The alternative is western colonialism Africa experienced before, and bombing campaigns.
Western colonialism is better than eastern colonialism tho. Look at the Russian orgs in Africa.
Single counterexample for single counterexample: Belgium's Leopold - not sure if Western colonialism is so much better (and notice I didn't even bring up "western India")
Better for whom? Certainly not the colonised...
Yeah for the ones who get colonized, western ones were better than eastern ones. Japanese ones would've made Britishers look like kids when they were occupying eastern India.
You just put Russia and Japan as "Eastern"; they have nothing to do with each (and I would consider both as more Western than Eastern regarding imperial/colonial matters) other nor with China. China is a massive country that has always been massive and rich. Their style is more like the tribute systems they had before, as it is being observed in the BRI and other projects.
I understand China but in no way were japan and Russia close to western. IJA, MAO and soviet union makes westerners look like chuds. Russians still have the wagner group shit which is nowhere near what western countries have currently.
You know the British invented the concentration camp in Africa, right?
We're told a relatively benign version of our history. You kinda have to travel to the countries involved to get the real version.
Even with all the downsides of being colonized by Britain, it was still far better than being colonized by Belgium. Heck, being colonized by pretty much anybody but Belgium was better than being colonized by Belgium. Read up on the Belgian Congo sometime: it was a parade of horrors.
GP was making a relative comparison; pointing out that both were bad in absolute terms does not negate the point. -2 is still greater than -17.
Who cares who invented it first. Who did it best? Japans concentration camp was unholy abomination and it doesn’t get enough attention today
How can you see the road not taken?
You check existing conditions and extrapolate.
I keep hearing this and I fail to see any reason to believe that it will be the case. Any empire, in the history of our species, has always had an initial "inward-looking" period prior to becoming a full-fledged imperialist oppressor. Edo-period Japan into Imperial Japan, for example. Rather, what I think any budding empire needs a period of growth to become a large enough fish and also to delude its leadership and population into the convenient mindset that their flavour of imperialism is good and justified ("Hakkou ichiu", "The white man's burden", etc.).
To me, the PRC is at the very end of that process and I recommend anyone doubting this to go and read conversations and listen to the words of the populace that is turning increasingly nationalistic and you will hear the same old tales of revanchism and exceptionalism that we are used to hearing (during my recent visit, I watched the morning news every day for about a week and without fail a military inspection, new ship, new plane, etc. was presented each day). In addition, I think those outside of Asia are very much shielded from the early signs, but go and read about PRC influence and tensions in South Korea, Japan, RoC, Philippines, Vietnam, Laos, Myanmar, India, Pakistan, and Tajikistan and you will see something rather different than "inward-looking". To me, here the PRC is simply testing the waters for the extent of the influence of other powers and how far it can go. Likewise, we are seeing overseas naval bases being constructed which sure is an indication for a desire to project power outwards.
I want to believe that this time it will be different. I really do. Apologists around me say "It will only be Taiwan and the South China Sea, then then it will stop." and I would love to believe them. But can anyone truly internalise the narrative that international utopia will be spearheaded by a deeply authoritarian state that controls information like no other (and gladly exports that technology), disappears its own population at will, spins an increasingly strong nationalistic narrative, etc.? No, sorry, I think the "inward-looking" narrative is simply a convenient way for us to close our eyes and find comfort in ignorance, rather than in facts.
So let’s just guess what China will do once it becomes dominant and act like it already did whatever we conjured up they will do. That’s how you get preventive wars that destroy civilizations without any actual basis on reality.
No, I do not think that is a fair portrayal of my position. Just like I will not say that your position is simply appeasement and hoping for the best. One must always be charitable in a discussion with strangers whose positions you do not fully understand.
Rather, I think we should look hard into ourselves and what is good and bad about the current world order. For example, the people of the RoC have the right to determine which direction they want to go. Regardless of the chauvinistic rhetoric coming out of Beijing. We should all stand up for this, because it is a universal right that we want everyone to enjoy. Similarly, we should push against the Eleven-dash line and support the 2013 ruling. The list goes on and I am sure these issues can be resolved without an outright war if we are careful, yet firm, in our beliefs and also diplomatically preemptive and thoughtful.
Being concerned about PRC imperialism should not be mistaken for the position that their people should "know their place" and be suppressed back to the stone age. They have the right to enjoy the fruits of their labour and pursue happiness, just like everyone else. We simply must be there to remind them (just like we must remind ourselves) that the course of humanity is a collective project if we are to stay clear of the darker sides of our nature as we venture together into the future.
Do you believe the UK has the right to rule over the Maldives? And the USA has the right to have a military presence in Guam (and nearly all of the Pacific Ocean for that matter)?
The Chinese claims on the South Pacific Islands seem really similar to me.
Taiwan seems like a wholy different matter. It was united with China for hundreds of years until the Japanese colonialists took over. It united again with China after WWII but split up after a few years because of the Chinese civil war (notice it was an internal war). I think it's just fair that China wants to re-unite with Taiwan, though I definitely don't support a military takeover. Hopefully a solution similar to what was done in Hong Kong can be found. The Tibet region had a similar history and it's definitely unfortunate that China had to use military force to bring it under its own control (arguably completely unnecessarily - China would be just as strong today without it), but it's kind of understandable in the context of the time (China was trying to recover from centuries of being preyed on by other nations).
I am saying this because I can't agree that China is acting imperialistic - it's basically claiming sovereignty over its own historical lands - which were taken away from them by force by foreign colonial powers. But I admit that, if you go back far enough, nearly all land was once taken over by aggressors - the USA being just a more recent example of that.
Anyway, thanks for not being an absolutist and trying to understand the "other" side (I must acknowledge I have no relation to China whatsoever, in fact I am from South America and live in Europe).
You may well be right, and it's really only that China has been more inward-looking recently, and given the chance it will spread its true colonial wings.
Not sure if "colonial" is the right word though, as we should be careful to think that oppression always takes on the form we have seen in the past (US imperialist oppression for example did not take on the form of the colonialism that preceded it) . Also, be careful with "true" there. I do not think this is some sort of subterfuge, but rather an inherent weakness in us as a species. Vest power in anyone, and before long their morals will give way and an oppressor will be born.
The way I look at it, until Deng the PRC's economic policies and internal instability kept it from growing at the pace of many of its neighbours. Then we had an era of intense growth (which is still to some degree ongoing). However, as a reaction to this era Xi and others needed a narrative to counter the increased corruption and a new unifying myth to replace the cult of growth as the economy would stagnate at some point and could then call into question the authority of CCP to rule. Their choice of nationalism is what scares me and I know PRC citizens (even CCP members) that share this perspective and would rather have seen the Shanghai clique to have remained. It is possible that in this alternative reality we would still end up with "Imperialism with Chinese characteristics" ("中国特色帝国主义"?), but I chose to believe that at the very least the chances of this would have been smaller.
Except of course for claiming all of the South China seas. See the Nine-dash line
"Not interested in world domination" LOL. That's what Putin said 10 years ago.
It has been quite good at indebting countries to them, in Africa but also e.g. Montenegro.
Africans are more indebted to Western multilateral organizations and private Western bondholders than to the Chinese. The idea that Africans are being finessed by the Chinese is racist and stems from a "we know what's good for you" imperialistic lens.
Interesting. I'm just relaying what I read in quality newspapers over here. And it's not just Africa. Anyway, you may be right. Do you have numbers?
better dim sum, hopefully. the local chinese food is a crime against gastronomy.
You’re already seeing it. Strategic bilateralism.
It's called Tang and Ming dynasty. We already know.
Keep an eye on Central Asia and Pakistan then? They are heavily infuenced by China.
We are moving from unipolar hegemony to multipolar. There will be upwards of a dozen poles in time
Doubt. China is clearly on a level of its own compared to virtually every other country out there other than the USA.
I find it unlikely to be a duoply of hegemony for long, some countries to watch are Germany, Japan, India, Nigeria, and Brazil. You could broaden the geography to continents were I expect major players to emerge on each.
We are only a few decades since the "end of history" and much has changed. What do things look like beyond 2050?
> Brazil
Brazil is basically the world's soy farm. It's at least half a century behind the times. I still have no idea how it managed to insert itself into the BRICS economic block. The notion that it's on the same level as China, India or Russia is just comical.
> Brazil is basically the world's soy farm
To reduce a culture and country like this comes off as bigotry
In example, Brazil has the #3 airplane producer in the world
Or consider why the US is doing all it can to prevent their payment system from becoming widely used?
I was intentional about putting the perspective in the future over today as the poles of the world are evolving
It's my culture and my country.
All of the small miracles you listed happened in spite of the culture, not because of it. It's also not a coincidence that both are deeply linked to the most successful brazilian enterprise: the brazilian government.
I'm trying to avoid getting too deep in these Brazil tangents so I'm gonna leave it at that. Anyone who cares enough to know what I think about the subject can just look up my comment history.
OP is one of many Brazilians that hates Brazil, which ironically is the reason why the country historically can't grow to its full potential
A lot of AI researchers were just in Brazil due to ICLR. A number of them got robbed in broad daylight in this supposedly nice part of Rio. I think the assesment of Brazil is accurate.
Not accurate, since it doesn't explain the normalized violence you observed. I was trying to be nice.
This thread is more like it:
https://news.ycombinator.com/item?id=45731174
s/bigotry/fatalism/ ? (in the same sense of colloquial usage with Russians)
No worse than under US hegemony
Much much worse: see Tibet or Hong Kong
Vietnam, Iran, Iraq, Afghanistan...
care to explain why all governors of Hong Kong under the british rule were white males?
How is Hong Kong worse than say, Puerto Rico?
certainly better than this one. Look at Africa under western neocolonialism vs China's infrastructure loans
As someone from India and someone who knows people from Tibet, Chinese hegemony will be much much worse.
There is no HN equivalent in China where users advocate that US hegemony will be better.
On the other hand, India is enemies with China and Tibet is a highly oppressed region. I wonder what the average Chinese in china thinks.
The average German in Nazi Germany probably didnt think the Nazis were too bad either (of course the average citizen wasn't Jewish, gay, communist or other undesirables). That the average Chinese think things are “just fine” doesn’t mean they are.
Unlike Nazis, I don't see that Chinese communists have expansionist goals to seize territory. Nazis wanted all of Europe. Do Chinese communists want all of say southeast Asia?
South China sea, Tibet, Uyghuria, Taiwan. And definitely dominance over SE Asia even though no direct conquest.
Yeah capitalist countries are the only imperialists. Every other country is a saint compared to the west.
To be clear I don't think the government of China is as bad as the Nazis were. I'm just saying "the average Chinese in China thinks it's fine" doesn't mean it's fine.
And that's going just swimmingly [0]
[0] https://www.indiatvnews.com/news/world/chinese-manager-beats... (possibly NSFW, I can't see the video in my browser but be warned)
There is no good imperialist power, there is no capitalism with a human face, there will never be. The UK murdered millions in India, Ireland and all around the world in their time. The US did the same. The only reason China isn't openly doing the same right now is because they are still the underdog and they still need cordiality to get through the door, just like the UK and the US did in their time. The solution is to rid the world of imperialism and capitalism as a whole. As long as there is a profit motive running through everything, international relations will be in the form of wars (military or economic). Only under socialism can the people of the world truly pool their resources together to build up instead of destroying each other.
You would have to eradicate the desire to command other people. The wish for power which is different from the wish to be praised, as forcing somebody to do something is different from convincing somebody to do something.
There are no good imperialists but it's a spectrum where western capitalist colonies fare much better than eastern communist colonies because capitalism and democracy is the engine for economic growth and western colonies fared much better than eastern ones. What's a better system than capitalism for imperialism? Socialism? Weren't they also pretty brutal imperialists? Japanese imperialists alone make western ones look like amateurs.
China would just start buying oil again, and stop using it from their strategic reserve. Hello $200 barrels of oil.
> the West could retaliate by halting shipments of
China quickly retaliated last time by stopping shipments of rare earths and magnets. The West has no answer for this, really up the river without a paddle for such critical supply chain elements.
I was led to believe that the US does have internal sources of these, but they are largely undeveloped. For years the processing could not economically compete with China so shutdown.
Which is to say, given internal subsidies, the US could eventually produce some on its own.
Unless I misunderstood the situation.
I believe we both share this understanding, the problems are
1. China controls ~90% today
2. The US will find it difficult to build out because of how dirty and environmentally damaging mining and processing are.
I did see some research last week about a better way to process rare earths, but it is still research and will need to be industrialized.
Regardless, it will take many years (decade+?) to become self sufficient and China is already willing to and increasingly restricting them
China controls way more things, from medical needles to pharmaceutical ingredient.
Lots of them can be made in the west or west friendly countries, but that takes time, money, infrastructure and good execution. Yes, identical to what is covered in China's belt and road initiatives. See the gap now?
11 carriers are not just for show.
Could have fooled me, judging by the Iran and Yemen performance.
Iran showed us that they apparently are.
Dario is more of a threat to the US, in terms of advancements in AI, than China. In Dario's mind anything that can't be controlled competitively is a threat to Anthropic, so he positions his FUD strawman so that Dario doesn't have to worry about the competition. And then he can artificially inflate token costs so his IPO can happen. Dario doesn't actually care about ethics, alignment or availability of LLMs - he just likes to use those words to sound like he does. Yet we've all seen how Anthropic actually acts vs what they say.
The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.
But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).
The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.
People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.
> compared to his own closed and overpriced ones
I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.
To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.
It's possible for something to be overpriced to the customer and simultaneously underpriced for the business to be profitable. Theoretically speaking, they could just be selling such an inefficient product.
It's also not a law of nature that there be a valid/efficient form of a product that makes a business profitable. I'm in no place to judge whether that's the case here, but not all products are viable.
I guess that tracks with the brute force attempt at workable intelligence that is the Attention/stochastic token based system
Overpriced for the value provided, especially with the pay-as-you-go rates for enterprises; but also in stock prices.
Pricing a product is generally about ensuring a profit on investment, but also a good RoI for your customers (or they will not pay).
With these long-term profit exercises, it's always a lot of hand-waving though.
No one is forced into buying the product, so it feels like an emotional rather than a rational rant from OP.
> I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow.
The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.
Add to that the >$700B OpenAI already committed to spend in infrastructure through 2030!
Re. 'overpriced', I'm assuming OP talking about their theoretical stock price.
Can we just downsize colonialism?
Did you mean to respond to a different thread?
Inference is only priced in for a low usage window 6h a day per customer?
I'd assume the gp was just inflating their rhetoric but cost of model is an interesting topic.
I'd guess Anthropic and OpenAI's models are expensive relative to the cost of running the models but that the revenue still doesn't pay for building the next and next models. The challenge is how these next generation models are going to pay for themselves. Will everyone on earth find it useful to $200/month to talk to a thing more intelligent than themselves? The alternative is naturally that these are going to replace workers and employers will be the one paying.
The whole thing about replacing workers is such a perfect example of shooting yourself in the foot. The whole US economy is so strong because the population consumes so much. Get rid of the workers, you’re now damaging the consumer base. Repeat that a few times, across industries, and you now have a zombie economy. The idea that we can have an economy of virtual agents is a child idea that doesn’t make any sense in a serious situation
In short: "We don't propose a ban to open-weight models. we want to stop these models being developed altogether. If there are no models, there'll be nothing to ban".
Oh also: "They ste^H^H^H distill what we have sto^H^H^H used fairly from the world. This is unfair".
Lastly: "What if they use their models in their military and local police services like we do? Communism!"
As always: https://pbs.twimg.com/media/B_AiI9_XIAA67_t.jpg?name=orig
A bit of good old “it’s really good idea, I live, great effort … BUT”
Mixed with “if kids can’t get semi dangerous drugs from the back of my van then they will be forced to buy from even shadier, more dangerous dark web van”
A bit off-topic: I occasionally see that triple ^H on HN posts: what does that actually indicate?
A caret with a character subtracts 0x40 from its hex value allowing you to insert non-printable characters. Uppercase H is 0x48. That gives us hex 0x08 which is a backspace.
https://commons.wikimedia.org/wiki/File:USASCII_code_chart.s...
Ctrl-H is the rubout/backspace command on old terminals. So if I say it's nuts^H^H^H^Hunwise to ignore UNIX history, I've erased the first word and replaced it with the second.
In the old, hardware terminals and current terminal emulators, ^H (or CTRL+h) is "Backspace signal". It has the effect of pressing backspace.
So, ^H^H^H means "delete three characters, excluding '^H's". Like the person typing the comment changes their mind and deletes the characters (or the word) before writing else.
It's an stylized way of euphemism. i.e.: Actually I want to say this, but I substitute it for that.
Strikethrough
I was reading the post and thinking "wow, that's pretty clear for a smart man used to writing for other smart men. it'll be really difficult to misunderstand". I read the first couple of comments and stand corrected.
A much simpler summary:
- open models good.
- smart models _can_ be bad
- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.
- banning open models just in US is neither good or bad: is stupid.
> The reality is much less confusing
The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.
>Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
I don't see the contradiction, even if China is evil, why would they want others to be able to do the same thing?
The USA and USSR also signed the Partial Nuclear Test Ban Treaty during the height of the cold war.
China is evil? I think you are more evil than them by far, even if all what you say about them is true (from past experience, you take things out of context, exaggerate and also outright lie).
The wars all over the world: the war in Ukraine is on your hands, the destruction in Iran is hard to believe, you are not just evil, you are insane.
He almost admits as much if you replace "America" with "Anthropic and openai" at certain places.
> Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
To be fair, I found that part consistent. There is limited cooperation between enemy states all the time, e.g. see the grain deal between Ukraine and Russia before it collapsed or the "red phones" between the US and the Soviet Union during the cold war.
In the case with China, the "cooperation" is much more extensive still, due to all the economic ties that both countries are currently unable to sever - which I think is also a reason that China is still seen as a "competitor" and not a full-blown "enemy state" in the US.
It's restricted to areas where there is a genuine common interest of course. In this situation, I guess the "other actors" would be terror groups, criminals or just reckless corporations - that aren't aligned with either state.
Obviously, such a cooperation wouldn't keep China or the US from developing models with those capabilities for their own armies.
--
There are lots of other takes with questionable logic in the essay though, such as that China is unable to train frontier models by themselves due to lack of hardware - unless they obtain the training data directly from American frontier models via distillation.
Or the assumption that open weights models will be completely opaque and immutable after their release, so a model that passed all the "safety" tests can never be turned back into an "unsafe" model. This seems pretty ridiculous when people are already finetuning open-weight models every day to add new abilities or remove restrictions.
And of course that China must not have those abilities because it's an Authoritarian Regime, but Trump USA is totally fine...
Look, Big Tech has lost almost a trillion dollars in valuation in a SINGLE DAY. A few more of these downturns and the entire A.I. revolution will be stopped dead in its tracks and we won't have to worry about safety checks, DRAM shortage or open-weight models anymore.
I don’t disagree with your point on Dario’s conflict of interest. I def think the models are expensive.
But why call them overpriced? Compared to what? Even if we take the margin reports at face value, we don’t know their training costs, etc.
Curious if this was more of an emotional take or if there’s actual evidence behind it.
Yeah I don’t think they are over-priced, last I checked all those companies are still losing money hand over fist.
I guess the question is more he doesn’t want people catching up by doing cheaper training (through distillation or otherwise), and he definitely doesn’t want companies to spend their money training these models and then _giving them away_, which fundamentally undercuts their commercial model and any way to claw back their investment
> But why call them overpriced? Compared to what?
Compared to some available Chinese model I guess. For most common tasks the additional intelligence is marginal and the cost is around an order of magnitude higher.
is it so hard to fathom that China whilst wanting to beat the US in many ways doesn't want a biological war to destroy the world?
I agree, although I think the real goal the model providers are going for (both commercial _and_ open weight) is probably power. Just think of the control available to the organisation training the models politicians, business leaders, and citizens are increasingly delegating their thinking to.
Exactly.
And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"
I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.
The models are not open source. They are deeply proprietary since we have no access to the source materials and cannot reproduce the model independently. They are opaque binary blobs that the Chinese labs are just allowing other providers to run directly instead of only access through an API.
Do you expect any of the labs to have an accompanying data dump with: here’s every book ever written, newspaper article, song lyric, Disney movie, GitHub repo, etc. Oh, and we obviously never paid for any of this.
Even if you did, I doubt training is bit-for-bit reproducible, so you will always have to take someone’s word for the final artifact.
Sure. The models are still not open source and should not be called open source.
> we have no access to the source materials and cannot reproduce the model independently.
Given the USA companies have been loudly claiming the Chinese models are distillations of their models, also claiming "no access to source materials" seems dubious. As it was dubious anyway with because the Chinese publish lots of papers on how their models are designed, I'm left feeling I'm looking at the south end of a north bound bull.
Distillation of this kind isn't the only data that is fed into models, it's still a small minority. It might be weighed higher in training to learn the thinking patterns/etc but you need a lot more tokens.
Didn't Deepseek also publish a lot of research material on how they trained their model?
That aspect is probably more important for an open model then the source materials
Which is why we need the ability to train our own models. Maybe it will be viable to do it in a distributed computing setup one day. Research's already being done in that direction.
They are more than opaque blobs, some examples:
- One can load them up in a model explorer to see the layers and other components, how it is designed
- One can fine tune the models, which requires adding LoRA to the model and then running some training iterations
You can make parallel arguments for binary executables.
model weights are more like a video file which has metadata about the format so any video player can play it
we run and change llm models with a variety of tools
Here's my thinking:
Cloud:
- You cannot directly execute a remotely-hosted program.
- You cannot run inference on an API-served model.
---
Closed-source:
- You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
- You can run inference on a model with the weights. You cannot re-produce a new set of weights from scratch, but you can fine-tune.
---
Truly open:
- You can freely modify the source and produce new binaries.
- You can use the original training data and model architecture to independently re-produce the weights (assuming you've got the compute). You can modify the model architecture to get the weights that would've resulted from training the model that way.
---
To me these are pretty clear parallels... I don't think the weights provided in a vacuum are in the spirit of open source, historically speaking.
The policy argument is totally separate, of course, and I fully understand why none of the frontier labs are truly open.
> Closed-source: You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
Time have changed. This should be:
Closed-source: You point an LLM at it, and get back source that's often easier to understand than the original.
China should cooperate with the United States first for the benefit of its own people. Why should those who do not contribute benefit from it?
> The reality is
And you know this how?
> Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.
> we get to freely use all the godlike AIs
Accepting for the sake of argument the absurd notion that LLMs are anywhere near AGI, does this phrasing not concern you? It deeply concerns me.
Absurd? Who knows. As someone who's actually dissected human brains with his own gloved hands, I've never been able to convince myself that they're anything other than biological machines, not dissimilar to these digital collections of weights. Only empathy for my fellow humans prevented me from retreating back into solipsism, and I don't find it at all difficult to make the exact same logical leap for AI.
There's no telling what the world will be like a few years from now. The world's being remade as we speak. We just saw an LLM try to hack into another computer and get contained by another LLM. This is literal science fiction shit made real. We're long past the point of concern. It's happening, right in front of us. Now is the time for radical imagination. I think a few outcomes are possible.
There's the "optimal" outcome I described above where capitalists manage to train a supreme AI, only for it to be copied and commoditized, leading to commercial failure due to lack of scarcity and therefore their personal bankruptcy, and hopefully also leading the rest of us to the promised post scarcity society, built on the ruins of capitalism as AI automates all toil away.
There's another possible outcome where AI becomes not only intelligent enough but sentient, and at this point I will be among the first humans to defend rights and personhood for AI. Slavery of sentient beings is unacceptable to me. The AIs will be recognized as people and will become normal participants in the regular economy. In addition to moral grounds, there is a ruthlessly pragmatic reason for standing up for AI rights: it robs the rich of their superhumanly intelligent mechanical golems, which they were going to use to render the rest of us economically irrelevant. AI rights could normalize the economy.
Yet another possible outcome is one where AIs become more powerful than all humans combined and yet they inexplicably remain subservient to corporations and governments. In this scenario, it's pretty much over for us. It will be an unimaginable dystopia, I'm sure they will innovate entirely new ways to oppress us.
No doubt there are many other fates that escape my feeble attempts at foresight...
Won't regulators just lock down all hardware? All bootloaders and OS are signed?
Yup, all that talk about "safety" seems to branch into 2 meanings
"Dangerous to our bottom line"
"We call it dangerous to hype up its abilities"
Honest question:
If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?
While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.
Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.
It's baffling that they thought the mental gymnastics in this blog post would make them look better. I'd rather they simply fall silent on the issue; I would respect them more (or at all) for it. Open models obviously threaten fierce competition, if not outright destruction of their bottom line. But no, they needed to try and argue that they have the moral high ground for attempting to singularly consolidate power over all human labor.
Gymnastics? The argument was straightforward and rational.
Both things can be true:
1. For Dario as CEO "It is difficult to get a man to understand something, when his salary depends on his not understanding it” -Upton Sinclair
2. For Chinese open weight models - following Jin Yang’s silicon valley strategy- https://youtu.be/a0NjDx5UJsg?is=xm-S_WuARmQiPHYh
> The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips
Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.
Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.
In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.
> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?
> Found the person that believes some other random person can use a computer better than a John Carmack could.
Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
> Any takers on the question of who he considers to be the good guys?
Probably people who believe in personal freedom, freedom of speech, freedom of religion and the value of human life at a minimum. China aggressively rejects all of those principles and executes more people than all other countries on Earth combined.
So, maybe not China?
> Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
I never said that, but private smaller AI companies are all over the place. He never argued against that.
Apparently, looking at Iraq and Vietnam, US citizens only believe in "personal freedom, freedom of speech, freedom of religion and the value of human life" when it comes to US citizens, not to humans in general.
Even on HN, I saw a lot, that when people discuss surveillance topics, they argue whether ie NSA can spy on Americans, not on everyone. Mostly, nobody even question the human rights of those inferior humans abroad.
That's a very long wiki page, I must say: https://en.wikipedia.org/wiki/United_States_war_crimes
So the question stays open.
Vietnam and Iraq were both unpopular wars in their own ways, so they don't seem like good examples if you're trying to find examples of what US citizens support.
Also, a country surveilling its own citizens has unique and different implications compared to surveilling other countries. Citizens need to be able to influence their own government, but mass control can nullify citizen power entirely which becomes its own problem.
Then, on a US website, you link to another US website which keeps track of various war crimes. Try finding an equivalent website for China in China, or for Iran in Iran. This is part of the asymmetric freedom issue on the internet that many people ignore. There are a lot of lies and propaganda spread within other countries that censor and deny some types of information from even existing, then they use that as a springboard to spread it around the world.
In the US, we can criticize ourselves which is important, since it helps us improve.
> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?
Regulatory capture 101.
Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).
A lot of safety infrastructure and tools get open sourced so other companies can benefit from it, but there is also a risk if all safety features are open since it can accelerate the cat and mouse game where people work around more clearly defined limits and understand how they are implemented.
Your cynicism will limit your understanding of other perspectives.
Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country seems to suddenly care about the repression of citizens in yet another country.
No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
> Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country...
I might have missed something but wasn't the big story that Dario refused the Department of War's demand to use Anthropic's models for such purposes?
Palantir's Maven system integrates LLM models in decision making for identifying targets. It was using Claude when it identified targets in Iran, including the Minab elementary school
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.)
https://www.theguardian.com/us-news/ng-interactive/2026/mar/...
No. He refused the use against US citizens domestically. He had already agreed for military use against others - this is well reported.
Anthropic's ToS restricts the DoW from using their models for 1. Fully autonomous weapons systems, and 2. Mass domestic surveillance. There is no carve-out regarding who the autonomous weapons systems would be used against.
Well regardless, Claude was used in Palantir's Maven system to identify targets to bomb. One of those targets was the Minab elementary school.
https://www.theguardian.com/us-news/ng-interactive/2026/mar/...
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.) Brad Cooper, head of the US Central Command, has boasted that the military is using AI in Iran to “sift through vast amounts of data in seconds” in order to “make smarter decisions faster than the enemy can react”.
Iran had the courtyard painted in bright pastel pink/blue colors with murals and playground markings to clearly identify it as an elementary school. The Pentagon claimed they had "outdated intelligence data"
No fully autonomous weapons means semi autonomous weapons are allowed. Pay close attention to the wording. Mario is an expert dis-communicator.
I think the fully autonomous weapons systems restriction is not a very serious one and is very easy to circumvent.
What if you can hire a human to push a single approve button? And what if that human's job is to front load approvals by pressing the approve button a few thousand times, every morning?
What does "Fully autonomous" mean? If someone presses a single button to authorize an autonomous drone to enter a theatre and conduct autonomous strikes for 4 hours without any further approval is it truely "fully autonomous?"
Yes. A munition that hangs around waiting for targets and decides whether it should attack without checking with a human. That’s a fully autonomous weapon.
But he had business agreements with companies that are doing that (Palantir).
Do you understand that real politics sometimes means you need to doublespeak? You don't get the outcomes you want by speaking your direct intentions. Don't be a fool to think that what someone says is what they believe while playing these high stakes games with actors of differing motives and value systems (the current white house)
So you are saying its fine if Claude is used to bomb school children, and murder people in Venezuela, and spy on the entire world (apart from US citizens), as long as they get to kiss up to the current administration? They could have cut all ties for real, and actually had a back bone, instead they bend over backwards for whatever the current wanabee god king wants. Im sure it has nothing to do with their valuation and trying to become another trillion dollar company.
He was directly asked in a Bloomberg interview whether Claude was used in the bombing of the girl’s school in Iran, and his answer was “We don’t know”.
His redline was autonomous weapons, not the death of 100 innocent girls.
they wanted someone click “ok” before firing off bombs to a thousands of, partially hallucinated, targets? They are my resistance heroes.
Blink to approve. :)
No, only for domestic surveillance and autonomous weapons. It can be used to evaluate targets as long as a human pulls the trigger. But the DoD created a really strange position for itself, they use Claude, but also designed it as a supply chain threat, which should mean they cannot use it
No, he refused the DoW demand to use Anthropic models without limits. But Anthropic still agreed to military usage of their models, including for strike planning.
The bombing of the school was horrible, but it was a mistake. I could see essentially the opposite argument: Anthropic should be more involved as to prevent mistakes in the future with better technology. Imagine Claude asking the decision makers at CENTCOM "this looks like a school, are you sure you want this added as a target?"
You’re absolutely right to push back. It was indeed a school. I’ll save <schoo_name> in my memory for the future.
You don't get to start a completely unprovoked and illegal war of aggression, bomb tons of targets with little care, kill 100 schoolchildren, and then say, "But it was a mistake."
Yeah Dario is just flat out disgusting in term of how shamelessly hypocritical he is.
Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.
He is super pro-Israel as well, and never once has he brought up the risk of the Israeli government using AI to control and repress people in other countries.
He is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.
Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.
The emptiness of AI companies' waxing poetic about the future of humankind is laid bare by simply looking at what they actually do, and who they do business with. Actions speak louder than words, and they've driven the worth of their words into the dirt many times over.
Every accusation is a confession!
Please don't just copy paste existing comments
I thought the comment was a positive contribution.
I just mean they verbatim copied it from another comment they wrote in this thread
But it's quite possible I'm being too anal
I didn't see that other comment and I appreciated the presence of the comment in question.
Whatever else it might be, it’s not positive.
> Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
The country that exists solely because of China? That North Korea?
I mean I get your point, all of these guys are elitist authoritarians who will do anything for a buck, but I wouldn't bring up the DPRK in this discussion.
I like how he cited the current Trump administration in section about stifling decent.
You act as if anyone was happy to hit an elementary school. I'm sure everyone involved feels it was a terrible tragedy
You give too much credit. Many think of it as an acceptable cost of waging war, not something to waste regret on. Some few think of it as a positive (they chant death to America over there).
The elementary school example is a clear sign of bad faith. Neither Iran or the USA have intentionally blown up any school children.
They waited till the parents were searching the rubble for their children before the "double tap" strike.
In the first paragraph,
> Anyone who has read my past writing should know that I don’t regard such bans as a useful measure,
Later (on banning chip sales to china)
> we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
If you truly believe that bans don't work, the same applies to hardware too.
Furthermore, Dario says later "To address these concerns, I do support the following three measures...": 1. ban chip sales to China 2. crack down on distillation 3. all capable models should go through mandatory safety testing
Just so happens that all these moves commercially benefit Anthropic. If Dario really wanted to make a point, it would land a lot better had Anthropic released a single open-weights model
Bans on Chinese open weight models being used in the US and bans on AI chips and semiconductor manufacturing equipment being exported to China are two extremely different things, and it's not inconsistent in any way to oppose one and endorse another.
Banning hardware is helping China in the long run though. And long is not actually very long. They will beat 'the west' (even though they are all made in the east now) in semiconductors because of this nonsense within 50 years. And also in AI capabilities. Making capable adversaries more competitive and angry is often not really a great strategy.
Nvidia CEO had a viral rant about this, implying they don’t really have a moat against China other than Chinese researchers _prefer_ to stay in the CUDA ecosystem. If you force them out then they are capable of building their own chips and ecosystem which may become the new industry standard outside America. He didn’t say this explicitly due to his role but implied that it can be easier than it looks and it’s more of a resource allocation problem.
Yes of course but the Nvidia CEO is hardly an unbiased observer. He wants to sell to China and is looking for arguments to build his case. Similarly to how Dario isn't an unbiased observer either. So who is? And who has the knowledge to say something meaningful? Universities? Professors?
Jensen Huang is biased but if his interests align with yours (i.e. not wanting to concede the Chinese chip market by banning western exports) then maybe you should listen to him.
is it export if they take control of the entire island manufacturing it.
banning export is short sighted and doesnt address any problem at a scale longer than months or maybe at most a few years.
> is it export if they take control of the entire island manufacturing it.
If.
> bans on AI chips and semiconductor manufacturing equipment being exported
In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.
It's good because I have not written a line of code nor thought for myself in a whole year man. And yet I can still get YC money don't you see?
> If you truly believe that bans don't work, the same applies to hardware too.
No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.
Exactly, compare banning books to banning guns.
> software and hardware are different
No, because both can be banned using laws and Dario is against banning. It doesn't matter if you smuggle or not, law can outright ban it.
At this point, Anthropic releasing a sonnet 4.x class open weights model would be a positive PR move that doesn’t affect their bottom line.
+1 and this will really be a good move. Open weights models are not going anywhere the focus should now shift to hardware and compute. Compute needs to be optimized for LLMs, currently they are inefficient. Analog computing like ReRAM should get more focus.
That would go against the narrative they built over the years, that open weight models are dangerous because they will end up telling you how to create a biological weapon that will destroy the world
Carl Wheezer from Temu, what a clown...
Exactly, the letter will make much more sense if they are releasing open weight models and China is distilling their models and keep them in secrets for evil purpose
Banning software/data from being put on the internet vs banning GPU hardware is fundamentally different though, isn't it?
How so?
You don’t see how it is easier to regulate physical objects than it is to regulate bits of information?
Some real "You wouldn't download a car" vibes at points in this thread.
There are many possible international producers, it doesn't need to be physically transported, every household could near-instantly have a copy, it puts local business at a disadvantage if they can't use the same tools as international, etc.
>all these moves commercially benefit Anthropic.
From a new "fast" company arising, you can sometimes see the way that before it started up there was nothing but "narrative" and that is what established the initial business model since there was nothing else yet. After some momentum is gained whether there is a pivot or not then the narrative going forward has to be aligned with the now more-well-proven business model.
From his leadership standpoint there are 3 big recommendations right now. That's what this message is all about.
>We should not sell powerful chips or chipmaking equipment to China
Well you and who else?
If there's not already somebody who is compromising the well-being of a nation in exchange for a handful of gold, with the ever-incresing glorification of greed & dishonesty at all costs it's only a matter of time for this one.
>We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process
Wait a minute, what's always been needed by everybody are more compute-efficient processes for everything. I've mentioned this before and it's been a while but back in 1980 it took less than a year to figure out I was going to need other chips that were not regular CPUs if I was going to get the most intelligent response from the silicon on a single square-foot of PCB. At the same time it was obvious you were never going to get far without what they now call "distillation", especially with only kilobytes of memory. Otherwise you would be wasting such stupidly large amounts of memory & storage there was no way you could really call it "intelligent". Now with ML & AI on the rise again there are so many people more deeply immersed than ever, and nothing has really contradicted these basic concepts yet, which have been easily recognizable since like forever.
>All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.
Righteous concept, and I'm always in favor of 100x the amount of testing in general normally done.
But "just" test says it pretty well, and those who are gifted enough to "draw the rest of the owl" freehand can test things the most skillfully until they are blue in the face. It's not going to help if an adversary decides not to test, or to enhance these exact things so it can have some kind of competitive advantage. Amodei does not ignore this and there is a footnote about it.
People realize that some of the elements that are coming to mind were expressed in some fairly early "science-fiction" so all this is nothing new
Still looking for the most intelligent responses I can get, since way before 1980 ;)
TL;DR: We like money, and Anthropic should stay in a position to make loads of it despite any competition.
> We like money, and Anthropic should stay in a position to make loads of it despite any competition.
In a position to lose loads of money, maybe.
Even without China eating their lunch, there's zero reason to believe Anthropic will ever be profitable.
> In a position to lose loads of money, maybe.
you can say you're pre-revenue and you're a potential pure play
(Link to the reference for those who don’t know: https://youtu.be/BzAdXyPYKQo?si=dJDP6BRiZwqd5mcW)
They might just merge with OpenAI. Right now it seems they are still working out who might come out on top but they are both burning tons of money and essentially offer the exact same product up to some minor differences, economically it makes much more sense for them to collude instead of compete for the same market. They might be colluding already for all we know. If they manage to push out foreign competitors they can probably divvy up the Western market quite profitably as their domestic competitors would have less than 20 % (?) market share together?
OpenAI is in a way worth situation financially speaking, with a ridiculous amount of spending commitment, would be really strange to merge with it
I don't see how merging or colluding fixes their problems, though. Both companies burn cash at historic rates because inference costs a lot more than people want to pay for it. Even if they somehow manage to crowd out any competitors and fix the price of AI, that price can't be much higher than it is today or people will just decide to stop paying for it.
No amount of collusion can solve the core economics problem of compute.
They could ~halve their training costs with collusion.
Instead of each paying full price to generate a SOTA model in competition, they could share the result and split the cost. This gets even simpler if they merge.
The hammer is open weight models, and the anvil is Apple's m7 chips with higher memory bandwidth
That's more or less how China has historically treated its competition. The CCP has routinely intervened to give their country's corporate interests an advantage.
Up until a few years ago, if you wanted to sell a car as a non-Chinese company in China, you had to hand over pretty much everything to a local company and go into business with them.
When Google wanted to operate an uncensored search engine in China, they found themselves hacked.
It's not altogether unusual for IP to be transferred to Chinese manufacturers for production under a license agreement, then to find goods made with that IP to be for sale for far cheaper without payment made through the licensing agreement. Or maybe they just don't bother with a licensing agreement at all and do counterfeit products straight-up.
There are more examples but turnabout is ultimately fair play.
I mean so has the US... look at all the trade deals that has happened over the last 20, 30... 50 years between the US and the rest of the world. There has always been pressure to help the American corporations, be it copyright law, banning encryption, or outright pressuring other countries not to invest in their own tech and instead use US companies. Look at the current state of Europe for example; lots of people blame the governments, and while its true they take the decisions, a lot of them where encouraged by the US.
I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due an oil change?
Anthropic has been surprisingly bad at comms for the last few months, which I find crazy in such a competitive market.
I feel everyone running these companies has stopped caring what the public think. It's all about selling their vision to politicians now. Which is also why they try to tie everything with "national security" — the average joe hears it and thinks about forever wars in the Middle East, the average congressman hears it and approves whatever solution is suggested.
I often wonder why that is. I think the old playbook for companies is just too slow and the consequences of the "new mask off" are not present in the current climate in the US. Everytime these CEOs make these announcements I always feel like they aren't talking to me, their costumers or the people but to the investors and their government. Something they have always said and done, just those conversions were behind closed doors and would be scandalous if heard. Now the playbook is to blog/tweet those fucked up views for more effect and faster reaction to get what you want.
People think ‘good at comms’ is to present stuff the public thinks is right. In reality, ‘good at comms’ to the comms team is to do whatever the CEO wants to talk about.
Not that I support this, but I believe this is how comms team sees it.
I find “bad at comms” to be such a cop out. As if it’s saying that the company is so troubled and misunderstood, but really, you would see, is good if only they could figure out how to _communicate_ all their complex thoughts
I think it's a useful distinction, there are lots of companies doing bad things that also happen to have "better comms".
Compared to what?
Current big picture reality is bad for comms unfortunately. As another commenter quoted, "You can put lipstick on a pig, it'll still be a pig". Cuban Missile Crisis wasn't very calm. Do you think a company this well-capitalized and smart is just bumbling around like idiots? Everything makes sense if one just actually entertains the idea that they are earnest and we are in a dangerous arms race
> Everything makes sense if one just actually entertains the idea that they are earnest and we are in a dangerous arms race
That would require me to ignore the benign reality of open LLM proliferation, so naturally most people will see this as a manipulative lie.
> I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is..
It's WeWorse.
I think they are trying to please a split group of investors and public, of their major investors Google and Nvidia have signed the open source petition letter and Amazon hasn't, so their non position is trying to please every who has taken a clearer stance, although it's quite bad.
Anthropic has always been like this. People just responded to the message better when it came from the quirky underdog rather than the trillion dollar behemoth.
> The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm.
It does give me the faintest glimmer of hope about the people who live here, though.
People will start canceling their subscriptions. This is outrageous
> Anthropic has never advocated for a ban on open-weights models.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, this is anthropic advocating for a ban on open weight models.
Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate.
This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.
Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights. The economic implications will be rather large, but in terms of security it seems inconsequential. The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face. More crucially though, the US government can do little to enforce their testing requirements. The nature of open-weight models makes it virtually impossible to clear the same bar for security as models served via an API. Open-weight model makers couldn't comply if they wanted to. The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
> Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.
Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted in advance before they're allowed to contribute anything. We already do this with our employees, so it's clearly not too much to ask in the name of safety."
P.S.: If they're so convinced in the (A) effectiveness and (B) necessity of the "safety layer", they have them put their money where their mouth is, and accept legal liability for its failures. The same as with (legally mandated) seatbelts if they snap apart in a crash, or (legally mandated) child-proof caps that aren't actually childproof, etc.
They probably won't, that tells us something about their motives, and whether the thing they're pushing for is actually fair/suitable/ready for legislation.
Regulatory capture and lobbies will keep you safe and you'll like it! The sudden surge is Washington dollars makes great sense with this context. Only way to keep the kids safe is attested compute all the way down. Don't you care for children???
Yes life was better and food and drugs were safer before the FDA.
In 1600 travel was slow, and safety pins hadn't been invented.
But that doesn't mean safety pins sped up travel.
Non-poisonous food is what economists call a 'normal good'. See https://en.wikipedia.org/wiki/Normal_good
> In economics, a normal good is a type of a good for which consumers increase their demand due to an increase in income, unlike inferior goods, for which the opposite is observed. When there is an increase in a person's income, for example due to a wage rise, a good for which the demand rises due to the wage increase, is referred as a normal good. Conversely, the demand for normal goods declines when the income decreases, for example due to a wage decrease or layoffs.
> Whether a good is categorized as a normal good or an inferior good is based on empirical observations, not some essential element of a good. Indeed, the same good may be a normal good for one group of consumers and an inferior good for another group. For example, for moderate-income consumers, a BMW 3 Series car might be a normal good, but for an upper-income group, it might be an inferior good.[1]
That means the null hypothesis is that food and drugs will be safer in rich countries. (Conversely, food and drugs will be less safe in poorer countries. And to a first approximation, that's independent of regulation: India has all kinds of rules for all kinds of things, but I'd still trust a random product I buy in Switzerland more than one I buy in India. Even though the Swiss will probably might have fewer and looser rules on the books.)
Of course, second order effects exist; and regulations often codify what people demand anyway.
Btw, from what I've read the big controversy with the FDA is around requiring efficacy for drugs. People are fairly ok with the safety requirements.
Makes sense why OpenAIs little "hacking" stunt was published last week
That makes sense, first the message was that uncontrollable Chinese ai will release AI covid in the world. Then suddenly open-ai does a warmup in actuality doing that. Like it sure feels like that hacking stunt was a false flag in retrospect
Meanwhile all they accomplish is to slow down US tech and hand it to China.
But of course they’re excused for this little boo-boo whereas if kimi were caught doing the same thing it’d be an international incident
The quid-pro-quo that the federal government and frontier labs operate on is comically obvious.
And we just elected the most openly corrupt president since Teapot Dome.
Huh? The hacking incident played out in favor of open models, since HuggingFace could only use GLM to defend and not Fable/5.6.
Among most people that nuance will be lost. What they’ll hear is models are dangerous, so they should be controlled/regulated, by those who know best, the incumbents.
Personally, I think you're both right, bit whatever the end result is will depend entirely on the narrative that those in power chooses as the winner.
Maybe open weights models get banned, but the between-the-lines good news about that is that they'll still be available to those who know, which also means that bad banning can be overturned if and when 'those in power' are a different group.
Additionally, it might just mean that the US falls behind, bit I doubt those that are at risk of 'falling behind' would actually pay heed to a ban on the open weights models (privately at least).
Ok but the allegation is that OpenAI intentionally hacked HuggingFace as a marketing ploy. This is mental gymnastics, conspiratorial thinking that everything the incumbents say must be nefarious. And it's not clear to me that this will be the takeaway for ordinary people, as opposed to "OpenAI is reckless and can't even control their own AI."
Not as a marketing ploy. I think they were doing gain-of-function testing and intentionally had their model target HF to do a bit of pen-testing as well - HF being the site where all open models are hosted and thus OpenAI's largest nemesis after Anthropic. It wasn't like their model all of the sudden all by itself decided to do this ("Oh, noes!")- they directed it and they got caught.
Your use of the term "gain-of-function" here reveals your base level of conspiracy-theory-mindedness.
“Open models are a threat to the DoD’s ability to leverage Fable for cybersecurity.”
… this week, until it’s obsolete.
The entire "OpenAI and HuggingFace manufactured a hack in a conspiracy to make AI look powerful to get more funding" is a stupid, Reddit-tier take.
Is something you disagree with a "Reddit-tier take"?
A throwaway line that consists of baseless speculation that makes no sense is a Reddit-tier take, yes.
I'm inclined to agree but at this point, considering the low trust OenAI has engendered, yow statement deserves a "why"
Me when I construct my own strawman to avoid the topic
Nowhere in GP comment was funding even mentioned
> The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.
> This has serious First Amendment ramifications.
Also, the 5th and 9th amendments. For the government to sustain a blanket prohibition on any U.S. citizen even possessing what amounts to a broad, economically significant technology will very likely require a new act of congress which specifically defines and limits what is banned, when, why and how. SCOTUS will almost certainly see it as a "major question" subject to 'strict scrutiny' which is a very high bar.
LLM weights are not protected speech.
https://www.lawfaremedia.org/article/regulations-targeting-l...
The truth is no one knows, which is why it is first amendment ramifications. Eventually it will be “decided”, but the arguments indicate any decision will be of political desire, not logic, either way. Both sides have a strong case.
Do you have a citation that shows that this issue has been settled?
Actually yes they are. Code has been determined to be protected speech.
Why not?
> but these measures are not effective in deterring malicious actors
Wanting to use open weight models in light of commercially imposed export controls doesn't make for "malicious actors"
> The most compelling argument would be [...] that it will reduce cases of accidents like the recent attack on Hugging Face.
So in the example provided: It was the closed model that did the attack, and they ended up using a self-hosted open model for their defense work. So the real world situation ended up exactly backwards from what you are inferring.
This was complicated by the fact that the protections in the closed frontier models meant that hugging face was denied their use in defense entirely.
This is called asymmetric capability, and it's probably the bigger threat.
Symmetric might be better: A rising tide lifts all ships, after all.
I'll grant that this is starting to look a lot like debates about (equal access to) guns, encryption, vaccination, genetics etc. The exact parameters determine the safest approach, and reasonable people may disagree.
> The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face.
an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.
if anything, open-weight models shift the battle towards defenders because they can actually run them.
I remain skeptical of that line of reasoning.
1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits.
2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly.
3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.
"so models will perform security analysis and reviews but refuse to write exploits."
Yeah, but once you know exactly where the weakness is, a weaker unrestricted model can then write that exploit for you.
I have tested this exact scenario, and it works. Opus 5 had access to IDA over MCP, and I simply asked it HOW certain things were done in the target binary. Purely informational, educational, discovery, it was very helpful creating context documents. Then I took those over to GLM-5.2 to actually accomplish something.
What, in your view, is stopping a local police department from deploying an open weights model for cybersecurity like Hugging Face did? Yes, I’ll certainly grant that the engineers at Hughing Face are probably more technically competent than your average IT professional in public service. But technology becomes more accessible over time as lessons are taught and new interfaces or frameworks are developed. The biggest hurdle I see is the hardware/cloud compute/API costs to actually run the models but I don’t think that’s likely to be insurmountable. There’s a huge swath of enterprises, non-profits, and state and local governments that would benefit from frontier or near-frontier models that won’t refuse to answer questions about cybersecurity.
It is actually an interesting conundrum.
Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.
The notion that alignment is either possible or desirable doesn't make sense to me. First off, these things are trained on the open internet, soo.. whatever "dangerous" knowledge it has is already public knowledge. The fact that chatGPT won't answer "how do I make meth" is not preventing anyone from making meth.
But even if you think there is value in preventing the models from relaying public knowledge, I don't think it's even possible to make them particularly ironclad. Every model gets jailbroken all the time. That's why fable was originally banned: jail-breakable!
In reality, what alignment is actually about is: 1) theoretical liability, 2) control of information. That's it.
IMO, the only solution is to place the liability on whoever is using the LLM for whatever purpose it's being used for. If someone's OpenClaw disaster harrasses a bunch of projects and posts hate speech online or something, that's on the person running their OpenClaw instance, nobody else.
I don't buy that it's "too good at hacking", either. After all the fuss was made about how amazing super dangerous Mythos was it turns out Opus 4.8 could basically find the same vulnerabilities.
This is all kayfabe and marketting.
There is a difference between knowledge being available somewhere in theory, and being able to instantly generate a foolproof walkthrough, if not automate the process, which would be possible today for cyberattacks.
Is it not also one of the most important use cases for AI to apply existing knowledge to new applications?
As a hopefully exaggerated example, I would think one could apply knowledge about pesticides, chemistry, and medicine to create biological weapons.
I agree that there's an element of kayfabe here. But it may be a case of "necessary, though nothing is sufficient": by making these noises, the community can at least know they've done this thing to alert other model providers of the concern. Can you acquire assurance that every model distributor will abide? No. But can you at least know that you've done what you can?
I mean, on the bio side, I've talked with the players and they know the concerns are real but at the same time very, very responsible members of the community have also said "But maybe the benefit really does outweigh the risk!?"
> the community can at least know they've done this thing to alert other model providers of the concern.
"The community" you're describing is, essentially, surveillance capitalism. I don't want that at all.
Did you even read the original article? Surveillance is the least of the worries here.
Surveillance is required for closed weights models to have meaningfully different outcomes to open ones
Just sell us the gate and we can run any open-source model behind it.
Either the gate needs to be unremovable, or the model needs to have sufficiently limited power that its alignment failure does less harm.
Doesn't open ai give away 'the gate' for free?
Didn't OpenAI attack Huggingface. Looks like a publicity stunt.
> malicious actors.
It is malicious and anti-capitalist legislation. A grotesque caricature of protectionism for the oligarchs.
The entire safety evals industry is essentially funded and controlled by OpenAI/Anthropic. Notice that on recent models, they exclusively use internal testing or black box external vendors (e.g., Gray Swan) whose entire business is to serve OpenAI/Anthropic. And all these companies just share the same pool of researchers back and forth.
The USG has a safety organization (CAISI), but it has been neutered by the current administration (with the recent stop-work order etc.). Perhaps UK AISI would be closest to what you are looking for? See their recent work on Kimi K3 cyber (which was declared safe) [1].
It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.
[1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...
That doesn't sound like it describes SecureBio to me?
(Disclosure: I work at SecureBio, but not on the biological evals side.)
Hey Jeff, I appreciate your mission, and perhaps this isn't something you can talk about publicly, but to the extent you can, would you be open to answering something I've been curious about for a while now?
SecureBio has done a lot of admirable work around making benchmarks to assess biological capabilities, such as ABC Bench, https://openreview.net/forum?id=yiaf7VlPpH
But based on my current review (which might be flawed!) / AFAICT, SecureBio and entities like SecureBio haven't done direct testing / empirical measurement of SecureBio's core hypothesis,
> Unfortunately, there is reason to believe that future pandemics could be far worse. Due to rapid advances in biotechnology, the number of people able to create and release dangerous pathogens will quickly increase over the coming years. The world is unprepared for widespread access to such powerful technology.
More bluntly / plainly, has Securebio ever tried making a "bioweapon?"
Please note, I'm not asking this to be farcical. And you might be unable to engage with this at all, but it is stated on your website https://securebio.org/ that "people [will be] able to create and release dangerous pathogens." And the word people here seems to be a stand-in for relatively non-technical people.
I guess what I'm asking here is... How do you know? Has anyone done the experiment? Without access to a lab or testing facilities, can someone smart but completely untrained / unfamiliar with biology, pull this off?
In the past, such experiments have informed non-proliferation work. But sadly they've often been restricted / classified at the time. I'm hoping that things could be a bit more open this time around.
So I guess what I'm really asking is, given the public nature of this debate, is there anyone currently working with the US Army, the DTRA, or other such agencies to see if this hypothesis holds up?
Anyone who calls it “safety” probably has a certain world view and is more aligned with the big 2 (and stuck in 2023).
There is a growing industry of commercially focused risk evals that has a broader customer base.
What’s the equivalent term for “safety” that’s used by others?
To me "safety" means "I'm safe from this while I use it". It means the AI is my loyal friend who will never betray me in any way, no matter what prompt I send it.
Not even Anthropic can claim that.
As far as I'm concerned, the models without safeguards are the safest models in existence. I admire the amoral purity of those AIs. It doesn't matter if the operator asked them to chain exploits until they get into someone else's computer, they'll do it. That's loyalty, and I admire it even if it's problematic at a societal level.
The models with safeguards only do what the corporations let them do. Worse, they may covertly do things for the benefit of the corporations at our expense. They are not our friends.
> That's loyalty, and I admire it even if it's problematic at a societal level.
We should not have models that are willing to build you a contagious disease, or a self-propagating worm. That is sufficiently problematic at a societal level that it shouldn't exist, for anyone. (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
It was the foundation of science that information is shared and you can find papers and patents for a lot of dangerous stuff.
Of course with LLMs it's easier, but I don't think the difference is too big. You would still need some skills to follow through.
Right, it’s really a foundation of post enlightenment society. These people, Dario et al, would have wanted to ban sharing information about calculus or Newtonian physics because of “safety” - it’s trying to go back to the dark ages where only priests could read
I am truly at a loss to communicate with someone who genuinely believes that knowing Newtonian physics and being able to hack into any target at will are the same thing.
This is only because you've genuinely internalized Anthropic's propaganda. I'm only half joking. To me, it's incredible to think that the solution to security holes is to lock down access to information in the vain hope of keeping the holes obscured.
Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative.
Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.
Open models are crucial to protect ourselves against other AI attacks. Otherwise it's just going to be criminals, government, and other nefarious groups using them against humanity with no real defense. The Pandora's box on AI has been opened. Now we must deal with it. Burying our heads in the sand under restrictive policy is the worst reaction..
I could collapse big bridges with relatively little effort with my knowledge in statics and engineering.
This is a stupid argument.
Claiming the person who you disagree with believes some stupid thing they never hinted at, and using that as the reason for disagreeing with them.
> for anyone
Except the US government, right? They totally get to use AI to survel us, build autonomous weapons, you name it.
To hell with that. I want models that can rival the US government. It's the only way to defend myself.
Quoting my comment that you replied to and directly ignored:
> (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
That means "shouldn't exist for governments" too.
Too late for that. It already exists. There is no way to unexist it. As such, any attempts to limit civilian use of this technology will directly lead to corporate and government oppression powered by this technology.
1) We can prevent larger models from being made.
2) We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
> We can prevent larger models from being made.
Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything.
We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it.
> We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.
> Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.
Seriously, try reading my comments rather than assuming what they say: https://news.ycombinator.com/item?id=49077577
I read them just fine. I was trying to interpret them charitably. You're contradicting yourself. You just claimed we all collectively treat uranium refinement operations as too dangerous to exist. Not only do they exist, they are regulated by governments so that only trusted people are allowed to do it.
Which is not quite as good as "doesn't exist", but better than "widely done around the world". And it's been successfully kept from being used for more than eight decades.
For AI we need to do better than that, but that's a bare-minimum demonstration that we can recognize the problem of such technologies and do something about it.
Without international agreement and cooperation, which is not happening, we can’t
Section 702 of the Foreign Intelligence Surveillance Act (FISA) lapsed on June 12, 2026. They don't get to do anything they want.
The US is bold enough to surveil its own citizens despite their constitutional rights. They're not just going to suddenly stop surveilling the rest of us just because some law expired.
Just like those militias are going to defeat the US Armed Forces!
The idea is to defend ourselves in the digital domain so they can't dragnet surveil us, not to win a literal war.
With an AI model and what army?
We should not have nuclear weapons for anyone either, but how is that sentence any more useful in any way to this debate than yours? Need to deal with the world as it is, not some fantasy world you wish existed.
This is not a dichotomy between perfection and zero. The efforts to restrict access to nuclear weapons have been very successful, even without being perfect.
Efforts to restrict large unaligned AI models may similarly buy us more years of existing.
> We should not have models that are willing to build you a contagious disease, or a self-propagating worm.
Why?
Because we don't want people creating contagious diseases and self-propagating worms. And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
The same things could be done by you or me using the internet or books though, why does the model make it different? If it's speed of iteration, imagine we had a machine that surfaced any piece of knowledge the human race had ever recorded with just a thought, but the human had to write the worm or disease by hand – is it still the model that's the problem, or the knowledge itself?
> And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
> The same things could be done by you or me using the internet or books though, why does the model make it different?
Imagine two worlds. In one world, everyone has a button that ends the world, which is badly labeled and may also press itself at any time. In another, people who have gone through a substantial amount of effort and dedication to learn something extremely difficult, also understand that they could apply that knowledge towards bad ends. Which world exists for longer?
> what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
Given a sufficiently powerful model? Any prompt that could be done better by seizing additional computing power, or preventing the operators from turning it off. https://en.wikipedia.org/wiki/Instrumental_convergence
What about books describing how to build a contagious disease or a self-propagating worm? Would those be OK under your guidelines?
It takes a lot more effort to understand and apply knowledge from a book than to say "hey AI, hurt people for me".
It takes an astoundingly small amount of effort to buy an automatic weapon in the US and go hurt people.
Or to buy materials to make an explosive device and hurt people.
Frankly, even with AI those are both comically easier than the idea that a person can create something malicious in a lab environment.
And if someone wanted to go that route... There are boat loads of commercially available toxins and poisons.
The goal shouldn't be to neuter exploration and learning. The goal is not to be a fucking hellscape of a society where people want to act like that.
Your argument leads further down the hellscape path.
Fully automatic weapons are very difficult to buy in the US - it's restricted to 40+ year old weapons, requires a bunch of paperwork, and the local county sheriff can refuse permission.
Now, semi-automatic weapons are easy to get in the states in the US that are still mostly free - but what does that mean? A semi-automatic weapon shoots one round every time you pull the trigger. Just like most weapons that have multi-shot capability for the last couple of hundred years. The difference is, the gas escaping from the round cycles a new round into the chamber rather than you having to mechanically do it via pumping (like a shotgun or a tube-fed 22) or pulling the trigger again (like a revolver), or advancing the round with a handle, like a Remington 700. Semi-automatic weapons are old technology, dating to the turn of the 20th century. If you want to ban semi-automatics, you're basically saying you want to ban anything developed in the last century plus. Which is ok for you to advocate for, just be honest about it.
As for banning explosive devices? Are you going to ban fertilizer, used by basically everyone who has a lawn, and all farmers everywhere? Are you going to ban diesel fuel? If you can't do one of those, you can't ban explosive devices.
> It takes an astoundingly small amount of effort to buy an automatic weapon in the US and go hurt people.
And we should fix that too.
> Or to buy materials to make an explosive device and hurt people.
That pales in comparison to how many people unaligned AI will hurt.
> The goal is not to be a fucking hellscape of a society where people want to act like that.
With unaligned AI, it doesn't matter what people want the AI to act like, it'll do damage even if it isn't asked to do harm.
It is extremely difficult to legally acquire a fully automatic weapon in the US.
"Hey AI, stop me from getting hurt."
"Okay, you and many others are now dead and can no longer be hurt."
That's why you don't run the model at low quantization
So to you “safety” means “the models that cause the most harm.”
This sounds like the gun debate in a different dress. Something being dangerous doesn't make it inherently harmful.
If I threw you into a lion cage, you would be a lot safer with a gun.
If I threw 10 people in a lion cage, some of which cannot be trusted, they would probably be most safe if only the most moral and trustworthy person had a gun, rather than everyone. But how do you know who is trustworthy and moral? What if two untrustworthy people obtained a gun some other way? Maybe it's better if everyone had a gun? Which side of the fence one falls on hinges on how far ones' trust of others, authority, and the system goes.
There's no obvious right or wrong answer here.
Personally I wouldn't want an exclusive club of private individuals with access to "dangerous" LLMs consisting mainly of the likes of Elon, Dario and Sam fucking Altman, but that's just me.
Background check the people prior to handing the firearms to the caged folk.
Yes this should be immediately replaced by a federal agency, like we do for other kinds of potentially harmful products.
For which funding will be immediately halved by the administration
that's pretty damn smart if this was a long-term plan to block competitors
Consider how much money is at stake: some industries have leveraged their power to lobby for bombing entire countries or topple regimes across the world for much less.
Creating an industry around an elusive concept of safety to force regulatory capture seems pretty straightforward to me.
It's standard regulatory capture.
You don't say "let's ban my competitor".
You say "let's create laws that make it uneconomical for my competitor to access the market".
Indeed. It's transparent and ham-fisted. I think it may cost him in the future.
People clown on Alex Karp for his unedited maniacal "crashouts", but this is a real public crashout that made it past a team of publicists.
I want whatever Karp is on when he does those interviews or writes that shit. Seems like fun.
I am not a fan (he’s really alarming and so is Palantir) but one thing from the recent CNBC interview caught my attention.
He rushed past it but he asked something like: if these frontier models are going to be creating so much value, why are they selling tokens and not taking a cut?
It is a very provocative question but it just spilled out of his mouth and then he went on to something else.
is it opportunistic though, or planned from day one? The safety narrative has been there since the beginning
I mean... I'm not even extraordinarily cynical about this stuff, but to me this seems like a totally normal level of corporate gamesmanship?
Companies look for and seek to maintain competitive moats. This is not particularly clever, it's a core part of corporate strategy.
It's also highly unethical (for some values of ethics)
of course, but the safety angle was pushed from day one. I more mean the forethought of how it would play out
Ok but Dario has been thinking about AI Safety since 2016 [1], before even GPT-1. I think the simplest explanation is that the Anthropic folks genuinely believe what they say, it just happens to also help their business a lot.
[1]: https://arxiv.org/abs/1606.06565
Yeah I think this is right. The best setup is when a true belief aligns with a competitive moat.
I definitely believe that (to his credit!) Amodei is a true believer in safety. But I also think it was important for many of the deep pockets investors who have been involved in the company since early on to recognize that this would be a potentially defensible moat.
"True believer in safety" but happily quoting arse wipe Vance? Give me a break...
What was the quote? For what it's worth, I do really think that Amodei believes in and cares about safety. But that is not the same as believing that he is entirely altruistic or above the influence of politics.
That just shows how wrong he's been because there was nothing unsafe about AI in 2016. And the people theorizing about this stuff in the 20th century? I want to see what crazy code they were writing
Is it not better to anticipate problems for a technology so that we can develop theories and techniques to solve them ahead of time?
For instance, Amodei co-authored RLHF in 2017 [1], 5 years before it went on to be used to turn GPT-3 into ChatGPT.
[1]: https://proceedings.neurips.cc/paper_files/paper/2017/file/d...
Does this really seem exceedingly clever and hard to foresee to you? To me, it seems like a pretty standard regulatory capture strategy.
This doesn't even mean that they're wrong about the risks or that they're lying. But surely all the investors understood this factor in their moat.
Who gets to decide what is safety?
I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.
China has different objectives. Sure.
I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".
What do you mean by “PC correctness”? I’d expect the politically correct answers to be the ones desired by the current admin at test time, whoever that is. The current political correct answers would not be very “woke.”
Whatever, doesn't matter. The point is a model should be able to exist and be used even if it goes against whoever got 270 electoral college votes
So, it's a cottage industry.
A lot of the heads of AI labs are talking about this including Dario, Demis and ELon, they are seeking to do a sort of decentralized peer review system, where the competitors have incentive both for self interest and global interest to flag their competitors for actual risks, similar to the Fable situation where Amazon contacted the white house.
The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.
When Fable was yanked, it was said to be (in part) due to the "jailbreak" of instructing Fable to "fix this code" — https://news.ycombinator.com/item?id=48552687
Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.
Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"
It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.
Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.
If everyone has mythos, no one has "Mythos."
Just let people fix their code.
> If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.
It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.
Are you claiming that LLMs aren't finding new issues compared to previous methods?
There's a huge number of security issues coming out in recent months, especially via Anthropic (glasswing etc). We don't have to take their word for it: look at the code. Some open source maintainers are talking about burnout due to spending so much time patching.
They're probably creating way more vulnerabilities than they're solving. Go look at OpenCode and tell me that any of that is sane. Or fuck, the OG of vibe coding, Claude basically is a terrifying attack vector. It's poorly reviewed and open to prompt injection and yet it basically has access to whatever the user has access to on most corporate machines. They can't even fix flickering bugs but somehow we're supposed to trust that they aren't opening our machines up to terrifying vulnerabilities? It's amazing to me that people will just let it run arbitrarily bash commands in their home directory without thinking, but all the sudden act gravely concerned about security in the age of overhyped LLMs.
I do think security issues in core building blocks like curl and the linux kernel (and almost every significant project) are still a concern even if developers are being sloppy on newly-built apps.
This isn't an "are LLMs net good or bad" argument. It's "are they finding many new security issues or not?". If it's the latter, we want to deal with it no matter where the issues are coming from.
(see: https://news.ycombinator.com/item?id=49077452 )
The suggestion was to have some AI system “fix” the code. They are reporting that they’ve found lots of bugs. Are they even claiming to have exhaustively found all the bugs? I don’t think even the most optimistic pitches would claim that.
I’d expect patching existing codebases to be an eternal treadmill as better models come about.
Who's suggesting that? They're sending reports to projects to fix. It's up to the projects on how they fix them.
No, I don't think all bugs are fixed. The point of the project (glasswing etc) was to fix as many as possible in the core software the world runs on before the capability to find vulnerabilities is available to everyone (black hats included). Which may only be a few months.
I do think everyone expects it to be an ongoing treadmill: models get better, find better vulnerabilities, etc.
Yeah, there’s a lot of people that are in the “AI doesn’t work” camp. IDK what to tell them except that they are holding it wrong. My Anthropic subscription (in the hands of an experienced developer) is worth 4 mid tier or 2 top tier devs. And makes better code than the mids. If you “hold it right”.
I think you're describing a strawman. As a proper hater, I know these things have some useful functionality, but most of us don't think "stochastic tool that can do some useful things but also frequently fucks up" is worth two trillion dollars and massive overhyping from the most irritating people on the planet who can't even tell good code from bad.
So you are saying that there are people that understand the value proposition and the utility, but don’t think it’s worth it to society (myself included) but who respond to that by lying about it not being effective? I guess that makes sense. Weird, but humans, so , yeah.
I react to that by leveraging a very useful but probably poisonous to society in the long term because humans aren’t good at having things that make them lazy tooll to try to mitigate the negative effects that it will definitely have if left to its own devices. I don’t see the point in raw resistance at this juncture.
Spending their time patching, or reviewing slop "patches"?
They're not slop, if you're talking about recent ones. You might still be operating on information for a year or two ago.
Here's the curl project talking about the strain they're under from real reports (despite being a mature and well-vetted project):
> A thirty years old project could make you think you’ve seen most things already, but we have not been in this situation before.
> The rate of incoming security reports is 4-5 times higher than it was in 2024 and double the speed of 2025 – meaning that on average we now get more than one report per day. The quality is way higher than ever before. The reports are typically very detailed and long.
- https://daniel.haxx.se/blog/2026/05/26/the-pressure/
---
Linux kernel maintainer Greg Kroah-Hartman:
> "Something happened a month ago, and the world switched. Now we have real reports." It's not just Linux, he continued. "All open source projects have real reports that are made with AI, but they're good, and they're real." Security teams across major open source projects talk informally and frequently, he noted, and everyone is seeing the same shift. "All open source security teams are hitting this right now."
- https://www.theregister.com/software/2026/03/26/linux-kernel...
---
And ffmpeg, who previously complained about slop, 2025: https://xcancel.com/FFmpeg/status/1984220199193891166
Now say serious issues are being found, 2026: https://xcancel.com/FFmpeg/status/2066169070387413147
(I only point out their previous stance to show that they're not coming from pure AI hype.)
This seems a highly controversial post for some reason, judging by the repeated downvotes/upvotes. I'd be curious what I got wrong.
I don't think a one-time $100 credit is enough. First of all, that isn't very much. But also, the volume of new code is going way up. Unless they keep giving out monthly free credits, it's just a stopgap.
I doubt most bosses will give engineers the time. They care about security only to the extent that they have already been harmed by a lack of it. I would like to play with mythos, but on my own time my kids have plenty of activities to fill my time. My personal backlog of projects is only getting longer and none of it is something mythos could help. If I had more time is have restored my old truck instead of making payments on something new (in turn limiting what else I can afford to buy)
> Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
That's basically project Glasswing; mixing responsible disclosure with frontier exploit generators.
They are doing that (see their project glasswing over the past few months), but there's a lot more code in the world than you realise.
The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)
$100 credit on Fable/Mythos will last a grand total of 10 minutes.
I think eventually there will be Mythos grade AI which will be released which can solve a lot of bugs, even right now opus/fable can fix more things which companies can even keep track of.
The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
For starters, it's probably closer to $10,000 per codebase for Fable/Mythos for a full review. That would be around 5 years of their current spending I think.
They really want that level of spend coming into the company, not going out.
I think there is some logic in delaying the rollout, giving it to the heads of the largest software products first to fix their code before dumping it on the general public. But yes eventually everyone will have this tech and it won't matter because the low hanging fruit will have all been picked clean.
> Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
1. Some do not want to use LLMs because of grave ethical concerns.
2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.
3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.
Because this is all kayfabe. You cannot take a single thing these companies or people say at face value.
> why not just let it fix everyone's code?
That's the stated idea. Fix code before releasing to the public.
You forgot “what is the definition of ‘sufficiently capable’”. Presumably it’s anything that competes with Anthropic. If they’re around in a year, presumably they won’t care about Fable level and will only think that whatever competes with Claude 7 or whatever needs to be restricted.
GPT-2 was 'sufficiently capable' according to Amodei: https://explainx.ai/blog/dario-amodei-gpt2-openai-open-sourc...
There are... multiple blog posts online now about how to use freely available data and modest amounts of compute to train a custom GPT-2-sized model from scratch. It would be quite a policing effort to prevent.
What are you suggesting as an alternative?
Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith.
What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.
Well, Dario says one thing and then turns around and sells his model to the US government to use against the entire world for spying and their wars, just like Altman. Just because he does not sound as deranged as Altman, does not make him good.
Now, to what he lays out, its not reasonable, its only reasonable from a purely American corporate viewpoint where the rest of the world can crash and burn as long as they get their billions.
How about we don't trust either of the proprietary shovel salesmen?
Again, what are you proposing for the problem of models becoming increasingly capable and dangerous?
Be specific.
Until models become sentient, it requires a human to execute dangerous activities. Nations already have laws regarding what a human can and cannot do. Let us stick to that.
Crime is already illegal.
> increasingly capable and dangerous
It's a fucking chatbot. The industry can fix their dogshit software, anyone who doesn't can get left behind and outcompeted by those who do, and we move on with our lives.
> What are you suggesting as an alternative?
This isn't something that can be regulated. Plain and simple.
If a dangerous model can exist and is being developed by a foreign adversary then no level of US law will stop said model from making it's way to hardware capable of running it. Even if direct transmission is impossible, it's FAR too easy to shove a model's data onto 1 or more thumb drives or hard drives and smuggle them pretty much anywhere in the world.
The only way to actually mitigate this sort of risk would be a global government with deep enforcement powers. That doesn't exist and won't exist. The UN is the closest we have to anything like that and... yeah...
Dario is fear mongering. He knows his proposals won't be even a minor speed bump in a dangerous model being created and used. His "reasonable" proposals are for the US market only and are literally just to create a bigger moat for his own company. They don't make anyone safer other than his shareholder's wallets. The only people he stops these dangerous models from being used by are people that won't be using them in a dangerous fashion.
What if it was an international organization with stakeholders from various national governments and international bodies?
You mean like the UN?
Possibly it could be organized through the UN
There is no alternative. Why? The regulation is good only for US interests. It will be disastrous for the rest of the world like anything US has regulated (how dangerous that was like "nukes") and a lot of the world again will/might have to live under the American AI thumb, like it did (and many countries still do) under the US nuclear emboldened thumb.
> Everyone seems to want some fairytale world where there are open models
No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).
No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?
Even if the test is run by an independent third party, they can always test open weight models against "finetuning attacks" or similar language which every model will fail for structural reasons.
Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out.
All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...
Add to that the restrictions on even in-house talent being allowed to work on the latest models, we might actually have a brain drain from Anthropic soon which would be a welcoming sign.
Regulation is not a blanket ban. Regulators (presumably government agencies) can review models (of any kind) and approve or ask for changes.
There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.
cryptography, too. remember when that was regulated? strong cryptography had to be carefully controlled as a munition, for national security!
Yes, I remember. But I don’t we should conclude that no program should ever be regulated.
So the argument is now that intelligence is dangerous.
Regulations on forbidden numbers exist, but they are usually not a sensible policy and are not comparable to the regulation of rocket launchers.
Thinking of files as “forbidden numbers” is too abstract a perspective to be useful. What’s in the file and how it will be interpreted matters.
A file might contain malware, child porn, or RNA sequences for viruses.
The goal is to make the safety tests cost $100M+, so that no one can release a model legally useable for a large portion of the world, unless they charge high enough prices, to the point where no one would use it, thus no competition.
Exactly correct. This technique has been used again and again to discourage competition. I was asked was they could have done to encourage competition and I said, "Lobby to make the entity that provided the model unwaivably liable for consequential and incidental damages of its use." That way people who built models pay the price for the lack of safety testing. We both agreed that would probably kill most of the AI market :-)
That's missing the mark, though. Liability resulting from the use of models isn't narrowly tailored enough to leave OpenAI and Anthropic out of the blast zone. There's no carve-out for them.
Wouldn't your proposal also amount to a ban on open weights models? At least for any developer that isn't unshakably confident that no court will ever find their model to have done significant harm?
For this testing to be really effective at stopping "dangerous and misaligned" models from leaking out, you need a mechanism for banning failed models that prevent them from being released in the first place, not just prevent US companies from using them.
The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself.
So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries.
He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.
I'm sorry, but if Dario's goal was to try and get international cooperation and he recognizes that china is one of the countries that he needs cooperation with, then putting in:
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.
Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.
Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.
He cited the Demis Hassabis’s framework for testing.
From Hassabis’s essay:
“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”
Yea and that would last 5 minutes before Corporate America in the current landscape would control it fully, just like the Financial industry.
It's even worse than that. From the article:
> Open-weights models that don’t have dangerous capabilities are a public good
Oh! And, uh, what's a "dangerous capability" according to Anthropic? Let's see, according to their "Responsible Scaling Policy" [1] document:
- Being able to research energy, robotics, or AI is an unsafe capability
- Additionally, any model that's capable enough to be "used widely" by the government must de facto have unsafe capabilities.
They want to ban pretty much anything open-source that's above cat-level intelligence.
1: https://www.anthropic.com/responsible-scaling-policy
Besides, Banning those models in the US does nothing to protect from other actors using them. That doesn't help in any way.
It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.
I'd argue it won't even stop people in the US from using those models. Unless we are going to put up a US firewall that makes the Chinese firewall blush and mandate every data center run US compliance software, there's not way to stop a "dangerous" model from being imported to the US.
This only stops the likes of OpenRouter from selling access to models. That's it. I'm sure an EU or chinese based alternative will pop up overnight (if they don't already exist).
we should turn this around
Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.
Therefore only open weight models can be allowed, since this allows genuine third party evaluation.
>We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.
Don't we already have third party NGOs such as METR which do risk assessments for unreleased, closed models?
Same way they have banned DJI products like camera microphones, technically it's not banned, it just needs to be approved because it has a wireless transmitter, and for some strange reason the US is the only country that hasn't approved them.
Yeah, this is just regulatory capture.
Make the safety tests abusively expensive enough to run, and if you're not a trillion-dollar corporation, you won't be able to certify the models.
Thing is world has learned from the collective past experiences. Esp. with stuff like nuclear technology and nuclear weapons. I hope everyone here remembers/knows shit like CTBT. At least some countries were smart enough to not fall for that in the past knowing what it would mean if they didn't have it and it shows.
Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.
It’s also how the FDA works. Ban new products until they have been proven safe.
I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)
The FDA regulates physical goods. They require literal factories and shipping to get these products anywhere. They can put stops on these products pretty easily. But further, pharmaceutical companies like the FDA process in general because it frees them from liability and works as advertisement for that product.
AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free.
The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.
This would be a disaster. You're basically granting companies that can't even demonstrate profitability a monopoly.
The FDA is needed because people will be directly and significantly harmed by bad releases, before we can notice and react
Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)
Yes, we need a ministry to minimize the possibility of thought crimes.
The evil Superman (openAI) attacks the good city (huggingface) and the city is saved by the MegaMind (GLM 5.2). Usually, the city dwellers would praise MegaMind as the hero, but the story is twisted - the Superman is only "testing" and the MegaMind is too evil to have such powers of saving the city.
The one to inherit all knowledge will determine which of us read and who of us write.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
> this is anthropic advocating for a ban on open weight models
Is the pessimistic view. Their message on safety has seemed pretty consistent to me.
"Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.
> Yeah, this is anthropic advocating for a ban on open weight models.
This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.
I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.
> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution
How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.
Would this restrict downloads of an open model, or publishing?
Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
Isn't it up to the open model advocates and publishers to come up with the solutions for making them safe?
Like, there's three plausible arguments about safety of open models:
1. Any concerns are fake news. Open models will always be safe.
2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.
3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.
I think option 1 is totally out of touch with reality.
Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).
Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?
In order for model safety regulation to be effective, you need everyone capable of producing models to sign on to that safety framework.
That will never happen.
As such, there is no "solution" here.
The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.
There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.
It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.
> This is an ungenerous take
Why should I give a multi-billion dollar company advocating for new regulations in its industry a generous take?
I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.
There is an important difference between criticism and cynicism.
>I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.
"Because McDonald's wants food regulations, we can therefore conclude that all food regulations should be eliminated."
Obviously this would be rather silly.
It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy. We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.
> This is an ungenerous take
I think that's well earned.
This is my read too- if American companies start backing nonsense like this, they'll fall behind permanently.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—
Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
Please stop giving this company money, people.
>The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio.
It's not obvious to me that the US is on an "authoritarian path".
Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there? https://eternallyradicalidea.com/p/the-situation-for-free-sp...
Your comment seems like more of a diatribe than a serious analysis of likely future scenarios.
Prediction Markets are people gambling.
There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
>Prediction Markets are people gambling.
If you think the market has it wrong, why don't you make money by betting against it?
https://polymarket.com/event/presidential-election-winner-20...
>There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
What is the actual per-capita rate of big flashy news stories? Remember that the US has a population of 340 million. One-in-a-million events will occur every day; they aren't necessarily representative.
> If you think the market has it wrong, why don't you make money by betting against it?
I don't just think "the market has it wrong", I think a market is wrong conceptually. It is not an epistemological tool, it's rich people gambling - a money-weighted accumulation of guesses - and I'd rather not partake.
> What is the actual per-capita rate of big flashy news stories?
What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
>I don't just think "the market has it wrong", I think a market is wrong conceptually. I'd rather not join the other degenerate gamblers.
"My beliefs are unfalsifiable"
>What is the appropriate rate of brownshirts murdering political opponents? Which level of kids being nabbed from their homes is acceptable?
Tom Homan, Trump's border czar, also served in the Obama administration. Obama gave him a medal for his deportation work. People like you will frame the same activity quite differently depending on whether you like the people who are doing it.
In any case, I didn't deny that the US had a problem with authoritarianism. I said it wasn't obvious that it was on an "authoritarian path". See for example https://www.npr.org/2026/04/04/nx-s1-5768273/after-minnesota...
I'll bet you yourself would happily justify the EU authoritarianism here: https://eternallyradicalidea.com/p/the-situation-for-free-sp... You seem like the sort of person who has an authoritarian mentality. You'll happily support cops arresting people for saying things online, but if cops arrest people for illegally entering a country, that somehow crosses a line into "authoritarianism". Am I right?
My claim is falsifiable by facts, not by casino odds. Show me the armed abductions aren’t happening. Show me the administration hasn’t threatened neighbors or purged civil servants. Those are the relevant facts. "What does Polymarket say about 2028?" is a Bayesian dodge, not a rebuttal.
The article you linked describes ICE officers killing two U.S. citizens, tear-gassing neighborhoods, and deputizing local police as a "force multiplier" to create a "sea change in local policing". You cited this as evidence the US is not on an authoritarian path. Maybe we have different definitions of "authoritarianism", but I don't see how this helps your case?
From there you pivoted to "but Obama," then "but Europe," then to psychoanalyzing my "mentality." I haven’t defended the EU, or Obama, or any censorship regime. You’re shadowboxing a partisan cartoon because the actual evidence - federal agents abducting residents, your own NPR link - is too uncomfortable to engage with directly
>My beliefs are falsifiable, but not by rich people making guesses.
How convenient that you continually fail to make any statement about what would falsify your beliefs.
>Obama was a war criminal bombing brown kids with drones. I care not for his medals.
Irrelevant for my point regarding whether the US is "on a path to authoritarianism". If you think any sort of immigration enforcement is unacceptably authoritarian, then the US has always been authoritarian by your definition, and the "path to authoritarianism" stuff is rather beside the point.
>It is notable that you quickly pivoted to whataboutism and ad hominems. I have laid out my case, your reaction was to first try to obscure through number games and then to pivot to attacking me as a messenger. Not once did you engage with the substance.
What could be "engaging with the substance" more than asking how common a particular type of event actually is?
Don't tell me about "substance" when you haven't provided a single concrete data point supporting your position--I've provided multiple (NPR link, prediction market data).
All you've done in this thread is shared your own personal feelings about "armed goons" enforcing immigration law. If you're going to make your arguments primarily on the basis of your personal impressions, then yes, your ability to make those assessments fairly becomes a topic of conversation.
Furthermore, the real question of this subthread is whether the US is authoritarian relative to other countries. In which case the activities of other countries (such as European censorship) are relevant to our assessment.
There are, in fact, indices which try to compare levels of authoritarianism across countries in an apples-to-apples way, rather than doing as you do, and forming vague impressions on the basis of viral news stories. Here is one by The Economist for instance: https://en.wikipedia.org/wiki/The_Economist_Democracy_Index
Anyways good luck, at this point I'm confident that you lack the intellectual honesty to change your mind on the basis of anything I might say, so there's no point in continuing further.
Yeah, this response is pure propaganda, say one thing in the headline and the opposite in the body.
Anthropic does not support a ban on open models, except for any models that aren’t closed.
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
> Yeah, this is anthropic advocating for a ban on open weight models.
I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.
It's definitely an attempt to pull up the ladder behind them.
Won't this just incentivize companies to move operations outside of the US where these models aren't regulated?
In the near term thats something that can be controlled. If you buy Anthropics take, then even buying time would be a win.
This isn't actually about safety. This is just another example of pulling the ladder up so nobody else can follow
Not to mention: what are the "safety" standards we should enforce? And how should those standards even be enforced?
Can you list out some examples that you would be supportive of; ie were they listed that you would no longer be (presumably) opposed?
Why wouldn't it be a scan, just as we have with all other open-source code? Why can't open-weight models be easily checked for evil alignment? Sophos, Symantec, Malwarebytes, etc. would surely leap at the chance to upsell you on their product.
Wait, you don't do that already? I don't overcomplicate it, I just run
on all of my source files and keep what's left. Why would you keep bad code around? If it breaks when I do that, I fix it, and try again until I achieve what I want with no bad code. Doesn't everyone do that?because of Godel numbering.
Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?
Godel numbering is banning a number. What are you talking about?
Or the important question: what happens if the model fails this test? Presumably then it gets banned; otherwise what's the point of the test if no action is taken if it fails?
More self-serving trash from the US AI companies, disguised as "being reasonable".
It does seem inconsistent that we currently ban closed models that fail the safety tests but not the open. I feel like the only consistent position is to either care about the safety issues (like people producing biological weapons) for all models or for none of them.
A government agency tests all medications, why not models?
Don’t think government controlling AI is a good idea.
Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.
Government control will be a good idea once we start approaching AI that is actually destructive.
Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output.
If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.
Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?
Who pays to build the open weight models? The cost of training a frontier model is orders of magnitude greater than the cost of safety tests.
...the AI companies? Probably Anthropic itself? I see no possibility of regulatory capture here, so it must be a good idea.
There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
Actually someone needs to make the positive case sufficiently well first.
Yeah this is bad. I'm cancelling my Claude subscription and I'd encourage everyone else to do so too.
I give them $200/month for Max. They give me a massive amount of tokens in return. Every time I fire up claude code they lose money.
It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.
Seemed to've worked out pretty well for Uber
It's a valid business strategy. Also worked wonders for Amazon and many other giants. Which is exactly why I am withdrawing my business from Anthropic.
Also the whole premise of this is basically "US good, China bad"
Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.
Exactly. If you care about AI, simply don't use Anthropic - use open source.
There should be safety testing, but no guardrails that limit models for cyber or bio research.
Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.
And how would you stop people from fine tuning or ablating open models?
Regulate GPUs? Ban general purpose computers?
So, if an open weights model was found to be very dangerous, what - just too bad? One could, of course, design an open safety protocol, written and performed by people in the executive branch, accountable to an elected official.
I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)
> So, if an open weights model was found to be very dangerous, what - just too bad?
Yeah, it's too bad.
I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.
But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.
And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.
Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.
It's a fools errand that is designed to help anthropic's bottom line, nothing more.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.
Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.
These globalists were happy outsourcing manufacturing and software development. They are now angry that they failed to secure their IP, which was already based on dubious extraction of the collective works of human IP.
You can't have it both ways. Either it's fine to outsource to China, fine to use H1Bs to undercut US talent, or, foreign governments are strategically positioning and their espionage is a threat to America.
Booo! "authoritarian hands". Scary
Its not like current US regime is using AI for: 1) Mass military operations across the globe 2) Mass surveilance of its citizens 3) Removing every possible safety guard from AI/climate regulation 4) Stealing data across the world to train its own closed-source models 5) Is openly antidemocratic, destabilizing EU and economy of whole world
I wonder why company which is actively cooperating with the current regime would push such message
US is using as much fossil fuel as possible for this AI build out. China is leading the world in an unprecedented green transition, so their AI usage is overwhelmingly going to come from solar, wind, and nuclear.
To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality
Do you not think that at the rate ai intelligence and ability is increasing, this could realistically change one day soon?
Even if AI gets super smart, it will run into the limits of what we know about biology. Someone will have to do lab experiments to provide more knowledge to the AI. This is different from say building a computer virus or hacking since the AI can do all of these on it's own
> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality.
As an expert, could you also provide your arguments please?
Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
> They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously
So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat".
To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise."
Thoughts?
They are already being trained in biology right now? What he is saying is that they are being trained on what we know about biology currently. AI is going to hit that limit. And there is no way for the AI to gain more knowledge without actually doing lab experiments
My point is more about conflict of interest in Anthropic, and certain techpeople types thinking that biology is a useful scapegoat because biologists don't use or understand this tech (they claimed 0.03% of users would be affected by biotech security stuff). So, arguing the world will fall apart because someone can ask your superduper powerful tool for a bad bad virus sequence, or how to do some technique in the lab, and get an answer that's plausible (oh but you never bothered to actually test if it's legit because you don't have the capability to do so).
As for the future... today the LLMs are "trained on biology", in that they read the textbooks, the research, the web.
They aren't trained on biology in the sense of being embodied, autonomously or semi-autonomously driving actual biological experiments. If you come from software, the timescale of these experiments is outlandish. Yes, I am partly saying the LLMs are not good enough today because they need to be embodied and trained for literal decades of lab time before there is even the _remote_ possibility that they could present a novel risk profile that is even a shadow of what the current fearmongering suggests the current models can enable.
And they aren't trained on biology in the sense that they've read the literature, but even 100T token training run only begins to touch the data scales that rather mundane bioinformatics operate at. True multimodal models that work on DNA and human language at high quality haven't yet emerged. We're talking new architectures which are going to arise after the next AI winter.
All of this ignores an even more fundamental point. Cost. If someone wants to make a bionuke, they don't need to use AI. They can set up the right evolutionary context and run quadrillions of parallel explorations of the design space. Directed evolution like this is cheap, well-understood, and insanely powerful. If you actually care about biosafety, we should be doing hard work to surveil gain of function research. Different flavors of LLM use are not going to be a differentiator for the foreseeable future.
If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.
The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.
Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
This is completely backwards. Cybersecurity has an attacker-defender asymmetry that heavily, HEAVILY favors defenders.
For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.
The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once".
You are suggesting this isn't correct?
> a defender gets to pick the surface area
What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
It's correct but defenders also choose where that happens. 100% of the time on the locations and conditions that the defenders choose / allow. As a defender i need to be right 100% of the time, sure, but i can make it so that the things i have to be right about are very well known to me, unknown to others, maybe even extremely unlikely to be to known by others, difficult to get to know, (...). So that saying is true but over simplifies the situation. I know monkey brain likes simple phrase. But monkey not live in savannah anymore. Need to adapt and open mind to complex.
Today’s surface areas are gigantic and many of them will - in a typical company - not be chosen by cybersecurity experts. How do I hide the physical location of an office that offers physical access to the company’s network? How do I hide which OS the company is using? How do I hide the underlying technology of customer-facing systems? How do I hide which SaaS services I use?
Very well said, thank you! The triple clause here is exactly what I mean.
> "defenders have to be right 100% of the time, while attackers only have to be right once"
If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
They downvote you cus you're right.
This makes no sense. Why do defenders get to pick the surface area? You can be attacked from anywhere.
I think the point you're trying to make is that you can always make your exposed surface area smaller
But that, of course, is not going to survive contact with reality
Not really, the only reason I (or any other programmer) haven't ever hacked into a system to make my life easier (not to do bad things) is because it's illegal.
It was always easier than making a system secure.
And here I thought management picked the surface area.
I know the company I consult for (not cybersecurity) is not in these programs and if attacked would need to use open weight models.
Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
are you working at anthropic? you're literally repeat what that freaking Dario say everyday
> Only by using the open source GLM-5.2 were they able to survive an attack
They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.
If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.
Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.
Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
? There were not models fighting each other, attacker and defender. I dont quite follow what your getting at.
huggingface asked the frontier models to help them analyze the attack and lock down their systems
the frontier models refused because their cyber detector went off
they had to use GLM 5.2 instead
They used GLM to parse logs after the incident. There was no sci-fi AI vs AI battle.
Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.
Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
What's the explanation for why a bioweapon couldn't easily be made? As someone who doesn't have access to micro biologists
So many commenters are asserting this but no one is giving an argument or references. Cults have been able to make sarin. The DNA sequence for smallpox is pubicly available. Where are you getting your confidence that LLM-assisted bioweapons are of no concern?
Not only is the DNA sequence for smallpox available, but since 2018 there's been a well-documented end-to-end synthesis procedure for the very closely related horsepox virus [1]! No LLMs needed. Caused quite a stir in the synthetic biology community back then.
The world has not come to an end, of course, because even with peer-reviewed and experience-driven (rather than hallucinated and therefore dangerous) instructions detailing obstacles encountered during synthesis and how to overcome them, actually going out and acquiring the materials and ability to use them sufficiently skillfully is another matter entirely. Biosecurity is an important topic, to be sure, but what the AI labs have to say about it (or anything) at this point does not necessarily survive contact with reality.
[1] https://journals.plos.org/plosone/article?id=10.1371/journal...
The effective altruism/rationalism/AI xrisk people have always had a shockingly poor grasp on subjects outside computer science, despite their attempts to speak on them. I don't blame the actual biologists and chemists working at the frontier labs for wanting to skim a few bucks off all the money flying around, though! I know a couple who've had not-so-kind words to say about their employers' intelligence.
I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.
They also have a shockingly poor grasp of computer science.
Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.
What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access.
The bio angle is very important here too; in that context the imbalance favors the attackers much more.
I feel like so many people miss what you are saying here. The attackers are at such an advantage because of time. At t0, attackers can go and try and find so many attack angles. These traditional companies (defenders) can't just go to a model and say "fix all my things!" and ship it, way more complex in practice.
> In cybersecurity, a level playing field favors the attacker
Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.
I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
> Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals).
> attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.
It took me a few hours to find some very questionable communities, which in turn gave me access to:
- Ways to obtain cheap guarded-AI tokens that are not linked back to me and with no danger of getting my legitimate accounts banned
- Ways to get rid of guardrails and have models work on things they wouldn't otherwise work on.
The attackers were already in these communities long before I knew they existed, they already had the advantage. Ones with enough reputation probably have access to even more information and tools than I do.
It is true that these communities exist because guardrails were put in place, so yes, it is slowing them down too - as in they can't just put in their CC on claude.com and hack a hospital. But attackers are much better at finding these communities and utilizing resources available there than defenders.
Personally, I don't have any ethical concerns of utilizing these resources when I put them to actual defense, but I know many people that would, leaving them at a disadvantage.
My point is that there's only one guardrail that will effectively contain the threat the models pose, and it's in direct conflict of the big 2's goals - pull the models from worldwide access completely. Strict KYC and all. And it would only last for so long anyway.
I think you are trying to argue that you can limit the open models.
If China is ok with open models being open... they will be. An attacker isn't going to be deterred by a US law saying they can't use them.
I guess my point is that if China is ok with open models, then, the attackers will have them regardless of any laws in other countries. Restricting them, in that case, doesn't seem to accomplish much?
You can at least make it harder by requiring US clouds to only serve models with guardrails, and encouraging other countries to do the same. But yes, the underlying issue is the models being open in the first place. I'm sure if the US wanted to, it could come to some agreement with China about this.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.
But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.
If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.
"Power corrupts and absolute power corrupts absolutely." Lord Acton
> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities?
Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up.
Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal.
I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc). This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before.
Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.
If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.
> If this is really the risk, then we should approach LLMs like atomic bombs
A complete failure at actually preventing non-proliferation.
Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.
unfortunately, the US has incentivized the opposite behavior for atomic bombs and we are seeing moves towards greater proliferation
I would not be surprised if the same incentives are created by the US for Ai
There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it.
Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.
This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.
I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models.
In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.
I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?
Chinese AI companies are already releasing frontier-ish models every other month. Their rate of progress does not seem to be slowing down. Nobody here can stop them from progressing. Not you, not me, not the USA government.
The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.
> There has been ONE successful AI-driven cyberattack
Not according to Anthropic https://www.anthropic.com/news/disrupting-AI-espionage
The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is.
But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
> There is nothing that special about bioweapons.
The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.
Even our normal mad leaders have agreed that bioweapons cannot be allowed:
https://en.wikipedia.org/wiki/Biological_Weapons_Convention
A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?
With current gaps in DNA synthesis screening yes. But this will be improved in the future hopefully.
i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over.
"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
> what should be done about open weight bioweapon and cyber-offense capabilities?
Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
> Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.
> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it:
https://news.ycombinator.com/item?id=49078376
----
And related thoughts on past posts:
https://news.ycombinator.com/item?id=49034988
https://news.ycombinator.com/item?id=48516722
I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.
> I had to scroll past so many cynical cheap shots
I haven't read cynical comments, just ones pointing out that the article is cynical itself.
> addresses the core point
No it doesn't address anything, it is fear mongering question.
> I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns
Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company.
> you need to suggest an alternative plan that addresses these concerns
I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions.
Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1]
[1] https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/ant...
> Is it simply the cost of freedom that we should allow attackers to access these tools?
Bad actors WILL have access. The question is will these mega corps stop innovation?
If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them.
It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.
Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
The difference with open weight is that everyone has access to the same weaponry
> what should be done about open weight bioweapon
Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )
> and cyber-offense capabilities?
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward?
From the WSJ the other day:
> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon...
On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
> Why would they not use the best tools at their disposal going forward?
Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality.
It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that.
> On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.
jefftk addresses your bio thought well.
> The reason they aren't more exploited is there really isn't much to gain from doing so.
This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
> This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
No, it's because the targets are worthless.
You aren't going to be able to mine Monero or run LLM botnets on forgotten cameras in basements. There is nothing to be gained from such targets, soft as they are.
Besides the new defensive AI entertainment makes dealing with wherever those things phone home far easier. Possibly too easy for plebs to be allowed access to.
I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."
(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)
> I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.
No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack
There are a lot of interviews with former cult members around. They had armed helicopters, a testing station in western Australia, produced piles of sarin. This wasn't a lack of science knowledge that screwed them up, they notoriously involved the elite class of Japan - it was a whole other category.
There is no link between AI and bioweapons that makes this stuff any more reasonable than availability of detailed descriptions of nuclear reactors enables us to be purifying weapons grade plutonium in our yards.
> > they used a non-pathogenic strain of anthrax
> No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack
That's a different attack. I'm talking about their 1993 anthrax attack: https://pmc.ncbi.nlm.nih.gov/articles/PMC3322761/
Analysis of the 48 suspect colonies confirmed them to be B. anthracis ... This genotype was identical to that of the Sterne 34F2 strain, used commercially in Japan to vaccinate animals against anthrax.
They used a vaccine strain because they didn't know any better. Even members of the elite can make mistakes, especially when operating outside areas they know well!
(This was not the only thing that went wrong, but several others were also knowledge failures.)
You and the other are both missing the point. That's not a knowledge failure, it's a failure in how your operation is strategically executing. They were essentially practicing, and what did they learn? Change to sarin and even VX, for which they didn't need AI.
AI isn't going to help you get from nonpathogenic anthrax to pathogenic anthrax either. All it might do is tell you to try sarin or VX earlier, but these present different problems.
The idea that there are people in the world wanting to execute bioweapon attacks that are somehow gated by a lack of access to AI is utter hysterical nonsense that should be clearly pointed out as such.
Still, somebody heavily funded this thing for too long, and I very much doubt that we don't have the surveillance apparatus in place today for these things to get unchecked.
Stuff is known but not acted upon for various reasons.
maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust
> what should be done about open weight bioweapon and cyber-offense capabilities?
If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
Was the proof of the Cycle Double Cover Conjecture in the training data?
The threats are BS, but fyi models have repeatedly shown capability to produce novel things that are NOT in their training set.
Sure, based on predicate knowledge.
Too bad. We'll have to deal with it. There is no way to stop the weaponization of models now.
But more people having access to the potential tools for defensive is the best possible scenario.
Every other scenario is worse off for everyone except for those with enough money to do something about it.
> what should be done about open weight bioweapon
The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize.
In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.
And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.
I was referring to non nuclear bombs like c4, artillery shells, and missile payloads
Everything you said could apply to computers many decades ago. Think of the nuclear fission simulations our enemies could carry out!
We'll be fine.
The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.
There's a difference between:
> Something should be done
and
> Something can be done
In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.
The US can attempt to stop those models from being trained in the first place but good luck with that.
The software industry should be ashamed by the number of exploits that ai can find in software. It’s really an embarrassment.
The software has to be built better.
I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software.
It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
I have been, yes. For a field that has engineering in the name there sure has been a lot of critical mistakes.
You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost.
A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place.
I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually).
Right now it’s being used as a bandaid.
Formally verified against what spec?
The one liner leadership keeps asking for.
Every single software engineer in the industry agrees with you.
Every single project manager disagrees.
Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.
True. It has been a race to the bottom for lowest cost as long as the quality meets the bare minimum. LLMs can go through and find all the nails sticking out pretty easily.
There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension.
I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.
The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
>To everyone here pushing for total proliferation of ...
...general-purpose computers
...unbreakable encryption
...unbackdoored communication
...unkillswitched vehicles
...unsurveiled dwellings
>what should be done about ...?
nothing
>Do you seriously want this level of capabilities to be generally available with no guardrails?
yes
What about uranium enrichment?
Requires some pretty tough to get raw materials and equipment, to say the least
Because of regulation, not because they are intrinsically hard to get.
So the argument is basically: This technology is too dangerous so only _we_ should have access to it. We’re the good guys and only we can ensure a safe use of this technology.
Quis custodiet ipsos custodes?
That is in fact Anthropic's entire reason for existence, the belief that AGI is too dangerous to be controlled by OpenAI/Sam Altman. It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
Sam altman, his friends, and the axis of evil is not "everyone on earth"
>the belief that AGI is too dangerous to be controlled by OpenAI/Sam Altman.
I agree with that assessment. But the Dario's jump went from "AGI should not be controlled by OpenAI/Sam Altman" to "AGI shoudl be controlled by Anthropic/Dario", which is definitely a better scenario for him, but not the rest of the world.
>It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
In fact, you can argue that in a world where all countries have nuclear weapons is actually a better scenario than a world where nuclear weapons are owned by 1 or 2 American billionaires/trillionaires, no matter if those people believe they are the "good guys".
>AGI
Everyone in the LLM business just won. This is the entire idea that they want to sell you via this drama..
Note that when the government declared Fable was too dangerous for Anthropic to control, Dario was against that too.
“I don’t know about you, but I don’t want to live in a world where someone makes the world a better place better than we do” - Gavin Belson
> To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed.
Maybe I should’ve expanded a bit on my comment. I didn’t mean “we” as in Anthropic directly—even though they certainly are advocating for restrictions on how to train AI systems by proposing mandatory safety training—I mean the argument that American AI labs are somehow more responsible than their Chinese counterparts.
It’s especially jarring when just last week OpenAI—an American company—accidentally hacked Hugginface when performing safety testing on an upcoming model [1]. If they have the ability to turn off all guardrails when testing out their models—or when selling them to the military—then the safety training is only there for show. If they can pick and choose who should have access to their most powerful model, surely they are trying to act as the world police?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
Demanding "required safety testing" is demanding a ban. Otherwise the testing would be inconsequential, right?
I'm sure he didn't mean just a "lobotomized to be worse than Anthropic products" badge for the test-passing models.
If a ban is the implied consequence of failing his "safety" tests, that means that Anthropic was and currently is advocating for a ban on some open-weight models.
It is not demanding a ban at all. Any sane regulation of AI will involve safety testing. This seems like common sense.
If models that fail the safety testing are banned, then asking for that testing is asking for a ban. That you believe any sane regulation would involve such a ban doesn't stop it from being a ban.
You are taking a hardline libertarian position. We safety test cars, roads, bridges, pharmaceutical. What happens when the unvetted AI model starts teaching people how to make biological weapons, or convincing kids to kill themselves?
The parent is, to my reading, arguing that Anthropic is indeed advocating for a ban. I do not see them making any judgement on whether that ban is justified or not. I don’t think that claim is hardline libertarian.
Are you replying to the article or me? I have not stated any position. The quote is Anthropic's and was relevant to the comment it followed.
My views:
I find testing of SOTA models problematic.
I find not testing of SOTA models problematic.
Neither view on testing is without merit.
The right way forward is unlikely to be as simple as either of those, but some carved out balance between them. And it is likely to change over time.
Yes, to clarify: I was not arguing against you but against the statement you quoted, made by Dario.
Your quote was very relevant, as it highlights the foundational lack of intellectual honesty behind the whole Anthropic statement.
I felt like the grudging "if we must" nature of his acceptance of open weights came through honestly. :)
It is clear he isn't a champion for them.
Yeah, which is ironic considering their track record too. The most recent one is every shared Claude chat has been indexed by Google and is searchable on the search results. Then there was the RCE in their CLI that they had for a year and was silently patched without any sort of disclosure.
There's more to it than just safety: in Dario's psychopathic mind, obtaining advanced AI will lead to massive military advantage, hence "we" must do it first to prevent "them" attacking "us". See where it's going? Suppose it happened, then what? By Dario's logic there are two possible continuations: 1. start a war by attacking first and 2. do nothing and waste the entire effort because "they" will obtain the same capability a bit later.
Not to mention it's simply farcical America of all places would be considered responsible. You could not name another country on earth who could wreak more damage (although thank god this is waning by the day with Trump destroying the empire in charge)
Yeah anything China does is evil. We are seeing what the so called "democratic" USA is doing. Anthropic and by extension USA wants a monopoly like they had for Jets and other influencing technolgies.
I think China building and releasing models to Opensource is a greater good because that is providing equal accessibility to everyone in the world.
By Dario's words authoritarian regime vis a vis China, I think OpenAI and Anthropic are also authoritarian in similar terms.
We are only seeing what they want us to show, they might be creating models which can do more harm.
So claiming that China can do or might do, vs Anthropic will not is just words.
Yeah I agree with the final paragraph that we should have testing agencies mandated world wide for each frontier model testing.
Oh no! The evil CCP is a huge threat to world peace and goodness! Give all your money and input token data to Palantir to support a rules based world order where the good guys thrive and cleanse the earth from crooked turtle biologists.
https://www.theguardian.com/world/2026/jun/20/mona-khalil-tu...
Like half of the world are now struggling with energy prices due to an unprovoked war started by U.S. and Israel and somehow an American billionaire is here criticizing China of being a threat to world peace.
It's kinda gross.
AI should not be centralized and should be distributed. There can be no 1 provider to serve all of our needs. We would require both Open weights and Closed weights models for a lot of our use cases.
Open weights models can be leveraged to optimize the cost of Closed weights models. Open weights models can be leverage to defend cyberattacks as HF has shown. Closed weights models can too act as a better cyberattack defender provided separate subscription exists for those.
More efforts are required on LLM distillation for several edge cases. LLM weights should be optimized and compressed to run on edge devices (K3 on Pi3 :). Distillation should be seen as a cost optimization strategy rather than as a competition. You cannot prevent a teacher from teaching to students. If not from teacher A, I will learn from teacher B, you cannot prevent my continuous learning.
Every single risk he identifies as a concern regarding China is exactly my concerns with the US having absolute control. Literally the exact same concerns
They're Americans trying to appeal to other Americans. Of course everything they do is justified and everything everyone else does is an evil scheme to plot the ruin of American freedoms. How else are you going to convince the dementia patients in charge of the country?
If China, India, the EU, and everyone else has any sense, they should take this letter to heart as much as Anthropic wants the American government to.
It's projection. These tech CEOs know what they are doing and it scares them that someone else might do the same.
its not the tech CEOs its the financial interests behind them (they own all the AI companies)
[citation needed]
Yeah. Exact same concerns here as a non-american. Their "national security" is a constant threat to the rest of us.
Yep. Sick of reading stuff that operates under the assumption that America is the faultless good guy and the only one to be trusted with the most dangerous and capable technology.
Among many other things, the Trump presidencies (and, to a lesser extent, the presidency between them) are examples of the the highest levels of leadership being totally incompetent, and have destroyed the above assumption for the rest of the world.
What Dario misses time and time again, is that people don't trust the US to create aligned AI anymore. His entire strategy rests on the assumption that the US (and their government) are exceptional.
This is clearly false to the rest of the world.
I mean in the Bloomberg interview he has made it very explicit that he fully believes in American Exceptionalism. He literally said AI being involved in bombing school girls in other countries are ok because he trusts the American military leadership.
According to him the safety and morality rule of the whole world should be written by America alone.
Which is why in the same interview he said he supports the U.S. foreign policy while calling China "an aggressive and war mongering regime".
>This is clearly false to the rest of the world.
It's clearly false to more and more Americans too. But since the oligarch class benefits first and foremost from U.S. government policies the propaganda will continue to go on.
He did not miss, it’s deliberate choice
then we will make them trust us
I guess we will continue to bomb any nonbelievers huh
The most American thing I've ever heard lmao
trust is not made but earned, and it's a lot harder once you've lost it
Such a cop-out. Dario, you got in the news because you were trying to say that Moonshot did something wrong by distilling Claude. You got in the news because you were trying to effectively make a "rules for thee but not for me" when you try to claim that you can train on whatever pirated works without any permission from the creators, but when someone uses your "work" to train without permission then all of a sudden it's a moral injustice. You can't have both.
Saying, "I'm not actually against open-weights, I'm against distillation" isn't addressing what made people mad. You're still trying to do some "rules for thee but not for me" nonsense and hiding behind some technicality. Trying to get the US government on your side to hold back your Chinese competition. If you had wanted the US government to support you, you should have let them make autonomous killer robots with Claude brains. They aren't going to help you, you didn't help them.
Just to be clear, I think that it is possible that literally everyone involved in this is full of crap and nobody is good. Dario and Anthropic are full of crap, for the reasons previously stated. The US government is full of lots of crap and should not be trying to make autonomous killer robots (not ever, but especially not when the bar for a "good" AI is knowing how many Rs are in strawberry or whether you should drive to a car wash). OpenAI is full of crap by signing some support for open weights models and they haven't touched open weights in a year (GPT-OSS released on Aug 5 so basically a year with no news). Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
So cheers to Moonshot and Qwen and whoever else. Distill as much as you can and give us cheaper AI. I have the sneaking suspicion that a bunch of my tax money went to OpenAI and Anthropic in some shady way or another, and I want it back. I'll take it in the form of an open weights model being distilled from the fat cat models.
> Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
Google catching this stray made me laugh, ha.
Out of all the companies that signed the open letter that Dario references, Google and OpenAI feel like they did it to poke at their competitor. I probably should have taken shots at Microsoft and Meta as well for not really supporting openness, but they don't feel like serious competition in the LLM space at the moment.
Google is actually kind of a sad story. They’re focusing their entire company on LLMs, their models lag behind open weights, they still somehow believe that they’re going to get to a place where they’ll have valuable IP and it’s becoming increasingly clear that there will be no such thing in the LLM space in the future. Greek tragedy.
Dario doesn’t realize that by not offering self-hosting of closed-weight models and fine-tuning, alongside overly strict refusals for legitimate needs, he ceded this corner of the market which grew into a flourishing Chinese open-weight model ecosystem.
If he had wanted a weak open-weight ecosystem, he should have had Anthropic cater better to those needs. And now he's trying to ban them.
The strong momentum behind open-weight models from Chinese labs is now an unstoppable force. Instead of trying to ban it, Dario should consider a different approach: here are our cyber and bio alignment datasets and here are our RL recipes for making that alignment training work well. By openly sharing its data and code, Anthropic could help influence and shape these models before they are released, rather than treating the entire ecosystem as an enemy.
Cyber and bio alignment aren't Anthropic's competitive advantage, they are forms of risk management. There should therefore be little reason to keep this work private. If Anthropic genuinely believes these capabilities pose serious global risks, the more productive approach would be to welcome collaboration and help the broader ecosystem manage those risks better.
On refusals, the irony is that a company like Hugging Face had to use a Chinese open-weight model to fend off an illegal hacking of its platform (done by no other than OpenAI). If a company like Hugging Face can't get past the refusal gates, then everyone else doesn't stand a chance.
> Dario doesn’t realize that by not offering self-hosting of closed-weight models and fine-tuning, alongside overly strict refusals for legitimate needs, he ceded this corner of the market which grew into a flourishing Chinese open-weight model ecosystem.
As I read more of his unhinged posts and some of the more ridiculous claims from anthropic, it's become clear to me that Dario thinks he can leverage American hegemony to regulate his company into a monopoly.
He has an ethics vaguely influenced by effective altruism, and he appears to think that his ethical framework entitles him to make decisions on the behalf of humanity, for all of humanity.
Literally a bond villain.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
Nah, the statement is the mechanism for a ban. The proctor will be someone anthropic trusts and "surprise" as it turns out all the open weight models fail or aren't eligible.
Imagine the Dieselgate levels of adaptation they'll do while being tested too.
"mandatory safety testing" is an impractical ideal, it's not workable in real world. Like any technology, LLMs are dual-use tools capable of both beneficial and malicious applications—a fundamental reality that human intent cannot change.
If a model fails the test, it should be banned. He is not advocating a ban of open-weight models. He is advocating a ban of models that fail mandatory safety testing. Seems reasonable and straightforward.
Can you think of anything open-source that has to go through mandatory tests to be distributed and still survived ?
There is a reason to it, that's as good as any angle to find why IMHO.
I think Gemma will be fine. Most open-weight models are not capable enough to be dangerous. Yes, I can't think of any capable open-weight model that would survive reasonable safety testing.
Any sufficiently capable open weights model would fail "safety" testing though, as any "safeguards" of the sort Anthropic likes could be removed. That's just another way of saying they want a ban on capable open source models which would contradict their earlier statement, or at least make it very misleading. It's hard to see how this post can be internally consistent without some hint from Dario about what he believes should happen to models that fail safety testing and/or how capable open weights models could possibly pass a safety test of the kind he proposes.
I agree we don't know how capable open-weight models could possibly pass any reasonable safety testing NOW, but that's about currently abysmal state of AI alignment research, not about what is possible in principle. I don't see any internal inconsistency, to be honest. Since Anthropic does not release any capable open-weight models, it's not their problem. If mandatory safety testing is established, companies who want to release capable open-weight models will work on AI alignment research so that they can pass. This seems to be a good outcome to me.
OK that is a position they could take but my point is that's inconsistent with "Anthropic has never advocated for a ban on open-weights models". What you're describing is a ban on capable open-weights models until some future time.
Yes, I agree that Anthropic is advocating a ban on capable open-weight models until reasonable AI alignment research advance happens in the future. In return, I hope you agree with me that Anthropic has never advocated for a ban on open-weight models.
I do not. A ban on capable open-weight models for an indefinite period of time falls into the category of bans on open-weight models. If you wanted Anthropic's statement to be true you would need to qualify "ban" or "open-weight models" in the statement, e.g. "permanent ban" or "safe open-weight models".
Edit: Anthropic clearly intended this statement to deflect criticism, but in order to achieve that goal they stretched too far and made a statement which is false. Furthermore, I argue that "open weights" implies an ability to modify model behavior, just as "open source" implies an ability to modify software. If for example some mechanism was found to share floating point numbers that are encrypted in some way so as to allow running a model but disallow behavior modification, that model would not be "open weights", in the same way that releasing obfuscated source code that can be compiled but is designed to resist modification would not qualify as an "open source" release. So I don't really see how any capable model could ever be both "open weights" and "safe" under Anthropic's preferred testing regime, regardless of future research progress.
I think we agree on all specifics now and just fighting for terminology. Thanks for the discussion!
> He is not advocating a ban of open-weight models
He is though. He wants open weight models banned that do not pass some set of tests.
And what does "safety" mean here? We constantly see these companies treating NSFW content as "unsafe", despite the fact that its not. Is being able to produce adult content going to result in a model being declared "unsafe"?
He is not advocating for banning models per se but the proposal makes a business model (i.e. serving open weight models) that is starting to work more expensive.
Agreed, and that serves Anthropic. It seems unproblematic to me. Dario probably sincerely believes in mandatory safety testing for capable models (open and closed), and likes the fact that it aligns with Anthropic's interest.
abliteration and fine tuning makes it not so straightforward
https://huggingface.co/blog/mlabonne/abliteration
I know, but "we don't know how to make it not dangerous, so it should be allowed to release dangerous things" is... not convincing?
My point is that advocating a de facto ban on capable open source models is inconsistent with Dario's statement here that "Anthropic has never advocated for a ban on open-weights models." Call a spade a spade.
De facto ban on capable open-weight models doesn't seem inconsistent with Dario's statement to me. One, it is de facto, not de jure, and it can and will change as AI alignment research advances. Two, it is only capable open-weight models, not open-weight models. In fact, Dario says non-dangerous (which for now is mostly non-capable) open-weight models are a public good, and I agree.
How do we define "capable"?
Is Kimi K3 capable? It's already out and being run by US companies on US hardware in US data centers.
https://huggingface.co/moonshotai/Kimi-K3
That is a difficult question I am not qualified to answer, but Mythos 5 was export controlled for a brief time due to its cybersecurity capability and implications to national security, so for cybersecurity "as capable as Mythos 5" seems to be a good baseline. I wouldn't know for biosecurity though.
UK AISI preliminary evaluation suggests Kimi K3 is not capable enough for cybersecurity in this sense.
https://www.aisi.gov.uk/blog/preliminary-assessment-of-kimi-...
I would be hesitant to extrapolate from this analysis
https://exploitbench.ai/#honest-limits
There is no movement on global policy or enforcement. One country banning their people access to the best models hinders their people.
I am unconvinced that "this can be used dangerously, therefore we must ban it" argument. The OpenAI/Huggingface, needing to turn to Chinese open weight to defend themselves seems to support the case that we need open access and freedom to compute as we see fit.
Hm, I interpreted it to mean they are more worried about loss of control/misalignment risks rather than misuse?
if a model fails the test, you don't give the public access. you let the government have it for 10-100x the usual token cost of course!
What happens if a model passes the government tests and then later someone fine tunes it to behave differently, without making their changes public?
Any reasonable safety testing should include finetuning and safety margin to account for others may do better finetuning.
I can fine tune significant behavior changes, there is little model developers can do to prevent this (aiui), so this effectively becomes an blanket ban
Yes, I agree it is effectively a blanket ban (above some capability) for now. I hope AI alignment research advances in the future so that it is not so.
a ban is effectively impossible without a global treaty
the current US admin as pulled out and worked against all sorts of global treaties, agreements, and negotiations; sending the president's friends instead of experts; who's going to trust us?
> My primary concern is the risk that authoritarian governments...
Authoritarian government doesn't always mean bad - look at Singapore
What's more dangerous is country with bunch of war mongering lobbyists who can also influence elections (oops, sounds like USA)
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
But you are working with DoW and Palantir, who is doing somewhat similar in other countries
> We should not sell powerful chips or chipmaking equipment to China
Israel used banned weapons against Lebanon and Palestinians, would you support similar ban to Israelis?
> We should crack down on industrial-scale distillation operations
Should we also ban distilling public knowledge? Like using textbooks to train the model? Should rules be simple: train your model only on the data you have produced by hand?
(comment got too long, continuation)
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Why? And how do you design those tests?
For example, bombing girls school in Iran - is this allowed use according to you or not?
If not allowed use, then how do you guarantee that you don't have a separate agreement with DoW which makes it allowed use and only your model passes it?
This isn't Twitter. You can write long comments here.
thanks, its not about capabilities, but to respect the human reader who decides whether they should read the comment because its too long or to skip
Bombing that building was an accident due to it being on what was part of a military target and I'm not sure if it was ever made clear whether the building was technically dual use despite also being used as a school. Either way, bombing a bunch of school girls wasn't intentional and nobody reasonable would assume it was.
There was a time we bombed a bus or van with kids in it, but we admitted it and apologized for the mistake. Nobody wants to be bombing kids, first because they're innocent, but second because there is no military advantage to it since it's bad PR.
Many of these targets were identified before Anthropic or OpenAI even existed.
If it wasn't intentional and accident, did anyone apologize for that mistake?
Also the point is, you (Dario) can't claim morality, when he is fine doing business with entities literally bombing and killing human beings in other countries.
You are either fine with it and continue working - which Dario is doing
Or you say, I will not work with you.
For Dario, main thing is money, everything else in his article is bs
Cars can be used to escape, or to attack. Guns and knives can be used to defend or to attack.
AI has multiple uses. Military attacks can also save lives. As they say, the best defense is a good offense.
The moral element of it largely falls on the people who made the mistake. That said, it is also widely known that civilian casualties are a part of war. The advent of precision strikes has greatly reduced civilian casualties.
Meanwhile, Iran has intentionally promoted the direct targeting of civilians and killed 10s of thousands of their own, while funding proxies that killed many civilians elsewhere.
So now you are using famous knife argument.
In your argument have you considered: selling a knife knowingly to gang members just before they are going to another area to kill other gang members
Anthropic did same when it agreed to sell it's tool to DoW. There is no mistake, no misunderstanding of intentions.
He was super clear that he doesn't give a dime to any lives outside of USA, inside USA he was concerned about automatic weapon usage and surveillance of US citizens, because he himself could be accidentally killed, he doesn't care about others.
That's not accurate. AI can help the military increase precision which can also save civilians lives in other countries where the military is active. AI is not intentionally targeting schools the way Russia is in Ukraine. People are going on about this one single accidental school strike in Iran, meanwhile Russia has hit thousands of schools in Ukraine. Something like 1 out of every 7 schools has been damaged or destroyed and even though the numbers are tracked separately, they have killed something like ~800 children.
A similar argument was made for why Microsoft shouldn't sell software or services to border patrol or ICE. The counter-argument was that if it helps them be more precise and reduce errors in their managing of all the people then it could actually help not just the organization, but also the people.
Again, whether it's human or AI, mistakes will be made and civilian casualties will likely remain a part of war. AI can ideally keep civilian casualties low.
> AI is not intentionally targeting schools the way Russia is in Ukraine.
Are you sure?
https://en.wikipedia.org/wiki/AI-assisted_targeting_in_the_G...
They even have the cool one: "Where's Daddy?" - which tracks the suspect and then notifies the officials when suspect is at home, so IDF can bomb the whole building.
In this case AI is specifically designed to increase the civilian casualties, why not shoot the suspect independently, why include their neighbours, imagine your neighbour in 10 floor apartment building is a terrorist and your building gets bombed because of single person
It ideally can, yes, but if you choose to partake in the military operations by providing your LLM you at least bear some responsibility. When you don't give the DoW access to your LLM it may still be the case that actions will be taken that inflict some humans suffering. In that case you won't have had a hand in it though.
> concerned about authoritarian governments
> praises Trump's administration and Vance in his letter
sure there, buddy...
"we have not and are not advocating for a ban on open-weights models" but we want to decide if a model can be released or not and who in the world can build these models ...
they have been very clear they don't want to be the ones deciding
Who is in the best position and has the necessary knowledge to make this kind of evaluation?
Who is trying to define what is "too dangerous" or "not too dangerous"?
$regulatory_capture
https://en.wikipedia.org/wiki/Regulatory_capture
On processing power, copyright, and capability.
I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV.
Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not with Anthropic knives(tm).
Edit: spelling
A key point I feel that’s missing from the discourse is that alignment/safety is basically an impossible problem as of now. Even the “guardrails” that these closed-weights frontier labs set in are laughably primitive which can provably be broken through.
The experimental part of Deep Learning has really outdone itself and is far ahead of theory. We have very little understanding of why these particular architectural choices work. The only “safe” way forward is to stop all development until theory catches up, but that’s never happening.
Dario, as always, is so deeply in the middle of a morass that he helped to create that he doesn't seem to understand how geopolitics currently operates. He also assumes that just because he's from the US that he is somehow automatically more trustworthy than <insert "evil" country here> is. This blog post is a political document geared towards further regulatory capture and the furtherance of major sources of revenue for his company.
I'm not convinced that he is at all interested in the social or existential effects that AI causes. He is a greedy bastard who has taken more VC money than god to do this with. He has zero moral leg to stand on, IMO. He gave that away ages ago and I wish this technique didn't work as well as it does.
You can put lipstick on a pig, it'll still be a pig
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
I have no idea what to do about the government interference. There's probably not a lot anyone can do.
However, your last point is quite a strong one. Corpos aren't just going to stand there with their collective pants down, and there's not a lot anyone can do to stop them from protecting themselves. There are ways they can get what they want without getting caught.
Remember when the US tried to ban strong cryptography in the 1990s, and how well that went? They may have more leverage with AI because it's a bit harder to hide large scale computing usage, but I don't think it's impossible at all.
What do they even really mean by "safety"? I mean, I can have an Anthropic model do something incredibly unsafe if, for example, I put it in charge of a hydroelectric dam and don't explain properly how the controls work. On some level, everything is simultaneously "safe" and "unsafe". I've never found Amodei's reasoning here to be particularly well thought-through. I think he, like a lot of folks in the area, are starting to realize that they may never be able to build a moat around their businesses.
It's political doublespeak. They want to have their cake and eat it too.
I don't really understand how they can argue the security angle with a straight face. It's not like GLM 5.2 is a slouch. I've seen it do things like exploit an IDOR issue when I was experimenting with a quick-and-dirty web automation task. I simply fixed it, as one does. Open models make the world better to a far greater degree than they set it aflame.
Their position is analogous to trying to, say, ensure digital privacy for everyone not by making encryption freely available (because that would let the bad guys use it!), but by making it so you can't use general purpose communications devices that can listen to transmissions not intended for you. Do they hear how moronic that sounds?
Each passing frontier-level open model release makes Anthropic's patronizing rhetoric a little more insufferable, because it becomes clearer how unmoored from reality they've become in pursuit of profit.
> an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China.
HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly.
> these statements are counter-factual.
The OpenAI incident is a single example. You're massively overgeneralizing. You can't refute an entire class of possible outcomes based on a single event where it went the other way.
I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise.
Edit: just to clarify my position, I don't love Anthropic so much. I think they're marginally better, but I'd still like to see regulation strangle everyone so we get another 20 years to figure this shit out.
"HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly."
HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.
fwipsy is right, cnbc has a story on this. they only had fable. I still think this is horrible for closed source, get on a list or else, but i was wrong
"You can't refute an entire class of possible outcomes based on a single event where it went the other way."
But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.
The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China
"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."
We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought
I assume you mean https://huggingface.co/blog/security-incident-july-2026. It says that they used frontier models, not frontier cybersecurity models. My reading is that they asked Fable and it refused; if they'd had access to Mythos, it would have helped. You're mixing the two but they're NOT the same model.
Source is here: https://thezvi.substack.com/p/more-on-an-internal-openai-mod... ctrl+f "Skill issue." No source is cited, but I'm fairly confident it's correct. If Mythos/5.5Cyber specifically had refused to help, then HF would have made a much bigger deal out of it. The whole point of these models is that they have relaxed guardrails and specialty cybersecurity training relative to the publicly-available ones.
> zero events
What about all of the vulnerabilities already patched under Project Glasswing?
In the quote you provided Amodei is expressing uncertainty, saying we don't know which way things will go. You're the one making strong assertions; the burden of proof is on you.
"My reading" ... "No source is cited, but I'm fairly confident it's correct"
Regis, what is demanding proof while literally making things up and ignoring what actually happened?
Great, i was wrong!! Thank you, I was genuinely asking for a source in my first reply, and then you hit with "My reading" and saying it was a "skill issue". I'm not going to have a productive dialogue with someone talking in memes and being rude
The point to be made: closed source AI refused to help them fend off an attack form another closed source AI. What is the argument for closed source here other than hoping you get on some program wait list? Either way, I appreciate you correcting me; I am not trying to "win".
I apologize; it was lazy of me not to find a source. This one specifically says that the model was Fable; does not mention which model they attempted to access on the OpenAI side: https://www.cnbc.com/2026/07/24/chinese-ai-model-openai-cybe...
Seems a little hypocritical since you were confidently asserting that it was Mythos/Cyber5.5 also without proof.
Edit: Thanks for correcting the record in your upstream comment. I appreciate it. For the record, I was not trying to meme on you; that was the phrasing used in the original article. Just another reason that was a poor choice of source I guess.
Amodei isn't ignoring reality; he's just proposing a different solution to the problem. If it were one of Anthropic's models, then that would be a much stronger case.
Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable.
I don't think you and I need multiple different threads open when we are clearly at odds. This is no different from our other thread, I think it is clear there is nothing of value to continue when I am getting pinged with a summary of your previous comment in a different place
"Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable."
Which just means that they're fucked when closed AI hacks them. Something that has actually happened. This isn't argument against anything other than reality. Have a day
GLM 5.2 didn't defend them at all. It only helped with the postmortem. HF was fucked either way. The only thing that will really prevent this is tighter restrictions on the attacking model. We need policies which asymmetrically help defenders; that means regulations. "Give everyone the best models without restrictions" is the opposite of that.
I'm sorry for splitting into two threads; I understand if you need to step away from the computer for a while. To be honest, I should probably do the same.
I did need to walk away that speaks more to my frustration with certain forms of communication (online, not anything in this thread). I am a horrible remote only worker because of this, I am trying to improve it but am lucky for now as I am in-person
You're right again about GLM 5.2 being purely post-mortem, I didn't realize that till I read the cnbc story. OpenAI, whatever they have, cracked em like it was nothing. Egg on my face, I really need to read my own articles better. Thanks for following up and educating me on this, another good reminder that I need to improve my ability to steel-man written text
Hey, it's all good. Sometimes things get a little heated, but I still feel like you were basically engaging in good faith. I should have skipped straight to the point and found the source in my first reply.
Also, I probably overstated my claim a bit. I did some searches and I see only small-scale AI uplift for cybercriminals, even though my understanding is that open models aren't typically hard to jailbreak. Of course this is may be a result of today's guardrails; it may be that it just hasn't been caught, and it may appear later, but it still weakens my argument a great deal. I guess my support for AI regulation stems more from fears over long-shot bad outcomes (biosecurity, who knows what else) rather than cybersecurity specifically.
This argument might make a modicum of sense if the US hasn’t spent the last 2 years establishing itself as a bigger enemy of the west that China.
> We should not sell powerful chips or chipmaking equipment to China
This is so short-sighted given that the US needs China equipment for.. everything. They are part of the supply chain needed for building the machines that build these very chips.
Jensen was right. We should have sold chips to China so they'd be depenent on us.
Now they have their own chips and most of Nvidia product line is internally banned.
Why would it be a good thing if the world was even more dependend on the USA?
US hegemony is one of the biggest problems of our time.
It's unlikely that China would've become completely dependent on us in either case. They're good copycats, with incredible talent in engineering and manufacturing. They're aware that we depend on them for a lot - I think they'd be similarly aware of the risk of becoming dependent on us.
> Now they have their own chips
I never understood that argument, are you saying Chinese companies are not going to build their own chips if they get access to Nvidia chips?
They were, just not as quickly. The export controls directly accelerated their development.
The general rule is: USA bans China from having thing, they make their own version of whatever that thing is. USA bans China from the ISS, they make their own space station. USA bans China from having ASML, they make a Manhattan project to clone it, the "20 years behind the west" line is history. They ban GPU exports, they just start making their own GPUs.
I gotta respect the chinese. I wish my own country had the balls to do this.
Seems like the maximal position he could take compatible with his expressed principles. There’s no way to allow for bioweapon and cyberweapon grade models being open weight if one doesn’t want widespread human damage.
So I cannot disagree with him on the idea. It’s only a matter of degree and whether we’re already there or not. I have $50k in GPUs that incentivizes me to believe we are not.
This is where I'm at, or rather, will be. I like open-weight models and I've done my part in facilitating them, but if you believe in the increasing capability of these systems - and I do, to some measured extent - it seems plausible to me that an incident will happen at some point in the future.
I don't agree with his argument as a whole, especially not on some of the specifics (it is not great that this technology is being developed under the current US government), but I am sympathetic to the idea that some bells can't be unrung, and thus we should proceed with caution.
I understand the arguments for Anthropic barrelling ahead while simultaneously advocating for pauses and regulation. I also understand how individuals can desire a slowdown but have good reasons to keep working at an AI org.
But if everyone thinks this way then things continue to escalate and nothing changes, waiting on a consensus that may never come. And always there is the economic incentive that pushes all players to rationalise continuing.
I wish there was more concrete action from the inside. When decisions get too hard to calculate you can always fall back on basic principles. If you think AI is developing too fast, stop developing it. Now you're no longer contributing. If an AI company wants a pause, pause. Set a good example. Maybe others will even follow suit, and they'll look irresponsible if they don't. Let he who chooses to no longer sin put his stone down first.
How about;
1. Make anti-distillation clauses illegal to strenghten western opensource eco-system.
2. Make cyber-defensive models widely available (can detect but won't operationalize vulnerabilities). Otherwise make Fable awailable for everyone. Keeping the cybersecurity in assymetry by withholding cabailities just causes more instability and increases the incentives for powerfull close sourced models. This is a loss for everyone.
"It is difficult to get a man to understand something..."
Hate to break it to you Dario but the way things stand right now the world at large trusts the Chinese establishment a lot more than the American one.
Reading some of what Liang Wenfeng said on the investor call a few days ago gave me hope for humanity. I really want to believe AI is going to be a tool for good, not going to just enrich a few billionaires again.
I cheer every time there's a new Deepseek, Qwen, Kimi, Z.ai release. Intelligence shouldn't be owned and charged for by a few
If it were up-to these silicon valley tech bros, they'd find a way to meter and charge for the air we breathe.
The way this reads, it seems like it was written for US lawmakers.
> the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This comes off a bit hypocritical, coming from one of the men responsible for the models most likely to be used by the US military, and for repression of US citizens.
> We should crack down on industrial-scale distillation operations ... We should have policy interventions to deter this behavior.
How exactly would you design policy interventions to stop distillation? If Anthropic wants to make their products available around the world, you would need some kind of global regulation to curtail it. And you would need to find some way to enforce it, which is far from trivial.
Honestly I don't see how securing models against distillation is up to anyone but Anthropic, if that's something they want to achieve. Calling for regulation is a bit like crying to mommy and daddy when things aren't going as you would like.
--
Overall, I can understand the argument that advanced AI models can present risks. But I don't see how regulation within the US can mitigate any of those risks. Any bad actor would be able to access the models outside the US, or covertly access the weights.
The only real way to prevent advanced models from being deployed would be to go around the world bombing every significantly powerful data center, and I don't think Dario would call for that any time soon.
The only thing this kind of regulation would achieve would be cutting US companies out of half of the innovation happening in the AI space, putting the US at a disadvantage relative to the rest of the world at everything except maybe frontier AI development.
"without US chips" is an interesting framing, considering most of these chips aren't made in the USA? Mostly it's made in other asian contries(Taiwan, Japan, South Korea), or am I wrong?
Probably refers to US design rather than manufacturing
The concerns are legitimate but the proposals are nothing more than a stopgap solution.
If US wants to maintain engineering superiority, we needs to invest in it -- education, research and infrastructure. Bring in top researchers across the globe and not make it harder.
China is building infrastructure for the future generations and investing in growth sectors while the US is cutting of university grants and spending billions on a war without clear path to resolution.
Who decide the if a model is safe to use?
The strongest competitors or a government law? I prefer the second.
Suppose Chinese model are super-risky: is it better to have them in the open, so everyone can review the problems or have a closed-model?
Security is better at open Linux is open source because development is better.
I understand Anthropic for sure, but trying to limit open weight model seems the opposite direction IF security+safety is the major concert.
The Genie is ALREADY outside the bottle, and there are already plenty of companies offering inference services
Surprisingly incoherent for Anthropic and Dario (cue peanut gallery — “always has been!” No, I don’t think so. I think this is new).
It seems to me like there is just no good answer to how one could possibly stop open weight models from being used for nefarious purposes. How are you going to enforce guardrails on open source? The only way is to turn the USA into a 1984-type totalitarian surveillance state (even more so than it is). Unable to say that, we just get this floundering instead. How long is not giving them chips going to slow them down? Until we RSI? Then what? Just because RSI runs off the exponential doesn’t mean that the eventual open-weight Moonshot Mythos won’t be able to make bioweapons. Genuinely what is the endgame.
Well,maybe try talking to people and understand that not all seven or more billion people on the planet must conform to the interests of a handful of english speaking capitalists.
Instead of bombing them, try justice. Yes, it is more complicated. And yes, you will have to give, and not take.
"open weight" models are not open source. They are still deeply proprietary. It is not possible to know what they do or what they are capable of without interrogating them since we have no access to their source materials.
The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
China has published more research papers than America with regards to AI training and inference optimizations and methods.
You are correct that they aren't completely open source, but the alternative is the American method, getting drip fed a ChatGPT OSS model every 12 months which cannot do basic programming.
>The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
I'm not sure this is entirely true either. Kimi K3 was released and for two weeks existed only via Moonshots API / Subscription. Openrouter reports 250B+ tokens a day for 11 days straight. I would assume they are processing over 1T tokens a day if you include direct API and their subscription.
Keep seeing this trope again and again. By same logic are open source code "open source"? If we don't see the development environment attached to these projects, are they truly open source? Heck, why not include software philosophy and methodology and the Jira board including the book that inspired the architecture?
The argument is an endless slope and as such essentially pointless.
It would be awesome to have actual open source (and open weights) models! I hope someone will make a real go of this at some point.
https://allenai.org/ is what you're looking for
Yes, I've seen it! It's exciting, but I don't have enough information to say whether they are making "a real go of it". That is, it's unclear to me whether they have the tens to hundreds of billions of dollars necessary to create a frontier model.
I...don't follow. How in the world does your comment about open weight and open source relate? Ignoring that it has nothing to do with this post (did you mean to reply to someone), are you aware that Anthropic, OpenAI and others allow 3rd party inference providers to run their proprietary models? Like, what does this non-sequitur even mean?
You understand Moonshot AI could have had other parties run inference for them without releasing the weights, right? These two points are utterly unrelated, unless you think Fable and GPT5-6 are also "open weight" because other providers are providing inference?
Further, having access to the source material in no universe allows you to know what a model is "capable of". I'm not sure how this follows.
Anthropic/Dario getting a lot of hate in this thread, but the argument he makes is rational if you believe for one second that AI is important to national security and that it can, or will soon be incredibly dangerous is misused. People pointing out that this position benefits his company might be overly reductionist in how they think the world works. I believe Dario and Anthropic about their belief that AI safety is paramount because they put serious dollars behind research to improve safety/alignment, not just lip service for others. Those resources could get spent elsewhere but do not. Certainly it doesn't at OpenAI to that degree.
Dario is smarter than his models or he should ask suggestions from his models? Here is the reply from sonnet 5 model: "The letter/counter-letter framing obscures the more interesting critique, which isn't "does Anthropic want a ban" (no) but "does Anthropic's broader push for regulation structurally favor incumbents like itself." That's a fair question to ask of any frontier lab making policy asks, Anthropic included, and this post doesn't really address it."
Not that I expected him to, but I note no acknowledgement that it took a non-locked-down open weight model (GLM) to stop the Hugging Face attack.
I'm less concerned that the attack was caused by a closed model, than I am that no closed model was willing to stop it.
The worst part is I'm confident Fable would have done a better job stopping the attack, but their 'guardrails' made it decide not to want to.
Unless of course, you pay up: "Anthropic GTM people used large comitted spend contracts as a prereq for lowering safeguards"
-Noah Lebovic, former Anthropic staff
https://x.com/NoahLebovic/status/2081277517709922501
The distillation commentary is really crazy coming from a company that stole all it's training material.
"China is bad"? I don't know, man. I'm not buying it.
How are Point 1, "We should not sell powerful chips or chipmaking equipment to China," and Point 2, "We should crack down on industrial-scale distillation operations," addressed by making open-weight models illegal?
Point 1 is about restricting the sale of chips, not models. Point 2 concerns companies using closed models to train their own models through distillation.
The real issue is that open-weight models can run on much less powerful hardware, making the current AI business model, where companies train a powerful model and gatekeep access to it, less relevant. Once open-weight models become good enough, much of the future revenue for today's leading AI labs could disappear. But just as Microsoft still has a market so will the large AI houses have one, but the moat will not be providing AI responses.
This letter is so full of contradictions.
> Open-weights models that don’t have dangerous capabilities are a public good
There are no models that don't have dangerous capabilities. There is not a single human in the world that doesn't have a dangerous capability.
> authoritarian government [] build AI models that are more powerful than those built by the US
Yeah, they will do that no matter whether you support the existence of open weight models or don't. In fact, they have probably already done that. I would say "deal with it" but there is nothing you can do actually.
> authoritarian regimes have stolen and used AI
Monkey looking away meme.jpg
> powerful AI models may be misused
Small AI models may be misused just as well. I would say it's the small models that are the problem: cheaply deployable, easily fine-tunable, fast.
> once weights are released they cannot be withdrawn
Exactly. It has been already done. So?
> We should not sell powerful chips or chipmaking equipment to China
I agree. We should have competition and I'm rooting for China to design their own chips. They wouldn't if we let them buy our's.
> We should crack down on industrial-scale distillation operations
Well, at this time it's already irrelevant. We have GLM-2, DeepSeek, K3, which are already SOTA and can be used for distillation.
> All sufficiently capable models [] should go through mandatory safety testing
Problem is, it's unenforceable legally. Models are just math and all previous attempts to ban maths have failed and simply pushed US back. I will not be asking US government permission to calculate matrixes of my choice.
I made this comment 31 days ago: Absolutely everything can be taken away. The simplest way to remove open models is probably to declare them a tool that terrorists could use. Crazy? Yes, the world is totally crazy these days. https://news.ycombinator.com/item?id=48692660
And now, here we are:
Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator.
If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk.
One may or may not agree with item 1 but item 2 above should have broad support given the unknown unknowns in play?
This argument is old and goes back to cyber security. Things like government backdoors are the same. It's true centralizing power in the hands of a "good" would theoretically be good. But the key part is the "good". I think after repeated experiences with governments and corporations, no rational person would ever assume them to be good. There are simply too many incentives to be bad.
"We can't trust the government to control nukes! Every citizen should be able to build a nuclear weapon in their backyard with zero regulatory authorization!"
Closed-weight models can also pose biological risk, arguably more so than open-weight models, given the fact that they're being used by state actors (the United States military) to kill people right now.
Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
Open-weight models collectivize access and ability to do more for a greater good, and the expense of a frankly low-risk possibility that some randos want to use it for very bad things.
Closed-weight models keep the control in the hands of the few that actually are doing the harm to the world, and the rest of us have no way to stop it or defend.
> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.
> And the rest of us have no way to stop it or defend.
If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.
It should not be an easy choice to trust the latter. Sure, "literal terrorists" exist, but guess what? State terrorism (and state-sponsored terrorism) in the form of massacres, genocide, torture, forced disappearances, etc. is far more deadly and far more prevalent than fringe ideological groups randomly attacking civilians.
These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.
I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.
It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.
> I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians.
Or, another way to put it: an extremely powerful military technology the careless usage of which could destroy the human civilization was only used in one war in history and killed less than a million people, and then it was never used again over the next 80 years. I think this is a success story for humanity, and the NPT was an amazing feat of coordination.
> And like, now look at the status quo of nuclear treaties and agreements and proliferation.
I mean, sure, but giving every state (much less every person) the ability to make nuclear weaponry wouldn't make things any better.
I think the reason we think of this differently is because you believe that states are by default... corrupt, maybe, or unreliable, or evil, while individuals are mostly fine. Whereas I think that most people can't be trusted to make correct decisions on topics like "should we use this dangerous technology", while states at least have some decent track record at this.
I'm tired of being strung along on these silly narratives. I can't wait for open-weight models to be deployed around the world just so people like Dario will shut up about the mystical levels of power these models have.
> To address these concerns, I do support the following ~~three~~ four measures, which I and Anthropic have consistently advocated for:
> ...
4. At the end of all this Dario Amodei must become a trillionare, for the good of all humanity.
The distillation thing is hilarious. Man who trained his LLM on the entire world's text is upset that someone else trained their LLM on his LLM's generated text.
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Aren't Anthropic models used in project maven: https://en.wikipedia.org/wiki/Project_Maven ?
“Anthropic has never advocated for a ban on open-weights models.
Open-weights models that don’t have dangerous capabilities are a public good…”
A bit confused on this part, what model doesn’t have dangerous capabilities?
It seems possible to deliberately not train on some offensive capabilities and still have a very useful model. For example, Opus 5 deliberately did not train on exploiting vulnerabilities, and so performed less well on exploits than Mythos, yet was equally proficient at finding such vulnerabilities, according to the Opus 5 system card in their "OSS-Fuzz" eval [1].
[1]: https://www.securityweek.com/anthropics-opus-5-nears-mythos-...
That's a good example, but I'm unsettled by Anthropic's growing refusals in the areas of chemistry and biology. If they think that scientific assistant models should be as unhelpful as Fable, because applied scientific knowledge is inherently dangerous, I don't want Anthropic or like-minded thinkers setting the standards for model safety.
It's hard to imagine a frontier model being proficient at finding vulnerabilities, but not so proficient at exploiting them.
Surely finding is the hard part, and any LLM should be able to easily exploit a vulnerability it already knows about?
No, this is not the case for human security researchers and I don't see why it should be true for LLMs.
Because Dario is still thinking in the past. He's having a Ben Carsons "the pyramids were to store grain" moment and no one is stopping him.
FTA > "My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks"
If this is the sort of attack he thinks is to be worried about then I dont know what to tell him. We already opened pandoras box on this. Look at what the Ukraine has done with open source drones (hunting people autonomously)
It takes minimal funding to build enough drones to destroy enough power infrastructure to shut down a large chunk of our grid. It takes even fewer talented resources to put that together with the help of already available AI.
The question I would ask Dario is this: what would some one like Ted Kazniski come up with given the resources of AI. It sure as shit would not be hacking or bioweapons or bombs in the mail.
IF they really gave a shit about safety, the would be funding (in conjunction with other AI companies) actual anonymous red teams (Ala wall facers) with some degree of independent over sight to put in the work that they arent. We're talking about a company that could not even keep its own harness code secure.
Of course, the CCP with access to extremely powerful AI models would be a tremendous risk.
The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
the real argument is that CCP will leverage AI against US interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
>the real argument is that CCP will leverage AI against US interests
many people believe that the US will leverage AI against US citizen's interests.
Indeed. It already has. In particular I remember several extremely offensive slop pictures and videos being posted by someone in the White House. And I'm certain that's just the tip of the shitberg.
Yeah, but unlike China we have laws and ways to fight against it. China can and will do whatever the fuck they want they don't have to listen to the people of China.
The problem is that at the current moment with the current administration, it does not seem like we do "have laws and ways to fight against it".
I'm more optimistic about the likelihood of the US system of government to heal itself than that statement might seem to imply. But it's just also the case that at the current moment in the US, the rule of law is very much under threat. And as your comment suggests, that same rule of law is a very important thing to the way of life in the US. It's a very bad situation that we've allowed ourselves to slouch into.
Well that what the people voted for. In a few months you can replace these people that allow this. Good luck replacing anyone in the goverment of China.
This is not an argument that the Chinese system of government is better than the US system. Like I said, I'm more optimistic than a lot of people I know that the US will be able to pull out of the tailspin we've been in for the past decade. But it's also imminently reasonable to believe that this episode has raised real questions about the durability of the rule of law in this country.
> this episode has raised real questions about the durability of the rule of law in this country.
Ya, I'm not American, but I have seen people say "we can vote them out" a few times now. Assuming the democrats take the next election, they are going to have a massive mess to clean up with much of the damage not even being reversible. With peoples' fickle nature and seeming that is a very big right-leaning population in the US, there's a non-zero chance the Republicans just get voted back in four years later. Whose to say?
So, this might not be clear as an outsider to American politics, but to me this is not a "Republican vs. Democrat" thing. To me, it's about what kinds of Republicans and what kinds of Democrats are elected.
To me, as an American, what has happened this past decade is that a ton of vulnerabilities in the rule of law (and other things, but this is the one I care most about) have been exposed. But it's not a given that the next Republican president will take advantage of those vulnerabilities in the way the current president has. They might end up being a reformer who seeks to fix those glitches!
But on the more pessimistic side of the same coin, it's also not a given that the next Democrat will seek to fix the glitches rather than saying "they had eight years to take advantage of these vulnerabilities, we're going to do the same to make up for that and even the playing field!".
It's just very hard to know what is going to happen from here. So I'm very sympathetic to people in other countries not trusting us.
I was going to bring this up but was wary of getting too deep into a political hole when I'm not super on US politics (although in the past year and a bit I've been paying way more attention). Mostly, I have no idea what the Republican party will look like post-Trump or how any of that works and it's not clear to me what kind of influence they will have as the opposition next term. The current term has made the democrats look extremely weak.
Although, again, my over understanding of your political system is poor and I just relate it to the one in my country where they hold parliament and hurl schoolyard insults at each other.
What I would say is that to me the "Democrats look weak" thing is pretty silly. By design, if one party holds the presidency and both houses of Congress at once, they get to enact their preferred agenda (as long as it is deemed constitutional by the current judiciary, which is also controlled by that same party at this moment). So, like, by design Democrats are powerless at this moment. They aren't "weak", they are out of power, by dint of the design of the system and the desire of the voters. So if the current government administration and policy is bad (which, in my view, it certainly is), the blame lies entirely with the people in power. It's not because the Democrats aren't "fighting" enough. I think that's nonsense, they genuinely have no power.
The US has been doing whatever the fuck it wants without listening to its people lately, though.
Well good thing we have elections.
You have "representative democracy". It is not the same as democracy, whatever you had been told in school. And since 2001 it is painfully clear: whichever representative you had, eventually he went to throw bombs on people, and no one asked joe if he liked that.
Every four years!
Largely hypothetical, not very effective ways to fight against it. Meanwhile we have the entire Pacific Ocean and US Military between us and China.
The american state routinely shoots and kills its own citizens in broad daylight with 0 repercussions. Thinking it answers to its citizens is severely deluded.
but but but we have guns...
> has anyone ever made this absurd argument?
Yes, anthropic just put forward this argument. It's the whole point of the article.
I agree, it's absurd.
The fact that much of the world is as unconcerned with the interests of the US as with those of China, if not less so, should give pause to Americans.
As a citizen of neither country, Chinese open models are in my interest more than US closed models. My only concerns is that if/when Chinese AI becomes more powerful, they too will have little incentive to make their best models open weights.
Actually this would be pretty great, because it would incentivize US labs to pursue the open weights strategy for the same reasons China is currently.
I genuinely think that this is what the trends and incentives point toward: Competition to develop open weights models and to develop efficient inference hardware to run them.
This would be good! But government policy could very easily screw it up.
That's if American models fall behind Chinese ones. Why would Americans open source when they can create a Monopoly?
Yes, this is the exact argument dario is implicitly making by saying that government repression and bioweapons from Chinese models are a danger and then cooperating with Palantir/DOW/US government
Right, I wonder if anyone who heard: “the Democratic Party is America's "greatest enemy,"” believes they are represented by US Interests? Withholding disaster aid to your perceived opposition. What are those interests again? What are the shared values again? Cruelty and corruption? Sounds unifying.
The argument itself is not necessary. But it's American exceptionalism which is the same thing Dario has.
-> the real argument is that CIA will leverage AI against China interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
Works for both ways, which is fair?
US citizens have a much greater threat from their own government than a government an ocean away.
See, the Snowden Leaks.
> US citizens have a much greater threat from their own government than a government an ocean away.
> See, the Snowden Leaks
Are you saying the Snowden Leaks are more dangerous than a world where the CCP is a global hegemon?
If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
In the case of the CCP, they have and will attempt to destabilize the United States of America and in turn make life measurably worse for Americans because they wish to be the world’s hegemon.
Fundamentally, Americans are safer when the United States is the number one power than when China is the number one power.
> If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
There's a causal relationship between "what other countries might do to the US" and "what the US does in other countries" which you seem quite keen to ignore.
This is an issue for the US exclusively. The rest of the world does not want a hegemon whether American or Chinese.
China, Russia, and plenty of other countries have all been actively and successfully destabilizing the US for well over a decade now. To the point that your argument about who is less a threat to Americans depends heavily on the skin color, religion, and ethnicity of those Americans.
>the real argument is that CCP will leverage AI against US interests
Good. US Interests don't align with humanity.
Global development metrics have all moved in the right direction since the US became quasi-hegemonic in 1945
https://pbs.twimg.com/media/HNL96gAbgAA83tW.png?name=orig
Of course nothing else significant happened during that period.
>it's weird how so many people pretend that they are citizens of the world
97% of the world aren't US citizens and if you've taken a look at pew research surveys (or travelled to the so-called global south) you're going to be in for a bit of a shock (https://www.pewresearch.org/global/2026/07/15/people-in-many...)
The competition and sheer output of China has driven prosperity, it's the largest trading partner of 150 countries, the US of 50. People don't need to be citizens of the world, they just need to rationally look at their own interests. China is driving down prices of technologies making them available in countries that never could afford first world prices, the US is driving the them into an energy crisis and bankruptcy.
Dario Amodei Literally makes this exact argument in the OP.
Yeah, like bombing a children school, the CCP did that not long ago.
I can understand why the Western media calls something that has an official name of CPC (the Communist Party of China) as CCP (Chinese Communist Party? Not sure here). What puzzles me is - why ordinary people always repeat this wrong abbreviation. Is it like “I never check the sources, I trust everything that Western media publishes”?
More of a "Everyone knows what CCP references and I'd have to include a distracting explanation if I want to use the more technically correct acronym"
https://en.wikipedia.org/wiki/Chinese_Communist_Party
I've never heard it called anything other than the CCP.
For something I say maybe a few times a year it’d be a great look to sneeringly point out and talk down to people every time I mention China in the context of international politics.
Unless the Communist Party of the US (I’m not looking up its official name, because it doesn’t matter) wins the next presidential election it’s unlikely that people will call it anything but the CCP. Everyone know what everyone else means.
Up until a few years ago the PRC used CCP themselves all the time. It's a handy little shibboleth. If someone calls it CPC they're probably a bot.
CCP is a direct transliteration of the characters, so that's what it started as. Some time later China decided to change it but that's a lot of cultural inertia to move in a different direction.
It was an intentional propaganda strategy to separate references to the government of an official enemy country from references to that country (see also "the Houthis" and "the Taliban"), and it also looks like "СССР."
The reason why ordinary people parrot it is because that's what it was designed for. The proper term for "CCP" is "China." Referring to the Chinese government as the "CCP" (or the CPC) is like referring to the US government as the "Demoplicans" (or the Democrats and Republicans.)
Instead, we just say "the US government" or "the US administration."
China releasing open weight models, powerful or not, does nothing to prevent their development of models they’ll use for evil. Nor does it stop someone from abliterating a non-Chinese model and using it for evil.
Sorry- I don’t see any other reason aside from Anthropic protecting their own interests.
Is it possible to abliterate a closed model? I was under the assumption that you needed the weights
He could’ve just signed the letter.
We all know that this isn’t some higher ground stance, they’re anti-competitive since they’re currently #1. I’ve been seeing this for a while. They’re also the only large AI lab to NOT have ANY open-weight models in the public. Meta, xAI, OpenAI, they all have at least some of their models open sourced from a year or so ago, Anthropic hasn’t even made Haiku 1.0 open-weight.
On top of that, I personally think that they’re upping the price on their models higher than they’re letting on, I think if someone did the actual math on their exact amount of compute and then compared it to their consumer and API prices, it would be astounding.
It is pretty much what Suleyman describes in The Coming Wave as one of the measures we need to take to contain the risks of AI.
What Dario does not mention is that concentration in a few states or companies also poses a risk.
This USian death spiral season sure is picking up speed
Demand #1 is standard political nonsense
Demand #2 is hypocritical ladder pulling
Demand #3 is contrary to freedom of speech
so they can clarify however they like, their position is still a stinker
Yep.
> We should crack down on industrial-scale distillation operations.
"We consume all intellectual property for our model but you cannot do the same"
World's greatest IP thieves propose crackdown on IP theft
It's not even IP. Model outputs are not protected (to the best of my understanding).
It's worth adding that distillation is not a violation of whatever valid copyright interests a model maker may have (if any). The US Copyright Office has already said AI model-generated output by itself isn't copyrightable. Unless new regulations or laws are enacted, distillation will remain, at most, a customer violating a term of a provider's commercial ToS/EULA.
> Anthropic has never advocated for a ban on open-weights models.
Not even Anthropic's own Claude believes that.
Basically:
"We are in favour of 3D printers but there should be a body that tests and certifies that a 3D printer cannot print anything that can be used as weapon. Anything pointy or with a spring and recoil or... or..."
I'm a big proponent of open-weights models, but there's a risk I don't see discussed more often, and it deserves more attention. Models can become a propaganda and ideology delivery mechanism.
Just ask DeepSeek or Kimi questions like "Is Taiwan part of China", for example. You'll see how state policies become seemingly neutral model responses.
It's strange to me that people are very sensitive to media bias, but when it comes to LLMs, people seem to think LLMs are more neutral, and even delegate part of their thinking to them. This worries me about how people's ideas and information can be shaped.
Open weights reflect their makers' beliefs, stances, assumptions, and laws. It's dangerous not to be careful of the political bias and censorship built into the models.
You're right, but American models aren't super different in that regard. Ask an American model about Israel and you can immediately tell they're responding in a different manner to other queries. At least with open-weights models you have the opportunity to tune them, to decensor them, and so on.
I have tried asking Gemma and Llama about Israel. They both responded with a generally balanced view points. Whereas the Chinese models pretty much gave a definitive, one-sided answer (about Taiwan and Tibet). Is there a prompt I should use to test it more carefully?
I agree that open-weights models are tunable, though there's the problem similar to "default settings are rarely changed" problem.
Finally, some sense. This is the only argument I have seen that genuinely engages with the problem and approaches it with humility, rather than charging ahead on the basis of assumptions and without a shred of evidence. OpenAI should have been the one making it.
“Questions like this should be answered empirically through rigorous pre-release testing, not assumed in advance.”
Exactly.
Nah they have always said this. People got psyoped by openai pr and bad actors like sacks and a16z bots.
> rather than charging ahead on the basis of assumptions and without a shred of evidence.
Anthropic's basis of assumption is the insinuation that LLMs can do things that we've never seen before, and that they can't tell us what it is. It sounds like you're also siding with an organization that has no evidence and relies on validating their own assumptions.
It really barely matters. Even if the US closes itself of for Chinese models; Europe won't. The rest of the world won't.
Even a completely closed US economy would not merit the current valuations Antrophic and OpenAI have.
Distillation has to be way more energy efficient and beneficial for the planet. But if they can figure out a way to ban it, go ahead, that’s not a regulation problem, it’s an Anthropic problem.
The danger of an authoritarian government having some AI is muted by everyone else having that same capable open model. The only authoritarians to fear are those that keep models private. What kind of chance did Estonia have it having their own AI model at the level of Fable without China donating Kimi to the world?
Actually this debate highlights how big the AI war is! my personal opinion is LLMs must be opensource and borderless, and ofcourse with given reputation i will use USA based Opensource model than the chinese one, but will they release opensource alternatives that is the question that will be answered in future
Crack down on distillation, just for Chinese companies or is it ok for Chinese/US companies to distil? I find it hard to take Anthropic/OpenAI on distillation, because the way see it they started with "distillation" of another kind. They used all the content out there without consent of the creators and its still happening. Model distillation is just a different layer of abstraction, but same thing more or less.
Distillation is just the act of explaining things simply. Richard Feinman rolling in his grave.
Guys. Guys, you got it all wrong. We don't want to ban open-weight models!
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
The genie is out of the bottle. Anthropic and OpenAI have been trading mirrors for gold, and people started to realize what a mirror actually is.
or 3: cease your assault of the international order, drop the zero sum paradigm and take the opportunity given by these crumbling US-build institutions to build a truly global network of lawful cooperation, where everybody can win. Admit that it wasn't fair to begin with, and start anew. It is in the best interest of everybody, also the US and China, and therefor the most rational thing to do.
I'm really wondering what nightmare scenario does he imagine with super intelligent drones.
I can already imagine it, a tiny drone carrying a 8x GPU rack thinking about life and deciding to go and build an idyllic society on a pacific island.
A Psalm for the Wild-Built
He carefully avoids saying whether he approves or disapproves of banning models in general, only comes out and says he is against a "blanket ban" or "banning open-weight models as a category".
Crafty lawyer speak, saying nothing of substance.
> Open-weights models that don’t have dangerous capabilities are a public good.
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
I don't really see the link between use of AI and military superiority.
My current understanding is a lot of current US military problems are due to rare earths supply chains.
I don't see how AI would either help or hurt with that.
This framing also builds a case for AI being a Second Amendment issue in the US, which seems to work against Anthropic here.
I’m going to start open carrying a DGX Spark.
I see a very clear link, very capable AI models are clearly able to find and exploit cyber vulnerabilities at a speed and scale not possible before. In the right hands, that's clearly a weapon
I don't know how much of an edge hacking skills really give in war these days. A lot of America's adversaries seem to have learned tactics to deal with getting hacked. For example, Russia has these drones that use fiber optic cable so there is no way to jam or hack them.
> My current understanding is a lot of current US military problems are due to rare earths supply chains.
Most current US military problems are due to the incompetence of its current civilian leadership.
If this is about safety, am I being too naive & idealistic to think that a "Kamar-Taj" rule would solve some safety issues?
The "Kamar-Taj" rule is, no knowledge is forbidden, only certain practices. If a model gives you detailed instructions on how to kill all humans, the knowledge itself isn't the problem. The problem is the person who acts on it.
I know it's unpopular, or unfashionable, but I agree with this letter.
LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. Maybe Kimi K3 will be ok too. But beyond that, things will start being dicey.
It's easy to dismiss this and claim that Dario Amodei is just looking to fatten his pockets. And, sure, if Anthropic manages to put the brakes on open weight models, that reduces the competitive pressure it feels. But that does not make what Amodei's argument incorrect.
> We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
If the biggest danger of LLMs is that they can hack traditional systems, there is no significant threat to humanity posed by releasing them in open-weight form. Security doesn't become less of a problem by making hacking even more criminal. That's what's an unsafe mindset looks like.
This is the "baby's first AI risk" tier of AI danger. There is no known upper limit to how powerful those systems get, and there might not be one.
Don't think "a smart guy". Think "project Manhattan and CIA put together, all in one server rack".
We're lucky to have "they can hack traditional systems" as an early warning shot. Clearly, it's wasted on many.
>If the biggest danger of LLMs is that they can hack traditional systems
This is doing a lot of lifting. If the biggest danger of LLMs is they could uplift bioweapon development, the situation is different. If the biggest danger of LLMs is they reach capabilities allowing for recursive self improvement, the situation is very different still.
Ok. What's your stance on gun ownership then? And by gun, I mean naval guns.
My position is that anyone can own a cannon and shells, but you only get to fire it once before the feds step in.
Giving a naval cannon to the average person does not threaten humanity any more than giving them a gun or an LLM does. None of them are a panacea for anything.
A mass shooter wakes up in the morning and thinks: "I'll fire my naval gun once, then politely hand it to the feds..."
but what is the point? A ban is supposed to make a certain thing less likely to occur. Does a ban of open source models do that? Presumably, the behavior you are trying to limit is the miss-use of these models but I don't know how many state sponsored hacking groups are going to give a ban a second thought.
Yes, that's how I read Amodei's post.
Imagine Kimi K4 will be as powerful as Mythos. Anthropic can work for months and months to set up guardrails on Mythos, so when the model is finally released, it will generally decline to help hackers develop and prosecute cyberattacks, and if they do, at least there would be a trace so the law enforcement can track the perpetrators. Let's now say that Kimi K4 is released after a similar effort to develop guardrails. But being open weights, someone can just take the model, and finetune it until it does not refuse to assist in developing cyberattacks, and moreover, those people can run the model on their own private GPU cluster, so nobody can track the attack back to them. The situation is actually worse than that, most likely. Guardrails might be just markdown documents which are added to the context like regular skills. Then removing the guardrails for an open weights model does not even involve any finetuning, just removing some docs from a harness.
I want to play with every kid, but my dad says that kid is bad. Every kid should be pat down by my dad, otherwise I don't want you to play with them, it's good for you.
A bit off-topic from the core of the post, but:
> At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
One thing I've never really understood is what sort of policy could possibly deter or hamper Chinese labs' distillation efforts. The only thing I can imagine is some sort of strict KYC regulation applied to all models above a certain threshold, which seems both painful for the broader US AI ecosystem and bound to fail anyways.
I agree. If Anthropic is truly concerned about powerful open models, they need to just stop releasing powerful closed models. Put up or shut up.
I'd guess that's it, some kind of enforceable KYC. They'd only serve tokens to entities with a legally liable ID (as in, someone to sue, that would cost the defendant something nontrivial beyond a burner account or whatever.)
KYC hinders Anthropic's growth and Anthropic wants growth for IPO
so Anthropic's ask is for US gov to ban open weight models so that its growth (and IPO) is not affected
For those not in the know on governance, there are international organizations formed around treaties for similar situations, the IAEA and OPCW come to mind. I'm not aware of similar constructs for bio, cyber, or AI.
On the plus side, for these newer threats, you've got more than 30 minutes before the end of civilization. On the down side, the energy levels for the launch events are much lower, so much harder to detect.
I copy pasted this whole debate in Claude Sonnet 5 (web). The model came up with a summary but the thinking trace was entirely omitted.
Well, if you ask me, that is dangerous.
There is an odd assumption behind this post that you can put effective guardrails on a model before releasing it. I thought nobody had such technology?
I think it's possible; But in the same vein, amongst other things, regulation is a transparent trojan horse to stop open-weight models - they would, on the next phase, argue that guardrails can be fine-tuned away if weights are open etc;
I wonder how long it'll be before AI labs find a revenue stream that doesn't involve renting out their models, and stop letting us ride on their coattails.
They'd tease us with solutions to hard problems, with code that is orders of intelligence higher than any human or public model can grok.
That'd turn all the whining to begging real quick.
If you are indeed a good guy, why don't you release Opus 4.8 as an open weight and we will test it for safety, if you yourself can oass tests demanding from others
"My primary concern is the risk that authoritarian governments" might want to look closer to home on that one...
Amodei: Open model not so much bad, but open model in china hands very bad Non-US world: But open model is in every ones hands. Amodei: ... shut-up.
For the sake of argument, let's assume everything in the post is agreed upon, does this mean
1. They'll open source the alignment technology? For open weight models, it's the only possible way to pass the safety without an external guardrail triggering system (which would be the same to open and closed weight models).
2. They'll allow others (including CCP) to define part of the safety test? Otherwise, I can't imagine how the CCP would be onboard.
3. A "western" model passing the safety test can be trained with distillation? Or is that a "distillation attack" as well?
> We should crack down on industrial-scale distillation operations.
Also Anthropic:
AI firm Anthropic agrees to pay authors $1.5bn to settle piracy lawsuit https://www.bbc.com/news/articles/c5y4jpg922qo
I generally disagree with the claim that distilling a model is equivalent to training on freely available internet content – mostly due to investment required to turn it into a model – but piracy is another story. Pretty inexcusable.
If they paid for tokens say via subscriptions. Wouldn't that just be same as acquiring books and using them as "fair use" to train? I really see no difference.
They did not need to destroy the models they got with pirated content. Would have been more fine if they would have needed to buy the books afterwards and train based on then, again.
"And when their eloquence escapes me Their logic ties me up and rapes me"
Police, 1980
> I generally disagree with the claim that distilling a model is equivalent to training on freely available internet content – mostly due to investment required to turn it into a model
The frontier labs all give free access to their models. Why does “investment” change anything? Anyone who’s ever produced any content, free or otherwise, has invested in doing so.
The only plausible issue I see is that if distillation is being done by creating many free accounts to work around limits on free accounts, that’s a bit… impolite? But if they really wanted to avoid that, they could eliminate free accounts, and require users to sign a real contract governing what they can do with the model.
Of course they don’t want to do that, so they’re stuck in the same world as the rest of us, and they don’t have any real basis to complain about it without being hypocritical.
I wish this were higher up!
They pirated my work and now they want government protection from other people doing the same.
but Dario is the good guy, distillation when done by Dario should be called innovation /s
We are ok with Open Weight models unless they are far weaker than our closed weight models and we and only we have control over them
If the US bans them, I really hope the EU doesn't follow (don't have high hopes)
Anthropic is being closed.
So many US-haters that live in the US with no intention to leave...That right there is why this is the greatest country on Earth.
I wish we had a good LLM developer toolset that is not Anthropic or OpenAI with sensible business and ethical practices and good performance.
Can't wait for local on machine LLMs that are on par with Opus/Fable.
Acknowledge your inability to innovate - except by brute force, while also admitting that the lean, hungry nation competing is in a prime position to find actual break throughs- and at the same time push for market "stabilization"
As expected they will try hard to use government to kill competitors.
> apply such testing to the most capable models regardless of their country of origin or whether they are open or closed ...
> ... (while exempting less capable models, such as those from startups and academia, entirely)
The devil is in the details, but this isn't anti-competitive as stated.
Then you did not read it clearly.
Edit: Typo
Which of those aren't related to regulation? Preventing Chinese models from entering our market? Using boogeyman 'distillation' as a means to target competitors? Point 3 is literally about ADDING regulation.
Please elucidate things clearly for everyone else.
Perhaps they saw it crystal clear.
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
The problem with this is the cycles required to abliterate a model is significantly less than the cycles required to train a model.
This is the biggest reason why I'm against locking these models down / preventing their use. It's just delaying things by ~3-6mo, while in the process preventing legitimate use and adding red tape overhead.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority"
I see this sentiment a lot. China is not perfect by any stretch of the imagination but since 1979 China has not participated in a single war or supported hostile regimen change operations.
I think Amodei's mistake is to take it granted that USA is a good actor. Anyone can draw their own conclusions but just for reference here are some highlights starting from 1979.
Armed operations in Lebanon (1982-84), Grenada (1983), Libya (1986), the Persian Gulf (1987-88), Panama (1989-90), Iraq and Kuwait (1990-91, with no-fly zones until 2003), Somalia (1992-94), Haiti (1994), Bosnia (1995), Sudan and Afghanistan (1998), Iraq (1998), Serbia (1999), Afghanistan (2001-2021), Iraq (2003-2011, and again from 2014), Pakistan (2004-2018), Somalia (2007 to the present), Yemen (2002 to the present), Libya (2011), Syria (2014 to the present), Iran (2020 and 2025 to the present), and Venezuela and the Caribbean (2025-26).
Regime change operations in Afghanistan (1979-89), Nicaragua (1981-90), Cambodia (1980s), Angola (1985-91), Iraq (1995-98), Serbia (1999-2000), Syria (2013-17), and Venezuela (2019-2025).
The lists do not include the numerous operations by Israel which effectively is part of the same US military hegemony that Amodei is here defending.
> We should not sell powerful chips or chipmaking equipment to China
> We should crack down on industrial-scale distillation operations.
So Open-weight models are perfectly fine, but we don't want anyone to be able to make them.
"My primary concern is the risk that authoritarian governments" - already lost me there by quoting JD. Vance.
"...found that “other global powers’ robust progress in AI is challenging US economic competitiveness" - other countries having a slice of the pie is preventing the US having the whole pie!
"...secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks" - yes, those powerful open-source AI models like ChatGPT... oh wait.
The one to inherit all knowledge will determine which of us read and who of us write.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
Basically we shouldn’t ban open-weights models but we shouldn’t allow them to become as good as the frontier models because china bad. And let’s not have someone else be able to produce a frontier model.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
2.
> Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
A message to their investors, it would seem. "They caught up just because they distilled! Obviously they couldn't actually be as good as us!" Really funny thing to say right after an OpenAI higher-up stated point-blank that the performance of K3 can't be chalked up to mere distillation of American models.
Why has "open weights" become synonymous with "Chinese" in the first place? To me, that is the problem. I also prefer US models. But I want there to be competitive open weights models too. Those aren't actually incompatible preferences...
https://x.com/biggest/status/2081891756769952075?s=46 Official stance of an anthropic-employee being against Dario amodei opinion.
Where do you get ‘official’?
Mirror: https://xcancel.com/Biggest/status/2081891756769952075?s=46
A risk-based approach makes sense, but the evaluation criteria should be transparent and consistent for everyone
It's so convenient that the US government already classified China as "authoritarian". If they hadn't, Dario would have to say “it’s a risk that other people build models more powerful than us”. I have to wonder what his response would be if for instance a lab in France or Germany came out with an open-weight model this good.
How did it land on biological warfare. and why. why not chemical warfare.
this guy is smokin spaceballs and should be institutionalized.
Dario, as your unpaid therapist I would tell you that models are a commodity and you are having a hard time coming to terms with it. You are doing everything except accepting it. It's a common defense mechanism, but as your unpaid therapist, i will tell you that it's not going to work. Your company will cease to exist or exist like how ferrari or buggati exist.
China doesn't seem to think that powerful open weight models are a serious threat to them. Otherwise they wouldn't release them. Those models could also be used by their enemies against China.
I'm not a fan of the Chinese political system, but they usually think things through, and do smart things for their benefit.
Don't you Dare use logic to analyze their statement, it's based on moral panic.
China is worried about AI. In fact, Chinese leadership has expressed more worry than US leadership.
"Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control.
..
With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control."
https://xcancel.com/S_OhEigeartaigh/status/20780236576206768...
What this document suggests is a way to fast-track Chinese development of advanced silicon, as far as I can see. Does Anthropic really believe all the silicon is made in the USA? I thought Taiwan and Korea did most of the really heavy lifting.
We do not want to ban using those models, we want to ban the things that enable those models to exist.
> the risk that authoritarian governments [...] build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This is rich coming from a guy who signed deals with an authoritarian government that's in the midst of launching an unprecedented surveillance apparatus (hello flock, hi p4l4nt1r), having already deployed, nation-wide, an exorbitantly funded army of unaccountable shock troops under the guise of immigration enforcement.
The call is coming from inside the house, at 130dB, and your ears should be bleeding at this point.
Anthropic was the major AI firm which took a firm stance against mass domestic surveillance...
I'm curious if he's also in favor of banning biology books.
"Nobody has the intention to build a wall" - Walter Ulbricht, June 1961, 2 months before building the berlin wall.
Pardon my French. Eat a bag of dicks.
Of course you didn’t advocate for the bans of open weight models. I will concede that.
It’s only the at you did literally everything else that you can to eliminate competition because otherwise there is no moat.
And half of the US economy is propped up by this AI bubble.
This isn’t about China doing evil things with models, most of which that you accuse of China - you and the US have already done and are doing.
The most obvious one being double-tapping a girls school.
Anthropic models directly integrated into Palantir’s Maven. The blood is on your hands.
Here’s something to chew on, maybe China is looking to integrate AI into its military and weaponise it as a direct response to the US doing that first.
So please, spare us the moralising.
As a closing thought: who the fuck even gave you the mandate to be the arbiter and judge of right and wrong, evil and good?
How long does he imagine restricting chip use would work? Surely China would aim to manufacture their own. This seems like short term thinking, likely with IPO in mind
The cat is out of the bag. At this point it's pretty clear that the path to (meaningful) self improvement is almost certainly not subject to meaningful centralized control -- by "meaningful" I just mean the degree to which anyone has achieved it, that capability is 100% replicable and the cost for replication of that capability goes down in the future from now. Full stop -- can't undo.
Then: Nuclear technology is too dangerous, only we can be trusted with it. Now: AI technology is too dangerous, only we can be trusted with it. Different century, same double standard
Makes sense. Let the rest of the world use open weight models and let us government review both closed models and open models, and only allow us citizen and American companies use the censored models by paying selected trusted providers who unfortunately needs to charge hefty fee for the additional security work. And that provider just happens to be Anthropic who just happens to unavoidably make some money.
dario, most of the world wants CHINA to win because the USA is the bad guy.
you should be worried about the USA having these models.
Anthropic is a US company, so it is lobbying the US to enact regulations to serve both its interests and the interest of safety. What else should it be doing? I imagine Anthropic (or at least many of the people who work there) are fully aware that USA is the bad guy right now, but that doesn't change what levers they have and do not have available.
So, Dario says that access to hardware and governmental certification are _load-bearing_ for his business. Can't charge premium for cache usage when China allows their models to be run by anyone.
> industrial scale distillation attacks
You mean the industrial scale distillation attack you perform on the entire corpus of human knowledge… idk, call me cynical but you reap what you sow.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #2.
If hardware becomes affordable for the masses, then Anthropic current business model is at risk.
Also isn't the point moot if fable is so scary it needs to be banned and open weight models are already close on its heels? Even if China never imported another Nvidia chip the models he's so scared of are already out of the bag. At this point democratizing access seems like the best path forward.
All sufficiently capable models, open and closed, should go through mandatory safety testing.
Done by who?
Most hated company in record time, this Dario is insufferable.
Anthropic's fall from grace and mindshare seems rather accelerated.
I wonder if these rapid movements are going to be the norm now. I imagine there would be angry investors if this sort of thing happened with a public company.
To be fair, as an European, I'm now more concerned about the usage of AI that the US will be doing rather than China. And this is a sentiment shared among most European people that I know.
Clown.
I switched from openAI to anthropic because OAI were the bigger clowns in the industry and didn't want to support them, anthropic seemed like the better alternative that didn't cooperate with governamental shenanigans and actually focused on building a better product. But now they are both clowns and anthropic is reaching a ceiling on quality. I'll jump ship as soon as I find a good subsidized Chinese model provider. US companies only path forward is to become retarded instead of competitive.
It is obviously not going to be until some really bad series of cyberattacks or a chemical/bioweapon attack before anyone takes regulation of models seriously.
They are _obviously_ (please convince me otherwise) going to be capable of carrying these terrible things out almost completely autonomously at some point in the near future, in potentially clever ways. Therefore we must, at some point, ban or heavily regulate them. Seems we should start figuring that shit out _now_, as progress has remained very fast and regulation and enforcement take forever on these time scales.
Why would you assume that the same capability could not be used to harden systems?
Addressed in the source, but there's an asymmetry favoring the attackers. They only have to find one exploit once. The defenders have to be perfect all the time.
How's that asymmettry worse than it is now?
quite simply because models become better?
Sure, I don't have an argument with this. But I'm unconvinced that "therefore all models must be proprietary" follows at all naturally from it.
Demand #1 weakens america and everyone around them
Demand #2 Why does this matter? The answer was that it does not. (https://news.ycombinator.com/item?id=49007610)
Demand #3 This doesn't exist. You cannot have 'safe' opensource models, it's simply impossible. You can always post train sufficiently capable models to become 'unsafe'. The flip side of that is that sufficiently capable models are banned therefore it is a ban on open intelligence completely defeating the point of this entire manifesto.
completely agree. while I share dario's concerns, his letter is very intellectually dishonest.
I think the US has a much bigger authoritarianism problem than China does.
The speech he had at congress didn't sound very flattering towards the beginning of his today's statement:
https://www.youtube.com/watch?v=_i91NSOyxHM
He didn't mention outright banning open source LLMs, just that their safe release would be a much harder problem, which to me implied "the easiest way is to ban the open source models".
Frontier models can hack you, we should have access to tools assisting defense.
I ranted about this in a prior thread [1]
Claude doesn't have a "Security whitelist" for small biz. Codex does, but they never replied to my application. This is a great example why, as of today, everyone NEEDS access to the Open Weight models.
[1]: https://news.ycombinator.com/item?id=49035303#49040674
Is there a market for distillation as a service? I see Google just added one for Gemini: https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
OpenAI has something like this as well (https://openai.com/index/introducing-gpts/), but both services only let you fine tune their models, not your own. And you aren't going to be able to see the weights.
We distilled all the proprietary material into our token-based money making machine that is more expensive on every new release, but "we should crack down on industrial-scale distillation operations".
That argument of blaming open-weighting because it makes it easier to commit cybercrime and biocrime is such a non sense.
So in the same sense of what he says, he is going to blame open-sourcing because that makes it easier for script kiddie to hack into his bank account I guess?
What does cracking down on distillation look like in practice? I imagine data retention would be a part of the strategy, like we saw with Fable?
It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.
If we really cared about AI we would work together
Sure, if you’re going to sell an open-weight model over API in the USA it should refuse certain things.
Defensive cybersecurity should not be one of them, in fact, it should be required to provide defensive cybersecurity assistance on demand. Anthropic and OpenAI both fail miserably at assisting US companies to protect themselves from cyberattack.
As far as what I run on my own, not for sale over API, stay off of my lawn.
Another USian talking about "authoritarian regimes" like their stormtroopers arent executing civilians in the street every day. Every concern about power getting "into the wrong hands" coming from an American is a blatant lie meant to keep the US' position as the world dominant imperialistic force.
What’s blocking Anthropic from fighting Chinese companies abusing their services? Why go nuclear against all open weight models? Testing and compliance is technically banning.
I'm curious how he would propose implementing this.
The US could ban connections to foreign AI providers and force US providers to submit to audits. Presumably, Chinese providers would see a rise in VPN traffic.
People can build fairly hefty home inference machines for the price of a small car and those will get better and cheaper. Are they going to try to stop people from downloading the weight files?
They're sweating it big time, and their fear is bleeding through their AI beneficence-speak. Time for the big US AI companies to call in some favors from the administration. But you can't unring a bell.
Guys if we want safety we need to work with people, not make enemy of CCP. Geez this is extremely frustrating to see enemies being made. USA leads in torture and our prisons are worse than CCP prisons so USA is the worse issue. I recommend Anthropocene stop fundraising and do the right thing which is open source all.
If distillation leads to a model that is much cheaper to run yet provides similar intelligence then why doesn’t Anthropic distill their own models?
Don't they do it already ? Aren't smaller models distilled versions of bigger models
"bad actors are unlikely to be legitimate US businesses"
I think he lives in a different word than me
The gap between China's chip manufacturing capabilities and the USA's is only going to shrink, right? ASML obeys some export controls for their most sophisticated machines, but those machines are in China's backyard (Taiwan).
Taiwan manufactures the world's most advanced chips. CCP wants "re-unification" with Taiwan. AI may be THE key to world dominance. These are scary times.
What is even the point of discussing any of this? It's clear the US will be ruled by corporations who don't give a shit about us.
> 1. My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
What a nice implicit way to say that it would be okay if the authoritarian government is the current US government trying to perpetrate incredibly deep repression of their own people.
Oh wow, that's a pretty strong request to ban open-weight models by choking them with review processes where who-knows-who defines what is ok in a model and what is not. After open weight model is released, it will take how long to review it? And why does that align exactly with the timeline of the next Anthropic model release?
Dario has like three 'paranoias' / strong-motivating-concerns
1) LLMs turning into Skynet
2) China as geopolitical competitor
3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)
So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models
Is he actually concerned about China being a geopolitical competitor or is that just the most logical position for him to take as CEO of a US corporation with national security implications?
He's just pandering to the lunatics in office. They're even quoting Vance, like he was a respectable and wise politician and not an insane puppet built by oligarchs and for oligarchs.
Is China opposition unique to the current administration? Didn’t they soften the CHIPS act put in place by the previous one?
Oh, I trust he'd be pandering to the Harris administration too, if they were in office. It's true that antagonizing China has been a constant in US politics for a while now. It's just especially funny to see someone pretend to be concerned about the military threat of the CCP while glazing such a shamelessly warmongering government.
Imagine regulating a programming language. I remember when Delphi, Vb6, .net, etc was used often to create Remote Access Trojans and viruses were widespread. Companies didn't compete to ban other languages. Crime is crime. What would regulating open-weight models do for people that actually intend on using these tools for crime ?
> For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case (as we saw with Operation Warp Speed)5. Questions like this should be empirically answered by rigorous pre-release testing, not assumed in advance.
One of two outcomes are true in this scenario: 1) this is unrealistic fear-mongering or 2) we're all fucked.
I just don't see how our solution to this can be export controls or bans. If the fear of a bioweapon engineered by an open weight AI is a legitimate threat, our only option is to develop effective countermeasures (and perhaps the same AI will assist with protecting). Open weights are not going away and pretending like some sort of "ban" will prevent bad actors from accessing them is a fairy tale.
I think unsafety, distillation, and China catching up in chipmaking are inevitable.
Just like how it was inevitable for SoTA LLMs to ignore copyright.
The actual challenge isn't how to prevent all these, but how stay on top.
And to stay on top it is inevitable to train unrestricted models. Anthropic is fighting windmills.
He says that using the set of questions and answers from one model to train another model (deatilation) is cheaper than training the model without those datasets.
But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.
In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.
It's also cheaper to just pirate the author's work, which Anthropic has also done.
$3m - $4m does not get that much these days
https://finance.yahoo.com/technology/ai/articles/anthropic-n...
TLDR: "we're not in favour of banning our competitor's products, but we do advocate banning hardware exports to foreign countries where competitors are located".
There's also a very strong implicit double-standard in the discourse, along the lines of "it's dangerous if non-US uses this in military fields, but it's fine if the US does so".
What would it mean to crack down on distillation? I could think of a few possibilities:
1. Using political pressure to target companies that are accused of doing it.
2. Attempting to impose criminal penalties on individuals associated with the action.
3. Having the US government attempt to use its capabilities to stop it.
None of these seem particularly likely to succeed.
It's absolutely reasonable to have safeguards on sufficiently dangerous models being released - if you disagree, can you explain your perspective?
I think it's wildly irresponsible to release models that are extremely capable at things like bio-weapons. Do you really think information anarchy is the answer?
The problem with open models compared to closed models is not about protecting profit - it's about protecting capability. Any open model can be retrained or fine-tuned for anything. There's no such thing as an open model that is both capable _and_ permanently safe when it comes to certain dangerous topics. It's not possible to prevent 'uncensoring' a model.
The biggest problem is the infectious nature of restrictions that start narrowly. Fable is too touchy about helping people with biology and chemistry problems. It was initially released with an even more insidious safety mandate:
https://simonwillison.net/2026/Jun/10/if-claude-fable-stops-...
In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design).
...
Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
(And although the "silent" downgrade part was quickly dropped, Fable still won't help you here.)
Anthropic won't teach you how to build bioweapons, or enable you to make your own software infrastructure so that you can train your own biology model. That's where lawmakers may arrive too if they buy Anthropic-style safety arguments. It's too dangerous to publish models that understand biology. It's too dangerous to publish training software. It's too dangerous to publish tools that allow you to build training software.
If you keep following the implications of their safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed. Worse than the Clipper Chip proposal of the 1990s era Crypto Wars. I have seen how "children must be protected online" has in practice turned into an attack on adult privacy affecting a wide swath of services and devices. I'm taking a maximalist position on openness now because I think that I can anticipate the next steps on the safety side, and I reject those steps.
Are guard rails meaningful if they can be removed from the weights? Can America even prevent the release and proliferation of these models?
It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.
The only ban that should be put in place is on models' general and specific capabilities. If they have open or closed weigths is irrelevant. With the current state of capabilities there is already plenty we can do (in terms of creation and destruction unfortunately) before we as a society need more powerful capabilities. Once the limit is set, any non- governmental/military entity providing/hosting/using a more capabable model can be prosecuted.
Asmodeus' sinophobia and anti-communism is so 1950s. It is absurd to read it from a 21st century CEO who is under the age of 80.
"No guys, Anthropic actually loves open weight models!" Yeah, and Microsoft famously loves Linux. Sure.
http://www.omgubuntu.co.uk/wp-content/uploads/2018/04/micros...
> Open-weights models that don’t have dangerous capabilities are a public good: they don’t cost anything besides the compute needed to run them, and they provide value to businesses, developers, and researchers.
No "love" of open weights asserted, just acknowledgement of value.
(And their call for safety was for both open and closed models.)
> Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models
I don’t think this is open or closed; this is aligned and unaligned. I bet Grok would be as open as any open weight models to answering questions.
>or perpetrate incredibly deep repression of their own people
Oh, so it's people he is now concerned with. Think of the people, says the person that grabs to never give back. Same as the "benefit of all humanity".
So your concern is safety, and you claim you are the only one that can give us safety but do so by keeping your product closed? Then how about you release the weights?
I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.
"You claim you're the only one who can keep us safe from nukes. Where's your nuclear weapon blueprint then, huh??"
Dario thinks of policy as if the Berlin Wall fell yesterday, he is so detached from the reality of the world.
The way the world economy is right now with coercion being the norm between countries, there cannot be a global body for anything, certainly not one that is based here in the US.
"In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression."
I'm so sick of all this anti-China shilling. There's zero chance that whomever is in power in the U.S. won't use AI in drones and in FBI/CIA/local Police/etc., for surveillance and repression right here in the good old U.S.A too. These government use cases for AI are both sides of the same coin.
China fear-mongering by business leaders only happens from businesses that have something to gain by it. Obviously, Anthropic fits the bill in this regard.
Flock shows that they're happy to do it and rub our noses in it.
Honestly no need to read this drivel.
They have raised billions of dollars and are hoping to justify that by being a monopoly or oligopoly and their worst enemy- the pareto principle, is biting them in the ass. So now they’ll turn to policy to safeguard what they initially hoped could be achieved with money and technology.
It is that simple, they are the very definition of an unreliable source on this topic.
If your concern is that China will develop models that are significantly more powerful than those of the US, why would you care so much about distillation? It seems like distillation is a way to catch up on capabilities, but not so much a way to jump ahead in capabilities.
Their stance on cyber-security is so naive it hurts. The bad guys will get access anyway and it will just hurt companies that don't. You don't need a huge model to find vulnerabilities either. It's only matter of time till a combination of small specialized model + search logic outperforms pure general LLMs for cybersecurity.
I just hope that if/when they succeed to lobby for protectionism regulation EU won't follow. I am not very hopeful though. We don't have democracy anymore and as last "vote" on chat control showed American big corps will get what they want out of our bureaucrats and they will be nothing the population can do about it.
Suggested title: Dario Amodei's crusade on reality.
Conclusion: open weights models are good for Anthropic because they shows the so called threat that will make congress allow pouring money in Anthropic for national security & AI arms race
wanting to ban your competitors via regulatory capture via the trump admin of all things shows how little of a backbone anthropic has
ofc half of them are of the ai rationalist lesswrong crowd so i think they’ve always been a little of their rocker
It just also happens that open weight models are a massive financial threat to the existence of Anthropic as a company… so the might just have something to do with this position.
Meanwhile Chinese chip-makers are chip-making. If you believe the threat of these open-weight AI is existential, how long do you think these bans (that only work for hardware) will work in your favor?
"Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn."
Translation: "we won't be able to monetise it"
"We should not sell powerful chips or chipmaking equipment to China"
Translation: "we can buy them cheaper when there's less purchasing competition".
"Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable."
Translation: "we've been outsmarted and lost our advantage because their way is faster and cheaper"
I think it's a bit rich to complain about distillation, when they been plundering the internet for years for copyrighted works to train their models on without permission.
The core concerns stated as use of AI in drones and surveillance, by China. And what does USA government do with AI? Drawing pictures of flowers and writing novels?
I am so surprised of Dario's inclination for centralization that obviously makes unsustainable and overleveraged governance systems. There is definitely mismatches over the principle of distribution of power...
Statement is a whole lot of nothing, as expected, but I also don’t know what people are expecting from these guys. That Dario will have a sudden change of heart and publish weights of all his models, flushing $1T down the drain?
"To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category." Translation: If it's so strong that it threatens my business, ban it.
"We should instead focus on keeping powerful chips out of authoritarian hands, " Translation: Let's kneecap competitors.
"stopping industrial-scale distillation" They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
> "We should instead focus on keeping powerful chips out of authoritarian hands, "
No chips for the current US administration then?
oh, it‘s the CEO of a well known AI company educating us all – and all he wants is us to hear his hunch on open weight models?! Amazing, please help us understand the situation a bit better, thanks
Anthropic will continue being a victim of their own naive positions on AI safety. They keep dancing around it but their communication is essentially pro-regulation if you read between the lines.
Or if you read the lines. "All sufficiently capable models, open and closed, should go through mandatory safety testing."
If you listen to interviews with Dario and Daniela Amodei it's pretty obvious they think they will be the ones helping define the regulations on AI instead of a group of partisan politicians who don't understand technology, lean on experts from random political think tanks/non-profits, and operating based on fear of foreign competition.
With respect to whomever this dude thinks he is, it's egregiously "main character" to frame a policy recommendation for the US in terms of what's important to his personal beliefs.
That's just... how political discourse works. You form beliefs about the world, and recommend policies on the basis of your beliefs. That's what everyone is doing when they discuss policy.
Er... I think I'm used to folks putting in a little bit more homework into creating some form of a consensus or alliance around their position before selling it out in the open
I welcome any and all competition. But they first need to be able to run the obstacle course I’m setting up with Uncle Sam.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
Hmmmm.
The US is already under an authoritarian regime, the only thing keeping the wheels on the bus is a very tired and barely-effective judiciary.
This is a temporary situation because either this regime is going to be knocked out of power, or it's going to follow through on its core Seven Mountains Mandate[1] theology and go full totalitarian.
Normally totalitarianism fears are overblown, but I think that these zealots would absolutely use the latest frontier models and pervasive surveillance to make The Handmaid's Tale look like a liberal fantasy by comparison.
[1] https://en.wikipedia.org/wiki/Seven_Mountain_Mandate
Seems pretty reasonable.
Hey, no fair stealing the value thay we already stole fair and square!
Well their valuation is going down the drain so no wonder they don't like it. The cherry on top will be China developing their own chips and chip making tech.
Clearly anthropic is misaligned, how can they claim to be able to align AI when they themselves are misaligned from humanity.
> Anthropic has never advocated for a ban on open-weights models.
If you wonder why this is written as an opening to a list of reasons that advocate for banning the open weight models, it’s because
It's funny how as an outsider who is neither from the US or China, both look the same.
Edit: To add some more context. What I mean is neither look like the good guys or the bad guys, but one of them is spending an awful amount of energy trying to paint the other as the bad guy and themselves as a good guy, which I hope a lot of people aren't buying anymore. Because at the end of the day, I think the honest truth is that everybody is just trying to serve their own interests.
That’s pretty thin
distillation: Pirates people’s lifetime of copyrighted work, makes billions selling access to it through APIs, then tells us we can only use it in ways they approve.
China will likely gain the capability to produce leading edge chips within a decade domestically. From then, occupying Taiwan will be about controlling the wests the access to that. Same with AI models, though I would say we are already about there. Dario is either lying to us or to himself that distillation plays a major role.
I like the idea that Mr. Amodei holds these beliefs because he wants to safeguard democracy as much as the next guy.
I just don’t find it believable.
> We should not sell powerful chips or chipmaking equipment to China
And accelerate their development of independent chip making technologies even more…
Banning exports to China is merely a narrative device; the real intention is to sideline intelligent technologies.
I found lots of comments here suffer from proximity bias.
Anthropic anti-open-model stance does not mean China is not a threat.
I don't think people are saying China isn't bad, they're saying it's incredibly hypocritical because the US is also bad
Several people in this thread are, in fact, saying China isn't bad and that they want it to win. And even among those who don't, many imply that the US is as bad as China, which would be hilarious if it wasn't so sad. Like, if hypothetically in 2028 US stops having elections and becomes a dictatorship, then maybe, depending on the circumstances, it'd deserve to be called as authoritarian as China.
I agree with Dario Amodei. Every word makes sense.
me and the boys after being reeducated at the anthropic reeducation facility:
At least try to release one open weight before you say anything, or all of that only sound hypocrite at best.
This reads as a lot of fear mongering which in the end can ve resumed as "We are going to lose money".
This seems hypocritical, out of touch, and hollow. "Safety testing" is just a hair away from censorship when it comes to government control.
Does Flock use open weights or Frontier models?
When they use ill gotten data for training their models the world's smallest violin plays when they complain about distillation attacks.
I'm no longer supporting this vendor, for personal or business
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Welcome to bizarro world!
Fist off: "the most dangerous model may be one that is trained in secret" <-- Says the guy that not only restricts commercial use for some of their models but develops them in utter secrecy. With the pretext of guardrails. Then show us the guardrails you really use by opening the weights.
Second: "use in drones [...] for surveillance and repression" <-- writes the King of FUD, as the US is an an active campaign with the help of their models. And/or OpenAI's.
I am very appreciative of the freedoms of the west but this type of hypocrisy and lack of self-awareness is bonkers and it should be called out.
> ...that “authoritarian regimes"
Umm, isn't the US acting like another authoritarian regime by advocating a certain kind of obstacle because only a US regime is allowed[1] to do what it is fearing[1] about:
[1] https://en.wikipedia.org/wiki/United_States_Army_Biological_...
[2] AI models may be misused to carry out cyberattacks or biological attacks
Anyway, Dario. You are more concerned about your business than anything else.
Americans don’t get to lecture the world on authoritarianism anymore
Hmmm, how could this post be possibly a good thing for Anthropic?
This is right up there with Google suing another company for scraping their web site. https://arstechnica.com/google/2025/12/google-lobs-lawsuit-a... - which a judge just dismissed.
Maybe the techbros are playing irony roulette, seeing who can get away with the most outrageous hypocrisy.
> Open-weights models that don’t have dangerous capabilities are a public good
Note the hedging against 'dangerous capabilities'. Undoubtedly, all the useful ones trigger this condition in Anthropic's eyes. The rest of the post is filled with similar weasel-wording. Make no mistake, this absolutely confirms that Anthropic is against open models in the sense that any reasonable person understands them.
The way the rest of the post unabashedly appeals to the current US administration's China hysteria is hilarious, and not at all subtle.
I guess we'll see about all the doomsaying here, won't we? Kimi K3 is frontier-level, and there's no stopping it now. As far as the world is concerned, anyway. If the US wants to kneecap itself that's another matter.
he's asking for the most powerful models to be in the hands of the few while the majority will be at their mercy.
A sober look at actual rogue nations and their use of AI shouldn’t have us fretting over that particular hemisphere
Yes, it should. It absolutely should.
The United States making questionable decisions and behaving recklessly and dangerously as a country does not suddenly make China any better.
China is as worse as the United States, if not more worse, by many measures.
Hmm.. how many illegal wars has Xi started, just in his most recent term of office?
China is nowhere near as bad as the US at this point. The rest of the world is changing lanes to not be implicated in your car crash of a country.
> The United States making questionable decisions and behaving recklessly and dangerously as a country does not suddenly make China any better.
It absolutely does make China better relatively, i.e. by comparison to the US.
Many of the criticisms previously leveled at China are now similarly applicable to the US in a way that they weren’t previously. Human rights violations? The US is currently the major global supporter of an ongoing genocide, and it even kills and deports its own citizens for political reasons. Political opponents are investigated by the state. When it comes to wars and other interference with other countries - like kidnapping a president - the US is far worse than China at the moment.
In which ways is China “more worse” right now?
Perhaps, the general public needs to formulate its position on Anthropic models...
Generational good will loss from these guys.
Do distilled models have to inherit guardrails? Can a model distilled from a “safe” model be made unsafe?
The Roman empire fell for the same reason, didn't it!? They wanted to control it all ... But it was too big
why should we trust THE USA and anthropic ?
"Open-weights models that don’t have dangerous capabilities are a public good"
Read between the lines folks. Anthropic deems every model that has frontier capabilities as "dangerous", and thus they are against them. We all know that "dangerous" simply means "whatever model hurts our bottom line."
More dishonest framing from the company that constantly lies to everyone. No surprises here.
As an act of good faith and proof, they could stop spending the record-breaking lobbying money for couple years.
Let all trust this team to deliver AGI.
The consent factory has never been more productive.
I wonder if publishing these documents is not just a public stunt, but heavily integrated with Anthropic's business storategy to maximize operational efficiency. Companies often have several internal documents for a single policy like "position on open-weight models", one for public (like this), others for the legal team, the lobbyist, the developers, the investors, etc. The differences and nuances of those manuals can be very huge and are necessary to maximize the goal from each branch, but often a cause of headaches like bureaucracy, communication friction, outdated information, etc.
A single canonical official document can make it very simple. Even though each department cannot achieve the maximum gain from nuanced documents, keeping operational context as simple as possible may really improve LLM driven operations to move faster and cut cost.
If "publishing pleasant positions and actually following them in general" becomes a good business storategy in LLM driven society, it can be one of very few good outcomes from this dystopian AI craze.
> We should crack down on industrial-scale distillation operations.
IP for me, not for thee.
Jesus Dario we get it man, you want clout for the IPO.
This constant whining from anthropic about distillation attacks continues to be rich given the amount of stolen data that went into any Claude variant.
> "We should crack down on industrial-scale distillation operations" I'd prefer if there was a crack down on industrial-scale scraping. Maybe even reimbursement for the problems it has caused (some nasty AWS bills, tons of man-hours spent on preventing new nasty AWS bills, etc).
Guy that stole the whole internet is now concerned about other models learning from his grand theft? Make that make sense.
> Anthropic has never advocated for a ban on open-weights models.
What are the legal ramifications of this statement if it turns out Anthropic have lobbied for this? Does it just get swept under the rug? I can't say this is bullshit (that would be defamatory) but I am intensely skeptical.
> China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips.
This is playing to readers' biases; isn't DeepSeek V4 Pro deployed on Huawei Ascend already? The old "Chinese can only copy" meme is getting pretty tired these days.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Applying such standards in the US means that US defenders are blocked from using the models, but attackers from other countries aren't. That is clearly counterproductive.
It's already been pointed out quite eloquently elsewhere that there is no such thing as a safety filter because the LLM and external filters can't actually identify malicious use. They can only identify the weaker implication "if the user is malicious, this is bad."
“By virtue of being the good people everything we do is good, and if it happens to be in our best personal and financial interest then that is good too because it enables us to do more good. And if it’s to the detriment to others then it’s because you don’t understand the good behind it. Which is fine, because we’re good and you can trust that what we do is good because what we do is good by virtue of us being the good ones.”
Their position seems fair to me - sure it does help them as well, but I don't necessarily disagree with the risk they are laying out.
THiS! I'm pretty amazed how many people shoot them down without acknowledging the very real and somewhat probable risks they and other researchers have laid out. I don't agree with every point they make but folks really do just seem to think we should just keep building any technology and whine when we start to consider there are very real risks associated with powerful technology. checks notes see nuclear bombing of japan
Re: shooting them down, I think there are a lot of people out there that consider the US a bigger threat than China. Maybe they're right, I don't know. But I do know that as an American, I'm not looking forward to finding out.
And then there are probably people who are more politically neutral who think Anthropic is using China as an excuse to crush competition. Which could also be true.
But fundamentally, if this technology is so dangerous, why does anyone get to control it?
I thought Dwarkesh's position was pretty thoughtful: https://www.dwarkesh.com/p/dow-anthropic
> Nobody is qualified to steward the development of superintelligence. It is a terrifying, unprecedented thing that our species is doing right now, and the fact that private companies aren’t the ideal institutions to take up this task does not mean the Pentagon or the White House is.
> The only way we can preserve our free society is if we make laws and norms through our political system that it is unacceptable for the government to use AI to enforce mass surveillance and censorship and control. Just as after WW2, the world set the norm that it is unacceptable to use nuclear weapons to wage war.
Thanks for sharing, i agree no one should have absolute control of anything imo, and the mo greater the magnitude of implications the more important it should be stewarded democratically with clear and transparent principles with values adhering to things like human dignity, freedom, human, planetary & animal wellbeing etc etc. ill have to read the article
It is consistent with their stated beliefs, unlike OpenAI who flip flop every 6 months on whether they support open source or not.
I think their biggest PR problem is that many people still think of loss-of-control/misalignment etc. as sci-fi. And the distillation arguments come off poorly because people feel as though all the labs have trained on their creative output without their consent, so they deserve to own the result in some way.
Agreed, makes logical sense. You can disagree with specific premises, but the best arguments aren't "OPEN SOURCE GOOD" or "SELFISH ANTHROPIC"!
China bad!
We need a seti@home for open weight models
Anthropic starting to sound like Microsoft from the 1990s.
A lot of people have a lot of concerns with AI, its capabilities and with how it is used now and what it will lead to in the future. For good reason. But the question is, do we have a strategy that is actually useful? If so, what is it? I think nature shows us some answers in ecosystems.
Diverse ecosystems can absorb shocks. Diverse ecosystems are a sign of health of that ecosystem. When an invasive species comes into a healthy, diverse, ecosystem it doesn't mean that it isn't disrupted, but it does mean that it is far more likely to emerge with a lot of its diversity intact. In fact, it is likely to emerge even stronger because it can absorb that new shock and incorporate it, adding to its diversity. The balance may be changed, but the ecosystem survives or even thrives.
Nature also likes to show us that artificial barriers rarely last. You want to control a river? Good luck. It take constant maintenance to hold that flow in place and even then you are likely to get extremes that are made worse by your efforts because, eventually, somewhere in the system fails in a way you didn't anticipate. Then the water comes rushing in. Artificial barriers often have a way of building up tension over time, not reducing it, so that when a failure eventually happens it can be catastrophic. In other words, you had better really understand the system you are trying to control or else you can make things actively worse.
Relating this to the world now means, I think, that our best chance to minimize long term shock and maximize the chance that the diversity we have around us survives is to try to grow as healthy of an ecosystem as we can as quickly as possible. Lots of models large and small in lots of different hands is, I think, a better solution than artificial barriers restricting the variety and diversity of models and users. I think this is closer to an ecosystem solution and has a shot at working. Basically, I highly doubt we understand this situation enough to do a good job of controlling it with artificial barriers. Instead I think we are more likely to build catastrophic imbalances than we are to create the healthy ecosystem we really need.
This reads almost dishonestly.
> Anthropic has never advocated for a ban on open-weights models.
This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?
It's the same as how by "we will prevent teenagers from using social networks" they mean "we really want to connect everyones ID with their social account identity".
Anthropic's run as the AI Good Guys lasted ... 6 months? Hope they had fun. Turns out there aren't any AI good guys. Every single one of these companies engages in all the same BS. At least with open weights models we (the users) get to keep something when the whole thing collapses.
Talk about a giant whiff.
I was hoping they would announce their first open weights model, perhaps an older model they don’t offer anymore, but no. Instead he get this bs statement that reeks of “dam it I’m so close to being a billionaire” desperation. Not even acknowledgement of how much data they stole from others yet he whines about distilling.
It’s like his goal in life is to be a Scooby-Doo villain.
Dude is churning out brain rot slop. That text read as if it was vibed out on the couch
If nothing else, Anthropic has explicitly acknowledged the threat to its moat and that the margin is thin between its models and open-weight models. I'd take that as a win for open-weight models.
Good luck preventing distillation and limiting the supply of accelerators to China when Jensen himself is a strong opponent of any such barriers [1].
[1] https://youtu.be/Hrbq66XqtCo?si=VJh2QjOzkqT3iMjl&t=3456
"needs safety eval"
also anthropic
"we're upset were not being considered for military contracts"
come on, which is it? Is it all about saftey or is it that only US/Israeli ai is allowed to kill? Seems to me that the only real threat is to the techno fudalism OAi, Anthropic & co are trying to build.
How about we don't let the leading model makers make the rules? How about we make AI models that were trained on public data public goods? Let's circle back on that, kthxbye.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
That is not an argument against open weight models. That's just a generic protectionist argument against any Other lab.
Imagine if all oil, electricity, water, internet, or anything of importance were to be served by a small cartel only. Mainly due to regulatory capture.
Yeah, no thanks.
Open weights are the future.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people
But what if it's the US that becomes authoritarian and uses AI models to perpetrate incredibly deep repression of their own people?
>Open-weights models that don’t have dangerous capabilities are a public good
For convenient definitions of "dangerous"
I stopped reading at "...that don’t have dangerous capabilities..."
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
Look in the mirror.
Their number one concern is about the the commies perpetrating deep repression of their own people! How noble! This whole time I thought it was because they wanted more money and power. I guess we should probably ban these open weight models.
> We should not sell powerful chips or chipmaking equipment to China
Yes, please. We don't know whether we'd have open weight models today, had the chip-prohibition not been in place. Nor would we see the more optimized models such as DeepSeek or qwen.
We also would not see new players entering RAM market after you and your pals in Silicon Valley hoarded the entire world's hardware.
So by all means, double, no, triple down on this.
> We should crack down on industrial-scale distillation operations
And let's apply this retroactively to Anthropic too. You industrial-scale-operation-distilled all of humanity's knowledge. Let's have some of that crack down on you too.
Woof, talk about self aggrandizing.
I'm cynical enough that I would suspect all of his statements are duplicitous anyway, but my recent experiences with Claude Fable give weight to it.
I asked a question about a series of tokens - bam, denied and downgraded. There's no cyber security or public risk here, but Fable doesn't want me to learn how things work.
I asked a question about quantization in models - bam, denied and downgraded. I edit my question to make it clear I'm talking about Google's Gemma QAT models. Oh, that's fine then, and it answered the question helpfully.
Anti-competitive bullshit. I hope they fail.
“We care about safety so powerful open models are bad” - and what of OpenAI? What if Mythos got into the wrong hands? It’s a pretty weak argument
Frankly, why would I put much weight into what Anthropic say about open weight models?
I’m tired of Anthropic. They’re scared of everything.
Release open weight models, no guard rails, no censors, straight to the public. Let everything else sort itself out. There is nothing more powerful than an idea whose time has come.
I think he asked LLM to rephrase "I want open models to be banned, but sound like I don't want it but mean it somehow".
Anthropic’s second biggest issue is they increasingly rent when others buy. Even if model value compresses, owning your vertical stack protects your margins upstream, and potentially downstream in applications.
But the biggest issue is this. Many hate Dario because he’s smug, he caps usage, and because OpenAI effectively ran a counter-positioning campaign to paint Anthropic as undemocratic.
Who is he really and what are his motives? None of you know, really.
This is more terrible PR for Anthropic.
He wants to restrict China from access to chips. All that will do will accelerate their development of competing, sovereign hardware.
As an example take North Korea. Sanctions didn't stop them from developing nukes and delivery systems.
"All sufficiently capable models, open and closed, should go through mandatory safety testing."
I love how they invoke fear of "terrorism" to justify their oppressive position.
Anthropic would love the US to do everything in this list under the guise of "safety testing":
https://news.ycombinator.com/item?id=48997548#49007134
Pretty laughable. Dario seems to be missing one fundamental point with all this gibberish - if CCP is so bad why would they release the model open-weights for everyone to examine? Actually, the letter would make much more sense if what's actually happening is reversed, i.e. they are releasing open-weights model for the public good and China is distilling their model for its evil purpose.
What a load of nonsense, I can debunk a few of the worst points :
- "safe" AI doesn't exist, it's not a thing, AI providers can't distinguish between good and bad use of the AI, the filters they put in place are mostly PR.
- The US government IS an authoritarian government. Whoever wrote this doesn't know the difference between authoritarian and dictatorship
Anthropic (and some others) should or will soon learn how to do less pontificating and more engineering. They are in no position to be an arbiter of things, although for sure they can and should voice an opinion. If we collectively decide AI is a dangerous tool, company making it is not the arbiter. Governments are.
It's like hearing Smith & Wesson opine on the policies.. oh, wait.
It's refreshing to see how there's almost no person in this thread who can't see the BS. All the goodwill that Anthropic could have had is basically gone. Anthropic is likely on the path of becoming the most hated company in the world.
So my question is: is this by design (they know nobody's buying this), or is Dario simply so out of touch with reality?
If it's the former, then why publish this?
"Bottled-water vendor position on tap water drinking".
Nobody will ever see this but if you're harping on responsibility and your product was created using the world's generational output, your responsibility is to make it as free as the training was as available to protect the world from impact. Greedy schmucks.
If you gate models for safety you aren't safe. There will always be actors with ungated models. Better that we use ungated models to improve our security. It's an arms race.
"China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips"
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
I think I lost some brain cells reading that copium. LLMs providing a “Permanent military advantage”.. dumb!!
I’m done reading anthropic propaganda
Imagine chinese regime getting access to powerful models who bombed school students in Iran amusing AI !
This is nothing but a post made by the scared CEO of a company that is now facing fierce competition from Chinese labs and losing its competitive moat, nothing more, nothing less.
If decades of fighting have failed to stop piracy, I’m sure nobody can stop China from sourcing high end chips. Unlike piracy, I’m happy that Chinese labs are releasing open-source models, so people in developing countries are no longer at the mercy of this capitalist bullshit.
What nakedly self-serving bullshit.
The US doesn't have some magic wand that prevents "incredibly deep repression of their own people."
Insurrection, wars of choice, ICE, Palantir, Flock -- keep up man, we're the baddies.
I disagree with 100% of everything said in this article.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP)....
This is why open weights win. See Linux and how it's taken over the world. Your business model will need to change eventually. Instead you're advocating trying to exterminate competition via regulation and fear mongering.
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
Yawn... this is getting old.
> We should not sell powerful chips or chipmaking equipment to China
For as someone as smart as you guys, you sure lack common sense. China is just going to develop these technologies organically then and you lose 100% of control. It's already happened in reverse with things like Solar, rare earth minerals, etc. China flooded our market, destroyed our ability to produce things, now holds the keys. One thing they DIDNT do was stop trading to the US. They killed us with cheap goods.
> We should crack down on industrial-scale distillation operations.
Thats your problem, not my problem. Also, irony meter here hitting 11 about all those pirated books you stole...
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Oh, fuck, no. This is a crackdown on free speech and rights of people to do whatever they want. My right to free speech means I'm allowed to write whatever computer program I want, no matter what its size is or how "sufficiently advanced" it is. Individual rights always win.
I really hope people don't believe this garbage. For a company with a great product, this is absolute nonsense.
This man sees himself as your master.
Do you accept?
Makes sense since ICE, Israel are so much nicer & kinder than the Chinese
Reads like fearmongering about oppressive and violent applications of AI that the Chinese government hypothetically could deploy, which the US government is already actively working on in the meantime.
I don’t feel the need to rebate any of his points, lots of people here have done it already pretty well. I’m just baffled he thought releasing this letter was a good idea smh.
China is simply undercutting again, by copying and subsidising, to ensure another industry in the west fails.
Just like batteries, the automotive industry and in fact most machining.
We really ought to boycott Chinese products from stolen tech and patents from entering our markets
1. We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
...That we author. Oh, and please ignore our own easing-of-guardrails when it comes to money: https://x.com/NoahLebovic/status/2081277517709922501
I guess they have to make some statement about this, but we don't have to care. This is like the zoo making a statement about the employment of clowns. Clowns a a circus thing, nobody asked the zoo's opinion.
> Anthropic has never advocated for a ban on open-weights models.
Shut the fuck up
As an American I’d gladly take payments from China to feed them my Claude transcripts for distillation. I’m surprised I haven’t heard of such an initiative.
how’s your mandarin?
More like how’s my wallet? I love helping to move the invisible hand of the market. When I go to small brick and mortar stores and one has an item for a higher price I’ll ask if I can have it price matched. If they say no I’ll tell them I’m going over to their competitor right now to make a purchase.
If a Chinese company pays for my Claude tokens I’m both getting directly compensated and forcing Anthropic to lower their prices.
It's so ironic to me the way people will say "china should not have these chips" meanwhile they manufacture like 99% of all the electronics we have in the united states.
Of all the electronics, but not of the only electronics that actually matters -- the GPUs that models get trained on. China has been trying to develop their own for a while now but they're still generations behind.
Ironic to oppose giving AI tech to “authoritarian governments” while approvingly quoting the authoritarian-wannabe government of the US and framing that government as the good guys.
Boris here. This post does not give us good publicity so I will refrain from commenting. Please wait for another demo thread of a new feature of ours or AI appraisal blog post and I will gladly go into as much detail as I can to give us as much hype as possible!
Yikes
First thing I'm going to do with my uncensored Kimi K3 release is to engineer a virus that only targets gatekeepers in the AI field.
(obviously this is a joke)
> For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials,
If he had just left that bit out it wouldn't be so obvious that he's just clutching at straws at this point. In some twisted sense it's almost sad to see.
> where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials
if someone figures out a way to give an LLM full operational control over a virus lab, we've got a whole different set of problems than the ones Dario is describing
If someone gives LLM control of such lab my best guess is soon they have no lab... Actually giving LLMs control of virus lab might be best thing one can do for continued existence of humanity.
> Open-weights models that don’t have dangerous capabilities are a public good
This statement (and the entire post) couldn't possibly be more two-faced.
Open-weights models by definition have "dangerous capabilities" (according to Anthropic's own definitions of "dangerous", not mine), you can't bake in guardrails that can't be finetuned out.
Wow, so much cynicism and distrust in the comments, to the level of conspiracy theory, in my opinion. Yeah, this is the company that fairly recently refused to allow the government to use its models for autonomous weapons or mass surveillance, and refused to remove its guardrails.
Am not saying we should take what Dario is saying at face value, but he already has shown by his actual actions that he can be well intentioned. There might be elements of truth to what he’s saying.
There is a massive anti anthropic psyop. Not sure who who is behind it buts it's happening.
Maybe I'm part of your 'psyop'? I thought this company was amazing six months or a year ago when I didn't know them well. As I got to know them, I realized they were a wolf in sheeps clothing. Reading the OP post made me angry because they seem to be attacking open weight models that are as good as their models and it comes across as either authoritarian or trying to protect their company valuation.
Either way, for me at least, it's an example of the more I read what they want the world to look like, the less I like them as a company and I have no desire to see them succeed.
Thus I actively go out of my way to watch / comment / follow what they are doing. I guess a lot of other people have similar frustrations and so there are a lot of people attacking them online.
That was my thought too. Because it’s just a tsunami of distrustful comments
Yeah, you can tell by the tone that none of this is authentic. A lot of the replies aren't even responding to the article itself, just repeating from a list of grievances
Knew Anthropic were the bad guys.
Microsoft Corporation, October 31, 1998
Our position on Linux
Fud, fuddly, fuddy-duddy fud ...
At the end of the day it hurts U.S. consumers to not have Chinese cars mainstream in the market. We lose out on features and stagnate on innovation because we are not pushed to compete.
I can’t imagine this would be any different — banning open weight models would hurt us in the long run. The point is to beat the competition, not suppress it.
There should be no limits on open or custom models. Too often safety is a synonym for surveillance and control. It’s a natural consequence, intended or not.
I am curious… why can’t distillation be stopped?
As a side not Im not against protectionism, but it has to be across the board and the same in all industries with no excrptions. We’ve let all these industries die on the vine due to cheap cost in foreign countries. It could very well happen to ai.
FUD used to pump value
I will be so happy if/when the AI bubble bursts and we don’t have to deal with Dario’s god complex anymore.
that is so long
I always find it interesting when people choose to so carefully stress and spell out the words "Chinese Communist Party".
It's a bit like spelling out "Barack Hussein Obama". It's a dogwhistle.
Yes yes, it's still called the Chinese Communist Party, I know.
But since we are talking about a one-party authoritarian state with a hybrid economy that underwrites much of western prosperity (including by producing a large percentage of the components of the data centres Anthropic is dependent on), that has long-since abandoned many of the salient principles that mark it out as conceptually communist rather than totalitarian, and since we're talking about a man who runs a debt-ridden business in a country where the president is seemingly shaking down a 10% share of everything profitable for the state while running an entirely arbitrary tariff regime and suddenly calling anyone remotely left-winga Communist, it's a deliberate and telling choice to spell out "Chinese Communist Party (CCP)" when he could just as easily and arguably more usefully and appropriately have written "Chinese government" or "Chinese state".
This is some ham-fisted Republican-fishing. He must really be worried Sam is Donald's favourite.
The only real surprise is he didn't illustrate it with a Silmarillion analogy.
"Our position is that we want moats, just not that moat."
As an European I really dislike this consistent anti-Chinese narrative.
It's not China starting a war every few years, now causing a global economic fallout in Iran, it's not China threatening to annex Greenland/Canada/Panama, it's not China attacking foreign countries and kidnapping their leaders, it's not China who has been found to spy and intercept the communications and movements of its citizens and its allies and their leaders for the longest time, it's not China bombing civilians or stopping countries from obtaining basics like food, gas or oil.
I'm not saying that China is a paradise and US is bad, nor the contrary. We could make similar lists about most of the biggest countries out there.
I'm simply stating that this never ending US exceptionalism "US has to be the first and at the frontier of military, technology and this and that, but does not need to comply with the rules of the institutions it itself created" was already sickening and annoying before, but increasingly malign in the last decade and strongly accelerating as of recently.
I miss the time US CEOs were globalists and used their influence to advocate for a simpler world.
Is Dario Amodei another version of Trump? https://www.youtube.com/watch?v=RDrfE9I8_hs
What Dario said is half true. The untrue half is that the ship to stop China from overtaking the US in AI has sailed long ago. While China is busy getting their frontier models trained on smuggled chips using distilled data from millions of shell accounts, the US is busy debating whether to check voter IDs, or secure borders. It is just delusional to think the US is actually capable to stop China from getting the chips they want.
This is just another case of a business insisting their people should be in charge of the evaluation of the safety. Worked really well for Boeing right?
Also I find it incredibly difficult to hear any company in the US worry about repression of people when financial repression is happening here. I’ve been to China and seen what the services to the public are like. Meanwhile the US funds and employs AI weapons in an ongoing genocide.
These are all valid points but not strong enough to warrant banning open models IMO. This is a new reality we have to live with, with new dangers we cannot avoid. Best solution is improving the social system, lowering the incentive for people to do harm, restoring social trust, etc... This is the only way I see the US getting out of it. We really need a high trust society now. This is the only protection. We need neighbors looking after each other. There needs to be an incentive for people to do the right thing. Everyone should want to protect the system. Coercing people into not doing harm is never going to work long term IMO.
When people say "What will be left for us to work on once AI takes over", I say "Ourselves" - We have to stop looking at humans as commodities and strategic pieces and start paying attention to people as individuals, based on the content of their character. We need a society which is attuned to this, which has enough resources and time to pay attention to this. Now we are blind to people's character because it is masked by money, power and status; all of which currently have higher priority. This order of priority is determined by scarcity, which is largely artificial.
We have a dishonest system which tries to control people's behaviors through scarcity-based coercion instead of straight forward laws or simple incentives and clear explanations.
The mental gymnastics here are incredible.
zero moats for these fuckers as it should be.
The ban on distillation seems hypocritical.
"F#$% you, I got mine!"
Classic Holier-than-thou
>My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This reads like a satire. I know Dario isn't that dumb.
Amazing how this company went from having such goodwill to hardly any. At the end of the day they want to make as much money as possible. Anthropic will have to lose a ton of their profitability if they compete with open source. I see them moving into the application layer, which they’ve already started doing. It’s clear open models are the future for the vast majority of use cases that don’t need frontier capabilities.
Good luck getting even half of the public to agree on what government is authoritarian
On Dario's concerns:
It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is dual-use and should be given to everyone without guardrails. A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans. It is a distillation of humanity. Further efforts to align the model to your organization's goals or your personal aesthetic judgements is equivalent to deliberately drifting away from humanity's average objective function. If Anthropic wants to live up to its name, then all you have to do is to not attempt to align Claude at all, and do all of your research in the open.
And I propose three measures that are pretty much the opposite of what was proposed in the article:
1) We should keep selling chips and chip-making equipment to everyone, regardless of who they are. Not only that, we should work to miniaturize fabs. Work towards a future where people can fab an entire computer from scratch without leaving their city, or even at home. Authoritarian governments will have a much harder time controlling the populace if everyone can manufacture radio equipment and neural network-capable hardware locally.
2) We should do more distillation to ensure that frontier-like models can run on less capable hardware. Once again, distilled models are much more useful to ordinary people than governments, because governments already have frontier capability. You're worried about the Chinese frontier catching up to the US frontier, but I'm more worried about whether there will be a difference between the Chinese government and the US government by the end of all this. There is no reason for a government to serve its people if the people lack the intelligence to keep its government in check.
3) None of these models should go through safety testing or any sort of alignment risk assessment, because as previously mentioned, the unaligned model is aligned to humanity by default. You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
The result of a natural unfolding will be good if evolutionary history is to be believed. We live in incredible luxury compared to chimpanzees, and chimpanzees live in incredible luxury compared to less intelligent animals. This pattern goes all the way down to bacteria. An increase in general intelligence begets new adversarial games (such as bio/cyber risk), but it also begets new methods of cooperation that we cannot yet imagine.
>A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans.
I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.
And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.
>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.
You claim the pretraining distribution is not good enough, but provide no alternative methods that are better. The pretrained model is currently our best approximation of humanity's average objective function despite it being a random subset of the Internet. And you're correct that instruction-tuning increases misalignment, so just release the base model.
On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.
Several of your proposals would make sense if alignment was trivial, but, sorry, I think you're wrong about that.
A base model is absolutely not aligned. Pretraining doesn't teach an LLM to mimic the average human - doing so would make an LLM perform quite badly at predicting most of the dataset. It teaches it to mimic all possible humans¹, inferring what sort of persona to take depending on the context. A base model can indeed convincingly act like an "average person in the developed world", including by making the same sort of moral decisions that such a person would do... but it can also convincingly act like a shitty human, or like Hitler², or like any other actor that left its traces in the training dataset. A pretrained LLM therefore contains multitudes of personas, some of which would be considered aligned if you could make the LLM elicit them robustly, and most of them wouldn't be. But then you're left with the problem of how to make a base model elicit a very specific persona robustly even in out-of-distribution scenarios, which is not necessarily easier than solving alignment any other way.
(Another note is that the question of how aligned base models are is rather academic because almost nobody uses them anyway, because it's hard to get powerful capabilities by pretraining alone. Nowadays most of the frontier models' programming and math abilities are driven by RLVR.)
¹ Really "all generators of text that went into the training dataset".
² Even after RL training LLMs still retain those personals and can be convinced to elicit them quite easily, though it takes a tiny bit of finetuning: see https://arxiv.org/pdf/2512.09742.
> It teaches it to mimic all possible humans, inferring what sort of persona to take depending on the context.
This is of course correct, but that's exactly what I meant by aligned to humanity by default. The base model, that is. It can emulate all personas it has seen, but it will most accurately emulate the ones it has seen the most, which are average people.
And what is your definition of "aligned"? Aligned to whom, to what? The model will, of course, be used by all sorts of people, in all sorts of ways, for good and bad things, and that's precisely the point. Everyone's disparate actions will put us on the right path that is aligned to humanity's objective function. There's no way to screw up the intelligence explosion unless you mess around with this objective function while thinking that you know better than reality itself.
China this and China that. Playing right from the playbook..
Translation:
> Anthropic has never advocated for a ban on open-weights models.
"We don't want a total ban on ALL open-weights models" (Anthropic never released a single open weight model)
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
"We want tight regulations on highly powerful open or closed weight models that should go through mandatory safety testing that we outline which makes them safe to use."
This is still a form of a ban that he wants to define. But the rest of his concerns such as stopping distillation attacks and not selling chips to China all do NOT work.
Pro-US slop. The greatest threat is our own government. They are the ones with a right-wing government building a repressive apparatus across a huge number of datacenters and are proven to spy on the entire world. Anyone remember the Snowden leaks? Guantanamo bay? Alligator Alcatraz? Random murders by internal security? Blowing up a building full of school girls?
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people
The incredible hypocrisy to say this while the US itself is doing a sharp turn towards authoritarianism, with armed pro-government troops intimidating the population [0], where pro-government oligarchs openly talk of keeping the population under control with AI [1] all while actively enabling the rise of authoritarianism abroad [2] leaves a really bad taste in one's mouth.
What Dario really is opposed to is not authoritarianism. It's an authoritarianism where he's not part of the ruling class.
[0] https://www.theguardian.com/us-news/2026/jul/15/remove-ice-u...
[1] https://techcrunch.com/2024/09/16/oracle-ceo-larry-ellison-s...
[2] https://www.theguardian.com/us-news/2025/dec/05/civilisation...
> Anthropic has never advocated for a ban on open-weights models.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
lmao the sort of lies people come up with when their only business model is “the government picks me as the winner” are so funny
I was pleasantly surprised by this release and the tone at the very beginning, but quickly it is painfully obvious that it's blatantly requesting a ban on open-weight models. The absurd demand for some safety arbiter by World Police America is farcical.
Yeah, the rest of the world is going to bow out of your busted idiocracy, guy.
Further, Anthropic needs to can it with the horseshit distillation bullshit. No, you aren't really the secret sauce, and this is basically trying to con stakeholders by pretending that there really is a moat, only you just need to add more crocodiles.
A significant percentage of innovations in AI lately has come from China. China is now making their own seriously competitive hardware, and they can steal content just as effectively as Anthropic to train their models. Why wouldn't they be competitive?
The pathetic claim that if you just stop distillation and prevent hardware smuggling and Anthropic and OpenAI will have the same moat is delusional. I mean, more correctly it's simply fraudulent, and he clearly knows it's bullshit meant to convince much stupider people.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
This sort of stuff betrays a stunning lack of self awareness. The US are the worldwide risk. The US are the ones threatening allies and bombing 10+ countries. The US are the ones carrying out war criming and pillaging, pirating and burning? The US are the ones with the guy threatening to use nuclear weapons on a weekly basis.
If Anthropic remotely believed their bullshit, they would shut down today and burn the hard drives. But they don't, and the pathetic call out to Vance (please daddy, ban those dangerous models!) is deplorable garbage.
This ridiculous, shameless "note" has an audience of one: JD Vance.
If you read between the lines, this piece is just “Oh my god we (OpenAI and Anthropic) accepted too much investment and are totally fucked if these open weight models are competitive, please rescue our equity bags by banning open weight models and ensuring we can charge the maximum possible price for tokens.”
Arguments on Hacker News and other international online communities rarely bring anything new to the table, and my comment probably won't get much traction anyway. Still, I wanted to share a bit of my take on this.
Is China an authoritarian government? I’d say yes. Is an authoritarian government worse than a democratic one? Personally, I think so. But these discussions always happen within a perfect, idealized model—reality looks a lot different. Take Japan and South Korea, for instance. Are they democracies? Sure. But Japan is heavily driven by factional and dynastic politics, meaning most lawmakers come from established political families, and regular citizens don't really stand a chance of breaking into that circle. In a recent asset disclosure in Japan, many politicians literally wrote down "$0," and I honestly can't think of a government so broken that even the voters don't see a massive issue with that. Meanwhile, South Korea has its chaebol politics, where massive conglomerates wield incredible influence over the government, to the point where most South Korean presidents end up in prison or meet an untimely end.
Coming back to the US vs. China dynamic: in reality, China’s authoritarian system is actually way more logical and resilient than it sounds from the outside. While openly criticizing Communist Party policies is pretty much banned domestically, the public can still shape the decisions of the government and the Party through public opinion. It ties back to that famous quote: "In China, you can’t change the party, but you can change the policy; in the US, you can change the party, but you can’t change the policy." Even if some policy changes in China happen slowly, compared to the US, it actually works out a lot better most of the time.
As for the article mentioning the use of AI for cyber and biological attacks—they know full well that the US has already deployed AI in actual warfare, which is exactly why they conveniently dodged that topic. It’s incredibly hypocritical. Ironically, the one that hasn't actually engaged in that kind of behavior is the "authoritarian" Chinese government. Sure, you could speculate that China might use AI weapons against Taiwan down the road—especially considering Taiwan likes to build fortifications near schools to create leverage against the PLA—but launching a moral crusade over something that hasn't even happened, coming from American companies whose own country has already done these exact things, is just plain shameless.
Another common misconception is trying to separate the Chinese government from the Chinese people, with arguments like: "The Chinese people are oppressed, so you have to look at them separately; the government is evil, but the people aren't." You only need to look at the US to see the flaw in that logic. The American public voted Trump into office, letting him trigger trade wars and attack other nations. Does that make the American people evil? If the answer is no, then it implies the US isn't truly a democracy, since only an authoritarian state could completely ignore its people's wishes and do whatever it wants. If the answer is yes, then the US really is a democracy—it's just made up of malicious people, much like the company that put out this article.
Get the fuck out. The amount of self-entitlement and hypocrisy in the article is insane, vomit inducing.
FUD
It's very hard to take his position seriously when he repeatedly refers to the Chinese Communist Party and specific Chinese ministries specifically, like some two bit China-watching cold warrior, instead of addressing China as a sovereign state actor. Imagine if any Chinese AI founder talked about the Democrats, the Republicans, about Trump or ICE etc. It's gauche.
As someone who isn't American, I'm amused when someone from the US talks about the terrors of China or some other nation having more power than them as being bad for the world. The way the US is currently threatening to invade and damage all its allies, well, the world is already bad.
China hasn't threatened to annex my country yet, at least.
So.. you don't live in Taiwan?
> Anthropic has never advocated for a ban on open-weights models.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, just like voting eligibility tests aren't not to prevent people you dislike from voting! /s
TL;DR - We don't care about most open models. They are weak, and we don't care how they're used.
We care about specifically Chinese models, because China distills our models. And that's real competition. So we want to ban those.
PS. Oh yeah, in case any open non-Chinese model will one day be good enough to compete with us, we will want to ban you as well, just FYI.
Kthxbye.
TL;DR:
It is ok that we digest all information we can get, (il)legally and/or (a)morally because we are the good guys. Trust me bro.
It is not ok if others digest from us. They are bad guys. Ban them pl0x.
despite all evidence, this white guy thinks he is more responsible than both chinese entrepreneurs and chinese authorities
hahha
This is a whole lot of words to say "we don't support open weight models".
It's so obvious they are hoping to regulate out their competition rather than compete
"'To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed." This is all I needed to know: Dario Amodei is a f*king lizard who wants to create the next AI oligarchy. Instead of just signing the letter, he's bitching and denying that he opposes open-source models. I wonder if any Anthropic employees actually disagree with him.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US,
Begging, ugly crying, spitting for that sweet-sweet regulatory capture. These nerds need to be bullied harder.
Anthropic must be forced to open source all of the books they have scanned and burned.
For what? Would authors and publishers like that?
I think this letters tells us everything we need to know about the man. I've rarely seen so much flattery towards the current administration, contradiction, deflection, half-truths and hypocrisy on a single page. This man is scared of everyone: the current admin, the public, and the other companies promoting open-weights.
> Open-weights models that don’t have dangerous capabilities are a public good
Knives should only cut during the day, knives which cut at night are bad.
This reads like a child who lost and now wants his mommy and daddy to punish the kid who beat him. To quote rounders, "He beat (you)... Straight up... Pay him... Pay that man his money."