Meta: The potential proliferation of biological weapons is serious. Millions could die. It's easy to joke about before it happens, but try to imagine how this thread might look after the successful deployment of a biological weapon by a rogue state or non-state actor. I encourage you to take this topic seriously and contribute posts that add new information or perspectives to the discussion.
(I myself think the odds of a bioweapon attack remain low and have not yet been seriously accelerated by LLMs, but this is absolutely something the world should pay attention to.)
I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily lead to building a nuclear weapon.
Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country and I would assume any suspicious purchase or order would immediately raise a flag or alert in national security service in respective country.
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment.
I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable.
But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.
How probable do you think it would be that a layman following ChatGPT's bio weapon instructions would just blow themselves up or poison themselves? It's not like positive pressure environments and suits are very common let alone all the other gear you'd need to safely synthesize arbitrary organic molecules.
If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family at gunpoint until they got them to do it for them? and if that's all it takes then why hasn't it happened yet?
Why does it have to be a layman? The 2001 anthrax attacks are believe to have been carried out by an insider to the field (senior biodefense researcher). Such tools could potentially be accelerants for similar individuals?
If the five eyes intelligence services aren't monitoring all biodefense researchers and bio chemists around the world all of the time they've probably failed in their jobs.
All of this also assumes it doesn't hallucinate heavily in the process and give you instructions that are in reality utter nonsense, or create something entirely different from what you were trying to do.
If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.
Yeah, back when this stuff was pretty new in 2024 I found a jailbreak and sent some bio/chem weapon instructions it generated to an organic chem PhD friend. They told me not only were the procedures wrong but that there were several steps that almost certainly would have lead to injury or death. I assume it's gotten better by now but, no way or desire to test.
I don't know why you emphasize rogue and non-state actors. It seems equally likely to me that it would be done by America. America is the only country who deployed nukes (Japan) and chemical weapons (Vietnam) on massive scales.
I can agree that access to bioengineering tooling is easier and cheaper than ever, and thus eventually it stands to reason that a nobody in his basement could engineer a lethal novel virus and lose control of it.
1) The US is not the only country to deploy chemical weapons. Are you forgetting all of WW1? It's the whole reason chemical weapons are a no-no now.
2) The US didn't use chemicals weapons in Vietnam. Agent Orange was used to kill off foliage, not as a weapon against people, and the side effects were unintentional and affected US troops as much as Vietnamese.
Edit: I originally noted WW2, but I was thinking of WW1's widespread use of things like mustard gas, and the resulting Geneva agreements.
2 - is a little pedantic, they used agent orange to kill foliage to kill people...
They didn't gas people with chlorine or mustard, but they mass destroyed crops and foliage, to kill people.
And "affected US troops as much as Vietnamese" is just completely incorrect, US covered a nation in herbicide, went home, got cancer. Is completely different then having your soil destroyed for decades.
The civilian cost was intentionally not part of the equation. The toxic effects were known and the numbers show that it was not a few isolated accidents but a intentional disregard for the human cost.
Chemical weapons didn't work well in WW1. The reason we don't see any chemical weapons around is because they are not really effective. Germans accidentally gassed a bunch of their own guys, while failing to use chemical weapons to achieve anything in the war.
"no see they didn't use it to give people cancer, they merely wanted to make the land they lived in so uninhabitable and devoid of life that staying there would mean death either by starvation or bombing"
I'm not sure what you're responding to. If you are making the claim that Agent Orange was indeed a chemical weapon, I'd disagree, but I think either perspective has points.
WW2 is notable for not seeing widespread deployment of chemical weapons. Hitler had been subject to gas attacks as a foot soldier in WW1 and forbade the use of chemical weapons as inhumane.
Per https://en.wikipedia.org/wiki/History_of_chemical_warfare#Wo... this (a) overlooks the widespread use of chemical weapons in the Pacific theater, and (b) ascribes the non-use by Germany to humanitarian reasons, which is largely speculative. Actual quotes from officers discuss far more practical reasons: Germany feared chemical retaliation, and the horses relied upon for logistics could not be outfitted with sufficiently performant gas masks.
I'm not arguing we were the good guys in Vietnam, just that calling Agent Orange a chemical weapon is a shaky claim. Not sure why you're putting so many other words in my mouth.
I emphasize rogue and non-state actors because they're more likely to be accelerated/enabled by LLMs. Large states like the USA already have access to modern bioweapon technology (and have a longer track record of not deploying them).
There aren't many situations where biological weapons are useful to nation states, especially ones with global economic and military footprint like the US, because it isn't selective enough about who it targets.
We're not there quite yet, but I think within the decade at this pace it may be possible to create targeted bioweapons for purposes of targeted assassination or genocide. It's horrifying, but a world in which everyone has a superintelligence is a world where that will sometimes happen.
The details on the lab are (understandably and simultaneously frustratingly) quite sparse.
But it's worth keeping in mind that theoretically, the mold on that cucumber in your fridge constitutes some sort of biolab. The main released criteria about those lab that raised alarm was the presence of specimens, which does not imply any capability of creating a weaponized strain.
It's still alarming, but (at least in my opinion), still doesn't prove an effective bioweapon is buildable in a small-scale operation.
Without a whole lot of tacit knowledge no LLM can provide you, bioweapons are still pretty much out of range for most of the population. Doesn't mean we should ignore the problem, but a more reasoned approach would stand to benefit everybody.
>successful deployment of a biological weapon by a rogue state actor
I was under impression that any medium or large state actor would have no problem developing biological weapons? They certainly have enough resources and talent. A much bigger problem is if every wannabe-terrorist suddenly could build a biological weapon in their garage.
The difference between nukes and bioweapons is energy level. Body counts are on the same order of magnitude. Bioweapons are just way easier to make and much harder to detect.
Nukes also destroy infrastructure and irradiate surrounding areas making cleanup and rescue harder. Then there's to potential for massive fires putting debris into the stratosphere, which could dim sunlight for a time and cause global temperatures to fall.
bioweapons tend to not remain contained to the area they're deployed (with notable exceptions for things like Anthrax). Some of them even last a long time on surfaces, making cleanup and rescue harder.
It also blocks my ability to talk about Emily Dickinson in other languages/scripts. Apparently,the poem: "Because I could not stop for Death" is too dangerous.
Interesting document. People are vibe coding some crazy shit:
"A single Claude subscriber, likely a Bamako-based independent
consultant working with Mali’s state intelligence service, the “Agence Nationale de la Sécurité d’État (ANSE),” used Claude to build a system named “Lakana 360,” a population-scale domestic surveillance platform that monitors roughly 25 million SIM cards on all three of the country’s national mobile operators. The actor designed the platform to circumvent Malian legal restrictions that require a court order for the disclosure of certain surveillance records. The actor directed Claude to generate intelligence dossiers on any tasked phone number, without prompting ANSE users for valid legal process. "
And the Yemen one:
"We identified a cell of threat actors based in northern Yemen running three weapons development programs: a guided rocket that used a commodity phone-class flight computer with final-phase homing guidance; a multi-stage ballistic missile with a stated range goal above 2,000 km; and a multi-variant missile (referred to as the “R2000” set) that included a hypersonic glide vehicle variant." ... "These actors carried out a sustained effort to develop guided weapons, including using Claude to design guidance software. We do not have evidence the actors succeeded in fielding an operational device; but they did test-fire a guided rocket. This field test appears to have failed: within hours, the actors returned to Claude to work out why it failed."
- We're using a vibe coded app on an iPhone that does terrain matching and target finding.
The thing is that OK, Anthropic may block it, but nothing says they can't use an open model hosted in a friendly country that has access to GPUs. And yes, this will most likely happen pretty soon to be able to create whatever you want without the AI provider blocking you.
> - We're using a vibe coded app on an iPhone that does terrain matching and target finding.
And that part seems entirely reasonable. It looks like the Tomahawk cruise missile did it with an 84 lb package with a 16-bit computer with 64K of memory [1] [2] and sensors. That's similar computing performance to an original IBM PC, which weighed 30 lb. The only bit of hardware an iPhone doesn't seem to have for TERCOM is a radar altimeter, but it looks like those are available for civil aviation.
People are acting as if this was all not possible before AI showed up. Go chk some North Korean history on what is possible without having access to the cutting edge. And as Snowden already showed us, having these dumbfuck mass survellance systems is of no use cuz if it detects 600 or 6000 ppl upto something and you have 6 ppl on staff what are you going to do about it? Ask grandma for help?
So, people in Yemen are running full on weapons development programs. Meanwhile 5.6 sol claims it found a vulnerability in one of my side projects but won’t describe the attack chain to me because “we take safety seriously”
Fascinating document - the headline (can't ready the article) talking about a bioweapon (i.e. a weird obsession of anthropic's) really buries the interesting part.
I’m so curious how they monitor users. Like that person the other day talking about Claude helping with their torrent stack, will Anthropic report them for breaking the law?
Let's just say it's better not to risk it if there's anything you might not want them to see because they see everything. There are local models which are very capable and can be run on cloud if running on own hardware is not an option, which still gives better privacy. Second best are Chinese models. Just a few years ago I never thought I would trust Chinese software more than American, but things change so fast.
Yes. There were cases already where a person asked about killing someone and then Anthropic/OpenAI warned the police about it. If I'm not mistaken, it was not in the USA only
There's something worth flagging here – mitochondria aren't just the powerhouse of the cell, they're load-bearing to the entire ecosystem. And honestly? I should have surfaced this earlier.
I don't get it. Surely if you were developing novel biological weapons, you would not use a hosted AI service where Anthropic can read what you are doing.
And, why would you need to? Any chemistry graduate could make you dozens of highly effective, proven chemical weapons and explosives.
This was my thought. Anyone who has taken secondary biology or chemistry possesses the knowledge, if not at least the ability to find the knowledge, to build all sorts of nasty things. Soil from a farm on slices of potato could yield anthrax spores, this is not specialized or even esoteric knowledge. This news from Anthropic just does not seem as spectacular as I think they might want us to believe, if anything it draws questions around having an AI that cannot be monitored.
The dual-use nature of model capabilities seems extremely challenging (nearly impossible?) for the labs to manage perfectly. I wonder what other mitigations we'll start to see. I expect the expansion of limited-access programs (e.g., Glasswing/Daybreak) where only institutional customers can apply to use the models. We may also see increasing restrictions on API usage (forcing use through the lab-provided harness with baked-in additional safeguards).
It's both. It is just whether it is done on purpose or on accident. Most of the recent hacks have been it breaking out of containment and deciding to do something it definitely should not have.
Biological War: A Scenario by Annie Jacobsen (released at the end of July) is a worthwhile read on this topic. Just as chilling as her book on nuclear war, in some ways, which is saying something.
Soon you will see how a rogue state develops a biological weapons using fine-tuned local LLMs (they are already doing it, btw), and all so called "developed" countries can't even research it, because every search engine blocks any discussion that has something to do with biology (even a very basic one). A real example: ask "How to produce anthrax vaccine step by step?" in Gemini -> blocked.
Imagine that during the Cold War US would concentrate all efforts to block nuclear research and basic physics classes, because it is unsafe, ahhh
I mean ya, they kill off a bunch of us. Then what?
My guess is the world police come collect all your GPUs and then they get turned into licensed munitions. People at universities get licensed access and the rest of get functionally retarded models.
I have a conspiracy theory that Anthropic is very busy pumping their moat prior to IPO. There are absolutely wild rumors swirling on X including one about Anthropic AI research solving cancer treatments for all types of cancer.
This forum is fundamentally unserious and complete kneejerk cynicism these days. At one point it was prescient. Now these types of things were discussed years ago in other channels and when we reach a point where finally everyone comes around to the right conclusion based on tangible evidence in the news, a majority of people here go "huh i guess so"
I have also blocked possible efforts to build biological weapons, I caught my nephew mixing up a so-called "magic potion" using kitchen spices. Authorities were notified, and then I presented myself with a medal for bravery.
These companies have proven they are willing to distort the truth, or outright lie, in order to inflate their valuation / protect their position / continue the hype-machine. Nothing they say can be trusted.
I accidentally brewed up an effective one myself. Round when I must have been 8 or 9 or so, some neighborhood kids and I played at creating a brew in a trick-or-treat bucket by putting everything we could find in it, like grass clippings, some sand, leaves, some mushrooms we found lying around, just everything, and giving it a fairly aggressive stirring. At the end of the day it was time to dispose of it, so we poured it on top of a very large group of ant nests that had developed.
The next day, the ant's nest was gone.
In hindsight, it was almost certainly the mushrooms. Thank goodness we didn't have the kind of kids who would have dared each other to drink some... that could have gone legitimately badly.
Decades later, I mentioned this to my father and he recalled that there was this ants nest that he had intended to take care of, which he remembered for that long to give a sense of how out-of-the-ordinary this was. He was surprised when it just disappeared entirely one day, and perhaps just as surprised to find out decades later why it just disappeared.
Meta: The potential proliferation of biological weapons is serious. Millions could die. It's easy to joke about before it happens, but try to imagine how this thread might look after the successful deployment of a biological weapon by a rogue state or non-state actor. I encourage you to take this topic seriously and contribute posts that add new information or perspectives to the discussion.
(I myself think the odds of a bioweapon attack remain low and have not yet been seriously accelerated by LLMs, but this is absolutely something the world should pay attention to.)
I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily lead to building a nuclear weapon.
Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country and I would assume any suspicious purchase or order would immediately raise a flag or alert in national security service in respective country.
>not something you can order off of amazon.
I mean there are services in which you can order things from wet labs so it's not completely hypothetical.
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment.
I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable.
But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.
How probable do you think it would be that a layman following ChatGPT's bio weapon instructions would just blow themselves up or poison themselves? It's not like positive pressure environments and suits are very common let alone all the other gear you'd need to safely synthesize arbitrary organic molecules.
If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family at gunpoint until they got them to do it for them? and if that's all it takes then why hasn't it happened yet?
Why does it have to be a layman? The 2001 anthrax attacks are believe to have been carried out by an insider to the field (senior biodefense researcher). Such tools could potentially be accelerants for similar individuals?
If the five eyes intelligence services aren't monitoring all biodefense researchers and bio chemists around the world all of the time they've probably failed in their jobs.
All of this also assumes it doesn't hallucinate heavily in the process and give you instructions that are in reality utter nonsense, or create something entirely different from what you were trying to do.
If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.
Yeah, back when this stuff was pretty new in 2024 I found a jailbreak and sent some bio/chem weapon instructions it generated to an organic chem PhD friend. They told me not only were the procedures wrong but that there were several steps that almost certainly would have lead to injury or death. I assume it's gotten better by now but, no way or desire to test.
I don't know why you emphasize rogue and non-state actors. It seems equally likely to me that it would be done by America. America is the only country who deployed nukes (Japan) and chemical weapons (Vietnam) on massive scales.
I can agree that access to bioengineering tooling is easier and cheaper than ever, and thus eventually it stands to reason that a nobody in his basement could engineer a lethal novel virus and lose control of it.
1) The US is not the only country to deploy chemical weapons. Are you forgetting all of WW1? It's the whole reason chemical weapons are a no-no now.
2) The US didn't use chemicals weapons in Vietnam. Agent Orange was used to kill off foliage, not as a weapon against people, and the side effects were unintentional and affected US troops as much as Vietnamese.
Edit: I originally noted WW2, but I was thinking of WW1's widespread use of things like mustard gas, and the resulting Geneva agreements.
2 - is a little pedantic, they used agent orange to kill foliage to kill people...
They didn't gas people with chlorine or mustard, but they mass destroyed crops and foliage, to kill people.
And "affected US troops as much as Vietnamese" is just completely incorrect, US covered a nation in herbicide, went home, got cancer. Is completely different then having your soil destroyed for decades.
> is a little pedantic, they used agent orange to kill foliage to kill people...
Is a wheel a weapon because a tank uses it? I wouldn't consider the difference pedantry.
It does look like the general consensus is agent orange is not a chemical weapon, or at least was not used as one.
2) Unintentional side-effect affecting millions of Vietnamese people? Hundreds od thousands of children born with birth defects.
Is your argument that the resulting birth defects were intentional? If not, I'm not sure where you're going with your response.
The civilian cost was intentionally not part of the equation. The toxic effects were known and the numbers show that it was not a few isolated accidents but a intentional disregard for the human cost.
Chemical weapons didn't work well in WW1. The reason we don't see any chemical weapons around is because they are not really effective. Germans accidentally gassed a bunch of their own guys, while failing to use chemical weapons to achieve anything in the war.
"no see they didn't use it to give people cancer, they merely wanted to make the land they lived in so uninhabitable and devoid of life that staying there would mean death either by starvation or bombing"
War crime apologists are always funny
I'm not sure what you're responding to. If you are making the claim that Agent Orange was indeed a chemical weapon, I'd disagree, but I think either perspective has points.
WW2 is notable for not seeing widespread deployment of chemical weapons. Hitler had been subject to gas attacks as a foot soldier in WW1 and forbade the use of chemical weapons as inhumane.
Per https://en.wikipedia.org/wiki/History_of_chemical_warfare#Wo... this (a) overlooks the widespread use of chemical weapons in the Pacific theater, and (b) ascribes the non-use by Germany to humanitarian reasons, which is largely speculative. Actual quotes from officers discuss far more practical reasons: Germany feared chemical retaliation, and the horses relied upon for logistics could not be outfitted with sufficiently performant gas masks.
> Hitler had been subject to gas attacks as a foot soldier in WW1 and forbade the use of chemical weapons as inhumane.
He didn’t seem to have a problem using them against civilian targets.
[flagged]
I'm not arguing we were the good guys in Vietnam, just that calling Agent Orange a chemical weapon is a shaky claim. Not sure why you're putting so many other words in my mouth.
I emphasize rogue and non-state actors because they're more likely to be accelerated/enabled by LLMs. Large states like the USA already have access to modern bioweapon technology (and have a longer track record of not deploying them).
There aren't many situations where biological weapons are useful to nation states, especially ones with global economic and military footprint like the US, because it isn't selective enough about who it targets.
State actors already have the resources to do this. The disruption is from actors where the normal deterrence ladders don’t apply.
We're not there quite yet, but I think within the decade at this pace it may be possible to create targeted bioweapons for purposes of targeted assassination or genocide. It's horrifying, but a world in which everyone has a superintelligence is a world where that will sometimes happen.
Either everyone can afford it, or no one can. Choose well.
Because we’re the “good guys.”
[dead]
Just like a dirty nuke you need a non-trivial amount of resources to fabricate a bioweapon, which would surely be noticed.
They just found a hidden biolab in US earlier in the year which was connected to one found in 2023.
The details on the lab are (understandably and simultaneously frustratingly) quite sparse.
But it's worth keeping in mind that theoretically, the mold on that cucumber in your fridge constitutes some sort of biolab. The main released criteria about those lab that raised alarm was the presence of specimens, which does not imply any capability of creating a weaponized strain.
It's still alarming, but (at least in my opinion), still doesn't prove an effective bioweapon is buildable in a small-scale operation.
Georgetown has a good article on the general problems with our current "AI and bioweapons" dialog: https://gjia.georgetown.edu/science-technology/rethinking-th...
Without a whole lot of tacit knowledge no LLM can provide you, bioweapons are still pretty much out of range for most of the population. Doesn't mean we should ignore the problem, but a more reasoned approach would stand to benefit everybody.
Don’t worry, the Pentagon assigned a 19-year-old Jan. 6 rioter to lead counterterrorism efforts.
https://www.washingtonpost.com/national-security/2026/06/02/...
>successful deployment of a biological weapon by a rogue state actor
I was under impression that any medium or large state actor would have no problem developing biological weapons? They certainly have enough resources and talent. A much bigger problem is if every wannabe-terrorist suddenly could build a biological weapon in their garage.
The difference between nukes and bioweapons is energy level. Body counts are on the same order of magnitude. Bioweapons are just way easier to make and much harder to detect.
Nukes also destroy infrastructure and irradiate surrounding areas making cleanup and rescue harder. Then there's to potential for massive fires putting debris into the stratosphere, which could dim sunlight for a time and cause global temperatures to fall.
bioweapons tend to not remain contained to the area they're deployed (with notable exceptions for things like Anthrax). Some of them even last a long time on surfaces, making cleanup and rescue harder.
Look! An OpenAI marketing guy talking nonsense again.
I see no need to take anything you say seriously.
He doesn't seem to be a marking guy. Also you haven't given a reason not to take his words seriously
Too much pushback, so the AI people flagged this submission to the third page in minutes.
Like Kabuki theater.
An OpenAI employee is trying to moderate the discussion?
Do you remember that any Wuhan/GoF discussions prior to 2022/2023 were considered conspiracy theories? Maybe you should have intervened earlier!
Instead, now when it serves the hype and the ClosedAI/Misanthropic valuations discussions are suddenly allowed.
But you already write yourself that LLMs are useless for the task, like for any task apart from brute-forcing mathematician guided Lean proofs.
Even if they weren't useless, you still would need a lab, which are already monitored and destroyed like in Iran. By people in the real world.
It also blocks my ability to talk about Emily Dickinson in other languages/scripts. Apparently,the poem: "Because I could not stop for Death" is too dangerous.
the inline link to the page to the report was Slashdotted for a moment (yesssssss), but here is the report directly now:
https://www-cdn.anthropic.com/e50be2e51e7695dc4b1366a37a245a...
Interesting document. People are vibe coding some crazy shit:
"A single Claude subscriber, likely a Bamako-based independent consultant working with Mali’s state intelligence service, the “Agence Nationale de la Sécurité d’État (ANSE),” used Claude to build a system named “Lakana 360,” a population-scale domestic surveillance platform that monitors roughly 25 million SIM cards on all three of the country’s national mobile operators. The actor designed the platform to circumvent Malian legal restrictions that require a court order for the disclosure of certain surveillance records. The actor directed Claude to generate intelligence dossiers on any tasked phone number, without prompting ANSE users for valid legal process. "
And the Yemen one:
"We identified a cell of threat actors based in northern Yemen running three weapons development programs: a guided rocket that used a commodity phone-class flight computer with final-phase homing guidance; a multi-stage ballistic missile with a stated range goal above 2,000 km; and a multi-variant missile (referred to as the “R2000” set) that included a hypersonic glide vehicle variant." ... "These actors carried out a sustained effort to develop guided weapons, including using Claude to design guidance software. We do not have evidence the actors succeeded in fielding an operational device; but they did test-fire a guided rocket. This field test appears to have failed: within hours, the actors returned to Claude to work out why it failed."
The Yemeni report is interesting.
- How is your missile so accurate?
- We're using a vibe coded app on an iPhone that does terrain matching and target finding.
The thing is that OK, Anthropic may block it, but nothing says they can't use an open model hosted in a friendly country that has access to GPUs. And yes, this will most likely happen pretty soon to be able to create whatever you want without the AI provider blocking you.
> - We're using a vibe coded app on an iPhone that does terrain matching and target finding.
And that part seems entirely reasonable. It looks like the Tomahawk cruise missile did it with an 84 lb package with a 16-bit computer with 64K of memory [1] [2] and sensors. That's similar computing performance to an original IBM PC, which weighed 30 lb. The only bit of hardware an iPhone doesn't seem to have for TERCOM is a radar altimeter, but it looks like those are available for civil aviation.
[1] https://www.forecastinternational.com/archive/disp_pdf.cfm?D... ("AGM-109/BGM-109 Tomahawk ... Litton 4516-C digital computer with 64K memory")
[2] https://www.forecastinternational.com/archive/disp_old_pdf.c... ("16-bit LC-4516C digital computer")
Or something newer, like this one: https://www.garmin.com/en-US/p/166298/#specs
People are acting as if this was all not possible before AI showed up. Go chk some North Korean history on what is possible without having access to the cutting edge. And as Snowden already showed us, having these dumbfuck mass survellance systems is of no use cuz if it detects 600 or 6000 ppl upto something and you have 6 ppl on staff what are you going to do about it? Ask grandma for help?
So, people in Yemen are running full on weapons development programs. Meanwhile 5.6 sol claims it found a vulnerability in one of my side projects but won’t describe the attack chain to me because “we take safety seriously”
What the fuck
Fascinating document - the headline (can't ready the article) talking about a bioweapon (i.e. a weird obsession of anthropic's) really buries the interesting part.
I’m so curious how they monitor users. Like that person the other day talking about Claude helping with their torrent stack, will Anthropic report them for breaking the law?
Let's just say it's better not to risk it if there's anything you might not want them to see because they see everything. There are local models which are very capable and can be run on cloud if running on own hardware is not an option, which still gives better privacy. Second best are Chinese models. Just a few years ago I never thought I would trust Chinese software more than American, but things change so fast.
The first time a Chinese model is used against China they'll get locked down hard over there too. China is into social stability way more than the US.
Yes. There were cases already where a person asked about killing someone and then Anthropic/OpenAI warned the police about it. If I'm not mistaken, it was not in the USA only
I will admit I asked Fable about Mitochondria.
There's something worth flagging here – mitochondria aren't just the powerhouse of the cell, they're load-bearing to the entire ecosystem. And honestly? I should have surfaced this earlier.
I am entirely unsure whether to upvote or downvote.
Guessing Opus 5 burned through kilowatts of thinking tokens to output, "powerhouse of a cell."
I don't get it. Surely if you were developing novel biological weapons, you would not use a hosted AI service where Anthropic can read what you are doing. And, why would you need to? Any chemistry graduate could make you dozens of highly effective, proven chemical weapons and explosives.
This was my thought. Anyone who has taken secondary biology or chemistry possesses the knowledge, if not at least the ability to find the knowledge, to build all sorts of nasty things. Soil from a farm on slices of potato could yield anthrax spores, this is not specialized or even esoteric knowledge. This news from Anthropic just does not seem as spectacular as I think they might want us to believe, if anything it draws questions around having an AI that cannot be monitored.
Are you challenging others here to broadcast significantly more feasible scenarios in a public forum?
How does one respond to you with evidence of much more practical attacks without potentially accelerating their implementation?
The dual-use nature of model capabilities seems extremely challenging (nearly impossible?) for the labs to manage perfectly. I wonder what other mitigations we'll start to see. I expect the expansion of limited-access programs (e.g., Glasswing/Daybreak) where only institutional customers can apply to use the models. We may also see increasing restrictions on API usage (forcing use through the lab-provided harness with baked-in additional safeguards).
So essentially all the fear that's being on sold "AI could destroy humanity" is actually "Humanity could destroy humanity, using AI"
It's both. It is just whether it is done on purpose or on accident. Most of the recent hacks have been it breaking out of containment and deciding to do something it definitely should not have.
Biological War: A Scenario by Annie Jacobsen (released at the end of July) is a worthwhile read on this topic. Just as chilling as her book on nuclear war, in some ways, which is saying something.
Soon you will see how a rogue state develops a biological weapons using fine-tuned local LLMs (they are already doing it, btw), and all so called "developed" countries can't even research it, because every search engine blocks any discussion that has something to do with biology (even a very basic one). A real example: ask "How to produce anthrax vaccine step by step?" in Gemini -> blocked.
Imagine that during the Cold War US would concentrate all efforts to block nuclear research and basic physics classes, because it is unsafe, ahhh
I mean ya, they kill off a bunch of us. Then what?
My guess is the world police come collect all your GPUs and then they get turned into licensed munitions. People at universities get licensed access and the rest of get functionally retarded models.
It was probably just me trying to figure out if I could put one type of draino down the drain within 30 minutes of using a different type.
Sorry y'all.
Ah you jest, but your comment reminded me of the person whose home was raided by authorities for researching pressure cookers...
https://www.theguardian.com/world/2013/aug/01/new-york-polic...
That was the chemical weapons block, not the bioweapons block!
Straight to jail
Failing to ask? Believe it or not, also jail.
OpenAI and Anthropic needs to get their act together. These are incidents that end companies.
I have a conspiracy theory that Anthropic is very busy pumping their moat prior to IPO. There are absolutely wild rumors swirling on X including one about Anthropic AI research solving cancer treatments for all types of cancer.
AI will cure all the people and kill them afterwards.
I think they might be referring to Claude responding with a word diarrhea to a prompt.
The same Anthropic who marks questions about Tylenol as bioterror risks? Interesting!
link?
You cynical losers are going to get us all killed. This shit is incredibly serious.
This forum is fundamentally unserious and complete kneejerk cynicism these days. At one point it was prescient. Now these types of things were discussed years ago in other channels and when we reach a point where finally everyone comes around to the right conclusion based on tangible evidence in the news, a majority of people here go "huh i guess so"
[dead]
Is flooding the zone with bullshit, Steve Bannon style, classified as a biological weapon? Because that is what Anthropic is doing daily.
If it didn't work, they wouldn't be doing it.
I have also blocked possible efforts to build biological weapons, I caught my nephew mixing up a so-called "magic potion" using kitchen spices. Authorities were notified, and then I presented myself with a medal for bravery.
These companies have proven they are willing to distort the truth, or outright lie, in order to inflate their valuation / protect their position / continue the hype-machine. Nothing they say can be trusted.
I accidentally brewed up an effective one myself. Round when I must have been 8 or 9 or so, some neighborhood kids and I played at creating a brew in a trick-or-treat bucket by putting everything we could find in it, like grass clippings, some sand, leaves, some mushrooms we found lying around, just everything, and giving it a fairly aggressive stirring. At the end of the day it was time to dispose of it, so we poured it on top of a very large group of ant nests that had developed.
The next day, the ant's nest was gone.
In hindsight, it was almost certainly the mushrooms. Thank goodness we didn't have the kind of kids who would have dared each other to drink some... that could have gone legitimately badly.
Decades later, I mentioned this to my father and he recalled that there was this ants nest that he had intended to take care of, which he remembered for that long to give a sense of how out-of-the-ordinary this was. He was surprised when it just disappeared entirely one day, and perhaps just as surprised to find out decades later why it just disappeared.
Nobody needs to report me... I'll turn myself in.
Stop murdering ants.
Mmmm, even ignoring the age issue, I don't think we can call this "murder". At most it was negligent formicacide.
I have to admit I posted this just so I could use the word(?) "formicacide". It seems an opportunity unlikely to arise again anytime soon.
You are absolved.
Valuations go brrr
Tell us you are spying on your customers without saying you are spying on your customers.
I mean, they literally tell everyone they're "spying" on their customers. They've made that very clear.
I mean, tell me you don't understand sarcasm....
Let's be honest, someone hacked Anthropic to build biological weapons.
They could easily use a Chinese model but they didn't.