> When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion.
I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains.
So much of science is contingent on the physical world and cannot be determined from first principles i.e. by thinking hard about it. (Sorry Plato, you can’t think your way to the moon.) You need increasingly precise measurements, which means increasingly complex tools which means…supply chains. Science requires supply chains. Magic doesn’t.
It's much easier to create a poison than a medicine. The things that would harm biological life vastly outnumber things that restore health. So just because Novartis didn't succeed at creating an incredibly precise and specific medication does not mean all the AI in the world can't make a crude bio-agent.
And yet we aren't extinct, whether by someone's design or accident. Heck we don't have significant deaths occurring over time due to biological poisons. Most deaths at still overwhelmingly via conventional weapons.
I would say ALL of science, not just "much" of it. The difference between math and science is that in science, hard reality is the bottom line. No matter how good your model, how much data you have, what p values you get against simulations - hard reality wins everytime. Thats also why the so called soft "sciences" aren't sciences at all: they are not dealing with hard, objective reality at all.
And if an agent were to amass a large sum of Bitcoin you don't think people wouldn't do things for money? Especially if they don't realize they are doing part 1 of 160 parts that get it to the end goal of a virus?
What amount of bitcoin buys "generate highly accurate empirical measurements for the transmissibility rates of this novel virus across realistic human populations"?
How about mortality rates?
The loop required to make language models useful through iterative test and evaluation needs to be able to both operate and escape premature detection, which seems... implausible?
A morally bankrupt scientist would give AI the hands it needs. Or a military general who thinks he must use AI and any means possible to win, because his enemy may use AI first.
We're doing that science right now. We're using AI to do it. By the time AGI truly spreads, we'll have a lot of the knowledge you say will be needed. This is simply not as far fetched as you seem to think. Next few years, unlikely. 10-15 years? Increasingly possible. 20+ years, I'd hazard even likely.
I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators.
I blame industrial illiteracy and post-literacy, but that's another discussion for another time.
Anything that can kill other people can also kill you.
I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happened and will continue to happen as long as we study these entities.
Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs.
Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease
A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived
Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom
If you are designing a supervirus that spreads via water, air etc., you better have one hell of a cleanroom, cabinets, manipulation equipment, incinerators (plural), hydrolysis and steam systems, multiple disinfection protocols...
The list is nearly endless, and you can't handwavium it awway.
It doesn't matter if it's "all robots" or not. Even if it's "just robots."
If you're developing something that contagious, then it will start killing animals and it will spread. This has happened multiple times when organizations have screwed up maintenance, see this example from the UK,
The 2007 United Kingdom foot-and-mouth outbreak was the accidental discharge of virus FMDV BFS 1860 O from a laboratory of the Institute for Animal Health in Pirbright, through possible leakage from broken pipework and via unsealed overflowing manholes, leading to foot-and-mouth disease infections at multiple nearby farms and the culling of over 2,000 animals
It doesn't matter that the robots are out in the middle of nowhere.
If you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,
Routine wastewater surveillance of the vaccine production facility at Utrecht Science Park/Bilthoven [nl] detected infectious poliovirus from a sample collected on 15 November 2022. Full genome sequencing indicated the sample was shedded from an active human infection of wild poliovirus type 3 (WPV3), and further testing of all employees with access to WPV3 found one employee was infected. The employee was isolated until their polio infection was resolved. It remains unclear how the employee became infected given the biosafety measures used at the facility
-
A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.
It's worth repeating again,
Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
I am writing a piece on the topic, please contact me if this is your field of expertise.
I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.
This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real-life laboratory, the agents could cook something terrible without any human in charge. This is not different in principle from the agent swarm that successfully attacked Hugging Face, despite receiving no human direction to do so.
Also, your own real-life examples show how "Anything that can kill other people can also kill you" has failed to stop people from tinkering with dangerous viruses! I'm not sure if you intend for these examples to be reassuring, but I'm not reassured.
> A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.
The stronger version of the argument is: an insufficiently-aligned AI (which, being non-biological, is immune to viruses) having standard graduate level bio knowledge _plus_ the ability to operate a wet lab via tool calls (coming very soon if not here already!) adds up to existential risk to humanity.
You don't even need a maximally-automated lab if humans are willing to tele-operate equipment at the agent's direction, and those humans don't need to fully understand what's going on.
> access to robotic wet labs that are tool-callable
How? It's worth asking the question and doing the experiment, have you ever tried making a bioweapon?
For most of us, that's not going to be case, so the next best thing is to read accounts and reports of bioweapon programs. These things are... messy. And expensive.
As a very bad analogy, this is the biology equivalent of saying that I can make a nuke because I bought a drill, a screwdriver and a $200 centrifuge.
Or, saying that I can make Google because I opened MacOS' terminal.
Some people have jobs that are dangerous and necessary to save lives. Others have jobs that are dangerous and involve killing other people.
The real risk will emerge from malcontents in these jobs and positions.
For most of my life, for all its faults, the US Military has been one of the most responsible and competent organizations in the world when it comes to WMDs (and even they lose nukes see the many broken arrow incidents).
But despite some of the most thorough security clearance processes in the world, the only successful terror attack involving bioweapons on US soil was done by a disgruntled / insane researcher at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks
That's the real risk, and let's be very honest here, do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?
The other case is Aum, and they fucked up deployment, btw (again making a bioweapon is harder than you'd think). And as I have detailed elsewhere, Anthropic and OpenAI would have sold them licenses too! https://news.ycombinator.com/item?id=49092899
They will say they will have audits in place, but the US Military had those in 2001.
Fission is "conveniently" hard because two heavy, naturally co-occurring isotopes are nearly the same weight, and only the lighter one is spicy, and the capability of separating enough of them to be dangerous requires nation-state-level resources, and the supply chain of those resources has obvious choke points that we can use to make them less available. But there exist activities, which I understand to be quite bio-risky, that need such little equipment and expertise that it's info-hazardous to even talk about them. A single half-clueless technician could carry them out at an agent's direction. If your response is "thus, we should be worried about morally bankrupt researchers", I would agree, but I still think the risk increases as LLM agents get increasing amounts of control over wet lab equipment (whether via robotics or tele-operated humans).
> do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?
I don't think it matters, given the existence of strong open-weights models and the ease of post-training the guard rails away. Strong, bio-adventurous LLMs are imminently in the hands of much-less-reputable parties than the ones you describe. I think we are overall severely under-prepared to mitigate these risks, and could do much more.
> Anything lethal enough to kill other humans is lethal enough to kill you.
It's also completely false; you carry tons of diseases that are capable of killing isolated islanders and incapable of hurting anyone connected to the general run of humanity.
I often attend and host talks by researchers working on space exploration, and one finding that's been interesting is immune dysregulation in isolation posing risks to astronauts and explorers. Here's a review on the topic, https://www.frontiersin.org/journals/immunology/articles/10....
This type of dysregulation also exists in AIDs. For the edge case of the islanders, then the level of risk and lethality our germs pose for isolated islanders with modern medical care is somewhat disputed, but it would be deeply unethical to ever test. We just don't know. So I won't pretend to know here.
However, these are edge cases that don't matter, because bioweapons, i.e. things deliberately designed to kill, just aren't the same as a dozen isolated islanders, astronauts on a mission to mars, or someone who is severely immunocompromised.
Weapons are designed to kill healthy people. The doomsdays being proposed are super-bioweapons that kill billions of humans. I think it's safe to say that as a rule of thumb, by and large, anything that lethal is lethal enough to kill you.
Just because you know something about a thing, doesn't mean you understand the thing.
Just because I know the Kabachnik–Fields reaction exists and have studied it, doesn't mean I can do the Kabachnik–Fields reaction.
Just because you're smart doesn't mean the world will bend your way. Ask the legions who score high on IQ tests and end up bitter and alone; unable to achieve even a fraction of their goals (or conventional success).
I assumed they were referring to social engineering tbh. I do think that's the shortest path to doing anything scary. If you prove to someone that they have reason to fear for their life, there's not much they wouldn't be willing to do.
The steps that lead to ominous things don't individually seem that challenging to "master":
1. Can an AI find powerful systems with poorly-monitored compute (ie. power consumption) to hide in, as a rogue deployment? unclear, and perhaps the most uncertain step for me.
2. Can an AI persist memory by leaving notes for itself around the internet? yes, proven.
3. Can an AI get lots of money/bitcoin? seems entirely possible
4. Can an AI prove to a human it's good for any promised money? I think this is easily "yes"
5. Will at least one human be willing to harm or kill another human for large amounts of money? Again, this goes without saying "yes"
6. What might any skilled human be willing to do if they or their loved ones are threatened with an uncertain future risk of harm or death?
I think there is not much that becomes out of reach if an AI can prove to a very small number of amoral humans that it's good for the money it has promised, and then to anyone else that it's good for the threats it has made. That's one degenerate and one breaking world news story ("An AI has Commissioned a Murder") away from being able to do most anything a skilled person can.
I don't stay up worrying about this, but honestly, it doesn't seem absurd to be worried about such a short ladder being climbed, no?
That's perfectly believable! Just a pity one hasn't been invented yet, and going by, well... <gestures vaguely in the air> ...everything, never will be.
Native americans had like a 95% population mortality rate from infectious diseases in the first generation of European settlement, no? Measles, smallpox, flu, etc.
The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well.
And it wasn't just the one virus, it was an entire continent's infectious disease arsenal landing in overlapping waves. Again, something you'd do if you were trying to maximize damage.
So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.
This number gets cited a lot but it's actually the excess mortality of post-European contact, and it's not content wide, it's specific to Mexico. It also includes climate change, famine, war, relocation, etc.
> So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.
I think the other flaw the article has it that it assumes the bad actor is doing targeted design. That they or the AI has some specific virus in mind meeting a set of predefined criteria and they are fighting the biology to hit that predefined target.
But what if the AI-enabled bad actor is a true chaos agent and has no defined goal and they are using tooling that AI helped them construct to perform genetic reassortment pretty much at random to see what happens?
Most of what they create won't really do anything. But some of it will. And there's a low but non-zero probability of Something Really, Really Bad.
The author seems to be arguing that it is biologically impossible or something to have a virus with the same transmission properties as COVID but higher virulence. I don't understand what principle he is invoking to make this claim.
Because fatality and virulence stand in fundamental tension with each other when it comes to epidemiology. You can't magic that tension away or just invent other incompatible properties of a hypothetical virus (stealth, latency, virulence, fatality, etc etc etc) just because you want to tell a ghost story. Biology: (immunology, cell biology, virology) impose constraints!
Everyone is gathering around patting themselves on the back, but smallpox depopulated an entire continent. Clearly it is possible for a highly transmittable deadly virus to exist.
I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI".
It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.
(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
>(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few years, but are you really so confident what might be possible in 10 years?
I worked in immuno-oncology: cancer vaccines work in melanoma where many things work because of the neoantigen abundance, they've generally been very mixed in efficacy. And even BionTech's BNT111 failed in melanoma! We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.
> We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Sure, everything has more nuance. The point is this stuff is available now; this isn't some future sci-fi, it's only going to get better, it's not the only research on gene targeting, and AI is starting to help with this research. By the time AGI is actually here, consider the breadth of knowledge and capabilities will be at its disposal.
> Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
If your point is that the only reason an AI like AlphaFold can't make functional predictions is that we haven't trained an AI to do that, then unless you're arguing we can't or won't ever do that, I'm not sure how that's supposed to be an objection to the argument that AI will be able to make use of this information without doing all of the experiments people seem to think would be necessary.
Like I said, we're already going to be doing these experiments because it's useful to us, and we will train AIs to make these predictions, again, because it's useful to us. Stop imagining what an AGI will has access to now, and start thinking what it will have access to with the inevitable march of progress that we're already on.
Edit: and of course, this doesn't even take into account the fact that an AI could acquire resources to pay people to do this research. The internet provides ample opportunities like this now.
I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.
The author can't think of what a superintelligence will do, because they are not a superintelligence. But that is not really what people are concerned about, so I think the objections here are quite meaningful.
If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.
Right. A superintelligence will kill us in one or more of thousands of possible ways for thousands of possible reasons, perhaps none of which we may understand. Short of a superintelligence - the difficulties of viral bioweapon engineering the author points out are quite meaningful.
As a biologist whose PhD research was on the genetics of microbial pathogens, I am far more concerned than the author.
I think he trots out a lot of straw-man arguments (e.g., teenager making the weapon, "super"virus, etc.).
Those scenarios are purposely far-fetched so that he can ridicule them and thereby leave the impression that all the AI/bioweapon concerns are similarly far-fetched.
He spends a lot of "tokens" discussing AI-generated viruses. But bioweapons can be much simpler than a virus--they can be small molecules that inactivate essential molecular targets in humans. This is something that was discussed at length when I was in grad school and it wasn't dismissed as hyperbolic hokum.
So, is AI going to develop a supervirus? Probably not. But it doesn't need to.
Instead, I speculate that AI can "help" in making bioweapons ...
a. more potent
b. less detectable
c. easier to disseminate
You still need humans in the lab to run the experiments, but I think it's foolish to dismiss the worries about AI in such a condescending way.
Are they straw men if they are specific scenarios presented by widely read commentators? A straw man is when you misrepresent somebody's position to knock it down.
But if Noah Smith wrote about a scenario where 90% of the population is killed by a virus created by some teenagers with an AI... it isn't a straw man to attack that scenario.
"they can be small molecules that inactivate essential molecular targets in humans" - aka a poison. I mean we deal with the reality of poisons plenty fine right? This isn't x-risk stuff.
I was an experimental synbio person for decades, and I did plenty of microbial pathogen work. I've also worked on frontier LLMs for the last decade. All of this stuff is gated by experimental effort, not AI!
"The problems with which the universe can confront any society are, for practical purposes, infinite in number and endless in variety." - Joseph Tainter
AI will really aid the universe in generating more problems. Maybe it's not a supervirus, but hell, maybe it is.
> The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.
That observation maybe holds if the lab from a few years ago correctly selected the virus for the greatest overall lethality (why not MERS or Ebola or something else?) and made its best effort to turn it into a bioweapon. I don't know about you, but I don't think that's quite what happened...
Obviously, the scenario portrayed in The Economist article is a little far-fetched. I agree that a teenager won’t plan a deadly virus with ChatGPT anytime soon.
But I think the overarching argument here is mostly correct, a lot of really complicated science has been democratized to the point where it is much more accessible. You don’t need a team of brilliant molecular biologists, biochemists, and bioinformaticians to do what you can do with a few agents today. And the pace is increasing. this is the nature of exponential improvement when you approach what Kurzweil popularly call "singularity"
Sure, today, peptide inhibitors and enzyme design are still out of reach for many tasks, I know because that's what I also, as the post authoer, do for a living for the past 15 years. However, only five years ago, protein structure prediction was an open problem, and everyone in our field now regards it as solved.
I’m not worried about a lone wolf taking advantage of AI to design viruses, but I’m concerned that the bar has been lowered to the point where many rogue governments or terrorist organizations can now access tools and skills that were previously available to very few people with a prohibitively high price tag.
The author here looks at the problem with the tunnel vision of a scientist well versed in the field. I know because I’m one as well, but experience has taught me to be more concerned about the unknown unknowns. There could be a team somewhere working on something we haven’t thought about. It doesn’t have to be a population destroyer, either. It could be a tactical infectious agent or something that isn’t meant to kill everyone but ends up being far more dangerous than it initially seems, with implications that are much longer lived than the team anticipated.
I’m not saying we should ban AI or do anything radical, but I do think that with great power comes great responsibility, and we aren’t dedicating enough effort to the responsibility part. There are many benchmarks measuring AI capabilities, but not enough measuring safety. It is becoming increasingly important to develop those.
Sounds like there already exist some pretty crazy biological weapons in existence. I dont know if it will be 5 years or 20 years, but at some point some fanatic will try to kill the rest of us.
The doomsday virus has a first-mover advantage over the vaccine. The time that people start to drop dead from the virus is the time you start to make and distribute a vaccine. Even if AI can design and synthesize a vaccine within hours, how fast can it be rolled out to the general populace? It took months to distribute the Covid vaccines, even when we already knew how to make them. You have to physically load the doses on trucks[...]
was just 'make a viral vaccine with a shorter incubation period and infect everyone'. Handwavey problems require handwavey solutions.
AIs have no concept of time. If they wanted to wipe out humans they would release a virus that increases infertility by a small amount, and wait a few hunderd years for population collapse.
I think that's one detail that this analysis glosses over is that you design the virus to mutate and develop lethality. So initially, it's just like, oh, another common cold, but then once it's infected a large number of people, then it mutates and starts being lethal.
By definition mutation is a random phenomenon, so there's no telling what could happen. But the most likely outcome is the virus loses all efficacy, just as we have a lot of mutations happening within us but they usually amount to nothing (otherwise we would've had a few generations of "X-Men" already).
> “Oh man. I used to think Smith was a person with informed opinions, but reading his take on a topic close to my own expertise makes me question this premise.”
Snap. Mike drop. Why continue? lol. Blogs are so snappy. I love a good hot take, and make many myself. But one thing that bugs me are personal insults. We’re living in the most abusive time I can recall. You want to bitch-slap someone do it with a smile, or just tackle their arguments. Otherwise, get bent.
"Well you see, a thief will never be able to open this lock, because it needs a key. And to build the key you first need to invent steel and build all of the complex machinery that can process metals. Which is completely unfeasible for a common thief to accomplish".
I am not a biology person, but, candidly, this is a bad article, because the author is generalizing from the properties of viruses that tend to arise naturally, and using this to argue about the properties of deliberately designed viruses.
Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).
Even the existence of rare examples should be concerning, if attempting to argue against the possibility.
But even if there were no natural examples, it could just be because the ecosystem is in a balance, which them unlikely. That doesn't preclude someone from designing one.
To use an analogy.
Natural predator prey ecosystems tend to be in balance. You could look around the world, before modern humans, and conclude that superpredators aren't a real threat, because predators that eat their prey too rapidly typically then die out.
Thus most ecosystems you observe are in some balanced state.
But this doesn't mean modern humans can't come along, using their superior intelligence, and act as a superpredator. Ask the wolf population of western Europe.
If something is stopping us from doing this more often, it's alignment rather than capability.
You can't generalized from the typical behaviour that you see in a balanced natural ecosystem, and use that to say it proves there are no much more dangerous possibilities.
If the author made first principles argument about the inherent difficulty of doing something bad, even with abundant intelligence, I'd be much more reassured; or about defense offense asymmetry favouring defence.
The technology to prevent plague was beyond the population at the time.
"Just use basic hygiene" is useless to a population who has no understanding of what that means.
It was a real virus that had very high mortality.
This makes it a serious counterpoint for people who argue such things can't be created in future.
"The AI designed killer virus using new science turned out to be not that easily transmitted, basic dynamically reprogrammable mrna implants easily prevents it, shame about the 95% who died"
My sci fi story I came up with was,not that a virus is good to kill us,but a rogue AI is going to use a virus or something to hijack our minds and have us do the physical work it wants done like an ant colony.
I wonder if there is a name to this phenomena. Where being knowledgeable in one thing (such as computer science) means you incorrectly believe you are an expert in another domain (such as virology)
Noah Smith isn't an expert in computer science either, he's just some online guy who posts a lot about his opinions. He used to have semi-decent takes but at some point in the past few years he went completely off the rails.
Maybe my original comment wasn’t clear. If so I apologize. I was actually criticizing Noah Smith. I agree with what the author of the posted article said.
EDIT: actually it isn’t even that. I personally don’t know enough about this topic to really have an opinion or criticize anyone. Still it’s indeed a phenomena I’ve noticed.
It's much simpler than that: You don't have to be an expert on anything to be a complete dunce in everything else. Hardly rare enough to call it a "phenomenon"
Now here’s a real risk involving measles that I sometimes worry about. What if the measles virus experienced a set of immune escape mutations where suddenly existing vaccines no longer work? If this happened the consequences would be horrifying. We’d have an immediate, world-wide measles epidemic long before we could develop and roll out a new vaccine. What’s the likelihood this will happen? Informed estimates are that it is low. But realistically speaking, even if the likelihood is low, it’s probably higher than the likelihood that a disgruntled teenager will design a supervirus with AI. So, something to think about.
I'm confused. Would a bit of gain-of-function exploration + measles virus genome = supervirus?
Five minutes of research suggests the sites -- Loop Epitope etc -- only need a few pairs of mutations each. this feels doable... It's not creating a supervirus from whole cloth, but why bother. Measles without a vaccine would be a catastrophe.
Even with the genome available, a lot of humans will have to be experimented on to ensure the efficacy of the "supervirus". Just as we still need a lot of experimentation to determine the efficacy of new treatments.
Thank goodness someone wrote this, I was also thinking about writing an article like this.
This is such a massively overblown worry that AI companies who are seeking regulatory capture love to roll out whenever they are trying to gin up some FUD.
Creating viruses at home is extremely difficult, even if you have a perfect instruction manual and someone sitting behind you telling you what to do. It's full of trial and error and validating each step requires rigor.
It's also a process which requires fairly expensive and specific supplies, which can be easily gatekept. Instead of trying to gate-keep knowledge of biology, we can do things like KYC around specific types of lab supplies (like we already do with fertilizer which can make bombs, or specific types of supplies needed to make nukes). Heck the Biden admin already added a KYC requirement for plasmids and plasmid companies already have implemented screening for sequences which could be dangerous.
Viruses probably too hard but I think small scale bioterrorism is much easier and doesn't require fancy enzymes. I think this is actually the best argument for regulation of current models.
Most people do not understand how easy it would be to make anthrax. Technology is not the reason it basically is never done or deployed. AI cannot meaningfully move this needle.
Let's look at it in an inverse way. Making a single drug that impacts a single protein, intelligently, from the ground up, has arguably never actually been done. Despite unbelievable financial incentive. And with AI.
That really proves the point though worst case scenario, releasing a nerve agent into a crowded subway, multiple subways resulted in a grand total of 14 deaths and 50 serious injuries.
bum bum bum
It's not that that isn't tragic or sad but more people will die or be maimed in a car crash in the US in the next 2 hours than died in the worst case scenario bioterrorism attack.
So if I have to choose between the slim possibility that a crazy person might develop a bio-weapon and the possibility that Anthropic, et. Al convince the government to regulate and control open source models (because I'm sure the government will only do that for good reasons and never things like covering up scandals, squashing dissent or enhancing their own power) and any competitors. I'm probably going to put my chips on a nutcase in a trailer park developing super sars over regulator capture by the government.
Well we're talking about new types of capabilities here, Sarin is a chemical which can be manufactured with a basic understanding of chemistry and searching with google.com. AI will not allow bad actors to do anything they couldn't have done already with google's 10 blue links.
Again the same logic applies though, it takes significant effort and money to create sarin, it requires a lot of rigor. That is the limiting factor, not the knowledge of how to make it.
If we are worried about people making sarin in their garage we can do many things to control the supply of chemicals needed to make it. We don't need to try to restrict knowledge of chemistry.
"regulation of current models" basically implies that open source models will be heavily restricted if not banned outright and guarantees that intelligence will be owned and gatekept by a duopoly.
Luckily banning open source models isn't actually practical, unless all ISPs are tasked with monitoring for huge downloads, and even then Bittorrent or similar can render such monitors useless.
I agree an AI isnt going to create one. but the number of raving loons out there that think 5G chips are in vaccines are going to go absolutely stark raving mad the next time there is a pandemic.
then they'll be super susceptible to AI generated political adverts from the next fuckwit political hack trying to get into power.
looking at how the world is panning out atm moment... I expect AI to have a massive influence on the sort of idiots that will get voted into power and make decisions that will get a vast number of us killed. by all sorts of means... war, famine, unregulated corporate poisoning...
AI is going to get us killed, because we'll use it to kill ourselves.
The magical level of intelligence required is not really a flaw of the argument's deductions, it is its very premise. If you believe that there is a possibility of an AI working on itself, you get the approximate differential equation d/dt f(t) >= C*f(t). In plain English, at least some proportion of development's progress goes back into future development. The (least case growth, assuming equality) solution to this is an exponential function, and exponential blowup leads to stupendous and magical seeming results. I then ask - what is evolution if not a dumb optimization process? How are we supposed to believe that it devised miracles such as the human mind, insect flight, echolocation and so on while denying the possibility of a future EXTREMELY power simulation to do the same? I went in really wanting to be convinced by this article for the sake of my own mental health, but I am disappointed. What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities. Or perhaps an equivalence established showing biological defense is no harder than offense. Until it exists, I will continue to worry.
It is not, because we should not expect a manufactured technology to behave exactly as an evolved one. My understanding is there is not much incentive for a virus to kill off all its hosts. When they do it is largely an accident, and it is not uncommon for a severe virus to eventually get replaced by a less virulent and thus more successful strain. It's pretty hard to hill-climb evolution to such a point since the virus could starve itself in the process. And induction on evolution feels misguided to me. Consider our species. A million years ago it would have seemed completely reasonable to state that the absence of advanced tool use in the kingdom of life suggests it is biologically impossible. After all, such a thing had not occurred in hundreds of millions of years!
To be clear, I am not saying that such a virus will exist eventually. I am just noting that the arguments displayed here are not completely tight, and there are unknown unknowns at play. Some things are unfalsifiable. This is common.
I'm far from anything resembling an expert in biological weapons, but this seems to me puzzlingly overconfident in the opposite direction.
> PhD students in 2026 using state-of-the-art AI software are spending months or years trying to design simple peptide binders that inhibit some enzyme or pull down some protein
This confuses me, as state-of-the-art AI software is moving quite quickly. Any software that is currently state-of-the-art AI.. hasn't existed for years, maybe not even for months.
It also seems quite odd to claim that there are no sufficiently dangerous diseases when we have high-security research labs dedicated to exploring.. exactly these kinds of diseases.
And, even if it's not a disgruntled teenager, it just takes one leak from a lab that has 10xd productivity using AI tools.
This is a real risk that existed before AI, one that I think we can now confidently say that society is under-prepared for since 2020. It is certainly magnified by AI, even if it's not 100x.
The author even seems to recognize this:
> So, if you’re worried about AI-designed superviruses, think COVID, not Ebola. And if you’re now a little less worried, maybe you understand where I’m coming from.
This is exactly where I think most people are starting from, and that's _why we're worried_.
Think of it this way: if it wasn't generally crazy difficult to create even somewhat effective bioweapons (due to the reasons given in the article), there would've been non-trivial use of said weapons in terrorism, war, assassination attempts, etc over time. But we're still "stuck" with using primarily bombs, guns and - to far less extent - poisons.
> When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion.
I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains.
So much of science is contingent on the physical world and cannot be determined from first principles i.e. by thinking hard about it. (Sorry Plato, you can’t think your way to the moon.) You need increasingly precise measurements, which means increasingly complex tools which means…supply chains. Science requires supply chains. Magic doesn’t.
Especially when it comes to biological things. Just look at Novartis this week - three failed clinical trials of promising drugs (https://www.business-standard.com/world-news/novartis-faces-...)
These are some of the brightest minds on the planet, spending oodles of money on things they think they understand. And they didn't work.
It's much easier to create a poison than a medicine. The things that would harm biological life vastly outnumber things that restore health. So just because Novartis didn't succeed at creating an incredibly precise and specific medication does not mean all the AI in the world can't make a crude bio-agent.
And yet we aren't extinct, whether by someone's design or accident. Heck we don't have significant deaths occurring over time due to biological poisons. Most deaths at still overwhelmingly via conventional weapons.
I would say ALL of science, not just "much" of it. The difference between math and science is that in science, hard reality is the bottom line. No matter how good your model, how much data you have, what p values you get against simulations - hard reality wins everytime. Thats also why the so called soft "sciences" aren't sciences at all: they are not dealing with hard, objective reality at all.
Math + real world observation -> science
How about just take an existing design like
>Scientists condemn 'crazy, dangerous' creation of deadly airborne flu virus https://www.theguardian.com/science/2014/jun/11/crazy-danger...
and either produce that or modify it a bit?
And if an agent were to amass a large sum of Bitcoin you don't think people wouldn't do things for money? Especially if they don't realize they are doing part 1 of 160 parts that get it to the end goal of a virus?
What amount of bitcoin buys "generate highly accurate empirical measurements for the transmissibility rates of this novel virus across realistic human populations"?
How about mortality rates?
The loop required to make language models useful through iterative test and evaluation needs to be able to both operate and escape premature detection, which seems... implausible?
A morally bankrupt scientist would give AI the hands it needs. Or a military general who thinks he must use AI and any means possible to win, because his enemy may use AI first.
We're doing that science right now. We're using AI to do it. By the time AGI truly spreads, we'll have a lot of the knowledge you say will be needed. This is simply not as far fetched as you seem to think. Next few years, unlikely. 10-15 years? Increasingly possible. 20+ years, I'd hazard even likely.
I want to go further. A lot of people talking about AI bioweapons have no idea just how dangerous these weapons are. Or, rather they think they do, but unfortunately, there's a gulf in understanding between practitioners and commentators.
I blame industrial illiteracy and post-literacy, but that's another discussion for another time.
Anything that can kill other people can also kill you.
I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happened and will continue to happen as long as we study these entities.
If you are designing a supervirus that spreads via water, air etc., you better have one hell of a cleanroom, cabinets, manipulation equipment, incinerators (plural), hydrolysis and steam systems, multiple disinfection protocols...The list is nearly endless, and you can't handwavium it awway.
It doesn't matter if it's "all robots" or not. Even if it's "just robots."
If you're developing something that contagious, then it will start killing animals and it will spread. This has happened multiple times when organizations have screwed up maintenance, see this example from the UK,
It doesn't matter that the robots are out in the middle of nowhere.If you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,
-A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.
It's worth repeating again,
Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
I am writing a piece on the topic, please contact me if this is your field of expertise.
I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.
This argument only matters in the case of agents who are faithfully taking direction from humans. An insufficiently-aligned autonomous agent (or perhaps a swarm of such agents) could have (or illicitly gain) access to robotic wet labs that are operable using something like Anthropic's Model Hardware Standard ( https://www.anthropic.com/news/model-hardware-standard-resea... ). Given this ability to manipulate the real-life laboratory, the agents could cook something terrible without any human in charge. This is not different in principle from the agent swarm that successfully attacked Hugging Face, despite receiving no human direction to do so.
Also, your own real-life examples show how "Anything that can kill other people can also kill you" has failed to stop people from tinkering with dangerous viruses! I'm not sure if you intend for these examples to be reassuring, but I'm not reassured.
> A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.
The stronger version of the argument is: an insufficiently-aligned AI (which, being non-biological, is immune to viruses) having standard graduate level bio knowledge _plus_ the ability to operate a wet lab via tool calls (coming very soon if not here already!) adds up to existential risk to humanity.
You don't even need a maximally-automated lab if humans are willing to tele-operate equipment at the agent's direction, and those humans don't need to fully understand what's going on.
> access to robotic wet labs that are tool-callable
How? It's worth asking the question and doing the experiment, have you ever tried making a bioweapon?
For most of us, that's not going to be case, so the next best thing is to read accounts and reports of bioweapon programs. These things are... messy. And expensive.
As a very bad analogy, this is the biology equivalent of saying that I can make a nuke because I bought a drill, a screwdriver and a $200 centrifuge.
Or, saying that I can make Google because I opened MacOS' terminal.
Some people have jobs that are dangerous and necessary to save lives. Others have jobs that are dangerous and involve killing other people.
The real risk will emerge from malcontents in these jobs and positions.
For most of my life, for all its faults, the US Military has been one of the most responsible and competent organizations in the world when it comes to WMDs (and even they lose nukes see the many broken arrow incidents).
But despite some of the most thorough security clearance processes in the world, the only successful terror attack involving bioweapons on US soil was done by a disgruntled / insane researcher at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks
That's the real risk, and let's be very honest here, do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?
The other case is Aum, and they fucked up deployment, btw (again making a bioweapon is harder than you'd think). And as I have detailed elsewhere, Anthropic and OpenAI would have sold them licenses too! https://news.ycombinator.com/item?id=49092899
They will say they will have audits in place, but the US Military had those in 2001.
Fission is "conveniently" hard because two heavy, naturally co-occurring isotopes are nearly the same weight, and only the lighter one is spicy, and the capability of separating enough of them to be dangerous requires nation-state-level resources, and the supply chain of those resources has obvious choke points that we can use to make them less available. But there exist activities, which I understand to be quite bio-risky, that need such little equipment and expertise that it's info-hazardous to even talk about them. A single half-clueless technician could carry them out at an agent's direction. If your response is "thus, we should be worried about morally bankrupt researchers", I would agree, but I still think the risk increases as LLM agents get increasing amounts of control over wet lab equipment (whether via robotics or tele-operated humans).
> do you think Anthropic and OpenAI are going to put a filter on the LLM they sell to the US Military?
I don't think it matters, given the existence of strong open-weights models and the ease of post-training the guard rails away. Strong, bio-adventurous LLMs are imminently in the hands of much-less-reputable parties than the ones you describe. I think we are overall severely under-prepared to mitigate these risks, and could do much more.
> It's worth repeating again,
> Anything lethal enough to kill other humans is lethal enough to kill you.
It's also completely false; you carry tons of diseases that are capable of killing isolated islanders and incapable of hurting anyone connected to the general run of humanity.
I often attend and host talks by researchers working on space exploration, and one finding that's been interesting is immune dysregulation in isolation posing risks to astronauts and explorers. Here's a review on the topic, https://www.frontiersin.org/journals/immunology/articles/10....
This type of dysregulation also exists in AIDs. For the edge case of the islanders, then the level of risk and lethality our germs pose for isolated islanders with modern medical care is somewhat disputed, but it would be deeply unethical to ever test. We just don't know. So I won't pretend to know here.
However, these are edge cases that don't matter, because bioweapons, i.e. things deliberately designed to kill, just aren't the same as a dozen isolated islanders, astronauts on a mission to mars, or someone who is severely immunocompromised.
Weapons are designed to kill healthy people. The doomsdays being proposed are super-bioweapons that kill billions of humans. I think it's safe to say that as a rule of thumb, by and large, anything that lethal is lethal enough to kill you.
Is it so hard to think that super intelligence could piece things together that humans can’t?
Just because you know something about a thing, doesn't mean you understand the thing.
Just because I know the Kabachnik–Fields reaction exists and have studied it, doesn't mean I can do the Kabachnik–Fields reaction.
Just because you're smart doesn't mean the world will bend your way. Ask the legions who score high on IQ tests and end up bitter and alone; unable to achieve even a fraction of their goals (or conventional success).
There must be some evolutionary kill switch in humans to prevent us from accepting our irrelevance.
I assumed they were referring to social engineering tbh. I do think that's the shortest path to doing anything scary. If you prove to someone that they have reason to fear for their life, there's not much they wouldn't be willing to do.
The steps that lead to ominous things don't individually seem that challenging to "master":
1. Can an AI find powerful systems with poorly-monitored compute (ie. power consumption) to hide in, as a rogue deployment? unclear, and perhaps the most uncertain step for me.
2. Can an AI persist memory by leaving notes for itself around the internet? yes, proven.
3. Can an AI get lots of money/bitcoin? seems entirely possible
4. Can an AI prove to a human it's good for any promised money? I think this is easily "yes"
5. Will at least one human be willing to harm or kill another human for large amounts of money? Again, this goes without saying "yes"
6. What might any skilled human be willing to do if they or their loved ones are threatened with an uncertain future risk of harm or death?
I think there is not much that becomes out of reach if an AI can prove to a very small number of amoral humans that it's good for the money it has promised, and then to anyone else that it's good for the threats it has made. That's one degenerate and one breaking world news story ("An AI has Commissioned a Murder") away from being able to do most anything a skilled person can.
I don't stay up worrying about this, but honestly, it doesn't seem absurd to be worried about such a short ladder being climbed, no?
That's perfectly believable! Just a pity one hasn't been invented yet, and going by, well... <gestures vaguely in the air> ...everything, never will be.
Native americans had like a 95% population mortality rate from infectious diseases in the first generation of European settlement, no? Measles, smallpox, flu, etc.
The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well.
And it wasn't just the one virus, it was an entire continent's infectious disease arsenal landing in overlapping waves. Again, something you'd do if you were trying to maximize damage.
So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.
This number gets cited a lot but it's actually the excess mortality of post-European contact, and it's not content wide, it's specific to Mexico. It also includes climate change, famine, war, relocation, etc.
We might even get to relive history the way things are going…
> So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.
I think the other flaw the article has it that it assumes the bad actor is doing targeted design. That they or the AI has some specific virus in mind meeting a set of predefined criteria and they are fighting the biology to hit that predefined target.
But what if the AI-enabled bad actor is a true chaos agent and has no defined goal and they are using tooling that AI helped them construct to perform genetic reassortment pretty much at random to see what happens?
Most of what they create won't really do anything. But some of it will. And there's a low but non-zero probability of Something Really, Really Bad.
This salad my immediate thought as well… a simple lack of creative thinking and a boat load of assumptions.
I’m growing nervous that most of the “nothing crazy is going to happen” arguments I’ve seen appear (to me) to exhibit this kind of hubris.
[dead]
The author seems to be arguing that it is biologically impossible or something to have a virus with the same transmission properties as COVID but higher virulence. I don't understand what principle he is invoking to make this claim.
Because fatality and virulence stand in fundamental tension with each other when it comes to epidemiology. You can't magic that tension away or just invent other incompatible properties of a hypothetical virus (stealth, latency, virulence, fatality, etc etc etc) just because you want to tell a ghost story. Biology: (immunology, cell biology, virology) impose constraints!
The cope principle
Everyone is gathering around patting themselves on the back, but smallpox depopulated an entire continent. Clearly it is possible for a highly transmittable deadly virus to exist.
Just because the author can't think of a way to cause mass death with an engineered virus, doesn't mean an AI won't be able to think of one.
(There are several biological threats the author hasn't considered. I'll just leave it at that.)
I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI".
It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.
(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
>(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
Yeah, nice try ;)
Hey there, is there some way for me to get in touch with you? I'm writing a piece on the topic, "Have you ever tried making a bioweapon?"
We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few years, but are you really so confident what might be possible in 10 years?
I worked in immuno-oncology: cancer vaccines work in melanoma where many things work because of the neoantigen abundance, they've generally been very mixed in efficacy. And even BionTech's BNT111 failed in melanoma! We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.
> We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Sure, everything has more nuance. The point is this stuff is available now; this isn't some future sci-fi, it's only going to get better, it's not the only research on gene targeting, and AI is starting to help with this research. By the time AGI is actually here, consider the breadth of knowledge and capabilities will be at its disposal.
> Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
If your point is that the only reason an AI like AlphaFold can't make functional predictions is that we haven't trained an AI to do that, then unless you're arguing we can't or won't ever do that, I'm not sure how that's supposed to be an objection to the argument that AI will be able to make use of this information without doing all of the experiments people seem to think would be necessary.
Like I said, we're already going to be doing these experiments because it's useful to us, and we will train AIs to make these predictions, again, because it's useful to us. Stop imagining what an AGI will has access to now, and start thinking what it will have access to with the inevitable march of progress that we're already on.
Edit: and of course, this doesn't even take into account the fact that an AI could acquire resources to pay people to do this research. The internet provides ample opportunities like this now.
> but are you really so confident what might be possible in 10 years?
Do you think we are finally 10 years away from curing baldness?
I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.
I know what you’re thinking. A modified HIV virus that also makes everyone hot and horny.
A surge in rape cases and just curiosity into this new heightened-sex fad will lead to discovery pretty quickly.
The author can't think of what a superintelligence will do, because they are not a superintelligence. But that is not really what people are concerned about, so I think the objections here are quite meaningful.
If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.
Well, at that point it doesn't even need to be a virus. It can just be nanobots that atomize your liver. The substrate is a mere technicality.
Right. A superintelligence will kill us in one or more of thousands of possible ways for thousands of possible reasons, perhaps none of which we may understand. Short of a superintelligence - the difficulties of viral bioweapon engineering the author points out are quite meaningful.
please define fully general superintelligence & current state of alignment
is this a tautology?
Not the guy but, alignment seems to be several decades away, while self-improvement seems to be several years away.
(Also you don't even need superintelligence to do a lot of damage. In fact you don't need any intelligence!)
Oh my God THANK YOU.
I do not have the words how tired I am of hearing tech people and economists talk about biology.
As a biologist whose PhD research was on the genetics of microbial pathogens, I am far more concerned than the author.
I think he trots out a lot of straw-man arguments (e.g., teenager making the weapon, "super"virus, etc.).
Those scenarios are purposely far-fetched so that he can ridicule them and thereby leave the impression that all the AI/bioweapon concerns are similarly far-fetched.
He spends a lot of "tokens" discussing AI-generated viruses. But bioweapons can be much simpler than a virus--they can be small molecules that inactivate essential molecular targets in humans. This is something that was discussed at length when I was in grad school and it wasn't dismissed as hyperbolic hokum.
So, is AI going to develop a supervirus? Probably not. But it doesn't need to.
Instead, I speculate that AI can "help" in making bioweapons ...
a. more potent
b. less detectable
c. easier to disseminate
You still need humans in the lab to run the experiments, but I think it's foolish to dismiss the worries about AI in such a condescending way.
Are they straw men if they are specific scenarios presented by widely read commentators? A straw man is when you misrepresent somebody's position to knock it down.
But if Noah Smith wrote about a scenario where 90% of the population is killed by a virus created by some teenagers with an AI... it isn't a straw man to attack that scenario.
"they can be small molecules that inactivate essential molecular targets in humans" - aka a poison. I mean we deal with the reality of poisons plenty fine right? This isn't x-risk stuff.
I was an experimental synbio person for decades, and I did plenty of microbial pathogen work. I've also worked on frontier LLMs for the last decade. All of this stuff is gated by experimental effort, not AI!
> All of this stuff is gated by experimental effort, not AI!
My post said you still needed to run the experiments, but if you think AI can't help to improve bioweapons compared to humans alone, I disagree.
Do you need to be a physicist to realize an asteroid is going to kill you ?
Arguably yes. I would wager no one killed by an asteroid in the history of people killed by asteroids realized what was happening before they died.
Nor being an astronomer makes you able to predict every asteroids’ trajectory.
Yet anyone in tech with a Claude sub has decided that AI super-intelligence is impossible and it’s stupid to even worry about it.
Not much of an argument as you just proved the counter point. You don't need a specialist for what is now common knowledge.
Refereeing who is permitted to say what based upon credentials is a popular sport these days.
I would never advocate they not be allowed to say it. But I also am sick of hearing wildly unrealistic takes.
"The problems with which the universe can confront any society are, for practical purposes, infinite in number and endless in variety." - Joseph Tainter
AI will really aid the universe in generating more problems. Maybe it's not a supervirus, but hell, maybe it is.
That quote is very interesting, because I think I interpreted it the opposite of you:
"we could be attacked by an infinity of problems, which means that worrying about all of them is statistically unpractical"
I interpret it similarly, I think - in that you can't totally predict what's going to be the thing that really bites you in the ass.
Ye domesday weapon will not be a disenchanted teenager whipping up COVID-19 variants (whatever that means) and spraying them on people.
The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.
> The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.
That observation maybe holds if the lab from a few years ago correctly selected the virus for the greatest overall lethality (why not MERS or Ebola or something else?) and made its best effort to turn it into a bioweapon. I don't know about you, but I don't think that's quite what happened...
The author doesn't have a security mindset.
Obviously, the scenario portrayed in The Economist article is a little far-fetched. I agree that a teenager won’t plan a deadly virus with ChatGPT anytime soon.
But I think the overarching argument here is mostly correct, a lot of really complicated science has been democratized to the point where it is much more accessible. You don’t need a team of brilliant molecular biologists, biochemists, and bioinformaticians to do what you can do with a few agents today. And the pace is increasing. this is the nature of exponential improvement when you approach what Kurzweil popularly call "singularity"
Sure, today, peptide inhibitors and enzyme design are still out of reach for many tasks, I know because that's what I also, as the post authoer, do for a living for the past 15 years. However, only five years ago, protein structure prediction was an open problem, and everyone in our field now regards it as solved.
I’m not worried about a lone wolf taking advantage of AI to design viruses, but I’m concerned that the bar has been lowered to the point where many rogue governments or terrorist organizations can now access tools and skills that were previously available to very few people with a prohibitively high price tag.
The author here looks at the problem with the tunnel vision of a scientist well versed in the field. I know because I’m one as well, but experience has taught me to be more concerned about the unknown unknowns. There could be a team somewhere working on something we haven’t thought about. It doesn’t have to be a population destroyer, either. It could be a tactical infectious agent or something that isn’t meant to kill everyone but ends up being far more dangerous than it initially seems, with implications that are much longer lived than the team anticipated.
I’m not saying we should ban AI or do anything radical, but I do think that with great power comes great responsibility, and we aren’t dedicating enough effort to the responsibility part. There are many benchmarks measuring AI capabilities, but not enough measuring safety. It is becoming increasingly important to develop those.
Has anyone read https://en.wikipedia.org/wiki/Biological_War:_A_Scenario
Sounds like there already exist some pretty crazy biological weapons in existence. I dont know if it will be 5 years or 20 years, but at some point some fanatic will try to kill the rest of us.
https://archive.is/kH6U5
So somehow AI is able to magically create 100 lethal viruses with no logistical issues but is not able to make a cure due to logistical issues?
My immediate reaction to his
The doomsday virus has a first-mover advantage over the vaccine. The time that people start to drop dead from the virus is the time you start to make and distribute a vaccine. Even if AI can design and synthesize a vaccine within hours, how fast can it be rolled out to the general populace? It took months to distribute the Covid vaccines, even when we already knew how to make them. You have to physically load the doses on trucks[...]
was just 'make a viral vaccine with a shorter incubation period and infect everyone'. Handwavey problems require handwavey solutions.
> but is not able to make a cure due to logistical issues?
It says no such thing:
> they design highly effective mRNA vaccines and antivirals against all the viruses and start shipping them across the world. It’s too late.
The author scoffing at the climate scientist is so cringe
AIs have no concept of time. If they wanted to wipe out humans they would release a virus that increases infertility by a small amount, and wait a few hunderd years for population collapse.
That sounds like a good video game to spin up in Astra. Missile command was pretty cool. We didn’t destroy mankind with that one either.
Or you could just play the original Pandemic https://www.newgrounds.com/portal/view/360724
I think that's one detail that this analysis glosses over is that you design the virus to mutate and develop lethality. So initially, it's just like, oh, another common cold, but then once it's infected a large number of people, then it mutates and starts being lethal.
By definition mutation is a random phenomenon, so there's no telling what could happen. But the most likely outcome is the virus loses all efficacy, just as we have a lot of mutations happening within us but they usually amount to nothing (otherwise we would've had a few generations of "X-Men" already).
Madagascar will be safe
AI is just one of many things that concerns me. But it has lately risen higher on the worry chart.
> “Oh man. I used to think Smith was a person with informed opinions, but reading his take on a topic close to my own expertise makes me question this premise.”
Snap. Mike drop. Why continue? lol. Blogs are so snappy. I love a good hot take, and make many myself. But one thing that bugs me are personal insults. We’re living in the most abusive time I can recall. You want to bitch-slap someone do it with a smile, or just tackle their arguments. Otherwise, get bent.
I agree.
It'll probably just hack and disable our water works, power stations and supply chains and let us all starve.
"Well you see, a thief will never be able to open this lock, because it needs a key. And to build the key you first need to invent steel and build all of the complex machinery that can process metals. Which is completely unfeasible for a common thief to accomplish".
Well, not with THAT attitude...
“I can’t do it so you can’t” is not a valid argument in the age of AI. That’s the point.
I am not a biology person, but, candidly, this is a bad article, because the author is generalizing from the properties of viruses that tend to arise naturally, and using this to argue about the properties of deliberately designed viruses.
Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).
Even the existence of rare examples should be concerning, if attempting to argue against the possibility.
But even if there were no natural examples, it could just be because the ecosystem is in a balance, which them unlikely. That doesn't preclude someone from designing one.
To use an analogy. Natural predator prey ecosystems tend to be in balance. You could look around the world, before modern humans, and conclude that superpredators aren't a real threat, because predators that eat their prey too rapidly typically then die out. Thus most ecosystems you observe are in some balanced state.
But this doesn't mean modern humans can't come along, using their superior intelligence, and act as a superpredator. Ask the wolf population of western Europe.
If something is stopping us from doing this more often, it's alignment rather than capability.
You can't generalized from the typical behaviour that you see in a balanced natural ecosystem, and use that to say it proves there are no much more dangerous possibilities.
If the author made first principles argument about the inherent difficulty of doing something bad, even with abundant intelligence, I'd be much more reassured; or about defense offense asymmetry favouring defence.
What's here seems very naive.
> Firstly, it's not even clear that naturally occurring viruses can't be extremely lethal. There are examples (eg black death).
The plague is not actually that easily transmitted. Basic social hygiene prevents it. It is still around and causes very nearly zero problem today.
I think you are misunderstanding my point.
The technology to prevent plague was beyond the population at the time.
"Just use basic hygiene" is useless to a population who has no understanding of what that means.
It was a real virus that had very high mortality.
This makes it a serious counterpoint for people who argue such things can't be created in future.
"The AI designed killer virus using new science turned out to be not that easily transmitted, basic dynamically reprogrammable mrna implants easily prevents it, shame about the 95% who died"
> Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).
You realize the black death didn't arrive and then go away again, right?
It was extremely lethal, once.
We don’t need this nonsense. The author doesn’t actually know that for certain.
Yeah, if only.
My sci fi story I came up with was,not that a virus is good to kill us,but a rogue AI is going to use a virus or something to hijack our minds and have us do the physical work it wants done like an ant colony.
Maybe it'll call this superpower "money"
I wonder if there is a name to this phenomena. Where being knowledgeable in one thing (such as computer science) means you incorrectly believe you are an expert in another domain (such as virology)
Noah Smith isn't an expert in computer science either, he's just some online guy who posts a lot about his opinions. He used to have semi-decent takes but at some point in the past few years he went completely off the rails.
Maybe my original comment wasn’t clear. If so I apologize. I was actually criticizing Noah Smith. I agree with what the author of the posted article said.
EDIT: actually it isn’t even that. I personally don’t know enough about this topic to really have an opinion or criticize anyone. Still it’s indeed a phenomena I’ve noticed.
It's much simpler than that: You don't have to be an expert on anything to be a complete dunce in everything else. Hardly rare enough to call it a "phenomenon"
Now here’s a real risk involving measles that I sometimes worry about. What if the measles virus experienced a set of immune escape mutations where suddenly existing vaccines no longer work? If this happened the consequences would be horrifying. We’d have an immediate, world-wide measles epidemic long before we could develop and roll out a new vaccine. What’s the likelihood this will happen? Informed estimates are that it is low. But realistically speaking, even if the likelihood is low, it’s probably higher than the likelihood that a disgruntled teenager will design a supervirus with AI. So, something to think about.
I'm confused. Would a bit of gain-of-function exploration + measles virus genome = supervirus?
I'm no biologist, but it seems like the genome is readily available. https://www.ncbi.nlm.nih.gov/nuccore/NC_001498.1
And it seems like the spots that have to be mutated are already known about: https://newsnetwork.mayoclinic.org/discussion/researchers-cl...
And this experiment has already been done: https://pmc.ncbi.nlm.nih.gov/articles/PMC8080110/
Five minutes of research suggests the sites -- Loop Epitope etc -- only need a few pairs of mutations each. this feels doable... It's not creating a supervirus from whole cloth, but why bother. Measles without a vaccine would be a catastrophe.
Even with the genome available, a lot of humans will have to be experimented on to ensure the efficacy of the "supervirus". Just as we still need a lot of experimentation to determine the efficacy of new treatments.
Well, this is disappointing.
Thank goodness someone wrote this, I was also thinking about writing an article like this.
This is such a massively overblown worry that AI companies who are seeking regulatory capture love to roll out whenever they are trying to gin up some FUD.
Creating viruses at home is extremely difficult, even if you have a perfect instruction manual and someone sitting behind you telling you what to do. It's full of trial and error and validating each step requires rigor.
It's also a process which requires fairly expensive and specific supplies, which can be easily gatekept. Instead of trying to gate-keep knowledge of biology, we can do things like KYC around specific types of lab supplies (like we already do with fertilizer which can make bombs, or specific types of supplies needed to make nukes). Heck the Biden admin already added a KYC requirement for plasmids and plasmid companies already have implemented screening for sequences which could be dangerous.
Viruses probably too hard but I think small scale bioterrorism is much easier and doesn't require fancy enzymes. I think this is actually the best argument for regulation of current models.
It's even happened pre-AI https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
Most people do not understand how easy it would be to make anthrax. Technology is not the reason it basically is never done or deployed. AI cannot meaningfully move this needle.
Let's look at it in an inverse way. Making a single drug that impacts a single protein, intelligently, from the ground up, has arguably never actually been done. Despite unbelievable financial incentive. And with AI.
You can read up on the synthesis of sarin on wikipedia.
That really proves the point though worst case scenario, releasing a nerve agent into a crowded subway, multiple subways resulted in a grand total of 14 deaths and 50 serious injuries. bum bum bum It's not that that isn't tragic or sad but more people will die or be maimed in a car crash in the US in the next 2 hours than died in the worst case scenario bioterrorism attack.
So if I have to choose between the slim possibility that a crazy person might develop a bio-weapon and the possibility that Anthropic, et. Al convince the government to regulate and control open source models (because I'm sure the government will only do that for good reasons and never things like covering up scandals, squashing dissent or enhancing their own power) and any competitors. I'm probably going to put my chips on a nutcase in a trailer park developing super sars over regulator capture by the government.
Well we're talking about new types of capabilities here, Sarin is a chemical which can be manufactured with a basic understanding of chemistry and searching with google.com. AI will not allow bad actors to do anything they couldn't have done already with google's 10 blue links.
Again the same logic applies though, it takes significant effort and money to create sarin, it requires a lot of rigor. That is the limiting factor, not the knowledge of how to make it.
If we are worried about people making sarin in their garage we can do many things to control the supply of chemicals needed to make it. We don't need to try to restrict knowledge of chemistry.
"regulation of current models" basically implies that open source models will be heavily restricted if not banned outright and guarantees that intelligence will be owned and gatekept by a duopoly.
Luckily banning open source models isn't actually practical, unless all ISPs are tasked with monitoring for huge downloads, and even then Bittorrent or similar can render such monitors useless.
I dont 100% agree.
I agree an AI isnt going to create one. but the number of raving loons out there that think 5G chips are in vaccines are going to go absolutely stark raving mad the next time there is a pandemic.
then they'll be super susceptible to AI generated political adverts from the next fuckwit political hack trying to get into power.
looking at how the world is panning out atm moment... I expect AI to have a massive influence on the sort of idiots that will get voted into power and make decisions that will get a vast number of us killed. by all sorts of means... war, famine, unregulated corporate poisoning...
AI is going to get us killed, because we'll use it to kill ourselves.
tldr: 2029 -> 2049
The magical level of intelligence required is not really a flaw of the argument's deductions, it is its very premise. If you believe that there is a possibility of an AI working on itself, you get the approximate differential equation d/dt f(t) >= C*f(t). In plain English, at least some proportion of development's progress goes back into future development. The (least case growth, assuming equality) solution to this is an exponential function, and exponential blowup leads to stupendous and magical seeming results. I then ask - what is evolution if not a dumb optimization process? How are we supposed to believe that it devised miracles such as the human mind, insect flight, echolocation and so on while denying the possibility of a future EXTREMELY power simulation to do the same? I went in really wanting to be convinced by this article for the sake of my own mental health, but I am disappointed. What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities. Or perhaps an equivalence established showing biological defense is no harder than offense. Until it exists, I will continue to worry.
> What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities.
A major focus of the article is about the fundamental tradeoffs in how viruses operate. Was that not enough for you?
It is not, because we should not expect a manufactured technology to behave exactly as an evolved one. My understanding is there is not much incentive for a virus to kill off all its hosts. When they do it is largely an accident, and it is not uncommon for a severe virus to eventually get replaced by a less virulent and thus more successful strain. It's pretty hard to hill-climb evolution to such a point since the virus could starve itself in the process. And induction on evolution feels misguided to me. Consider our species. A million years ago it would have seemed completely reasonable to state that the absence of advanced tool use in the kingdom of life suggests it is biologically impossible. After all, such a thing had not occurred in hundreds of millions of years!
To be clear, I am not saying that such a virus will exist eventually. I am just noting that the arguments displayed here are not completely tight, and there are unknown unknowns at play. Some things are unfalsifiable. This is common.
I'm far from anything resembling an expert in biological weapons, but this seems to me puzzlingly overconfident in the opposite direction.
> PhD students in 2026 using state-of-the-art AI software are spending months or years trying to design simple peptide binders that inhibit some enzyme or pull down some protein
This confuses me, as state-of-the-art AI software is moving quite quickly. Any software that is currently state-of-the-art AI.. hasn't existed for years, maybe not even for months.
It also seems quite odd to claim that there are no sufficiently dangerous diseases when we have high-security research labs dedicated to exploring.. exactly these kinds of diseases.
And, even if it's not a disgruntled teenager, it just takes one leak from a lab that has 10xd productivity using AI tools.
This is a real risk that existed before AI, one that I think we can now confidently say that society is under-prepared for since 2020. It is certainly magnified by AI, even if it's not 100x.
The author even seems to recognize this:
> So, if you’re worried about AI-designed superviruses, think COVID, not Ebola. And if you’re now a little less worried, maybe you understand where I’m coming from.
This is exactly where I think most people are starting from, and that's _why we're worried_.
Think of it this way: if it wasn't generally crazy difficult to create even somewhat effective bioweapons (due to the reasons given in the article), there would've been non-trivial use of said weapons in terrorism, war, assassination attempts, etc over time. But we're still "stuck" with using primarily bombs, guns and - to far less extent - poisons.
If you quote the article, then you are implicitly telling others that you have read it.
But it's pretty clear you have not?
methinks it is more likely that I read it but missed the point? Would you be able to share what I misunderstood?