Login
The scariest AI doom scenario you haven’t thought about
AI will be the death of me. And you. And everyone we know.
At least, this is what a growing number of technologists and policymakers fear.
Last week, a lead researcher at Anthropic declared that there was a greater than 10 percent chance of AI “killing all humans” within the next decade. This nerve-racking forecast, combined with recent AI-initiated cyberattacks, have fueled calls for slowing the progress of artificial intelligence — a proposal that Anthropic and OpenAI subsequently embraced.
Not all prophets of AI doom agree about precisely how this apocalypse would come about. Many focus on scenarios reminiscent of The Terminator films, in which superintelligent machines conspire to exterminate humanity — perhaps in the same not-quite-intentional way that humans laid waste to the dodo, or passenger pigeon.
Key takeaways
- Synthetic pandemics are among the biggest existential risks facing humanity.
- AI could make bioweapons more powerful and easier to build.
- Governments can make us safer from both natural and artificial pathogens by investing more into pandemic preparedness.
But there is another, arguably more intuitive way that AI could bring about mass death, if not human extinction: by making it easier for people to develop and deploy catastrophic bioweapons.
Importantly, this hypothetical doesn’t require AI models to become conscious or superintelligent; it merely requires them to accelerate preexisting trends in biotechnology.
For those alarmed by this hypothetical, the threat AI poses is essentially twofold. First, when combined with technological progress in gene editing and DNA synthesis, chatbots could vastly increase the number of people capable of assembling dangerous viruses already known to medical science. And the more potential bioengineers there are in the world, the higher the odds that one will prove to be a diligent psychopath.
Second, AI could help highly trained biologists to discover — or engineer — a custom supervirus, which could then fall into the wrong hands.
Recent headlines have lent some credence to both of these fears.
Earlier this year, biologists warned that, with a few simple prompts, they’d gotten chatbots to dispense detailed instructions for engineering a treatment-resistant strain of an infamous pathogen, step-by-step protocols for recreating a once-pandemic virus, and advice on dispersing biological payloads via weather balloon.
In August, a team of scientists revealed that an AI model trained on DNA sequences had generated blueprints for deadly viruses unseen in nature (albeit, viruses deadly to bacteria, not animals). And last week, Anthropic announced that it had thwarted several suspected attempts by state actors to use Claude for bioweapons research.
Precisely how much AI is increasing the risk of an artificial pandemic is unclear. Even before ChatGPT, advances in synthetic biology were already making bioengineering tools more potent and accessible. And even after four years of rapid AI progress, the technical barriers to engineering novel superviruses remain formidable.
Regardless, there is a great deal more that governments can — and should — be doing to protect us from synthetic pandemics, both manmade and natural. No matter what AI’s ultimate impact on biosecurity proves to be, now is the time to both prepare for the worst-case scenario and, ideally, preempt it.
Why synthetic superviruses are scary
Before we dig into how AI could raise the risk of a man-made pandemic, it’s worth spelling out why that prospect is so unnerving.
Put simply, few things could more credibly destroy civilization — and/or, the human species — than a synthetic supervirus.
Infectious diseases have killed more humans than any other force on Earth. The Black Death wiped out at least 30 percent of all Europeans in the 1300s, while the 1918 flu killed 50 million people worldwide. You don’t need me to tell you what things were like in 2020 (or how much worse they could have been, if vaccines hadn’t been rolled out in record time).
And nature kicked off these catastrophes without even trying. A bioterrorist consciously seeking to maximize human death could theoretically improve on evolution’s handiwork. If biotech continues to advance — and scientists identify viral genomes that reliably yield high transmissibility and lethality — a malicious actor might one day be able to manufacture a virus that’s both as contagious as measles and as deadly as Ebola.
Natural selection generally disfavors viruses that are both highly transmissible and profoundly fatal; bugs that kill their hosts too reliably often have difficulty spreading. Yet a virus with a long asymptomatic period could escape this tradeoff. For example, HIV can live in a person’s body for a decade before generating noticeable symptoms. In theory, a highly contagious respiratory virus could similarly lie in wait for many weeks, months, or even years.
If bioterrorists ever managed to unleash an easily transmitted, highly fatal virus, they could shake the foundations of modern existence. In that scenario, reliably showing up for work might require accepting a 50-50 chance of getting yourself — and your family — brutally killed. Covid never required essential workers to assume risks anywhere near this high (the virus had a less than 1 percent case fatality rate among prime-age workers, before vaccines became available). Although there are many heroes among us, plenty of doctors, nurses, warehouse workers, power plant technicians, delivery drivers, and other essential laborers might not tolerate a coin flip’s chance of wiping out their loved ones. And if enough sheltered in place, systems for producing and distributing healthcare, food, water, and power could swiftly break down.
Humanity’s “Joker” problem
All this invites the question: Who would ever want to do this?
After all, viruses are terrible weapons: They’re difficult to make, impossible to control, and jeopardize the lives of those who cultivate and distribute them. Partly for these reasons, biological weapons have hardly ever been used; the last fatal bioterror attacks were the anthrax mailings of 2001.
To this, anxious biosecurity experts might reply, to quote the former British intelligence officer Alfred Pennyworth: “Some men just want to watch the world burn.”
Mass shooters regularly forfeit their lives to the cause of killing others at random. And brilliant iconoclasts periodically develop murderous ideologies; Ted Kaczynski, the Harvard mathematician turned Unabomber, wished to bring about the collapse of industrial civilization.
And every once in a while, an aspiring supervillain gets graduate training in virology. In the 1990s, the Kyoto University-educated genetic engineer Seiichi Endo joined Japan’s Aum Shinrikyo death cult. Drawing on the group’s $1 billion in funds, Endo sought to weaponize Ebola, Q fever, botulinum, and anthrax — with the aim of killing as many nonbelievers as possible.
Of course, Endo failed. And lone wolf psychopaths have never gotten their hands on anything more lethal than an assault weapon, bomb, or airplane.
But some scientists believe that technological advances — including generative AI — may provide the Endos and Kaczynskis of tomorrow with catastrophic power.
How AI and biotech could democratize bioterrorism
MIT biologist Kevin Esvelt is among his field’s most prominent Cassandras. And he has voiced two fundamental concerns. The first is that breakthroughs in synthetic biology and AI are rapidly expanding the pool of people who can engineer known viruses.
Over the past 15 years, the cost and difficulty of editing genomes have plummeted. In 2012, to modify a specific site in a genome, you often needed access to custom-engineered proteins that cost upward of $5,000 and still tended to work poorly, according to Olivia Scharfman, a biotechnology fellow at the Institute for Progress. Thanks to the gene-editing tool CRISPR, a scientist with minimal lab training can execute such a modification for as little as $30. Meanwhile, DNA synthesis has also grown radically more affordable.
This might not be such a big deal, if dangerous biological materials were hard to legally access. But Esvelt and his colleagues recently demonstrated that this isn’t the case. As an experiment, they had a student place orders for pieces of the 1918 influenza genome using a pseudonym and fake position. Thirty-six of 38 DNA synthesis companies shipped the genomes without asking whether the applicant had government permission or sound intentions. Together, these fragments were enough to create the virus many times over.
Crucially, in Esvelt’s account, reverse engineering known viruses using modern tools does not require elite skills. Any random biology graduate student may be up to the job.
And AI may be making such rudimentary viral engineering even easier.
In theory, a chatbot can help you plan a bioweapons attack in much the same way that it can walk you through fixing a dishwasher: By finding and synthesizing the internet’s vast trove of information, offering step-by-step instructions for navigating unfamiliar challenges, and fielding all troubleshooting queries that might subsequently arise.
And there’s evidence that AI models will in fact provide would-be bioterrorists with precisely this kind of assistance.
In a 2023 study, MIT researchers asked students to seek guidance on engineering a pandemic from that era’s chatbots. Within an hour, the AI models had suggested four promising pathogens, offered detailed protocols for manufacturing them with with synthetic DNA, provided a list of DNA synthesis retailers with low screening standards, and advised that — if users still found themselves unable to make the virus, even with all this help — they could purchase the assistance of a contract research organization to do some of the bioengineering for them.
And of course, frontier models have gotten a lot better since 2023. Last year, SecureBio, Esvelt’s biosecurity nonprofit, found that one OpenAI model outperformed 94 percent of expert virologists on troubleshooting complex laboratory problems.
Anthropic’s internal research has produced similar results. In one experiment, the company tasked amateurs with drafting a bioweapons acquisition plan. It gave some participants access to basic internet resources and others the assistance of Claude models (with the safeguards turned off). Those working with AI developed far more viable blueprints, according to the biodefense experts who scrutinized the proposals for critical errors.
Thus, there’s reason to think that the number of people with the skills and resources necessary for reverse engineering a virus from genomic blueprints is sharply rising — and with it, the risk that a bloodthirsty nihilist or ideologue gets their hands on a dangerous pathogen.
AI might help scientists learn too much
What’s even worse than bringing back smallpox? Esvelt’s second concern is that these same technological forces — combined with top scientists’ often reckless ambitions — will lead to the discovery or development of a supervirus, a new or custom-designed pathogen even more dangerous than what nature has come up with in the past.
Today, some scientists are actively trying to discover — and publicly identify — novel viruses circulating in nature that could theoretically trigger the next pandemic. Others conduct so-called gain-of-function experiments, altering pathogens to discern which mutations make them more transmissible, lethal, or difficult to detect (some have alleged that Covid-19 emerged from precisely this kind of research, though that theory is widely rejected among epidemiologists). This work is generally intended to anticipate future epidemiological threats, so that we can develop vaccines before they arrive. But their work could inadvertently provide bad actors with recipes for reverse engineering mass-casualty bioweapons.
Separately, advances in gene editing and AI may enable breakthroughs in viral engineering that yield apocalyptic pathogens.
A recent study from researchers at Stanford and the Arc Institute fed such fears. The scientists trained an AI called “Evo” on the DNA sequences of bacteriophages, the simple viruses that infect bacteria. They then used this “genomic language model” to generate blueprints for hundreds of bacteriophages found nowhere in nature. Sixteen of these designs yielded viable pathogens, some of which were better at killing bacteria than their natural ancestors.
What is clear, however, is that novel pathogens are among the greatest security threats our species faces.
Theoretically, scientists may one day be able to engineer novel human viruses using the same basic techniques. What’s more, even if genomic-language models fail at that task, they might still be able to help users tweak existing viruses in ways that render them more transmissible, deadly, or vaccine-resistant.
These discoveries could theoretically filter down to amateur bioterrorists or well-funded extremist groups like Aum Shinrikyo. Alternatively, a state-sponsored weapons program could develop a catastrophic supervirus, then unleash it on the world through a lab leak or (perhaps, more improbably) a deliberate attempt at genocide.
There is some precedent for the former scenario: The Soviet Union tried to weaponize smallpox and the Marburg virus, despite the considerable hazards and dubious military utility of such bioweapons. In the process, Soviet scientists accidentally triggered a smallpox outbreak during testing, while also letting anthrax leak from a laboratory, killing at least 64 people.
Bioweapons ain’t easy
All this said, there are reasons to question whether AI is truly making a synthetic pandemic more likely.
Chatbots may solve some of the challenges facing an amateur biologist with omnicidal aspirations. But it doesn’t necessarily clear the biggest obstacles in that person’s path.
Viral engineering is a complex, difficult, and physical undertaking. Performing it successfully requires what scientists call “tacit knowledge” — practical skills and understandings that are difficult to explain in words: knowing how to suppress the hand tremors that could rip delicate DNA strands, or how to recognize when the DNA in your solution has reached the proper concentration, or how to smell when a sample has been contaminated.
And these are things that Claude still can’t reliably impart. In studies, amateurs with access to a large language model tend to perform better on written biological challenges — but not on actual lab work. In a 2026 randomized study, non-scientists working with frontier AI systems were no more likely to complete a multistep laboratory assignment than those relying on the internet alone.
Of course, tacit knowledge wouldn’t necessarily be a problem for aspiring bioterrorists in the mold of Seiichi Endo — graduate-trained scientists with extensive lab experience. But at a minimum, chatbots don’t appear to have turned every malcontent who once took AP Bio into a competent virologist.
And while AI will likely help top biologists advance their field’s frontiers, the barriers to custom engineering doomsday viruses remain formidable. The “Evo” experiment is arguably a case in point. Although the model did generate working blueprints for a few novel bacteriophages, nearly all of its suggested genomes didn’t work. Researchers had to arduously test hundreds of DNA sequences to arrive at 16 viable strands. And engineering slightly modified versions of an extremely small, well-known bacteriophage capable of infecting E. coli in a lab — and creating a novel virus more fearsome than any nature has yet devised — aren’t especially similar tasks.
To offer a strained analogy: The fact that I can dunk on Victor Wembanyama in NBA 2K26 does not imply that I’m making progress toward being able to dunk on him in real life.
Even in the age of CRISPR and ChatGPT, biology remains an exceptionally difficult discipline that resists reliable modeling. The effects of modifying a single gene can often yield unpredictable and inconsistent results, varying with host biology, immune responses, and environmental conditions that scientists only partly understand. AI may help us overcome these complexities. It also might not.
We need to be preparing for the next pandemic
Scientists disagree about how serious a risk synthetic pandemics pose — and how much AI is magnifying it.
What is clear, however, is that novel pathogens are among the greatest security threats our species faces. Even if synthetic biology and artificial intelligence did not exist, natural selection would keep churning out new viruses — and periodically hitting upon ones that befuddle human immune systems.
For this reason, there would be an urgent need to make our societies less vulnerable to future pandemics, irrespective of AI’s implications for viral engineering. In a world where chatbots and CRISPR are plausibly making bioterrorism easier, failing to invest aggressively in public health is all the more reckless.
The good news is we already know — from hard-earned experience — many steps that we can take to reduce the risk of a new pandemic and prepare to respond to one if needed.
Governments can limit the destructive power of novel contagions by investing in vaccines that protect against entire families of viruses (or ideally, all of them) and broad-spectrum antivirals; raise our odds of catching outbreaks early by funding meticulous wastewater and clinical surveillance; ensure that essential workers don’t have to choose between staying alive and keeping civilization going amid a pandemic by developing and then stockpiling foolproof personal protective equipment; make indoor spaces less hospitable to airborne viruses by upgrading ventilation and filtration systems; and reduce the the threat of bioterrorism by mandating rigorous screening of synthetic-DNA orders.
Fears that humanity will be done in by superintelligent machines are currently galvanizing media attention and political concern. And this may be warranted. But we should pay at least as much attention to our species’ age-old, organic adversaries as we do to our hypothetical, robot ones — and not just because the latter could start teaching the former new tricks.
