As a biologist, I want to zone in on one particular part of the post here because I find it very off base.
Whatever expert interviews were done to get the list of "tacit knowledge" seems deeply flawed, as the difficulty of the listed examples is overstated by orders of magnitude. I get the impression that this list was compiled by talking to ancient PIs that haven't done experiments in decades and have lost much of their sense of proportion. This is a semi-common phenomenon.
Specifically,
Pipetting isn't difficult. You push a button up and down. Students get it in a few tries. Pipetting volumes extremely precisely requires some days of practice but is practically never necessary.
Fragile materials exist but are pretty rare and honestly not mythically difficult to handle. The one case I encountered came down to not using the vortex machine on it and pipetting it slowly. This really isn't relevant all that often and it's not super severe when important.
Most biology works fine or a bit less effectively, if the concentrations are off by 20%. If it's not working you just vary the concentrations in a "grid search" till it works
Contamination is a regular problem in lots of wetlab settings. It definitely requires some practice to keep things sterile consistently. That said, students in university microbiology lab classes routinely pull this off first try with just a Bunsen burner.
It's true that much (but by no means all) biology work requires weeks or months of finicky optimization before you can get it working reliably, but this has little to do with these (to me bizarre) claims.
Pipetting is hard based on the type of task you need to do. Obviously, pipetting is super easy when the thing itself is basic. The difficult is a function of the braoder task you're doing. Fragile materials, concentrations, contamination: all this is to illustrate what "lab hands" mean to non-bio people people, as well as why some things in textbooks are much harder in real life.
None of this is to say that making a basic bioweapon is impossible for people who have an institution, basic skills, the ability to order controlled things, and the desire. The people who have a list of all of those things, plus the skills for very specific types of virus sets, are a smaller crowd, in general, which people seem to not be understanding. Hence, see my BOTEC of those people in Article 1.
I'd love to know what "bizarre" claims you mean. A good number of the biologists were actively doing wet lab work. It's been publicly commended predominantly biologists and domain experts.
This post feels like it is saying that lone actors are currently incapable of creating an X-risk bioweapon. I agree. I think almost everyone in the X-risk scene agrees.
The concern is instead about whether this capability will exist soon. To show that being impossible (or very unlikely), IMO you need to show that there is at least one critical step that is not getting democratized. IE, bio-expertise is getting cheaper because of increasingly powerful open source models. Robots with sufficient dexterity to do bio experiments are getting more capable and cheaper. Where is the bottleneck that will remain unsolved for 2+ decades, and why are we confident it stays unsolved?
To give an analogy, 2 years ago non-experts couldn't solve major open math problems. Today, with access to frontier models, they can. How can we be confident that the bottlenecks against independently creating novel & powerful plagues will not also fall?
Hi Dan. I expect terrorist attacks to be super unlikely to cause human extinction in the absence of very strong evidence suggesting otherwise. For the attacks killing more than 10 people among the 202 k listed on the Global Terrorism Database (GTB), the fraction killing more than 10*N people is around 1 % of the fraction killing more than N people. This suggests a Pareto distribution with a tail index of 2. Around 0.001 % of attacks killed more than 1 k people. Assuming the same Pareto distribution (straight line fitted to the section of the graph below with more than 10 deaths), more than 1 billion deaths would be 10^-12 (= 0.01^(9 - 3)) times as likely as more than 1 k deaths. So the Pareto distribution implies only 10^-17 (= 10^(-5 - 12)) of terrorist attacks cause more than 1 billion deaths. I think the actual risk is way higher. At the same time, I believe a significant risk of a terrorist attack causing human extinction, like 1 % over the next 20 years, would depart a lot from the most natural uninformed ways of extrapolating past data. A pandemic causing human extinction looks much more plausible based on past data (although I still guess the risk of human extinction in the next 20 years is much lower than 1 %).
Due to 2, I think I might go as high as 0.1% risk over the next 20 years (with incredibly wide error bars). That still seems high enough to seriously consider, although I agree other X-risk concerns are likely even higher.
I interviewed dozens of biosecurity experts. The fear that AI will enable lone wolves to build pandemics is a case study in how domain expertise gets sidelined.
(Emphasis mine) How many AI experts did you interview? How about AIxBio experts?
Roughly 10, ~8 of which I interviewed at one point at the same time so to red team each other. At least 4 were in AIxbio. More than one were x-risk/ea-prioritization-focused.
[Subtitle.] I interviewed dozens of biosecurity experts. The fear that AI will enable lone wolves to build pandemics is a case study in how domain expertise gets sidelined.
Some of my friends think we are going to die because AI will make building a bioweapon easier.
The fear is often that AI will walk an ordinary person through unleashing a pandemic. Or that AI will facilitate the creation of an engineered pandemic that could kill millions or billions on Earth.
I spent ten years as a US diplomat. My job was often to check claims before Washington acted. I wanted to double check this too.
I interviewed dozens of biosecurity experts who had decades at the lab bench.
Yes, AI makes virology work a little easier. But it is still extremely difficult. Media headlines often say “AI can make this step easier” without the broader context that a bioweapon involves a lot of hard steps that are not easier.
The public doesn’t hear this context because of our information ecosystem.
Let me first explain why I’m no longer worried [about lone wolves and non-state actors]:
Why bioweapons are rare
Since the 1980s, there has been exactly one fatal bioterror attack.[1]
They are rare because 1. they are difficult, and 2. they are bad weapons.
People who want to cause harm still do cost-benefit thinking. For almost any goal they have, a bomb, a gun, a chemical, or a cyberattack is cheaper, faster, and more controllable.
Practitioners flagged several more barriers:
You have to be comfortable killing your own people. Pathogens kill indiscriminately. Few groups are this malicious and extreme. Groups that are tend toward paranoia and rigid hierarchies that make them less effective.
You need significant resources. Bioweapons capable of harming many people need equipment, chemicals, and professionals trained in different fields.
You need an institution for access to specialized equipment and materials. Many items are too niche for black markets. Essential equipment like biosafety cabinets or fume hoods requires professional installation and ongoing manufacturer support. Some materials are restricted; for example, viruses need host cells, cell lines are sold only to institutions, and plasmids require institutional credentials. In the United States, only tens of thousands of people have sustained access to such facilities and supplies.
Your team must be both highly competent and completely secret. A contamination incident could destroy the pathogen or infect the team.
You need to convince and retain experts from different domains e.g. biology, chemistry, engineering, aerosolization, or fieldcraft to carry out attacks. Recruiting without detection is hard too.
Even well-resourced groups with scientific talent still fail. Aum Shinrikyo, a Japanese cult with ~$1 billion, failed in all ten of their attempts to release biological agents.
Why AI models don’t change this
AI can supply biology and chemistry knowledge, but not hands-on skill.
Reading every surgery textbook doesn’t make you a good surgeon. The biology laboratory works in a similar apprenticeship model. You learn by failing repeatedly under a mentor who has also failed repeatedly.
To get a sense of the physical nature of biology work, I explored one aspect: moving liquids.
Biologists often use pipettes, very precise eye-dropper-like tools. Students will often try fifty times to get it right. Practitioners develop a feel for specific materials over the years.
This skill appears in almost every bioweapon protocol. It is hard because:
Materials are often fragile. Some DNA strands can shear from hand tremors. Some proteins are so temperature sensitive that the pressure of moving the liquid generates enough heat to destroy them.
Concentration has to be exact. If a sample is too dilute, DNA strands may never find each other. Instructions might say “check with a spectrophotometer,” but a practiced scientist recognizes when a bubble or residual chemical is corrupting the reading.
Contamination in biology is unforgiving. A one-in-a-million contaminant in chemistry is usually negligible. In biology, this can matter because organisms multiply.
Creating a bioweapon involves many more complex steps. Converting viral DNA into a live virus is hard even for researchers. Stabilizing a pathogen into an effective aerosol often requires specialists from different fields. AI can describe the process but doesn’t give you the skills to execute it. (I walk through the full chain of steps here.)
Even success probably wouldn’t be catastrophic
A motivated team of scientists with institutional access could keep trying and eventually succeed at making a pathogen.
But success would not necessarily produce a new pandemic. Any pathogen that a team could plausibly assemble is one that already exists in nature. A lab-assembled version of COVID-19 would have about the same effect as an infected person coughing in a crowded room.
The more serious worry is that someone could revive a pathogen no longer in circulation, like the 1918 influenza or engineer something entirely new.
The 1918 flu infected roughly a third of the world’s population and killed an estimated 50 million people. Releasing it today would be very bad. It would not be 1918-bad. Most people’s immune systems recognize flu descendants. We also now have antivirals and antibiotics to treat the secondary bacterial pneumonia that killed most victims. Today’s vaccines offer cross-protection.
Engineering a virus that is both extremely lethal and highly contagious, worse than anything nature has produced, would be bad. But many biologists doubt this is possible even with perfect technology. People have very complex immune systems that often react differently. Viruses mutate as they spread. But more importantly: biology has built-in tradeoffs: the more lethal a virus is, the less transmissible it is (because it kills its human host).
When it comes to designing such a virus, any edits to make a virus deadlier could make the virus unstable or incompatible with human life. In physics, you can tell when an experiment is getting closer to a goal. Biology doesn’t work that way, meaning you can spend a long time on an endpoint that is not biologically possible.
Nature makes new viruses constantly, but by brute force. It spins out countless mutations and lets billions of them die every day.
Why this isn’t conventional wisdom
When I published this research, leaders in biosecurity and national security organizations thanked me privately. “My organization found the same conclusion, hence why we don’t work on that.”
Null results aren’t often published. Looking at the media sphere specifically: “bioweapons are hard” is not news. It isn’t a headline; it’s always been true.
Advocacy organizations have similar constraints. If you are pushing for AI oversight, a finding that undercuts urgency on one specific threat does not help.
Biologists, the people best positioned to explain this, mostly don’t write for general audiences. Academic incentives don’t reward op-eds, especially about things that are obvious within the field.
Our information systems get less accurate as issues become politicized. Alarm, not context, ends up on the news partly because of who is incentivized to speak up.
I’ve worked through the evidence across three Golden Gate Institute for AI pieces, if you want to go deeper:
Update: To be clear, biosecurity is still critical! My point above is narrow, aimed at a claim common in EA and x-risk circles. Even EA-affiliated organizations like CLTR and GovAI have published a similar point: an outsized focus on lone-wolf bioterrorism and engineered pandemics can crowd out other threats, like smaller bioweapons, non-transmissible pathogens, chemical weapons, and explosives. Broad biological resilience helps across many of these scenarios, especially naturally occurring pandemics, which carry a 2-3% chance each year.
Note: This post was crossposted from the Coefficient Giving Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
Subtitle: On vegan advocacy, effective altruism, and FIFA
I turn 40 today. Here are some hot takes.
On factory farming
1. Our biggest challenge is salience. If factory farming led the evening news, it wouldn’t last lo...
Impressions from the 21st Annual U.S. Cluster Headache Patient Conference, organized by Clusterbusters.
On the morning of the last day of the conference, E. was hanging out with other “clusterheads” outside the conference hotel when he started having a cluster headache attack. Before he could go back to his hotel room, J. offered E. his DMT v...
TL;DR:
The movement has an evidence problem. To solve it, for each intervention we need to know:
* What types of evidence that intervention needs
* What types of evidence different sources can give us
In this post, we outline a framework for evidencing an intervention through five stages:
...
As a biologist, I want to zone in on one particular part of the post here because I find it very off base.
Whatever expert interviews were done to get the list of "tacit knowledge" seems deeply flawed, as the difficulty of the listed examples is overstated by orders of magnitude. I get the impression that this list was compiled by talking to ancient PIs that haven't done experiments in decades and have lost much of their sense of proportion. This is a semi-common phenomenon.
Specifically,
It's true that much (but by no means all) biology work requires weeks or months of finicky optimization before you can get it working reliably, but this has little to do with these (to me bizarre) claims.
Pipetting is hard based on the type of task you need to do. Obviously, pipetting is super easy when the thing itself is basic. The difficult is a function of the braoder task you're doing. Fragile materials, concentrations, contamination: all this is to illustrate what "lab hands" mean to non-bio people people, as well as why some things in textbooks are much harder in real life.
None of this is to say that making a basic bioweapon is impossible for people who have an institution, basic skills, the ability to order controlled things, and the desire. The people who have a list of all of those things, plus the skills for very specific types of virus sets, are a smaller crowd, in general, which people seem to not be understanding. Hence, see my BOTEC of those people in Article 1.
I'd love to know what "bizarre" claims you mean. A good number of the biologists were actively doing wet lab work. It's been publicly commended predominantly biologists and domain experts.