Here is a new article I wrote with Chris Bakerlee for Vox's Future Perfect on engineered pathogens and global catastrophic biorisks. We make the case for why this is a particularly important challenge, why it has so far been mostly neglected and poorly dealt with, and we discuss some steps we can take to reduce the risk. It might be useful for EAs interested in pandemic risk and risks from biotechnology as well as those interested in global catastrophic risks broadly. https://www.vox.com/future-perfect/2018/12/6/18127430/superbugs-biotech-pathogens-biological-warfare-pandemic
Reactions
Comments5
Interesting, scary stuff. I've been reading up on biotech/bioweapons a bit as part of my research on AI strategy. They're interesting both because there could be dangerous effects from AI improving bioweapons*, and because they're a relatively close analogue to AI by virtue of their dual-use, concealability, and reasonably large-scale effects.
Do you know of good sources on bioweapons strategy, offense-defense dynamics, and potential effects of future advances? I'm reading Koblentz's Living Weapons right now and it's quite good, but I haven't found many other leads. (I'd think there would be more papers on this; maybe they're mostly kept secret, or maybe I'm using the wrong keywords.)
*My impression from Koblentz is that foreseeable advances in biotech aren't hugely destabilizing, since bioattacks aren't a good strategic threat; military locations can be pretty effectively hardened against them for not-unbearable costs. One danger I'm curious about is the scope of potential attacks in 20-30 years; could there be devastating, hard-to-trace attacks on civilian populations?
I don't think there is much publicly available on this topic besides Koblentz's work (also check out his 2003 article in International Security). The "strategy of conflict" as it pertains to bioweapons is something we thought about, but we don't discuss it much in our paper. Some thoughts:
Historically bioweapons research has focused on diseases that are not transmissible person to person like Tularemia, Anthrax, Q Fever, and Botulism. If you dump a bunch of anthrax spores from an airplane over a city, you would kill a lot of people (I recall seeing a study where they estimated that if you dumped anthrax over a large city you would kill ~200,000 people) even though it's not transmissible person to person.
Japanese Unit 731 used Plague, which is transmissible person to person, but used it well behind enemy lines on enemy cities to limit collateral damage. This made this sort of weapon more of a poor man's strategic bombing campaign that could wreak havoc on civilian populations by dumping swarms of plague infested fleas on enemy territory. There is a lot of uncertainty on the actual numbers, but Unit 731 may have killed more civilians in China through these sorts of attacks than the US killed Japanese civilians in Japan from the nuclear bombs and firebombings (the exact numbers are hard to know in large part because it's hard to attribute civilian deaths directly to the released bioweapons because the weapon is meant to present itself as a natural epidemic, as we mention in the paper). There is evidence that the Japanese did sustain some collateral damage from their plague attacks against China during the war: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1200679/.
Starting a pandemic transmissible Person to Person in the enemy's country is probably a bad idea because it could turn into a global pandemic and there is significant collateral damage to your own country. Though, it should be noted that using other WMDs like nuclear weapons also has significant, and possibly similar scale, collateral damage. One of the dangers of future development in biotechnology is that it could make biological weapons more attractive as a weapon.
Determining strategic behavior and implications of using weapons that impose significant collateral damage to the country using them and to non-combatant countries is actually an active research topic of mine.
Curated and popular this week
·
Once we expand to other star systems, we may begin a self-propagating expansion of human civilisation throughout the galaxy. However, there are existential risks potentially capable of destroying a galactic civilisation, like self-replicating machines, strange matter, and vacuum decay. Without an extremely widespread and effective governance system, the eventual creation of a galaxy-ending x-risk seems almost inevitable due to cumulative chances of initiation over time across numerous independent actors. So galactic x-risks may severely limit the total potential value that human civilisation can attain in the long-term future. The requirements for a governance system to prevent galactic x-risks are extremely demanding, and they need it needs to be in place before interstellar colonisation is initiated.
Introduction
I recently came across a series of posts from nearly a decade ago, starting with a post by George Dvorsky in io9 called “12 Ways Humanity Could Destroy the Entire Solar System”. It’s a fun post discussing stellar engineering disasters, the potential dangers of warp drives and wormholes, and the delicacy of orbital dynamics.
Anders Sandberg responded to the post on his blog and assessed whether these solar system disasters represented a potential Great Filter to explain the Fermi Paradox, which they did not[1]. However, x-risks to solar system-wide civilisations were certainly possible.
Charlie Stross then made a post where he suggested that some of these x-risks could destroy a galactic civilisation too, most notably griefers (von Neumann probes). The fact that it only takes one colony among many to create griefers means that the dispersion and huge population of galactic civilisations[2] may actually be a disadvantage in x-risk mitigation.
In addition to getting through this current period of high x-risk, we should aim to create a civilisation that is able to withstand x-risks for as long as possible so that as much of the value[3] of the univers
·
There is dispute among EAs--and the general public more broadly--about whether morality is objective. So I thought I'd kick off a debate about this, and try to draw more people into reading and posting on the forum! Here is my opening volley in the debate, and I encourage others to respond.
Unlike a lot of effective altruists and people in my segment of the internet, I am a moral realist. I think morality is objective. I thought I'd set out to defend this view.
Let’s first define moral realism. It’s the idea that there are some stance independent moral truths. Something is stance independent if it doesn’t depend on what anyone thinks or feels about it. So, for instance, that I have arms is stance independently true—it doesn’t depend on what anyone thinks about it. That ice cream is tasty is stance dependently true; it might be tasty to me but not to you, and a person who thinks it’s not tasty isn’t making an error.
So, in short, moral realism is the idea that there are things that you should or shouldn’t do and that this fact doesn’t depend on what anyone thinks about them. So, for instance, suppose you take a baby and hit it with great force with a hammer. Moral realism says:
1. You’re doing something wrong.
2. That fact doesn’t depend on anyone’s beliefs about it. You approving of it, or the person appraising the situation approving of it, or society approving of it doesn’t determine its wrongness (of course, it might be that what makes its wrong is its effects on the baby, resulting in the baby not approving of it, but that’s different from someone’s higher-level beliefs about the act. It’s an objective fact that a particular person won a high-school debate round, even though that depended on what the judges thought).
Moral realism says that some moral statements are true and this doesn’t depend on what people think about it. Now, there are only three possible ways any particular moral statement can fail to be stance independently true:
1. It’s
·
Summary
Arkose is an early-stage AI safety fieldbuilding nonprofit focused on accelerating the involvement of experienced machine learning professionals in technical AI safety research through direct outreach, one-on-one calls, and public resources.
Between December 2023 and June 2025, we had one-on-one calls with 311 such professionals. 78% of those professionals said their initial call accelerated their involvement in AI safety[1].
Unfortunately, we’re closing due to a lack of funding. We remain excited about other attempts at direct outreach to this population, and think the right team could have impact here.
Why are we closing?
Over the past year, we’ve applied for funding from all of the major funders interested in AI safety fieldbuilding work, and several minor funders. Rather than try to massively change what we're doing to appeal to funders, with a short funding runway and little to no feedback, we’re choosing to close down and pursue other options.
What were we doing? Why?
* Calls: we ran 1:1 calls with mid-career machine learning professionals. Calls lasted an average of 37 minutes (range: 10-79), and we had a single call with 96% of professionals we spoke with (i.e. only 4% had a second or third call with us). On call, we focused on:
* Introducing existential and catastrophic risks from AI
* Discussing research directions in this field, and relating them to the professional’s areas of expertise.
* Discussing specific opportunities to get involved (e.g. funding, jobs, upskilling), especially ones that would be a good fit for the individual.
* Giving feedback on their existing plans to get involved in AI safety (if they have them).
* Connecting with advisors to support their next steps in AI safety, if appropriate (see below).
* Supportive Activities:
* Accountability: after calls, we offered an accountability program where participants set goals for next steps, and we check in with them. 114 call participants set goals for check
I don't think there is much publicly available on this topic besides Koblentz's work (also check out his 2003 article in International Security). The "strategy of conflict" as it pertains to bioweapons is something we thought about, but we don't discuss it much in our paper. Some thoughts:
Historically bioweapons research has focused on diseases that are not transmissible person to person like Tularemia, Anthrax, Q Fever, and Botulism. If you dump a bunch of anthrax spores from an airplane over a city, you would kill a lot of people (I recall seeing a study where they estimated that if you dumped anthrax over a large city you would kill ~200,000 people) even though it's not transmissible person to person.
Japanese Unit 731 used Plague, which is transmissible person to person, but used it well behind enemy lines on enemy cities to limit collateral damage. This made this sort of weapon more of a poor man's strategic bombing campaign that could wreak havoc on civilian populations by dumping swarms of plague infested fleas on enemy territory. There is a lot of uncertainty on the actual numbers, but Unit 731 may have killed more civilians in China through these sorts of attacks than the US killed Japanese civilians in Japan from the nuclear bombs and firebombings (the exact numbers are hard to know in large part because it's hard to attribute civilian deaths directly to the released bioweapons because the weapon is meant to present itself as a natural epidemic, as we mention in the paper). There is evidence that the Japanese did sustain some collateral damage from their plague attacks against China during the war: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1200679/.
Starting a pandemic transmissible Person to Person in the enemy's country is probably a bad idea because it could turn into a global pandemic and there is significant collateral damage to your own country. Though, it should be noted that using other WMDs like nuclear weapons also has significant, and possibly similar scale, collateral damage. One of the dangers of future development in biotechnology is that it could make biological weapons more attractive as a weapon.
Determining strategic behavior and implications of using weapons that impose significant collateral damage to the country using them and to non-combatant countries is actually an active research topic of mine.