Summary: If you’re a longtermist (i.e you believe that most of the moral value lies in the future), and you want to prioritize impact in your career choice, you should strongly consider either working on AI directly, or working on things that will positively influence the development of AI.
Epistemic Status: The claim is strong but I'm fairly confident (>75%) about it. I think the main crux is how bad biorisks could be and how the risk profile compared with the AI safety one, which I think is the biggest crux of this post. I've spent at least a year thinking about advanced AIs and their implications on everything, including much of today's decision-making. I've reoriented my career towards AI based on these thoughts.
The Case for Working on AI
If you care a lot about the very far future, you probably want two things to happen: first, you want to ensure that humanity survives at all; second, you want to increase the growth rate of good things that matter to humanity - for example, wealth, happiness, knowledge, or anything else that we value.
If we increase the growth rate earlier and by more, this will have massive ripple effects on the very longterm future. A minor increase in the growth rate now means a huge difference later. Consider the spread of covid - minor differences in the R-number had huge effects on how fast the virus could spread and how many people eventually caught it. So if you are a longtermist, you should want to increase the growth rate of whatever you care about as early as possible, and as much as possible.
For example, if you think that every additional happy life in the universe is good, then you should want the number of happy humans in the universe to grow as fast as possible. AGI is likely to be able to help with this, since it could create a state of abundance and enable humanity to quickly spread across the universe through much faster technological progress.
AI is directly relevant to both longterm survival and longterm growth. When we create a superintelligence, there are three possibilities. Either:
- The superintelligence is misaligned and it kills us all
- The superintelligence is misaligned with our own objectives but is benign
- The superintelligence is aligned, and therefore can help us increase the growth rate of whatever we care about.
Longtermists should, of course, be eager to prevent the development of a destructive misaligned superintelligence. But they should also be strongly motivated to bring about the development of an aligned, benevolent superintelligence, because increasing the growth rate of whatever we value (knowledge, wealth, resources…) will have huge effects into the longterm future.
Some AI researchers focus more on the ‘carrot’ of aligned benevolent AI, others on the ‘stick’ of existential risk. But the point is, AI will likely either be extremely good or extremely bad - it’s difficult to be AI-neutral.
I want to emphasize that my argument only applies to people who want to strongly prioritize impact. It’s fine for longtermists to choose not to work on AI for personal reasons. Most people value things other than impact, and big career transitions can be extremely costly. I just think that if longtermists really want to prioritize impact above everything else, then AI-related work is the best thing for (most of) them to do; and if they want to work on other things for personal reasons, they shouldn’t be tempted by motivated reasoning to believe that they are working on the most impactful thing.
Objections
Here are some reasons why you might be unconvinced by this argument, along with reasons why I find these objections unpersuasive or unlikely.
You might not buy this argument because you believe one of the following things:
You want to take a ‘portfolio approach’
Some EAs take a ‘portfolio approach’ to cause prioritization, thinking that since the most important cause is uncertain, we should divide our resources between many plausibly-important causes.
A portfolio approach makes sense when you have comparable causes, and/or when there are decreasing marginal returns on each additional resource spent on one cause. But in my opinion, this isn’t true for longtermists and AI. First, the causes here are not comparable; no other cause has such large upsides and downsides. Second, the altruistic returns on AI work are so immensely high that even with decreasing marginal returns, there is still a large difference between this opportunity and our second biggest priority.
There’s a greater existential risk in the short term
You might think that something else currently poses an even greater existential risk than AI. I think this is unlikely, however. First, I’m confident that of the existential risks known to EAs, none is more serious than the risk from AI. Second, I think it’s unlikely that there is some existential risk that is known to a reader but not to most EAs, and that is more serious than AI risk.
In The Precipice, Toby Ord estimates that we are 3 times more likely to go extinct due to AI than due to biological risks - the second biggest risk factor after AI (in his opinion). Many people - including me - think that Ord vastly overestimates biorisks, and our chances of going extinct from biological disasters are actually very small.
One of the most critical features that seem to be crucial to extinction events via viruses is whether the virus is stealth or not and for how long. I think we’re likely to be able to prevent the ‘stealth viruses’ scenario happening in the next few years thanks to metagenomic sequencing which should make extinction from stealthy pathogens even less likely; therefore, I believe that the risk of extinction from pathogens in the next few decades is very unlikely. If there's any X-risk this century, I think it's heavily distributed in the second half of this century. For those interested, I wrote a more detailed post on scenarios that could lead to X-risks via biorisks. I think that the most likely way I could be wrong here is if the minimum viable population was not 1000 but greater than 1% of the world population or if an irrecoverable collapse was very likely even above these thresholds.
On the other hand, transformative AIs (TAIs) will probably be developed within the next few decades according to Ajeya Cotra’s report on biological anchors (which is arguably an upper bound of the development of TAI).
Others have argued that nuclear war and climate change, while they could have catastrophic consequences, are unlikely to cause human extinction.
A caveat: I’m less certain about the risks posed by nanotechnology. However, I don’t think this poses a comparable risk to AI, although I’d expect this to be the second biggest source of risk after AI.
See here for a database of various experts’ estimates of existential risk from various causes.
It’s not a good fit for you
I.e., you have skills or career capital that make it suboptimal for you to switch into AI. This is possible, but given that both AI Governance and AI Safety need a wide range of skills, I expect this to be pretty rare.
By wide range, I mean very wide. So wide that I think that even most longtermists with a biology background who want to maximize their impact should work on AI. Let me give some examples of AI-related career paths that are not obvious:
- Community building (general EA community building or building the AI safety community specifically).
- Communications about AI (to targeted public such as the ML community).
- Increasing the productivity of people who do direct AI work by working with them as a project manager, coach, executive assistant, writer, or other key support roles.
- Making a ton of money (I expect this to be very useful for AI governance as I will argue in a future post).
- Building influence in politics (I expect this to be necessary for AI governance).
- Studying psychology (e.g. what makes humans altruistic) or biology (e.g evolution). These questions are relevant for AI to make our understanding of optimization dynamics more accurate, which is key to predicting what we may expect from gradient descent. PIBSS is an example of this kind of approach to the AI problem.
- UX designer for EA organizations such as 80k.
- Writing fiction about AGI that is about plausible scenarios that could happen (rather than, e.g., terminator robots) - the only example I know of this type of fiction is Clippy.
There is something that will create more value in the long-term future than intelligence
This could be the case; but I give it a low probability, since intelligence seems to be highly multipurpose, and a superintelligent AI could help you find or increase this other thing more quickly.
It’s not possible to align AGI
In this case, you should focus on stopping the development of AGI or tried to develop beneficial unaligned AGI.
AGI will be aligned by default
If you don’t accept the orthogonality thesis or aren’t worried about misaligned AGI, then you should work to ensure that the governance structure around AGI is favorable to what you care about and that AGI happens as soon as possible within this structure, because then we can increase the growth rate of whatever we care about.
You’re really sure that developing AGI is impossible
This is hard to justify: the existence of humans proves that general intelligence is feasible.
Have I missed any important considerations and counter-arguments? Let me know in the comments. If you’re not convinced of my main point, I expect this to be because you disagree with the following crux: there isn’t any short term X-risk which is nearly as important as AGI. If this is the case- especially if you think that biorisks could be equally dangerous - tell me in the comments and I’ll consider writing about this topic in more depth.
Non-longtermists should also consider working on AI
In this post I’ve argued that longtermists should consider working on AI. I also believe the following stronger claim: "whatever thing you care more about, it will likely be radically transformed by AI pretty soon, so you should care about AI and work on something related to it". I didn’t argue for this claim because this would have required significantly more effort. However, If you care about causes such as poverty, health or animals, and you think your community could update based on a post saying “Cause Y will be affected by AI”, leave a comment and I will think about writing about it.
This post was written collaboratively by Siméon Campos and Amber Dawn Ace as part of Nonlinear’s experimental Writing Internship program. The ideas are Siméon’s; Siméon explained them to Amber, and Amber wrote them up. We would like to offer this service to other EAs who want to share their as-yet unwritten ideas or expertise.
If you would be interested in working with Amber to write up your ideas, fill out this form.
I think this is a bit too strong of a claim. It is true that that overwhelming majority of value in the future is determined by whether, when, and how we build AGI. I think it is also true that a longtermist trying to maximize impact should, in some sense, be doing something which affects whether, when, or how we build AGI.
However, I think your post is too dismissive of working on other existential risks. Reducing the chance that we all die before building AGI increases the chance that we build AGI. While there probably won't be a nuclear war before AGI, it is quite possible that a person very well-suited to working on reducing nuclear issues could reduce x-risk more by working to reduce nuclear x-risk than they could by working more directly on AI.
Thanks for the comment.
I think it would be true if there were other X-risks. I just think that there is no other literal X-risk. I think that there are huge catastrophic risks. But there's still a huge difference between killing 99% of people and killing a 100%.
I'd recommend reading (or skimming through) this to have a better sense of how different the 2 are.
I think that in general the sense that it's cool to work on every risks come precisely from the fact that very few people have thought about every risks and thus people in AI for instance IMO tend to overestimate risks in other areas.
"no other literal X-risk" seems too strong. There are certainly some potential ways that nuclear war or a bioweapon could cause human extinction. They're not just catastrophic risks.
In addition, catastrophic risks don't just involve massive immediate suffering. They drastically change global circumstances in a way which will have knock-on effects on whether, when, and how we build AGI.
All that said, I directionally agree with you, and I think that probably all longtermists should have a model of the effects their work has on the potentiality of aligned AGI, and that they should seriously consider switching to working more directly on AI, even if their competencies appear to lie elsewhere. I just think that your post takes this point too far.
Just tell me a story with probabilities of how nuclear war or bioweapons could cause human extinction and you'll see that when you'll multiply the probabilities, it will go down to a very low number.
I repeat but I think that you don't still have a good sense of how difficult it is to kill every humans if the minimal viable population (MVP) is around 1000 as argued in the post linked above.
"knock-on effects"
I think that it's true but I think that on the first-order, not dying from AGI is the most important thing compared with developing it in like 100 years.
I have a slight problem with the "tell me a story" framing. Scenarios are useful, but also lend themselves general to crude rather than complex risks. In asking this question, you implicitly downplay complex risks. For a more thorough discussion, the "Democratising Risk" paper by Cramer and Kemp has some useful ideas in it (I disagree with parts of the paper but still) It also continues to priorities epistemically neat and "sexy" risks which whilst possibly the most worrying are not exclusive. Also probabilities on scenarios in many contexts can be somewhat problematic, and the methodologies used to come up with very high xrisk values for AGI vs other xrisks have very high uncertainties. To this degree, I think the certainty you have is somewhat problematic
Yes, scenarios are a good way to put a lower bound but if you're not able to create one single scenario that's a bad sign in my opinion.
For AGI there are many plausible scenarios where I can reach ~1-10% likelihood of dying. With biorisks it's impossible with my current belief on the MVP (minimum viable population)
Sketching specific bio-risk extinction scenarios would likely involve substantial info-hazards.
You could avoid such infohazards by drawing up the scenarios in a private message or private doc that's only shared with select people.
I think that if you take these infohazards seriously enough, you probably even shouldn't do that. Because if everyone has a 95% likelihood to keep it secret, with 10 persons in the know is 60%.
I see what you mean, but if you value cause prioritization seriously enough, it is really stifling to have literally no place to discuss x-risks in detail. Carefully managed private spaces are the best compromise I've seen so far, but if there's something better then I'd be really glad to learn about it.
I think that I'd be glad to stay as long as we can in the domain of aggregate probabilities and proxies of real scenarios, particularly for biorisks.
Mostly because I think that most people can't do a lot about infohazardy things so the first-order effect is just net negative.
Yes I mostly agree but even conditional on info hazardy things I still think that the aggregate probability of likelihood of collapse is a very important parameter.
I'm not sure what you mean - I agree the aggregate probability of collapse is an important parameter, but I was talking about the kinds of bio-risk scenarios that simeon_c was asking for above?
Do I understand you right that overall risk levels should be estimated/communicated even though their components might involve info-hazards? If so, I agree, and it's tricky. They'll likely be some progress on this over the next 6-12 months with Open Phil's project to quantify bio-risk, and (to some extent) the results of UPenn's hybrid forecasting/persuasion tournament on existential risks.
Thanks for this information!
What's the probability we go extinct due to biorisks by 2045 according to you?
Also, I think that things that are extremely infohazardy shouldn't be thought of too strongly bc without the info revelation they will likely remain very unlikely
I'm currently involved in the UPenn tournament so can't communicate my forecasts or rationales to maintain experimental conditions, but it's at least substantially higher than 1/10,000.
And yeah, I agree complicated plans where an info-hazard makes the difference are unlikely, but info-hazards also preclude much activity and open communication about scenarios even in general.
And on AI, do you have timelines + P(doom|AGI)?
I don't have a deep model of AI - I mostly defer to some bodged-together aggregate of reasonable seeming approaches/people (e.g. Carlsmith/Cotra/Davidson/Karnofsky/Ord/surveys).
I think that it's one of the problems that explains why many people find my claim far too strong: in the EA community, very few people have a strong inside view on both advanced AIs and biorisks. (I think that's it's more generally true for most combinations of cause areas).
And I think that indeed, with the kind of uncertainty one must have when one's deferring , it becomes harder to do claims as strong as the one I'm making here.
I don't think this reasoning works in general. A highly dangerous technology could become obvious in 2035, but we could still want actors to not know about it until as late as possible. Or the probability of a leak over the next 10 years could be high, yet it could still be worth trying to maintain secrecy.
Yes, I think you're right actually.
Here's a weaker claim which is I think it true:
- When someone knows and has thought on a infohazard, the baseline is that he's way more likely to cause harm via it than to cause good.
- Thus, I'd recommend anyone who's not actively thinking about ways to solve to prevent classes of scenario where this infohazard would end up being very bad, to try to forget this infohazard and not talk about it even to trusted individuals. Otherwise it will most likely be net negative.
Luisa's post addresses our chance of getting killed 'within decades' of a civilisational collapse, but that's not the same as the chance that it prevents us ever becoming a happy intergalactic civilisation, which is the end state we're seeking. If you think that probability is 90%, given a global collapse, then the effective x-risk of that collapse is 0.1 * <its probability of happening>. One order of magnitude doesn't seem like that big a deal here, given all the other uncertainties around our future.
That's right! I just think that the base rate for "civilisation collapse prevents us from ever becoming a happy intergalactic civilisation" is very low.
And multiplying any probability by 0.1 also does matter because when we're talking about AGI, we're talking about things are >=10% likely to happen for a lot of people (I put a higher likelihood than that but Toby Ord putting 10% is sufficient).
So it means that even if you condition on biorisks being the same as AGI (which is the point I argue against) for everything else, you still need biorisks to be >5% likely to lead to a civilizational collapse by the end of the century for my point to not hold, i.e that 95% of longtermists should work AI (19/20 of the people + assumption of linear returns for the few first thousands ppl).