Epistemic status: Thinking out loud. My biggest priority is to minimise x-risk, I’m interested first in understanding where to focus the bulk of my attention. To some extent that's simply a function of my estimated timelines for AGI, but writing this helped me sort through how I think about this which will help me if my estimates change or I need to communicate to others with different estimates.
TLDNR: I define some terminology to make arguments about which x-risks longtermists should prioritise as a function of their risk and personal tractability estimates.
Theory
Assumptions:
- AI and Bio x-risks pose significantly more risk than any others (therefore we can just discuss those risks).
- Any x-risk that is much larger than any other should be prioritised by those interested in minimising x-risk.
- (modifying 2.) It is possible for the potential contribution of an individual to be greater in one x-risk than another to the extent that this should overcome the imperative to work on the greatest x-risk. (eg: A professor of virology with lots of political capital might be best placed on bio-security).
Terminology:
- RAI is the risk posed by AI (Probability that AI causes human extinction before 2100)
- RBio is the risk posed by Biosecurity threats / GBCRs etc (Probability that Biological Threats causes human extinction before 2100)
- > is greater (< 10x greater)
- >> is much greater (~ 100 x greater)
- >>> is much much greater (~1000 x greater or more)
- ~> is greater in any sense.
- TAI is the tractability of the individual working on AI (eg: Expected reduction in extinction probability by choosing to specialise on this x-risk for x years)
- TBio is the tractability of the individual working on Biosecurity threats
Note: Since a lot of arguments could be reversed and AI is generally considered more dangerous, I’ll only consider scenarios where RAI is larger than RBio by different magnitudes.
If RAI > RBio then a portfolio approach in general makes sense. For example, iirc, in The Precipice, Toby Ord suggests that RAI ~ 3 RBio and so would probably want anyone with TBio > TAI working on Bio.
However, if RAI >> RBio or RAI >>> RBio then unless TAI~0 or TBio>>>TAI, that individual will probably be most efficiently placed on AI.
My intuition tells me that TBio >>> TAI is a distinct possibility for some individuals, but not that it is likely. It is probably less likely to be true for early career individuals. It has been argued that max(Ti/Tj) < 100 (the maximum ratio of tractability on any given cause area is less than 100), this was done within the context of the ITN framework which isn’t quite the same thing as individual tractability.
I’d like to note that the definition and estimate of Ti seems to be really important. When I started thinking about where I could work directly I had this vague notion of the worlds in which I did work on a cause and comparing those to the worlds in which I didn’t work on a given cause. On further inspection however, it feels like maybe there are two separate notions here:
- The inherent tractability of a problem. This is the part of the tractability that can be easily discussed on the EA forum because it relates more to the problem itself and can be estimated with public or expert information.
- The personal tractability of a problem. This is the part of the tractability that is associated with the individual's contribution. This seems really messy to estimate for a bunch of reasons.
Application
A concrete example will demonstrate how the above arguments appear to be already playing out on the EA forum. In “Longtermists Should Work on AI - There is No "AI Neutral" Scenario - EA Forum” the argument is put forward that RAI >> RBio or even RAI>>>RBio and that simultaneously TBio is unlikely to be significantly less than TAI
I.e., you have skills or career capital that make it suboptimal for you to switch into AI. This is possible, but given that both AI Governance and AI Safety need a wide range of skills, I expect this to be pretty rare.
By wide range, I mean very wide. So wide that I think that even most longtermists with a biology background who want to maximize their impact should work on AI.
The most upvoted comment replies with the sentiment that while RAI > Ri, for many individuals Ti >> TAI. (i not Bio because the example was nuclear x-risk).
However, I think your post is too dismissive of working on other existential risks. Reducing the chance that we all die before building AGI increases the chance that we build AGI. While there probably won't be a nuclear war before AGI, it is quite possible that a person very well-suited to working on reducing nuclear issues could reduce x-risk more by working to reduce nuclear x-risk than they could by working more directly on AI.
I thinks it's clear that there are two (albeit related) conversations happening simultaneously here. A question about relative risk and hypothetical relative tractability.
Note: Important caveats here are that maybe there’s lots of information on the bio side that is being filtered because of information hazards and that few people share a deep understanding of both RAI and RBio. This might make it particularly hard to compare RAI and RBio
Conclusion
The value of this kind of formalisation could be:
- It makes it easier to communicate precisely the key ideas or points of disagreement. Making ideas concise and explicit makes it easier to see their flaws/assumptions.
- Algebraic decision rules can enable alignment on “in theory” arguments which will lead to cleaner back-forth discussions that focus more on estimates or hypothetical individuals making decisions.
Questions:
- Does this seem like a useful conception of the relevant concepts relating to choice of direct work for longtermists? If not, is there a better way to think about it?
- How problematic are the assumptions?
- What is the probability that TAI ~ 0? If an individual can’t find a role then I assume TAI is 0, but it seems to me like that might be short term. Given the importance of the problem, we should be working towards maximum reduction which may involve hiring pretty much everyone interested in working on AI. (eg: policy work, advocacy).
