I have been thinking about this a lot recently, and it seems like a pretty perennial topic on the forum. This post raises good points--I'm especially interested in the idea that EA might be somewhere like a local maximum in terms of cause prioritization, such that if you "reran" EA you'd likely end up somewhere else--I but there are many ways to come at this issue. The general sentiment, as I understand it, is that the EA cause prioritization paradigm seems insufficient given how core it is.

For anyone who's landed here, here's a few very relevant posts, in reverse chronological order:

The "Meta Cause" - Aug '22
The Case Against "Cause Areas" - July '21
Why "cause area" as the unit of analysis? - Jan '21
The Case of the Missing Cause Prioritization Research - Aug '20
The ITN framework, cost-effectiveness, and cause prioritisation - Oct ‘19
On "causes" - June '14
Paul Christiano on Cause Prioritization research - March '14
The case for cause prioritization as the best cause

The apparent lack of a well-organized, public "cause ontology," or a world model which tries to integrate, systematically and extensibly, the main EA theories of change, seems like a research gap, and I've been unable to find any writing which satisfactorily resolves this line of inquiry for me.

Note that it's not just "cause prioritization" per se that this is relevant to, but really any sort of cause ontologies or frameworks for integrating/comparing theories of change. It has to do with the bigger question of how EA ought to structure its epistemic systems, and is also thus relevant to e.g. metascience and collective behavior "studies," to name a few preparadigmatic proto-disciplines.

I was going to write this post, so I definitely agree :)

In general, EA recommendations produce suboptimal herding behavior. This is because individuals can't choose a whole distribution over career paths, only a single career path. Let's say our best guess at the best areas for people to work in is that there's a 30% chance it's AI, a 20% chance it's biosecurity, a 20% chance it's animal welfare, a 20% chance it's global development, and a 10% chance it's something else. Then that would also be the ideal distribution of careers (ignoring personal fit concerns for the moment). But even if every single person had this estimate, all of them would be optimizing by choosing to work in AI, which is not the optimal distribution. Every person optimizing their social impact actually leads to a suboptimal outcome!

The main countervailing force is personal fit. People do not just optimize for the expected impact of a career path, they select into the career paths where they think they would be most impactful. Insofar as people's aptitudes are more evenly distributed, this evens out the distribution of career paths that people choose and brings it closer to the uncertainty-adjusted optimal distribution.

But this is not a guaranteed outcome. It depends on what kind of people EA attracts. If EA attracts primarily people with CS/software aptitudes, then we would see disproportionate selection into AI relative to other areas. So I think another source of irrationality in EA prioritization is the disproportionate attraction of people with some aptitudes rather than others.

A useful thought experiment is to imagine 100 different timelines where effective altruism emerged. How consistent do you think the movement’s cause priorities would be across these 100 different timelines?

This is a useful exercise. I think that in many of these timelines, EA fails to take AI risk seriously (in our timeline, this only happened in the last few years) and this is a big loss. Also probably in a lot of timelines, the relative influence of rationality, transhumanism, philosophy, philanthropy, policy, etc. as well as existing popular movements like animal rights, social justice, etc. is pretty different. This would be good to the extent these movements bring in more knowledge, good epistemics, operational competence, and bad to the extent they either (a) bring in bad epistemics, or (b) cause EAs to fail to maximize due to preconceptions.

My model is something like this: to rank animal welfare as important, you have to have enough either utilitarian philosophers or animal rights activists to get "factory farming might be a moral atrocity" into the EA information bubble, and then it's up to the epistemics of decision-makers and individuals making career decisions. A successful movement should be able to compensate for some founder effects, cultural biases, etc. just by thinking well enough (to the extent that these challenges are epistemic challenges rather than values differences between people).

I do feel a bit weird about saying "where effective altruism emerged" as it sort of implies communities called "effective altruism" are the important ones, whereas I think the ones that focus on doing good and have large epistemic advantages over the rest of civilization are the important ones.

The more time I've put into giving career advice and trying to work out my own career planning, the more I'm drawn to Holden's aptitude advice. In addition to feeling more actionable, identifying aptitudes you could excel at (and then testing those ) is more robust to this cause area uncertainty. 

I mostly agree with this post, but I take issue with the summary:

EA cause prioritisation frameworks use probabilities derived from belief which aren't based on empirical evidence. This means EA cause prioritisation is highly uncertain, which means that the optimal distribution of career focuses for engaged EAs should be less concentrated amongst a small number of "top" cause areas.

As a Bayesian, you have to assign some subjective probabilities to things, and sometimes there just isn't empirical evidence. To argue that e.g. 80k doesn't have enough uncertainty (even if you have reasons to believe that high uncertainty is warranted in general), it's necessary to argue that their methodology is not only subject to irrationality, it's subject to bias in the direction you argue (overconfidence rather than underconfidence in top causes).

E.g. if their numbers for I, T, and 1/N are based on point estimates for each number rather than expected values, then you underrate causes that have a small chance to be big / highly tractable. (I don't know whether 80k actually has this bias.)

The main body of the post does address this briefly, but I would want more evidence, and I think the summary does not have valid standalone reasoning.

To take the concern that “some of them could easily be wrong by a couple of points” literally, this could mean that factory farming could easily be on par with AI, or land use reform could easily be more important than biosecurity.

minor point, but land use reform wouldn't be more important (in the sense of higher scale) than biosecurity, since they differ by 3 points (30x) in overall score but 6 points (1000x) in scale.

Nice post.

All in all, I think it is still the case one should maximise expected value, and terminate deliberation based on resilience, not certainty.

Hi, thanks for writing this. As others have pointed out I am a bit confused how the conclusion (more diversification in EA careers etc) follows from the assumption (high uncertainty about cause prioritisations.

1. You might think that we should be risk averse with respect to our difference-making, i.e. that the EA community does some good in many worlds. See here a summary post from me which collects the arguments against the "risk averse difference-making" view. One might still justify increased diversification for instrumental reasons (e.g. welcomingness of the community), but I don't think that's what you explicitly argue for.
2. You might think that updating that we are more uncertain means that we are more likely to change our minds about causes in the future. If we change our minds about priorities in e.g. 2 or 10 years , it is really advantageous if X members of the community already worked in the relevant cause area. Hence, we should spread out.
   1. However, I don't think that this argument works. First, more uncertainty now might also mean more uncertainty later - hence unclear that I should update that it is more likely that we will change our mind
   2. Secondly, if you think that we can resolve that uncertainty and update in the future, then I think this is a reason for people to work as cause prioritisation researchers and not a reason to spread out among more cause areas.

Thanks for writing this!

I think you make some reasonable points in your post, but I don't think that you make a strong argument for what appears to be your central point, that more uncertainty would and should lead to greater diversity in cause areas.

I think I'd like to see your models for the following points to buy your conclusion

• How much EA uncertainty does the current amount of diversity predict that we have, is this less than you think we 'should' have? My sense is that you're getting more of a vibe we should have more causes, but

• Why does more diversity fall out of more uncertainty? This seems to kind of be assumed but I think the only argument that was made here was the timelines thing which feels like the wrong way to think about this (at least to me).

• A few concrete claims about uncertainty over some crux and why you think that means we are missing [specific cause area].

(You do point to a few reasons for why the diversity of causes may not exist which I think is very helpful - although I probably disagree with the object level takes)

I think if EAs better appreciated uncertainty when prioritising causes, people’s careers would span a wider range of cause areas.

I've got a strong intuition that this is wrong, so I'm trying to think it through. 

To argue that EA’s underestimate uncertainty, you need to directly observe their uncertainty estimates (and have knowledge of the correct level of uncertainty to have). For example, if the community was homogenous and all assigned a 1% chance to Cause X being the most important issue (I'm deliberately trying not to deal with how to measure this) and there was a 99% chance of cause Y being the most important issue, then all individuals would choose to work on Cause Y. If the probabilities were 5% X and 95% you'd get the same outcome. This is because individuals are making single choices. 

Now, if there was a central body coordinating everyone's efforts, in the first scenario, it still wouldn't follow that 1% of people would get allocated to cause Y. Optimal allocation strategy aside, there isn't this clean relationship between uncertainty and decision rules. 

I think 80 000 Hours could emphasise uncertainty more, but also that the EA community as a whole just needs to be more conscious of uncertainty in cause prioritisation.

I think 80k is already very conscious of this (based on my general sense of 80k materials). Global priorities research is one of their 4 highest priorities areas and it's precisely about having more confidence about what is the top priority. 

I think something that would help me understand better where you are coming from is to hear more about what you think the decision rules are for most individuals, how they are taking their uncertainty into account and more about precisely how gender/culture interacts with cause area uncertainty in creating decisions.