Interesting. Who might these people be who deliberately feed you biased information? How do they benefit from you focusing on cause area y instead of cause area z?

One simple model is:

• each person can choose only one cause area
• errors are iid across people
• the x-risk coming from each cause decreases with each additional person working on this cause

Whether diversification is better (in expectation) depends on how a cause's x-risk decreases as additional people work on this cause. If x-risk decreases linearly (the 1000th person makes the same marginal contribution as the 1st), then diversification is not better in expectation. But if the contribution to x-risk prevention is marginally decreasing in people, diversification is better. 

(By diversification I mean each person choosing their top estimated x-risk cause individually. But it can also mean that some people deliberately do not work on the cause with the highest aggregated risk estimate.)

Given the optimizer's cause, how do I optimally pick a cause area? Two observations:

1. In the standard model above, I cannot improve upon picking the cause with the highest estimate. While this cause will likely be overestimated by several OOMs and will likely not be the top x-risk cause, it is still the optimal cause to choose in expectation.
2. But in the extended model with grounded and speculative causes, the logic changes. Here, it can be optimal to pick the cause with the highest x-risk estimate in the grounded class, even if there are causes with higher x-risk estimates in the speculative class.

I think this has very interesting implications. 

2) implies that working in a more grounded cause area (like global health?) can be better than working on speculative x-risk. This is a powerful implication and I think EA should take this very seriously.

1) implies that even if everyone makes an individually optimal cause-prioritization decision, some people's top causes will still look highly implausible to others. 

Here is a power law pattern for causes of death in the US. 22 causes have a share of total deaths between 1% and 10%, and many more causes have much smaller shares. Makes sense, right? 

Okay, so power law here means: Of all the possible causes of event Y, most will have a relatively small probability of causing Y, and only a few will have a higher probability of causing Y. To me, this makes sense, and I would expect this to hold across many domains.

Questions for clarification: 

1) "This means that probability values that are 10 times higher are 10 times common." Shouldn't it be probabilities that are 10 times lower are 10 times more common?

2) In the section on speculative bias, you say that "grounded and speculative threats are identical in all ways, except that the speculative threats are much more uncertain". Shouldn't the frequencies of grounded actual (blue) and speculative actual (yellow) look the same then?
 

How can we be sure that there aren’t a couple dozen more zeroes in there?

I think that's a great point! I think the behavioral econ/psychology literature should make us cautious too:

• people overestimate small frequencies [^1]
• and even when told the true probability, people overweight small probabilities (this is standard prospect theory) [^2]

[^1] Lichtenstein, Sarah, Paul Slovic, Baruch Fischhoff, Mark Layman, and Barbara Combs. Judged Frequency of Lethal Events.

[^2] Barberis, Nicholas C. “Thirty Years of Prospect Theory in Economics: A Review and Assessment.” Journal of Economic Perspectives 27, no. 1 (2013): 173–96. https://doi.org/10.1257/jep.27.1.173.

 I think this depends on whether farmed or wild animal welfare matters more. I don't have an answer, so let's treat it as 50/50. 

1. If wild animals matter more, what could happen?  On the upside, AGI might enable us to help wild animals.  On the downside, it might lead to humans creating biospheres on other planets, which would increase the suffering of wild animals by many orders of magnitude.
2. If farmed animals matter more, the upside could be that AGI enables us to substitute farmed animals completely (cultivated meat, etc.). The downside could be that people get richer and want to eat more meat, or that AGI changes the production of farmed animals in a way that increases suffering. 

Again, I don't know whether the upside or downside in each scenario is more likely.  Let's say each is 50/50 again.  I think this makes 1) EV negative and 2) EV positive, with the aggregate being slightly EV negative.

Great post!

I keep thinking about labor shares though. Yes, wages might rise in a world with AGI, so far so good. But I still worry about the implications of a decreasing labor share.

1. I think my main concern is a political economy one. A higher capital share might mean that political power is concentrated among few capital owners, which in turn affects the welfare state and so on. This argument is not new, you can find it in this essay and elsewhere. I might try to turn this into a formal model. Do you know if such work exists already?
2. One could also make a reference point argument: My utility is determined by my consumption, but also by my standing relative to others. I might be worse of then if some people's income explodes, but mine rises only somewhat.