Yeah I agree - expected utility maximisation really starts to fall apart in this existential risk regime, even over trajectories rather than applied statically, and it only makes sense “locally” and at the margin. 

Personally I’m very happy to bite the bullet and not be rigorously utilitarian, but I’m also a global health focussed “old school EA” thinking about how much to diversify donations across charities ;)
 

I guess the obvious question is: Where do you draw the line regarding who to include in "the community", why, and how does it affect the results? 

An explicit swap is very clean and the counterfactual is known. Could they make a non-official, un-enforcable deal (I am very much not an expert on this at all and much less a lawyer!)?

Thinking about how ones situation impacts effective opportunities is also reasonable [1], but who is to say that they have not already assessed their specific viable options and are still convinced that their preferred choice is the best? People want to use their donations to "vote" and express an opinion, and that's probably valuable?

[1] I suspect this is relatively common among people donating to underfunded areas in Animal welfare.

While this sounds intuitively right, I think in the simplest utility maximising setting (iid additive errors with mean zero) your first claim does not seem true? The best looking noisy option is still most likely to be the best?

(I need to think more about the maths, but at least you need some kind of shrinkage to a prior that can change the ranking, which you’re unlikely to get, and if you’re maximising utility the solution is always fully concentrated?)

To be clear, it need not be deliberate and they need not benefit personally!
 

I was more referring to the diversification as implied by “don't focus the vast majority of efforts on one cause”, which to me meant more “if you’re a decision maker over some amount of resources, you should diversify the allocation across cause areas”. Which I agree with, but it’s quite hard to really justify.

Yes, via nonlinearity you can get to diversification, but this means making additional assumptions beyond sampling error/publication bias/optimiser curse  type effects. 
The nonlinearity you’re describing matters on a movement level, but not on a individual decision makers level. 
”Impact risk aversion” is another mechanism to get diversification which I think can be reasonable in cases where eg low impact reduces the probability of future donations or similar.

One channel I think is under explored and might work well as a justification for diversification in practice is something like this (I haven’t thought about this rigorously though): if I predictably optimise and my objective function is known to others, they will (in the worst case, possibly thru misaligned incentives) feed me biased information to influence my decision, and optimisation is very sensitive to noise, therefore I subject myself to adverse selection. So basically by not optimising but diversifying across good options you reduce the negative impact of this type of adverse selection. In this case, the “errors” are not iid. Hard to say how much diversification that yields. 

I like diversification as a reaction to this type of uncertainty, but it does not trivially follow? I might be missing something - do you have a favourite minimal set of assumptions that rigorously yield diversification as a function of this?

Or perhaps the conclusion could just be “if you want to try very hard, don’t forget to have good epistemics and course correct if needed”?

Cause prioritisation is a function of the marginal impact on the outcome per marginal dollar/hour spent or similar. 

This adds another layer of complexity because you can’t just eg “integrate out” the existential risk first and then reason about the impact, you kind of need to do it jointly. 

This means you also need to include sources of impact uncertainty. 
To me, this ~ removes many longtermist cause areas because it drags most of the “impact mass” too close to zero - but people have different opinions on this of course. 

Point 1 is interesting - you can do better if you have and incorporate a prior, but the question is where that comes from. I think often it’s easier to have priors over intervention success than existential outcomes per se. 

An important (and to me fairly open) related question is to what extent this ends up being action-guiding? 
Eg if I lower my probability estimate of this materialising by 10 percentage points, how much will it affect resources I spend on helping to prepare for the possible outcome? Perhaps people here have thoughts on that - my impression is that working on improving the future opportunity set for such donations is relatively robustly good right now?