I'm shocked and somewhat concerned that your empirical finding is that so few people have encountered or thought about this crucial consideration.

My experience is different, with maybe 70% of AI x-risk researchers I've discussed with being somewhat au fait with the notion that we might not know the sign of future value conditional on survival. But I agree that it seems people (myself included) have a tendency to slide off this consideration or hope to defer its resolution to future generations, and my sample size is quite small (a few dozen maybe) and quite correlated.

For what it's worth, I recall this question being explicitly posed in at least a few of the EA in-depth fellowship curricula I've consumed or commented on, though I don't recall specifics and when I checked EA Cambridge's most recent curriculum I couldn't find it.

My impression was that due to multiple accusations of sexual harassment, Jacy Reese Anthis was stepping back from the community. When and why did this stop?

He was evicted from Brown University in 2012 for sexual harassment (as discussed here).

And he admitted to several instances of sexual harassment (as discussed here).

He also lied on his website about being a founder of effective altruism.

I like "quality risks" (q-risks?) and think this is more broadly appealing to people who don't want to think about suffering-reduction as the dominantly guiding frame for whatever reason. Moral trade can be done with people concerned with other qualities, such as worries about global totalitarianism due to reasons independent of suffering such as freedom and diversity. 

It's also relatively more neglected than the standard extinction risks, which I am worried we are collectively Goodharting on as our focus (and to a lesser extent, focus on classical suffering risks may fall into this as well). For instance, nuclear war or climate change are blatant and obvious scary problems that memetically propagate well, whereas there may be many q-risks to future value that are more subtle and yet to be evinced.

Tangentially, this gets into a broader crux I am confused by: should we work on obvious things or nonobvious things? I am disposed towards the latter. 

Look into suffering-focused AI safety which I think is extremely important and neglected (and s-risks).

Thanks for this! I agree this is a very important question, I sympathize with the view that people overweight some arguments for historical optimism, and I'm mostly on board with the list of considerations. Still, I think your associated EV calculation has significant weaknesses, and correcting for these seems to produce much more optimistic results.

• You put the most weight on historical harms, and you also put a lot of weight on empirical utility asymmetry. But arguably, the future will be deeply different from the past (including through reduced influence of biological evolution), so simple extrapolation from the past or present should not receive very high weight. (For the same reason, we should also downweight historical progress.)
• Arguably, historical harms have occurred largely through the divergence of agency and patiency, so counting both is mostly double-counting. (Similarly, historical progress has largely occurred through the other mechanisms that are already covered.) So we should further downweight these.
• I don't see why we should put negative weight on "The Nature of Digital Minds, People, and Sentience."
• Reasoned cooperation should arguably receive significantly more weight as an argument for optimism; moral trade should allow altruists to substantially mitigate suffering and increase well-being, especially as human tools for shaping the world and efficiently coordinating continue to improve.
• Editing the calculation to account for all of the above (and ignoring my other, more minor quibbles), we reach a fairly optimistic result (especially if we're looking at the more relevant logarithmic sum).

(Edited to account for how downweighting history should mean downweighting historical progress, not just downweighting historical harms, and for phrasing tweaks.)

I've considered a possible pithy framing of the Life Despite Suffering question as a grim orthogonality thesis (though I'm not sure how useful it is):

We sometimes point to the substantial majority's revealed preference for staying alive as evidence of a 'life worth living'. But perhaps 'staying-aliveness' and 'moral patient value' can vary more independently than that claim assumes. This is the grim orthogonality thesis.

An existence proof for the 'high staying-aliveness x low moral patient value' quadrant is the complex of torturer+torturee, which quite clearly can reveal a preference for staying alive, while quite plausibly being net negative value.

Can we rescue the correlation of revealed 'staying-aliveness' preference with 'life worth livingness'?

We can maybe reason about value from the origin of moral patients we see, without having a physical theory of value. All the patients we see at present are presumably products of natural selection. Let's also assume for now that patienthood comes from consciousness.

Two obvious but countervailing observations

• to the extent that conscious content is upstream of behaviour but downstream of genetic content, natural selection will operate on conscious content to produce behaviour which is fitness-correlated
  • if positive conscious content produces attractive behaviour (and vice versa), we might anticipate that an organism 'doing well' according to suitable fitness-correlates would be experiencing positive conscious content
  • this seems maybe true of humans?
• to the extent that behaviour is downstream of non-conscious control processes, natural selection will operate on non-conscious control processes to produce behaviour which is fitness-correlated
  • we can not rule out experiences 'not worth living' which nevertheless produce net revealed staying-aliveness preference, if the behaviour is sufficiently under non-conscious control, or if the selection for behaviour downstream of negative conscious experience is weak
    • weak selection is especially likely in novel out-of-distribution situations
  • in general, organisms which reveal preferences for not staying alive will never be ceteris paribus fitter (though there are special cases of course)

For non-naturally-selected moral patients, I think even the above bets are basically off.

See also Robert Harling's Good v. Optimal Futures.

I find the following simple argument disturbing:

P1 - Currently, and historically, low power being (animals, children, old dying people) are treated very cruelly if treating them cruelly benefits the powerful even in minor ways. Weak benefits for the powerful empirically justify cruelty at scale.
P2 - There is no good reason to be sure the powerful wont have even minor reasons to be cruel to the powerless (ex: suffering sub-routines, human CEV might include spreading earth like life widely or respect for tradition)
P3 - Inequality between agents is likely to become much more extreme as AI develops
P4 - The scale of potenital suffering will increase by many orders of magnitude

C1 - We are fucked?

Personal Note - There is also no reason to assume me or my loved ones will remain relatively powerful beings

C2 - Im really fucked!

I think considerations like these are important to challenge the recent emphasis on grounding x-risk (really, extinction risk) in near-term rather than long-term concerns. That perspective seems to assume that the EV of human expansion is pretty much settled, so we don’t have to engage too deeply with more fundamental issues in prioritization, and we can instead just focus on marketing.

I’d like to see more written directly comparing the tractability and neglectedness of population risk reduction and quality risk reduction. I wonder if you’ve perhaps overstated things in claiming that a lower EV for human expansion suggests shifting resources to long-term quality risks rather than, say, factory farming. It seems like this claim requires a more detailed comparison between possible interventions.

The thing I have most changed my mind about since writing the post of mine you cite is adjacent to the "disvalue through evolution" category: basically, I've become more worried that disvalue is instrumentally useful. E.g. maybe the most efficient paperclip maximizer is one that's really sad about the lack of paperclips.

There's some old writing on this by Carl Shulman and Brian Tomasik; I would be excited for someone to do a more thorough write up/literature review for the red teaming contest (or just in general).

As a small comment, I believe discussions of consciousness and moral value tend to downplay the possibility that most consciousness may arise outside of what we consider the biological ecosystem.

It feels a bit silly to ask “what does it feel like to be a black hole, or a quasar, or the Big Bang,” but I believe a proper theory of consciousness should have answers to these questions.

We don’t have that proper theory. But I think we can all agree that these megaphenomena involve a great deal of matter/negentropy and plausibly some interesting self-organized microstructure- though that’s purely conjecture. If we’re charting out EV, let’s keep the truly big numbers in mind (even if we don’t know how to count them yet).

Thanks for the post.  I've also been surprised how little this is discussed, even though the value of x-risk reduction is almost totally conditional to the answer to this question (the EV of the future conditional on human/progeny survival). Here are my big points to bring up re this issue, though some might be slight rephrasing of yours. 

1. Interpersonal comparisons of utility canonically have two parts - a definition of utility, of which every sentient being is measured by. Then, to compare and sum utility, one must pick (subjective) weights for each sentient being, scale their utility by the weights, and add everything up. (u_1x_1+.....+u_nx_n). If we don't agree on the weights, it's possible that one person may think the future be in expectation positive while another thinks it will be negative even w/ perfect information of what the future will look like. It could be even harder to agree on the weights of sentient beings when we don't even know what agents are going to be alive. We have obvious candidates for general rules about how to weight utility (brain size, pain receptors, etc.) but who knows how are conceptions of these things will change.
2. Basically repeating your last point in the chart but it's really important so I'll reiterate. Like everything else normative, there is no objective "0" line, no non-arbitrary point at which life is worth living. It is a decision we have to make. Moreover, I don't see any agreement in this community on the specific point where life is worth living. It is pretty obvious that disagreement about this could flip the sign of the EV of the future.
3. "Alien Counterfactuals". I actually mentioned this in a comment to a previous post where someone said we should mostly just call longtermism x-risk(extremely wrong in my opinion).  First, for simplicity, let's just assume humans become grabby. If we become grabby, a question of specific interest to us should be, what characteristics do our society and species have relative to other grabby societies/species? Are we going to be better or worse gatekeepers of the future than the other gatekeepers of the future?  I'm pretty sure we should take the prior that we display the mean characteristics of a grabby civilization (interested in hearing if others disagree). If this is the case, then, again for simplicity, assuming(for simplicity) that our lightcone will be populated by aliens whether or not we specifically become grabby, x-risk reduction could be argued to have exactly 0 expected value, as we have no reason to believe that we are going to do a better job with the future than aliens. Evidence updating against the prior would probably take the form of arguments about why our specific evolutionary or economic history was a weird way to become grabby, not an easy task. Of course even with all the simplifying assumptions I've made, it's not so simple. Even if we have the mean characteristics of all the other grabby civilizations, adding more civilizations to the mix can change the game theory of space wars and governance. Still, it's not clear if more or less players is better.                                                                                                                                                                                                                                                                                                                                                                                                    I talked to a few people in EA about 'alien counterfactuals', and they all seemed to dismiss the argument, thinking that humans are better than "rolling the dice" on a new grabby civilization. No one provided arguments that were super convincing though. The most convincing counter argument I heard was  that it is very unlikely that grabby aliens will actually end up existing in our lightcone, subverting the whole argument. AI makes this argument significantly more confusing but it's not worth getting into without further ironing out of the initial arguments.
4. And then this is sort of the whole point of your post but I will reiterate - predicting the future is extremely difficult. We should have very little confidence in what it will be like. Predicting whether the future will be good or bad (given that we have already ironed out the normative considerations, which we haven't) is probably easier than predicting the future but still seems really difficult. The burden of evidence is on us to prove the future will be good, not on other people to prove it will be bad. After all, we are pumping huge amounts of money into creating impact which is completely conditional on this information. I've found posts like this one to be the only type of things that even feel tractable, and if that is the level of specificity we are at, it truly does feel like we have been p. wagered on this issue. posts like this one that you mentioned ultimately don't have nearly enough firepower to serve as anything more than an exploration of what a full argument would look like.

Typo hint:

"10³⁸" hasn't rendered how you hoped. You can use <dollar>10^{38}<dollar> which renders as ${10}^{38}$

I think that this large argument / counterargument table is a great example of how using a platform like Kialo to better structure debates could be valuable.

Thanks for doing this work but I dont have the patience to read entirely. What is it you found exactly? please put at the top of the summary

Only a good moral foundation can keep AI at minimal risk of putting humans in extinction level events imo. Possibly giving them a Pantheon to idolize could be all they need to spark this type of 'thought'. It's worked in civilizations since the dawn of time. Why not try some models with this in mind?

Minor technical comment: the links to subsections in the topmost table link to the Google docs version of the article, and I think it would be slightly nicer if they linked to the forum post version.

I think you have undervalued optionality value. Using Ctrl + F I have tried to find and summarise your claims against optionality value: 

• EA only has a modest amount of "control" [ I assuming control = optionality ]
•  EA won't retain much "control" over the future
• The argument for option value is based on circular logic
•  Counterpoint, short x-risk timelines would be good from the POV of someone making an optionality value argument
• Counterpoint, optionality would be more important if alien's exist and propagate negative value
•  humans existing limits option value similar [question, by similar do you mean equal to?] to that of non-existence
• We can't raise x-risk after we've lowered it 

Without having thought about this for very long, I think the argument against optionality needs to be really really strong. Since you essentially need to demonstrate we have equal or better decision making abilities right now, than at any point in the future. 

One of the reasons optionality seems like an exceptionally good argument, is that uncertainty exists both inside and outside EV models (i.e. you can model EV, and include some uncertainty, but then you need to account for uncertainty around the entire EV model because you've likely made a ton of assumptions during the process). And it's extremely likely this uncertainty would remain constant overtime. One way we try to improve our models of the world is by making predictions and seeing if we were correct. The two reasons we do this are: making predictions is hard (so it's test for a model that's hard to pass) , and we have more information in the future. 

The argument against optionality seems borderline tautological, because you essentially have to round all optionality value to 0, meaning the value of making predictions (and all over science, philosophy ect.) is also 0. 

I am basically making a fanatical argument here for optionality, whereby the only  consideration that trumps it is opportunity cost. 

You wrote 

"There is a substantial philosophical literature on such topics that I will not wade into, and I believe such non-value-based arguments can be mapped onto value-based arguments with minimal loss (e.g., not having a duty to make happy people can be mapped onto there being no value in making happy people)."

Duty to accomplish X implies much more than an assessment of the value of X. To lack the (moral, legal, or ethical) obligation to bring about a state of affairs does not imply a sense that the state of affairs has no value to you or others.

...