Hide table of contents

The idea that the future might contain astronomical amounts of suffering, and that we should work to prevent such worst-case outcomes, has lately attracted some attention. I've written this FAQ to help clarify the concept and to clear up potential misconceptions.

[Crossposted from my website on s-risks.]

General questions

What are s-risks?

In the essay Reducing Risks of Astronomical Suffering: A Neglected Priority, s-risks (also called suffering risks or risks of astronomical suffering) are defined as “events that would bring about suffering on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far”.

If you’re not yet familiar with the idea, you can find out more by watching Max Daniel's EAG Boston talk or by reading the introduction to s-risks.

Can you give an example of what s-risks could look like?

In the future, it may become possible to run such complex simulations that the (artificial) individuals inside these simulations are sentient. Nick Bostrom coined the term mindcrime for the idea that the thought processes of a superintelligent AI might cause intrinsic moral harm if they contain (suffering) simulated persons. Since there are instrumental reasons to run many such simulations, this could lead to vast amounts of suffering. For example, an AI might use simulations to improve its knowledge of human psychology or to predict what humans would do in a conflict situation.

Other common examples include suffering subroutines and spreading wild animal suffering to other planets.

Isn’t all that rather far-fetched?

At first glance, one could get the impression that s-risks are just unfounded speculation. But to dismiss s-risks as unimportant (in expectation), one would have to be highly confident that their probability is negligible, which is hard to justify upon reflection. The introduction to s-risks gives several arguments why the probability is not negligible after all:

First, s-risks are disjunctive. They can materialize in any number of unrelated ways. Generally speaking, it’s hard to predict the future and the range of scenarios that we can imagine is limited. It is therefore plausible that unforeseen scenarios – known as black swans – make up a significant fraction of s-risks. So even if any particular dystopian scenarios we can conceive of is highly unlikely, the probability of some s-risk may still be non-negligible.

Second, while s-risks may seem speculative at first, all the underlying assumptions are plausible. [...]

Third, historical precedents do exist. Factory farming, for instance, is structurally similar to (incidental) s-risks, albeit smaller in scale. In general, humanity has a mixed track record regarding responsible use of new technologies, so we can hardly be certain that future technological risks will be handled with appropriate care and consideration.

Which value systems should care about reducing s-risks?

Virtually everyone would agree that (involuntary) suffering should, all else equal, be avoided. In other words, ensuring that the future does not contain astronomical amounts of suffering is a common denominator of almost all (plausible) value systems.

Work on reducing s-risks is, therefore, a good candidate for compromise between different value systems. Instead of narrowly pursuing our own ethical views in potential conflict with others, we should work towards a future deemed favourable by many value systems.

The future

Aren't future generations in a much better position to do something about this?

Future generations will probably have more information about s-risks in general, including which ones are the most serious, which does give them the upper hand in finding effective interventions. One might, therefore, argue that later work has a significantly higher marginal impact. However, there are also arguments for working on s-risks now.

First, thinking about s-risks only as they start to materialize does not suffice because it might be too late to do anything about it. Without sufficient foresight and caution, society may already be “locked in” to a trajectory that ultimately leads to a bad outcome.

Second, one main reason why future generations are in a better position is that they can draw on previous work. Earlier work – especially research or conceptual progress – can be effective in that it allows future generations to more effectively reduce s-risk.

Third, even if future generations are able to prevent s-risks, it’s not clear whether they will care enough to do so. We can work to ensure this by growing a movement of people who want to reduce s-risks. In this regard, we should expect earlier growth to be more valuable than later growth.

Fourth, if there’s a sufficient probability that smarter-than-human AI will be built in this century, it's possible that we already are in a unique position to influence the future. If it’s possible to work productively on AI safety now, then it should also be possible to reduce s-risks now.

Toby Ord’s essay The timing of labour aimed at reducing existential risk addresses the same question for efforts to reduce x-risks. He gives two additional reasons in favor of earlier work: namely, the possibility of changing course (which is more valuable if done early on) and the potential for self-improvement.

Seeing as humans are (at least somewhat) benevolent and will have advanced technological solutions at their disposal, isn’t it likely that the future will be good anyway?

If you are (very) optimistic about the future, you might think that s-risks are unlikely for this reason (which is different from the objection that s-risks seem far-fetched). A common argument is that avoiding suffering will become easier with more advanced technology; since humans care at least a little bit about reducing suffering, there will be less suffering in the future.

While this argument has some merit, it’s not airtight. By default, when we humans encounter a problem in need of solving, we tend to implement the most economically efficient solution, often irrespective of whether it involves large amounts of suffering. Factory farming provides a good example of such a mismatch; faced with the problem of producing meat for millions of people as efficiently as possible, a solution was implemented which happened to involve an immense amount of nonhuman suffering.

Also, the future will likely contain vastly larger populations, especially if humans colonize space at some point. All else being equal, such an increase in population may also imply (vastly) more suffering. Even if the fraction of suffering decreases, it's not clear whether the absolute amount will be higher or lower.

If your primary goal is to reduce suffering, then your actions matter less if the future will 'automatically' be good (because the future contains little or no suffering anyway). Given sufficient uncertainty, this is reason to focus on the possibility of bad outcomes anyway for precautionary reasons. In a world where s-risks are likely, we can have more impact.

Does it only make sense to work on s-risks if one is very pessimistic about the future?

Although the degree to which we are optimistic or pessimistic about the future is clearly relevant to how concerned we are about s-risks, one would need to be unusually optimistic about the future to rule out s-risks entirely.

From the introduction to s-risks:

Working on s-risks does not require a particularly pessimistic view of technological progress and the future trajectory of humanity. To be concerned about s-risks, it is sufficient to believe that the probability of a bad outcome is not negligible, which is consistent with believing that a utopian future free of suffering is also quite possible.

In other words, being concerned about s-risks does not require unusual beliefs about the future.

S-risks and x-risks

How do s-risks relate to existential risks (x-risks)? Are s-risks a subclass of x-risks?

First, recall Nick Bostrom’s definition of x-risks:

Existential risk – One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.

S-risks are defined as follows:

S-risks are events that would bring about suffering on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far.

According to these definitions, both x-risks and s-risks relate to shaping the long-term future, but reducing x-risks is about actualizing humanity’s potential, while reducing s-risks is about preventing bad outcomes.

There are two possible views on the question of whether s-risks are a subclass of x-risks.

According to one possible view, it’s conceivable to have astronomical amounts of suffering that do not lead to extinction or curtail humanity’s potential. We could even imagine that some forms of suffering (such as suffering subroutines) are instrumentally useful to human civilization. Hence, not all s-risks are also x-risks. In other words, some possible futures are both an x-risk and an s-risk (e.g. uncontrolled AI), some would be an x-risk but not an s-risk (e.g. an empty universe), some would be an s-risk but not an x-risk (e.g. suffering subroutines), and some are neither.

    Yes No
X-risk? Yes Uncontrolled AI Empty universe
No Suffering subroutines Utopian future

The second view is that the meaning of “potential” depends on your values. For example, you might think that a cosmic future is only valuable if it does not contain (severe) suffering. If “potential” refers to the potential of a utopian future without suffering, then every s-risk is (by definition) an x-risk, too.

How do I decide whether reducing extinction risks or reducing s-risks is more important?

This depends on each of us making difficult ethical judgment calls. The answer depends on how much you care about reducing suffering versus increasing happiness, and how you would make tradeoffs between the two. (This also raises fundamental questions about how happiness and suffering can be measured and compared.)

Proponents of suffering-focused ethics argue that the reduction of suffering is of primary moral importance, and that additional happiness cannot easily counterbalance (severe) suffering. According to this perspective, preventing s-risks is morally most urgent.

Other value systems, such as classical utilitarianism or fun theory, emphasize the creation of happiness or other forms of positive value, and assert that the vast possibilities of a utopian future can outweigh s-risks. Although preventing s-risks is still valuable in this view, it is nevertheless considered even more important to ensure that humanity has a cosmic future at all by reducing extinction risks.

In addition to normative issues, the answer also depends on the empirical question of how much happiness and suffering the future will contain. David Althaus suggests that we consider both the normative suffering-to-happiness trade ratio (NSR), which measures how we would trade off suffering and happiness in theory, and the expected suffering-to-happiness ratio (ESR), which measures the (relative) amounts of suffering and happiness we expect in the future.

In this framework, those who emphasize happiness (low NSR) or are optimistic about the future (low ESR) will tend to focus on extinction risk reduction. If the product of NSR and ESR is high – either because of a normative emphasis on suffering (high NSR) or pessimistic views about the future (high ESR) – it’s more plausible to focus on s-risk-reduction instead.


Is the concept of s-risks tied to the possibility of AGI and artificial sentience?

Many s-risks, such as suffering subroutines or mindcrime, have to do with artificial minds or smarter-than-human AI. But the concept of s-risks is not conceptually dependent on the possibility of AI scenarios. For example, spreading wild animal suffering to other planets does not require artificial sentience or AI.

Examples often involve artificial sentience, however, due to the vast number of artificial beings that could be created if artificial sentience becomes feasible at any time in the future. Combined with humanity’s track record of insufficient moral concern for “voiceless” beings at our command, this might pose a particularly serious s-risk. (More details here.)

Why would we think that artificial sentience is possible in the first place?

This question has been discussed extensively in the philosophy of mind. Many popular theories of consciousness, such as Global workspace theory, higher-order theories, or Integrated information theory, agree that artificial sentience is possible in principle. Philosopher Daniel Dennett puts it like this:

I’ve been arguing for years that, yes, in principle it’s possible for human consciousness to be realised in a machine. After all, that’s what we are. We’re robots made of robots made of robots. We’re incredibly complex, trillions of moving parts. But they’re all non-miraculous robotic parts.

As an example of the sort of reasoning involved, consider this intuitive thought experiment: if you were to take a sentient biological brain, and replace one neuron after another with a functionally equivalent computer chip, would it somehow make the brain less sentient? Would the brain still be sentient once all of its biological neurons have been replaced? If not, at what point would it cease to be sentient?

The debate is not settled yet, but it seems at least plausible that artificial sentience is possible in principle. Also, we don’t need to be certain to justify moral concern. It’s sufficient that we can't rule it out.

Ok, I’m sold. What can I personally do to help reduce s-risks?

A simple first step is to join the discussion, e.g. in this Facebook group. If more people think and write about the topic (either independently or at EA organizations), we’ll make progress on the crucial question of how to best reduce s-risks. At the same time, it helps build a community that, in turn, can get even more people involved.

If you’re interested in doing serious research on s-risks right away, you could have a look at this list of open questions to find a suitable research topic. Work in AI policy and strategy is another interesting option, as progress in this area allows us to shape AI in a more fine-grained way, making it easier to identify and implement safety measures against s-risks.

Another possibility is to donate to organizations working on s-risks reduction. Currently, the Foundational Research Institute is the only group with an explicit focus on s-risks, but other groups also contribute to solving issues that are relevant for s-risk reduction. For example, the Machine Intelligence Research Institute aims to ensure that smarter-than-human artificial intelligence is aligned with human values, which probably also reduces s-risks. Charities that promote broad societal improvements such as better international cooperation or beneficial values may also contribute to s-risk reduction, albeit in a less targeted way.

[Disclaimer: I'm in close contact with the Foundational Research Institute, but I am not employed there and don't receive any financial compensation.]





More posts like this

Sorted by Click to highlight new comments since:

Thanks for writing this up! Having resources like this explaining ideas seems pretty uncontroversially good.

This is very helpful, as is Max's write-up linked to at the top.

One thing I find meta-interesting about s-risk is that s-risk is included in the sort of thing we were pointing at in the late 90s before we started talking about x-risk, and so to my mind s-risk has always been part of the x-risk mitigation program but, as you make clear, that's not how it's been communicated.

I wonder if there are types of risks for the long-term future we implicitly would like to avoid but have accidentally explicitly excluded from both x-risk and s-risk definitions.

the sort of thing we were pointing at in the late 90s before we started talking about x-risk

I'd be interested to hear more about that if you want to take the time.

My memory is somewhat fuzzy here because it was almost 20 years ago, but I seem to recall discussions on Extropians about far future "bad" outcomes. In those early days much of the discussion focused around salient outcomes like "robots wipe out humans" that we picked up from fiction or outcomes that let people grind their particular axes (capitalist dystopian future! ecoterrorist dystopian future! ___ dystopian future!), but there was definitely more serious focus around some particular issues.

I remember we worried a lot about grey goo, AIs, extraterrestrial aliens, pandemics, nuclear weapons, etc. A lot of it was focused on getting wiped out (existential threats), but some of it was about undesirable outcomes we wouldn't want to live in. Some of this was about s-risks I'm sure, but I feel like a lot of it was really more about worries over value drift.

I'm not sure there's much else there, though. We knew bad outcomes were possible, but we were mostly optimistic and hadn't developed anything like the risk-avoidance mindset that's become relatively more prevalent today.

This sentence in your post caught my attention: " Even if the fraction of suffering decreases, it's not clear whether the absolute amount will be higher or lower."

To me, it seems like suffering should be measured by suffering / population, rather than by the total amount of suffering. The total amount of suffering will grow naturally with the population, and suffering / population seems to give a better indication of the severity of the suffering (a small group suffering a large amount is weighted higher than a large group suffering a small amount, as I intuitively think is correct).

My primarily concern with this (simplistic) method of measuring the severity of suffering is that it ignores the distribution of suffering within a population (i.e there could be a sub population with a large amount of suffering). However, I don't think that's a compelling enough reason to discount working to minimize the fraction of suffering rather than absolute suffering.

Are there compelling arguments for why we should seek to minimize total suffering?

If I understand right, the view you're proposing is sort of like the 'average view' of utilitarianism. The objective is to minimize the average level of suffering across a population.

A common challenge to this view (shared with average util) is that it seems you can make a world better by adding lives which suffer, but suffer less than the average. In some hypothetical hellscape where everyone is getting tortured, adding further lives where people get tortured slightly less severely should make the world even worse, not better.

Pace the formidable challenges of infinitarian ethics, I generally lean towards total views. I think the intuition you point to (which I think is widely shared) in that larger degrees of suffering should 'matter more' is perhaps better accommodated in something like prioritarianism, whereby improving the well-being of the least well off is given extra moral weight to its utilitarian 'face value'. (FWIW, I generally lean towards pretty flat footed utilitarianism, as there some technical challenges with prioritarianism, and it seems hard to distinguish the empirical from the moral matters: there are evolutionary motivations (H/T Carl Shulman) why there should be extremely severe pain, so maybe a proper utilitarian accounting makes relieving these extremes worth very large amounts of more minor suffering).

Aside: in population ethics there's a well-worn problem of aggregation, as suggested by the repugnant conclusion: lots and lots of tiny numbers when put together can outweigh a big numbers, so total views have challenges such as: "Imagine A where 7 billion people live lives of perfect bliss, versus B where these people suffer horrendous torture, but TREE(4) people with lives that are only just barely worth living". B is far better than A, yet it seems repulsive. (The usual total view move is to appeal to scope insensitivity and that our intuitions here are ill-suited to tracking vast numbers. I don't think perhaps more natural replies (e.g. 'discount positive wellbeing if above zero but below some threshold close to it'), come out in the wash).

Unfortunately, the 'suffering only' suggested as a potential candidate in the FAQ (i.e. discount 'positive experiences', and only work to reduce suffering) seems to compound these, as in essence one can concatenate these problems of population ethics with the counter-intuitiveness of this discounting of positive experience (virtually everyone's expressed and implied preferences indicate positive experiences have free-standing value, as they are willing to trade off between negative and positive).

The aggregation challenge akin to the repugnant conclusion (which I think I owe to Carl Shulman) goes like this. Consider A: 7 billion people suffering horrendous torture. Now consider B: TREE(4) people enjoying lifelong eudaimonic bliss with the exception of each suffering a single pinprick. On a total suffering view A >>> B, yet this seems common-sensically crazy.

The view seems to violate two intuitions, the first the aggregation issue (i.e. TREE(4) pinpricks is more morally important than 7 billion cases of torture), but also the discounting of positive experience - the 'suffering only counts view' is indifferent to the difference of TREE(4) instances of lifelong eudaimonic bliss between the scenarios. If we imagine world C where no one exists a total util view gets the intuitively 'right' answer (i.e. B > C > A), whilst the suffering view gets most of the pairwise comparisons intuitively wrong (i.e. C > A > B)

How do you feel about the mere addition paradox? These questions are not simple.

Curated and popular this week
Relevant opportunities