Today, Forethought and I are releasing an essay series called Better Futures, here. It’s been something like eight years in the making, so I’m pretty happy it’s finally out! It asks: when looking to the future, should we focus on surviving, or on flourishing?
In practice at least, future-oriented altruists tend to focus on ensuring we survive (or are not permanently disempowered by some valueless AIs). But maybe we should focus on future flourishing, instead.
Why?
Well, even if we survive, we probably just get a future that’s a small fraction as good as it could have been. We could, instead, try to help guide society to be on track to a truly wonderful future.
That is, I think there’s more at stake when it comes to flourishing than when it comes to survival. So maybe that should be our main focus.
The whole essay series is out today. But I’ll post summaries of each essay over the course of the next couple of weeks. And the first episode of Forethought’s video podcast is on the topic, and out now, too.
The first essay is Introducing Better Futures: along with the supplement, it gives the basic case for focusing on trying to make the future wonderful, rather than just ensuring we get any ok future at all. It’s based on a simple two-factor model: that the value of the future is the product of our chance of “Surviving” and of the value of the future, if we do Survive, i.e. our “Flourishing”.
(“not-Surviving”, here, means anything that locks us into a near-0 value future in the near-term: extinction from a bio-catastrophe counts but if valueless superintelligence disempowers us without causing human extinction, that counts, too. I think this is how “existential catastrophe” is often used in practice.)
The key thought is: maybe we’re closer to the “ceiling” on Survival than we are to the “ceiling” of Flourishing.
Most people (though not everyone) thinks we’re much more likely than not to Survive this century. Metaculus puts *extinction* risk at about 4%; a survey of superforecasters put it at 1%. Toby Ord put total existential risk this century at 16%.
Chart from The Possible Worlds Tree.
In contrast, what’s the value of Flourishing? I.e. if near-term extinction is 0, what % of the value of a best feasible future should we expect to achieve? In the next two essays that follow, Fin Moorhouse and I argue that it’s low.
And if we are farther from the ceiling on Flourishing, then the size of the problem of non-Flourishing is much larger than the size of the problem of the risk of not-Surviving.
To illustrate: suppose our Survival chance this century is 80%, but the value of the future conditional on survival is only 10%.
If so, then the problem of non-Flourishing is 36x greater in scale than the problem of not-Surviving.
(If you have a very high “p(doom)” then this argument doesn’t go through, and the essay series will be less interesting to you.)
The importance of Flourishing can be hard to think clearly about, because the absolute value of the future could be so high while we achieve only a small fraction of what is possible. But it’s the fraction of value achieved that matters. Given how I define quantities of value, it’s just as important to move from a 50% to 60%-value future as it is to move from a 0% to 10%-value future.
We might even achieve a world that’s common-sensically utopian, while still missing out on almost all possible value.
In medieval myth, there’s a conception of utopia called “Cockaigne” - a land of plenty, where everyone stays young, and you could eat as much food and have as much sex as you like.
We in rich countries today live in societies that medieval peasants would probably regard as Cockaigne, now. But we’re very, very far from a perfect society. Similarly, what we might think of as utopia, today, could nonetheless barely scrape the surface of what is possible.
All things considered, I think there’s quite a lot more at stake when it comes to Flourishing than when it comes to Surviving.
I think that Flourishing is likely more neglected, too. The basic reason is that the latent desire to Survive (in this sense) is much stronger than the latent desire to Flourish. Most people really don’t want to die, or to be disempowered in their lifetimes. So, for existential risk to be high, there has to be some truly major failure of rationality going on.
For example, those in control of superintelligent AI (and their employees) would have to be deluded about the risk they are facing, or have unusual preferences such that they're willing to gamble with their lives in exchange for a bit more power. Alternatively, look at the United States’ aggregate willingness to pay to avoid a 0.1 percentage point chance of a catastrophe that killed everyone - it’s over $1 trillion. Warning shots could at least partially unleash that latent desire, unlocking enormous societal attention.
In contrast, how much latent desire is there to make sure that people in thousands of years’ time haven’t made some subtle but important moral mistake? Not much. Society could be clearly on track to make some major moral errors, and simply not care that it will do so.
Even among the effective altruist (and adjacent) community, most of the focus is on Surviving rather than Flourishing. AI safety and biorisk reduction have, thankfully, gotten a lot more attention and investment in the last few years; but as they do, their comparative neglectedness declines.
The tractability of better futures work is much less clear; if the argument falls down, it falls down here. But I think we should at least try to find out how tractable the best interventions in this area are. A decade ago, work on AI safety and biorisk mitigation looked incredibly intractable. But concerted effort *made* the areas tractable.
I think we’ll want to do the same on a host of other areas — including AI-enabled human coups; AI for better reasoning, decision-making and coordination; what character and personality we want advanced AI to have; what legal rights AIs should have; the governance of projects to build superintelligence; deep space governance, and more.
On a final note, here are a few warning labels for the series as a whole.
First, the essays tend to use moral realist language - e.g. talking about a “correct” ethics. But most of the arguments port over - you can just translate into whatever language you prefer, e.g. “what I would think about ethics given ideal reflection”.
Second, I’m only talking about one part of ethics - namely, what’s best for the long-term future, or what I sometimes call “cosmic ethics”. So, I don’t talk about some obvious reasons for wanting to prevent near-term catastrophes - like, not wanting yourself and all your loved ones to die. But I’m not saying that those aren’t important moral reasons.
Third, thinking about making the future better can sometimes seem creepily Utopian. I think that’s a real worry - some of the Utopian movements of the 20th century were extraordinarily harmful. And I think it should make us particularly wary of proposals for better futures that are based on some narrow conception of an ideal future. Given how much moral progress we should hope to make, we should assume we have almost no idea what the best feasible futures would look like.
I’m instead in favour of what I’ve been calling viatopia, which is a state of the world where society can guide itself towards near-best outcomes, whatever they may be. Plausibly, viatopia is a state of society where existential risk is very low, where many different moral points of view can flourish, where many possible futures are still open to us, and where major decisions are made via thoughtful, reflective processes.
From my point of view, the key priority in the world today is to get us closer to viatopia, not to some particular narrow end-state. I don’t discuss the concept of viatopia further in this series, but I hope to write more about it in the future.
It's worth noting that it's realistically possible for surviving to be bad, whereas promoting flourishing is much more robustly good.
Survival is only good if the future it enables is good. This may not be the case. Two plausible examples:
Survival could still be great of course. Maybe we'll solve wild animal suffering, or we'll have so many humans with good lives that this will outweigh it. Maybe we'll make flourishing digital minds. But I wanted to flag this asymmetry between promoting survival and promoting flourishing, as the latter is considerably more robust.
I think this is an important point, but my experience is that when you try to put it into practice things become substantially more complex. E.g. in the podcast Will talks about how it might be important to give digital beings rights to protect them from being harmed, but the downside of doing so is that humans would effectively become immediately disempowered because we would be so dramatically outnumbered by digital beings.
It generally seems hard to find interventions which are robustly likely to create flourishing (indeed, "cause humanity to not go extinct" often seems like one of the most robust interventions!).
A lot of people would argue a world full of happy digital beings is a flourishing future, even if they outnumber and disempower humans. This falls out of an anti-speciesist viewpoint.
Here is Peter Singer commenting on a similar scenario in a conversation with Tyler Cowen:
COWEN: Well, take the Bernard Williams question, which I think you’ve written about. Let’s say that aliens are coming to Earth, and they may do away with us, and we may have reason to believe they could be happier here on Earth than what we can do with Earth. I don’t think I know any utilitarians who would sign up to fight with the aliens, no matter what their moral theory would be.
SINGER: Okay, you’ve just met one.
This is pretty chilling to me, actually. Singer is openly supporting genocide here, at least in theory. (There are also shades of "well, it was ok to push out all those Native Americans because we could use the land to support a bigger population.)
I'm not an expert, but I think you've misused the term genocide here.
Putting aside that homo sapiens isn't one of the protected groups, the "as such" is commonly interpreted to mean that the victim must be targeted because of their membership of that group and not some incidental reason. In the Singer case, he wouldn't be targeting humans because they are humans, he'd be targeting them on account of wanting to promote total utility. In a scenario where the aliens aren't happier, he would fight the aliens.
I'm probably just missing your point here, and what you're actually getting at is that Singer's view is simply abhorrent. Maybe, but if you read the full exchange, what he's saying is that, in a war, he would not choose a side based on species but instead based on what would promote the intrinsic good. Importantly, I don't think he says he would invite/start the war, only how he would act in a scenario where a war is inevitable.
Even under that definition, I think the aliens sound to me like they intend to eliminate humans, albeit as a means to an end, not an end to itself. If the Armenian genocide happened to be more about securing a strong Turkish state than any sort of Nazi-style belief that the existence of Armenians was itself undesirable because they were someone inherently evil, it wouldn't mean it wasn't genocide. (Not sure what the actual truth is on that.) But yes, I am more bothered about it being abhorrent than about whether it meets the vague legal definition of the word "genocide" given by the UN. (Vague because, what is it to destroy "in part"? If a racist kills one person because of their race is that an attempt to destroy a race "in part" and so genocide?)
"Importantly, I don't think he says he would invite/start the war, only how he would act in a scenario where a war is inevitable." If someone signed up to fight a war of extermination against Native Americans in 1800 after the war already started, I'm not sure "the war was already inevitable" would be much of a defence.
We're just getting into the standard utilitarian vs deontology argument. Singer may just double down and say—just because you feel it's abhorrent, doesn't mean it is.
There are examples of things that seem abhorrent from a deontological perspective, but good from a utilitarian perspective, and that people are generally in favor of. The bombings of Hiroshima and Nagasaki are perhaps the clearest case.
Personally, I think utilitarianism is the best moral theory we have, but I have some moral uncertainty and so factor in deontological reasoning into how I act. In other words, if something seems like an atrocity, I would have to be very confident that we'd get a lot of payoff to be in favor of it. In the alien example, I think it is baked in that we are pretty much certain it would be better for the aliens to take over—but in practice this confidence would be almost impossible to come by.
I agree that this is in some sense part of a more general utilitarianism vs intuitions thing.
Are people generally in favour of the bombings? Or do you really mean *Americans*? What do people in liberal democracies like say Spain that didn't participate in WW2 think? People in Nigeria? India? Personally, I doubt you could construct a utilitarian defense of first dropping the bombs on cities rather than starting with a warning shot demonstration at the very least. It is true, I think that war is a case where people in Western liberal democracies tend to believe that some harm to innocent civilians can be justified by the greater good. But it's also I think true that people in all cultures have a tendency to believe implausible justifications for prima facie very bad actions taken by their countries during wars.
I don't know about globally, but there are a lot of Chinese people, and they generally support the bombings, which has to take us a fair bit of the the way towards general support. (I'm not aware of any research into the views of Indians or Nigerians). And the classic utilitarian defense is that there were a limited number of bombs of unknown reliability, so they couldn't be wasted - though to be honest, asking for warning shots seems a bit like special pleading. Warning shots are for deterring aggression in the first place - not for after the attacker has already struck, and shows no sign of stopping.
The overwhelming majority of Manhattan Project scientists, as well as the Undersecretary of the Navy, believed there should be a warning shot. It makes total sense from a game theory perspective to do warning shots when you believe your military advantage has significantly increased in a way that significantly change their own calculus.
My point wasn't necessarily that I believe that most people worldwide think the bombing was wrong, but rather that it's unlikely JackM has access to what "most people" think worldwide, and that it is plausible for obvious reasons that insofar as he does have a sense of what most Americans think about this, it's at least very plausible for standard reasons of nationalism and in-group bias that Americans have a more favourable view of the bombings than the world as whole. But "plausible" just means that, not definitely true.
As for the fact that they had few bombs: that is true, and I did briefly think it might enable the utilitarian defence you are giving, but if you think things through carefully, I don't think it really works all that well. The reason that the bombings pushed Japan towards surrender* is not, primarily, that it was much harder for Japan to fight on once Hiroshima and Nagasaki were gone, but rather the fear that US could drop more bombs. In other words, the Japanese weren't prepared to risk the US having more bombs ready, or being able to manufacture them quickly. That fear could certainly also have been generated simply by proof that the US had the bomb. I guess you could try and argue a warning shot would have had less psychological impact, but that seems speculative to me.
*There is, I believe, some level of historical debate about how much longer they would have held out anyway, so I am not sure whether the bombings alone were decisive.
That may be fair. Although, if what you're saying is that the bombings weren't actually justified when one uses utilitarian reasoning, then the horror of the bombings can't really be an argument against utilitarianism (although I suppose it could be an argument against being an impulsive utilitarian without giving due consideration to all your options).
I did not use the bombings as an argument against utilitarianism.
Yeah, I didn't meant to imply you had. This whole Hiroshima convo got us quite off topic. The original point was that Ben was concerned about digital beings outnumbering humans. I think that concern originates from some misplaced feeling that humans have some special status on account of being human.
I agree with the core point, and that was part of my motivation for working on this area. There is a counterargument, as Ben says, which is that any particular intervention to promote Flourishing might be very non-robust.
And there is an additional argument, which is that in worlds in which you have successfully reduced x-risk, the future is more likely to be negative-EV (because worlds in which you have successfully reduced x-risk are more likely to be worlds in which x-risk is high, and those worlds are more likely to be going badly in general (e.g. great power war)).
I don't think that wild animal suffering is a big consideration here, though, because I expect wild animals to be a vanishingly small fraction of the future population. Digital beings can inhabit a much wider range of environments than animals can, so even just in our own solar system in the future I'd expect there to be over a billion times as many digital beings as wild animals (the sun produces 2 billion times as much energy as lands on Earth); that ratio gets larger when looking to other star systems.
A counter argument to the wild animal point is that some risks may kill humanity but not all wild animals. I wonder if that’s the case for most catastrophic risks.