Existential Choices Symposium with Will MacAskill and other special guests

I agree with the framing.

Quantitatively, the willingness to pay to avoid extinction even just from the United States is truly enormous. The value of a statistical life in the US — used by the US government to estimate how much US citizens are willing to pay to reduce their risk of death — is around $10 million. The willingness to pay, therefore, from the US as a whole, to avoid a 0.1 percentage point of a catastrophe that would kill everyone in the US, is over $1 trillion. I don’t expect these amounts to be spent on global catastrophic risk reduction, but they show how much latent desire there is to reduce global catastrophic risk, which I’d expect to become progressively mobilised with increasing indications that various global catastrophic risks, such as biorisks, are real. [I think my predictions around this are pretty different than some others, who expect the world to be almost totally blindsided. Timelines and gradualness of AI takeoff is of course relevant here.] 

In contrast, many areas of better futures work are likely to remain extraordinarily neglected. The amount of even latent interest in, for example, ensuring that resources outside of our solar system are put to their best use, or that misaligned AI produces a somewhat-better future than it would otherwise have done even if it kills us all, is tiny, and I don’t expect society to mobilise massive resources towards these issues even if there were indications that those issues were pressing.

In some cases, what people want will be actively opposed to what is in fact best, if what's best involves self-sacrifice on the part of those alive today, or with power today. 

And then I think the neglectedness consideration beats the tractability consideration. Here are some pretty general reasons for optimism on expected tractability:

• In general, tractability doesn’t vary by as much as importance and neglectedness.
• In cause areas where very little work has been done, it’s hard for expected tractability to be very low. Because of how little we know about tractability in unexplored cause areas, we should often put significant credence on the idea that the cause will turn out to be fairly tractable; this is enough to warrant some investment into the cause area — at least enough to find out how tractable the area is.
• There are many distinct sub-areas within better futures work. It seems unlikely to me that tractability in all of them is very low, and unlikely that their tractability is very highly correlated.
• There’s a reasonable track record of early-stage areas with seemingly low tractability turning out to be surprisingly tractable. A decade ago, work on risks from AI takeover and engineered pathogens seemed very intractable; there was very little that one could fund, and very little in the way of promising career paths. But this changed over time, in significant part because of (i) research work improving our strategic understanding, and shedding light on what interventions were most promising; (ii) scientific developments (e.g. progress in machine learning) making it clearer what interventions might be promising; (ii) the creation of organisations that could absorb funding and talent. All these same factors could well be true for better futures work, too. 

Of these considerations, it’s the last that personally moves me the most. It doesn't feel long ago that work on AI takeover risk felt extraordinarily speculative and low-tractability, where there was almost nowhere one could work for or donate to outside of the Future of Humanity Institute or Machine Intelligence Research Institute. In the early days, I was personally very sceptical about the tractability of the area. But I’ve been proved wrong. Via years of foundational work — both research work figuring out what the most promising paths forward are, and via founding new organisations that are actually squarely focused on the goal of reducing takeover risk or biorisk, rather than on a similar but tangential goal — the area has become tractable, and now there are dozens of great organisations that one can work for or donate to. 

A broader coalition of actors will be motivated to pursue extinction prevention than longtermist trajectory changes... For instance, see Scott Alexander on the benefits of extinction risk as a popular meme compared to longtermism.

This might vary between: 

• The level of the abstract memes:
  • I agree "reducing risk of extinction (potentially in the near term)" may be more appealing than "longtermist trajectory change"
• The level of concrete interventions:
  • "Promoting democracy" (or whatever one decides promotes long term value) might be more appealing than "reducing risk from AI"^[1] (though there is likely significant variation within concrete interventions).

1. ^{^}

   Though our initial work does not suggest this.

I think what we should be talking about is whether we hit the "point of no return" this century for extinction of Earth-originating intelligent life. Where that could mean: Homo sapiens and a most other mammals get killed off in an extinction event this century; then technologically-capable intelligence never evolves again on Earth; so all life dies off within a billion years or so. (In a draft post that you saw of mine, this is what I had in mind.)

The probability of this might be reasonably high. There I'm at idk 1%-5%.

I think the "earth-originating intelligent life" term should probably include something that indicates sentience/ moral value. Perhaps you could read that into "intelligent" but that feels like a stretch. But I didn't want to imply that a world with no humans but many non-conscious AI systems would count as anything but an extinction scenario - that's one of the key extinction scenarios. 

Yeah, do you have other proposed reconceptualisations of the debate?

One shower thought I've had is that maybe we should think of the debate as about whether to focus on ensuring that humans have final control over AI systems or ensuring that humans do good things with that control. But this is far from perfect.

If we assume that natural risks are negligible, I would guess that this probably reduces to something like the question of what probability you put on extinction or existential catastrophe due to anthropogenic biorisk? Since biorisk is likely to leave much of the rest of Earthly life unscathed, it also hinges on what probability you assign to something like "human level intelligence" evolving anew. I find it reasonably plausible that a cumulative technological culture of the kind that characterizes human beings is unlikely to be a convergent evolutionary outcome (for the reasons given in Powell, Contingency and Convergence), and thus if human beings are wiped out, there is very little probability of similar traits emerging in other lineages. So human extinction due to a bioengineered pandemic strikes me as maybe the key scenario for the extinction of earth-originating intelligent life. Does that seem plausible?  

I think it hinges on whether our AI successors would be counted as "life", or whether they "matter morally". I think the answer is likely no to both^[1]. Therefore the risk of extinction boils down to risk of misaligned ASI wiping out the biosphere. Which I think is ~90% likely this century on the default trajectory, absent a well enforced global moratorium on ASI development.

1. ^{^}

   Or at least "no" to the latter; if we consider viruses to be life that don't matter morally, or are in fact morally negative, we can consider (default rogue) ASI to be similar.

The easiest way, in my view, to make a near-optimal future very likely, conditional on non-extinction, is if value is bounded above. 

There's an argument that this is the common sense view. E.g. consider:

Common-sense Eutopia: In the future, there is a very large population with very high well-being; those people are able to do almost anything they want as long as they don’t harm others. They have complete scientific and technological understanding. War and conflict are things of the past. Environmental destruction has been wholly reversed; Earth is now a natural paradise. However, society is limited only to the solar system, and will come to an end once the Sun has exited its red giant phase, in about five billion years.

Does this seem to capture less than one 10^22th of all possible value? (Because there are ~10^22 affectable stars, so civilisation could be over 10^22 times as big). On my common-sense moral intuitions, no.

Making this argument stronger: Normally, quantities of value are defined are in terms of the value of risky gambles. So what it means to say that Common-sense Eutopia is less than one 10^22th of all possible value is that a gamble with a one in 10^22 chance of producing an ideal-society-across-all-the-stars, and a (1 - 1/10^22) chance of near-term extinction, is better than producing Common-sense Eutopia for certain.

But that seems wild. Of all the issues facing classical utilitarianism, this seems the most problematic to me. 

 

There are funky axiologies where value is superlinear in resources — basically any moral worldview that embraces holism. If you think that the whole, arranged a particular way, is more valuable than the parts, then you will be even more precious about how precisely the world should be arranged than the total utilitarian.

Since others have discussed the implications, I want to push a bit on the assumptions.

I worry that non-linear axiologies^[1] end up endorsing egoism, helping only those whose moral patienthood you are most confident in or otherwise prioritizing them far too much over those of less certain moral patienthood. See Oesterheld, 2017 and Tarsney, 2023.

(I also think average utilitarianism in particular is pretty bad, because it would imply that if the average welfare is negative (even torturous), adding bad lives can be good, as long as they're even slightly less bad than average.)

Maybe you can get around this with non-aggregative or partially aggregative views. EDIT: Or, if you're worried about fanaticism, difference-making views.

1. ^{^}

   Assuming completeness, transitivity and the independence of irrelevant alternatives and each marginal moral patient matters less.

Unpacking this: on linear-in-resources (LIR) views, we could lose out on most value if we (i) capture only a small fraction of resources that we could have done, and/or (ii) use resources in a less efficient way than we could have done. (Where on a LIR view, there is some use of resources that has the highest value/unit of resources, and everything should be used in that way.)

Plausibly at least, only a tiny % of possible ways of using resources are close to the value produced by the highest value/unit of resources use. So, the thinking goes, merely getting non-extinction isn't yet getting you close to a near-best future - instead you really need to get from a non-extinction future to that optimally-used-resources future, and if you don't then you lose out on almost all value. 

 

Average utilitarianism is approx linear in resources as long as at least one possible individual's wellbeing is linear in resources. 

I.e. we create Mr Utility Monster, who has wellbeing that is linear in resources, and give all resources to benefiting Mr Monster. Total value is the same as it would be under total utilitarianism, just divided by a constant (namely, the number of people who've ever lived).

Surely any of our actions changes who exists in the future? So we aren't in fact benefiting them?

(Whereas we can benefit specific aliens, e.g. by leaving our resources for them - our actions today don't affect the identities of those aliens.) 
 

I tend to think that the arguments against any theory of the good that encodes the intuition of neutrality are extremely strong. Here's one that I think I owe to Teru Thomas (who may have got it from Tomi Francis?).

Imagine the following outcomes, A - D, where the columns are possible people, the numbers represent the welfare of each person when they exist, and # indicates non-existence.

A 5 -2 #  #
B 5  #  2   2
C 5  2  2  #
D 5  -2  6 #

I claim that if you think it's neutral to make happy people, there's a strong case that you should think that B isn't better than A. In other words, it's not better to prevent someone from coming to exist and enduring a life that's not worth living if you simultaneously create two people with lives worth living. And that's absurd. I also think it's really hard to believe if you believe the other side of the asymmetry: that it's bad to create people whose lives are overwhelmed by suffering. 

Why is there pressure on you to accept that B isn't better than A? Well, first off, it seems plausible that B and C are equally good, since they have the same number of people at the same welfare levels. So let's assume this is so.

Now, if you accept utilitarianism for a fixed population, you should think that D is better than C, since all the same people exist in these outcomes, and there's more total/average welfare. (I'm pretty sure you can support this kind of verdict on weaker assumptions if necessary.)

So let's suppose, on this basis, that D is better than C. B and C are equally good. I assume it follows that D is better than B. 

Suppose that B were better than A. Since D is better than B, it would follow that D is better than A as well. But we know this can't be so, if it's neutral to make happy people, because D and A differ only in the existence of an extra person who has a life worth living. The neutrality principle entails that D isn't better than B. But it's absurd to think that B isn't better than A. 

Arguments like this make me feel pretty confident that the intuition of neutrality is mistaken. 

How asymmetric do you think things are? I tend to deprioritise s-risks (both accidental and intentional) because it seems like accidental suffering and intentional suffering will be a very small portion of the things that our descendants choose to do with energy. In everyday cases I don't feel a pull to putting a lot of weight on suffering. But I feel more confused when we get to tail cases. Maximising pleasure intuitively feels meh to me, but maximising suffering sounds pretty awful. So I worry that (1) all of the value is in the tails, as per Power Laws of Value and (2) on my intuitive moral tastes the good tails are not that great and the bad tails are really bad.

Worth pointing out that extinction by almost any avenue we're discussing seriously would kill a lot of people who already exist.

It therefore seems better to prioritize our descendant moral patients conditional on our survival because there are far far more of them.

I think in practical terms this isn't mutually exclusive with ensuring our survival. The immediate way to secure our survival, at least for the next decade or so, is a global moratorium on ASI. This also reduces s-risks from ASI, and keeps our options open for reducing human-caused s-risk (i.e. we can still avoid factory farming in space colonization).

Will is thinking about this much more actively and will give the best answer, but here are some key crucial considerations:

• How tractable is extinction risk reduction and trajectory change work?
  • As a part of that, are there ways that we can have a predictable and persistent effect on the value of the long-term future other than by reducing extinction risk?
• How good is the future by default?
• How good are the best attainable futures?

These are basically Tractability and Importance from the INT framework.

Some of the biggest disagreements in the field are over how likely we are to achieve eutopia by default (or what % of eutopia we will achieve) and what, if anything, can be done to predictably shape the far future. Populating and refining a list of answers to this last question has been a lot of the key work of the field over the past few years.

I think another crucial consideration is how likely, and near, extinction is. If it is near, with high likelihood (and I think it is down to misaligned ASI being on the horizon), then it's unlikely there will be time for trajectory change work to bear fruit.

I think Will MacAskill and Finn Morehouse's paper rests on the crucial consideration that aligning ASI is possible (by anyone at all). They haven't established this (EDIT: by this I mean they don't cite to any supporting arguments for this, rather than personally coming up with the arguments themselves. But as far as I know, there aren't any supporting arguments for the assumption, and in fact there are good arguments on the other side for why aligning ASI is fundamentally impossible).

So in the debate week statement (footnote 2) it says "earth-originating intelligent life". What if you disagree that AI counts as "life"? I expect that a singleton ASI will take over and will not be sentient or conscious, or value anything that humans value (i.e. the classic Yudkowskian scenario).

Civ-Saturation seems plausible, though only if there are other agents in the affectable universe. I don't have a good view on this, and yours is probably better.

Civ-Similarity seems implausible. I at least have some control over what humans do in the future, so I can steer things towards the futures I judge best. I don't have any control over what aliens do. And there are large differences between the best and middling futures as I argue in Power Laws of Value .

Thanks! I haven't read your stuff yet, but it seems like good work; and this has been a reason in my mind for being more in favour of trajectory change than totla extinction reduction for a while. It would only reduce the value of extinction risk reduction by an OOM at most, though?

I'm sympathetic to something in Mediocrity direction (for AI-built civilisations as well as human-built civilisations), but I think it's very hard to have a full-blooded Mediocrity principle if you also think that you can take actions today to meaningfully increase or decrease the value of Earth-originating civilisation. Suppose that Earth-originating civilisation's value is V, and if we all worked on it we could increase that to V+ or to V-. If so, then which is the right value for the alien civilisation? Choosing V rather than V+ or V- (or V+++ or V--- etc) seems pretty arbitrary. 

Rather, we should think about how good our prospects are compared to a random draw civilisation. You might think we're doing better or worse, but if it's possible for us to move the value of the future around, then it seems we should be able to reasonably think that we're quite a bit better (or worse) than the random draw civ. 

It felt surprisingly hard to come up with important examples of this, I think because there is some (suspicious?) convergence between both extinction prevention and trajectory changing via improving the caution and wisdom with which we transition to ASI. This both makes extinction less likely (through more focus on alignment and control work, and perhaps slowing capabilities progress or differential accelerating safety-oriented AI applications) and improves the value of surviving futures (by making human takeovers, suffering digital minds etc less likely).

But maybe this is just focusing on the wrong resolution. Breaking down 'making the ASI transition wiser', if we are mainly focused on extinction, AI control looks especially promising but less so otherwise. Digital sentience and rights work looks better if trajectory changes dominate, though not entirely. Improving company and government (especially USG) understanding of relevant issues seems good for both.

Obviously, asteroids, supervolcanoes, etc work looks worse if preventing extinction is less important.

Biorisk I'm less sure about - non-AI mediated extinction from bio seems very unlikely, but what would a GCR pandemic do to future values? Probably ~neutral in expectation, but plausibly it could lead to the demise of liberal democratic institutions (bad), or to a post-recovery world that is more scared and committed to global cooperation to prevent that recurring (good).

Interesting argument - I don't know much about this argument, but my thoughts are that there's not much value in thinking in terms of conditional value. If AI Safety is doomed to fail, there's not much value focusing on good outcomes which won't happen, when there are great global health interventions today. Arguably, these global health interventions could also help at least some parts of humanity have a positive future.

I'll just share that for me personally the case rests on expected value. I actually think there is a lot that we can do to make AI existential safety go better (governance if nothing else), and this is what I spend most of my time on. But the expected value of better futures seems far higher given the difference in size between the default post-human future and the best possible future.

I think it rests a lot on conditional value, and that is very unsatisfactory from a simple moral perspective of wanting to personally survive and have my friends and family survive. If extinction risk is high, and near (and I think it is!) we should be going all out to prevent it (i.e. pushing for a global moratorium on ASI). We can then work out the other issues once we have more time to think about them (rather than hastily punting on a long shot of surviving just because it appears higher EV now).

I'm also pretty skeptical of convergence, largely because I'm a moral anti-realist. I don't see why we would converge to any view in particular, except by coincidence or mechanisms that don't track stance-independent moral truths (because there are none). It's just people weighing their own particular moral intuitions. Humans differ in our most basic moral intuitions and leanings.

Barring value lock-in, I suspect there would be convergence towards the recognition that unnecessary suffering is bad and worth preventing (when cheap enough) because this seems pretty widely held and something societies move towards, but I'd guess there will still be disagreement on some of these:

1. population ethics
2. hedonism vs preference views vs others
3. whether non-sentient/non-conscious things matter terminally, e.g. preserving nature
4. whether groups of moral patients have special status beyond their aggregates, e.g. ethnic groups, species
5. deontology vs consequentialism vs virtue ethics
6. what counts as conscious/sentient (I think this is partly normative, not just empirical)
7. decision theory, attitudes towards risk and ambiguity, fanaticism

Could you clarify what you mean by 'converge'? One thing that seems somewhat tricky to square is believing that convergence is unlikely, but that value lock-in is likely. Should we understand convergence as involving agreement in views facilitated by broadly rational processes, or something along those lines, to be contrasted with general agreement in values that might be facilitated by irrational or arational forces, of the kind that might ensure uniformity of views following a lock-in scenario? 

A bull case for convergence:

• Factory farming, and to a lesser extent global poverty, persist because there are some costs to ending them, and the rich aren't altruistic enough (or the altruists aren't rich enough) to end them. Importantly, it will not just be that factory farming itself ends, but due to cognitive dissonance, people's moral views towards nonhumans will likely change a lot too once ~no-one is eating animals. So there will predictably be convergence on viewing c2025 treatment of animals as terrible.
• There is an ongoing homogenization of global culture which will probably continue. As the educational and cultural inputs to people converge, it seems likely their beliefs (including moral beliefs) will also converge at least somewhat.
• Some fraction of current disagreements about economic/political/moral questions are caused just by people not being sufficiently informed/rational. So those disagreements would go away when we have ~ideal post-human reasoners.
  • A more ambitious version of the above is that perhaps post-humans will take epistemic humility very seriously, and they will know that all their peers are also very rational, so they will treat their own moral intuitions as little evidence of what the true/best/idealised-upon-reflection moral beliefs are. Then, everyone just defers very heavily to the annual survey of all of (post)humanity's views on e.g. population axiology rather than backing their own intuition.
    • (Arguably this doesn't count as convergence if people's intuitions still differ, but I think if people's all-things-considered beliefs, and therefore their actions, converge that is enough.)

But I agree we shouldn't bank on convergence!

I'm wondering whether we should expect worlds which converge on moral views to converge on bad moral views. 

From the space of world religions - we've seen a trend where we converge over time (at least from a high level of abstraction where we can refer to "Christianity" and "Islam" rather than "mega-church Christians" or whatever). Is this because the religions that succeed are exclusive and expansionary? Of all religions that have existed, I know that many of them don't much care if you also worship other gods. My empirical (ish) question is whether we should expect world in which a sizable fraction of the population follows the same religion to be one where the religion they follow is exclusive (you can't follow others) and expansionary (other people should also follow this religion). PS- I know that not all Christians or Muslims are exclusionary about other religions, this is over-simplified. 

This is relevant because, if this is a mechanism, we might expect the same thing of morality or political organisation - beliefs which demand you don't follow others, and that others follow the same beliefs as you, rather than tolerant beliefs. Perhaps this would make it more likely that futures which converge have converged on something extreme and closed, rather than exploratory and open. 

This is pretty vague - just wondering if others a) know more than me about the religion question and can speak to that or b) have had similar thoughts, or c) think that the existence of exclusive and expansionary (and wrong) ideologies might make convergence more likely. 

Hopefully they do not just converge on the same moral views, but also good ones!

A negotiated paretotopian future could create lots of moral value regardless of values not converging on their own.

Yes, this is yet another reason for a moratorium on further-AGI development imo. If everyone has a genie with unlimited wishes, and are all pushing the world in different directions, the result will be chaos. Yampolskiy's solution to this is everyone having their own private solipsistic universe simulations...