For Existential Choices Debate Week, we’re trying out a new type of event: the Existential Choices Symposium. It'll be a written discussion between invited guests and any Forum user who'd like to join in.
How it works:
- Any forum user can write a top-level comment that asks a question or introduces a consideration, the answer of which might affect people’s answer to the debate statement[1]. For example: “Are there any interventions aimed at increasing the value of the future that are as widely morally supported as extinction-risk reduction?” You can start writing these comments now.
- The symposium’s signed-up participants, Will MacAskill, Tyler John, Michael St Jules, Andreas Mogensen and Greg Colbourn, will respond to questions, and discuss them with each other and other forum users, in the comments.
- To be 100% clear - you, the reader, are very welcome to join in any conversation on this post. You don't have to be a listed participant to take part.
This is an experiment. We’ll see how it goes and maybe run something similar next time. Feedback is welcome (message me with feedback here).
The symposium participants will be online between 3 - 5 pm GMT on Monday the 17th.
Brief bios for participants (mistakes mine):
- Will MacAskill is an Associate Professor of moral philosophy at the University of Oxford, and Senior Research Fellow at Forethought. He wrote the books Doing Good Better, Moral Uncertainty, and What We Owe The Future. He is the cofounder of Giving What We Can, 80,000 Hours, Centre for Effective Altruism and the Global Priorities Institute.
- Tyler John is an AI researcher, grantmaker, and philanthropic advisor. He is an incoming Visiting Scholar at the Cambridge Leverhulme Centre for the Future of Intelligence and an advisor to multiple philanthropists. He was previously the Programme Officer for emerging technology governance and Head of Research at Longview Philanthropy. Tyler holds a PhD in philosophy from Rutgers University—New Brunswick, where his dissertation focused on longtermist political philosophy and mechanism design, and the case for moral trajectory change.
- Michael St Jules is an independent researcher, who has written on “philosophy of mind, moral weights, person-affecting views, preference-based views and subjectivism, moral uncertainty, decision theory, deep uncertainty/cluelessness and backfire risks, s-risks, and indirect effects on wild animals”.
- Andreas Mogensen is a Senior Research Fellow in Philosophy at the Global Priorities Institute, part of the University of Oxford’s Faculty of Philosophy. His current research interests are primarily in normative and applied ethics. His previous publications have addressed topics in meta-ethics and moral epistemology, especially those associated with evolutionary debunking arguments.
- Greg Colbourn is the founder of CEEALAR and is currently a donor and advocate for Pause AI, which promotes a global AI moratorium. He has also supported various other projects in the space over the last 2 years.
Thanks for reading! If you'd like to contribute to this discussion, write some questions below which could be discussed in the symposium.
NB- To help conversations happen smoothly, I'd recommend sticking to one idea per top-level comment (even if that means posting multiple comments at once).
The main question of the debate week is: “On the margin, it is better to work on reducing the chance of our extinction than increasing the value of the future where we survive”.
Where “our” is defined in a footnote as “earth-originating intelligent life (i.e. we aren’t just talking about humans because most of the value in expected futures is probably in worlds where digital minds matter morally and are flourishing)”.
I'm interested to hear from the participants how likely they think extinction of “earth-originating intelligent life” really is this century. Note this is not the same as asking what your p(doom) is, or what likelihood you assign to existential catastrophe this century.
My own take is that literal extinction of intelligent life, as defined, is (much) less than 1% likely to happen this century, and this upper-bounds the overall scale of the “literal extinction” problem (in ITN terms). I think this partly because the definition counts AI survival as non-extinction, and I truly struggle to think of AI-induced catastrophes leaving only charred ruins, without even AI survivors. Other potential causes of extinction, like asteroid impacts, seem unlikely on their own terms. As s... (read more)
A broader coalition of actors will be motivated to pursue extinction prevention than longtermist trajectory changes.[1] This means:
Is this a reasonable framing - if so which effect dominates or how can we reason through this?
For instance, see Scott Alexander on the benefits of extinction risk as a popular meme compared to longtermism.
I argeud for something similar here.
I agree with the framing.
Quantitatively, the willingness to pay to avoid extinction even just from the United States is truly enormous. The value of a statistical life in the US — used by the US government to estimate how much US citizens are willing to pay to reduce their risk of death — is around $10 million. The willingness to pay, therefore, from the US as a whole, to avoid a 0.1 percentage point of a catastrophe that would kill everyone in the US, is over $1 trillion. I don’t expect these amounts to be spent on global catastrophic risk reduction, but they show how much latent desire there is to reduce global catastrophic risk, which I’d expect to become progressively mobilised with increasing indications that various global catastrophic risks, such as biorisks, are real. [I think my predictions around this are pretty different than some others, who expect the world to be almost totally blindsided. Timelines and gradualness of AI takeoff is of course relevant here.]
In contrast, many areas of better futures work are likely to remain extraordinarily neglected. The amount of even latent interest in, for example, ensuring that resources outside of our solar system are put to ... (read more)
If the true/best/my subjective axiology is linear in resources (e.g. total utilitarianism), lots of 'good' futures will probably capture a very small fraction of how good the optimal future could have been. Conversely, if axiology is not linear in resources (e.g. intuitive morality, average utilitarianism), good futures seem more likely to be nearly optimal. Therefore whether axiology is linear in resources is one of the cruxes for the debate week question.
Discuss.
Starting my own discussion thread.
My biggest doubt for the value of extinction risk reduction is my (asymmetric) person-affecting intuitions: I don't think it makes things better to ensure future people (or other moral patients) come to exist for their own sake or the sake of the value within their own lives. But if future people will exist, I want to make sure things go well for them. This is summarized by the slogan "Make people happy, not make happy people".
If this holds, then extinction risk reduction saves the lives of people who would otherwise die in an extinction event, which is presumably good for them, but this is only billions of humans.[1] If we don't go extinct, then the number of our descendant moral patients could be astronomical. It therefore seems better to prioritize our descendant moral patients conditional on our survival because there are far far more of them.
Aliens (including alien artificial intelligence) complicate the picture. We (our descendants, whether human, AI or otherwise) could
&nbs... (read more)
That seems true, but I'm not convinced it's the best way to reduce s-risks on the margin. See, for example, Vinding, 2024.
I'd also want to see a fuller analysis of ways it could backfire. For example, a pause might make multipolar scenarios more likely by giving more groups time to build AGI, which could increase the risks of conflict-based s-risks.
What actually changes about what you’d work on if you concluded that improving the future is more important on the current margin than trying to reduce the chance of (total) extinction (or vice versa)?
Curious for takes from anyone!
Discussion topic: People vary a lot in the extent to which, and how likely it is, that post-AGI, different people will converge on the same moral views. I feel fairly sceptical about having a high likelihood of convergence; I certainly don't think we should bank on it.
[See my response to Andreas below. Here I meant "convergence" as shorthand to refer to "fully accurate, motivational convergence".]
(Crossposted from a quicktake I just did).
Clarifying "Extinction"
I expect this debate week to get tripped up a lot by the term “extinction”. So here I’m going to distinguish:
Human extinction doesn’t entail total extinction. Human extinction is compatible with: (i) AI taking over and creating a civilisation for as lo... (read more)
Do you think octopuses are conscious? I do — they seem smarter than chickens, for instance. But their most recent common ancestor with vertebrates was some kind of simple Precambrian worm with a very basic nervous systems.
Either that most recent ancestor was not phenomenally conscious in the sense we have in mind, in which case consciousness arose more than once in the tree of life. Or else it was conscious, in which case consciousness would seem easy to reproduce (wire together some ~1,000 nerves).
Position statement: I chose 36% disagreement. AMA!
My view is that Earth-originating civilisation, if we become spacefaring, will attain around 0.0001% of all value. This still makes extinction risk astronomically valuable (it's equivalent to optimising a millionth of the whole cosmos!), but if we could increase the chance of optimising 1% of the universe by 1%, this would be 100x more valuable than avoiding extinction. (You're not going to get an extremely well grounded explanation of these numbers from me, but I hope they make my position clearer.)
My view... (read more)
Part of what it means that I try to support thinking on this issue, e.g. by seed-funding NYU MEP and doing this discussion, and doing my own thinking on it.
At this stage the thing I'm most excited about supporting is market-based mechanisms for democratic AI alignment like this. Also excited about trying to get more resources to work on AI welfare, utilitarianism, and to groups like Forethought: A new AI macrostrategy group.
In practice I spend more resources on extinction risk reduction. Part of this is just because I'd really prefer not to die in my 30s. When an EA cares for their family taking away time from extinction risk they're valuing their family as much as 10^N people. I see myself as doing something similar here.
Thanks for saying this. I feel likewise (but s/30s/40s :))
Question: what level of extinction risk are people personally willing to accept in order to realise higher expected value in the futures where we survive? How much would the extinction coming in the next 5 years effect this? Or the next 1 year? How is this reflected in terms of what you are working on / spending resources on?
Do you agree that the experience of digital minds likely dominates far future calculations?
This leads me to want to prioritize making sure that if we do create digital minds, we do so well. This could entail raising the moral status of digital minds, improving our ability to understand sentience and consciousness, and making sure AI goes well and can help us with these things.
Extinction risk becomes lower importance to me. If we go extinct we get 0 value from digital minds which seems bad, but it also means we avoid the futures where we create them and the... (read more)
My position is that Timelines are short, p(doom) is high: a global stop to frontier AI development until x-safety consensus is our only reasonable hope (this post needs updating, to factor in things like inference time compute scaling, but my conclusions remain the same).
The problem is that no one has even established whether aligning or controlling ASI is theoretically, let alone practically, possible. Everything else (whether there is a human future at all past the next few years) is downstream of that.
Will MacAskill stated in a recent 80,000 hours podcast that he believes marginal work on trajectory change toward a best possible future rather than a mediocre future seems likely significantly more valuable than marginal work on extinction risk.
Could you explain what the key crucial considerations are for this claim to be true, and a basic argument for why think each of the crucial considerations resolves in favor of this claim?
Would also love to hear if others have any other crucial considerations they think weigh in one direction or the other.
Thank you for organizing this debate!
Here are several questions. They are related to two hypotheses, that could, if both significantly true, make impartial longtermists update the value of Extinction-Risk reduction downward (potentially by 75% to 90%).
For context... (read more)
I have a question, and then a consideration that motivates it, which is also framed as a question that you can answer if you like.
If an existential catastrophe occurs, how likely is it to wipe out all animal sentience on earth?
I've already asked that question here (and also, to some acquaintances working in AI Safety, but the answers have very much differed - it seems we're quite far from a consensus on this, so it would be interesting to see perspectives from the varied voices taking part in this symposium.
Less important question, but that may clari... (read more)
How much of the argument for working towards positive futures rather than existential security rests on conditional value, as opposed to expected value?
One could argue for conditional value, that in worlds where strong AI is easy and AI safety is hard, we are doomed regardless of effort, so we should concentrate on worlds where we could plausibly have good outcomes.
Alternatively, one could be confident that the probability of safety is relatively high, and make the argument that we should spend more time focused on positive futures because it's likely alre... (read more)
Yeah, I think a lot of the overall debate -- including what is most ethical to focus on(!) -- depends on AI trajectories and control.
For a given individual, can they have a higher probability of averting extinction (i.e. making the difference) or for a long-term trajectory change? If you discount small enough probabilities of making a difference or are otherwise difference-making risk averse (as an individual), would one come out ahead as a result?
Some thoughts: extinction is a binary event. But there's a continuum of possible values that future agents could have, including under value lock-in. A small tweak in locked-in values seems more achievable counterfactually than being the diffe... (read more)
This is a cool idea! Will this be recorded for people who can't attend live?
Edit: nevermind, I think I'm confused; I take it this is all happening in writing/in the comments.