Existential Choices Debate Week

Over the years I've written some posts that are relevant to this week's debate topic. I collected and summarized some of them below:

"Disappointing Futures" Might Be As Important As Existential Risks

The best possible future is much better than a "normal" future. Even if we avert extinction, we might still miss out on >99% of the potential of the future.

Is Preventing Human Extinction Good?

A list of reasons why a human-controlled future might be net positive or negative. Overall I expect it to be net positive.

On Values Spreading

Hard to summarize but this post basically talks about spreading good values as a way of positively influencing the far future, some reasons why it might be a top intervention, and some problems with it.

It seems like "what can we actually do to make the future better (if we have a future)?" is a question that keeps on coming up for people in the debate week.

I've thought about some things related to this, and thought it might be worth pulling some of those threads together (with apologies for leaving it kind of abstract). Roughly speaking, I think that:

• ~Optimal futures flow from having a good reflective process steering things
• It's sort of a race to have a good process steering before a bad process
  • Averting AI takeover and averting human takeover are both ways to avoid the bad process thing (although of course it's possible to have a takeover still lead to a good process)
• We're going to need higher powered epistemic+coordination tech to build the good process
  • But note that these tools are also very useful for avoiding falling into extinction or other bad trajectories, so this activity doesn't cleanly fall out on either side of the "make the future better" vs "make there be a future" debate

There are some other activities which might help make the future better without doing so much to increase the chance of having a future, e.g.:

• Try to propagate "good" values (I first wrote "enlightenment" instead of "good", since I think the truth-seeking element is especially important for ending up somewhere good; but others may differ), to make it more likely that they're well-represented in whatever entities end up steering
• Work to anticipate and reduce the risk of worst-case futures (e.g. by cutting off the types of process that might lead there)

However, these activities don't (to me) seem as high leverage for improving the future as the more mixed-purpose activities.

I want to see a bargain solver for AI alignment to groups: a technical solution that would allow AI systems to solve the pie cutting problem for groups and get them the most of what they want, for AI alignment. The best solutions I've seen for maximizing long run value involve using a bargain solver to decide what ASI does, which preserves the richness and cardinality of people's value functions and gives everyone as much of what they want as possible, weighted by importance. (See WWOTF Afterwards, the small literature on bargaining-theoretic approaches to moral uncertainty.) But existing democratic approaches to AI alignment seem to not be fully leveraging AI tools, and instead aligning AI systems to democratic processes that aren't empowered with AI tools (e.g. CIPs and CAIS'S alignment to the written output of citizens' assemblies.) Moreover, in my experience the best way to make something happen is just to build the solution. If you might be interested in building this tool and have the background, I would love to try to connect you to funding for it. 

For deeper motivation see here.

I really liked several of the past debate weeks, but I find it quite strange and plausibly counterproductive to spend a week in a public forum discussing these questions.

There is no clear upside to reducing the uncertainty on this question, because there are few interventions that are predictably differentiated along those lines.

And there is a lot of communicative downside risk when publicly discussing trading off extinction versus other risks / foregone benefits, apart from appearing out of touch with > 95% of people trying to do good in the world ("academic" in the bad sense of the word).

I have the impression we have not learned from the communicative mistakes of 2022 in that we are again pushing arguments of limited practical import that alienate people and limit our coalitional options.

Is this question really worth discussing and publicly highlighting when really getting more buy in into existential risk prevention work broadly construed would be extremely desirable and naturally, in the main, both reduce extinction risk and increase the quality of futures where we survive?

Clarifying "Extinction"

I expect this debate week to get tripped up a lot by the term “extinction”. So here I’m going to distinguish:

• Human extinction — the population of Homo sapiens, or members of the human lineage (including descendant species, post-humans, and human uploads), goes to 0.
• Total extinction — the population of Earth-originating intelligent life goes to 0.

Human extinction doesn’t entail total extinction. Human extinction is compatible with: (i) AI taking over and creating a civilisation for as long as it can; (ii) non-human biological life evolving higher intelligence and building a (say) Gorilla sapiens civilisation.

The debate week prompt refers to total extinction. I think this is conceptually cleanest. But it’ll trip people up as it means that most work on AI safety and alignment is about “increasing the value of futures where we survive” and not about “reducing the chance of our extinction” — which is very different than how AI takeover risk has been traditionally presented.  I.e. you could be strongly in favour of "increasing value of futures in which we survive" and by that mean that the most important thing is to prevent the extinction of Homo sapiens at the hands of superintelligence. In fact, because most work on AI safety and alignment is about “increasing the value of futures where we survive”, I expect there  won’t be that many people who properly understand the prompt and vote “yes”.

So I think  we might want to make things more fine-grained. Here are four different activities you could do (not exhaustive):

1. Ensure there’s a future for Earth-originating intelligent life at all.
2. Make human-controlled futures better.
3. Make AI-controlled futures better.
4. Make human-controlled futures more likely. 

For short, I’ll call these activities:

1. Future at all.
2. Better human futures.
3. Better AI futures.
4. More human futures.

I expect a lot more interesting disagreement over which of (1)-(4) is highest-priority than about whether (1) is higher-priority than (2)-(4).  So, when we get into debates, it might be worth saying which of (1)-(4) you think is highest-priority, rather than just “better futures vs extinction”. 

Heads up that the symposium event will be happening in the comments of this post, starting at 3pm (UK time). 

Now's a good time to leave questions or key considerations in the comments of the post, for Will MacAskill, Tyler John, Andreas Mogensen, Greg Colbourn and Michael St Jules to respond to.