Crosspost of my blog post. 

Note: the things I say here are largely based on this excellent report from Forethought about flourishing futures, and also based somewhat on this other excellent report from Forethought about the intelligence explosion. Conflict of interest alert: I’ve applied to work for them.

AI capabilities are advancing very quickly. The effective AI population is growing by about 25X per year, and maximum task length is doubling every seven months. For this reason, we may soon be in for terrifyingly explosive economic growth of a kind never before seen in history—rather like the impact of having the number of researchers grow 25X per year, and having them get smarter every single year. We might get a century’s worth of economic growth in the span of a decade or even less. Read the linked report if you want more detail (I found it very persuasive).

For this reason, space development is no longer off the table. If you previously thought it wasn’t implausible that we’d develop space sometime this century, you should now think it’s not implausible that we’ll develop it within two decades. After all, a century might only take a decade.

But after we develop space, there’s very serious risk of locking in our current values. Whomever gets to space first will gain control over vast resources which can be used to implement their will further in space. If AIs are given some goals in their development of outer space, then it’s pretty likely they’ll ruthlessly optimize with respect to space resources. Where currently there are many branching possibilities, it is not very unlikely that within a century or two, there will only be one. The pieces might still move around the board, but the future will ultimately be fixed. Nothing much will be able to change.

Does this terrify you? It should. We might soon be deciding the permanent future of humanity—what humans will do for the next many billions of years. And to quote our current president, we “don’t know what the fuck [we’re] doing.” We will be blindly groping about in the dark with almost limitless power.

Almost every society in history so far has engaged in serious moral error (every society if you ask me). For much of history, people owned slaves, beat children, and oppressed women. Barbarism and conquest have been the historical norm, not the exception. We should expect we are making similar moral errors. But if we colonize space, trying to optimize for whatever we care about, then any errors we make could be replicated—irreversibly—throughout the universe. Staggering numbers of digital minds will be created under our thumb, and they will be bound by the errors of their forebears—us.

Here’s a non-exhaustive list of the ways we might lose out on nearly all future value:

• We might fail to count the interests of digital minds and cause them to suffer horribly to appease a comparatively smaller number of biological overlords.
• We might fail to count the interests of non-charismatic digital minds, and thus cause them to suffer for the sake of charismatic digital minds.
• We might not think creating well-off people is very important, and thus try to make good lives for those who already exist without creating happy people. This would be a very bad thing if it’s good to create happy people (as I suspect).
• We might be wrong about well-being. We might, for example, try to maximize pleasure, when in reality, pleasure is only a small slice of what matters. Or, on the opposite end, we might maximize lots of different goods when only pleasure is important.
• We might just not be that concerned about ruthlessly optimizing for good stuff, preferring to produce a future that’s pretty nice for lots of people, rather than one that is optimal.
• There might be weird ways of producing lots of happiness that we just don’t care about—e.g. by creating lab universes or improving welfare of strange and simple organisms.
• We might spread wild-animal suffering to the stars.
• If I had my way, we might fail to spread beautiful nature to the stars. If wild animals live good lives, that could be terrible.
• We might spread factory farming to the stars.
• Even if the digital minds are well-off, we might violate their rights in serious ways.

You shouldn’t trust yourself to build utopia. You certainly shouldn’t trust whomever happens to get to space first to build utopia. Given the history of moral error, we should expect that the person who gets to space first will morally err very seriously. Any mistakes they make could be hugely significant. To build utopia, you have to get every important moral issue right. If not, you miss out on nearly all value. And no human is competent enough to be trusted to get almost every important moral issue right. Even Parfit rejected the repugnant conclusion.

For this reason, we should deliberate before we take over space in pursuit of some goal. We should spend a long time getting superintelligent AI philosophers to think about ethics—about what things are worth optimizing for. We should have something like a long reflection: a time to pause and think, to make sure that we’re getting ethics right.

One of the most important values to give the AI is taking moral uncertainty seriously. We should give it values that favor prolonged and careful reflection, rather than rushing ahead. This ought to be a critical feature of model specs.

This might be the most important thing we ever do. For if we don’t do this, then we might spend the next several billion years optimizing for something wildly suboptimal. That would be, in some sense, the ultimate tragedy.