Well put. I like to think of the digital world and outer space as the big multipliers of what's possible for the extent of sentient experience. Having control over what can scale in these two worlds seems essential to achieving the best futures and avoiding the worst.
One thought is that the most rapid scale-up of all time(?) happens in coming years/decades here in solar system since you're both densifying compute and scaling up from K=~0.7 to K=2
You might later scale to Kardashev 3 and 4 at 10 OOMs for each point you climb on Sagan-Kardashev scale but this takes on the order of a hundred thousand plus years and billions of years minimum respectively while I'm assuming you've saturated the tech level long before then.
The distinct eras in the universe are maybe:
pre-singularity, singularity to K=2+tech maturity (call it ~10-100 years), K=2+tech maturity to K=3 (maybe 100k years), K=3 to K=4 (~tens of billions of years)
I believe that artificial intelligence and other resources will have diminishing marginal utility. If that's the case, then it should be possible to do so that both humans and future digital minds are close to their maximum level of utility/happiness by splitting resources between them. The idea of "both digital minds and humans are important" would be therefore significantly more attractive for humans and realistic to accomplish than the idea of "digital minds are all of what matters".
On moral patience:
Morality is subjective.
The criteria for moral patience are subjective. You can say that a human is a moral patient, you can say that AI agent is a moral patient, or you can say that a stone is a moral patient. There's no objective truth about that.
However, we can define moral patience as "if I want to maximize my happiness/utility, whose happiness/utility should I care about when choosing my actions"? That is objective.
If a human wants to maximize their happiness, who should they care about?
Firstly, we need to answer what are the reasons why a human would have an interest to care about someone/something else.
The reasons are:
Game-theoretic - especially direct and indirect reciprocity.
Importantly, I think that agents have interest to care about weaker agents (ask me if you want to know why).
I also think that it's in the interest of a human to care about other sentient life, at least to some extent, because:
It's a norm that is useful - if we live by that norm, then we have that guarantee that we won't end up terribly, because as long as we are sentient, someone will care about us.
It's a norm that has a historical precedent and it's easier to sustain a norm than to establish a new one. But the historical precedent is that humans care about existing sentient life, not potential sentient life.
If the norm of respecting sentient life implied that humans have to commit all their resources for the benefit of future digital minds / AI agents, then that norm would stop to be useful to humans, so that norm would either become weaker to the point where human lives matter, or humans would stop creating new AI agents.
Emotional - you care about someone because you simply want them to be happy.
I think it's likely that there are game-theoretic and/or emotional reasons why we should care about AI agents / digital beings, but not to the point that they are all that we should care about and that we should ignore other agents or beings.
On number of agents:
How to count the number of agents? If we have multiple agents on different computers that cooperate and exchange knowledge, and then we have one agent that is distributed among many computers, then what's the difference between them? Does that count as one agent or many?
Let's suppose that we define agents by their goals (i.e. two computers with AI on it are considered different agents, if they aim to achieve different goals). If there are multiple agents with different goals (either human or AIs), then the best thing that they can do is to create one AI agent that will be optimized for the cumulative utility/happiness all of them. So, I expect that there won't be many separate agents with different goals in the future.
It’s unlikely that there are any digital minds right now, but the class of entities that will matter most in history is digital minds. Making sure their interests are taken seriously will be one of the main determinants of how well the future goes. And you should think this even if you assign a low probability to there being any digital minds. Compared to the welfare of expected digital minds, all other welfare is a rounding error.
Why think this? First, digital minds could be produced with amazing efficiency. Computing has grown vastly more efficient. If digital minds are possible, it should be cheap to create them in enormous numbers.
Second, in the far future, we should expect access to lots of energy. In the next many billions of years, we are likely to find ways of harnessing enormous amounts of energy. Nearly all the stuff in the universe that could be converted to value is outside of Earth.
It is a lot easier to convert space resources into computing than into biological organisms. Biological organisms need a stable habitat. Probably we’d need to terraform other planets to make them habitable for biological organisms. Yet this is costly and inefficient. For this reason, it would be hard to sustain very large populations of biological organisms. Not so for digital minds, which could be directly generated from space’s abundant resources.
These two factors mean there could be lots of future digital minds. One estimate put the number around 10^58, many orders of magnitude more than the number of humans, and there are reasons to think that estimate might be conservative. Don’t take this number too literally, but the important takeaway is that even if you were pretty sure that there couldn’t be digital minds, still roughly all expected future minds would be digital.
Earth is small compared to space (SOURCE???). In total it looks like we’ll be able to reach between 10^21 and 10^23 stars. Most welfare will be had, therefore, by the organisms that will populate space, which can by and large only be digital minds.
IMAGINE ALL THE UTILS, LIVING FOR TODAY. YOU MAY SAY I’M A DREAMER, BUT I’M NOT THE ONLY ONE.
The third factor that makes nearly all expected welfare digital is that digital beings could have a lot more welfare than biological beings. We’ll be in charge of designing digital minds—we won’t for biological beings. For this reason, we’d be able to:
Give digital minds vastly more intense experiences than humans have.
Slow down their experiences, so that they subjectively experience them for longer.
Give them very great quantities of all the goods that might make their lives go well. Make them knowledgeable, wise, able to enter valuable relationships with each other, happy, and high-achieving.
One final consideration is simply: the things we do matter a great deal to the welfare of digital minds. In the future, it’s very likely that we’ll try to safeguard human interests. In contrast, there are a range of future ways we might treat digital minds which range from slavery and torture to treating their welfare like our own. Humans don’t have a great track record of taking seriously the welfare of beings very different from ourselves. The stakes are very high.
Together, these factors lead to the following very simple argument for digital beings being the primary beings that matter:
If nearly all expected beings are of some type, and they will have vastly higher expected average welfare than other beings, then what happens to that type of being matters more than what happens to other beings.
Nearly all expected beings are digital, and they will have vastly higher expected average welfare than other beings.
So what happens to digital beings matters more than what happens to non-digital beings.
This conclusion strikes me as very simple, straightforward, and even somewhat obvious. And yet it’s routinely rejected.
First, you might doubt it’s good to create digital minds. You might think that there’s no proactive reason to bring someone into existence, and so we shouldn’t worry about filling space with happy digital minds. But this isn’t an adequate response.
First of all, I think there are many very strong philosophical arguments against such a view. I’ve already laid them out here, so I won’t repeat them. TLDR: the view that there’s no reason to create a happy person is plagued with structural problems, and conflicts with many obvious ethical judgments.
Second, this judgment at best undermines the case for there being a strong reason to create happy digital minds. But it wouldn’t undermine the claim that most of what matters in the future is what happens to digital minds. If the future will likely have a lot of digital minds, then even if there’s no reason to create them, it’s important that the ones that we do create have good lives.
Third, even if you’re unsure about population ethics, you should think there’s some chance it’s good to create a happy person. Otherwise it’s probably neutral. And yet if creating a bunch of happy digital minds might be good and might be neutral, then it’s very good in expectation.[1]
A second objection you might have: perhaps digital minds don’t matter because they’re not human. I don’t find this line of reply promising. If there are beings that are smarter, more virtuous, and happier than us, the fact that they’re made out of silicon shouldn’t lead us to discount their interests. If your mind got uploaded to a computer, assuming you retained consciousness, your welfare would still matter.
Even if you think members of our species matter somewhat more than members of other species, given just how extreme the predominance of digital welfare could be, you should still mostly care about digital minds. And the kind of extreme speciesism that says only biological humans matter is quite implausible; it would imply that Superman wouldn’t matter, nor would Lord of the Rings elves or dwarves.
A third concern: you might doubt that digital minds are possible in principle. Perhaps you think consciousness is substrate dependent, so that only certain kinds of material can give rise to it. I’m pretty skeptical of this.
First of all, I think there are a number of strong arguments for the possibility of digital sentience. This is the view held by most philosophers, and it’s implied by leading views of philosophy of mind. Expert forecasters who specialized in fields related to the existence of digital minds estimated a 90% chance digital minds were possible in principle. They estimated a 50% chance of digital minds existing by 2050. If you’re not an expert, you should probably just defer.
Second, even if you’re skeptical of substrate independence, you should still think digital minds are possible in principle. Even if you need a pretty special recipe to get a mind, there’s no strong argument for thinking that recipe requires carbon. This might make it harder to get digital minds, but it doesn’t affect the core argument.
Third, even if you were pretty sure that digital minds couldn’t be conscious, they might be so numerous that nearly all expected sentient beings are digital. Even just a 1% chance that there might be a population quintillions of times more numerous than people, that might either suffer or experience joy at incomprehensible levels, would be an enormous deal. The difference in scale is so massive that it dominates any concerns about the possibility of digital sentience.
You might be skeptical that digital minds are conscious or as morally important as people if you’re religious because you think they’re not made in the image of God. But you shouldn’t be confident in this, just as you shouldn’t be confident that if you came across human-like aliens, they weren’t made in God’s image. Catholic philosopher Brian Cutter elaborates on this argument here. And even if you think there’s only a 1% chance sophisticated minds are made in God’s image, still nearly all expected people made in God’s image would be digital. In addition, animals aren’t made in God’s image on standard views, but they still matter.
If I’m right about this, then having an adequate legal system for handling AI welfare is hugely important. More research should be done into which legal regimes could protect digital minds, without neglecting human interests. If you want to donate somewhere to help protect the interests of digital minds, Eleos is the best place to give. If you give at least 500 dollars straight up, or 30 a month to Eleos, in response to this article, you get a free paid subscription to the blog.
(I should note: to think Eleos is doing good work, you don’t need to think digital minds are the beings that matter most in the aggregate, and that’s not a position that I’ve ever heard Eleos advocate. All you have to think is that how we treat digital minds is important).
When explaining why you should give to shrimp welfare, I’ve given the following thought experiment: imagine that you could prevent 15,000 shrimp from freezing and suffocating to death by spending a dollar. Turns out that’s the effectiveness of a dollar given to the shrimp welfare project. But given just how numerous future digital minds could be, and the potential staying power of early efforts, giving a dollar to Eleos might be like preventing millions or billions of digital minds from being tortured, in expectation.[2]
This conclusion is a lot less weird than my views about shrimp and insects. Intuitively, because of their cognitive simplicity, we don’t value their interests much. But digital minds could be a lot more mentally complex than us. It’s not radical or extreme to think that if we may be in the process of creating beings vastly more numerous than humans, with vastly more welfare than humans, it’s important to make sure their interests are taken seriously.
There are also reasons to think that if you had many digital successors, they would all be you in the sense relative to prudence. Thus, creating lots of psychological digital successors of ourselves is very good even assuming the person-affecting view.
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...
Well put. I like to think of the digital world and outer space as the big multipliers of what's possible for the extent of sentient experience. Having control over what can scale in these two worlds seems essential to achieving the best futures and avoiding the worst.
One thought is that the most rapid scale-up of all time(?) happens in coming years/decades here in solar system since you're both densifying compute and scaling up from K=~0.7 to K=2
You might later scale to Kardashev 3 and 4 at 10 OOMs for each point you climb on Sagan-Kardashev scale but this takes on the order of a hundred thousand plus years and billions of years minimum respectively while I'm assuming you've saturated the tech level long before then.
The distinct eras in the universe are maybe:
pre-singularity, singularity to K=2+tech maturity (call it ~10-100 years), K=2+tech maturity to K=3 (maybe 100k years), K=3 to K=4 (~tens of billions of years)