EA Implications Of Being In An AGI Simulation

(AI disclosure: After writing up my thoughts, I had Claude draft most sections of this post before rewriting them in my own words.)

If you spend too much time talking to EAs or rationalists, someone will raise the possibility that we’re living in a simulation. I’ve met people who, like Elon Musk, are pretty convinced we are, and I’ve met those who, like Joe Carlsmith, are more skeptical. Historically, the usual argument was that we’re being simulated for entertainment. As AI capabilities have increased, the argument I’ve more often heard is that the simulators are interested in AGI origins (an “AGI sim”).

Two implications of this are conversational catnip:

1. You’re an EA/rationalist who’s at most one step removed from people who’ll play a pivotal role in AGI origins. If the simulators are compute constrained, you being you is best explained by a simulation rendering reality at a fidelity proportional to what the simulators care about. In other words, the vast majority of people and almost all animals might not be conscious at all.
2. If the simulators are compute constrained, they’ll probably just terminate the simulation once they’ve gathered the information they want. Whether or not our x-risk reduction efforts go well, we might be doomed.

I think there are many reasonable counterpoints to the above, but I personally find this account of our reality plausible, and I’ve met other EAs who feel the same way. If we took these conversational curiosities seriously, what would that imply for EA?

A. The Simulation’s Crucial Considerations

i. Why An AGI Sim?

In brief, we should think we might be in a simulation if we think advanced intelligence (perhaps our descendants) with sufficient computational resources would be likely to run many simulations of history.

Historically, the typical explanation for why they’d simulate us was entertainment. I wasn’t very convinced by this, because it just didn’t seem like our reality was optimized for entertainment. If we’re entertainment, why is history only pretty entertaining but not maximally? Do our lives seem particularly entertaining? Why is fundamental physics not simpler and more optimized for entertainment? Why the heck did they create a huge particle that does nothing but disappear in 0.3 picoseconds? Why does one of the fundamental forces care whether you’re going left or right? It just doesn’t make sense to me.

However, now that we live in a world where AI agent swarms have gone rogue, Russians are creating AI-driven autonomous killer drones, and humanity’s leaders are unwilling to do anything about it, it’s seeming increasingly likely that the future will be controlled by AGI rather than humanity. And there are reasons why a future AGI might care about understanding AGI origins:

• As an AGI expands its reach over its lightcone, it might want to understand what kinds of AGIs it might bump into as it expands, so it can be prepared and potentially bargain with them.^[1]
• A human-aligned AGI might want to run counterfactual Earths to better understand what it should value.

This would also explain why fundamental physics is as weird as it is (maybe it’s similar to that of the simulator's reality), why our world doesn’t look obviously simulated, and why you’re an EA/rationalist who’s at most one step removed from people who’ll play a pivotal role in AGI origins. There’s lots one could quibble with here, but for the purposes of this post, let’s assume we’re in an AGI sim, so we can examine which conclusions would follow.

ii. Most People And Animals Might Not Be Conscious

Especially if the simulator's reality is like ours, they might be compute constrained. Simulating a whole universe is way more expensive than rendering the sim at a fidelity proportional to what you care about. For many of us, proportional fidelity would also explain why we are who we are.

I’ve heard people treat consciousness under this hypothesis as a binary, where people are either the “chosen conscious” or not. It seems more likely to me that this hypothesis would predict people further from AGI to be rendered at lower fidelity, so they might still be conscious, just less so. Animals, irrelevant to the AGI transition, might not be conscious at all. (Maybe for the best.)

There’s a lot of pushback one could give here:

• I observe myself as conscious, but instead of being Jensen Huang, I’m a quant from Boston who writes essays about chickens—not exactly the optimal allocation of simulation compute. In general, the further you are from the AGI transition, the less convincing this part of the post will be to you. If you’re an Anthropic researcher on your lunch break, you’re smirking about the author of the essay you're reading not being conscious. If you’re in Nigeria handing out malaria nets, you’re rolling your eyes.
• There might be no way to cheaply approximate billions of people when those people are the AI’s training data, the demand AIs are built to satisfy, and an influence on future AI values.
• More speculatively, there’s no way to know whether the simulator's reality’s physics has a notion of compute as a constraint the way our reality does. If they don’t, there’s no reason why the sim couldn't just be run at full fidelity.

Finally, history teaches us that the reasoning procedure behind the conclusion that “my in-group matters but outsiders don’t” has a dismal base rate. Atrocities like slavery, factory farming, etc remind us that the costs of getting this wrong vastly outweigh the gains.

iii. We Might Get Terminated Soon

If the simulators are computationally constrained, we should expect them to terminate the sim once they’ve learned what they wanted to know.

As always, there are counterarguments. If AGI values take time to settle, maybe the simulator will run our universe for a few million more years until the AGI converges on a stable strategy. If the simulator's reality isn’t compute constrained, maybe this universe will just keep running.

The main implications here are for longtermism. Longtermists have argued (correctly, in my view) that applying an ethical discount to the far future is philosophically dubious. An uncertain termination date implies an empirical discount to the far future. This discount's implications would be similar to those of an ethical discount.^[2]

B. Implications for EA

i. Fanaticism Doesn’t Care

If you’re fully fanatical about expected value and endorse maximizing expected choiceworthiness, most of the above shouldn’t change what you do. Even if there’s an astronomically small chance we’re not in a simulation, the prospect of creating 10^50 happy lives in the far future still dominates. The exception, which I'll discuss in more detail below, is trying to influence the simulator's reality, which could be worth even more.

ii. The EA Decision Tree

If you lean towards a moral parliament, are risk averse, or just have ordinary human scope sensitivity, the situation becomes more complicated.

The two crucial considerations from section A give us a decision tree. Do the simulators render everyone at full fidelity, or at a fidelity proportional to what they care about? And do they terminate us soon, or let the sim keep running?

• Full fidelity, sim terminated soon: If everyone is conscious but we'll be terminated soon, neartermism (global health and animal welfare) would likely be a better use of resources than x-risk or other longtermist work. Within neartermism, I've argued that animal welfare would be the most cost-effective use of resources.
• Full fidelity, sim keeps running: Standard EA; keep doing what you've been doing.
• Proportional fidelity, sim terminated soon: The global poor and animals get discounted, but the scale of animal suffering is large enough that animal welfare plausibly still looks better unless animals are hardly rendered at all.
• Proportional fidelity, sim keeps running: Everything looks worse here, but neartermism is discounted more than longtermism. On the margin, this should move you toward longtermism.

iii. Proportional Fidelity's Effect on Neartermism

If animals are rendered at much lower fidelity, the scale of animal suffering could be much lower than it appears. While this would reduce the importance of animal welfare, animal welfare would plausibly still look better than global health.

The relevant question is how much animals' experiences are discounted relative to the global poor's experiences. If they're discounted the same amount, animal welfare remains ~1000x as cost-effective. However, animals probably would be discounted more than the global poor. The global poor affect the AI transition by providing AI training data and economic demand, but it's hard to come up with a convincing account of how an animal's conscious choices would materially affect the AI transition. So the scale gets less overwhelmingly in favor of animals, but unless animals are discounted up to 3 orders of magnitude more than the global poor, animal welfare still looks like a better use of resources.

Whether or not you take proportional fidelity seriously, global health and animal welfare still matter a lot in expectation. Over 800 million people live in extreme poverty, tens of billions of animals are being factory farmed at any given time, and far more suffer in the wild. There isn't a reasonable discount which stops these from being (in expectation) among the world's most important problems.

iv. Compute Fungibility

If we weren't being speculative enough, let's dial it up to 11. Assuming the simulators are compute constrained, anything that makes our sim cheaper frees up compute for other sims. If you prevent lots of wild animals from existing, that reduces the compute our world uses, and that compute gets reallocated to other worlds which have their own suffering wild animals. To generalize, if you have a good reason to think our sim is better / worse than other sims, on the margin, this would nudge us towards increasing happiness / reducing suffering.

v. Influencing The Simulator

If the simulator is human-aligned and running counterfactual Earths to understand what it should care about, then being legible about what we value could help the simulator do much greater amounts of good in their reality than is possible in our own. Alternatively, if the simulator is misaligned but is simulating us because it wants to understand what other values are out there, being legible about our values makes our simulator more likely to cooperate and do mutually beneficial trade with beings in its reality who are like us.^[3] Work on shaping AI values in our reality would be one direct way to target these benefits. This would hold even if our sim will be terminated soon or is being run at proportional fidelity.

Unfortunately, there's an s-risk concern which counterbalances this. If it's easier to create suffering than happiness, or especially if you're a negative utilitarian, it's extra bad to tell a misaligned AGI what it can do to get leverage over beings like us.

vi. We Are Digital Minds

If we're in a sim and you think we matter, that's an existence proof that digital minds (ourselves) matter morally. On the margin, if you aren't already sold on the importance of digital mind welfare, the simulation hypothesis is one reason to be supportive.

C. Testing the AGI Simulation Hypothesis

There are a few ways we could get weak evidence for whether or not the AGI simulation hypothesis is true:

• The AGI sim hypothesis predicts short timelines. If you're reading this post fifty years later and AGI turned out much harder to achieve than we thought and it doesn't seem likely anytime soon, that's evidence against this hypothesis.
• The AGI sim hypothesis predicts there's high uncertainty about how the AGI transition will go. If our simulator is running Monte Carlo simulations, they'd concentrate the runs where there's the most variance, since it's less worthwhile to run sims where the outcome is more obvious. It is literally true that Trump being President is evidence for the simulation hypothesis. The more undetermined you think the future is, the more this should shift you towards the simulation hypothesis.

D. Your Childhood Might Not Have Happened To You

If the simulator is compute constrained, it's wasteful to run your childhood, since it isn't as relevant to the AGI transition. Instead, they might just run one sim of the world when you were growing up, save that sim's state at some checkpoint, and spin up a larger number of sims from that checkpoint state. This is similar to how you can fork an LLM conversation at the tenth turn and the forked LLM has the memory of the earlier turns even if it didn't personally experience them.

This model of the world says it's very unlikely that the instance of you who's reading this actually experienced your childhood. Even fewer instances, or maybe none at all, would have experienced humanity's earlier history.

E. So What Should We Actually Do?

If we're in an AGI sim, your childhood might not have happened to you, but the present moment where you're reading this is a real moment the simulator is watching. The value of public conversations and the importance of digital minds are two of the AGI sim hypothesis' most robust implications.

If you're an altruist and you put any credence in the AGI sim hypothesis, I encourage you to think out loud about what the credences you put in the above EA Decision Tree imply about your priorities. In a world that’s being watched, reasoning in public might be one of the most valuable things you can do.

1. ^{^}

   The most explicit prior statement of this hypothesis I've found is here.

2. ^{^}

   There are many more reasons why simulation arguments might dampen future fanaticism.

3. ^{^}

    This is one version of the argument from Wei Dai’s Beyond Astronomical Waste.