Adding to Tristan's 'large enough multiverse' point, and re section iii ('We Might Get Terminated Soon'). There seem to be a few ways to think about personal continuity here. To describe two:

1. Continuity is causal, and termination of simulations contributes to some probability of "seeing nothing" next equal to the probability of being a simulated copy.
2. Continuity is content-based, and termination of simulations just means you continue to whatever remains and is sufficiently coherent to be your next moment, e.g. copies in base reality (related to Wei Dai's 'identical copy immortality').

I kinda lean toward 2 > 1. If there’s nothing special about causality that links you from one moment to the next, then what links them might just be that the later moment’s contents coherently follow on from the earlier one’s.

Continuity is ontologically tricky, but in practice we act as if there’s a fact about what we’ll experience next. Overall, I expect to see stuff even after simulations end.

Markov utilitarianism as universalized prudence

I recently proposed a view I call Markov utilitarianism. To me, it feels like a natural way to think about ethics. I came to it after thinking about how I care about my future, and how I really care about how I feel right now, from the first-person perspective.

1. I care about my actual subjective futures, weighted by how likely I am to find myself in each, rather than the physical futures someone looking from a third-person perspective implicitly treats as relevant to me. (Physical futures and my subjective futures might come apart because of copying and merging.)
2. I care about how I feel right now because of the contents of my experience, rather than any elusive "number of subjectively identical copies" someone looking from a third-person perspective would associate with me.

This is prudence, in the philosophical sense of rational concern for one's own wellbeing. And it has exactly the same form as the equation reinforcement learning uses for an agent's value of being in a state, the Bellman equation (hence 'Markov'):

V(t) = r(t) + γ Σ P(t → t') V(t') over types of experience t.

To make it an ethics that considers everybody, we universalize it and sum impartially across all unique perspectives, mine and everyone else's:

U = Σ V(t)

In slogan form: "Prudence is caring about how you feel right now, and about each possible next experience in proportion to how likely you are to find yourself there. Ethics is doing that for every qualitatively unique perspective, counting each once".

Or more precisely: "Count each qualitatively unique perspective once, and care about it as it would care about how it feels now and what it will experience next, if fully informed about its futures".

Markov utilitarianism is closely related to total utilitarianism, but it structurally prefers lives that get better over time, and once minds can be copied and merged, it avoids conclusions like it being permissible to merge people into a being fated for extreme suffering.

So it feels like a natural extension of how you already care about yourself, extended to every perspective. Starting an ethics from this first-person point of view seems better to me because it avoids the third-person assumptions hidden in total utilitarianism.

What if exact copies don't matter but future experiences still do?

I think an ethics resulting from the following two premises is worth exploring:

P1: The total moral value of subjectively indistinguishable observer-moments doesn't scale with the number of copies, i.e. they sum to a constant value.

• This is because subjectively indistinguishable observer-moments, in some sense, happen to the same "someone". One could think of themselves as all their copies. N observer-moments that are subjectively indistinguishable aren't felt N times by that same "someone". Only felt "once".
  • E.g. Even if there were multiple exact copies of me, of my current present experience, I might not really care personally because it doesn't seem to change anything for me. "I" don't notice anything.
  • See Wei Dai's "The Moral Status of Independent Identical Copies" for the tension this creates with standard utilitarianism.
• We can say that multiple "tokens" of subjectively indistinguishable observer-moments belong to the same "type" of observer-moment.

P2: What matters, and can be changed, for each type of observer-moment is the distribution of its subjectively "future" experiences.

• You can't "un-exist" the type of observer-moment itself, but you can change what it continues into.
  • A higher fraction of "future" observer-moments experiencing suffering is dispreferable.
  • A higher fraction of "future" observer-moments experiencing happiness/tranquility is preferable, even if only instrumentally to minimize subjectively "future" suffering.
• This is related to thinking about "anthropic immortality" / "quantum immortality" / "multiverse immortality" in which distributions of post-death observer-moments are of substantial concern.

Using these lenses, the moral situation of the world looks like:

• Consider a graph with types of observer-moments as nodes, such that one node is one type.
• Each type of observer-moment (each node) is associated with some value/disvalue attributed to its current experience.
  • Importantly, the value/disvalue of the current experience is not something we can change in order to help the observer-moment type. This is already fixed.
• Each type of observer-moment (each node) is also associated with any number of subjectively indistinguishable token observer-moments belonging to that specific type.
  • The absolute number of token observer-moments appears to only matter instrumentally insofar as it affects the frequency of expectations of other types of observer-moments.
• Edges represent subjective continuation. An edge from type A to type B exists if an observer-moment of type A can possibly expect to continue as an observer-moment of type B in the next moment.
  • The weight of an edge from type A to type B is the frequency at which an observer-moment of type A should expect to continue as an observer-moment of type B.
  • These weights are things that we might be able to change in order to help the observer-moment type.
• Perhaps the "future" welfare of each type / node could be considered with equal weight in the aggregate to keep things impartial.

In conclusion... well, I'm still thinking about what the objective function might look like.

There's some scale invariance here. E.g. suppose you duplicated the world. That duplication would multiply all token observer-moments by the same factor. This leaves you with the same types and the same distributions for future observer-moments.

In a small enough world, you can create or prevent the existence of new types of observer-moment. I think it's unlikely (30%) that we live in a small enough world.

On the other hand, in a large enough world, all possible types of observer-moments exist. The only thing you can change is adjusting the distributions for future observer-moments for each type of observer-moment. Try to lead types of observer-moments down paths of non-suffering etc. So, this seems like an ethics of flow (rather than stock).