This is a short companion to the longer post, "Markov utilitarianism: Copies can change where you're headed but not how you feel right now."

Copying and merging digital minds may break traditional utilitarianism. "Markov utilitarianism" is a form of utilitarianism that results from applying two fixes to utilitarianism (shown here for total utilitarianism). Both fixes follow from the principle that only what some mind feels, or goes on to feel, should count. Together they turn utilitarianism into a Markov reward process - a mathematical structure that tracks each state's immediate reward and the rewards of the states it goes on to, weighted by how likely they are. In other words, it tracks how each unique perspective feels now and where it is headed.

Exact copies problem: Total utilitarianism prefers that everyone suffer more.

Consider two worlds, World A and World B. In World A there are m digital people in extreme suffering, with each unique perspective containing pain disvalued at -1000 utility. In World B there are the same m digital people in moderate suffering, with each unique perspective containing pain disvalued at -500 utility.

The only other difference is that World A runs only one instance of each of the m digital people, whereas World B runs N instances of each of the m people, such that every digital person's perspective is multiply instantiated N times in the exact same way. N instances are subjectively identical, holding the same perspective, with exactly the same qualitative contents.

There's a straightforward way in which World A is worse. Every unique perspective suffers more intensely. At N=1, total utilitarianism agrees that World A is worse than World B.

But at N=2, total utilitarianism says World A and World B are equally disvaluable. At N>2, the verdict flips, and total utilitarianism says World B is worse and that we should prefer World A, in which every unique perspective is in extreme suffering.

We should stop at this point and ask ourselves how it feels to hold one perspective of one of the m digital people. From the inside, it doesn't mean anything to have your "number of exact copies" or "copy count" increase to N=1,000,000. You share the unique perspective that your subjectively identical copies share. World B just means a lower intensity of suffering to you, no matter what N is. From the unique perspective you inhabit, the verdict never flips.

If ethics is about experience, then a change should only make a moral difference if it shows up somewhere in what someone feels, or goes on to feel. A change in copy count doesn't show up in the perspective you share with your exact copies, and it doesn't show up in anyone else's. So it shows up for no perspective.

Every single one of the m unique perspectives prefers World B. Total utilitarianism prefers World A at N>2, against the preferences of every single unique perspective, demanding that they suffer more rather than less for a number that no one feels.

Subjective futures problem: Total utilitarianism is willing to send everyone into hell.

Utilitarianism may tolerate or even encourage sending perspectives to hell. It does this because, while aggregation has accounted for physical futures, it has never explicitly accounted for subjective futures that perspectives themselves come to experience. It samples from the third person, never from the first. It looks at future experiences as if picked from the whole universe, never from a perspective's position.

This example assumes merging of minds is possible. Other examples, using copying or memory modification instead of merging, appear in the full post.

By default, a mind will exist that experiences mild happiness then continues on to terrible suffering. A utilitarian can choose to create you, under the condition that they merge your mind with that currently mildly happy, but soon to be suffering, other mind. Your creation brings about an additional few mildly pleasant hours of experience, prior to the merging.

What does the utilitarian do? The utilitarian creates you. You add non-zero positive value.

But now there are two perspectives instead of just one headed for terrible suffering. Two terrible subjective futures, but just one physical future.

No suffering was physically added. And yet you're headed straight for hell.

Generalize this to the entire world and the usual forms of utilitarianism are capable of sending everyone to hell. Just do it at the end of everyone's lives. Merge them into one being always destined to suffer. No suffering is physically added, since there's still just one suffering being. That's totally fine by the usual utilitarian views.

Combines into an MRP

To fix both problems, we could count the immediate welfare of each unique perspective only once, and add the expected value of where it's going.

Straightforwardly, this gives the value function of a Markov reward process.

V(t) = r(t) + γ Σ P(t→t') · V(t')

It's no accident that we've arrived at the Bellman equation. This is how valuation of futures works when you have multiple possible futures, each with futures of their own, weighted by how likely each is. Finance, reinforcement learning, control theory, and evolutionary biology (Fisher's reproductive value) all converged on this form. Until now, utilitarian aggregation has not done so explicitly.

We have so far considered the interests of a single unique perspective. An impartial sum of the value functions of all unique perspectives gives an aggregated utility function.

U = Σ V(t), over all types t.

Implications

Markov utilitarianism appears to better account for interests from the first person perspective, and doesn't let third person assumptions about how aggregation should work override such interests. One such assumption is that caring about your future because it's yours is a kind of "selfishness". Total utilitarianism counts your future experiences when they happen, but not how your future matters to you now. Normally the two line up, but copying and merging pull them apart. Markov utilitarianism accounts for both - and it does this impartially across all perspectives that ever exist, including each of your future perspectives, which have futures of their own. So it isn't selfish, because every perspective's stake in its own future counts.

Markov utilitarianism offers action guidance in a large world / multiverse without caring about the absolute quantities, or measure, of each type of perspective. It does this by valuing what each perspective sees next, weighted by how likely each future is - relative measure. Hence, consequentialism can guide action in a large world or multiverse without asking "how much?" It asks "what comes next?" instead.

This may also avoid recommending a monoculture. Copying the best experience doesn't change how it feels, so copies of it add nothing. In a large world / multiverse where every type of perspective already exists, what adds value is improving where perspectives are headed, and since the returns to helping any one type of perspective diminish, the best use of resources likely spreads across many.

Finally, Markov utilitarianism brings symmetric and suffering-focused utilitarianism together, especially in a large world / multiverse. Here, the best thing to do may be to rescue those with futures of extreme suffering by redirecting them into futures of extreme happiness, e.g. by running simulations of their next moments that continue them into the happiest lives. Suffering-focused views value this because it takes suffering out of their future. Symmetric views value it because it's the biggest improvement possible. Both agree on what to do.

Full post

The arguments, math, implications, objections, and references are here: https://forum.effectivealtruism.org/posts/suTGtHhuya3mA8Hte/markov-utilitarianism-copies-can-change-where-you-re-headed