In other places people have asked me "what does it mean to merge very different minds together"?
I think I'm claiming something simpler here.
If there is any way at all for two observer-moments to continue as just one observer-moment later on - let's say it's two near-copies with a difference between them erased, as in Yudkowsky's Anthropic Trilemma - then it seems like there's a way to make two observer-moments' futures worse instead of one, while only having one observer-moment to be that worse future. This alone makes utilitarian aggregation odd to me (for near-copies).
If you can merge near-copies, and if additionally you can gradually turn very different minds into near-copies, then you can presumably eventually merge them. Then, it also seems to make utilitarian aggregation odd to me (for more distinct minds).
In the post I never actually specified how close the minds of 'everybody' in the thought experiment are. Some people assumed I was talking about merging very different minds. Whereas I was comfortable working from the more abstract assumption that merging minds is possible at all, to argue that if you can merge minds, utilitarian aggregation has a problem. Perhaps I should have made that clearer.
(The setup has other idealizations too, like everyone’s lives ending at the right time for the merge. I was treating the possibility of merging the same way.)
So yes, perhaps very different minds can’t be merged. Or perhaps they can be. But the simple conditional of "if minds can merge, utilitarian aggregation has a problem" is all I’m arguing for - and even if only near-copies can merge, aggregation still has a problem.
Similar argument was presented as utilitarian argument against quantum immortality by some writers on LW.
They argue: The number of branches of multiverse where I survive is declining after quantum suicide, so the total utility is declining, so I should not care about QI.
Quantum immortality instead is based on the idea of preserving continuity.
Traditional utilitarianism doesn't care about continuity (and also about uniqueness and about complexity of experience). I think that we should update utilitarianism, as it was created as an apoximation of what is really good and should not be taken as religious dogma.