In fairness to Richard, I think it comes across in text a lot more strongly than in my view it came across listening on youtube

I really like this piece, and I think I share in a lot of these views. Just on some fairly minor points:

1. Deep Incommensurability. It seems like incommensurability helps with regards to avoiding MPL, but not actually that much. For example, there seem many moral theories (ie something that is somewhat like Person Affecting Views) that are incommensurable (or indifferent) between different size worlds, but not different qualities. So they may really care if it is a world of humans, or insects, or hedonium.

I can imagine views (they do run into non-identity, but maybe there is ways of formulating them that don't) that this would be a real problem. For example, imagine a view that holds that simulated human existence if the best form of life, but is indifferent between that and non-existence. As such, they won't care whether we leave the universe insentient, but faced with a pair-wise choice between hedonium and simulated humans, they will take the simulated humans everytime. So they don't care much if we do extinct, but do care if the hedonistic utilitarians win. indeed, these views may be even less willing to take trades than many views that care about quantity. I imagine many religions, particularly universalist religions like Christianity and Islam, may actually fall into this category.

2. I think some more discussion of the 'kinetics' vs 'equilibrium' point you sort of allude to seems pretty interesting. I think you could reasonably hold the view that rational (or sensing or whatever other sort of beings) beings converge to moral correctness in infinite time. But we are likely not waiting infinite time before locking in decisions that cannot be reversed. Thus, because irreversible moral decisions could occur at a faster rate than correct moral convergence (ie the kinetics of the process is more important than what it would be at equilibrium), we shouldn't expect the equilibrium process to dominate. I think you gesture towards this, but I think exploration of the ordering further would be very interesting.

3. I also wonder if views that are pluralist rather than monist about value may make the MPL problem worse or better. I think I could see arguments either way, depending on exactly how those views are formulated, but would be interesting to explore.

Very interesting piece anyway, thanks a lot, and really resonates with a lot I've been thinking about

I'm sure I'll have a few more comments at some point as I revisit the essay.

Ye, I might be wrong, but something like Larry Temkin's model might work best here (been a while since I read it so may be getting it wrong)

I think averageists may actually also care about the long term future a lot, and it may still have a MPL if they don't hold (rapid) diminish returns to utility WITHIN lives (ie it is possible for the average life to be a lot worse or a lot better than today). Indeed, given (potentially) plausible views on interspecies welfare comparisons, and how bad the lvies of lots of non-humans seem today, this just does seem to be true. Now, its not clear they shouldn't be at least a little more sympathetic to us converging on the 'right' world (since it seems easier), but it doesn't seem like they get out of much of the argument either

I think a really important question in addressing this is something like - does the USA remain 'unfanatical' if the shackles are taken off powerful people. This is where I think the analysis of the USA goes a little bit wrong - we need to think about what the scenario looks like if it is possible for power to be much much more extremely concentrated than it is now. Certainly, in such a scenario, its not obvious thatit will be true post AGI that "even a polarizing leader cannot enforce a singular ideology or eliminate opposition"

You're sort of right on the first point, and I've definitely counted that work in my views on the area. I generally prefer to refer to it as 'making sure the future goes well for non-humans' - but I've had that misinterpreted as just focused on animals. I

I think for me the fact that the minds will be non-human, and probably digital, matter a lot. Firstly, I think arguments for longtermism probably don't work if the future is mostly just humans. Secondly, the fact that these beings are digital minds, and maybe digital minds very different to us, means a lot of common responses that are given for how to make the future go well (eg make sure they're preferred government 'wins' the ASI race) definitely looks less promising me. Plus you run into trickier problems like what Carlsmith discusses in his Otherness and Control series, and on the other end, if conscious AIs are 'small minds' ala insects (lots of small conscious digital minds that are maybe not individually very smart) you run into a bunch of the same issues of how to adequately treat them. So this is sort of why I call it 'digital minds', but I guess thats fairly semantic.

On you're second point, I basically think it could go either way. I think this depends on a bunch of things, including if, how strong and what type (ie what values are encoded) of 'lock in' we get, how 'adaptive' consciousness is etc. At least to me, I could see it going either way (not saying 50-50 credence towards both, but my guess is I'm at least less skeptical than you). Also, its possible that these are the more likely scenarios to have abundant suffering (although this also isn't obvious to me given potential motivations for causing deliberate suffering).

I wish more work focused on digital minds really focused on answering the following questions, rather than merely investigating how plausible it is that digital minds similar to current day AI's could be sentient:

1. What does good sets of scenarios for post-AGI governance need to look like to create good/avoid terrible (or whatever normative focus we want) futures, assuming digital minds are the dominant moral patients going into the future 1a) How does this differ dependent on what sorts of things can be digital minds eg whether sentient AIs are likely to happen 'by accident' by creating useful AIs (including ASI systems or sub-systems) vs whether sentient AIs have to be delibrately built? How do we deal with this trade off?

2. Which of these good sets of scenarios need certain actions to be taken pre-ASI development (actions beyond simply ensuring we don't all die)? Therefore, what actions would we ideally take now to help bring about such good futures? This includes, in my view, what, if any, thicker concept of alignment than 'intent alignment' ought we to use.

3. Given the strategic, political, geopolitical and technological situation we are in, how, if at all, can we make concrete progress to this? We obviously can't just 'do research' and hope this solves everything. Rather, we ought to use this to guide specific actions that can have impact. I guess this step feels rather hard to do without 1 and 2, but also, as far as I can tell, no one is really doing this?

I'm sure someone has expressed this same set of questions elsewhere, but i've not seen them yet, and at least to me, seem pretty neglected and important

Factory farming?

Just flagging, it seems pretty strange to have something about career choice in 'Community'

Pretty sure EA basically invented that (yes people were working on stuff before then and outside of it, but still that seems different to 'reinventing the wheel')