They model that, and after, I think, 1661 iterations of the human-AI trade game, the human-AI trade game accumulates enough wealth for humans that it would've been self-defeating for the humans to defect like that. I think it's still a Nash equilibrium but one where the humans give up perfectly good gains from trade. (Plus blockchain tech can make it hard to confiscate property.)

Thanks for the summary!

by promoting revanchism and ethnic hatred

Hiii! Do you know someplace where I can read up on that? Ty!

Oops, already subscribed. xD I'll try to pay more attention! Ty!

Just learned about it! Do you have a newsletter or something so I can try to be quicker next year? I'll be in Berlin around that time in case someone wants to meet up before or after! 

What has been your personal take-away from this line of thinking? This “standard case” is far from my own thinking, though I agree with the conclusion. Is it also far from your own thinking?

My take:

1. My axiology depends on unknown empirical results from further research into such things as Evidential Cooperation in Large Worlds, but at first approximation I'm skeptical that anything that runs counter to widely shared goals (or convergent drives, e.g., self-preservation) can be particularly good.
2. I endorse antisubstratism and AI rights.
3. I value reducing suffering very highly because I think that suffering is frequently very intense.
4. Bio-humans will have an extremely hard time surviving and reproducing in space or on other planets in our solar system and will have an extremely hard time reaching other solar systems.
5. The Long Reflection is necessary but impossible.
6. They key feature of a being that risks that it can experience suffering is reinforcement learning. Perhaps negative reinforcement can feel very mild or just like the absence of positive reinforcement, but perhaps it's intense.

So what I'm afraid will happen is that an artificial RL agent will seek out resources first elsewhere in our solar system and then elsewhere in the galaxy (something that would be difficult for bio-humans), will run into communication delays due to the lightspeed limit, and will hence split into countless copies, each potentially capable of suffering. Soon they'll be separated so far that even updates on what it means to be value-aligned would travel for a long time, so there'll be moral “drift” in countless directions.

What I would find reassuring is:

1. Making sure AIs want to minimize the risk of suffering of other AIs, or at least near copies.
2. Research into how we can measure and minimize the suffering in RL (at least to the point where it's mild and the pleasure dominates) and some way of applying that research broadly. Sadly it seems questionable to me that a low-suffering training regime is the most efficient training regime.
3. Faster-than-light communication but not transportation so the agent can remain monolithic.
4. Very energy efficient AIs, because they'll face a tradeoff between staying monolithic to avoid value drift but only being able to harvest energy sources in our solar system or even just from the sun vs. the opposite. And the more they can do with little energy, the longer they'll stay in our solar system, which might buy us and them time.
5. More time to get anyone interested in these things, research them, and apply them on a global scale before the lightspeed limits make it hard to disseminate the information.

Human extinction also seems bad on the basis that it contradicts the self-preservation drive that many/most humans have. Peaceful disenfranchisement may be less concerning depending on the details. But at the moment it seems random where we're headed in the coming years because hardly anyone in power is trying to steer these things in any sensible way. Again more time would be helpful.

Basic rights for AIs (and standing in court!) could also provide them with a legal recourse where they currently have to resort to threats, making the transition more likely to go smoothly, like you argue in another post. Currently we're nowhere close to having those. Again more time would be helpful.

I used the comment field in the form to note that a field in the form was marked as optional when it was actually mandatory. That comment got automatically published here, and out of context it made no sense whatsoever. I think it would've been clearer to not automatically transfer this form feedback here (some people might've even assumed that it's private feedback).

Thanks for writing all of this up in one place! 

One of my gripes with the community has long been that maximization is core to EA and we're still really clueless about what it implies and most of the community (outside RP, QURI, etc. and some researchers) seems to have given up on figuring it out. 

I feel like we're like this one computer science professor I had who seemed a bit senile and only taught the sort of things that haven't lost relevance in the last 30 years because he hadn't kept up with anything that happened since the 80s. He probably had good personal and neurological reasons for that, but we don't, right?

I haven't read any EA introductory materials in a few years, but I hope they contain articles about expected value maximization along with articles on how EV is usually largely unknowable due to cluelessness and often undefined due to Pasadena games. That stochastic dominance is arguably a much better approach to prioritization but that Christian Tarsney is so far more or less the only one who has bothered to look into it. That there is perhaps a way forward to figure out what's the best thing to do if we funded some big world-modeling efforts based on software like Squiggle but that hardly anyone outside RP and QURI (and Convergence?) currently bothers to do anything about it. (I've dabbled a bit in these fields but my personal fit doesn't seem to be great.)

Maybe there's even a way to scalably adjust for personal fit whatever recommendations this big model effort might yield. Maybe there are some common archetypes/personas plus quizzes that tell people which ones they are closest to.

Arguably this can wait until this whole AI thing is under control (if that's even possible), but few people will want to work on AI safety, so maybe it doesn't have to wait?

My takeaway has been mostly that I don't have a clue and so will go with some sort of momentary best guess gives me enough fulfillment, enjoyment, and safety. I've written more about it here.

That said, EA has had a great effect on my mental health.

I used to be a crying wreck because of all the suffering in the world. I spread myself thin trying to help everyone. I felt guilty about the majority of terrible things in the world that I was powerless to prevent. (Suicidal too, except that would've been self-defeating.)

Then EA came along and gave me an excuse to “pick my battles,” i.e. focus on a few things where I could make a big difference, taking into account my skills and temperament. Now, if someone went, “Hey, you should become a politician to prevent X and Y,” I could go, “No, I wouldn't be good at that and hate every second of it and it would come at a great cost to A, which I'm already doing.” EA, for the first time, allowed me to set boundaries.

EA also gave me an appreciation for the (perhaps, plausibly, who knows really) great absolute impact that I can have despite the minimal impact that I have (perhaps, plausibly, who knows really) relative to the totality of the suffering in the world. That made it much easier to find fulfillment.