Hide table of contents

EA is associated with maximising expected utility. Which leads to counterintuitive recommendations --- such  as (repeatedly) pressing a button that destroys the world with 99.99% chance but has a 1% chance of creating a (10 000+)-times better world. I claim that:

  • the assumptions underlying expected utility maximisation do not apply to these scenarios,
  • it's ok if you don't want to press that button,
  • it's ok if EA is not about pressing that button,
  • [and aaaaaah, for f...s sake, stop associating EA with pressing that button].[1]

More detailed explanation:

An argument for maximising expected utility goes something like this:

Suppose you were risk averse --- for example, suppose you prefferred (A) a  >90% chance of $1 to (B) a 1% chance of $1000. Well, then I expose you to a series of 100 bets like this, and [apply probability theory] suddenly we are comparing (A') a >90% chance of $90 with (B') a >90% chance of $100K. Presumably you don't feel so happy about choosing (A) now, eh? Maybe you want to go back on this risk aversion thing, and start maximising expected utility with every decision, like a proper Utilitarian?

However, this argument critically relies on repeating the bets. If there is only ever one bet, or a small number of them, Central Limit Theorem does not apply!

In particular:

  • In your personal life, there is nothing incoherent about being risk averse with respect to your overall happiness, taken over your whole life.
  • In terms of near-term social impact, there is nothing incoherent about being risk averse with respect to the planetary scale.
  • In longtermism, there is nothing incoherent about being risk averse with respect to the scale of the future trajectories of the universe.

[Also: uuuhm, maaaybe don't rely on things that you don't understand, particularly if they start telling you to jump off a cliff?]


Disclaimer: There is a hypothetical version of this post that includes references, is polished, isn't a rant, has a better example of risk aversion, makes all the disclaimers such as "yeah yeah, I know most people's opinions are more nuanced than what I make them", and that I ran through Granterly a few times. Sadly, that version is less fun, more work, and doesn't exist. So apologies to all offended readers.

Edit: changed the title to better reflect the main claim.

  1. ^

    Insert countless links here. Including by philosophers much smarter than me, who really should know better.

14

0
0

Reactions

0
0
Comments11


Sorted by Click to highlight new comments since:

I like your disclaimer, because I wanted to make a similar post, but since I'm a perfectionist I haven't even started yet 😅 (Edit: I'm not sure about the downvotes, I'm saying I think OP is doing better than me)

Agreed and strongly upvoted.

It may not be incoherent to be risk averse, but there are instances in which it is not expected utility maximizing. If you have a 51% chance of world doubling, the expected utility is greater if you take the bet. That's true, no matter how many times it was previously offered. I don't quite understand why the CLT is relevant. 

Yes, the expected utility is larger. The claim is that there is nothing incoherent about not maximising expected utility in this case.

To try rephrasing: Principle 1: if you have to choose a X% chance of getting some outcome A, and a >=X% chance of a strictly better outcome B, you should take B. Principle 2: if you will be facing a long series of comparably significant choices, you should decide each of them based on expected utility maximisation. Principle 3: you should do expected utility maximisation for every single choice. Even if that is the last/most important choice you will ever make.

The claim is that: P1 is solid. P2 follows from P1 (via Central Limit Theorem, or whatever math). But P3 does not follow from P1/P2, and there will be cases where it might be justified to not obey P3. (Like the case with 51% chance of doubling the world's goodness, 49% chance of destroying it.)

Note that I am not claiming it's wrong to do expected utility maximisation in all scenarios. Just saying that it both doing and not doing it is OK. And therefore it is (very?) non-strategic to associate your philosophical movement with it. (Given that most people's intuitions seem to be against it.)

Does this explanation make sense? Maybe I should change the title to something with expected utility?

I don't think EV reasoning relies on CLT and many repeated incidences. Probabilities (in the important decision-guiding sense) are features of the map, not the territory.

Yes, sure, probabilities are only in the map. But I don't think that matters for this. Or I just don't see what argument you are making here. (CLT is in the map, expectations are taken in the map, and decisions are made in the map (then somehow translated into the territory via actions). I don't see how that says anything about what EV reasoning relies on.)

I'll respond with 2 orthonormal comments on a real life use of extremism, then moderation.

Re: COVID, the correct course of action (unless one was psychic) was to be extremely paranoid at the start (trying for total bubbling, sterilizing outside objects, etc) because the EV was very downside-skewed—but as more information came in, to stop worrying about surfaces, start being fine with spacious outdoor gatherings, get a good mask and be comfortable doing some things inside, etc.

That is, a good EA would have been faster than the experts on taking costly preventative acts and faster than the experts on relaxing those where warranted.

Some actual EAs seemed to do this well, and others missed in one direction or the other (there was a lot of rapid group house self-sorting in March/April 2020 over this, and then a slower process afterward).

To be concrete about my model, sterilizing groceries was the right call in March 2020 but not by June 2020 (when we knew it very probably didn't transmit through surfaces), and overall maximum-feasible alert was the right call in March 2020 but not by June 2020 (when we knew the IFR was low for healthy young people and that the hospitals were not going to be too overwhelmed).

"Be sure the act is effective" is not a good proxy for "take actions based on EV". In March 2020, the officials were sure (based on a bad model) that COVID wasn't airborne. We masked up all the same, not because we knew it would be effective but because the chance was large enough for the expected gain to outweigh the cost.

Also, money-pumping does happen, it's why payday loans exist and stock picking exists (while the EMH can't be literally true, you need very expensive strategies to beat the EMH.)

More generally, organizations like the Long Now are really reliant on no distributional shift, which in my view is probably not going to happen, and if that happens they're useless (like so many other attempts to shift the long-term future.

Agree with Acylhalide's point - you only need to be non-Dutchbookable by bets that you could actually be exposed to.

To address a potential misunderstanding: I agree with both Sharmake's examples. But they don't imply you have to maximise expected utility always. Just when the assumptions apply.

More generally: expected utility maximisation is an instrumental principle. But it is justified by some assumptions, which don't always hold.

I think the assumptions are usually true, though if they involve one-shot situations things change drastically.

Aren't x-risk interventions and causes basically one-shot?

Uhm, yes, that seems right. That's why this matters.

I do not agree, because in most cases bets are repeated bets, so this doesn't apply. (I also think by default the long-term future has extreme outcomes relative to today, but that's a different case.)

Do not be moderate in your actions (usually) go extreme.

Curated and popular this week
 ·  · 25m read
 · 
Epistemic status: This post — the result of a loosely timeboxed ~2-day sprint[1] — is more like “research notes with rough takes” than “report with solid answers.” You should interpret the things we say as best guesses, and not give them much more weight than that. Summary There’s been some discussion of what “transformative AI may arrive soon” might mean for animal advocates. After a very shallow review, we’ve tentatively concluded that radical changes to the animal welfare (AW) field are not yet warranted. In particular: * Some ideas in this space seem fairly promising, but in the “maybe a researcher should look into this” stage, rather than “shovel-ready” * We’re skeptical of the case for most speculative “TAI<>AW” projects * We think the most common version of this argument underrates how radically weird post-“transformative”-AI worlds would be, and how much this harms our ability to predict the longer-run effects of interventions available to us today. Without specific reasons to believe that an intervention is especially robust,[2] we think it’s best to discount its expected value to ~zero. Here’s a brief overview of our (tentative!) actionable takes on this question[3]: ✅ Some things we recommend❌ Some things we don’t recommend * Dedicating some amount of (ongoing) attention to the possibility of “AW lock ins”[4]  * Pursuing other exploratory research on what transformative AI might mean for animals & how to help (we’re unconvinced by most existing proposals, but many of these ideas have received <1 month of research effort from everyone in the space combined — it would be unsurprising if even just a few months of effort turned up better ideas) * Investing in highly “flexible” capacity for advancing animal interests in AI-transformed worlds * Trying to use AI for near-term animal welfare work, and fundraising from donors who have invested in AI * Heavily discounting “normal” interventions that take 10+ years to help animals * “Rowing” on na
 ·  · 3m read
 · 
About the program Hi! We’re Chana and Aric, from the new 80,000 Hours video program. For over a decade, 80,000 Hours has been talking about the world’s most pressing problems in newsletters, articles and many extremely lengthy podcasts. But today’s world calls for video, so we’ve started a video program[1], and we’re so excited to tell you about it! 80,000 Hours is launching AI in Context, a new YouTube channel hosted by Aric Floyd. Together with associated Instagram and TikTok accounts, the channel will aim to inform, entertain, and energize with a mix of long and shortform videos about the risks of transformative AI, and what people can do about them. [Chana has also been experimenting with making shortform videos, which you can check out here; we’re still deciding on what form her content creation will take] We hope to bring our own personalities and perspectives on these issues, alongside humor, earnestness, and nuance. We want to help people make sense of the world we're in and think about what role they might play in the upcoming years of potentially rapid change. Our first long-form video For our first long-form video, we decided to explore AI Futures Project’s AI 2027 scenario (which has been widely discussed on the Forum). It combines quantitative forecasting and storytelling to depict a possible future that might include human extinction, or in a better outcome, “merely” an unprecedented concentration of power. Why? We wanted to start our new channel with a compelling story that viewers can sink their teeth into, and that a wide audience would have reason to watch, even if they don’t yet know who we are or trust our viewpoints yet. (We think a video about “Why AI might pose an existential risk”, for example, might depend more on pre-existing trust to succeed.) We also saw this as an opportunity to tell the world about the ideas and people that have for years been anticipating the progress and dangers of AI (that’s many of you!), and invite the br
 ·  · 3m read
 · 
Hi all, This is a one time cross-post from my substack. If you like it, you can subscribe to the substack at tobiasleenaert.substack.com. Thanks Gaslit by humanity After twenty-five years in the animal liberation movement, I’m still looking for ways to make people see. I’ve given countless talks, co-founded organizations, written numerous articles and cited hundreds of statistics to thousands of people. And yet, most days, I know none of this will do what I hope: open their eyes to the immensity of animal suffering. Sometimes I feel obsessed with finding the ultimate way to make people understand and care. This obsession is about stopping the horror, but it’s also about something else, something harder to put into words: sometimes the suffering feels so enormous that I start doubting my own perception - especially because others don’t seem to see it. It’s as if I am being gaslit by humanity, with its quiet, constant suggestion that I must be overreacting, because no one else seems alarmed. “I must be mad” Some quotes from the book The Lives of Animals, by South African writer and Nobel laureate J.M. Coetzee, may help illustrate this feeling. In his novella, Coetzee speaks through a female vegetarian protagonist named Elisabeth Costello. We see her wrestle with questions of suffering, guilt and responsibility. At one point, Elisabeth makes the following internal observation about her family’s consumption of animal products: “I seem to move around perfectly easily among people, to have perfectly normal relations with them. Is it possible, I ask myself, that all of them are participants in a crime of stupefying proportions? Am I fantasizing it all? I must be mad!” Elisabeth wonders: can something be a crime if billions are participating in it? She goes back and forth on this. On the one hand she can’t not see what she is seeing: “Yet every day I see the evidences. The very people I suspect produce the evidence, exhibit it, offer it to me. Corpses. Fragments of