EA Forum Bot Site
EA Forum

Hide table of contents
Comment Permalink
Sharmake2
1
0

I think the assumptions are usually true, though if they involve one-shot situations things change drastically.

Reply
See in context
by Pivocajs
2 min read 11

14

Expected valueRationalityRisk aversion
Frontpage

EA is associated with maximising expected utility. Which leads to counterintuitive recommendations --- such  as (repeatedly) pressing a button that destroys the world with 99.99% chance but has a 1% chance of creating a (10 000+)-times better world. I claim that:

  • the assumptions underlying expected utility maximisation do not apply to these scenarios,
  • it's ok if you don't want to press that button,
  • it's ok if EA is not about pressing that button,
  • [and aaaaaah, for f...s sake, stop associating EA with pressing that button].[1]

More detailed explanation:

An argument for maximising expected utility goes something like this:

Suppose you were risk averse --- for example, suppose you prefferred (A) a  >90% chance of $1 to (B) a 1% chance of $1000. Well, then I expose you to a series of 100 bets like this, and [apply probability theory] suddenly we are comparing (A') a >90% chance of $90 with (B') a >90% chance of $100K. Presumably you don't feel so happy about choosing (A) now, eh? Maybe you want to go back on this risk aversion thing, and start maximising expected utility with every decision, like a proper Utilitarian?

However, this argument critically relies on repeating the bets. If there is only ever one bet, or a small number of them, Central Limit Theorem does not apply!

In particular:

  • In your personal life, there is nothing incoherent about being risk averse with respect to your overall happiness, taken over your whole life.
  • In terms of near-term social impact, there is nothing incoherent about being risk averse with respect to the planetary scale.
  • In longtermism, there is nothing incoherent about being risk averse with respect to the scale of the future trajectories of the universe.

[Also: uuuhm, maaaybe don't rely on things that you don't understand, particularly if they start telling you to jump off a cliff?]

Disclaimer: There is a hypothetical version of this post that includes references, is polished, isn't a rant, has a better example of risk aversion, makes all the disclaimers such as "yeah yeah, I know most people's opinions are more nuanced than what I make them", and that I ran through Granterly a few times. Sadly, that version is less fun, more work, and doesn't exist. So apologies to all offended readers.

Edit: changed the title to better reflect the main claim.

  1. ^

    Insert countless links here. Including by philosophers much smarter than me, who really should know better.

14

0
0

Reactions

0
0
Comments11


Sorted by
New & upvoted
Click to highlight new comments since:
Guy Raveh6
1
0

I like your disclaimer, because I wanted to make a similar post, but since I'm a perfectionist I haven't even started yet 😅 (Edit: I'm not sure about the downvotes, I'm saying I think OP is doing better than me)

Agreed and strongly upvoted.

Reply
1mkl32j2010915
1
0

It may not be incoherent to be risk averse, but there are instances in which it is not expected utility maximizing. If you have a 51% chance of world doubling, the expected utility is greater if you take the bet. That's true, no matter how many times it was previously offered. I don't quite understand why the CLT is relevant. 

Reply
Pivocajs1
0
0

Yes, the expected utility is larger. The claim is that there is nothing incoherent about not maximising expected utility in this case.

To try rephrasing: Principle 1: if you have to choose a X% chance of getting some outcome A, and a >=X% chance of a strictly better outcome B, you should take B. Principle 2: if you will be facing a long series of comparably significant choices, you should decide each of them based on expected utility maximisation. Principle 3: you should do expected utility maximisation for every single choice. Even if that is the last/most important choice you will ever make.

The claim is that: P1 is solid. P2 follows from P1 (via Central Limit Theorem, or whatever math). But P3 does not follow from P1/P2, and there will be cases where it might be justified to not obey P3. (Like the case with 51% chance of doubling the world's goodness, 49% chance of destroying it.)

Note that I am not claiming it's wrong to do expected utility maximisation in all scenarios. Just saying that it both doing and not doing it is OK. And therefore it is (very?) non-strategic to associate your philosophical movement with it. (Given that most people's intuitions seem to be against it.)

Does this explanation make sense? Maybe I should change the title to something with expected utility?

Reply
Linch2
1
1

I don't think EV reasoning relies on CLT and many repeated incidences. Probabilities (in the important decision-guiding sense) are features of the map, not the territory.

Reply
Pivocajs2
0
0

Yes, sure, probabilities are only in the map. But I don't think that matters for this. Or I just don't see what argument you are making here. (CLT is in the map, expectations are taken in the map, and decisions are made in the map (then somehow translated into the territory via actions). I don't see how that says anything about what EV reasoning relies on.)

Reply
Sharmake2
0
1

I'll respond with 2 orthonormal comments on a real life use of extremism, then moderation.

Re: COVID, the correct course of action (unless one was psychic) was to be extremely paranoid at the start (trying for total bubbling, sterilizing outside objects, etc) because the EV was very downside-skewed—but as more information came in, to stop worrying about surfaces, start being fine with spacious outdoor gatherings, get a good mask and be comfortable doing some things inside, etc.

That is, a good EA would have been faster than the experts on taking costly preventative acts and faster than the experts on relaxing those where warranted.

Some actual EAs seemed to do this well, and others missed in one direction or the other (there was a lot of rapid group house self-sorting in March/April 2020 over this, and then a slower process afterward).

To be concrete about my model, sterilizing groceries was the right call in March 2020 but not by June 2020 (when we knew it very probably didn't transmit through surfaces), and overall maximum-feasible alert was the right call in March 2020 but not by June 2020 (when we knew the IFR was low for healthy young people and that the hospitals were not going to be too overwhelmed).

"Be sure the act is effective" is not a good proxy for "take actions based on EV". In March 2020, the officials were sure (based on a bad model) that COVID wasn't airborne. We masked up all the same, not because we knew it would be effective but because the chance was large enough for the expected gain to outweigh the cost.

Also, money-pumping does happen, it's why payday loans exist and stock picking exists (while the EMH can't be literally true, you need very expensive strategies to beat the EMH.)

More generally, organizations like the Long Now are really reliant on no distributional shift, which in my view is probably not going to happen, and if that happens they're useless (like so many other attempts to shift the long-term future.

Reply
Pivocajs1
0
0

Agree with Acylhalide's point - you only need to be non-Dutchbookable by bets that you could actually be exposed to.

To address a potential misunderstanding: I agree with both Sharmake's examples. But they don't imply you have to maximise expected utility always. Just when the assumptions apply.

More generally: expected utility maximisation is an instrumental principle. But it is justified by some assumptions, which don't always hold.

Reply
Sharmake2
1
0

I think the assumptions are usually true, though if they involve one-shot situations things change drastically.

Reply
Michael St Jules 🔸4
1
0

Aren't x-risk interventions and causes basically one-shot?

Reply
Pivocajs1
0
0

Uhm, yes, that seems right. That's why this matters.

Reply
Sharmake-1
0
1

I do not agree, because in most cases bets are repeated bets, so this doesn't apply. (I also think by default the long-term future has extreme outcomes relative to today, but that's a different case.)

Do not be moderate in your actions (usually) go extreme.

Reply
Curated and popular this week
 ·  · 20m read
 · 
Once we expand to other star systems, we may begin a self-propagating expansion of human civilisation throughout the galaxy. However, there are existential risks potentially capable of destroying a galactic civilisation, like self-replicating machines, strange matter, and vacuum decay. Without an extremely widespread and effective governance system, the eventual creation of a galaxy-ending x-risk seems almost inevitable due to cumulative chances of initiation over time across numerous independent actors. So galactic x-risks may severely limit the total potential value that human civilisation can attain in the long-term future. The requirements for a governance system to prevent galactic x-risks are extremely demanding, and they need it needs to be in place before interstellar colonisation is initiated.  Introduction I recently came across a series of posts from nearly a decade ago, starting with a post by George Dvorsky in io9 called “12 Ways Humanity Could Destroy the Entire Solar System”. It’s a fun post discussing stellar engineering disasters, the potential dangers of warp drives and wormholes, and the delicacy of orbital dynamics.  Anders Sandberg responded to the post on his blog and assessed whether these solar system disasters represented a potential Great Filter to explain the Fermi Paradox, which they did not[1]. However, x-risks to solar system-wide civilisations were certainly possible. Charlie Stross then made a post where he suggested that some of these x-risks could destroy a galactic civilisation too, most notably griefers (von Neumann probes). The fact that it only takes one colony among many to create griefers means that the dispersion and huge population of galactic civilisations[2] may actually be a disadvantage in x-risk mitigation.  In addition to getting through this current period of high x-risk, we should aim to create a civilisation that is able to withstand x-risks for as long as possible so that as much of the value[3] of the univers
 ·  · 1m read
 · 
If you are planning on doing AI policy communications to DC policymakers, I recommend watching the full video of the Select Committee on the CCP hearing from this week.  In his introductory comments, Ranking Member Representative Krishnamoorthi played a clip of Neo fighting an army of Agent Smiths, described it as misaligned AGI fighting humanity, and then announced he was working on a bill called "The AGI Safety Act" which would require AI to be aligned to human values.  On the Republican side, Congressman Moran articulated the risks of AI automated R&D, and how dangerous it would be to let China achieve this capability. Additionally, 250 policymakers (half Republican, half Democrat) signed a letter saying they don't want the Federal government to ban state level AI regulation. The Overton window is rapidly shifting in DC, and I think people should re-evaluate what the most important messages are to communicate to policymakers. I would argue they already know "AI is a big deal." The next important question to answer is, "What should America do about it?"
 ·  · 13m read
 · 
  There is dispute among EAs--and the general public more broadly--about whether morality is objective.  So I thought I'd kick off a debate about this, and try to draw more people into reading and posting on the forum!  Here is my opening volley in the debate, and I encourage others to respond.   Unlike a lot of effective altruists and people in my segment of the internet, I am a moral realist.  I think morality is objective.  I thought I'd set out to defend this view.   Let’s first define moral realism. It’s the idea that there are some stance independent moral truths. Something is stance independent if it doesn’t depend on what anyone thinks or feels about it. So, for instance, that I have arms is stance independently true—it doesn’t depend on what anyone thinks about it. That ice cream is tasty is stance dependently true; it might be tasty to me but not to you, and a person who thinks it’s not tasty isn’t making an error. So, in short, moral realism is the idea that there are things that you should or shouldn’t do and that this fact doesn’t depend on what anyone thinks about them. So, for instance, suppose you take a baby and hit it with great force with a hammer. Moral realism says: 1. You’re doing something wrong. 2. That fact doesn’t depend on anyone’s beliefs about it. You approving of it, or the person appraising the situation approving of it, or society approving of it doesn’t determine its wrongness (of course, it might be that what makes its wrong is its effects on the baby, resulting in the baby not approving of it, but that’s different from someone’s higher-level beliefs about the act. It’s an objective fact that a particular person won a high-school debate round, even though that depended on what the judges thought). Moral realism says that some moral statements are true and this doesn’t depend on what people think about it. Now, there are only three possible ways any particular moral statement can fail to be stance independently true: 1. It’s
Relevant opportunities
6
Head of Communications at LEEP
Tatiana K. Nesic Skuratova
· · 5m read
0
0
6
Applications Closing Soon: Nonprofit Operations Accelerator
Deena Englander
· · 1m read
0
0
91
CEA is hiring for a Director of EA Funds
Zachary Robinson🔸
· · 5m read
0
0
View more