This post is part of Rethink Prioritiesâ Worldview Investigations Teamâs CURVE Sequence: âCauses and Uncertainty: Rethinking Value in Expectation.â The aim of this sequence is twofold: first, to consider alternatives to expected value maximization for cause prioritization; second, to evaluate the claim that a commitment to expected value maximization robustly supports the conclusion that we ought to prioritize existential risk mitigation over all else.
Introduction
RP has committed itself to doing good. Given the limits of our knowledge and abilities, we wonât do this perfectly but we can do this in a principled manner. There are better and worse ways to work toward our goal. In this post, we discuss some of the practical steps that weâre taking to navigate uncertainty, improve our reasoning transparency, and make better decisions. In particular, we want to flag the value of three changes we intend to make:
- Incorporating multiple decision theories into Rethink Prioritiesâ modeling
- More rigorously quantifying the value of different courses of action
- Adopting transparent decision-making processes
Using Multiple Decision Theories
Decision theories are frameworks that help us evaluate and make choices under uncertainty about how to act. Should you work on something that has a 20% chance of success and a pretty good outcome if success is achieved, or work on something that has a 90% chance of success but only a weakly positive outcome if achieved? Expected value theory is the typical choice to answer that type of question. It calculates the expected value (EV) of each action by multiplying the value of each possible outcome by its probability and summing the results, recommending the action with the highest expected value. But because low probabilities can always be offset by corresponding increases in the value of outcomes, traditional expected value theory is vulnerable to the charge of fanaticism, ârisking arbitrarily great gains at arbitrarily long odds for the sake of enormous potentialâ (Beckstead and Thomas, 2021). Put differently, it seems to recommend spending all of our efforts on actions that, predictably, wonât achieve our ends.
Alternative decision theories have significant drawbacks of their own, giving up one plausible axiom or another. The simple alternative is expected value maximization but with very small probabilities rounded down to zero. This gives up the axiom of continuity, which suggests for a relation of propositions A ⼠B ⼠C, that there exists some probability that would make you indifferent between B and a probabilistic combination of A and C. This violation causes some weird outcomes where, say, believing the chance of something is 1 in 100,000,000,000 can mean an action gets no weight but believing itâs 1.0000001 in 100,000,000,000 means that the option dominates your considerations if the expected value upon success is high enough, which is a kind of attenuated fanaticism. There are also other problems like setting the threshold for where you should round down.
Alternatively, you could go with a procedure like weighted-linear utility theory (WLU) (Bottomley and Williamson, 2023), but that gives up the principle of homotheticity, which involves indifference to mixing a given set of options with the worst possible outcome. Or you could go with a version of risk-weighted expected utility (REU) (Buchak, 2013) and give up the axiom of betweenness which suggests the order in which you are presented information shouldnât alter your conclusions.
Itâs very unclear to us, for example, that giving up continuity is preferable to giving up homotheticity, and neither REU or WLU really logically eliminate issues with fanaticism (even if it seems in practice, say, WLU produces negative values for long shot possibilities in general). It seems once you switch from pure EV to other theories, whether it be REU, WLU, expected utility with rounding down, or some other future option, there isnât an option thatâs clearly best. Instead, many arguments rely on competing, but ultimately not easily resolvable, intuitions about which set of principles are best. Still, at worst, it seems the weaknesses in these alternative options are similar in scope to the amount of weakness provided to pure EV logically suggesting spending (and predictably wasting) all of our resources not on activities like x-risk prevention or insect welfare, but on actions like interacting with the multiverse or improving the welfare of protons.
Broadly, we donât think decision theories with various strengths and weaknesses, axiomatic and applied, are the type of claim you can be highly confident about. For this reason, we ultimately think you need to be unreasonably confident that a given procedure, or set of procedures that agree on the types of actions they suggest, is correct (possibly >90%) in order for the uncertainty across theories and what they imply not to impact your actions. While there are arguments and counterarguments for many of these theories, weâre more confident in the broad claim that no arguments for one of these theories over all the others is decisive than we are in any particular argument or reply for any given theory.
So, we still plan to calculate the EV of the actions available to us, since we think in most cases this is identical to EV with rounding down. However, we wonât only calculate the EV of those actions anymore. Now, we plan to use other decision theories as well, like REU and WLU, to get a better understanding of the riskiness of our options. This allows us, among other things, to identify options that are robustly good under decision theoretic uncertainty. (As Laura Duffy notes in a general discussion of risk aversion and cause prioritization and in the case of only the next few generations, work on corporate campaigns for chickens fits this description: itâs never the worst option and rarely produces negative value across these procedures). Using a range of decision theories also helps us represent internal disagreements more clearly: sometimes people agree on the probabilities and values of various outcomes, but disagree about how to weigh low probabilities, negative outcomes, or outcomes where our gamble doesnât pay off. By formalizing these disagreements, we can sometimes resolve them.
Quantify, Quantify, Quantify
Weâve long built models to inform our decision-making. However, probabilities can be unintuitive and the results of more rigorous calculations are often surprising. Weâve discovered during the CURVE sequence, for instance, that small changes to different kinds and levels of risk-aversion can alter what you ought to do; and, even if you assume that you ought to maximize expected utility, making small adjustments to future risk structures and value trajectories have significant impacts on the expected value of the existential risk mitigation work. And, of course, before the present sequence, RP has built many models, for example, to try to estimate some moral weights for animals, finding significant variance across them.
Whatâs more, there are key areas where we know our models are inadequate. For example, itâs plausible that returns on different kinds of spending diminish at different rates, but estimating these rates remains difficult. We need to do more work to make thoughtful tradeoffs between, say, AI governance efforts and attempts to improve global health. Likewise, itâs less complex to assess the counterfactual credit due to some animal welfare interventions but extremely difficult to estimate the counterfactual credit due to efforts to reduce the risk of nuclear war. Since these kinds of factors could swing overall cost-effectiveness analyses, itâs crucial to keep improving our understanding of them. So, weâll keep investigating these issues as systematically as we can.
None of this is to say we take the outputs of these types of quantitative models literally. We donât. Nor is it to claim there is no place at all for qualitative inputs or reasoning in our decision-making. It is to say we think quantifying our uncertainties whenever possible generally helps us to make better decisions. The difficulty of accounting for all of the above issues are typically made worse, not better, when precise quantitative statements of beliefs or inputs are replaced by softer qualitative judgments. We think the work in the CURVE sequence has further bolstered this case that even when you canât be precise in your estimates, quantifying your uncertainty can still significantly improve your ability to reason carefully.
Transparent Decision-Making
Knowing how to do good was hard enough before we introduced alternative decision theories. Still, RP has to make choices about how to distribute its resources, navigating deep uncertainty and, sometimes, differing perspectives among our leadership and staff. Since we want to make our choices sensitive to our evidential situation and transparent within the organization, weâre committed to finding a decision-making strategy that allows us to navigate this uncertainty in a principled manner. Thankfully, there are a wide range of deliberative decision-making processes, such as Delphi panels and citizen juries, that are available for just such purposes. Moreover, there are a number of formal and informal methods of judgment aggregation that can be used at the end of the deliberative efforts.
We arenât yet sure which of these particular decision procedures weâll use and we expect creating such a process and executing it to take time. All of these procedures have drawbacks in particular contexts and we donât expect any such procedure to be able to handle all the specific decisions that RP faces. However, weâre confident that a clearly defined decision procedure that forces us to be explicit about the tradeoffs weâre making and why is superior to unilateral and intuition-based decision-making. We want to incorporate the best judgment of the leaders in our organization and own the intra- and inter-cause comparisons on which our decisions are based. So, weâre in the process of setting up such decision procedures and will report back what we can about how theyâre operating.
Conclusion
We want to do good. The uncertainties involved in doing good are daunting, particularly given we are trying to take an impartial, scope sensitive, open to revision approach. However, RP aims to be a model of how to handle uncertainty well. In part, of course, this requires trying to reduce our uncertainty. But lately, weâve been struck by how much it requires recognizing the depth of our uncertaintyâall the way to the very frameworks we use for decision-making under uncertainty. We are trying to take this depth seriously without becoming paralyzedâwhich explains why weâre doubling down on modeling and collective decision-making procedures.
In practice, we suspect that a good rule of thumb is to spread our bets across our options. Essentially, we think weâve entered a dizzying casino where the house wonât even tell us the rules of the game. And even if we knew the rules, weâd face a host of other uncertainties: the long-term payouts of various options, the risk of being penalized if we choose incorrectly among various courses of action, and a host of completely inscrutable possibilities where we have no idea what to think of them. In a situation of this type, it seems like a mistake to assume that one ruleset is correct and proceed accordingly. Instead, we want to find robustly good options among different plausible rulesets whenever we can. And when we canât, we may want to distribute our resources in proportion to different reasonable approaches to prioritization.
This isnât perfect or unobjectionable. But nothing is. RP will continue to do its best to make these decisions as transparently as we can, learning from our mistakes and continuing to try to advance the cause of improving the world.
Acknowledgements

The piece was written by Marcus A. Davis and Peter Wildeford. Thanks to David Moss, Abraham Rowe, Janique Behman, Carolyn Footitt, Hayley Clatterbuck, David Rhys Bernard, Cristina Schmidt Ibåùez, Jacob Peacock, Aisling Leow, Renan Araujo, Daniela R. Waldhorn, Onni Aarne, Melissa Guzikowski, and Kieran Greig for feedback. A special thanks to Bob Fischer for writing a draft of this post. The post is a project of Rethink Priorities, a global priority think-and-do tank, aiming to do good at scale. We research and implement pressing opportunities to make the world better. We act upon these opportunities by developing and implementing strategies, projects, and solutions to key issues. We do this work in close partnership with foundations and impact-focused non-profits or other entities. If you're interested in Rethink Priorities' work, please consider subscribing to our newsletter. You can explore our completed public work here.
I have increasingly become open to incorporating alternative decision theories as I recognize that I cannot be entirely certain in expected value approaches, which means that (per expected value!) I probably should not solely rely on one approach. At the same time, I am still not convinced that there is a clear, good alternative, and I also repeatedly find that the arguments against using EV are not compelling (e.g., due to ignoring more sophisticated ways of applying EV).
Having grappled with the problem of EV-fanaticism for a long time in part due to the wild norms of competitive policy debate (e.g., here, here, and here), I've thought a lot about this, and I've written many comments on the forum about this. My expectation is that this comment won't gain sufficient attention/interest to warrant me going through and collecting all of those instances, but my short summary is something like:
I dislike using "objective" and "subjective" to make these distinctions, but for simplicity's sake / for lack of a better alternative at the moment, I will use them.
For more on this thread, see here: https://forum.effectivealtruism.org/posts/WSqLHsuNGoveGXhgz/disentangling-some-important-forecasting-concepts-termsÂ
I advocate for something like this competitive policy debate, since "fighting EV fire with EV fire" risks "burning the discussion"âincluding the educational value, reputation of participants, etc. But most deliberations do not have to be made within the artificial constraints of competitive policy debate.