This post is part of Rethink Priorities’ Worldview Investigations Team’s CURVE Sequence: “Causes and Uncertainty: Rethinking Value in Expectation.” The aim of this sequence is twofold: first, to consider alternatives to expected value maximization for cause prioritization; second, to evaluate the claim that a commitment to expected value maximization robustly supports the conclusion that we ought to prioritize existential risk mitigation over all else.
Introduction
RP has committed itself to doing good. Given the limits of our knowledge and abilities, we won’t do this perfectly but we can do this in a principled manner. There are better and worse ways to work toward our goal. In this post, we discuss some of the practical steps that we’re taking to navigate uncertainty, improve our reasoning transparency, and make better decisions. In particular, we want to flag the value of three changes we intend to make:
- Incorporating multiple decision theories into Rethink Priorities’ modeling
- More rigorously quantifying the value of different courses of action
- Adopting transparent decision-making processes
Using Multiple Decision Theories
Decision theories are frameworks that help us evaluate and make choices under uncertainty about how to act. Should you work on something that has a 20% chance of success and a pretty good outcome if success is achieved, or work on something that has a 90% chance of success but only a weakly positive outcome if achieved? Expected value theory is the typical choice to answer that type of question. It calculates the expected value (EV) of each action by multiplying the value of each possible outcome by its probability and summing the results, recommending the action with the highest expected value. But because low probabilities can always be offset by corresponding increases in the value of outcomes, traditional expected value theory is vulnerable to the charge of fanaticism, “risking arbitrarily great gains at arbitrarily long odds for the sake of enormous potential” (Beckstead and Thomas, 2021). Put differently, it seems to recommend spending all of our efforts on actions that, predictably, won’t achieve our ends.
Alternative decision theories have significant drawbacks of their own, giving up one plausible axiom or another. The simple alternative is expected value maximization but with very small probabilities rounded down to zero. This gives up the axiom of continuity, which suggests for a relation of propositions A ≥ B ≥ C, that there exists some probability that would make you indifferent between B and a probabilistic combination of A and C. This violation causes some weird outcomes where, say, believing the chance of something is 1 in 100,000,000,000 can mean an action gets no weight but believing it’s 1.0000001 in 100,000,000,000 means that the option dominates your considerations if the expected value upon success is high enough, which is a kind of attenuated fanaticism. There are also other problems like setting the threshold for where you should round down.
Alternatively, you could go with a procedure like weighted-linear utility theory (WLU) (Bottomley and Williamson, 2023), but that gives up the principle of homotheticity, which involves indifference to mixing a given set of options with the worst possible outcome. Or you could go with a version of risk-weighted expected utility (REU) (Buchak, 2013) and give up the axiom of betweenness which suggests the order in which you are presented information shouldn’t alter your conclusions.
It’s very unclear to us, for example, that giving up continuity is preferable to giving up homotheticity, and neither REU or WLU really logically eliminate issues with fanaticism (even if it seems in practice, say, WLU produces negative values for long shot possibilities in general). It seems once you switch from pure EV to other theories, whether it be REU, WLU, expected utility with rounding down, or some other future option, there isn’t an option that’s clearly best. Instead, many arguments rely on competing, but ultimately not easily resolvable, intuitions about which set of principles are best. Still, at worst, it seems the weaknesses in these alternative options are similar in scope to the amount of weakness provided to pure EV logically suggesting spending (and predictably wasting) all of our resources not on activities like x-risk prevention or insect welfare, but on actions like interacting with the multiverse or improving the welfare of protons.
Broadly, we don’t think decision theories with various strengths and weaknesses, axiomatic and applied, are the type of claim you can be highly confident about. For this reason, we ultimately think you need to be unreasonably confident that a given procedure, or set of procedures that agree on the types of actions they suggest, is correct (possibly >90%) in order for the uncertainty across theories and what they imply not to impact your actions. While there are arguments and counterarguments for many of these theories, we’re more confident in the broad claim that no arguments for one of these theories over all the others is decisive than we are in any particular argument or reply for any given theory.
So, we still plan to calculate the EV of the actions available to us, since we think in most cases this is identical to EV with rounding down. However, we won’t only calculate the EV of those actions anymore. Now, we plan to use other decision theories as well, like REU and WLU, to get a better understanding of the riskiness of our options. This allows us, among other things, to identify options that are robustly good under decision theoretic uncertainty. (As Laura Duffy notes in a general discussion of risk aversion and cause prioritization and in the case of only the next few generations, work on corporate campaigns for chickens fits this description: it’s never the worst option and rarely produces negative value across these procedures). Using a range of decision theories also helps us represent internal disagreements more clearly: sometimes people agree on the probabilities and values of various outcomes, but disagree about how to weigh low probabilities, negative outcomes, or outcomes where our gamble doesn’t pay off. By formalizing these disagreements, we can sometimes resolve them.
Quantify, Quantify, Quantify
We’ve long built models to inform our decision-making. However, probabilities can be unintuitive and the results of more rigorous calculations are often surprising. We’ve discovered during the CURVE sequence, for instance, that small changes to different kinds and levels of risk-aversion can alter what you ought to do; and, even if you assume that you ought to maximize expected utility, making small adjustments to future risk structures and value trajectories have significant impacts on the expected value of the existential risk mitigation work. And, of course, before the present sequence, RP has built many models, for example, to try to estimate some moral weights for animals, finding significant variance across them.
What’s more, there are key areas where we know our models are inadequate. For example, it’s plausible that returns on different kinds of spending diminish at different rates, but estimating these rates remains difficult. We need to do more work to make thoughtful tradeoffs between, say, AI governance efforts and attempts to improve global health. Likewise, it’s less complex to assess the counterfactual credit due to some animal welfare interventions but extremely difficult to estimate the counterfactual credit due to efforts to reduce the risk of nuclear war. Since these kinds of factors could swing overall cost-effectiveness analyses, it’s crucial to keep improving our understanding of them. So, we’ll keep investigating these issues as systematically as we can.
None of this is to say we take the outputs of these types of quantitative models literally. We don’t. Nor is it to claim there is no place at all for qualitative inputs or reasoning in our decision-making. It is to say we think quantifying our uncertainties whenever possible generally helps us to make better decisions. The difficulty of accounting for all of the above issues are typically made worse, not better, when precise quantitative statements of beliefs or inputs are replaced by softer qualitative judgments. We think the work in the CURVE sequence has further bolstered this case that even when you can’t be precise in your estimates, quantifying your uncertainty can still significantly improve your ability to reason carefully.
Transparent Decision-Making
Knowing how to do good was hard enough before we introduced alternative decision theories. Still, RP has to make choices about how to distribute its resources, navigating deep uncertainty and, sometimes, differing perspectives among our leadership and staff. Since we want to make our choices sensitive to our evidential situation and transparent within the organization, we’re committed to finding a decision-making strategy that allows us to navigate this uncertainty in a principled manner. Thankfully, there are a wide range of deliberative decision-making processes, such as Delphi panels and citizen juries, that are available for just such purposes. Moreover, there are a number of formal and informal methods of judgment aggregation that can be used at the end of the deliberative efforts.
We aren’t yet sure which of these particular decision procedures we’ll use and we expect creating such a process and executing it to take time. All of these procedures have drawbacks in particular contexts and we don’t expect any such procedure to be able to handle all the specific decisions that RP faces. However, we’re confident that a clearly defined decision procedure that forces us to be explicit about the tradeoffs we’re making and why is superior to unilateral and intuition-based decision-making. We want to incorporate the best judgment of the leaders in our organization and own the intra- and inter-cause comparisons on which our decisions are based. So, we’re in the process of setting up such decision procedures and will report back what we can about how they’re operating.
Conclusion
We want to do good. The uncertainties involved in doing good are daunting, particularly given we are trying to take an impartial, scope sensitive, open to revision approach. However, RP aims to be a model of how to handle uncertainty well. In part, of course, this requires trying to reduce our uncertainty. But lately, we’ve been struck by how much it requires recognizing the depth of our uncertainty—all the way to the very frameworks we use for decision-making under uncertainty. We are trying to take this depth seriously without becoming paralyzed—which explains why we’re doubling down on modeling and collective decision-making procedures.
In practice, we suspect that a good rule of thumb is to spread our bets across our options. Essentially, we think we’ve entered a dizzying casino where the house won’t even tell us the rules of the game. And even if we knew the rules, we’d face a host of other uncertainties: the long-term payouts of various options, the risk of being penalized if we choose incorrectly among various courses of action, and a host of completely inscrutable possibilities where we have no idea what to think of them. In a situation of this type, it seems like a mistake to assume that one ruleset is correct and proceed accordingly. Instead, we want to find robustly good options among different plausible rulesets whenever we can. And when we can’t, we may want to distribute our resources in proportion to different reasonable approaches to prioritization.
This isn’t perfect or unobjectionable. But nothing is. RP will continue to do its best to make these decisions as transparently as we can, learning from our mistakes and continuing to try to advance the cause of improving the world.
Acknowledgements
The piece was written by Marcus A. Davis and Peter Wildeford. Thanks to David Moss, Abraham Rowe, Janique Behman, Carolyn Footitt, Hayley Clatterbuck, David Rhys Bernard, Cristina Schmidt Ibáñez, Jacob Peacock, Aisling Leow, Renan Araujo, Daniela R. Waldhorn, Onni Aarne, Melissa Guzikowski, and Kieran Greig for feedback. A special thanks to Bob Fischer for writing a draft of this post. The post is a project of Rethink Priorities, a global priority think-and-do tank, aiming to do good at scale. We research and implement pressing opportunities to make the world better. We act upon these opportunities by developing and implementing strategies, projects, and solutions to key issues. We do this work in close partnership with foundations and impact-focused non-profits or other entities. If you're interested in Rethink Priorities' work, please consider subscribing to our newsletter. You can explore our completed public work here.
I have increasingly become open to incorporating alternative decision theories as I recognize that I cannot be entirely certain in expected value approaches, which means that (per expected value!) I probably should not solely rely on one approach. At the same time, I am still not convinced that there is a clear, good alternative, and I also repeatedly find that the arguments against using EV are not compelling (e.g., due to ignoring more sophisticated ways of applying EV).
Having grappled with the problem of EV-fanaticism for a long time in part due to the wild norms of competitive policy debate (e.g., here, here, and here), I've thought a lot about this, and I've written many comments on the forum about this. My expectation is that this comment won't gain sufficient attention/interest to warrant me going through and collecting all of those instances, but my short summary is something like:
I dislike using "objective" and "subjective" to make these distinctions, but for simplicity's sake / for lack of a better alternative at the moment, I will use them.
For more on this thread, see here: https://forum.effectivealtruism.org/posts/WSqLHsuNGoveGXhgz/disentangling-some-important-forecasting-concepts-terms
I advocate for something like this competitive policy debate, since "fighting EV fire with EV fire" risks "burning the discussion"—including the educational value, reputation of participants, etc. But most deliberations do not have to be made within the artificial constraints of competitive policy debate.