Edit: If I were to re-write this post, I would re-frame it as a list of reasons for why we should lower our confidence in Bayesian estimates rather than as an attack on Bayesianism itself.
This is a crosspost from my blog post.
Many EAs use extensive “bayesian reasoning.” The basic idea behind it is that you should do the following:
- Be willing to assign probabilities to the likelihood of anything occurring or being true.
- Update these probabilities when you learn new information.
- Act on these probabilities if they suggest that certain actions are higher in value than all other actions.
Bayesian reasoning is helpful for a lot of everyday decision-making. If you’re trying to figure out whether to take a job in Los Angeles or in New York, it makes sense to try to guess how happy you’d be in each respective city. And, if you’re trying to compare career paths, you should assign probabilities to how likely you are to succeed in them.
But, to me, EAs take this kind of reasoning too far. EAs have variously tried to predict how many future humans there will be, asked non-domain experts how likely they think a catastrophic pandemic will be, and even tried to determine whether we should work on improving the lives of people in the far future.
Probably the most common (and representative) example of this, though, is the idea of AI timelines, which means trying to predict when “AGI” will be developed. Given that we have no clue what it actually takes to develop AGI or how many breakthroughs are required, it seems pretty unreasonable to me to assign an exact year to when such an outcome would occur.
But, setting that aside, these are my five reasons EAs should use less Bayesian reasoning.
Reason #1: If you know very little about something, your guesses are completely arbitrary.
Have you ever asked a child how far away they think New York is and then heard them guess “a hundred miles?” This child’s guess is inaccurate because they don’t know very much about the relative distances of places around the world.
I think this same line of reasoning can be applied to a lot of EA’s predictions about the future. For instance, a lot of EAs make guesses about the likelihood of AI takeover, but I don't think we have enough knowledge to know when we'll develop AGI, whether we’ll develop AGI, or what the development of AGI will look like. So, given this, it seems pretty unlikely that we could come up with any kind of accurate prediction about how likely AI takeover is.
Reason #2: We're bad at making guesses in general.
Some people have undergone extensive training to be good at making forecasts over short durations of time in situations where trends can be roughly extrapolated forwards.
Most people have not undergone this training, and are, in fact, pretty bad at making guesses in general.
Most people are bad at predicting how successful they will be, who will win elections, and how long it will take for them to finish projects.
Given this, it seems like we should expect our predictions on most things to be off by a reasonable extent.
Reason #3: If your guesses are completely arbitrary, updating won’t bring you to the correct probability.
Bayesians like to say that, if you have no clue how likely something is, you should just pick a random number and then update your beliefs from there. The problem, though, is that, if you’re updating relative to an arbitrary number, your arbitrary number might hold too much weight
For instance, if you think there’s a 10% chance that a pandemic this century will kill more than a billion people, you might be only willing to update by a single order of magnitude each time you learn new information.
Given this, if you learned that the Chinese government has decided to stop stockpiling masks, you might reduce your probability to 1%, but, if the real probability were 10^-7, you would need an overwhelming amount of information to update your beliefs to the correct probability.
Reason #4: We should expect most guesses about the future to be wrong.
People are generally familiar with the idea that we’re bad at predicting the future, but I think they fail to take seriously how significant of an issue this is.
The fact is that history has been determined by an extraordinarily complex interaction of social, political, environmental, economic, and circumstantial factors. And, as a result, historically, people were very bad at predicting the future. If you had someone in 1910 try to make predictions about how the century would go, they would probably be wrong in a vast myriad of ways. They likely wouldn’t have predicted two world wars and a cold war. They wouldn’t have guessed that we’d discover the existence of galaxies. And, they wouldn’t have been able to tell you that we’d become completely digitally interconnected. Given this, I think we should also consider our own predictions about the future to likely be very wrong.
Reason #5: If your guesses are based on other people’s guesses, you might all be wrong.
Humans experience an anchoring bias when it comes to making predictions, so if we hear someone make a prediction, we usually make ours relative to theirs. The problem with this is that, if one person makes a very prominent prediction that is completely off, everyone will be basing their prediction on that bad prediction.
I think this is particularly concerning in domains where only a few individuals have prominence, but there's very little information to go off of. If everyone is assuming those individuals know more than they do, then everyone will have very biased guesses.
Whenever I ask someone how likely they think something is, they pretty much never give a probability less than .1% unless that something is religious in nature. Given this, it seems like people systematically over estimate low probabilities because they fail to consider probabilities such as 10^-7 or 10^-53.
The post title somewhat confuses me since reasons 2, 4, and 5 (miscalibration, anchoring cascades, and granularity failure at the tails) are in-paradigm critiques.
I'm also confused by the link to the XPT tournament as if it illustrates the assertion that EAs take Bayesian reasoning too far, given it did in fact include domain experts and given its headline finding that the forecasts were discrepant between the groups and failed to converge after structured persuasion. Same confusion re: linked GPI paper.
I do think many of your points are correct. The strongest argument to your post title I can think of is sparse evidence + multiple models and you're uncertain between them -> precise priors are unwarranted -> use an interval instead, possibly quite wide like [10⁻⁶, 0.5] (just to make something up) -> updating may not collapse this wide interval -> so EV-maxxing becomes undefined -> so switch to other decision criteria, e.g. maybe robustness to harms, which is less Bayesian as per your post title -> choose robustly good actions, maintain option value, build capacity etc. Which is basically what most meta interventions are about, no?
Unfortunately I don't think there's really much of a case for "maintain option value, build capacity etc." being robustly good either, as argued here.
Thanks Anthony. Would it be fair to interpret your unawareness series as your steelman of OP's post title, or as being relevant?
I couldn't find on a quick look what decision-making approach you would (at least provisionally) endorse instead, bracketing maybe? For my own reference later:
I would be particularly interested in how you think meta- and/or longtermist-oriented grantmaking could be improved by your work. I have not been very impressed by the reasoning behind some of these (sometimes quite large) grants, at least on the rare occasions they've been shared publicly.
Bayesian inference and decision making are somewhat distinct steps.
EV maxxing is a specific (and specifically simple) objective function that you can plug Bayesian estimates into for decision making. It removes the need to think about distributions.
I think it can be helpful to separate out the problems - eg I believe that there aren’t really any plausible alternatives to Bayesian inference (over future trajectories of the world as a function of your actions or similar) but I think there’s much more room for debate regarding the objective function.
Thanks for the corrections. Agree there's plenty of room for debate re: objective function, I generally think people don't take this seriously enough (Nuno Sempere's estimating value series is what I have in mind by "take it seriously").
Richard Ngo's Towards a Formal Scientific Epistemology sketches out the beginnings of an alternative to Bayesian epistemology in case you're interested. I am not the right person to field questions about it unfortunately, I'll just quote his intro:
I interpret his follow-up essay Agents as webs of beliefs: Unifying beliefs, goals, and actions as sketching out the bigger picture.
People in EA should definitely read more Feyerabend! (Or ask llms what Feyerabend would say about a topic etc).
For example “a complete theory of scientific epistemology” is something he’d most likely reject even as an ideal.
Hi Mo.
This argument works with sharp probabilities too? If the distributions for the cost-effectiveness are very wide, the expected cost-effectiveness of decreasing uncertainty or building capacity would tend to be higher than the highest expected cost-effectiveness of the interventions under evaluation, even if these are all sharp values?
I do not want to give up completeness because it follows from 3 super intuitive premises.
I was making in-paradigm critiques, since, if something is internally invalid, it gives us good reason to think it is also externally invalid.
I linked to the XPT tournament just as an example of asking non-experts for predictions. I thought that pandemics were a good example since most people don't know much about pandemics at all so we should expect their guesses to be very off. I could see the argument that forecasting research has found forecasters to be better than experts so it makes sense to ask non-experts, but it's important to note that we haven't validated this over very long time periods.
I linked to the GPI paper since it seems to pretty absurd to me to try to guess how many future there will be. Even if we can come up with accurate estimates for carrying capacity for different regions of space, I have no clue how we could predict the likelihood of reaching carrying capacity in these regions of space.
In regards to your last point, I think my view is most basically that, by assigning probabilities to outcomes, we're giving ourselves excessive confidence when we often have too little knowledge to warrant the confidence, that correct decision theories should take into account our cluelessness to a much greater extent, and that people probably do Bayesian reasoning a lot worse than they think.
That said, I'm still figuring out my views on decisions theories so I probably should have researched this topic a lot more before making a post.
For your last point, I agree with you that we run into the problem of EV-maxxing being undefined, but I don't know where to go from there. It doesn't seem to make sense to me to do anything in regards to something with such a wide probability range because it seems like you'll just spend all your time chasing things that you know very little about but which suggest really high EV.
Thanks for the thoughtful comment.