I'm feeling confused.

I basically agree with this entire post. Over many years of conversations with Givewell staff or former staff, I can't readily recall speaking to anyone affiliated with Givewell who I can identify that they would substantively disagree with the suggestions in this post. But you obviously feel that some (reasonably large?) group of people disagrees with some (reasonably large?) part of your post. I understand a reluctance to give names, but focusing on Givewell specifically as much of their thoughts on these matters are public record here, can you identify what specifically in that post or the linked extra reading you disagree with? Or are you talking to EAs-not-at-Givewell? Or do you think Givewell's blog posts are reasonable but their internal decision-making process nonetheless commits the errors they warn against? Or some possibility I'm not considering?

I particularly note that your first suggestion to 'entertain multiple models' sounds extremely similar to 'cluster thinking' as described and advocated-for here, and the other suggestions also don't sound like things I would expect Givewell to disagree with. This leaves me at a bit of a loss as to what you would like to see change, and how you would like to see it change.

I haven't had time yet to think about your specific claims, but I'm glad to see attention for this issue. Thank you to contributing what I suspect is an important discussion!

You might be interested in the following paper which essentially shows that under an additional assumption the Optimizer's Curse not only makes us overestimate the value of the apparent top option but in fact can make us predictably choose the wrong option.

Denrell, J. and Liu, C., 2012. Top performers are not the most impressive when extreme performance indicates unreliability. Proceedings of the National Academy of Sciences, 109(24), pp.9331-9336.

The crucial assumption roughly is that the reliability of our assessments varies sufficiently much between options. Intuitively, I'm concerned that this might apply when EAs consider interventions across different cause areas: e.g., our uncertainty about the value of AI safety research is much larger than our uncertainty about the short-term benefits of unconditional cash transfers.

(See also the part on the Optimizer's Curse and endnote [6] on Denrell and Liu (2012) in this post by me, though I suspect it won't teach you anything new.)

Can you give an example of a time when you believe that the EA community got the wrong answer to an important question as a result of not following your advice here, and how we could have gotten the right answer by following it?

There's actually a thing called the Satisficer's Curse (pdf) which is even more general:

The Satisficer’s Curse is a systematic overvaluation that occurs when any uncertain prospect is chosen because its estimate exceeds a positive threshold. It is the most general version of the three curses, all of which can be seen as statistical artefacts.

I believe effective altruists should find this extremely concerning.

Well it does not change the ordering of options. You're kind of doing a wrong-way reduction here: you're taking the question of what project should I support and "reducing" it to literal quantitative estimation of effectiveness. Optimizer's curse only matters when comparing better-understood projects to worse-understood projects, but you are talking about "prioritizing among funding opportunities that involve substantial, poorly understood uncertainty".

In most scenarios where effective altruists encounter the optimizer’s curse, this solution is unworkable. The necessary data doesn’t exist.

We can specify a prior distribution.

I don’t think ignorance must cash out as a probability distribution. I don’t have to use probabilistic decision theory to decide how to act.

Well no, but it's better if you do. That Deutsch quote seems to say that it could allow people to take bad reasons and overstate them; that sounds like a problem with thinking in general. And there is no reason to assume that probabilistic decision makers will overestimate as opposed to underestimate. There have been many times when I had a vague, scarce prejudice/suspicion based on personal ignorance, and deeper analysis of reliable sources showed that I was correct and underconfident. If you think your vague suspicions aren't useful, then just don't trust them! Every system of thinking is going to bottom out in "be rational, don't be irrational" at some point, so this is not a problem with probabilism in particular.

The reason it's better is that it allows better rigor and accuracy. For instance, look how this post revolves around the optimizer's curse. Here's a question: how are you going to adjust for the optimizer's curse if you don't use probability (implicitly or explicitly)? And if people weren't using probabilistic decision theory, no one would have discovered the optimizer's curse in the first place!

Kyle is an atheist. When asked what odds he places on the possibility that an all-powerful god exists, he says “2%.”

Hey! I didn't consent to being included in your post!!!

Here's what it means, formally: given that I have an equal desire to be right about the existence of God and the nonexistence of God, and given some basic assumptions about my money and my desire for money, I would make a bet with at most 50:1 odds that all-powerful-God exists.

Placing hazy probabilities on the same footing as better-grounded probabilities (e.g., the odds a coin comes up heads) can lead to problems.

But in Bayesian decision theory, they aren't on the same footing. They have very different levels of robustness. They are not well-grounded and this matters for how readily we update away from them. Is the notion of robustness inadequate for solving some problem here? In the Norton paper that you cite later on this point, I ctrl-F for "robust" and find nothing.

When discussing these ideas with members of the effective altruism community, I felt that people wanted me to propose a formulaic solution—some way to explicitly adjust expected value estimates that would restore the integrity of the usual prioritization methods. I don’t have any suggestions of that sort.

All of your suggestions make perfect sense under standard, Bayesian probability and decision theory. As stated, they are kind of platitudinous. Moreover, it's not clear to me that abandoning these principles in favor of a some deeper concept of ignorance actually helps motivate any of your recommendations. Why, exactly, is it important that I embrace model skepticism for instance - just because I have decided to abandon probabilities? Does abandoning probabilities reduce the variance in the usefulness of different models? It can't, actually, because without probabilities the variance is going to be undefined.

In practice, I haven't done things with multiple quantitative models because (a) models are tough to build, and (b) a good model accommodates all kinds of uncertainty anyway. It's never been the case where I've found some new information/ideas, decided to update my model, and then realized "uh oh, I can't do this in this model." I can always just add new calculations for the new considerations, and it becomes a bit kludgy but still seems more accurate. So yeah this is good in theory but the practical value seems very limited. To be sure, I haven't really tried it yet.

If we want to test the accuracy of a model, we need to test a statistically significant number of the things predicted by the model. It's not sufficient for us to donate to AMF, see that AMF seems to work pretty well (or not), and then judge Givewell accordingly. We need to see whether Givewell's ordering of multiple charities holds.

Testing works well in some contexts. In others it's just unrealistic.

Improving social capacity tends to work better when society is trusted to actually do the right thing.

In my experience, effective altruists are unusually skeptical of conventional wisdom, tradition, intuition, and similar concepts.

But these are exactly the things that you are objecting to. Where do you think probability estimates of deeply uncertain things come from? If there's some disagreement here about the actual reliability of things like intuition and tradition, it hasn't been made explicit. Instead, you've just said that such things should not be expressed in the form of quantitative probabilities.

Perhaps a related phenomenon is that "adding maximal value on the margin" can look a lot like "defecting from longterm alliances & relationships" when viewed from a different framing?

Most clearly when ongoing support to longterm allies is no longer as leveraged as marginal support towards a new effort.

Late to the party, but I was re-reading this as it relates to another post I'm working on, and I realised I have a question. You write: (note that I say "you" in this comment a lot, but I'd also be interested in anyone else's thoughts on my questions)

The optimizer’s curse can show up even in situations where effective altruists’ prioritization decisions don’t involve formal models or explicit estimates of expected value. Someone informally assessing philanthropic opportunities in a linear manner might have a thought like:

Thing X seems like an awfully big issue. Funding Group A would probably cost only a little bit of money and have a small chance leading to a solution for Thing X. Accordingly, I feel decent about the expected cost-effectiveness of funding Group A.

Let me compare that to how I feel about some other funding opportunities…

Although the thinking is informal, there’s uncertainty, potential for bias, and an optimization-like process.

That makes sense to me, and seems a very worthwhile point. (It actually seems to me it might have been worth emphasising more, as I think a casual reader could think this post was a critique of formal/explicit/quantitative models in particular.)

But then in a footnote, you add:

Informal thinking isn’t always this linear. If the informal thinking considers an opportunity from multiple perspectives, draws on intuitions, etc., the risk of postdecision surprise may be reduced.

I'm not sure I understand what you mean by that, or if it's true/makes sense. It seems to me that, ultimately, if we're engaging in a process that effectively provides a ranking of how good the options seem (whether based on cost-effectiveness estimates or just how we "feel" about them), and there's uncertainty involved, and we pick the option that seems to come out on top, the optimizer's curse will be relevant. Even if we use multiple separate informal ways of looking at the problem, we still ultimately end up with a top ranked option, and, given that that option's ended up on top, we should still expect that errors have inflated its apparent value (whether that's in numerical terms or in terms of how we feel) more than average. Right?

Or did you simply mean that using multiple perspectives means that the various different errors and uncertainties might be more likely to balance out (in the same sort of way that converging lines of evidence based on different methodologies make us more confident that we've really found something real), and that, given that there'd effectively be less uncertainty, the significance of the optimizer's curse would be smaller. (This seems to fit with "the risk of postdecision surprise may be reduced".)

If that's what you meant, that seems reasonable to me, but it seems that we could get the same sort of benefits just by doing something like gathering more data or improving our formal models. (Though of course that may often be more expensive and difficult than cluster thinking, so highlighting that we also have the option of cluster thinking does seem useful.)

Do you have any thoughts on Tetlock's work which recommends the use of probabilistic reasoning and breaking questions down to make accurate forecasts?

Thanks for this!

I’ve come to believe there are serious, underappreciated issues with the methods the effective altruism (EA) community at large uses to prioritize causes and programs.

Is there a tl;dr of these issues?

I’m going to try to clarify further why I think the Bayesian solution in the original paper on the Optimizer’s Curse is inadequate.

The Optimizer's Curse is defined by Proposition 1: informally, the expectation of the estimated value of your chosen intervention overestimates the expectation of its true value when you select the intervention with the maximum estimate.

The proposed solution is to instead maximize the posterior expected value of the variable being estimated (conditional on your estimates, the data, etc.), with a prior distribution for this variable, and this is purported to be justified by Proposition 2.

However, Proposition 2 holds no matter which priors and models you use; there are no restrictions at all in its statement (or proof). It doesn’t actually tell you that your posterior distributions will tend to better predict values you will later measure in the real world (e.g. by checking if they fall in your 95% credence intervals), because there need not be any connection between your models or priors and the real world. It only tells you that your maximum posterior EV equals your corresponding prior’s EV (taking both conditional on the data, or neither, although the posterior EV is already conditional on the data).

Something I would still call an “optimizer’s curse” can remain even with this solution when we are concerned with the values of future measurements rather than just the expected values of our posterior distributions based on our subjective priors. I’ll give 4 examples, the first just to illustrate, and the other 3 real-world examples:

1. Suppose you have $n$ different fair coins, but you aren’t 100% sure they’re all fair, so you have a prior distribution over the future frequency of heads (it could be symmetric in heads and tails, so the expected value would be $1/2$ for each), and you use the same prior for each coin. You want to choose the coin which has the maximum future frequency of landing heads, based on information about the results of finitely many new coin flips from each coin. If you select the one with the maximum expected posterior, and repeat this trial many times (flip each coin multiple times, select the one with the max posterior EV, and then repeat), you will tend to find the posterior EV of your chosen coin to be greater than $1/2$, but since the coins are actually fair, your estimate will be too high more than half of the time on average. I would still call this an “optimizer’s curse”, even though it followed the recommendations of the original paper. Of course, in this scenario, it doesn’t matter which coin is chosen.

Now, suppose all the coins are as before except for one which is actually biased towards heads, and you have a prior for it which will give a lower posterior EV conditional on $k$ heads and no tails than the other coins would (e.g. you’ve flipped it many times before with particular results to achieve this; or maybe you already know its bias with certainty). You will record the results of $k$ coin flips for each coin. With enough coins, and depending on the actual probabilities involved, you could be less likely to select the biased coin (on average, over repeated trials) based on maximum posterior EV than by choosing a coin randomly; you'll do worse than chance.

(Math to demonstrate the possibility of the posteriors working this way for $k$ heads out of $k$: you could have a uniform prior on the true future long-run average frequency of heads for the unbiased coins, i.e. $p\left({\mu }_i\right)=1$ for ${\mu }_i$ in the interval $[0,1]$, then $p\left({\mu }_i|kheads\right)=(k+1){\mu }_i^k$, and $E\left[{\mu }_i|kheads\right]=(k+1)/(k+2)$, which goes to $1$ as $k$ goes to infinity. You could have a prior which gives certainty to your biased coin having any true average frequency $<1$, so any of the unbiased coins which lands heads $k$ out of $k$ times will beat it for $k$ large enough.)

If you flip each coin $k$ times, there’s a number of coins, $n$, so that the true probability (not your modelled probability) of at least one of the $n-1$ other coins getting k heads is strictly greater than $1-1/n$, i.e. $1-(1-1/2^k{)}^{n-1}>1-1/n$ (for $k=2$, you need $n>8$, and for $k=10$, you need $n>9360$, so $n$ grows pretty fast as a function of $k$). This means, with probability strictly greater than $1-1/n$, you won’t select the biased coin, so with probability strictly less than $1/n$, you will select the biased coin. So, you actually do worse than random choice, because of how many different coins you have and how likely one of them is to get very lucky. You would have even been better off on average ignoring all of the new $k\times n$ coin flips and sticking to your priors, if you already suspected the biased coin was better (if you had a prior with mean $>1/2$).

2. A common practice in machine learning is to select the model with the greatest accuracy on a validation set among multiple candidates. Suppose that the validation and test sets are a random split of a common dataset for each problem. You will find that under repeated trials (not necessarily identical; they could be over different datasets/problems, with different models) that by choosing the model with the greatest validation accuracy, this value will tend to be greater than its accuracy on the test set. If you build enough models each trial, you might find the models you select are actually overfitting to the validation set (memorizing it), sometimes to the point that the models with highest validation accuracy will tend to have worse test accuracy than models with validation accuracy in a lower interval. This depends on the particular dataset and machine learning models being used. Part of this problem is just that we aren’t accounting for the possibility of overfitting in our model of the accuracies, but fixing this on its own wouldn’t solve the extra bias introduced by having more models to choose from.

3. Due to the related satisficer’s curse, when doing multiple hypothesis tests, you should adjust your p-values upward or your p-value cutoffs (false positive rate, significance level threshold) downward in specific ways to better predict replicability. There are corrections for the cutoff that account for the number of tests being performed, a simple one is that if you want a false positive rate of $\alpha$, and you’re doing $m$ tests, you could instead use a cutoff of $1-(1-\alpha {)}^m$.

4. The satisficer’s curse also guarantees that empirical study publication based on p-value cutoffs will cause published studies to replicate less often than their p-values alone would suggest. I think this is basically the same problem as 3.

Now, if you treat your priors as posteriors that are conditional on a sample of random observations and arguments you’ve been exposed to or thought of yourself, you’d similarly find a bias towards interventions with “lucky” observations and arguments. For the intervention you do select compared to an intervention chosen at random, you’re more likely to have been convinced by poor arguments that support it and less likely to have seen good arguments against it, regardless of the intervention’s actual merits, and this bias increases the more interventions you consider. The solution supported by Proposition 2 doesn’t correct for the number of interventions under consideration.