Expected value estimates you can take (somewhat) literally

the difficulty in estimating some of the parameters makes me wary of applying it as is.

I agree that these expected-value estimates shouldn't be taken (even somewhat) literally. But I think toy models like this one can still be important for checking the internal consistency of one's reasoning. That is: if you can create a model that says X, this doesn't mean you should treat X as true; but if you can't create a reasonable model that says X, this is pretty strong evidence that X isn't true.

In this case, the utility would be in allowing you to inspect the sometimes-unintuitive interplay between your guesses at an estimate's R^2, the distributional parameters, and the amount of regression. While you shouldn't plug in guesses at the parameters and expect the result to be correct, you can still use such a model to constrain the parameter space you want to think about.

2) We need to know the means of the distribution to do the standardization - after all, if an intervention was estimated to be below the mean, we should anticipate it to regress upwards.

Trickier for an EA context, as the groups that do evaluation focus their efforts on what appear to be the most promising things, so there isn't a clear handle on the 'mean global health intervention' which may be our distribution of interest. To some extent though, this problem solves itself if the underlying distributions of interest are log normal or similarly fat tailed and you are confident your estimate lies far from the mean (whatever it is): log (X - something small) approximates to log(X)

Sadly I don't think a log-normal distribution solves this problem for you, because to apply your model I think you are working entirely in the log-domain, so taking log(X) - log(something small), rather than log(X - something small). Then the choice of the small thing can have quite an effect on the answer.

For example when you regressed the estimate of cost-effectiveness of malaria nets, you had an implicit mean cost-effectiveness of 1 DALY/$100,000. If you'd assumed instead 1 DALY/$10,000, you'd have regressed to $77/DALY instead of $97/DALY.

I noticed something else: you may have lost track of expectations when translating between log-normal and normal.

The log of the median of a log-normal distribution is the same as the median of the normal distribution which you get by taking logs, and the same as the mean of that latter distribution. But the log of the mean of the log-normal distribution will be higher.

This affects what happens with your regressions. Assuming the initial estimates are point estimates, you end up with a distribution of possible values after regressing. With normal distributions, everything behaves nicely, the mean is equal to the median, and your calculations are correct.

With log-normal distributions, we normally want to make decisions based on expected value, which will be higher than the median I think you've produced. In particular, the log-normal will be wider (and the mean:median ratio higher) when there is less correlation between true values and estimates. This looks like it might reduce the importance of the r-value relative to your calculations, at least if you do care about expectations, and at the upper end of the range.

I'm not sure how this all cashes out. But wanted to flag it as something else which may significantly affect the answer!

Couple of other comments:

• I'm not sold on estimating r^2 just by group (this is a totally outside view on error). I think you can often say something quite sensible about the relative strengths of different estimates you make for yourself. There should be a way to incorporate such information, and I'm not sure what it is.

• It looks like it might need more thought on how to generalise to distributions which aren't normal or log-normal. Generally I think we should expect tails to be a little thicker than log-normal; this should have the effect of reducing the impact of regression, but I don't off-hand see how to follow this through quantitatively.

1) The variances of the two distributions need to be standardized.

Less of a big deal, we we'd generally hope for and aim that our estimates of expected value are drawn from a similar distribution to the actual expected values - if it's not, our estimates are systemically wrong somehow.

Our estimates could be systematically wrong in that they represent, for instance, before-regression estimates. We don't even know the true distribution, and the generating mechanism for the estimates is sufficiently different that I wouldn't feel confident that the variances should look similar.

So, in our toy model with an R-square of 0.9, an estimate which is 1SD above the mean estimate puts the expected value at 0.9SD above the mean expected value.

I think there's a confusion here about what the different distributions are. Normally when thinking about regression I think of having a prior over cost-effectiveness of the intervention at hand, and a distribution representing model uncertainty, which tells you the likelihood of having got your model output, given varying true values for the parameter. If those are the distributions you have, and they're both normal with the same variance, then regression would end up centring on the mid-point (and so be linear with the number of SDs up you are).

I think that the distributions you are looking at, however, are the prior, and a prior over the distribution of the estimate numbers. Given this, the amount of regression is not linear with the number of standard deviations out. I think rather it goes up super-linearly.

Working with the perspective you are bringing in terms of the different distributions could be a useful angle on the problem. It's not obvious to me it's better than the normal approach to regression, though, mostly because it seems harder to give an inside view of correlation than of possible model error.

which have dropped by an order of magnitude every couple of years

This seems like a really important point, and I wonder if anyone has blogged on this particular topic yet. In particular:

• How should we expect this trend to continue?
• Does it increase the activation energy for getting involved in EA? (my interest in EA was first aroused by GW and how cheap they reckoned it was to save a life via VillageReach)
• Does it affect the claim that a minority of charities are orders of magnitude more effective than the rest?
• If we become able to put numbers to the effectiveness of a new area, such as xrisk or meta, would we expect to see the same exponential drop-off in our estimates even if we're aware of this problem?

I've now made that spreadsheet viewable by anyone with the link. If you want to use it you should save a copy you can edit.

It doesn't estimate regression as such; rather it's for keeping track of the inside-view uncertainty about model parameters, in a way which is quick and easy but roughly correct. This gets you to something like a total inside view of the error inherent in the model; if you also have a prior over effectiveness you could then use it to regress.

I talk more about the details in this post. The spreadsheet was made mostly for my own use when I want to make quick estimates with error-bars on log-normal distributions; I've tried to make it slighly user-friendly but I'm afraid it's not terribly so.