exploring sensitivity to how answers would look if the deworming RCT results had been higher or lower and that they change sensibly?
Do you just mean that the change in the posterior expectation is in the correct direction? In that case, we know the answer from theory: yes, for any prior and a wide range of likelihood functions.
Andrews et al. 1972 (Lemma 1) shows that when the signal B is normally distributed, with mean T, then, for any prior distribution over T, E[T|B=b] is increasing in b.
This was generalised by Ma 1999 (Corollary 1.3) to any likelihood function arising from a B that (i) has T as a location parameter, and (ii) is strongly unimodally distributed.
A lognormal prior (and a normal likelihood function) might be a good starting point when adjusting for the statistical uncertainty in an effect size estimate. The resulting posterior cannot be calculated in closed form, but I have a simple website that calculates it using numerical methods. Here's an example.
Worth noting that adjusting for the statistical uncertainty in an effect size estimate is quite different from adjusting for the totality of our uncertainty in a cost-effectiveness estimate. For doing the latter, it's unclear to me what likelihood function would be appropriate. I'd love to know if there are practical methods for choosing the likelihood function in these cases.
GiveWell does seem to be using mostly normal priors in the document you linked. I don't have time to read the whole document and think carefully about what prior would be most appropriate. For its length (3,600 words including footnotes) the document doesn't appear to give much reasoning for the choices of distribution families.
Do you just mean that the change in the posterior expectation is in the correct direction? In that case, we know the answer from theory: yes, for any prior and a wide range of likelihood functions.
Andrews et al. 1972 (Lemma 1) shows that when the signal
Bis normally distributed, with meanT, then, for any prior distribution overT,E[T|B=b]is increasing inb.This was generalised by Ma 1999 (Corollary 1.3) to any likelihood function arising from a
Bthat (i) hasTas a location parameter, and (ii) is strongly unimodally distributed.A lognormal prior (and a normal likelihood function) might be a good starting point when adjusting for the statistical uncertainty in an effect size estimate. The resulting posterior cannot be calculated in closed form, but I have a simple website that calculates it using numerical methods. Here's an example.
Worth noting that adjusting for the statistical uncertainty in an effect size estimate is quite different from adjusting for the totality of our uncertainty in a cost-effectiveness estimate. For doing the latter, it's unclear to me what likelihood function would be appropriate. I'd love to know if there are practical methods for choosing the likelihood function in these cases.
GiveWell does seem to be using mostly normal priors in the document you linked. I don't have time to read the whole document and think carefully about what prior would be most appropriate. For its length (3,600 words including footnotes) the document doesn't appear to give much reasoning for the choices of distribution families.
I've fully rewritten the spreadsheet tool, available at carlo.app:
The major improvements are:
Other improvements include: