Learning from our mistakes: how HLI plans to improve

Hi folks, in this post we’d like to describe our views as the Chair (Peter) and Director (Michael) of HLI in light of the recent conversations around HLI’s work. The purpose of this post is to reflect on HLI’s work and its role within the EA community in response to community member feedback, highlight what we’re doing about it, and engage in further constructive dialogue on how HLI can improve moving forward. 

HLI hasn’t always got things right. Indeed, we think there have been some noteworthy errors (quick note: our goal here isn’t to delve into details but to highlight broad lessons learnt, so this isn’t an exhaustive list):

• Most importantly, we were overconfident and defensive in communication, particularly around our 2022 giving season post. 
  • We described our recommendation for StrongMinds using language that was too strong: “We’re now in a position to confidently recommend StrongMinds as the most effective way we know of to help other people with your money”. We agree with feedback that this level of confidence was and is not commensurate with the strength of the evidence and the depth of our analysis. 
  • The post’s original title was “Don’t give well, give WELLBYs”. Though this was intended in a playful manner, it was tone-deaf, and we apologise.
• We made mistakes in our analysis.
  • We made a data entry error. In our meta-analysis, we recorded that Kemp et al. (2009) found a positive effect, but in fact it was a negative effect. This correction reduced our estimated ‘spillover effect’ for psychotherapy (the effect that someone receiving an intervention had on other people) from 53% to 38%  and therefore reduced the total cost-effectiveness estimate from 9.5x cash transfers to 7.5x.
  • We did not include standard diagnostic tests of publication bias. If we had done this, we would have decreased our confidence in the quality of the literature on psychotherapy that we were using. 
• After receiving feedback about necessary corrections to our cost-effectiveness estimates for psychotherapy and StrongMinds, we failed to update our materials on our website in a timely manner.

As a community, EA prides itself on its commitment to epistemic rigour, and we’re both grateful and glad that folks will speak up to maintain high standards. We have heard these constructive critiques, and we are making changes in response. 

We’d like to give a short outline of what HLI is doing next and has done in order to improve its epistemic health and comms processes.

1. We’ve added an “Our Blunders” page on the HLI website, which lists the errors and missteps we mentioned above. The goal of this page is to be transparent about our mistakes, and to keep us accountable to making improvements.
2. We’ve added the following text to the places in our website where we discuss StrongMinds: 
   • “Our current estimation for StrongMinds is that a donation of $1,000 produces 62 WELLBYs (or 7.5 times GiveDirectly cash transfers). See our changelog. 
     However, we have been working on an update to our analysis since July 2023 and expect to be ready by the end of 2023. This will include using new data and improving our methods. We expect our cost-effectiveness estimate will decrease by about 25% or more – although this is a prediction we are very uncertain about as the analysis is yet to be done. 
     While we expect the cost-effectiveness of StrongMinds will decrease, we think it is unlikely that the cost-effectiveness will be lower than GiveDirectly. Donors may want to wait to make funding decisions until the updated report is finished.” 
3. We have added more/higher quality controls to our work:
   • Since the initial StrongMinds report, we’ve added Samuel Dupret (researcher) and Dr Ryan Dwyer (senior researcher) to the team, providing more quantitative eyes to double-check all key research inputs and reproduce all key research outputs.
   • We will be more careful about communicating our work, including greater emphasis on the uncertainties in, and the limitations of, our analyses. 
   • In line with this, we’ve developed our process for how to classify the quality of evidence and the depth of our work in a more principled and transparent manner. This will be posted on our website in Q4 2023.
   • We have just added a changelog to the site to keep a public record of how our CEAs have changed over time and why.
   • We have sought advice from leading experts to develop our methods for handling publication bias. We will use these methods in our updated evaluation of psychotherapy, scheduled for Q4 of 2023.
   • We have also further developed our general research methodology to ensure we follow best practices, such as following more closely to recommendations from the Cochrane Collaboration for conducting systematic reviews and meta-analyses. We will post an article outlining our research methodology in Q4 2023.
4. Updating the StrongMinds analysis in Q4 2023 to include more rigorous methods, the most recent studies, and to address various points raised by feedback from the EA community, including to:
   • Conduct a full systematic review of the evidence, with an improved process for determining the weight we place on different sources of evidence.
   • Update the estimated spillover benefits for household members.
   • Update the cost estimates for StrongMinds.
   • Apply more appropriate checks and corrections for publication bias.
   • Conduct more robustness tests to determine how the effects change under various reasonable analytic approaches.
   • We have addressed the data entry error for spillovers and have updated our analysis with the correct value.
   • We have updated our website to reflect our corrected estimates, and we now note which analyses are being updated.
5. We are developing new content for our website that will summarise our charity evaluations to date. We produce long technical reports – which we think we need to continue to do  – but we realise that if we only produce these, it can be challenging for others to understand and review our work. We hope these summaries will make it easier for others to understand our evaluations. 
6. We’ve added an experienced Comms Manager, Lara Watson, to the team. Lara will be helping us improve both our community and academic reporting. We are striving to take a tone in our communications that is more cautious and less adversarial, and Lara will actively double-check for this. 
7. Since I (Peter) have now stepped down as CEO of Mind Ease and UpLift and handed over the reins, I’ll be resigning from the board to join HLI as a Managing Director in mid-September, where I’ll be providing strategy, comms, and management support to the org. Michael will become Research Director, freeing him up to spend more time on research; moving forward, HLI will have two Co-Directors.

Again, we are grateful to have received feedback from the EA community, and we are looking forward to doing good better. If you’d like to follow HLI’s progress, please sign up for the newsletter in the footer of our site here, and we’ll also be posting updates here on the forums as usual. 

As part of my (Peter’s) new role, I’ll be reaching out to various stakeholders and collecting further feedback on HLI work, what we should do, and what we should do better. If there are any suggestions or concerns you’d like to send to me directly, I’m [email protected] – I’d love to hear from you.

Peter Brietbart (Outgoing Chair of HLI Board of Trustees, Incoming Managing Director)

Michael Plant (Outgoing Director, Incoming Research Director)