This is really cool! I liked your approach to the model and I think your big-picture finding is at least moderately likely to be correct. Marshall alluded to this as well, but I think the upper end of your distribution for the average cost-effectiveness of new interventions is pretty unlikely (at least in global health and development), given that we know lots of interventions have already been tried and only a small number are >6x GiveDirectly's cost-effectiveness. But the general finding that we may be underinvesting in research seems robust even if you exclude the cases where the average cost-effectiveness of new interventions is anywhere close to the current best causes.
I looked at a similar but much smaller issue recently in evaluating whether it would be a cost-effective use of funding to replicate the 1998 deworming trial in Kenya that's been used to generate GiveWell's cost-effectiveness estimates of deworming. My finding there was that it was indeed likely to be a good use of funds, and I concluded that we were probably more generally underinvesting in replication research. I think that finding would fit with this one, in that the underinvestment in replication research may be part of an overall underinvestment in research.
This is very neat, thanks for sharing! Some comments.
First, the term “academic research” is used a lot in the text. Does this modeling speak to a need for more academic research or more research and development more generally?
It’s not clear to me that academia (as opposed to say, the nonprofit sector or even for-profit business) can claim the best track record of creating effective interventions. Academia might be involved at early stages but interventions often transition out of the academy when they scale up.
In other sectors, development (after the basic science research is done) costs the most and makes a product commercially viable. Maybe a majority of R&D spending should go towards development of interventions after initial proof-of-concept research is done?
Secondly, this modeling seems sensitive to assumptions about the efficacy of new interventions.
"...we found that, on average, investing at least 50% of the initial annual budget in scientific research is optimal even if the new interventions are only about half as cost-effective as the best existing intervention, on average…"
One could argue that new interventions are unlikely to be, on average, even half as effective as the best existing interventions, given that the best current interventions are recognized to be outliers (maybe even extreme outliers). Could you use some historical data to model average effectiveness of new interventions? There is a lot of cost-effectiveness data out there for public health interventions.
The simulations do not distinguish between scientific research and R&D projects outside of academia. The relative usefulness of these two types of research is beyond the scope of the model. The main assumption of the simulations is that the research projects are selected strategically for their potential to enable or produce more cost-effective interventions.
I agree that the assumption about the cost-effectiveness of new interventions can and should be validated empirically. Estimating it from historical data is an important direction for future work, and I am planning to pursue it. I think the expected cost-effectiveness will be vastly different depending on the extent to which the research builds on established knowledge and techniques. In the extreme case of refining the best existing intervention, the expected cost-effectiveness of the new intervention would definitely be larger than 50%.
Thanks, your points make a lot of sense to me! The case does seem to be stronger for R&D generally and it's helpful to know that you're not arguing for investment in a specific stage of research. I also agree that targeting existing interventions for improvement could be very high yield :).
Thank you, Stuart! Your post strongly resonated with me. :) I agree that EA funding agencies should develop farsighted strategic agendas for the entire process of generating and utilizing knowledge and innovations that are crucial for humanity's long-term survival and flourishing. I think this process should start with use-inspired basic research on crucial questions. The next step should be to translate the discoveries of that research into new interventions. Then those interventions should be tested on an increasingly larger scale and rigorously compared against the best existing interventions in terms of their cost-effectiveness. Once we have completed those steps to generate better interventions and knowledge about their effectiveness, we can exploit having a substantially improved set of opportunities to do good. A single program of a single funding agency could support all of these steps and thereby coordinate and guide the process of discovery, intervention development, and evaluation research towards what we most need to maximally improve the future of humanity. This farsighted strategic approach is not unheard of. The Development Innovation Ventures program and the Grand Challenges for Human Flourishing program are steps in that direction. I think EA Funding agencies could learn something from the design of those programs and combine their strategic approach with the latest insights and principles of Effective Altruism and good scientific practice. I think that would be extremely valuable.
Very interesting post, thank you for the research.
Based on your model, should Open Phil etc. be aiming for 50% research in every year? Or should it be aiming for a very high level of research funding now, knowing that it can take actions on better opportunities in the future? Maybe the research percentage by year should be something like 100%, 95%, 90% etc?
The recommendation of 50% already takes into account that better opportunities will be available in the future. This statistic means that the amount of money we invest into research in total across time and funding agencies should be roughly equal to the total amount of money that has been or will be invested into existing interventions. This global, long‐term 50-50 split can be achieved in many ways. One or more EA funding agencies temporarily investing much more than 50% into research could be a good way to implement it.
I expect the optimal fraction of resources going to research being lower for less neglected areas, whose ideas space has been more explored. So, for example, I guess the fraction of resources going into research should be higher for AI safety than for global health and development.
That being said, as you pointed out, if the current fraction of resources going into research is lower than the optimal fraction f, at the margin, it makes sense to direct more resources into research. This could mean, for instance, EA funders of global health and develoment moving much more than f to research.
I think it is also worth noting that, if I understand correctly, this analysis assumes the budget of each cause area is constant. However, moving resources to different activities (research or not) may also influence future funding. If non-research activities lead to more funding than research ones (I do not know whether this is the case), the optimal fraction of resources going to research will tend to decrease.
I agree that the optimal percentage of research funding is higher/lower for areas where less/more science and R&D have been done so far. We don't really know yet how different areas correspond to which of the simulated scenarios. I think establishing this correspondence will be a crucial next step in our project. Moreover, some topics and potential interventions within a broad cause area, such as global health and development, might have been researched much less than others. Therefore, it probably makes most sense to apply our analysis at the level of specific research topics or interventions.
Thank you for pointing out that the amount of available resources can change. Contrary to your intuition, I suspect that taking this into account would tilt the analysis in favor of even more research if the amount of funding an area receives depends on the cost-effectiveness and scalability of its best intervention. Successful research results in new interventions that are more cost-effective or more scalable than the best previous ones. This can significantly increase how much the cause area appeals to the EA community and how much finding it can absorb. Suppose, research in a cause area without any highly cost-effective interventions leads to the development of an intervention that is more cost-effective than the best interventions in any other area. That would probably increase the amount of money that will be donated to the cause. Or suppose that the research makes a highly effective intervention much more scalable. That would likely increase the amount of money that will be donated to the corresponding cause as well.
[Disclaimer: Joel McGuire contributed in a personal capacity unrelated to the HLI.]
EA grantmakers dramatically differ in how they split their funds between doing good with the tools and knowledge we already have versus improving them through scientific research.[1] This raises the question of which proportion of its funds the EA community should invest in scientific research. Are we investing the correct percentage of our funds in scientific research? Should it be lower? Or should it be higher than it currently is?
We recently found thatfunding scientific research that aims to enable or create improved interventions can be 2-5 times as cost-effective as investing in the interventions that already exist. This seems plausible given that scientific research gave us the tools and knowledge we can now use to improve the future of humanity. If previous generations had not funded scientific research and R&D projects, there would be no medicines for preventing malaria, no long-lasting insecticide treated bed nets, no supplements for preventing vitamin A deficiency, no deworming tablets, and no vaccines. Strategically funding relevant scientific research could generate even more cost-effective opportunities to do good. But the more money we invest in research the less money we will have left for applying our best interventions. To help funding agencies and the EA community navigate this dilemma, we investigated how the total investment into a cause should be split between scientific research that might enable us to do more good in the future and doing good with the tools and knowledge that happen to be already available today.
Assumptions
Our analysis makes the following three assumptions:
On average, additional research and R&D projects produce interventions that are substantially less cost-effective than the best interventions that already exist.
The cost-effectiveness of newly developed interventions is variable. Therefore, research occasionally succeeds in inventing superior interventions, and the cost-effectiveness of those interventions can be substantially higher.
When a more cost-effective intervention is invented, then the funding that the best previous intervention would have received will be invested into the new intervention instead.
We measure the cost-effectiveness of research in the same units as the cost-effectiveness of the existing interventions it might help us improve on. The expected increase in the amount of good we can do with the improved interventions is a lower bound on the value of research.
Approach
We performed more than 300 million computer simulations to determine which way of splitting a fixed budget between developing better interventions and deploying interventions would generate the most good in total in expectation across different scenarios. For each scenario, we estimated the total good that could be achieved by first funding 0, 1, 2, 3, …, or 1000 research projects and then optimally allocating the remaining funds between the best interventions that resulted from the research and the best intervention that were available prior to the research. We then derived the optimal size of the research budget from the number of funded research projects that maximized the total good that the funding agency accomplished on average across 10,000 simulations. To derive robust recommendations, we replicated the main findings of our simulations across a wide range of conservative assumptions about the effectiveness of scientific research, the variability in the cost-effectiveness of new interventions. Concretely, we varied the average cost-effectiveness of new interventions between 10% and 110% of the cost-effectiveness of the best existing interventions with a standard deviation between 1% and 100%. In addition, we varied the total amount of funding in the simulated cause area between 10 million dollars and 1 billion dollars. We assumed uniform priors over these different scenarios. Moreover, we considered that the cost-effectiveness of new interventions could either follow a heavy-tailed distribution or a Normal distribution.
Results
1. We should initially spend more than half of our resources on scientific research. According to our simulations, the optimal philanthropic strategy is to first strategically improve the repertoire of interventions that are available for a given cause by funding research projects and then optimally allocate the remaining funds between the best new interventions and the best previous interventions. Concretely, our simulations suggest that the optimal strategy for causes with a budget of 100 million dollars or more is to initially allocate at least 52% of their annual spending to scientific research, and later invest around 45% of the initial endowment into the most effective interventions that the research produced.[2] Moreover, as the budget increases to 1 billion dollars, the optimal size of the initial research budget increases to 64-72% (see Figure 1). This is because the larger the budget, the larger the total gain from improvements in cost-effectiveness of each dollar spent.
Figure 1. Relative size of the research budget in % as a function of the size of the total budget, depending on whether the cost-effectiveness of new interventions follows a heavy-tailed distribution or not.
2. We should fund many research projects. According to our simulations, we should fund at least one research project for every 10 million dollars that are available for a given cause (see Figure 2). As illustrated in Figure 2, the optimal number of research projects increases even more rapidly with the total amount of funding that is available for a given cause or cause area when the cost-effectiveness of new interventions follows a heavy-tailed distribution.[3] On average across all simulated scenarios, the optimal number of research projects on causes backed by at least 50 million dollars is around 48 to 111 depending on whether the cost-effectiveness of new interventions follows a Normal distribution or a heavy-tailed distribution, respectively.
Figure 2. Optimal number of research projects as a function of the total amount of funding available for a given cause or cause area. Please note that both the x-axis and the y-axis are on a log-scale.
3. Why is this a good idea? According to our simulations, the recommended level of research funding would likely produce interventions that are 78% to 252% more cost-effective at doing good than the best already available interventions. While research takes time, once new interventions have been developed, they will, on average, allow us to do about twice as much good with each of the remaining dollars as is currently possible (see Figure 3). The resulting increase in the total good produced by our annual donations and investments will add up over time. Thus, even if investing into research temporarily reduced donations to charities that deploy existing interventions, we would be able to quickly make up for that and then still have money left to do more good than we could have done without the research. Concretely, according to our simulations, this strategy achieves between 23% and 56% more good with the initial budget than investing everything into the application of interventions that already exist today. Moreover, having developed more cost-effective interventions would also amplify the benefit of additional future donations and could attract more funding to the cause.
Figure 3. The long-run value of investing into scientific research. The orange line shows the total amount of good that can be accomplished by investing 50% of the annual budget into research for the decade, and then investing everything into the best new interventions. The blue line shows how much good we would accomplish if we always invested everything into the interventions that already exist today. Comparing the two lines shows that investing into research pays off in the long run. This is true even if investing into research would temporarily reduce how much money we can invest into the deployment of interventions that already exist.
4. The EA community's investments in scientific research are much lower. So far, major EA funding agencies, such as GiveWell, Open Philanthropy, have spent much less than 50% of their resources on scientific research (see Figure 4). Only 1.6% of the grant funding that GiveWell has provided since 2014 supported projects with a research component. Open Philanthropy has invested about 25% of its funds into scientific research and R&D projects, and the Gates Foundation has invested approximately 30%. The Global Health and Development Fund, the Animal Welfare Fund, and the EA Infrastructure Fund invested between 13% and 28% of their funds into scientific research and R&D. For the FTX Future Fund the proportion of the investments that have gone into research and R&D projects is close to optimal. Only the Long-Term Future Fund invested more than 50% of its resources into scientific research and R&D projects (68%). However, this does not bring the total investment in research anywhere near the recommended 50% because the Long-Term Fund's budget was only 0.01% of all funds.[4] Crucially, the total research funding provided jointly by all those grantmakers was only about 30%. Without the Gates Foundation, it would have been only about 18%.
Figure 4. Comparison of the relative size of the research budget of two existing grant makers with the optimal size of the research budget. The bar labeled “All” refers to the grants of GiveWell, Open Philanthropy, EA Funds, and the FTX Future Fund combined. For the FTX Future Fund the error bar shows the proportion of the publicly declared investments into research out of the total spending. For the other grantmakers the error bars show the 95% confidence intervals of our empirical estimates we obtained by annotating a random sample of 200-600 of their grants. The red error bar shows the proportion of the publicly declared investments into research out of the publicly declared investments only.
5. The recommendations are robust to uncertainty about the effectiveness of research. As shown in Figure 5 and Figure 6 the recommendations of our analysis hold for a wide range of assumptions about the expected cost-effectiveness of new interventions and its variability across different research projects. Concretely, we found that, on average, investing at least 50% of the initial annual budget in scientific research is optimal even if the new interventions are only about half as cost-effective as the best existing intervention, on average (see Figure 5). This recommendation holds almost regardless of whether the distribution of the cost-effectiveness of new interventions is heavy-tailed or not.
Figure 5. Optimal size of the research budget as a function of the expected cost-effectiveness of new interventions relative to the cost-effectiveness of the best existing interventions.Figure 6. Optimal size of the research budget as a function of the variability of the cost-effectiveness of new interventions relative to the cost-effectiveness of the best existing intervention.
Discussion
We introduced a general method for determining how the grand total of the funds of a given cause or cause area should be split between charities that apply existing knowledge and interventions versus strategically selected basic and applied projects that might lead to more cost-effective interventions. We hope that, in the long run, the prudential application of our method to grant-making decisions will enable the EA community to do even more good by investing its funds more wisely. In particular, our analysis raises the question whether the EA community should invest more funds into scientific research and R&D projects. This question is very timely because our simulations suggest that the proportion of resources invested into research should increase with the total amount of funding available to a given cause area.
Some might object that academic research is already well funded. However, the focus areas and priorities of academic funding agencies are often misaligned with the goals of EA. This can make it extremely difficult for academics in certain fields to receive funding for EA-aligned research. Consequently, many academics who would be thrilled to have a more significant positive impact become discouraged and resign themselves to working on conventional topics. On the bright side, this creates an easy opportunity for the EA community to increase the value of academic research. All we need to do is to rectify the lack of funding opportunities for scientific research on crucial EA topics. Suppose we do decide to fund more research. In that case, we will face the practical challenge of determining which topics and interventions additional research should focus on could become a practical obstacle to increasing research funding to its optimal level. We have recently begun to develop a method for identifying impactful research topics, but much more work remains to be done.
Because there are usually multiple funders in each cause area, achieving the optimal split does not require that every grantmaker allocates the same proportion of their budget to scientific research. Rather, some funders may specialize in funding projects applying existing interventions, whereas others may specialize in funding scientific research and R&D projects. From this perspective, the results of our analysis can be interpreted as a recommendation for how the total funds of a given cause area should be allocated between funding agencies specializing in research versus intervention. In practice, many funding agencies, such as Open Philanthropy, support both.
Grantmakers that follow this approach could use the results of our analysis to determine whether they could increase their positive impact by increasing or decreasing the proportion of funds they allocate to research on a cause-by-cause basis. For each area, the optimal adjustments will depend on how funds are currently allocated in that area and how much research has already been conducted. Usually, the budget of a single program or grantmaker is only a fraction of the total funding for a given area. Thus, the implications of our analysis for an individual program or grantmaker depend on how other funders split their funds between research and applications. In highly neglected cause areas, most of the funding might come from within the EA community. In other areas, such as global health and development, governmental, corporate, and private philanthropy outside the EA community should also be considered to the extent that its funds are used effectively. Thus, when the other funders in a given cause area spent much less than 50% of their resources on developing more cost-effective interventions, it might be warranted to invest an even more significant fraction of the budget into scientific research. Conversely, suppose previous investments in a given cause area have almost exclusively focussed on scientific research. In that case, it might be optimal for an EA grant maker to focus primarily or solely on the findings’ applications.
Because the implications of our analysis depend on how funds are currently allocated in a given cause area, researching how funds are currently allocated in different areas is an important direction for future work. This research should be accompanied by determining the optimal fund allocation for a given cause area based on the potential gains that additional research could achieve. Through this investigation, we might identify which cause areas should increase their investment in scientific research and R&D projects and which, if any, should decrease it.
The simulations reported in this post make a number of rather abstract, theoretical assumptions about the cost-effectiveness of new interventions that might result from additional research and development projects. We believe that those theoretical assumptions can and should be replaced by data-driven estimates that reflect the effectiveness of scientific research and R&D projects tackling crucial problems in the cause area(s) that specific funding agencies or programs are working on. In fact, for many cause areas, both the expected value and the standard deviation of the relative difference between the cost-effectiveness of new interventions and the cost-effectiveness of the best existing intervention can be estimated from the effect sizes reported in relevant meta-analyses. Moreover, whether the distribution of the cost-effectiveness is heavy-tailed can be gauged from histograms of the cost-effectiveness of newly developed interventions and the prior distribution on the available budget can be replaced by the actual budget of a specific funding agency or program.
If you would like to know more about the methods and results of our simulations, you can read our detailed technical report. If you would like to use our method, feel free to reach out to us. We are happy to share the code with you and help you adapt it to your questions.
Author Contributions
Falk Lieder conducted the simulations and analysis and wrote the first draft. Joel McGuire gave feedback on the draft and contributed the ideas for the Assumptions section and recommended comparing the optimal versus actual sizes of grant maker’s research budgets. Joel was only involved in a personal capacity unrelated to his work at HLI.
Acknowledgements
Matthias Stelter gave insightful feedback on an earlier draft and made concrete suggestions for how to improve it. This analysis is part of theresearch priorities project, which is an ongoing collaboration with Izzy Gainsburg, Philipp Schönegger, Emily Corwin-Renner, Abbigail Novick Hoskin, Louis Tay, Isabel Thielmann, Mike Prentice, John Wilcox, Will Fleeson, and many others. The model is partly inspired byJack Malde’s earlierpost on whether effective altruists should fund scientific research. We would also like to thank Michael Plant from the Happier Lives Institute, Emily Corwin-Renner, andCecilia Tilli for helpful pointers and discussions.
On the low end of the spectrum, only 1.6% of GiveWell’s investments supported scientific research or the development of new interventions. On the high end of the spectrum, the Long-Term Future Fund provided about 70% of its funds to research and R&D projects.
How many more research projects we should fund on a given topic also depends on how much research has already been conducted on that topic. We simulated the effect of prior research by adjusting the distribution of the difference between the cost-effectiveness of new interventions and the cost-effectiveness of existing interventions.
Even if we don’t account for the grants made by the Gates Foundation, the budget of the Long-Term Future Fund is still only about 0.2% of the total funds invested according to the principles of EA.
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
I think right now EAs might be making a significant mistake by paying insufficient attention to the political realm. As EAs we tend to figure out what’s most impactful for us to work on and focus hard. That’s great! But there are various actions that are ‘non-delegatable’ - the extent to which an individual can do the action is limited (like voting, going to a protest, making hard money contributions to particular campaigns). It might be useful if we were all more in the habit of doing variou...
New Video from AI in Context: The Fall and Rise of Sam Altman
If you want to skip straight to the video, here it is!
AI in Context is excited to be back with our fourth video! For those just hearing from us, we make videos for 80,000 Hours, telling stories about transformative AI...
This is really cool! I liked your approach to the model and I think your big-picture finding is at least moderately likely to be correct. Marshall alluded to this as well, but I think the upper end of your distribution for the average cost-effectiveness of new interventions is pretty unlikely (at least in global health and development), given that we know lots of interventions have already been tried and only a small number are >6x GiveDirectly's cost-effectiveness. But the general finding that we may be underinvesting in research seems robust even if you exclude the cases where the average cost-effectiveness of new interventions is anywhere close to the current best causes.
I looked at a similar but much smaller issue recently in evaluating whether it would be a cost-effective use of funding to replicate the 1998 deworming trial in Kenya that's been used to generate GiveWell's cost-effectiveness estimates of deworming. My finding there was that it was indeed likely to be a good use of funds, and I concluded that we were probably more generally underinvesting in replication research. I think that finding would fit with this one, in that the underinvestment in replication research may be part of an overall underinvestment in research.