This essay was jointly written by Peter Hurford and Marcus A. Davis.
Note that because of technical length restrictions on the EA Forum, this essay is the second part of three parts: Part 1, Part 2, and Part 3. To see all three parts in one part, you can view the article on our research site.
Previous articles also include an analysis of how beneficial vaccines have been, an analysis of how much it costs to roll-out a vaccine, how much it costs to research and develop a vaccine, and how long it takes to research a new vaccine. However, this series is structured so that starting at Part 1 should be all you need to do.
3. Why do these estimates vary? What assumptions are we making?
How do we assess the cost-effectiveness of developing a new vaccine, from scratch, before anything is known about the vaccine? As mentioned by Dalton (2016) (p13), research is “by nature the exploration of the unknown” and thus is naturally difficult to estimate.
For our models, we constructed ranges for cost-effectiveness estimates instead of point values, because there was a wide range of assumptions for each vaccine and it wasn’t clear which assumption would be most correct. When looking at cost-effectiveness, we should consider all the ways in which the cost-effectiveness of a vaccine may differ and what assumptions are being made.
Throughout our modeling, we identified eleven key sources of variation and assumption-making:
| Assumption | How do we account for it? | Overall impact on estimate |
|---|---|---|
| What population are you targeting? | Not accounted for in individual models, but tested over several modeled scenarios | Individual range could be off by 20 - 100% in either direction when applied to a different population. |
| How are you calculating the DALY burden for that population? | Mostly accounted for in individual models via Guesstimate range parameter | The true cost-effectiveness may be worse than the model range makes it appear (that is, the range is biased downwards) due to this effect. |
| How do you want to handle benefits that are not captured by DALYs? | Not accounted for by Guesstimate models | The true cost-effectiveness may be better than the model range makes it appear (that is, the range is biased upwards) due to this effect. |
| How effective are you assuming the vaccine will be? | Accounted for in individual models via Guesstimate range parameter | Should be accounted for within estimate range. |
| How expensive do you assume the roll-out will be? | Accounted for in individual models via Guesstimate range parameter | Should be accounted for within estimate range, but the true cost-effectiveness has some risk of being worse due to this effect. |
| What assumptions are you making about the R&D cost? | Mostly accounted for in individual models via Guesstimate range parameter | Should be accounted for within estimate range, but the true cost-effectiveness has some risk of being worse due to this effect. |
| How do you want to account for changing population and DALY burden into the future? | Not accounted for by Guesstimate models | The true cost-effectiveness has some risk of being worse than the model range makes it appear because of a shrinking DALY burden per person, but also some risk of being better than the model range makes it appear due to a growing population. |
| How do you want to handle “leveraged funding”? | Not accounted for by Guesstimate models (models take a total “all costs” approach) | The true cost-effectiveness may be better than the model range makes it appear due to this effect, perhaps by a factor of 25% to 100%, though a risk of being biased upwards by this effect is also possible. |
| How do you want to handle discount rates? | Not accounted for by Guesstimate models (models take a total “all costs” approach) | The true cost-effectiveness may be worse than the model range makes it appear due to this effect. |
| For how many years should you consider benefits? | Variation in this assumption is not accounted for by Guesstimate models. Guesstimate models assume 20 years of consideration. | Final range by model could be off by a large amount in either direction, depending on whether you think more or fewer years of benefits should be taken into account. It’s not clear which direction this goes in, though, as assuming both more and fewer years of benefits is reasonable. |
| How do you adjust for other counterfactuals? | Not accounted for by Guesstimate model (models take a total “all costs” approach) | The true cost-effectiveness may be worse than the model range makes it appear due to this effect. |
Taking all of this into account is essentially some attempt to itemize our model uncertainty, and it suggests that the true 90% confidence interval for our model would be much wider than the ones offered by our models.
For example, while our standard model gives a 90% confidence interval for the cost-effectiveness of the malaria vaccine as $23 - $52 / DALY, changes in these above assumptions could justify numbers perhaps as low as $0.30 / DALY using very optimistic assumptions across the board, and as perhaps high as $3500 / DALY using very pessimistic assumptions across the board.
This doesn’t necessarily mean that our model is wrong or that our 90% interval is not a true 90% interval – it just means that you have to be careful about your assumptions and what you are defining when you’re talking about “cost-effectiveness” in a particular context. A very wide notion of “all things considered” cost-effectiveness that captures all the variations in these assumptions is likely not useful, but itemizing and considering these assumptions is. Further exploration of these parameters is available in the discussion below and in the comparison to other literature found in Appendix A.
(If this table is enough for you, you can skip to the next section on takeaways. Otherwise, read on for in-depth discussions of each assumption.)
3.1.) What population are you targeting?
The overall cost-effectiveness of a vaccine can vary a lot depending on what population you are targeting. We explored two different scenarios – targeting enough people in SSA to achieve a 60% vaccination rate and targeting enough people for complete eradication. However, there are many possibilities. For example, one could try to target just an especially high-risk population, and this may be more cost-effective per person though help less people. Additionally, one could try to achieve more vaccination than just 60% of the SSA, though less than outright eradication. Depending on the precision and scale of the targeting and the DALY burden per person of the targeted population, the ultimate cost-effectiveness could differ.
For one concrete example, consider this table varying the targeting of the malaria vaccine. We look at our estimates for our two scenarios – vaccinating 60% of SSA and outright eradication, but we also consider a third, more targeted scenario – vaccinating 60% of a country with high malaria prevalence, like the Democratic Republic of the Congo.
| Population | Cost-effectiveness | Cost-effectiveness, excluding R&D costs | Guesstimate |
|---|---|---|---|
| 60% of DR Congo | $31 - $140 / DALY | $18 - $84 / DALY | Link |
| 60% SSA | $23 - $52 / DALY | $21 - $49 / DALY | Link |
| Eradication | $11 - $18 / DALY | $11 - $18 / DALY | Link |
A more targeted roll-out can be more cost-effective by eliminating more DALYs per person but could be less cost-effective when including R&D costs, as the R&D costs are spread across fewer people. On the other hand, the savings from outright eradication could make targeting the most people more cost-effective, even on a per person basis. For another exploration of how targeting can change cost-effectiveness estimates, see Appendix B on estimating the cost-effectiveness of Ebola.
3.2.) How are you calculating the DALY burden for that population?
The DALY burden per person can change how cost-effective a particular vaccination is, so calculating the burden correctly is important. We have dedicated an entire article to exploring that. However, it can be tricky to figure out what the correct burden should be for our vaccine scenarios. We’re trying to model the amount of DALYs that vaccines would reduce but we can’t always use the DALY figures as of today since some of these vaccines in reality already exist and thus already have dramatically reduced the DALY burden of diseases. For example, smallpox has been completely eliminated, so the current DALY burden is 0! Thus we have to carefully rewind the clock and extrapolate to what the DALY burden would be. On the other hand, we do need to include the impact of modern trends in vaccine alternatives, like how the malaria DALY burden has been substantially reduced by bednets.
For our calculations, we try to either estimate the DALYs, such as in the cases of smallpox and measles, or we try to use a range of DALYs from Global Burden of Disease (2016d). We think this helps capture uncertainty in the correct DALY burden to use, but that it risks overestimating the size of the DALY burden, especially extrapolating into the future.
3.3.) How do you want to handle benefits that are not captured by DALYs?
As reviewed in our assessment of the benefits of vaccines, there are multiple economic benefits from health savings due to reduced health care costs and reduced spending on combating the disease. Similarly, there may be substantial medium-term benefits from economic development via a healthier population. None of these effects are accounted for in our DALY-based modeling.
Additionally, if we vaccinate enough to achieve a significant degree of herd immunity but not enough to outright eradicate a disease, the benefit of this herd immunity is also not accounted for in our models. This would likely not affect our 60% SSA estimate much but may affect our eradication estimate.
We found these benefits to be too hard to pin down to be worth directly modeling, but we must note that their exclusion likely biases our estimate downward (making it appear less cost-effective than it should) to some degree.
Lastly, and this goes without saying, long-term effects such as the ramifications of shifting population change, or the effect of economic development on the trajectory of civilization, are not accounted for. It’s not clear what the impact of these longer-term effects would be.
3.4.) How effective are you assuming the vaccine will be?
Vaccine effectiveness rates are empirical values that can be assessed using trials. For example, the current malaria vaccine candidate in Phase III trials has an effectiveness of 13.7% - 46.9% after four years (Winskill, Walker, Griffin, and Ghani, 2017). However, vaccines usually aren’t rolled out with such low efficacy, and vaccines tend to increase in effectiveness over time with further R&D, and this effect is not fully accounted for in the current value.
On the other hand, it may not make sense to extrapolate the current efficacy of the rotavirus vaccine (or other very new vaccines like HPV and malaria) as anything other than the initial efficacy. Vaccines are improved and refined in part by being actually implemented and manipulating the timing of the shots, the number of shots, and by learning from impact in a real population. Accounting for this would change the estimate of roll-out DALYs prevented since it may be preferable, in such a modeling exercise, to use a longer time horizon and some probability distributions on the potential efficacy of the given vaccine over time.
Additionally, taking the efficacy of a new vaccine literally may risk extrapolating it too far. For example, our models estimate the benefits across a twenty-year time horizon, but the efficacy of the malaria vaccine has currently only been established for a four-year time horizon (Winskill, Walker, Griffin, and Ghani, 2017). It’s possible that the overall efficacy across the twenty-year time horizon could decline.
Our Guesstimates use ranges for effectiveness that include the empirical values, but with an assumption that these vaccines will become more effective with time. Some of the vaccines that are experimental and haven’t yet been rolled out, like the malaria vaccine, have very wide ranges for the estimate of effectiveness. We think it’s possible this extrapolation for the newer vaccines (e.g., HIV, malaria, Ebola) could be overestimating or underestimating the final vaccine efficacy found on actual roll-out due to these discussed factors.
3.5.) How expensive do you assume the roll-out will be?
The roll-out costs of a vaccine are empirical values that we have estimated, but they also tend to decrease over time with further R&D and further price negotiations, especially as the costs of the vaccine get recouped by the vaccine companies. The roll-out cost can also be very sensitive to the remoteness of the regions being targeted, and thus can vary depending on the population being targeted.
We have tried to account for these effects using a range of estimates for the roll-out costs. However, there still remains some risk that our model overcorrects for the remoteness of populations, as we did not fully research how those costs break down in our targeted population. There is also some risk that we have not adequately accounted for the declining cost of roll-out in our estimates.
3.6.) What assumptions are you making about the R&D cost?
Similar to roll-out costs, the R&D costs of a vaccine are empirical values that we have estimated (or at least guessed, where historical data was more sparse). R&D costs do not decrease over time, but they decrease per person and per DALY saved as the cost can be spread (amortized) across more people. Thus vaccinating a larger population will spread costs further and be “penalized” less by a high R&D cost. This should be fully accounted for in our models and scenario testing.
What we don’t take into account, however, is the degree to which R&D costs might be double counted in the roll-out costs. If the roll-out costs include a price per dose and the price per dose is meant to compensate companies for R&D into these vaccines (even explicitly so in the case of an advanced market commitment), we would count the R&D costs twice.
On the other hand, we’re unsure if the price of the vaccine does adequately capture the R&D cost or would be sufficient to incentivize the creation of the vaccine without additional outside philanthropy. Empirically, we observe philanthropy going to both vaccine R&D and vaccine roll-out, regardless of the existence of what should be a vaccine market that could incentivize R&D on its own. Outterson (2006) cites research that estimates the “cost recovery percentage” of R&D to be only 17%, which would suggest that we still should substantially consider R&D costs in our model, but it’s not clear how representative this estimate is of the particular vaccines we are attempting to model.
Overall, this might bias our model upwards (making our model appear less cost-effective than it should), but it’s not clear by how much. Given that R&D costs seem to be a very small component of the model and are typically dominated by much higher roll-out costs, it’s likely this bias is small.
3.7.) How will the population size and DALY burden change over time?
Population growth proved very difficult to model, as it depends on the growth specifically of the population that would receive the vaccine in the areas the vaccine is delivered, which may differ substantially from the more general population statistics that are freely available and that we used to calculate the DALY burden. We’re unsure how an increasing population and decreasing DALY burden will interact over our 20 year time span and to what degree each effect will dominate.
3.8.) How do you consider "leveraged funds" / "unlocking funding" / "crowing out funding"?
So far, we’ve attempted to look at creating rough estimates for the total pool of funding to create the vaccine and bring it to the necessary population, including some fixed cost to build up delivery infrastructure and then compared it to the total benefits. However, if we were an individual financial actor or institution in the wide web of vaccine funding, we would have to consider much more than the total rate.
This might be similar to a distinction made by Dalton (2016), which differentiates between the question “how cost-effective is it to spend an extra dollar on projects you can contribute to today”, called marginal ex-ante analysis versus “how cost-effective, on average, have all total contributions been to the project historically?”, called average ex-post analysis. There are likely huge differences in the two numbers, and just because something may look ineffective in average ex-post analysis does not mean it might be ineffective on a marginal ex-ante analysis (and vice versa). Since new philanthropic investments are made on a marginal ex-ante basis, we should put substantial effort into figuring out how marginal funding might be different than total funding. Considerations like leverage help with this.
A good model for this is discussed by Snowden (2018), where we might itemize all the individual funders contributing to the vaccine and decide what they would have donated to instead, and then increase or reduce the cost-effectiveness estimate accordingly. We could then consider a dependency web of funding, figuring out to what degree some funding “unlocks” other funding or “crowds out” other funding. However, actually carrying this out in detail is beyond the scope of this article, outright impossible for earlier vaccines where detailed funder information is unknown, and still exceedingly difficult for modern vaccines, like malaria, that do itemize all the relevant funders.
Current mechanisms for funding vaccines involve substantial “co-funding” from the governments of the countries receiving the vaccines and the foreign aid departments of developed nations. Thus private philanthropic investment (say by the Bill and Melinda Gates Foundation) could net “unlock” significant amounts of government funding that would likely be spent on net less cost-effective pursuits. On the other hand, difficulty in finding funding opportunities and the general perception that there is no “room for more funding” in this area could suggest that funding actually just “crowds out” funding from other would-be interested donors.
Our general intuitions are that, taking these two effects together, we’d guess overall that this net effect points more toward “unlocking” than “crowding out”, for a total effect that could substantially reduce the cost-effectiveness of a vaccine to be around 25% to 100% better (i.e., the adjusted cost effectiveness would be somewhere between 4/5ths to half the original unadjusted value) than what it would be if you did not consider this effect, from the point of view of a private foundation donor (or a private donor pooling their individual donation with such a foundation). We’re unsure of this, however, and the real effect could be substantially in the other direction.
Additionally, our analysis doesn’t offer a way to evaluate the impact of existing vaccination charities or interventions operating given existing vaccination delivery infrastructure and commitments from governments and non-state actors. For example, a charity that increased the efficiency of vaccination distribution through public outreach would not be responsible for the costs of vaccination doses. Instead, they would only be responsible for their personal staff costs and costs spent on outreach, and thus the cost-effectiveness of that charity would not be within the scope of our analysis of the cost-effectiveness of vaccines themselves.
We adapted the Guesstimate for malaria to include a leverage factor and explored some possible ranges:
| R&D Leverage | Roll-out Leverage | Total cost-effectiveness |
|---|---|---|
| 1 (Normal) | 1 (Normal) | $23 - $53 / DALY |
| 1 (Normal) | 1.25 (Crowd out by 25%) | $28 - $66 / DALY |
| 1 (Normal) | 0.75 (Unlock 25%) | $18 - $41 / DALY |
| 1 (Normal) | 0.5 - 0.75 (Unlock 25-50%) | $14 - $35 / DALY |
| 1 (Normal) | 0.5 (Unlock 50%) | $12 - $28 / DALY |
| 0.75 (Unlock 25%) | 0.5 (Unlock 50%) | $12 - $28 / DALY |
| 0.5 (Unlock 50%) | 1 (Normal) | $22 - $52 / DALY |
| 1 (Normal) | 0 (Unlock 100%) | $1.50 - $3.60 / DALY |
From this exploration, we can see that leverage on the roll-out costs dominates any leverage on the R&D costs.
3.9.) How do you handle discount rates, if at all?
When making this model, we did not apply any discount rate to future benefits. Furthermore, we don’t think a discount rate should be applied here, despite it being fairly standard to do so, for reasons discussed in Ord and Wiblin (2013). We hope that the lack of discount rate and the fact that we also did not consider population growth will roughly cancel each other out.
3.10.) For how many years should you consider benefits?
It is not clear for how long we should continue to consider benefits, since the benefits of vaccines would potentially continue indefinitely for hundreds of years. Perhaps these benefits would eventually be offset by some other future technology, and we could try to model that. Or perhaps we should consider a discount rate into the future, though we don’t find that idea appealing.
Instead, we decided to cap at an arbitrary fixed amount of years set to 20 by default, though adjustable as a variable in our spreadsheet model (or by copying and modifying our Guesstimate models). We picked 20 because it felt like a significant enough amount of time for technology and other dynamics to shift.
It’s important to think through what cap makes the most sense, though, as it can have a large effect on the final model, as seen in this table where we explore the ramifications of smallpox eradication with different benefit thresholds:
Smallpox Eradication Cost-effectiveness
| Benefits Considered | Cost-effectiveness | Guesstimate |
|---|---|---|
| 10 Years | $0.79 - $7.30 / DALY | Link |
| 20 Years | $0.41 - $3.50 / DALY | Link |
| 30 Years | $0.26 - $2.40 / DALY | See “10 years” |
| 50 Years | $0.16 - $1.50 / DALY | See “10 years” |
Malaria 60% SSA Cost-effectiveness
| Benefits Considered | Cost-effectiveness | Guesstimate |
|---|---|---|
| 10 Years | $24 - $56 / DALY | Link |
| 20 Years | $23 - $53 / DALY | Link |
| 30 Years | $20 - $47 / DALY | See “10 years” |
| 50 Years | $19 - $45 / DALY | See “10 years” |
The longer you consider for your benefits window, the better a vaccine will be. For smallpox eradication, the increased cost per year is considered to be ~$0 and the benefits in our model scale linearly. However, for the malaria 60% SSA model, the costs and benefits both scale with the window, leading to a much steadier increase in cost-effectiveness as the benefits window is increased.
It’s not clear that this consideration biases the model in any direction, making it systematically more likely to be an overestimate or an underestimate, but taking variability in this assumption into account should widen our confidence bars more than is reflected in our models. It seems reasonable one could make an argument for a lower benefits window (perhaps via a discussion of counterfactuals, see below) or a larger benefits window. Perhaps a more robust model may taper benefits over time due to other medical advances.
3.11.) How do you adjust for counterfactuals?
Some previous analysis, such as Dalton (2016), has looked at the benefits of funding a vaccine by assuming that this vaccine would have eventually been funded anyway by someone else, and that one’s philanthropy is merely “bringing the vaccine forward”, or making the vaccine available a few years earlier than it otherwise would have inevitably been made available by someone else, later. For example, if Salk hadn’t developed his polio vaccine, Sabin’s vaccine was just under six years away. Thus, Salk only “brought forward” a polio vaccine by about six years. This may be a big deal to the extra people spared polio over those years of additional vaccine time, but it limits the overall impact of Salk’s work.
This can be explicitly considered in our model by changing the “years to consider benefits” variable to the number of years you think the vaccine will be brought forward. However, we’re not sure that thinking in terms of bringing the vaccine forward is a good way to think about it, as any funding of a vaccine by a philanthropist now will forever free up the funding that a future philanthropist would have spent, allowing the philanthropist to instead spend money on something else potentially worthwhile, assuming that philanthropist would spend the money as effectively (or more) as you would. Thus this is not really a question of bringing the vaccine forward, but rather a special subset of the question of “crowding out” funding that we discussed earlier (see section 3.8).
4.) Our takeaways
Overall then, ignoring the economic benefits of vaccination, the total cost-effectiveness of the vaccines we observed varies widely from under $0.50 to over $1600 per DALY. Averaging out all the values we consider reasonable, produces a range of $18 - $7000 / DALY for the cost-effectiveness of a “typical” / average vaccine. Further changes in assumptions and contexts could dramatically change these numbers further in both directions.
In all cases, there was considerable variability both within and between vaccines based on uncertain inputs. This became especially difficult when trying to determine the historic impact of vaccines, as in the case of measles and smallpox, as there are many assumptions and decisions that need to be made in order to estimate the cost-effectiveness of vaccines across time. These assumptions can highly alter the final cost-effectiveness analysis and thus the best our analysis can do is justify narrowing the range of possible cost-effectiveness for these vaccines, as opposed to reaching a narrow answer with high confidence. We detailed a large section on model uncertainty to understand these comparisons and tried to compare against other analyses (see Appendix A).
Overall, while vaccination does have some clear wins, it does not seem to be a universal “best buy” 100% across the board. Smallpox eradication appears to be one of the most cost-effective interventions of all time, but HPV and Ebola vaccination (and even eradication) do not appear competitive with current opportunities in global health (see Appendix B for Ebola forecasting, Appendix C for comparisons between vaccination and current opportunities, and Appendix D for some analysis of GAVI). The only ongoing vaccine that could appear competitive to current global health opportunities is work on the malaria vaccine (see Appendix C), though other considerations could potentially make some other vaccine-related work more effective (see Appendix D). However, the multiple ways in which funding a concrete charity on the margin may differ from the abstract total cost-effectiveness of vaccination may make this comparison somewhat misleading altogether.
The desirability of funding a vaccine at the outset then seems to vary heavily on the underlying burden of the disease, but also on how efficiently a vaccine can be brought to market. If the cost of creating a vaccine is very high, as in the case of HIV, or the underlying burden of the disease is relatively low, as is the case with HPV or Ebola, the ultimate cost-effectiveness will suffer.
There are a lot of additional potential takeaways here, such as general lessons learned about cost-effectiveness modeling, which we’ll discuss in a final wrap-up essay.
Note that because of technical length restrictions on the EA Forum, this essay is broken up into three parts. Please continue on to Part 3 to see all the appendices and endnotes. To see all three parts in one part, you can view the article on our research site.
Thanks to Max Dalton, Joey Savoie, Tee Barnett, Palak Madan, and Christina Rosivack for reviewing this piece.