I feel like as president of 1Day Sooner I should probably chime in -- first, I wanted to say this type of work -- critiquing advocacy campaigns and analyses from EA or EA-aligned groups -- is very valuable and should be encouraged. I'm appreciative of SoGIve for publishing this and think they should be commended for spending the time to conduct this analysis. I think creating a healthy ecosystem for disagreement and the right incentives to encourage criticism and full-throated debate is important. 

On the object-level question, I'm obviously biased but I think most of the difference in cost-effectiveness in the SoGive analysis goes away if you adjust for the fact that vaccines are only given to children under 5 but only ~15-20% of bednets cover children under 5. Because 75% of malaria mortality is in children under 5, bednets are cheaper per person protected but the vaccines are much more targeted to people whose protection is most valuable. (The development benefit effects of reducing morbidity in children are also age-skewed in vaccines' favor though that's less dramatic). 

Insecticide resistance (probably reduces bednets' effectiveness to about 80% of what they'd otherwise be) and durability (GW estimates each bednet purchased provides about 1.7 year of coverage) are probably also relevant. The AMF tab of the GiveWell spreadsheet is a useful resource in thinking through these questions. 

For more of my thinking, here's my side of the email correspondence with Sanjay at SoGive. (I didn't include text from other people on the thread because I haven't asked their permission to share). An interesting meta-question is what should be the norm about making these sort of red-teaming or adversarial post-review correspondences public. My guess is it's probably a good thing to default to because it incentivizes people to be on their best behavior (and the benefits to confidentiality of being able to speak frankly don't seem that strong in these cases). But I don't think it's obvious either way and would be curious what other people think.

Overall, I'm eager to see more analysis done digging into the Imperial/Oxford modeling of cost benefit of the R21 vaccine (which comes to about 630 lives saved per 100K vaccinated, see Table 2) and what's publicly available about the WHO estimate of 13% all-cause mortality reduction from RTS,S. (Here's an older preprint that finds a smaller benefit -- more recent data that was publicly reported is apparently higher). So I think generally the follow-on research plan Sanjay discusses makes sense from my perspective, and I'd be personally supportive of anyone who wants to contribute to that work. 

Thanks SoGive for the post! We wanted to share some of GiveWell's current thinking around malaria vaccines in case it's helpful. We also wrote a report on RTS,S in 2022 here and have recommended a couple grants for vaccine rollout and research.

On a cost-per-person-reached basis, we agree ITNs and SMC are superior to either of the two WHO-approved malaria vaccines. However, we think there's less of a differential in cost-effectiveness than this post implies, for a number of reasons:

• The difference in all-in delivery costs is probably less substantial: We think it roughly costs $6 to deliver an ITN to a household^[1], and roughly $7 to provide a child with a full course of SMC. (See also impact metrics here). We estimate the total costs for fully immunizing a child range from $23-$43, depending on the choice of vaccine.^[2] 
• We would approach the comparison for the reduction in malaria incidence differently: Using the effect size from the Pryce et al meta-analysis for ITNs (which consists mostly of trials that lasted one year) and comparing it with the incidence reduction observed after 12 months in RTS,S/R21 trials is not as straightforward as is seems:
  • The reduction found in Pryce et al has to be considerably updated in light of recent developments, notably new types of nets (PBO, dual active ingredient) and increased insecticide resistance. 
  • The RTS,S Clinical Partnership trial includes results from a follow-up four years after the start of the intervention and finds a reduction in clinical malaria from 28% (3 dose group) to 36% (4-dose group) at endpoint. Our malaria team recommends these results rather than the earlier snapshots, as they are less noisy.
  • Our best guess is that R21 and RTS,S are similarly effective at preventing malaria. Available evidence suggests that short-term effectiveness is broadly similar between RTS,S and R21. No data has been published for the impact of R21 on malaria incidence over the long run (>20 months after the first dose). Because the short-term outcomes are broadly similar, and both vaccines employ the same mechanism to induce an immune response in the vaccinated person^[3], our best guess is that four doses of R21 probably offer similar levels of protection as four doses of RTS,S over the long run.
  • We believe the apparent difference cited in the post is most likely due to the different setup between trials (Datoo 2022, unlike the RTS,S trial, was carried out in a seasonal setting). Note also that results from a phase III trial on R21 are now available (Datoo 2023 (preprint)).
• The duration of protection differs: Due to factors such as attrition and physical decay, we currently estimate an ITN to provide between 1.2 - 2.0 years of effective protection. As indicated above, we estimate that malaria vaccines offer around 30% protection over four years. 

There are, of course, additional factors that need to be taken into account to get a full picture (for example, what coverage levels are achievable for each intervention?). However, our current best guess is that even with those included, nets (and SMC) will be more cost-effective than malaria vaccines - just not by an order of magnitude.

1. ^{^}

   Costs per child reached are much higher, roughly $15-$26.

2. ^{^}

   The price per dose of RTS,S was $9.30 in 2022, and estimates for R21 indicate a price per dose of $3.90. We expect that, on average, 70% of children who received three doses will also get a booster shot, which implies vaccine costs per child between $14-$37. The best costing estimates for the delivery of the doses suggest around $9 per child.

3. ^{^}

   “The leading malaria vaccine in development is the circumsporozoite protein (CSP)-based particle vaccine, RTS,S, which targets the pre-erythrocytic stage of Plasmodium falciparum infection. It induces modest levels of protective efficacy, thought to be mediated primarily by CSP-specific antibodies. We aimed to enhance vaccine efficacy by generating a more immunogenic CSP-based particle vaccine and therefore developed a next-generation RTS,S-like vaccine, called R21. The major improvement is that in contrast to RTS,S, R21 particles are formed from a single CSP-hepatitis B surface antigen (HBsAg) fusion protein, and this leads to a vaccine composed of a much higher proportion of CSP than in RTS,S.” Collins et al. 2017, “Abstract”

Thank you for posting this, really interesting! I am pretty excited for malaria vaccine roll-out, but think that posts which go against current thinking are valuable, and I appreciate this being posted.

Three things that I'd be super curious about (and i think might have the effect of being more optimistic); 

(1) IIRC the WHO found a 13% reduction in all-cause mortality (!) with a vaccination coverage rate of ~65%. This was in Ghana/ Kenya and Malawi, I think. I assume these areas already had some availability of SMC/ bednets, and this makes me think that there's likely quite a substantial impact beyond nets/ SMC? 

(2) Beyond immediate impacts of the vaccine- will the malaria vaccine likely bring eradication forward beyond the impact of nets/ SMC? (my current understanding is yes, and I think that bringing eradication forward is super important.)

(3) As you already point out, where would money have otherwise been spent. My impression here is that the money which funds malaria vax roll-out would likely come from a mix of sources, rather than nescessarily pulling from SMC/ bednets (but I still have some concern/ uncertainty here). 

I said this in another comment, but in case it gets missed, I just want to highlight that 1Day Sooner has shown an excellent attitude. When we reached out to them, they were consistently welcoming of the criticism and had constructive useful comments. I've found these virtues to be more common in the EA community than elsewhere, but I still like to call them out when I see it.

I always enjoy some good malaria cost-effectiveness chat, however I think we need to move beyond prioritisation of the roll-out of health interventions at the global/donor level. While there is a place for generalised cost-effectiveness analysis, on the whole it’s better to think of cost-effectiveness not as a generalisable property of a technology but as a product of the technology and the context. There is no doubt malaria vaccination (R21 or RTS,S) will be highly cost-effective in some populations and less so in others. The same goes for bed nets and SMC due to mosquito type, seasonality and many other factors. Plus, as others have said, the original post too often treats malaria interventions as alternatives when, in many cases, more important malaria planing will be about where to focus interventions rather than which technology to chose. 

There are two broad ways to approach linking cost-effectiveness with context: i) large-scale multi-country, multi-intervention, cost-effectiveness analyses at national or sub-national level. There is a place for this and many organisations do great work in this space, including Givewell. ii) Countries undertake their own analyses and set priorities compared to other healthcare investment options, not so much other malaria interventions both other healthcare. For highly endemic regions, it could well be that nets and vaccination are both very cost-effective and that an MoH should prioritise this over, say, certain oncology treatments. If the MoH still accepts donated vaccine or earmarked vaccine financing (and why wouldn't it), then the donors aren’t really buying more people vaccinated, they’re buying more people treated for cancer - which might still be a positive contribution, but not what they think they are buying and not the best option. 

From where I’m sitting, it seems that global health is rightly moving away from campaigns to back single technologies or disease areas and looking at how to provide less fragmented and more foundational support. That’s not as easy a story to tell to philanthropic donors but there’s increasing recognition of the harm that hundreds of separate earmarked offers of support for this technology or that disease area does. Even in most low-income countries, the government is the main funder of health services and those looking to help should identify appropriate auxiliary programmes to support, not displace or distract. 

(Note all the above refers to *roll-out* of vaccines. For R&D a best-guess overall cost-effectiveness estimate makes sense and so does intervention-focused advocacy/engagement.)

Tl;dr it is usually better to consider cost-effectiveness of technologies as context-dependent, rather than an property of the technology, and we should consider systems support over financing technologies that would/should have been bought with domestic financing anyway. We need to empower local planners to consider the full range of health investment options, not try to decide for them. 

--

A few bonus comments on some questions in the post:

“Is there a reason why including a mixture of cost-effective and somewhat less cost-effective interventions together could be a high-impact outcome?”

 - Yes, for example if the combination of interventions brings local R below 1. But it might also be less than expected too. Effectiveness for infectious diseases is non-linear so it’s better to simulate transmission effects. This has been done by several groups. 

“Will the cost-effectiveness of bednets/SMC decline over time (e.g. because of diminishing marginal returns of some sort)? If so, might vaccine cost-effectiveness overtake it? “

 - Yes, insecticide or drug resistance can result in declining effectiveness of bed nets or prophylaxis. 

And on the age-factor of bed net vs vx cost-effectiveness - I would certainly expect this to be taken into account in all decent modelling so would be wary of over correcting in botecs. 

The burden of malaria as it exists right now, exists even amidst the decades of use of all the available interventions (including ITNs and chemotherapies). Is there any progression of To add to that, compliance in the use of the ITNs is also challenge (several reports exists on how ITNs in Africa end up being used for fishing, or how some resell them. The ITNs reduce all cause mortality by 17% and the RTS'S vaccine reduces mortality by 13% and the projection for R21 is expected to be higher when the data becomes available.

I am not 100% certain that I follow all the math. However, when it comes to the malaria burden and which intervention is most cost effective, the general rule I'd apply is the permanence of the intervention. An intervention such as the use of ITNs may appear cost effective if you view it in a period of maximum 3 years. But because such an intervention is not permanent, the need to replenish ITN supplies to households repeatedly means that nets are a recurring cost. This implies that the actual cost of ITNs can't fully be quantified in the sense of how long malaria as a disease will continue to exist. Vaccines on the other hand are more permanent and have a predictive cost even when measured against waning immunity. 

Vaccines will bring us closer towards herd immunity and potentially the elimination of malaria and vaccines must be used complimentary to other interventions in the malaria toolkit.

As much as I am for malaria vaccine, I feel it does not solve the issues caused by mosquitoes. Where does the vaccines solution leave the pain caused by mosquito sting? The bed nets protect us from both malaria and physical pain caused by mosquito bite.

How long do the effects of the vaccine last / how was this incorporated into your estimates?

My naive guess is that this should make vaccines more cost effective on the time scale of years (since you won't need to re-vaccinate people), but maybe it's already incorporated into the efficacy estimate?

My two cents, leaning (weakly) towards there being a danger of comparing apples to oranges, and 'red teaming' being more valuable when more data is available.

• Much of the cost of the malaria vaccine will be borne by governments or non-EA international aid (current $25 cost-estimate of a programme may also include committed govt health spend on vaccination infrastructure) 
  • afaik the marginal cost per impact of a hypothetical EA vaccine rollout funding program is currently unknown and likely lower than current total cost per vaccination estimate 
  • other organizations involved already fund many interventions with weaker cost/benefit and ring fence causes, so them diverting funding to malaria vaccination may be significantly net positive even if better interventions exist
• Cost effectiveness has already been a consideration for those involved in the programmes.(most notably the Gates Foundation dropped out of funding RTS'S primarily based on cost-effectiveness related concerns. I am not sure of their position on R21 or if it is still evolving). So it hasn't been neglected and public-facing critical analysis may already be available.
• It's unlikely 'red teaming' medical trials will uncover the sort of methodological flaws present in less rigorous RCTs and analysis of more speculative causal relationships.
• Vaccination is intended to be a complement rather than a substitute for nets/SMC and an idealised antimalarial intervention should include both.
  • BOTEC calculation based on above figures $27 for an 88% reduction; likely to compare very favourably with most global health interventions
  • I suspect most of the funding for the vaccination program won't be diverted from bednets or other extremely cost effective interventions, and organisations which backed vaccine development continue to spend on bed nets
• The efficacy criteria are not equivalent as the 55% of [severe] malarial cases not prevented by nets represent cases where the nets have no effect (e.g. daytime infections) whereas the vaccination may still have had a positive immune effect on the the 23% of clinical malarial cases still diagnosed post-vaccination. Long term economic and health impacts of a vaccine attenuating [still severe] malaria infection even where symptoms exist are outside the scope of evidence collected, but plausibly significant.
• Malaria vaccination is a new programme involving many non-EA stakeholders and facing numerous political and logistical challenges to roll out. Caution is needed to avoid "red team" material being misinterpreted as arguing R21 is ineffective or poor value relative to the median global healthcare intervention rather than relative to the most cost-effective interventions studied; perfect can be the enemy of good. Current evidence suggests the interventions are cost effective relative to many global health programs with public support, and on a cost-neutral basis R21 appears significantly more effective than other standalone antimalarial interventions
  • Downside risk is greater if the preliminary data overstates marginal costs or underestimates long-term benefit (plausibly both)
  • Likelihood and salience of misinterpretation lower (and benefit of analysis of more up-to-date data/estimates higher) after govt commitments made, with research more likely to be appropriately used by EAs exploring a specific donation option
  • I would update the other way if there are major institutional donors ready but unwilling to commit in the absence of thorough independent study. Ideally, these are the sort of donors that ought to be willing to fund you too!

(Disclaimer: not a sector expert. I may help 1Day Sooner with their request for ad hoc research task support in the near future but opinions and any errors are entirely mine)

Executive summary: The cost-effectiveness of malaria vaccines like R21 is uncertain and potentially much worse than bednets or seasonal malaria chemoprevention. More research is warranted before advocating for rollout.

Key points:

1. Malaria vaccines may cost over $25 per person protected versus under $3 for bednets, with similar efficacy. This 10x difference merits investigation.
2. Key uncertainties include future costs as production scales, efficacy duration, and impact on severe malaria. Interaction effects with nets are also unclear.
3. An updated cost-effectiveness analysis is proposed, focusing on R21 versus nets, incorporating new data, but avoiding full red-teaming.
4. Even if less cost-effective, advocacy might be reasonable if it unlocks additional funding or enables elimination.
5. More evidence is needed before confidently advocating for rollout over alternative interventions.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.