I always enjoy some good malaria cost-effectiveness chat, however I think we need to move beyond prioritisation of the roll-out of health interventions at the global/donor level. While there is a place for generalised cost-effectiveness analysis, on the whole it’s better to think of cost-effectiveness not as a generalisable property of a technology but as a product of the technology and the context. There is no doubt malaria vaccination (R21 or RTS,S) will be highly cost-effective in some populations and less so in others. The same goes for bed nets and SMC due to mosquito type, seasonality and many other factors. Plus, as others have said, the original post too often treats malaria interventions as alternatives when, in many cases, more important malaria planing will be about where to focus interventions rather than which technology to chose.
There are two broad ways to approach linking cost-effectiveness with context: i) large-scale multi-country, multi-intervention, cost-effectiveness analyses at national or sub-national level. There is a place for this and many organisations do great work in this space, including Givewell. ii) Countries undertake their own analyses and set priorities compared to other healthcare investment options, not so much other malaria interventions both other healthcare. For highly endemic regions, it could well be that nets and vaccination are both very cost-effective and that an MoH should prioritise this over, say, certain oncology treatments. If the MoH still accepts donated vaccine or earmarked vaccine financing (and why wouldn't it), then the donors aren’t really buying more people vaccinated, they’re buying more people treated for cancer - which might still be a positive contribution, but not what they think they are buying and not the best option.
From where I’m sitting, it seems that global health is rightly moving away from campaigns to back single technologies or disease areas and looking at how to provide less fragmented and more foundational support. That’s not as easy a story to tell to philanthropic donors but there’s increasing recognition of the harm that hundreds of separate earmarked offers of support for this technology or that disease area does. Even in most low-income countries, the government is the main funder of health services and those looking to help should identify appropriate auxiliary programmes to support, not displace or distract.
(Note all the above refers to *roll-out* of vaccines. For R&D a best-guess overall cost-effectiveness estimate makes sense and so does intervention-focused advocacy/engagement.)
Tl;dr it is usually better to consider cost-effectiveness of technologies as context-dependent, rather than an property of the technology, and we should consider systems support over financing technologies that would/should have been bought with domestic financing anyway. We need to empower local planners to consider the full range of health investment options, not try to decide for them.
--
A few bonus comments on some questions in the post:
“Is there a reason why including a mixture of cost-effective and somewhat less cost-effective interventions together could be a high-impact outcome?”
- Yes, for example if the combination of interventions brings local R below 1. But it might also be less than expected too. Effectiveness for infectious diseases is non-linear so it’s better to simulate transmission effects. This has been done by several groups.
“Will the cost-effectiveness of bednets/SMC decline over time (e.g. because of diminishing marginal returns of some sort)? If so, might vaccine cost-effectiveness overtake it? “
- Yes, insecticide or drug resistance can result in declining effectiveness of bed nets or prophylaxis.
And on the age-factor of bed net vs vx cost-effectiveness - I would certainly expect this to be taken into account in all decent modelling so would be wary of over correcting in botecs.
I mostly agree. Though having worked in research, policy and now policy analysis, it's also the case that sectors can get stuck in established practices. Big-picture /blue sky thinking is of course also important. But I agree that the value of work to get the basics/details right is under recognised. So thumbs up for your post, except the framing of "boring"; one person's boring is another's wildly fascinating - who are we to judge.
Thanks for sharing - it’s an interesting interview. My first reaction is that interdepartmental bureaucracy is quite a different beast to an evidence-to-policy process. I agree that splitting development policy/programmes across multiple government depts causes lots of problems and is generally to be avoided if possible (I’m thinking about the UK system but imagine the challenges are similar in the US and elsewhere).
Of course you do need some bureaucracy to facilitate evidence-to-policy too, but on the whole I think it’s absolutely worth the time. For public policy we should aim to make a small number of decisions really well. The idea a small efficient group who just know what to do and crack on is appealing; it’s a more heroic narrative than a careful weighing of the evidence. Though I can’t imagine the users of this forum need persuading of the importance of using evidence to do better than our intuitions and overcome our biases.
Incidentally, I feel this kind of we-know-what-to-do-let’s-crack-on instinct is more acceptable in development policy than domestic, and in my view development policy would benefit from being much more considered. We cause a lot of chaos and harm to systems in LMICs in the way we offer development assistance, even through programmes that are supporting valuable services. I think all of the major GHI’s do great work, but all could benefit from substantial reforms. Though again, this is somewhat separate from the point about interdepartmental bureaucracy.
Thanks for your comment and I agree. Modelling (even rigorous modelling) is just that, a model. It's a simplification of a more complex reality. We should not mistake the map for the territory, but equally, not using a map would be foolish.
Let me first re-emphasise that this blog is very much in a “yes and…” spirit. As I say in the blog, I believe EA is a positive influence on philanthropy and global development and has more potential to continue to shake things up for the better.
Thank you for this detailed response and the clarification that some of your analysis and grant making is indeed context-specific; great to see, particularly as someone who cut their teeth on CEA of malaria interventions. My impression is still that context-differentiated analysis perhaps the exception rather than the rule within the EA space - i.e. including organisations beyond GiveWell and topics beyond global health. It doesn’t yet come out strongly in the back-and-forth in forums like this one nor in presentations at EA Global conferences and it’s usually not emphasised in recommendations of EA donors - perhaps including GiveWell if we look at the evidence in the Top Charities page (as you note). Solving this challenge is not straightforward of course and remains an issue for other donors in the development space.
I don’t mean to suggest there are off-the-shelf lessons from HTA - on this or the other differences I highlight - that could be adopted directly by EA organisations. Equally, the field of HTA has been developing approaches for cost-effectiveness-based decision-making for several decades and may be fertile ground to explore for the development of EA prioritisation.
Ok gotcha this time. Similar to some of the response to Jason below I would say that the assessment of value and the assessment of gaps etc are separate steps. We shouldn’t think about adopting a value framework because it allows us to find a practical niche. I suppose it does beg the question; whose value framework should we use? I don’t have that answer, but I do think that it links to the 4th difference in my blog; about institutionalisation and participation.
Btw, you imply that the rest of the aid landscape uses this HTA-like approach to prioritisation.. Sadly this is certainly not the case! I worked in the UK Department for International Development for several years, in a team that was tasked with providing evidence to the rest of the organisation. We did our best but it was a long way from perfect. Indeed I do see a significant opportunity for EA movement and organisations to influence other donors to take a more systematic, evidence-informed approach. For me this absolutely does mean using cost-effectiveness - but it means using it not in a blunt way but as part of an appropriate process and so that - wherever possible - we work with, not around, local systems and institutions.
Yes I essentially agree - though the other option available is to decide to work more through local institutions to prioritise and fund health services. Again yes this may be better done at scale. Though with the rate of growth in EA funding, I wouldn’t rule out a role for this kind of EA engagement. Or perhaps there is a role for EA as a kind of catalytic funder, investing in frameworks/platforms for more effective aid policy from states and major multilaterals.
I think you mean that if there are gaps for highly cost-effective services that aren’t being delivered by other donors (or indeed the national health system, we should remember that even in the very poorest countries, national institutions are still likely to be the main healthcare funder and provider, not the donors) then it makes sense for EA to fill those gaps. I think gap-filling and prioritisation are separate steps. With the additional criteria we’re still trying to assess value and therefore in a sense a certain kind of more broadly conceived assessment of cost-effectiveness. For example, if intervention A has a better cost per DALY than intervention B but equity considerations mean the HTA committee strongly prefers intervention B, then B is rightly considered a higher priority. We could then look at the practical issues of gaps, organisations that can deliver etc.
On specificity of recommendations, yes that’s fair. As you say this is just a blog so perhaps for future work. I also considered working out a clearer taxonomy of what is being prioritised (causes/programmes/interventions/charities etc). But this one was on the long side for a blog as it is
First off - I see HTA as a kind of process wrapper for cost-effectiveness analysis. So in a sense It's CEA+, rather than an alternative.
I'm not sure I followed you points about moral uncertainty and emulation of pluralism. Would you try again to say what you mean?
On Justin's blog, CGD welcomes - even encourages - a diversity of views within its staff. I don't agree with everything in this blog. For example HTA (which as I say, encompasses CEA) is precisely the tool used by many countries in price negotiation. I think there will be a follow up CGD blog with the opposing view, but I would say that CEA/HTA is now adopted by a large and increasing number of countries to inform health policy. If we're saying it's good enough domestic policy but not aid policy, we need to be really clear why.
I would summarise your overall message as “we’re not ready for this yet” and I would partly agree. But EA community and orgs could be part of developing the solution and influencing other donors. We shouldn't consider EA orgs as separate from bilateral, multilateral or other philanthropic donors. At this point, EA is not an insignificant voice in the global health ecosystem. Also the shift doesn't need to happen for the sector as a whole, it could be country by country and some countries probably are ready for such an approach. We're discussing it with a couple.
To some of your more specific points:
I disagree that fungibility is the biggest challenge. Excepting cases of corruption, this simply means the country is effectively choosing the spend on other priorities. Indeed some folks in CGD have argued against the marginal aid approach precisely because they favour the fungible quality of current donations. This isn’t my view but it speaks to an opinion I do share, which is that, in many cases, countries should be given more control over healthcare priority setting.
Of course a common concern is whether country institutions are good faith actors and have the necessary capabilities to adopt a marginal aid approach. This will differ between countries and can be considered on a case by case basis - and will change over time (inc with donor support to strengthen priority setting capabilities, ensure comprehensive delivery on the core package etc). No country administrations are perfect, including in many donor countries, but we need a better framework for how countries transition from receiving support to deliver essential health services to doing so themselves. This is an active process in many countries today (inc where EA supported orgs operate) and has been exacerbated by the recent squeeze on global development financing.
Lastly, you touch on some practical challenges with funding marginal services. I agree that donors with country presence will be best placed participate in the coordination mechanisms necessary for a marginal aid approach, but EA individuals or orgs could i) choose to support intermediaries. There are none at present but it’s not hard to imagine existing organisations adopting the approach. For example, the Global Financing Facility (GFF) has done much to work with countries and donors on better coordinated and prioritised investments (with some success and some challenges). ii) they could also choose to support catalytic investments to strengthen country capabilities rather than earmarked aid for specific services.
I would challenge EAs to rethink the rationale for doing global level analysis to set health priorities in low income countries. In absence of local prioritisation it can be better than nothing, and initiatives like the Disease Control Priorities Project have been helpful. But such approaches are both technically and philosophically limited, and inferior to building local capabilities. Before too long, and with the growing decolonisation/localisation movements, I expect this top-down approach will seem increasingly out-dated.
I always enjoy some good malaria cost-effectiveness chat, however I think we need to move beyond prioritisation of the roll-out of health interventions at the global/donor level. While there is a place for generalised cost-effectiveness analysis, on the whole it’s better to think of cost-effectiveness not as a generalisable property of a technology but as a product of the technology and the context. There is no doubt malaria vaccination (R21 or RTS,S) will be highly cost-effective in some populations and less so in others. The same goes for bed nets and SMC due to mosquito type, seasonality and many other factors. Plus, as others have said, the original post too often treats malaria interventions as alternatives when, in many cases, more important malaria planing will be about where to focus interventions rather than which technology to chose.
There are two broad ways to approach linking cost-effectiveness with context: i) large-scale multi-country, multi-intervention, cost-effectiveness analyses at national or sub-national level. There is a place for this and many organisations do great work in this space, including Givewell. ii) Countries undertake their own analyses and set priorities compared to other healthcare investment options, not so much other malaria interventions both other healthcare. For highly endemic regions, it could well be that nets and vaccination are both very cost-effective and that an MoH should prioritise this over, say, certain oncology treatments. If the MoH still accepts donated vaccine or earmarked vaccine financing (and why wouldn't it), then the donors aren’t really buying more people vaccinated, they’re buying more people treated for cancer - which might still be a positive contribution, but not what they think they are buying and not the best option.
From where I’m sitting, it seems that global health is rightly moving away from campaigns to back single technologies or disease areas and looking at how to provide less fragmented and more foundational support. That’s not as easy a story to tell to philanthropic donors but there’s increasing recognition of the harm that hundreds of separate earmarked offers of support for this technology or that disease area does. Even in most low-income countries, the government is the main funder of health services and those looking to help should identify appropriate auxiliary programmes to support, not displace or distract.
(Note all the above refers to *roll-out* of vaccines. For R&D a best-guess overall cost-effectiveness estimate makes sense and so does intervention-focused advocacy/engagement.)
Tl;dr it is usually better to consider cost-effectiveness of technologies as context-dependent, rather than an property of the technology, and we should consider systems support over financing technologies that would/should have been bought with domestic financing anyway. We need to empower local planners to consider the full range of health investment options, not try to decide for them.
--
A few bonus comments on some questions in the post:
“Is there a reason why including a mixture of cost-effective and somewhat less cost-effective interventions together could be a high-impact outcome?”
- Yes, for example if the combination of interventions brings local R below 1. But it might also be less than expected too. Effectiveness for infectious diseases is non-linear so it’s better to simulate transmission effects. This has been done by several groups.
“Will the cost-effectiveness of bednets/SMC decline over time (e.g. because of diminishing marginal returns of some sort)? If so, might vaccine cost-effectiveness overtake it? “
- Yes, insecticide or drug resistance can result in declining effectiveness of bed nets or prophylaxis.
And on the age-factor of bed net vs vx cost-effectiveness - I would certainly expect this to be taken into account in all decent modelling so would be wary of over correcting in botecs.
I mostly agree. Though having worked in research, policy and now policy analysis, it's also the case that sectors can get stuck in established practices. Big-picture /blue sky thinking is of course also important. But I agree that the value of work to get the basics/details right is under recognised. So thumbs up for your post, except the framing of "boring"; one person's boring is another's wildly fascinating - who are we to judge.
Thanks for sharing - it’s an interesting interview. My first reaction is that interdepartmental bureaucracy is quite a different beast to an evidence-to-policy process. I agree that splitting development policy/programmes across multiple government depts causes lots of problems and is generally to be avoided if possible (I’m thinking about the UK system but imagine the challenges are similar in the US and elsewhere).
Of course you do need some bureaucracy to facilitate evidence-to-policy too, but on the whole I think it’s absolutely worth the time. For public policy we should aim to make a small number of decisions really well. The idea a small efficient group who just know what to do and crack on is appealing; it’s a more heroic narrative than a careful weighing of the evidence. Though I can’t imagine the users of this forum need persuading of the importance of using evidence to do better than our intuitions and overcome our biases.
Incidentally, I feel this kind of we-know-what-to-do-let’s-crack-on instinct is more acceptable in development policy than domestic, and in my view development policy would benefit from being much more considered. We cause a lot of chaos and harm to systems in LMICs in the way we offer development assistance, even through programmes that are supporting valuable services. I think all of the major GHI’s do great work, but all could benefit from substantial reforms. Though again, this is somewhat separate from the point about interdepartmental bureaucracy.
Thanks for your comment and I agree. Modelling (even rigorous modelling) is just that, a model. It's a simplification of a more complex reality. We should not mistake the map for the territory, but equally, not using a map would be foolish.
Greetings GiveWell Colleagues,
Let me first re-emphasise that this blog is very much in a “yes and…” spirit. As I say in the blog, I believe EA is a positive influence on philanthropy and global development and has more potential to continue to shake things up for the better.
Thank you for this detailed response and the clarification that some of your analysis and grant making is indeed context-specific; great to see, particularly as someone who cut their teeth on CEA of malaria interventions. My impression is still that context-differentiated analysis perhaps the exception rather than the rule within the EA space - i.e. including organisations beyond GiveWell and topics beyond global health. It doesn’t yet come out strongly in the back-and-forth in forums like this one nor in presentations at EA Global conferences and it’s usually not emphasised in recommendations of EA donors - perhaps including GiveWell if we look at the evidence in the Top Charities page (as you note). Solving this challenge is not straightforward of course and remains an issue for other donors in the development space.
I don’t mean to suggest there are off-the-shelf lessons from HTA - on this or the other differences I highlight - that could be adopted directly by EA organisations. Equally, the field of HTA has been developing approaches for cost-effectiveness-based decision-making for several decades and may be fertile ground to explore for the development of EA prioritisation.
Happy to keep in touch as useful.
Ok gotcha this time. Similar to some of the response to Jason below I would say that the assessment of value and the assessment of gaps etc are separate steps. We shouldn’t think about adopting a value framework because it allows us to find a practical niche. I suppose it does beg the question; whose value framework should we use? I don’t have that answer, but I do think that it links to the 4th difference in my blog; about institutionalisation and participation.
Btw, you imply that the rest of the aid landscape uses this HTA-like approach to prioritisation.. Sadly this is certainly not the case! I worked in the UK Department for International Development for several years, in a team that was tasked with providing evidence to the rest of the organisation. We did our best but it was a long way from perfect. Indeed I do see a significant opportunity for EA movement and organisations to influence other donors to take a more systematic, evidence-informed approach. For me this absolutely does mean using cost-effectiveness - but it means using it not in a blunt way but as part of an appropriate process and so that - wherever possible - we work with, not around, local systems and institutions.
Thanks Jason, appreciate your thoughts.
Taking your points in order
Hi Ben
First off - I see HTA as a kind of process wrapper for cost-effectiveness analysis. So in a sense It's CEA+, rather than an alternative.
I'm not sure I followed you points about moral uncertainty and emulation of pluralism. Would you try again to say what you mean?
On Justin's blog, CGD welcomes - even encourages - a diversity of views within its staff. I don't agree with everything in this blog. For example HTA (which as I say, encompasses CEA) is precisely the tool used by many countries in price negotiation. I think there will be a follow up CGD blog with the opposing view, but I would say that CEA/HTA is now adopted by a large and increasing number of countries to inform health policy. If we're saying it's good enough domestic policy but not aid policy, we need to be really clear why.
Thanks,
Thanks Ryan!
I see you like the capability approach - I'm a fan too
Thanks for this considered response Jason.
I would summarise your overall message as “we’re not ready for this yet” and I would partly agree. But EA community and orgs could be part of developing the solution and influencing other donors. We shouldn't consider EA orgs as separate from bilateral, multilateral or other philanthropic donors. At this point, EA is not an insignificant voice in the global health ecosystem. Also the shift doesn't need to happen for the sector as a whole, it could be country by country and some countries probably are ready for such an approach. We're discussing it with a couple.
To some of your more specific points:
I disagree that fungibility is the biggest challenge. Excepting cases of corruption, this simply means the country is effectively choosing the spend on other priorities. Indeed some folks in CGD have argued against the marginal aid approach precisely because they favour the fungible quality of current donations. This isn’t my view but it speaks to an opinion I do share, which is that, in many cases, countries should be given more control over healthcare priority setting.
Of course a common concern is whether country institutions are good faith actors and have the necessary capabilities to adopt a marginal aid approach. This will differ between countries and can be considered on a case by case basis - and will change over time (inc with donor support to strengthen priority setting capabilities, ensure comprehensive delivery on the core package etc). No country administrations are perfect, including in many donor countries, but we need a better framework for how countries transition from receiving support to deliver essential health services to doing so themselves. This is an active process in many countries today (inc where EA supported orgs operate) and has been exacerbated by the recent squeeze on global development financing.
Lastly, you touch on some practical challenges with funding marginal services. I agree that donors with country presence will be best placed participate in the coordination mechanisms necessary for a marginal aid approach, but EA individuals or orgs could i) choose to support intermediaries. There are none at present but it’s not hard to imagine existing organisations adopting the approach. For example, the Global Financing Facility (GFF) has done much to work with countries and donors on better coordinated and prioritised investments (with some success and some challenges). ii) they could also choose to support catalytic investments to strengthen country capabilities rather than earmarked aid for specific services.
I would challenge EAs to rethink the rationale for doing global level analysis to set health priorities in low income countries. In absence of local prioritisation it can be better than nothing, and initiatives like the Disease Control Priorities Project have been helpful. But such approaches are both technically and philosophically limited, and inferior to building local capabilities. Before too long, and with the growing decolonisation/localisation movements, I expect this top-down approach will seem increasingly out-dated.