For context, GiveWell's relationship with CHAI dates to 2022, when GiveWell Managing Director Neil Buddy Shah departed to become CEO of CHAI. According to GiveWell's announcement, "this transition does not mark the end of Buddy’s relationship with GiveWell. It is important that GiveWell maintain strong connections with leading organizations in the global health sector." (Incidentally, Shah is also a member of Anthropic's long-term benefit trust.)

GiveWell announced Shah's departure in April 2022; Shah apparently started at CHAI in June; and in August GiveWell announced its first grant recommendation to CHAI, $10m for a new incubator program "to identify, scope, pilot, and ultimately scale cost-effective programs that GiveWell might fund". As planned, the incubator led to later GiveWell grant recommendations to CHAI, like CHAI's tuberculosis contact management program, and multiple grants to CHAI's oral rehydration and zinc distribution program.

Assuming you're correct that this grant is atypical for GiveWell, I would presume it's a result of their special relationship with Shah.

Thanks for sharing your critique of our recent grants with Open Philanthropy for technical support units (TSUs). We really appreciate this thoughtful pushback! We've recommended (and are considering) a number of grants to help respond to the current situation with cuts in US foreign health assistance. So, getting critiques like yours is helpful since it encourages us to pause and consider whether we’re making the right tradeoffs in these grants. While we share some of your perspectives on the uncertainties of this work, we're still excited about our decision in this case.

While this grant’s impact is particularly uncertain, we see this as a difference in degree, not kind, compared to other grants we recommend. Most of our funding still goes to Top Charities - proven programs backed by strong evidence and our cost-effectiveness analysis. But we also recommend opportunities through the All Grants Fund. The goal of this fund is to find and fund what we believe are the highest-impact uses of marginal dollars, even when those opportunities are riskier or harder to model. This grant fits squarely in that approach. We’ve funded technical assistance from the All Grants Fund before, alongside grants that are uncertain for other reasons. For example, sometimes we're trying to generate new evidence, while at other times we're recommending high expected value bets even when we know we’re unlikely to get a definitive answer on their impact.

We agree that the evidence base for TSUs is thin. In general, we think it’s challenging to evaluate technical assistance programs because 

• particular programs might not have very convincing controls or comparators and
• technical assistance packages can vary widely so aggregating results from many programs might not be that informative, even if they’re targeting the same outcome

So even if the review that Nick cites had found good evidence for past TA programs, we still might not feel sure that it would generalize to the TSUs we recommended funding. 

But as discussed above, we don’t consider high uncertainty to be a dealbreaker in grants funded from the All Grants Fund. We still think it can be worth funding TA (see, e.g., our maternal syphilis grants) and we’re very interested in building up our ability to learn about programs like this over time. (We’re working on a project looking back on a subset of technical assistance grants we’ve funded, but don’t have a publication date for that yet.) 

While we don't have detailed theories of change, we still think it's plausible that TSUs could be impactful. We are excited about this grant because we think it could help governments to make difficult prioritization and program adaptation decisions in countries affected by US government funding freezes and cuts. We expect that the details of how this could look will vary by country and so we don’t feel confident that any particular mechanism will cash out in impact. But for example, we think TSUs could help governments to: 

• Adapt programs away from more expensive models implemented with USG support to more cost-efficient models (e.g., by integrating disease-specific verticals or moving from international to local implementing organizations)
• Target programs (e.g., to certain regions or populations) based on cost-effectiveness analyses
• Crowd in philanthropic or other funding by identifying funding gaps 

While we think the above examples are plausible, we agree that the theory of change for these programs is not tightly specified. However, we spoke with senior Ministry of Health officials in each country about this grant, and overall governments voiced support and demand for the proposed TSUs and were eager to have CHAI and PATH's support on this work. We also think both organizations are well-placed to support this work, as both have supported on malaria-specific TSUs in the past, have teams with specific focus on health systems and health financing, and have established relationships with the governments they’re supporting. 

Budget - Thank you for sharing your estimates - this is helpful for us as we continue to update how we review budgets. We’ll share a high level budget breakdown for this grant with our public grant write-ups (which are coming - see below!). One quick clarification (which wasn’t clear in the podcast) is that costs for Nigeria reflect support in seven states as well as national support. 

Outside of that, we think that the higher budget reflects both higher salaries and a higher non-salary budget share (to account for travel, coordinating stakeholder engagement, and support from global technical teams). Our understanding is that salaries are set based on globally-benchmarked salary ranges and localized equity adjustments to account for organizational equitable pay standards and differential cost of living across different geographies. A portion of the compensation costs is also due to benefits (such as health insurance) that may be standard to each location.

Learning - We agree that we should try to learn about the impact of these grants and also agree with commenters and Nick’s revision that an RCT isn’t an appropriate strategy. We’ve asked CHAI and PATH to track and report out on the following. 

• Process indicators (staff hiring, government engagement, TSU recommendations made and implemented);
• Case studies of how government prioritization decisions are made;
• Promising funding opportunities identified through the TSU work;
• Contexts where cost-effectiveness considerations are or aren't helpful for government decision-making. 

We’ll attempt to triangulate these reports through speaking with other stakeholders, though we expect we’ll still have substantial uncertainty about impact given the lack of counterfactuals. 

No CEA and grant write up. These are coming! We typically have a lag between making grants and publishing our write-ups, but wanted to share about this grant sooner because we’ve received a lot of interest in our response to the funding cuts. We expect to publish pages for CHAI and PATH (including a rough BOTEC) by the end of June. 

Urgency. We see the urgency here as being specifically related to governments’ needs to adapt to frozen or cut US health assistance: we heard when investigating this grant that governments were already beginning this planning process and that lighter touch versions of the support offered by TSUs were already being provided on nights and weekends by CHAI staff in certain countries. We also think this kind of grant is inherently uncertain and it didn’t seem likely that we’d reduce that uncertainty by spending additional time investigating. So, with apparent demand for support at the time and since we didn't think waiting would lead to a better decision, we chose to recommend funding relatively quickly.

A quick drive-by comment on "4. Missed RCT Opportunity": The sample size seems way too small for a RCT to be worth it. There's not much statistical power to work with when researchers are studying a messy intervention with only 6 countries. And I imagine they'd struggle to attribute changes to the Technical Support Units unless it was something truly transformative (at least within the framework of the RCT).^[1]

More broadly, I'm not aware of any commonly accepted way to do "small n" impact evaluation yet, especially with something as customized as Technical Assistance. This blog post from 3ie, a NGO to promote evidence-based policy making, talked about the issue 13 years ago and I think it's still broadly true. The impact evaluation toolkit works best with (1) a precisely defined intervention, (2) a decent sample size (say n > 100), and (3) a very homogeneous sample. This grant, so far, looks to be the opposite of all 3.

1. ^{^}

   I also recall the math for statistical inference gets strange when using very small sample sizes (say n<15) and may require assumptions that most people consider unrealistic. But I could be wrong here.

Thanks for this Nick! I always appreciate your posts on forum! I haven't listened to the full podcast episode and don't have expertise in TSUs, but a few things that stood out to me/additional context on the timing of the USAID cuts:

• GiveWell mentioned that governments were specifically requesting this kind of support, which makes me more inclined to believe these TSUs will fill temporary gaps that were "outsourced" in a way from MoHs to USAID/other funders over the years or to address vulnerabilities that are rapidly appearing... so it could be more of a triage situation than setting even a short/medium term strategy for governments.^[1]
  • I share the sentiment of "not another dashboard!!" but I'm wondering if the way that USAID pulled the plug so quickly has made it so that governments have little visibility into their own programs and need to rapidly (re)build this capacity so they can reprogram funds accordingly - but also so they can go to other bilateral donors and apply for emergency funds that are/may become available through multilateral funders like the Global Fund or World Bank/IDA.
  • Despite claims to the contrary by Secretary of State Marco Rubio and others, the US government is moving to significantly reduce USAID's global health work^[2] In addition, nearly all USAID staff will be terminated on July 1 (some foreign service officers at post and administrative staff in DC will remain until September 1 to wind down other operations). 

I hope others with more knowledge about this will comment and thanks again for keeping global health on the front page :)

1. ^{^}

   One really bad example that comes to mind is when the Kenya MoH temporary lost access health data stored in the Electronic Medical Records (Kenya EMR) system - because it was physically hosted on the servers of a USAID contractor under stop work order. https://www.data4sdgs.org/blog/data-crisis-following-usaids-withdrawal-opportunities-reimagine-data-systems 

2. ^{^}

   I have a draft post about the FY26 President's Budget Request, proposed $900m rescissions to FY25 funds, and impending "pocket rescission" of FY24 funds (~$10 billion) and how these all interact, but haven't gotten around to finalizing.

Appreciate you putting this out with humility and curiosity, Nick. (And hi from South Africa!)

You're right that technical assistance (TA) in general lacks the type of rigorous evidence base (e.g. RCTs) that underpins most GiveWell top charity picks. And it is the case that the TSU model has its roots in more linear, project-based infrastructure settings, which may not map easily to health systems, which are ongoing, more transaction-intensive, deeply influenced by human behavior, etc.

Fully agree on the dashboard point–too often, evidence use in government is imagined as “provide better information and better decisions will result,” which is mostly unsupported by actual evidence or lived experience.

That said, I think the how of TA matters as much as the what. Whether support is effective depends on design. Do these units build state capability or hollow it out through parallel structures? Do they align with real incentives faced by actors inside the system? Are they embedded in actual decision-making processes? All these questions are core to our work supporting governments on economic growth.

So I don’t know if this is the right bet for GiveWell, but I do think there's value in experimenting with more adaptive, politically-aware, learning-oriented embedded support to governments. It’ll require thoughtful measurement and iteration (and not necessarily via a six-country RCT, which would fail to yield meaningfully insights), but it’s a space worth exploring if done intentionally.

What’s unique about these grants?: These grants are a good illustration of how GiveWell is applying increased flexibility, speed, and risk tolerance to respond to urgent needs caused by recent cuts to US foreign assistance. Funded by our All Grants Fund, the grants also demonstrate how GiveWell has broadened its research scope beyond its Top Charities while maintaining its disciplined approach—comparing each new opportunity to established interventions, like malaria prevention or vitamin A supplementation, as part of its grantmaking decisions.

The grants were explicitly made from the all grants fund, which is the place people donate when they are happy for GiveWell to make riskier decisions and hold themselves to lower standards than for top charities. I personally donate to the all grants fund over the top charities fund, am a fan of a more risk tolerant approach, and I'm happy to defer to GiveWell's judgement. I think your post is holding this grant to the standard of a top charity, which I think is unreasonable and would not be worth the effort and expense of GiveWell staff time

I don't have too much context on the actual object details of the grant, so don't have strong takes on most of your criticisms (you definitely know more about this domain than me!). But I find it pretty plausible that lots of high importance decisions get made after a disaster like the USAID cuts, and that this was urgent. And I also expect that there are, in general, a bunch of grants that are time sensitive in response to the USAID cuts and endorse GiveWell moving fast here and maximising expected value.

Good post Nick. I think the question mark about the timing of the experiment considering cuts to many robustly good programmes is a particularly good one

I don't think the Centre for Effective Aid Policy is a particularly accurate comparison, as I think there's a significant difference between the likely effectiveness of a new org lobbying Western governments to give money to different causes (against sophisticated lobbyists for the status quo and government-defined "soft power" priorities) and orgs with established relationships providing technical recommendations to improve healthcare outcomes to LEDC governments that actually express interest in using them. I think the lack of positive findings in the wider literature links you provide are more interesting, although suspect the outcomes are highly variable depending on level of government engagement, competence of organizations, magnitude of problems they purport to solve and whether the shifts they are promoting are even in the right direction. It would be interesting in that respect to see how GiveWell evaluated the individual organizations. I do agree that budgeting dashboards don't necessarily seem like an area relatively highly paid outsiders are best placed to optimise. 

I suspect the high cost reflects use of non-local staff, which of course has a mixture of advantages and disadvantages beyond the higher cost.

I'm sceptical of the value of RCTs between nations that have different healthcare policies and standards and bureaucracies to start with (particularly as I don't think there's a secular global trend in the sort of outcomes TSUs are supposed to achieve, and collecting data on some of them feels like it would involve nearly as much effort as actually providing the recommendations). A lot of policy and government optimization work - effective or otherwise - is hard to RCT especially at national level. Which doesn't mean there can't be more transparency and non-RCT metrics

Please give the full names of organisations you refer to with acronyms (CHAI and PATH). That's particularly important for CHAI since there is also the "Centre for Human-compatible Artificial Intelligence" with the same acronym, so it can be confusing for readers who are familiar with one but not the other (like me until I read a similar article earlier this year).