Some very common failure modes, when working independently:
Hey this is super cool!
I think this could be a great intervention, but I disagree on the RCT front. I strongly believe that in the medium to long-term, this intervention should, and will only go to scale after high quality RCTs.
Yours and RP's reasons for not doing it aren't compelling to me. I'm almost a bit confused.. This might actually be one of the easier RCTs to do in the development world. If I was a funder, I would be asking why you aren't in the middle of a big RCT at the moment, or at least why you aren't getting started on one. I agree with the funders.
"Funders often tell us to come back with an RCT, which we estimate would need around 72 hospitals across three countries, US$5 million and three to four years." If this is what's needed, then yeah do it. You can get the 5 million - have you asked GiveWell? But I'm skeptical you really need 72 hospitals. Mortality rates are high enough in massive hospitals I would imagine if you are in national referral hospitals in capital cities, or similar, then this number would drastically drop. Maybe you are considering a bunch of smaller hospitals?
From your article
"conventional RCT is hard to run for this intervention. It can't be blinded, because health workers know when monitors are on the ward. It has to randomise whole hospitals, because the central patient overview and the positive impact on workload affects care for every child on the ward, not only those attached to a monitor. Death is a relatively rare outcome, and baseline neonatal mortality across our sites ranges from 4.3% to 27.8%, which makes a trial harder to power. Ministries are also reluctant to keep hospitals in a control arm for years while neighbouring hospitals have monitoring. So is an RCT needed before scaling? If the effect is real, waiting three to four years for trial results means children in hospitals that could have had monitoring go without it. Last but not least it is important to emphasize that continuous monitoring is already considered standard of care in most parts of the world, so perhaps it is less about if it works but rather to what extent it should be prioritized amidst many other priorities and gaps.
From RP
"Note that a traditional randomized controlled trial may be genuinely hard to run in this context: a monitoring system cannot be disguised as a placebo, and it may change how the whole ward operates. For example, staff know when the monitors are present, and the intervention is bundled with training and supervision. The observed effect size therefore reflects both the device plus the attention surrounding it (i.e., the Hawthorne effect[5]),
1. Your current evidence is fairly weak, and low on the evidence ladder. Before and after studies are notorious for being unreliable because there are so many other factors involved. Hospitals are always improving their systems and there are many reasons why mortality could drop. Seeing consistency between studies is encouraging but evidence needs to be better.
2. Blinding is not very important - the staff seeing the monitors is part of the intervention no? I'm not sure why RP and you think this is a problem? If it changes how the whole ward operates that is part of the benefit of the intervention. In fact that might be where you main benefit is! I'm confused about this comment by RP here and not sure what point they are making.... The training and supervision front could be a big deal though, especially when you believe your main impact to be ward spillovers. That's why I think training and supervision can't really last more than a few months, then you need to largely leave the hospital be with the monitors with minimal supervision and see if mortality reductions are sustained over 2-3 years. This isn't that long an RCT!
4. Randomising whole hospitals isn't an ethical issue. Ministries will not refuse to allow hospitals being in a control arm (they are getting free monitors) and the evidence isn't nearly good enough yet to argue an RCT is unethical. If the difference in mortality between hospitals is big after year 2 say, the study will be triggered to stop for ethical reasons anyway.
5. The range of mortality rates doesn't make powering a study harder I don't think? (might be missing something) Its low mortality rates and low sample sizes that might be the issue here. But there are lots of hospitals where multiple kids die every week (I have one 500 meters from my home) If you are in the busiest hospitals where 150+ kids die every year, then I would think you wouldn't need neadly that amount (I might Claude this later). If you need to do a multi-country RCT so be it, that seems likely needed.
6. The urgency argument seems unrealistic. Because of the low level of evidence intervention isn't "obviously" better than other things we could spend money on. I don't think your case is nearly strong enough here to make a "bypass the RCT" argument. I think actually even framing it like this could make people lose confidence in the intervention. It feels more like a sales pitch than a good argument.
7. What is considered "Standard of care" in rich countries is close irrelevant in healthcare interventions in low-income countries. I don't think it should even be a factor really. There are 100 "standard of care" things you could introduce to a pediatric ward in Malawi (Better antibiotics, Dr. nos, nursing ratios, frequency checks, infection control, blood cultures, blood gases etc.). This doesn't mean you don't study their benefit in a low-income setting.
As an unrelated point, I feel like the presentation of the intervention here might be too branded. The intervention is a monitoring device, yet you are at pain to emphasise how your intervention is different from other devices. If your plan is to scale through government, its the monitoring system that matters not the branding. It might be better even to brand machines as "Ministry of Health" machines and think of yourself as a support org as much as an innovation org? This is a tricky one though I know.
I absoutely love this intervention - I'd be keen to get on a call and chat about it even if you might be. Sadly someone ffrom your org was pegged to meet me at EAG New York, but I can't go because of travel restrictions due to ebola :(
Will MacAskill's "Saturation View" write-up describes a view that tries to get around the Mere Addition Paradox and the Repugnant Conclusion by valuing variety.
This fails "Non-Anti-Egalitarianism" i.e. it sometimes places a lower value on a world where the same people have a more equal welfare and also a greater average welfare. He claims that it satisfies a weaker version in which the people don't change their "types" but I think that's not correct.
For his toy example (see screenshot): Call the top-left and bottom-right squares in the 3x3 grid red and blue. Consider two worlds:
The unequal world has total welfare 16 while the equal world has total welfare 18. The cumulative welfare of the lowest-welfare N members is also greater for the equal world for all N from 1 to 9.
However, Toy-Saturationism awards the unequal world a value of 24/5 = 4.8, which is greater than the equal world's value of 232/5 ≈ 4.2.
The choice of
In reality, I expect saturation views suffer badly from this problem of creating inequality for the sake of the "variety" values they introduce.
Really proud of this response love it! After a slow couple of weeks responding to the flurry of EA press, it's great to see some positive, plain English responses over the last week, and this might be one of the most important.
Also I think it's becoming more clear how important GiveWell and global health is for the reputation and public view of AI safety work. They are an almost unarguable anchor for do-most-good thinking that helps the public take the AI claims more seriously.
EA often gets criticized for taking small probabilities too seriously. But people take actions with small probabilities of big success all the time! Asking a girl out? Small chance she'll become your wife. Trying to affect political chance? Fat chance of that. Launching a startup? Good luck buddy. The difference between EA and the standard way of acting is that standard psychology deals with the problem of small probabilities by being highly overconfident. Of course _my_ startup/political movement/YouTube channel is going to make it big. You cannot maintain high agency while ignoring small probability options.
The EA approach is simply more honest.
Hmm...
It’s interesting: on the one hand, your hypothetical could be the way it works out, and it doesn't sound that bad. But on the other hand, as I say in the post, I think a sensible way to understand the badness of failing to make the hire is that it’s the difference between a) a 55-person org which is growing and b) a 56-person org which is growing. In your framing, it sounds like the org is only trivially worse off. In my framing, it’s clear that b) is much better off!
I thought about it for a little while. Here are three reasons why I think the costs of failing to make the hire are much higher than your comment makes them sound.
1. Repeating the hiring round fairly soon and making an equivalently strong hire is unusually fortunate
Here’s what might happen instead:
The manager learns they can’t hire for the role they wanted, and scopes out a smaller one. They hand off the procedural parts of the function and keep the rest. What they wanted was for the function to grow into a strong one, with the attention of someone who can oversee it well and make sure it moves to meet the needs of an aggressively growing organisation. It doesn’t get that. And because the smaller role is filled, the pressure to fix it goes away, so the situation tends to continue indefinitely: the function is weaker than it should be, and the manager keeps spending time on it.
Someone else accepts the role. The candidate is, in expectation, weaker, and in these rounds the gap between the top candidate and the next is often surprisingly large (Rob Wiblin worked through this).
The manager just keeps it. No re-run, no re-scope. The function sits at manager-level attention and the manager’s cycle stays blocked. This happens a lot. I’ve had functions that I ended up running with a small fraction of my time, going badly, and that situation continued indefinitely, sometimes in and out of failed hiring rounds.
2. Blocking other functions being scaled
Here’s a cycle which managers in high-growth orgs often spend their attention on: (1) scope out a function (understand what to do and how to do it, see if you can systematise it without hiring, learn what someone overseeing it would look like and what kind of person could do it); (2) hire someone to oversee the function; (3) onboard and train them; (4) use the freed-up time to repeat 1–3 for the next function.
The manager in your hypothetical has failed to get past (2) for this function, so they’ll move on to this cycle for other functions. That's fine, but failing to make the hire means that they'll be slower, in two ways. First, they’ll be spending (say) 10 hours a week overseeing the function for as long as it has no owner. Second, at some point they’ll do a hiring push to find the person, and that push will take something like 70–200 hours. All of that is time they could have spent spinning up the next function. In a fast-growing org, there's always some sense of haste in trying to get these functions going: because as soon as you get someone overseeing it, they're able to make it work well, then grow it at a similar compound growth rate to what the org is (so that it can meet the needs of the org).
3. For whatever time there is no hire, the function is being run poorly by the manager and falling behind org needs
The function which the hiring manager failed to hire for was among the highest priority functions to get more attention to. That's why it was being hired for!
“The manager covers it for a little while” means it runs on (say) ten hours a week of the manager’s time until someone is hired into it. It will fall further behind what the fast-growing org needs for that whole time, and might become a real mess (and/or maybe take more of the hiring manager's attention).
It being increasingly poorly run creates increasing problems for the org, and as it gets more messy it'll take whoever is eventually moved onto it longer to get it on the right track again.