Epistemic status: I’m confident in the question, unsure of the answer. This comes from working on healthcare AI in Nigeria, not from a dataset. I’d like to be proven wrong.
A clinic, not far from here
Picture a primary health clinic in northern Nigeria. One doctor, a few nurses, a queue that doesn’t end. Someone has installed an AI triage tool, and it’s good. It reads the symptoms, flags the urgent cases, and suggests a diagnosis in seconds. For a clinic that sees too many patients per hour, it’s a gift.
One afternoon it tells a young nurse that a child’s fever is routine. The nurse is three months into the job. She learned medicine mostly from the tool, because there was no time to sit with a senior and watch slowly how it’s done. She has no gut feeling to set against the screen. The screen says routine, so she writes routine.
I want to be clear that this scene is imagined... now to you reading this (one real moment from your own work where someone trusted an output they couldn’t check, or where someone caught something the system missed. Two or three sentences, concrete)......
But I don’t think this is far-fetched, and I think we’re building toward it without looking.
The situation on the ground
In my last post I argued that Africa is missing from the AI safety conversation. Since then I’ve kept running into a second gap, one that sits underneath the first.
Where I work, people learn by apprenticeship. Not only in markets, where a boy attaches to an oga and carries tools for a year before touching an engine. Doctors, engineers and lawyers learn the same way, just in nicer clothes: the junior years, the ward rounds, the first-draft grind. You learn what wrong looks like by being wrong cheaply while someone experienced stands nearby.
AI is now doing that junior layer first. And at the same time, many of our most experienced people are leaving. Health-worker emigration out of Nigeria, the japa wave, has been going on for years (Here).So the pipeline is thinning at the bottom while the senior layer thins at the top.
Why this is an AI safety problem
Almost every safety plan I’ve read leans on the same sentence: a human will check it. Human feedback, scalable oversight, audits, clinician sign-off, red-teaming. All of it assumes people exist who can look at an AI output in a domain and say, “No. That’s wrong.”
That sentence is an empirical claim about people. And as far as I can tell, nobody is measuring it.
We track model capability constantly: benchmarks, leaderboards, evals every month. We don’t track whether the humans in the loop are keeping their judgment, whether the next cohort is developing it, or whether “human oversight” in a given hospital is real or just a signature on a form.
Think about what that means for a place like Nigeria, where three things stack:
- Thin supply of experts, and the pipeline for new ones is under pressure.
- Models built elsewhere, trained on data that under-represents our languages, diseases and contexts. The blind spots are largest exactly where we most need to see them.
- Weak local ability to audit, so what the vendor says is accurate becomes the working truth.
When those line up, “human in the loop” can quietly become “human in the room.” Nothing looks broken until the error is big enough to count.
The study I want to see run
I don’t want to write a post that only warns. I want to propose something small and testable.
Take junior professionals in a high-stakes field, such as clinicians, engineers or lawyers. Split them into groups with different amounts of AI help while they learn. A year later, test each group on the same thing: can they independently catch planted errors, in cases where the AI is confidently wrong?... will @Centre for Effective Altruism fund this??
- Run it in more than one country. Include Nigeria and at least one other African country, because training-data gaps and senior-staff shortages are real variables here, not footnotes.
- Include cases where the AI is wrong in culturally specific ways, such as local disease presentations or names and languages the model handles badly.
- Publish the result either way
What each outcome means:
- If people keep their judgment while using AI, that’s good news. It makes the “humans will oversee” assumption more credible, and we can say so with evidence.
- If people lose it, then oversight plans need a human-capacity component, and funders have a reason to build one.
Either way, the field learns something it currently has to guess.
What I’d like from this Forum
- Is anyone already doing this? I’d love to build on existing work in automation and skill atrophy (aviation, medicine, the ironies of automation literature) rather than reinvent it. I haven’t done a full search, and I’d like to hear what exists.
- Researchers, hospitals, or training programs willing to run a pilot. I can help with Nigerian partners and the governance cohorts I already run.
- Maybe AI tutors will build judgment better than apprenticeship ever did. Maybe verification by other AI systems makes the human bottleneck moot. I’d like to see the strongest version of those arguments.
What I’m not claiming
I’m not claiming this beats existing priorities, and I’m not saying anything about slowing AI down. I’m saying that if safety depends on people being able to catch errors, then whether people can still do that is a load-bearing question, and it’s currently unmeasured.
So here’s the question I’ll leave you with: the next time an AI system is confidently wrong in a clinic, a court or a control room, will there be anyone in the room who knows enough to say no?
