Thanks Morgan! I appreciate the detailed response from you, Filip, and Vicky. Glad the patterns were useful. I'll look out for your email and am happy to dig into whatever direction would be most useful on your end.
Thanks Toby, both the specific guidance and the offer are super helpful.
The asymmetric-reach point is something I had not thought of, but makes perfect sense. Even when a misunderstanding would be the org's to own, the critique travels further than the correction, so the pre-share is doing real work beyond politeness. I'll treat it as the default going forward, and I'll take you up on the DM route next time rather than trying to find a warm lead myself. That removes my main hesitation.
And good catch on the link-post vs cross-post distinction. I didn't get that, either, somewhat conflated them. For the next one I'll put the full text on the Forum with a canonical link back. You're right that the comments are where the methodology actually gets pressure-tested, and honestly, my blog has no commenting enabled, so the Forum is clearly where the conversation should live. Appreciate you taking the time to flag both of these.
I sent Vicky an email at the time of publishing rather than sharing a draft in advance. I didn't pre-share for two reasons: I don't have an existing relationship with anyone at AIM, so cold-emailing a stranger and asking them to review a draft on a deadline felt like a heavier ask than a heads-up at publication, and I tried to write the post so that any factual claim can be verified in a few minutes from the public spreadsheets (the cell references in each finding are there for that purpose). Vasco Grilo's recent methodology posts on AIM CEAs are part of the same public conversation, and Vicky has engaged with those publicly on the Forum, which is part of why the published-with-heads-up route felt appropriate for a contribution in the same vein.
That said, I'm new to posting on the Forum and I'd take your read on the convention seriously. If the norm here is that pre-sharing should happen even when the analysis is grounded in public data, I'd want to know for next time.
If you read my blog post, I go into detail about why this is not a model issue. It's about how you frame the question much more than what the model contains. For this purpose any decent model would have had the same result. The main benefit that Claude gives is direct in terminal code writing and execution.
The model I use did have an earlier cut off for it's data, but that isn't relevant for what I am doing. My write up actually surfaced several things that they didn't see at all. That's the point, really. And for the verication, it did not look at GiveWell at all. My verification sources are all listed in the code if you are concerned. All reputable sources.
Good to hear! All of my work is there on github. Please have a look at the results. If my pipeline found something that yours didn't, it might be worth integrating the methodology.
I'd be very happy to discuss with you at your convenience. I'm in Central EU time (Italy.) I also sent you an email via [email protected]. Hannah says she will pass it on to you.
If you want help converting to a database let me know. It looks like a weekend project. We could also develop a front end for easier input, if you want. I would be happy to assist you.
When I ran my multi-agent pipeline on the data, I had to create unique parsing rules for each data set due to inconsistencies in the spreadsheets. One of my suggestions going forward would be to standardize how they collect and store the data to make it more machine readable. But you are right. Claude Code can sift through it and sort it out, with the right prompting, at the right stage. It is just one more context to manage.
Hi — I took you up on the invitation to try an alternative AI red teaming approach.
I built a multi-agent pipeline (decomposition → investigation → verification → quantification → adversarial testing → synthesis) and ran it against all three interventions where you published detailed AI output: water chlorination, ITNs, and SMC.
Results across three runs:
Signal rates: 84% (water), 100% (ITNs), 82% (SMC) — vs your reported ~15-30%
Zero hallucinated citations (the key architectural change: Investigators generate hypotheses without citing evidence, then a separate Verifier searches for real evidence)
Each surviving critique includes parameter mappings to specific CEA spreadsheet cells with computed sensitivity ranges
The most interesting finding was cross-intervention: three structural patterns appeared independently in all three analyses. All three CEAs model dynamic phenomena with static parameters (adherence decay, resistance evolution, efficacy degradation). All three collapse meaningful within-category variation into single aggregate parameters. And the two malaria interventions both lack mechanisms to capture biological adaptation by the target organism.
Your post mentions you covered six grantmaking areas total. The other three — CMAM, syphilis, and malaria vaccines — could be run through the pipeline as well. It doesn't strictly require your AI output to function; that feeds into novelty filtering and baseline comparison, but the critiques themselves are generated independently. I've reached out separately about this.
Happy to discuss methodology, and happy to hear where you think the pipeline's findings miss the mark — several of the cross-intervention patterns may reflect deliberate modeling choices rather than oversights, and I'd be interested to know which.
Thanks Morgan! I appreciate the detailed response from you, Filip, and Vicky. Glad the patterns were useful. I'll look out for your email and am happy to dig into whatever direction would be most useful on your end.
Tsondo
Thanks Toby, both the specific guidance and the offer are super helpful.
The asymmetric-reach point is something I had not thought of, but makes perfect sense. Even when a misunderstanding would be the org's to own, the critique travels further than the correction, so the pre-share is doing real work beyond politeness. I'll treat it as the default going forward, and I'll take you up on the DM route next time rather than trying to find a warm lead myself. That removes my main hesitation.
And good catch on the link-post vs cross-post distinction. I didn't get that, either, somewhat conflated them. For the next one I'll put the full text on the Forum with a canonical link back. You're right that the comments are where the methodology actually gets pressure-tested, and honestly, my blog has no commenting enabled, so the Forum is clearly where the conversation should live. Appreciate you taking the time to flag both of these.
I sent Vicky an email at the time of publishing rather than sharing a draft in advance. I didn't pre-share for two reasons: I don't have an existing relationship with anyone at AIM, so cold-emailing a stranger and asking them to review a draft on a deadline felt like a heavier ask than a heads-up at publication, and I tried to write the post so that any factual claim can be verified in a few minutes from the public spreadsheets (the cell references in each finding are there for that purpose). Vasco Grilo's recent methodology posts on AIM CEAs are part of the same public conversation, and Vicky has engaged with those publicly on the Forum, which is part of why the published-with-heads-up route felt appropriate for a contribution in the same vein.
That said, I'm new to posting on the Forum and I'd take your read on the convention seriously. If the norm here is that pre-sharing should happen even when the analysis is grounded in public data, I'd want to know for next time.
If you read my blog post, I go into detail about why this is not a model issue. It's about how you frame the question much more than what the model contains. For this purpose any decent model would have had the same result. The main benefit that Claude gives is direct in terminal code writing and execution.
The model I use did have an earlier cut off for it's data, but that isn't relevant for what I am doing. My write up actually surfaced several things that they didn't see at all. That's the point, really. And for the verication, it did not look at GiveWell at all. My verification sources are all listed in the code if you are concerned. All reputable sources.
Good to hear! All of my work is there on github. Please have a look at the results. If my pipeline found something that yours didn't, it might be worth integrating the methodology.
I'd be very happy to discuss with you at your convenience. I'm in Central EU time (Italy.) I also sent you an email via [email protected]. Hannah says she will pass it on to you.
If you want help converting to a database let me know. It looks like a weekend project. We could also develop a front end for easier input, if you want. I would be happy to assist you.
When I ran my multi-agent pipeline on the data, I had to create unique parsing rules for each data set due to inconsistencies in the spreadsheets. One of my suggestions going forward would be to standardize how they collect and store the data to make it more machine readable. But you are right. Claude Code can sift through it and sort it out, with the right prompting, at the right stage. It is just one more context to manage.
Hi — I took you up on the invitation to try an alternative AI red teaming approach.
I built a multi-agent pipeline (decomposition → investigation → verification → quantification → adversarial testing → synthesis) and ran it against all three interventions where you published detailed AI output: water chlorination, ITNs, and SMC.
Results across three runs:
The most interesting finding was cross-intervention: three structural patterns appeared independently in all three analyses. All three CEAs model dynamic phenomena with static parameters (adherence decay, resistance evolution, efficacy degradation). All three collapse meaningful within-category variation into single aggregate parameters. And the two malaria interventions both lack mechanisms to capture biological adaptation by the target organism.
I wrote up the full results here: tsondo.com/blog/three-interventions-same-structural-patterns/
Phase 1 write-up (methodology explanation): tsondo.com/blog/give-well-red-team/
The full pipeline, prompts, and results are open source: github.com/tsondo/givewell_redteam
Your post mentions you covered six grantmaking areas total. The other three — CMAM, syphilis, and malaria vaccines — could be run through the pipeline as well. It doesn't strictly require your AI output to function; that feeds into novelty filtering and baseline comparison, but the critiques themselves are generated independently. I've reached out separately about this.
Happy to discuss methodology, and happy to hear where you think the pipeline's findings miss the mark — several of the cross-intervention patterns may reflect deliberate modeling choices rather than oversights, and I'd be interested to know which.