This has a strong flavour of something I see a lot from AI safety grantees. It's something along the lines of 'something about our field/org is unique and therefore the usual rules of the road don't apply'.

For example, it is almost universally accepted that independent analyses are better than in-house analyses, for obvious reasons. I'm very surprised to see you make the opposite claim, which runs counter to widely accepted wisdom. The author points out many ways in which the current monitoring is weak, prone to bias, incomplete etc.

Importantly, Coefficient Giving is a grantmaker and UHNW advisor, not an impact evaluator. Their teams aim to assess the best grants of the available options in order to achieve a goal. This is related to but distinct from evaluating the counterfactual impact of an organisation. (As an analogy, GiveWell assesses grant opportunities including ORSZ distribution, but it doesn't measure the impact of ORSZ distribution. It relies on specialist measurement and evaluation of ORSZ distribution.)

I think the author laid out what would be interesting and useful. It would be reassuring to have more confidence in the counterfactual value of tens of millions of dollars of grants.

This is a very apposite and useful post (it correlates with a small set of survey results from Future Matters here - https://future-matters.org/updates/legibility-in-the-coming-wave-anonymous-philanthropists-barriers-to-funding-ai-governance/ )

I wrote up some of my experiences trying to assess AI safety and governance programmes here: https://fundinganthropalypse.com/p/donors-dont-check-whether-their-grants

In general, the standard of M&E is very low, and independent views like this are very helpful. As you point out, it would be relatively low lift to produce better quality data. I suspect that short timelines lead to a lot of leadership and grantmaking on trust, as people think something like 'given the stakes, we should just throw money and time at our best bets'.

Thanks for writing it.

It should be possible fairly easily to get separate funders to use the same indicators - the right MEL plan should address the 'true progress' of the project, and so should be of interest to all funders. 

Given the power dynamics, it might be easier for one funder to negotiate this with another, rather than you having to do this as the project.

I have sat on the charity side and I actually do think this can be done in under half a day, at least if the indicators don't require bespoke data curation. The plan we included as an example above would take <10 minutes to report on ('passed', 'didn't pass', 'passed'). Of course, most grantees then want to editorialise on top ('we didn't achieve X because of Y'), so maybe I am being a bit simplistic.

Overall, I agree that funders should use the same indicators more.

These are certainly a good start, yes.

In one or two cases, I would argue that they could be improved. For example, I don't think placing people into target institutions for 24 months is enough. I think you need to show that the people you place are able to influence policy or decision-making. 

I recall one example where an org reported that one of the people they placed had written the memo on an AI safety decision that went to the National Security Council. Unfortunately, this was about a week after the Administration had completely ignored the memo and done the exact opposite of what it recommended.

Nonetheless, these indicators are a decent first stab, and could be refined to be quite effective with a small amount of work/expert input/negotiation.

Nice to know the word of day calendar is paying off.

I think donors should use reasonable indicators to judge evidence if an RCT isn't available, while also being clear with themselves that this makes the grant higher risk. This is relatively easy in global health, where most places are doing decent monitoring, evaluation and learning, and running pre/post studies or difference-in-difference studies of their work. It's harder in other cause areas, as I just posted about.

If donors have the money, I would love to see more funding specifically for RCTs. I think a funder or group of funders dedicated to this could do an enormous amount of good. You do need a significant amount of capital, though - it's not uncommon for health RCTs to run to several million dollars per study.

Of course, my mid-stage global health fund aims to do good grantmaking in this gap, but we would need more capital to be the main funder of RCTs.

As you say, I wouldn't recommend going all-in on the single max-EV idea, but I would recommend backing a spread of them.

I think these a very defensible recommendations but, without knowing anything about a specific donor, I didn't want to give a maximalist recommendation. If the donor has the appetite and tolerance for it, these could be exciting options.

I would like to see this sort of diversity of approaches/opinions/lenses, yes.

Their announcement doesn't mention that as a motivation, although it would indirectly factor into their assessment of the field's 'room for more funding'.

https://coefficientgiving.org/research/increasing-our-2026-allocation-to-givewells-recommendations-to-1-billion/

This is a really useful counterweight to the prevailing thinking.

How do you think about trading off the downsides of growing too late (if the money materialises) versus growing too quickly (if it doesn't)?

My sense so far has been that the field is kind of doing business-as-usual, or at least was until cG gave GiveWell a billion dollars to get ahead of an expected windfall, anyway. A lot of places don't seem to have hired much or made firm moves as if they really expect the money to land (with a couple of notable exceptions).