The AI safety community has grown rapidly since the ChatGPT wake-up call, but available funding doesn’t seem to have kept pace.
However, there’s a more recent dynamic that’s created even better funding opportunities, which I witnessed as a recommender in the most recent SFF grant round.[1]
Most philanthropic (vs. government or industry) AI safety funding (>50%) comes from one source: Good Ventures. But they’ve recently stopped funding several categories of work (my own categories, not theirs):
* Many Republican-leaning think tanks, such as the Foundation for American Innovation.
* “Post-alignment” causes such as digital sentience or regulation of explosive growth.
* The rationality community, including LessWrong, Lightcone, SPARC, CFAR, MIRI.
* High school outreach, such as Non-trivial.
In addition, they are currently not funding (or not fully funding):
* Many non-US think tanks, who don’t want to appear influenced by an American organisation (there’s now probably more than 20 of these).
* They do fund technical safety non-profits like FAR AI, though they’re probably underfunding this area, in part due to difficulty hiring for this area the last few years (though they’ve hired recently).
* Political campaigns, since foundations can’t contribute to them.
* Organisations they’ve decided are below their funding bar for whatever reason (e.g. most agent foundations work). OP is not infallible so some of these might still be worth funding.
* Nuclear security, since it’s on average less cost-effective than direct AI funding, so isn’t one of the official cause areas (though I wouldn’t be surprised if there were some good opportunities there).
This means many of the organisations in these categories have only been able to access a a minority of the available philanthropic capital (in recent history, I’d guess ~25%). In the recent SFF grant round, I estimate they faced a funding bar 1.5 to 3 times higher.
This creates a lot of opportunities for other donors
Does he think all the properties of superintelligent systems that will be relevant for the success of alignment strategies already exist in current systems? That they will exist in systems within the next 4 years? (If not, aren't there extremely important limitations to our ability to empirically test the strategies and figure out if they are likely to work?)