Headline finding: I audited 17 AI Safety Talent programmes. Zero of 17 have published any comparison group, rejected-applicant follow-up, matched control or randomisation. Not one. Every programme that mentions a counterfactual does it by asking participants to self-report.
Background
At least $70 million...
Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm?
We’ve seen two large and extremely capable swarms from OpenAI in the last few months:
* 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched...
A year ago I posted that YCombinator (YC) companies didn’t seem to be growing faster since the release of ChatGPT in 2022. I reran that experiment and found that 2023+ YC companies are arguably growing a bit faster than pre-2023 companies (including AfterQuery, ...
At what level of compute spending will AI Safety research be cut off from being considered effective altruism (if any)?
Of course, saving humanity from misaligned AI could be argued to be close to priceless. But how many experiments have a direct theory of change (ToC) of how it's going to mitigate existential risk? Perhaps a general one is fine at low compute ("it only costs $10 and 'control research' is generally thought to be a good research agenda").
But what about $5,000? What about $10,000? These numbers start to compare to or surpass what organizations like Giving What We Can receive from someone who donates for a whole year. It also starts to compete with saving a human life via programmes like those in GiveWell's top charities.
What about $20,000? $30,000? $50,000? Over what time frame are we comfortable spending that much money on compute and still considering that money well (effectively) spent? A year? A month? A single experiment? What kind of discovery is worth $50,000 in AIS research? Should we expect a clear ToC?
I'm very pro AI Safety, but I'm worried about some of the numbers I'm hearing for compute budgets being thrown around (compared to the information gained). I'm wondering - is anyone else is worried about a movement being (famously) concerned with cost effectiveness continuing on this path? Should we encourage more accountability?
Is there a reason why you are focusing on compute and not salaries? The example numbers you use are rather low compared to the yearly salary of a single AIS researcher.
Fair question, I guess some of the numbers I've been hearing can wipe out a (high) yearly salary well within a month (or days).
Perhaps one layer deeper I generally "back" money spent on someone working on AIS full time for a year and think there will probably be some good to come out of that. Although it may happen quickly, it seems that at least some level of thought goes into which positions are needed to fill before the job posting goes out.
However, on individual experiments level, I think the level of scrutiny is much lower/potentially nonexistent.
There seem to be plausible arguments for paying market rate to retain top talent (although you may disagree with them), but I don't really think there's an argument to spend huge sums on experiments without even double-checking if there's a way this can reduce x-risk.
Right now I would give very little marginal philanthropic money to compute-based experiments. AI companies already do a lot of those, and I don't expect them to work anyway. ML experiments are not addressing the fundamental barriers to solving AI misalignment. A core problem is that experiments can't deal with the sharp left turn.
(I would make an exception for CaML-style alignment-to-animals work, but that's not about AI safety as it's normally construed.)
It seems very unlikely to me that there are AI safety projects that are both worth doing but also not worth doing if they spent $10k on compute. The human capital cost is typically far in excess of that.