BLUF:
* To determine whether AI is ‘improving exponentially’, ‘hitting the wall’, or any other claim which involves a quantity or magnitude (e.g. ‘This model was a big leap/small increment’). We need a good y-axis: an interval scale of AI capability which means +1 unit always represents the same degree of ‘how much better’, in the same way +1 degree Celsius is always the same amount of ‘how much hotter’.
* Yet there is no good y-axis for AI capability. All our...
Public service announcement
1. Applications are now open for our first ever round of the Charity Entrepreneurship Incubation Program dedicated exclusively to animal welfare. Learn more about what’s different this round here and apply...
TL;DR
* The Long-Term Future Fund is closing down, and EA Funds is launching the Transformative AI Fund with a new full-time team.
* The fund's primary focus is technical AI safety and AI governance (including post-AGI governance), as well as supporting fields such as field-building and forecasting. We'll also consider non-GCR implications of transformative AI such as flourishing futures and digital...
At what level of compute spending will AI Safety research be cut off from being considered effective altruism (if any)?
Of course, saving humanity from misaligned AI could be argued to be close to priceless. But how many experiments have a direct theory of change (ToC) of how it's going to mitigate existential risk? Perhaps a general one is fine at low compute ("it only costs $10 and 'control research' is generally thought to be a good research agenda").
But what about $5,000? What about $10,000? These numbers start to compare to or surpass what organizations like Giving What We Can receive from someone who donates for a whole year. It also starts to compete with saving a human life via programmes like those in GiveWell's top charities.
What about $20,000? $30,000? $50,000? Over what time frame are we comfortable spending that much money on compute and still considering that money well (effectively) spent? A year? A month? A single experiment? What kind of discovery is worth $50,000 in AIS research? Should we expect a clear ToC?
I'm very pro AI Safety, but I'm worried about some of the numbers I'm hearing for compute budgets being thrown around (compared to the information gained). I'm wondering - is anyone else is worried about a movement being (famously) concerned with cost effectiveness continuing on this path? Should we encourage more accountability?
Is there a reason why you are focusing on compute and not salaries? The example numbers you use are rather low compared to the yearly salary of a single AIS researcher.
Fair question, I guess some of the numbers I've been hearing can wipe out a (high) yearly salary well within a month (or days).
Perhaps one layer deeper I generally "back" money spent on someone working on AIS full time for a year and think there will probably be some good to come out of that. Although it may happen quickly, it seems that at least some level of thought goes into which positions are needed to fill before the job posting goes out.
However, on individual experiments level, I think the level of scrutiny is much lower/potentially nonexistent.
There seem to be plausible arguments for paying market rate to retain top talent (although you may disagree with them), but I don't really think there's an argument to spend huge sums on experiments without even double-checking if there's a way this can reduce x-risk.
Right now I would give very little marginal philanthropic money to compute-based experiments. AI companies already do a lot of those, and I don't expect them to work anyway. ML experiments are not addressing the fundamental barriers to solving AI misalignment. A core problem is that experiments can't deal with the sharp left turn.
(I would make an exception for CaML-style alignment-to-animals work, but that's not about AI safety as it's normally construed.)
It seems very unlikely to me that there are AI safety projects that are both worth doing but also not worth doing if they spent $10k on compute. The human capital cost is typically far in excess of that.