BLUF:
* To determine whether AI is ‘improving exponentially’, ‘hitting the wall’, or any other claim which involves a quantity or magnitude (e.g. ‘This model was a big leap/small increment’). We need a good y-axis: an interval scale of AI capability which means +1 unit always represents the same degree of ‘how much better’, in the same way +1 degree Celsius is always the same amount of ‘how much hotter’.
* Yet there is no good y-axis for AI capability. All our...
TL;DR
* The Long-Term Future Fund is closing down, and EA Funds is launching the Transformative AI Fund with a new full-time team.
* The fund's primary focus is technical AI safety and AI governance (including post-AGI governance), as well as supporting fields such as field-building and forecasting. We'll also consider non-GCR implications of transformative AI such as flourishing futures and digital...
Public service announcement
1. Applications are now open for our first ever round of the Charity Entrepreneurship Incubation Program dedicated exclusively to animal welfare. Learn more about what’s different this round here and apply...
Zero-bounded vs negative-tail risks
(adapted from a comment on LessWrong)
In light of the FTX thing, maybe a particularly important heuristic is to notice cases where the worst-case is not lower-bounded at zero. Examples:
Not that you should definitely not do things that potentially have large-negative downsides, but you can be a lot more willing to experiment when the downside is capped at zero.
Indeed, a good norm in many circumstances is to do lots of exploration and iteration. This is how science, software development, and most research happens. Things get a lot tricker when even this stage has potential deep harms -- as in research with advanced AI. (Or, more boundedly & fixably, infohazard risks from x- and s-risk reduction research.)
In practice, people will argue about what counts as effectively zero harm, vs nonzero. Human psychology, culture, and institutions are sticky, so exploration that naively looks zero-bounded can have harm potential via locking in bad ideas or norms. I think that harm is often fairly small, but it might be both important and nontrivial to notice when it's large -- e.g., which new drugs are safe to explore for a particular person? caffeine vs SSRIs vs weed vs alcohol vs opioids...
(Note that the "zero point" I'm talking about here is an outcome where you've added zero value to the world. I'm thinking of the opportunity cost of the time or money you invested as a separate term.)