Here's an idea on how funders in AI safety and governance could help applicants improve their applications and projects: Share statistics!
(What follows is a text I'm also sending directly to a funder.)
I understand you aren't really able to give individualized feedback. Though, as applicants, it would be really helpful to have some more clarity. I think you'd like to give more feedback, if you were able. In thinking about this, here's an idea I had.
You could create a score chart, where for every application you keep numerical or binary scores on the reasons it did well or not in the evaluation. Then you can release some statistics publicly.
You could for example be able to say things like:
"30% of applications were filtered out because they were not pursuing our AI safety objectives"
"20% we would fund if we had more money"
"40% we'd investigate more deeply if we had more money"
"x% were too hard to understand"
"x% had some flaw in the theory of change"
"x% ..."
(I picked more negative than positive slices in these examples, but the positive side is just as interesting.)
In fact, some of these statistics would be helpful for donors to <funder>.
The scores should track some pretty different kinds of dimensions.
Would this be hard to do? I think there can be some pretty different levels of ambition here and some are pretty easy. Releasing just a few pieces of statistics for example might be possible to do based on your current tracking.
Potentially, you could even let people request access to some subset of these scores for their application specifically. Something like, the email used to send the application can send an email to request an automated response with the scores you'd be prepared to divulge.
Could this create more opportunities for gaming? Well yes, but assuming your criteria are actually good proxies for value, then you also achieve: (1) Better applications (so you get to grant valuable things you might filter otherwise), and (2) Better projects (people make their projects have better theories of change etc).
The lack of two-way communication in funding seems like a large missed opportunity to me!
Grantmakers, even when you don't grant, wield a lot of influence. You shape incentives in the ecosystem.
From an altruistic cause prioritization perspective, existential risk seems to require longtermism, including potentially fanatical views (see Christian Tarsney, Rethink Priorities). It seems like we should give some weight to causes that are non-fanatical.
This seems clearly incorrect to me. I'm surprised to see this claim fronted prominently inside a highly upvoted comment. It also strikes me as uncharitable by invoking the "fanatical" frame.
Prioritizing x-risk merely requires thinking the risk of existential catastrophe is close enough in time.
It’s a widely held belief in the existential risk reduction community that we are likely to see a great technological transformation in the next 50 years. A technological transformation will either cause flourishing, existential catastrophe, or other forms of large change for humanity. The next 50 years will matter directly for most currently living people. Existential risk reduction and handling the technological transformation are therefore not just questions of the ‘far future’ or the ‘long-term’; it is also a ‘near-term’ concern.
Note that I wrote this short piece in 2020, before Chat GPT. I used "50 years" to, even at the point, work with a conservative time frame. Back then in 2020, I might have used 20-30 years personally. Now, in 2025, I might use 10 years personally.
This looks incorrect to me. Factory farming interventions winning over x-risk interventions requires both thinking (1) that animals have moral weight not too far from that of humans, and (2) that the amount of suffering in factory farming is more morally important than increasing the chances of humanity and life in general of surviving at all. These assumptions are not shared by everyone in EA.
Making the writing easy for myself: What's your response to Carl Shulman's take? Which is, that pushing for a pause too early might spoil the chance of getting people to agree to a pause when it would matter the most: at pivotal points where AI improvement is happening tremendously quickly. Carl Shulman on the 80k podcast.
X-risk: yes. The idea of fast AI development: yes. Knowing the phrase "takeoff speed"? No. For sure, this also depends a bit on the type of role and seniority. "Moral patienthood" strikes me as one of those terms where if someone is interested in one of our jobs, they will likely get the idea, but they might not know the term "moral patienthood". So let's note here that I wrote "language", and you wrote "concepts", and these are not the same. One of the distinctions I care about is that people understand, or can easily come to understand the ideas/concepts. I care less what specific words they use.
Digressing slightly, note that using specific language is a marker for group belonging, and people seem to find pleasure in using in-group language as this signals group belonging, even if there exists standard terms for the concepts. Oxytocin creates internal group belonging and at the same time exclusion towards outsiders. Language can do some of the same.
So yes, it's important to me that people understand certain core concepts. But again, don't overindex on me. I should've maybe clarified the following better in my first comment: I've personally thought that EA/AI safety groups have done a bit too much in-group hiring, so I set out how to figure out how to hire people more widely, and retain the same mission focus regardless.
Speaking as a hiring manager at a small group in AI safety/governance who made an effort to not just hire insiders (it's possible I'm in a minority -- don't take my take for gospel if you're looking for a job), it's not important to me that people know a lot about in-group language, people, or events around AI safety. It is very important to me that people agree with foundational ideas such as to actually be impact-focused and to take short-ish AI timelines and AI risk seriously and have thought about it seriously.
Thanks Holly. I agree that fixating on just trying to answer the "AI timelines" question won't be productive for most people. Though, we all need to come to terms with it somehow. I like your callout for "timeline-robust interventions". I think that's a very important point. Though I'm not sure that implies calling your representatives.
I disagree that "we know what we need to know". To me, the proper conversation about timelines isn't just "when AGI", but rather, "at what times will a number of things happen", including various stages of post-AGI technology, and AI's dynamics with the world as a whole. It incorporates questions like "what kinds of AIs will be present".
This allows us to make more prudent interventions: What technical AI safety and AI governance you need depends on the nature of the AI that will be built. Important AI to address isn't just orthogonality thesis-driven paperclip maximizers.
I think seeing the way AI is emerging, that it's clear some classic AI safety challenges are not as relevant anymore. For example, it seems to me that "value learning" is looking much easier than classic AI safety advocates thought. But versions of many classic AI safety challenges are still relevant. The same issue remains: if we can't verify that something vastly more intelligent than us is acting in our interests, then we are in peril.
I don't think it would be right if everyone would be occupied with such AI timelines and AI scenarios questions, but I think they deserve very strong efforts. If you are trying to solve a problem, the most important thing to get right is what problem you're trying to solve. And what is the problem of AI safety? That depends on what kind of AI will be present in the world and what humans will be doing with it.
Thank you, Zachary and team! I'm happy to see CEA take an ambitious stance. Your goals make perfect sense to me. The EA community is a very important one and your stewardship is needed.
I like this principles-first approach! I think it's really valuable to have a live discussion that starts from "How do we do the most good?", even if I am kind of all-in on one cause. (Kind of: I think most causes tie together: making the future turn out well.) I think it'd be a valuable use of the time of you folks to try and clarify and refine your approach, philosophy, and incentives further, using the comments here as one input.
Here's an idea on how funders in AI safety and governance could help applicants improve their applications and projects: Share statistics! (What follows is a text I'm also sending directly to a funder.)
I understand you aren't really able to give individualized feedback. Though, as applicants, it would be really helpful to have some more clarity. I think you'd like to give more feedback, if you were able. In thinking about this, here's an idea I had.
You could create a score chart, where for every application you keep numerical or binary scores on the reasons it did well or not in the evaluation. Then you can release some statistics publicly.
You could for example be able to say things like:
(I picked more negative than positive slices in these examples, but the positive side is just as interesting.)
In fact, some of these statistics would be helpful for donors to <funder>. The scores should track some pretty different kinds of dimensions.
Would this be hard to do? I think there can be some pretty different levels of ambition here and some are pretty easy. Releasing just a few pieces of statistics for example might be possible to do based on your current tracking.
Potentially, you could even let people request access to some subset of these scores for their application specifically. Something like, the email used to send the application can send an email to request an automated response with the scores you'd be prepared to divulge.
Could this create more opportunities for gaming? Well yes, but assuming your criteria are actually good proxies for value, then you also achieve: (1) Better applications (so you get to grant valuable things you might filter otherwise), and (2) Better projects (people make their projects have better theories of change etc).
The lack of two-way communication in funding seems like a large missed opportunity to me!
Grantmakers, even when you don't grant, wield a lot of influence. You shape incentives in the ecosystem.
This seems clearly incorrect to me. I'm surprised to see this claim fronted prominently inside a highly upvoted comment. It also strikes me as uncharitable by invoking the "fanatical" frame.
Prioritizing x-risk merely requires thinking the risk of existential catastrophe is close enough in time.
https://forum.effectivealtruism.org/posts/X5aJKx3f6z5sX2Ji4/the-far-future-is-not-just-the-far-future
Note that I wrote this short piece in 2020, before Chat GPT. I used "50 years" to, even at the point, work with a conservative time frame. Back then in 2020, I might have used 20-30 years personally. Now, in 2025, I might use 10 years personally.
This looks incorrect to me. Factory farming interventions winning over x-risk interventions requires both thinking (1) that animals have moral weight not too far from that of humans, and (2) that the amount of suffering in factory farming is more morally important than increasing the chances of humanity and life in general of surviving at all. These assumptions are not shared by everyone in EA.
Making the writing easy for myself: What's your response to Carl Shulman's take? Which is, that pushing for a pause too early might spoil the chance of getting people to agree to a pause when it would matter the most: at pivotal points where AI improvement is happening tremendously quickly. Carl Shulman on the 80k podcast.
You may have responded to this before. Feel free to provide a link.
This page is giving me a 404 right now: https://pauseai.info/mitigating-pause-failures
X-risk: yes. The idea of fast AI development: yes. Knowing the phrase "takeoff speed"? No. For sure, this also depends a bit on the type of role and seniority. "Moral patienthood" strikes me as one of those terms where if someone is interested in one of our jobs, they will likely get the idea, but they might not know the term "moral patienthood". So let's note here that I wrote "language", and you wrote "concepts", and these are not the same. One of the distinctions I care about is that people understand, or can easily come to understand the ideas/concepts. I care less what specific words they use.
Digressing slightly, note that using specific language is a marker for group belonging, and people seem to find pleasure in using in-group language as this signals group belonging, even if there exists standard terms for the concepts. Oxytocin creates internal group belonging and at the same time exclusion towards outsiders. Language can do some of the same.
So yes, it's important to me that people understand certain core concepts. But again, don't overindex on me. I should've maybe clarified the following better in my first comment: I've personally thought that EA/AI safety groups have done a bit too much in-group hiring, so I set out how to figure out how to hire people more widely, and retain the same mission focus regardless.
Speaking as a hiring manager at a small group in AI safety/governance who made an effort to not just hire insiders (it's possible I'm in a minority -- don't take my take for gospel if you're looking for a job), it's not important to me that people know a lot about in-group language, people, or events around AI safety. It is very important to me that people agree with foundational ideas such as to actually be impact-focused and to take short-ish AI timelines and AI risk seriously and have thought about it seriously.
Thanks Holly. I agree that fixating on just trying to answer the "AI timelines" question won't be productive for most people. Though, we all need to come to terms with it somehow. I like your callout for "timeline-robust interventions". I think that's a very important point. Though I'm not sure that implies calling your representatives.
I disagree that "we know what we need to know". To me, the proper conversation about timelines isn't just "when AGI", but rather, "at what times will a number of things happen", including various stages of post-AGI technology, and AI's dynamics with the world as a whole. It incorporates questions like "what kinds of AIs will be present".
This allows us to make more prudent interventions: What technical AI safety and AI governance you need depends on the nature of the AI that will be built. Important AI to address isn't just orthogonality thesis-driven paperclip maximizers.
I think seeing the way AI is emerging, that it's clear some classic AI safety challenges are not as relevant anymore. For example, it seems to me that "value learning" is looking much easier than classic AI safety advocates thought. But versions of many classic AI safety challenges are still relevant. The same issue remains: if we can't verify that something vastly more intelligent than us is acting in our interests, then we are in peril.
I don't think it would be right if everyone would be occupied with such AI timelines and AI scenarios questions, but I think they deserve very strong efforts. If you are trying to solve a problem, the most important thing to get right is what problem you're trying to solve. And what is the problem of AI safety? That depends on what kind of AI will be present in the world and what humans will be doing with it.
Thank you, Zachary and team! I'm happy to see CEA take an ambitious stance. Your goals make perfect sense to me. The EA community is a very important one and your stewardship is needed.
I like this principles-first approach! I think it's really valuable to have a live discussion that starts from "How do we do the most good?", even if I am kind of all-in on one cause. (Kind of: I think most causes tie together: making the future turn out well.) I think it'd be a valuable use of the time of you folks to try and clarify and refine your approach, philosophy, and incentives further, using the comments here as one input.
BERI is doing an awesome service for university-affiliated groups, I hope more will take advantage of it!