(This comment is written quickly, without too much thought. Sorry)
It's always been odd to me what grant-making orgs mean when they say now is a good time to hire more grant makers or they're grantmaker-constrained. When is it not a good time to hire more grant makers? What are the signs that we are actually grantmaker-constrained, and that scaling up grantmakers would clearly lead to more impact? And then does our current situation match one of those situations?
Scenario A: There are many strong applicants, but not very much money. We need grant makers to play the role of figuring out how to prioritize who to give money to. It seems like grant makers are pretty important in this situation, but also maybe you don't need that many of them.
Scenario B: There are many strong applicants and lots of money. I would assume we don't need very many grant makers because the existing grant makers can just have a low-ish bar and this be fine because there are many strong applicants. But there's an argument that we should have more grant makers in this situation in order to make sure we don't fund net negative projects. Also, the overall demand for grant making is simply higher, both from applicants and from higher-up funders. So I would imagine grant-making orgs would try to hire in this environment.
Scenario C: There are not very many strong applicants and not very much money. I guess the grant makers should try to do direct work and solve the fact that there aren't very many strong applicants. Maybe they should try to fundraise.
Scenario D: There are not very many strong applicants but lots of money. It's not obvious what grantmakers should be doing in this situation. Sounds like there isn't much work to do—grantmakers should stop hiring and their people should switch to direct work. Or maybe not?
As far as I can tell, Scenario D is the closest to describing the current situation where CG has lots of money and some orgs/applicants that CG is happy to fund; but CG is not able to give out as much money as it would like to because there are not enough obviously strong applicants. There is an excess amount of money compared to projects CG thinks will succeed/be good. It seems that CG's response to this scenario is what Luke calls "3 strategy-driven creation of new grantees." It's also sometimes called active grant making.
As our AI timelines shorten, we've shifted more focus to (3) since many critical gaps remain that we haven't gotten good applications for. We've had strong success with this so far, but the strategy work and headhunting of (3) requires far more staff capacity per dollar moved than (1) or (2) do, so we need to grow our grantmaker capacity as quickly as we can.
Personally, if I were considering joining, I would want much more evidence about how successful active grant making has been and how successful you expect it to be in the future. It seems like "whether active grant making works" is the key question for how good additional grantmakers are. My prior is that active grant making is very hard because most field building is very hard in my experience. In fact, maybe instead of active grant making, I would just go harder on funding normal field building efforts?
Perhaps CG or others have an internal production function relating applicants, funding, and grantmakers together to create impact. Without such a function, I think we should be skeptical when grantmakers say that they need more capacity. ~Everyone thinks they need more capacity. While I'm pretty sure Scenario D describes the current environment, I don't actually know what the correct hiring reaction to that environment is. Active grant making is one answer and a reasonable one, but it should require having confident beliefs about how effective this is. I have not myself seen evidence that justifies such beliefs.
As someone reviewing applications, how do you evaluate independent research produced outside of a fellowship or academic context?
I'm not sure how to answer this. I try to evaluate all work based on its quality; whether a project was single-authored also matters a fair amount (and high quality single-author work is an especially strong signal).
Is there a threshold of rigor or novelty below which it hurts more than it helps to include it?
Maybe for rigor, probably not for novelty. Applicants and other researchers should of course be up front about what are their contributions.
And do you have suggestions for how early-career people in this transition can get lightweight feedback on research directions before investing weeks into a project?
Weeks sound like it might be a lot. I encourage people to do Apart Research Sprints or other Hackathon-style things which are shorter. I'm not really sure about getting lightweight feedback. In my experience, when junior people ask me for feedback on a project idea, the project idea is usually too broad or vague for me to know if it's a good project, and they have usually put less than 30 min of effort into it. So maybe my advice there is something like "if you are going to ask a more established researcher for feedback on your project plan, you should have already put a couple hours into the project, including surveying the relevant literature, coming up with a detailed project plan, and doing a little bit of de-risking". I'm not sure, maybe that's more intense than I endorse. Fortunately, even without the goal of getting feedback from somebody else, these are useful steps to begin a project with.
I will also note that prior work does not always have to be extremely relevant. Academia exists and is by far the place where the most people learn research skills.
Sorry, it's a bummer to be scooped or even just feel scooped.
Most academic fields have, I think, less cross-org coordination than AI governance. I would be hesitant about trying to do generally more cross-org coordination in this space given that it's a departure from (what I view as) the norm in other fields.
As I am reading applications to hire AI governance researchers, one of my big questions is "has this person done relevant work before, successfully?". I don't think it would be much of a mark against that work if it was also similar to other work that was released at the same time, as long as it didn't seem like there was plagiarism and did seem like there were novel contributions.
Relatedly, multiple researchers teams taking independent stabs at the same question is often useful for reaching a higher quality of overall work, as they sometimes come up with different ideas/emphases/etc.
Some researchers have said (but I'm unsure where I land) that you almost never actually get scooped. Usually projects are a bit different in a way that is important and that you can emphasize in your output. Also you can boostrap from that work to make your project even better (but again, be clear about your original contributions vs. others').
I found over 10% of fellows did another fellowship after their fellowship. This doesn’t feel enormously efficient.
Seems plausibly fine to me. If you think about a fellowship as a form of "career transition funding + mentorship", it makes sense that this will take ~3 months (one fellowship) for some people, ~6 months (two fellowships) for others, and some either won't transition at all or will transition later.
I only skimmed the post, but I want to say that it seems good to write posts like this, and I am surprised and slightly disheartened by the limited engagement you have gotten here and in various comments. These seem like very important topics, thanks for working on them!
Strong upvoted but I figured I should comment as well. I agree with Ryan that the effect on chip supply and AI timelines is one of the most important dynamics, perhaps the most important. It's a bit unclear which direction it points, but I think it probably swamps everything else in its magnitude, and I was sad to see that this post doesn't discuss it.
I don't have the time right now to find exactly which comparison I am thinking of, but I believe my thought process was basically "the rate of new people getting AI PhDs is relatively slow"; this is of course only one measure for the number of researchers. Maybe I used data similar to that here: https://www.lesswrong.com/s/FaEBwhhe3otzYKGQt/p/AtfQFj8umeyBBkkxa
Alternatively, AI academics might be becoming more sociable – i.e. citing their friends' papers more, and collaborating more on papers. I don’t find either of the explanations particularly convincing.
FWIW, I find this somewhat convincing. I think the collaborating on papers part seems like it could be downstream of the expectations of # of paper produced being higher. My sense is that grad students are expected to write more papers now than they used to. One way to accomplish this is to collaborate more.
I expect if you compared data on the total number of researchers in the AI field and the number of papers, you would see the second rising a little faster than the first (I think I've seen this trend, but don't have the numbers in front of me). If these were rising at the same rate, I think it would basically indicate no change in the difficulty of ideas, because research hours would be scaling with # papers. Again, I expect the trend is actually papers rising faster than people, which would make it seem like ideas are getting easier to find.
I think other explanations, like the norms and culture around research output expectation, collaboration, how many references you have to have, are more to blame.
Overall I don't find the methodology presented here, of just looking at number of authors and number of references, to be particularly useful for figuring out if ideas are getting harder to find. It's definitely some evidence, but I think there's quite a few plausible explanations.
Language models have been growing more capable even faster.But with them there is something very special about the human range of abilities, because that is the level of all the text they are trained on.
This sounds like a hypothesis that makes predictions we can go check. Did you have any particular evidence in mind? This and this come to mind, but there is plenty of other relevant stuff, and many experiments that could be quickly done for specific domains/settings.
Note that you say "something very special" whereas my comment is actually about a stronger claim like "AI performance is likely to plateau around human level because that's where the data is". I don't dispute that there's something special here, but I think the empirical evidence about plateauing — that I'm aware of — does not strongly support that hypothesis.
Point of clarification, it seems like FutureSearch is largely powered by calls to AI models. When you say "we", what do you mean? Has a human checked the entire reasoning process that led to the results you present here?
(This comment is written quickly, without too much thought. Sorry)
It's always been odd to me what grant-making orgs mean when they say now is a good time to hire more grant makers or they're grantmaker-constrained. When is it not a good time to hire more grant makers? What are the signs that we are actually grantmaker-constrained, and that scaling up grantmakers would clearly lead to more impact? And then does our current situation match one of those situations?
Scenario A: There are many strong applicants, but not very much money. We need grant makers to play the role of figuring out how to prioritize who to give money to. It seems like grant makers are pretty important in this situation, but also maybe you don't need that many of them.
Scenario B: There are many strong applicants and lots of money. I would assume we don't need very many grant makers because the existing grant makers can just have a low-ish bar and this be fine because there are many strong applicants. But there's an argument that we should have more grant makers in this situation in order to make sure we don't fund net negative projects. Also, the overall demand for grant making is simply higher, both from applicants and from higher-up funders. So I would imagine grant-making orgs would try to hire in this environment.
Scenario C: There are not very many strong applicants and not very much money. I guess the grant makers should try to do direct work and solve the fact that there aren't very many strong applicants. Maybe they should try to fundraise.
Scenario D: There are not very many strong applicants but lots of money. It's not obvious what grantmakers should be doing in this situation. Sounds like there isn't much work to do—grantmakers should stop hiring and their people should switch to direct work. Or maybe not?
As far as I can tell, Scenario D is the closest to describing the current situation where CG has lots of money and some orgs/applicants that CG is happy to fund; but CG is not able to give out as much money as it would like to because there are not enough obviously strong applicants. There is an excess amount of money compared to projects CG thinks will succeed/be good. It seems that CG's response to this scenario is what Luke calls "3 strategy-driven creation of new grantees." It's also sometimes called active grant making.
Personally, if I were considering joining, I would want much more evidence about how successful active grant making has been and how successful you expect it to be in the future. It seems like "whether active grant making works" is the key question for how good additional grantmakers are. My prior is that active grant making is very hard because most field building is very hard in my experience. In fact, maybe instead of active grant making, I would just go harder on funding normal field building efforts?
Perhaps CG or others have an internal production function relating applicants, funding, and grantmakers together to create impact. Without such a function, I think we should be skeptical when grantmakers say that they need more capacity. ~Everyone thinks they need more capacity. While I'm pretty sure Scenario D describes the current environment, I don't actually know what the correct hiring reaction to that environment is. Active grant making is one answer and a reasonable one, but it should require having confident beliefs about how effective this is. I have not myself seen evidence that justifies such beliefs.
I'm not sure how to answer this. I try to evaluate all work based on its quality; whether a project was single-authored also matters a fair amount (and high quality single-author work is an especially strong signal).
Maybe for rigor, probably not for novelty. Applicants and other researchers should of course be up front about what are their contributions.
Weeks sound like it might be a lot. I encourage people to do Apart Research Sprints or other Hackathon-style things which are shorter. I'm not really sure about getting lightweight feedback. In my experience, when junior people ask me for feedback on a project idea, the project idea is usually too broad or vague for me to know if it's a good project, and they have usually put less than 30 min of effort into it. So maybe my advice there is something like "if you are going to ask a more established researcher for feedback on your project plan, you should have already put a couple hours into the project, including surveying the relevant literature, coming up with a detailed project plan, and doing a little bit of de-risking". I'm not sure, maybe that's more intense than I endorse. Fortunately, even without the goal of getting feedback from somebody else, these are useful steps to begin a project with.
I will also note that prior work does not always have to be extremely relevant. Academia exists and is by far the place where the most people learn research skills.
A couple of my thoughts, written quickly:
Seems plausibly fine to me. If you think about a fellowship as a form of "career transition funding + mentorship", it makes sense that this will take ~3 months (one fellowship) for some people, ~6 months (two fellowships) for others, and some either won't transition at all or will transition later.
I only skimmed the post, but I want to say that it seems good to write posts like this, and I am surprised and slightly disheartened by the limited engagement you have gotten here and in various comments. These seem like very important topics, thanks for working on them!
Strong upvoted but I figured I should comment as well. I agree with Ryan that the effect on chip supply and AI timelines is one of the most important dynamics, perhaps the most important. It's a bit unclear which direction it points, but I think it probably swamps everything else in its magnitude, and I was sad to see that this post doesn't discuss it.
I don't have the time right now to find exactly which comparison I am thinking of, but I believe my thought process was basically "the rate of new people getting AI PhDs is relatively slow"; this is of course only one measure for the number of researchers. Maybe I used data similar to that here: https://www.lesswrong.com/s/FaEBwhhe3otzYKGQt/p/AtfQFj8umeyBBkkxa
FWIW, I find this somewhat convincing. I think the collaborating on papers part seems like it could be downstream of the expectations of # of paper produced being higher. My sense is that grad students are expected to write more papers now than they used to. One way to accomplish this is to collaborate more.
I expect if you compared data on the total number of researchers in the AI field and the number of papers, you would see the second rising a little faster than the first (I think I've seen this trend, but don't have the numbers in front of me). If these were rising at the same rate, I think it would basically indicate no change in the difficulty of ideas, because research hours would be scaling with # papers. Again, I expect the trend is actually papers rising faster than people, which would make it seem like ideas are getting easier to find.
I think other explanations, like the norms and culture around research output expectation, collaboration, how many references you have to have, are more to blame.
Overall I don't find the methodology presented here, of just looking at number of authors and number of references, to be particularly useful for figuring out if ideas are getting harder to find. It's definitely some evidence, but I think there's quite a few plausible explanations.
This sounds like a hypothesis that makes predictions we can go check. Did you have any particular evidence in mind? This and this come to mind, but there is plenty of other relevant stuff, and many experiments that could be quickly done for specific domains/settings.
Note that you say "something very special" whereas my comment is actually about a stronger claim like "AI performance is likely to plateau around human level because that's where the data is". I don't dispute that there's something special here, but I think the empirical evidence about plateauing — that I'm aware of — does not strongly support that hypothesis.
Point of clarification, it seems like FutureSearch is largely powered by calls to AI models. When you say "we", what do you mean? Has a human checked the entire reasoning process that led to the results you present here?