• I found over 10% of fellows did another fellowship after their fellowship. This doesn’t feel enormously efficient.

Seems plausibly fine to me. If you think about a fellowship as a form of "career transition funding + mentorship", it makes sense that this will take ~3 months (one fellowship) for some people, ~6 months (two fellowships) for others, and some either won't transition at all or will transition later. 

I only skimmed the post, but I want to say that it seems good to write posts like this, and I am surprised and slightly disheartened by the limited engagement you have gotten here and in various comments. These seem like very important topics, thanks for working on them! 

Strong upvoted but I figured I should comment as well. I agree with Ryan that the effect on chip supply and AI timelines is one of the most important dynamics, perhaps the most important. It's a bit unclear which direction it points, but I think it probably swamps everything else in its magnitude, and I was sad to see that this post doesn't discuss it. 

I don't have the time right now to find exactly which comparison I am thinking of, but I believe my thought process was basically "the rate of new people getting AI PhDs is relatively slow"; this is of course only one measure for the number of researchers. Maybe I used data similar to that here: https://www.lesswrong.com/s/FaEBwhhe3otzYKGQt/p/AtfQFj8umeyBBkkxa 

Alternatively, AI academics might be becoming more sociable – i.e. citing their friends' papers more, and collaborating more on papers. I don’t find either of the explanations particularly convincing. 

FWIW, I find this somewhat convincing. I think the collaborating on papers part seems like it could be downstream of the expectations of # of paper produced being higher. My sense is that grad students are expected to write more papers now than they used to. One way to accomplish this is to collaborate more. 

I expect if you compared data on the total number of researchers in the AI field and the number of papers, you would see the second rising a little faster than the first (I think I've seen this trend, but don't have the numbers in front of me). If these were rising at the same rate, I think it would basically indicate no change in the difficulty of ideas, because research hours would be scaling with # papers. Again, I expect the trend is actually papers rising faster than people, which would make it seem like ideas are getting easier to find. 

I think other explanations, like the norms and culture around research output expectation, collaboration, how many references you have to have, are more to blame. 

Overall I don't find the methodology presented here, of just looking at number of authors and number of references, to be particularly useful for figuring out if ideas are getting harder to find. It's definitely some evidence, but I think there's quite a few plausible explanations. 

Language models have been growing more capable even faster. But with them there is something very special about the human range of abilities, because that is the level of all the text they are trained on.

This sounds like a hypothesis that makes predictions we can go check. Did you have any particular evidence in mind? This and this come to mind, but there is plenty of other relevant stuff, and many experiments that could be quickly done for specific domains/settings. 

Note that you say "something very special" whereas my comment is actually about a stronger claim like "AI performance is likely to plateau around human level because that's where the data is". I don't dispute that there's something special here, but I think the empirical evidence about plateauing — that I'm aware of — does not strongly support that hypothesis. 

We estimate that

Point of clarification, it seems like FutureSearch is largely powered by calls to AI models. When you say "we", what do you mean? Has a human checked the entire reasoning process that led to the results you present here? 

My understanding of your main claim: If AGI is not a magic problem-solving oracle and is instead limited by needing to be unhobbled and integrated with complex infrastructure, it will be relatively safe for model weights to be available to foreign adversaries. Or at least key national security decision makers will believe that's the case. 

Please correct me if I'm wrong. My thoughts on the above:

Where is this relative safety coming from? Is it from expecting that adversaries aren't going to be able to figure out how to do unhobbling or steal the necessary secrets to do unhobbling? Is it from expecting the unhobbling and building infrastrucure around AIs to be a really hard endeavor? 

The way I'm viewing this picture, AI that can integrate all across the economy, even if that takes substantial effort, is a major threat to global stability and US dominance. 

I guess you can think about the AI-for-productive-purposes supply chain as having two components: Develop the powerful AI model (Initial development), and unhobble it / integrate it in workflows / etc. (Unhobbling/Integration). And you're arguing that the second of these will be an acceptable place to focus restrictions. My intuition says we will want restrictions on both, but more on the part that is most expensive or excludable (e.g., AI chips being concentrated is a point for initial development). It's not clear to me what the cost of both supply chain steps is: Currently, it looks like pre-training costs are higher than fine-tuning costs (point for initial development); but actually integrating AIs across the economy seems very expensive to do, the economy is really big (point for unhobbling/integration) (this depends a lot on the systems at the time and how easy they are to work with). 

Are you all interested in making content or doing altruism-focused work about AI or AI Safety? 

I'll toss out that a lot of folks in the Effective Altruism-adjacent sphere are involved in efforts to make future AI systems safe and beneficial for humanity. If you all are interested in producing content or making a difference around artificial intelligence or AI Safety, there are plenty of people who would be happy to help you e.g., better understand the key ideas, how to convey them, understand funding gaps in the ecosystem, etc. I, for one, would be happy to help with this — I think mitigating extinction risks from advanced AI systems is one of the best opportunities to improve the world, although it's quite different from standard philanthropy. PS I was subscribed to Jimmy back at ~10k :) 

Poor people are generally bad at managing their own affairs and need external guidance

That seems like a particularly cynical way of describing this argument. Another description might be: Individuals are on average fine at identifying ways to improve their lives, and if you think life improvements are heavy tailed, this implies that individual will perform much less well than experts who aim to find the positive tail interventions. 

Here's a similar situation: A high school student is given 2 hours with no distractions and told they should study for a test. How do you think their study method of choice would compare to if a professional tutor designs a studying curriculum for them to follow? My guess is that the tutor designed curriculum is somewhere between 20% and 200% better, depending on the student. Now that's still really far from 719x, but I think it's fine for building the intuition. I wouldn't necessarily say the student is "bad at managing their own affairs", in fact they might be solidly average for students, but I would say they're not an expert at studying, and like other domains, studying benefits from expertise.