Thesis: Artificial general intelligence (AGI) is far away because general intelligence requires the ability to learn quickly and efficiently. General intelligence is not just a large set of skills learned inefficiently. Current AI systems learn incredibly slowly and inefficiently. Scaling them up won’t fix that.
Preamble: AGI is less than 0.1% likely by 2032
My current view is that there is significantly less than a 1 in 1,000 chance of artificial general intelligence (AGI) being developed before the end of 2032, with upwards of 95% confidence. Part of the reason I think AGI within 7 years[1] is so unlikely and why my confidence is so high is that in the sort of accounts that attempt to show how we go from current AI systems to AGI in such a short time, people stipulate things that are impossible based on current knowledge of how AI works or their accounts contradict themselves (which also makes it impossible for them to actually happen). In response to objections to these accounts that attempt to point out these difficulties, there have been no good answers. This leads me to conclude that my first impression is correct that these accounts are indeed impossible.
If I stopped to really think about it, my best guess of the probability of AGI by the end of 2032 might be less than even 1 in 10,000 and my confidence might be 99% or more. My impulse to make these numbers more cautious and conservative (higher probability, lower confidence) comes only from a desire to herd toward the predictions of other people, but a) this is a bad practice in the first place and b) I find the epistemic practices of people who believe very near-term AGI is very likely with high confidence tend to have alarming problems (e.g. being blithely unaware of opposing viewpoints, even those held by a large majority of experts — I’m not talking about disagreeing with experts, but not even knowing that experts disagree, let alone why), which in other contexts most reasonable people would find disqualifying. That makes me think I should disregard those predictions and think about the prediction I would make if those predictions didn’t exist.
Moreover, if I change the reference class from, say, people in the Effective Altruism Forum filter bubble to, say, AI experts or superforecasters, the median year for AGI gets pushed out past 2045, so my prediction starts to look like a lot less of an outlier. But I don’t want to herd toward those forecasts, either.
Humans learn much faster than AI
DeepMind’s AI agent AlphaStar was able to attain competence at StarCraft II competitive with the game’s top-tier players. This is an impressive achievement, but it required a huge amount of training relative to what a human requires to attain the same level of skill or higher. AlphaStar was unable to reinforcement learn StarCraft II from scratch — the game was too complex — so it first required a large dataset of human play (supplied to DeepMind by the game’s developer, Blizzard) to imitation learn from. After bootstrapping with imitation learning, AlphaStar did 60,000 years of reinforcement learning via self-play to reach Grandmaster level. How does this compare to how fast humans learn?
Most professional StarCraft II players are in their 20s or 30s. The age at which they first achieved professional status will also be less, on average, than their current ages. But to make my point very clear, I’ll just overestimate by a lot and assume that, on average, StarCraft II players reach professional status at age 35. I’ll also dramatically overestimate and say that, from birth until age 35, professional players have spent two-thirds of their time (16 hours a day, on average) playing StarCraft II. This comes out to 23 years of StarCraft II practice to reach professional status. Excluding the imitation learning and just accounting for the self-play, this means humans learn StarCraft II more than 2,500x faster than AlphaStar did.
The domain of StarCraft II also helps show why the speed of learning is also relevant in terms of what it means to have a skill. The strategy and tactics of StarCraft II are continually evolving. Unlike testing a skill against a frozen benchmark that never changes, opponents in StarCraft II respond to what you do and adapt.
Anecdotally, some top-tier StarCraft II players have conjectured that the reason AlphaStar’s win rate eventually plateaued at a certain point within the Grandmaster League is that there are few enough Grandmasters (only 200 per geographical region or 1,000 worldwide) that these players were able to face AlphaStar again and again and learn how to beat it.
It’s one thing for the AI to beat a professional player in a best of 5 matchup, as happened on two or three occasions. (Although the details are a bit complicated and only one of these represented fair, typical competitive play conditions.) A best of 5 matchup in one sitting favours the AI. A best of 100 matchup over a month would favour the human. AlphaStar is not continually learning and, even if it were, it learns far too slowly — more than 2,500x more slowly than a human — to keep up. Humans can learn how AlphaStar plays, exploit its weaknesses, and turn around their win rate against AlphaStar. This is a microcosm of how general intelligence works. General intelligence means learning fast.
Scaling can’t compensate for AI’s inefficiency
It is generally not disputed that AI learns far more slowly and more inefficiently than humans. No one seems to try to claim that the speed or efficiency at which AI learns is improving fast enough to make up the gap anytime soon, either. Rather, the whole argument for the high likelihood of near-term AGI relies on that efficiency disadvantage being overcome by exponentially growing amounts of training data and training compute. So what if it takes AI more than 2,500x more data or experience than humans to learn the same skills? We’ll just give the AI that 2,500x more (or whatever it is) and then we’ll be even! But that is not how it works.
First, this is physically impossible. According to calculations by the philosopher (and co-founder of effective altruism) Toby Ord, just scaling the reinforcement learning of large language models (LLMs) by as much again as it has already been scaled would require five times more electricity than the Earth currently generates in a year. It would also require the construction of 1 million data centres. What would you estimate the probability of that happening before the end of 2032 is? More than 1 in 1,000?
But keep in mind this would only achieve a modest performance gain. You might not even notice it as a user of LLMs. This is not about what it takes to get to AGI. This is about what it would take just to continue the scaling trend of reinforcement learning for LLMs. That’s a very low bar.[2]
Second, it’s technologically impossible given the current unsolved problems in fundamental AI research. For instance, AI models can’t learn from video data, at least not in anything like the way LLMs learn from text. This is an open research problem that has received considerable attention for many years. Does AGI need to be able to see? My answer is yes. Well, we currently don’t have AI models that can learn how to see to even the level of competence LLMs have with text (which is far below human-level) and it’s not for a lack of compute or data, so scaling isn’t a solution.
Sure, eventually this will be solved, but everything will be solved eventually, barring catastrophe. If you’re willing to hand-wave away sub-problems required to build AGI, you might as well hand-wave all the way, and just assume the overall problem of how to build AGI will be solved whenever you like. What year sounds interesting? Say, 2029? It has the weight of tradition behind it, so that’s a plus.[3]
Third, it’s practically impossible given the datasets we currently lack. Humans have a large number of heterogeneous skills. The amount of text available on the Internet is an unusual exception when it comes to the availability of data that can be imitation learned from, not the norm.
For instance, there are almost no recordings or transcripts (which, it should be noted, lose important information) of psychotherapy sessions, primarily due to privacy concerns. Clinical psychology professors face a difficulty in teaching their students how to practice psychotherapy because of the ethics concerns of showing a recording of some real person’s real therapy session to a classroom of students. If this scarcity of data poses challenges even for humans, how could AI systems that require three or more orders of magnitude more data to learn the same thing (or less) ever learn enough to become competent therapists?
That’s just one example. What about high-stakes negotiations or dealmaking that happens behind closed doors, in the context of business or government? What about a factory worker using some obscure tool or piece of equipment about which the number of YouTube videos is either zero or very few? (Not that AI models can currently learn from video, anyway.) If we’re talking about AI models learning how to do everything… everything in the world… that’s a lot of data we don’t have.
Fourth, many skills require adapting to changing situations in real time, with very little data. If AI systems continue to require more than 2,500x as much data than humans to learn the same thing (or less), there will never be enough data for AI systems to attain human-level general intelligence. If the strategy or tactics of StarCraft II changes, AI systems will be left flatfooted. If the strategy or tactics of anything changes, AI systems will be left flatfooted. If anything changes significantly enough that it no longer matches what was in the training data, AI systems that generalize as poorly as current AI systems will not succeed in that domain. Arguably, nearly all human occupations — and nearly all realms of human life — involve this kind of continuous change, and require a commensurate level of adaptability. Artificial general intelligence has always been a question of generalization, not just learning a bunch of narrowly construed skills that can be tested against frozen benchmarks or a frozen world — the world isn’t frozen.
This gets to the question of what “having a skill” really means. When we say a human has a certain skill, we implicitly mean they have the ability to adapt to change. If we say that MaNa can play StarCraft II, we mean that if he faces another professional player who suddenly tries some off-the-wall strategies or tactics never before seen in the game of StarCraft, he will be able to adapt on the fly. The element of surprise might trip him up in the first round, or the first five, but over the course of more games over more time, he will adapt and respond. He isn’t a collection of frozen weights instantiating frozen skills interacting with a frozen world, he’s a general intelligence that can generalize, evolved in a world that changes.
When we talk about about what an AI system “can do”, what “skills it has”, we are often bending the definition so what capability or skill means no longer fits the real world, everyday definition we apply to humans. We don’t think about whether, as is always true for humans, the AI has the ability to adapt on the fly, to change in response to change, to generate non-random, intelligent, reasonable, novel behaviour in response to novelty. If AI can hit a fixed target, even though all targets in the real world are always and forever moving, we say that’s good enough, and that’s equivalent to what humans do. But it isn’t. And we know this. We just have to think about it.
One of the most talked about imagined uses cases of AI is to use AI recursively for AI research. But the job of a researcher is one of the most fluid, changing, unfrozen occupations I can think of. There is no way an AI system that can’t adapt to change with only a small amount of data can do research, in the sense that a human does research.
Fifth, even in contexts where the datasets are massive and the problems or tasks aren’t changing, AI systems can’t generalize. LLMs have been trained on millions of books, likely also millions of academic papers, everything in the Common Crawl dataset, and more. GPT-4 was released 2 years and 8 months ago. Likely somewhere around 1 billion people use LLMs. Why, in all the trillions of tokens generated by LLMs, is there not one example of an LLM generating a correct and novel idea in any scientific, technical, medical, or academic field? LLMs are equipped with as close as we can get to all the written knowledge in existence. They are prompted billions of times daily. Where is the novel insight? ChatGPT is a fantastic search engine, but a miserable thinker. Maybe we shouldn’t think that if we feed an AI model some training data, it will have mastery over much more than literally exactly the data we fed it. In other words, LLMs’ generalization is incredibly weak.
Generalization is not something that seems to be improved with scaling, except maybe very meagerly.[4] If we were to somehow scale the training data and compute for LLMs by another 1 million times (which is probably impossible), it’s not clear that, even then, LLMs could generate their first novel and correct idea in physics, biology, economics, philosophy, or anything else. I reckon this is something so broke scaling ain’t gonna fix it. This is fundamental. If we think of generalization as the main measure of AGI progress, I’m not sure there’s been much AGI progress in the last ten years. Maybe a little, but not a lot.
There have been many impressive, mind-blowing results in AI, to be sure. AlphaStar and ChatGPT are both amazing. But these are systems that rely on not needing to generalize much. They rely on a superabundance of data that covers a very large state space, and the state space in which they can effectively operate extends just barely beyond that. That’s something, but it’s not general intelligence.
Conclusion
General intelligence is (or at least, requires) the ability to learn quickly from very little new data. Deep learning and deep reinforcement learning, in their current state, require huge quantities of data or experience to learn. Data efficiency has been improving over the last decade, but not nearly fast enough to make up the gap between AI and humans within the next decade. The dominant view among people who think very near-term AGI is very likely (with high confidence) is that scaling up the compute and data used to train AI models will cover either all or most of the ground between AI and humans. I gave five reasons this isn’t true:
- Physical limits like electricity make it unrealistic to even continue existing scaling trends, let alone whatever amount of scaling might be required to make up for the data inefficiency of AI relative to humans.
- AI can’t learn from video data (at least not in any way like how LLMs learn from text) and this is not a scaling problem, it’s a fundamental research problem. An AGI will need to see, so this needs to be solved.
- We don’t have anything anywhere close to datasets encompassing all the skills in the world, and certainly not of the huge size that deep learning models require.
- To really possess a skill, an AI system must be able to adapt to change quickly based on very little data. This means that even with continued scaling, even with the ability to learn from video and other modalities, and even with datasets encompassing every skill, AI systems would still quickly break in real world contexts.
- Not even considering adaptation to change, current AI systems’ lack of generalization mean they already fail at real world tasks, even where plenty of text data exists, and even where the task more or less stays the same over time. This appears to be a fundamental feature of deep learning and deep reinforcement learning as we know them (although new fundamental ideas within those paradigms could one day change that).
It is always possible to hand-wave away any amount of remaining research progress that would be required to solve a problem. If I assume scientific and technological progress will continue for the next 1,000 years, then surely at some point the knowledge required to build AGI will be obtained. So, why couldn’t that knowledge be obtained soon? Well, maybe it could. Or maybe it will take much longer than 100 years. Who knows? We have no particular reason to think the knowledge will be obtained soon, and we especially have no reason to think it will be obtained suddenly, with no warning or lead up.
More practically, if this is what someone really believes, then arguably they should not have pulled forward their AGI forecast based on the last ten years of AI progress. Since almost all the energy around near-term AGI seems to be coming as a response to AI progress, and not a sudden conversion to highly abstract and hypothetical views about how suddenly AGI could be invented, I choose to focus on views that see recent AI progress as evidence for near-term AGI.
So, that amounts to arguing against the view that all or almost all or most of the fundamental knowledge to built AGI has already been obtained, and what remains is entirely or almost entirely or mostly scaling up AI models by some number of orders of magnitude that is attainable within the next decade. Scaling is running out of steam. The data is running out, supervised pre-training has been declared over or strongly deemphasized by credible experts, and training via reinforcement learning won’t scale much further. This will probably become increasingly apparent over the coming years.
I don’t know to what extent people who have a high credence in near-term AGI will take this as evidence of anything, but it seems inevitable that the valuations of AI companies will have to come crashing down. AI models’ capabilities can’t catch up to the financial expectations those valuations are based on. I think people should take that as evidence because the real world is so much better a test of AI capabilities than artificially constructed, frozen benchmarks, which are always, in some sense, designed to be easy for current AI systems.
In general, people curious about the prospects of near-term AGI should engage more with real-world applications of AI, such as LLMs in a business context or with a robotics use case like self-driving cars, since the real world is much more like the real world than benchmarks, and AGI is defined by how it will perform in the real world, not on benchmarks. Benchmarks are a bad measure of AGI progress and without benchmarks, it’s not clear what other evidence for rapid AGI progress or near-term AGI there really is.
- ^
I chose the end of 2032 or around 7 years from now as a direct response to the sort of AGI timelines I’ve seen from people in effective altruism, such as the philosopher (and co-founder of effective altruism) Will MacAskill.
- ^
I haven’t really given any thought to how you’d do the math for this — obviously, it would just be a toy calculation, anyway — but I wouldn’t be surprised if you extrapolated the scaling of reinforcement learning compute forward to get to some endpoint that serves as a proxy for AGI and it turned out it would require more energy than is generated by the Sun and more minerals than are in the Earth’s crust.
For example, if you thought that AGI would require reinforcement learning training compute to be scaled up not just as much as it has been already, but by that much again one more time, then 1 trillion data centres would be required (more than 100 per person on Earth), and if by that much again two more times, then 1 quintillion data centres would be required (more than 100 million per person on Earth). But I suspect even this is far too optimistic. I suspect you’d start getting into the territory where you’d start counting the number of Dyson spheres required, rather than data centres.
Combinatorial explosion never stops producing shocking results. For instance, according to one calculation, all the energy in the observable universe (and all the mass, converted to energy), if used by a computer as efficient as physics allows, would not be sufficient to have more than one in a million chance of bruteforcing a randomly generated 57-character password using numbers, letters, and symbols. Reinforcement learning is far more efficient than brute force, but the state space of the world is also astronomically larger than the possible combinations of a 57-character password. We should be careful that the idea of scaling up compute all the way to AGI doesn’t implicitly assume harnessing the energy of billions of galaxies, or something like that.
- ^
My point here isn’t that we know it’s extremely unlikely the problem of how to learn from video data will be solved within the next 7 years. My point is that we have no idea when it will be solved. If people were saying that they had no idea when AGI will be created, I would have no qualms with that, and I wouldn’t have written this post.
- ^
Please don’t confuse, here, the concept of an AI model being able to do more things (or talk about more things) because it was trained on data about more things. That’s not generalization, that’s just training. Generalization is a system’s ability to think or to generate intelligent behaviour in situations that go beyond what was covered in the data it was trained on.
For what it's worth, I think "less than 0.1% likely by 2032" is PROBABLY also not in line with expert opinion. The Forecasting Research Institute, where I currently work has just published a survey of AI experts and superforecasters on the future of AI, as part of our project LEAP, the Lognitudinal Expert Panel on AI. In it, experts and supers median estimate was a 23% chance of the survey's "rapid scenario" for AI progress by 2030 would occur. Here's how the survey described the rapid scenario:
"By the end of 2030, in the rapid-progress world, AI systems are capable of competing with the best human minds and workers, and can surpass them.
Human creativity and leadership remain valued, but mostly for setting high-level vision—day-to-day execution can be left to silicon-based systems. Autonomous researchers can collapse years-long research timelines into days, weeks, or months, creating game-changing technologies, such as materials that revolutionize energy storage, or bespoke cancer cures. No human freelance software engineer can outperform AI. The same goes for customer service (e.g., call center and support chat), paralegal, and administrative workers (e.g., bookkeepers or scheduling assistants).
Indeed, models have become so capable that AI can create an album of the same caliber as the Grammy Album of the Year. Additionally, a single AI agent can generate a Pulitzer- (or Booker Prize-) caliber novel according to current (2025) standards, adapt the book into an engaging two-hour movie, negotiate the resulting book and movie contracts, and launch the marketing campaigns for both while its sibling agents manage the book publishing company and movie studio at the level of highly competent CEOs.
Not only do Level-5 robo-taxis exist, but they are, on average, 99.9% safer than human-piloted cars and can venture anywhere off-road that a competent human driver can. Meanwhile, robots can navigate an arbitrary home anywhere in the world, make a cup of the most popular local hot beverage, clean and put away the dishes according to the local custom, fix any plumbing issues that arise while they’re doing the dishes--and they can do it all faster and more reliably than most humans and without human guidance. Robots in advanced factories can autonomously perform the full range of tasks requiring the highest levels of dexterity, coordination, and adaptive decision-making."
I don't think that necessarily amounts to "AGI", because, for example, what about robotics, maybe the AIs still can't replace manual labour, for software as well as hardware related reasons. But I do think it's fair to infer that if the survey-takes thought there was a 23% chance of this scenario by 2030, it's pretty unlikely they don't think AGI is >0.1% likely by 2032.
Very interested to look at the survey. Can you link to it?
I think there’s no chance of the rapid scenario, as in, much less than a 1 in 10,000 chance. I think an outlandish scenario like we find out JFK is actually still alive is more likely than that. Simply put, that will not happen. (99%+ confidence.)
There is an ambiguity about "capabilities" versus deployment here to be fair. Your "that will not happen" seems somewhat more reasonable to me if we are requiring that the AIs are actually deployed and doing all this stuff versus merely that models capable of doing this stuff have been created. I think it was the latter we were forecasting, but I'm not 100% certain.
I think including widespread deployment in the rapid scenario makes it modestly more unlikely but not radically so. The fundamental issue is that these capabilities cannot be developed within 5 years. Is there a small chance? Yes, sure, but a very small one.
https://leap.forecastingresearch.org/ The stuff is all here somewhere, though it's a bit difficult to find all the pieces quickly and easily.
For what it's worth, I think the chance of the rapid scenario is considerably less than 23%, but a lot more than under 0.1%. I can't remember the number I gave when I did the survey as a superforecaster, but maybe 2-3%? But I do think chances are getting rather higher by 2040, and it's good we are preparing now.
". I think an outlandish scenario like we find out JFK is actually still alive is more likely than that"
If you really mean this literally, I think it is extremely obviously false, in a way that I don't think merely 0.1% is.
No, I mean it literally. I literally think something as bizarre as it turning out somehow JFK has really been alive all this time and his assasination was a hoax is more likely than the rapid scenario. I don’t think that’s obviously false. I think it’s obviously correct.
For instance, Toby Ord has calculated it’s physically impossible to continue the scaling trend of RL training for LLMs. Bizarre and outlandish things are more likely than physically impossible things. That’s not all there is to say about the subject, but it’s a good start.
That's pretty incomprehensible to me even as a considerable skeptic of the rapid scenario. Firstly, you have experts giving a 23% chance and it's not moving you up even to like over 1 in a 100,000, let's say, although probably the JFK scenario is a hell of a lot less likely than that, even if his assassination was faked, despite there literally being a huge crowd who saw his head get blown off in public, still he have to be 108 to still be alive. Secondly, in 2018, AI could do to a first approximation basically nothing outside of highly specialized uses like chess computers, that did not use current ML techniques. Meanwhile, this year, I, a philosophy PhD, asked Claude about an idea that I had seriously thought about turning into a paper one day back when I was still in philosophy, and it came up with a very clever objection that I had not thought of myself. I am fairly, even if not 100% sure that this objection is not in the literature anywhere. Given that we've gone from nothing to "high quality philosophical arguments at times" in like 7 years, and there are some moderately decent reasons for thinking models good at AI research tasks could set off a positive feedback loop, and far more money and effort is being thrown at AI than ever before, it seems hard to me to think it is 99,999 in 100,000 sure that we won't get AGI by 2030, even though the distance to cross is still very large, and current success on benchmarks somewhat misleading.
I didn’t update my views on the survey because I haven’t seen the survey. I did ask for the survey so I could see it. I haven’t seen it yet. I couldn’t find it on the website. I might change my mind after I see it. Who knows.
I agree the JFK scenario is extremely outlandish and would basically be impossible. I just think the rapid scenario is more outlandish and would also basically be impossible.
Everything you said about AI I just don’t think is true at all. LLMs are just another narrow AI, similar to AlphaGo, AlphaStar, AlphaFold, and so on, and not a fundamental improvement in generality that gets us closer to AGI. You shouldn’t have updated your AGI timelines based on LLMs. That’s just a mistake. Whatever you thought in 2018 about the probability of the rapid scenario, you should think the same now, or actually even less because more time has elapsed and the necessary breakthroughs have still not been made. So, what was your probability for the rapid scenario in 2018? And what would your probability have been if someone told you to imagine there would be very little progress toward the rapid scenario between 2018 and 2025? That’s what I think your probability should be.
To say that AI’s capabilities were basically nothing in 2018 is ahistorical. The baseline from which you are measuring progress is not correct, so that will lead you to overestimate progress.
I also get the impression you greatly overestimate Claude’s capabilities relative to the cognitive challenges of generating the behaviours described in the rapid scenario.
AI being able to do AI research doesn’t affect the timeline. Here’s why. AI doing AI research requires fundamental advancements in AI to a degree that would make something akin to AGI or something akin to the rapid scenario happen anyway. So, whether AI does AI research can’t accelerate the point at which we reach the rapid scenario. There are no credible arguments to the contrary.
The vast majority of benchmarks are not just somewhat misleading if seen as evidence about AGI progress. They are almost completely meaningless in terms of AGI progress, with perhaps the sole exception of the ARC-AGI benchmarks. Text Q&A benchmarks are about as meaningful an indication of AGI progress as AI’s ability to play go or StarCraft.
There is also the physical impossibility problem. If continuing scaling trends is literally physically impossible, then how can the probability of the rapid scenario be more than 1 in 10,000? (By the way, I said less than 1 in 10,000, not less than 1 in 100,000, although I’m not sure it really matters.)
Someone should try to do the math on what it would take to scale RL training compute for LLMs to some level that could be considered a proxy for AGI or the sort of AI system that could make the rapid scenario possible. You will likely get some really absurd result. For example, I wouldn’t be surprised if the result was that the energy required would mean we’d have to consume the entire Sun, or multiple stars, or multiple galaxies. In which case, the speed of light would render the rapid scenario impossible.
Combinatorial explosion is just that crazy. There isn’t enough energy in the entire universe to brute force a 60-character password. RL is not the same thing as trying random combinations for a password, but there is an element of that in RL, and the state space of real world environments from the perspective of an AI agent is much, much larger than the possible combinations of a 60-character password.
Ok, there's a lot here, and I'm not sure I can respond to all of it, but I will respond to some of it.
-I think you should be moved just by my telling you about the survey. Unless you are super confident either that I am lying/mistaken about it, or that the FRI was totally incompetent in assembling an expert panel, the mere fact that I'm telling you the median expert credence in the rapid scenario is 23% in the survey ought to make you think there is at least a pretty decent chance that you are giving it several orders of magnitude less credence than the median expert/superforecaster. You should already be updating on there being a decent chance that is true, even if you don't know for sure. Unless you already believed there was a decent chance you were that far out of step with expert opinion, but I think that just means you were already probably doing the wrong thing in assigning ultra-low credence. I say "probably" because the epistemology of disagreement IS very complicated, and maybe sometimes it's ok to stick to your guns in the face of expert consensus.
-"Physical impossibility". Well, it's not literally true that you can't scale any further at all. That's why they are building all those data centers for eyewatering sums of money. Of course, they will hit limits eventually and perhaps soon-probably monetary before physical. But you admit yourself that no one has actually calculated how much compute is needed to reach AGI. And indeed, that is very hard to do. Actually Epoch, who are far from believers in the rapid scenario as far as I can tell think quite a lot of recent progress has come from algorithmic improvements, not scaling: https://blog.redwoodresearch.org/p/whats-going-on-with-ai-progress-and Text search for "Algorithmic improvement" or "Epoch reports that we see". So progress could continue to some degree even if we did hit limits on scaling. As far as I can tell, most of the people who do believe in the rapid scenario actually expect scaling of training compute to at least slow down a lot relatively soon, even though the expect big increases in the near future. Of course, none of this proves that we can reach AGI with current techniques just by scaling, and I am pretty dubious of that for any realistic amount of scaling. But I don't think you should be talking like the opposite has been proven. We don't know how much compute is needed for AGI with the techniques of today or the techniques available by 2029, so we don't know whether the needed amount of compute would breach physical or financial or any other limits.
-LLM "Narrowness" and 2018 baseline: Well, I was probably a bit inexact about the baseline here. I guess what I meant was something like this. Before 2018ish, as a non-technical person, I never really heard anything about exciting AI stuff, even though I paid attention to EA a lot, and people in EA already cared a lot about AI safety and saw it as a top cause area. Since then, there has been loads of attention, literal founding fathers of the field like Hinton say there is something big going on, I find LLMs useful for work, there have been relatively hard to fake achievements like doing decently well on the Math Olympiad, and College students can now use AI to cheat on their essays, a task that absolutely would have been considered to involve "real intelligence" before Chat-GPT. More generally, I remember a time, as someone who learnt a bit of cognitive science while studying philosophy, when the problem with AI was essentially being presented as "but we just can't hardcode all our knowledge in, and on the other hand, its not clear neural nets can really learn natural languages". Basically AI was seen as something that struggled with anything that involved holistic judgment based on pattern-matching and heuristics, rather than hard-coded rules. That problem now seems somewhat solved: We now seem to be able to get AIs to learn how to use natural language correctly, or play games like Go that can't be brute forced by exact calculation, but rely on pattern-recognition and "intuition". These AIs might not be general, but the techniques for getting them to learn these things might be a big part of how you build an AI that actually is, since the seem to be applicable to large variety of kinds of data: image recognition, natural language, code, Go and many other games, information about proteins. The techniques for learning seem more general than many of the systems. That seems like relatively impressive progress for a short time to me as a layperson. I don't particularly think that should move anyone else that much, but it explains why it is not completely obvious to me, why we could not reach AGI by 2030 at current rates of progress. And again, I will emphasize, I think this is very unlikely. Probably my median is that real AGI is 25 years away. I just don't think it is 1 in a million "very unlikely".
I want to emphasize here though, that I don't really think anything under the 3rd dash here should change your mind. That's more just an explanation of where I am coming from, and I don't think it should persuade anyone of anything really. But I definitely do think the stuff about expert opinion should make you tone down your extremely extreme confidence, even if just a bit.
I'd also say that I think you are not really helping your own cause here by expressing such an incredibly super-high level of certainty, and making some sweeping claims that you can't really back up, like that we know right now that physical limits have a strong bearing on whether AGI will arrive soon. I usually upvote the stuff you post here about AGI, because I genuinely think you raise good, tough questions, for the many people around here with short timelines. (Plenty of those people probably have thought-through answers to those questions, but plenty probably don't and are just following what they see as EA consensus.) But I think you also have a tendency to overconfidence that makes it easier for people to just ignore what you say. This come out in you doing annoying things you don't really need to do, like moving quickly in some posts from "scaling won't reach AGI" to "AI boom is a bubble that will unravel" without much supporting argument, when obviously, AI models could make vast revenues without being full AGI. It gives the impression of someone who is reasoning in a somewhat motivated manner, even as they also have thought about the topic a lot and have real insights.
I think your suspicion toward my epistemic practices is based simply on the fact that you disagree very strongly, you don’t understand my views or arguments very deeply, you don’t know my background or history, and you’re mentalizing incorrectly.
For example, I have a detailed collection of thoughts about why I think AI investment is most likely in a bubble, but I haven’t posted about that in much detail on the EA Forum yet — maybe I will, or maybe it’s not particularly central to these debates or on-topic for the forum. I’m not sure to what extent an AI bubble popping would even change the minds of people in EA about the prospects of near-term AGI. How relevant is it?
I asked on here about to what extent the AI bubble popping would change people’s views on near-term AGI and the only answer I got is that it wouldn’t move the needle. So, I’m not sure if that’s where the argument needs to go. Just because I briefly mention this topic in passing doesn’t mean my full thoughts about the topic are really only that brief. It is hard to talk about these things and treat every topic mentioned, even off-handedly, in full detail without writing the whole Encyclopedia Britannica.
Also, I am much, much less sure about the AI bubble conclusion than I am about AGI or about the rapid scenario. It is extremely, trivially obvious that sub-AGI/pre-AGI/non-AGI systems could potentially generate a huge amount of profit and justify huge company valuations, and indeed I’ve written something like 100 articles about that topic over the last 8 years. I used to have a whole blog/newsletter solely about that topic and I made a very small amount of money doing freelance writing primarily about the financial prospects of AI. I actually find it a little insulting that you would think I have never considered that AI could be a big financial opportunity without AGI coming to fruition in the near term.
Here is Toby Ord on the physical limits to scaling RL training compute for LLMs:
This is not what it would take to get to AGI, it’s what it would take to get from Grok 4 to Grok 5 (assuming the scaling trend were to continue as it did from Grok 3 to Grok 4).
I am willing to say that, if Toby’s calculation is correct, it is very close to an absolute certainty that this level of scaling of RL training compute for LLMs — using 5x the world’s current annual electricity supply and 1 million datacentres — will not happen before the end of 2030.
My comments about extrapolating scaling to AGI potentially requiring galaxies is not really the actual main point I’m trying to make about scaling, it’s just to emphasize the problem with runaway exponential growth of this kind and the error in extrapolating its long-term continuation. This is for emphasis and illustration, not a strongly held view.
A number of prominent experts like OpenAI’s former chief scientist Ilya Sutskever have said self-supervised pre-training of LLMs has run out of steam or reached a plateau. Anthropic’s CEO Dario Amodei said that Anthropic’s focus has shifted from pre-training to RL training. So, at this point we are relying quite a lot on scaling up RL training for LLMs to get better as a result of training. (More discussion here.) Inference compute can also be scaled up and that’s great, but you have to pay the inference cost every query and can’t amortize it across billions or trillions of queries like you can with the training cost. Plus, you run into a similar problem where once you scale up inference compute 100x and 100x again after that, the next 100x and the next 100x starts to become unwieldy.
On the philosophy of mind and cognitive science topics, I have been spiritually enthralled with these topics since I was a teenager in the 2000s and 2010s (when I first read books like Daniel Dennett’s Consciousness Explained and Douglas Hofstadter’s I Am A Strange Loop, and incidentally was also interested in people talking about AGI like Ray Kurzweil and Nick Bostrom, and even was reading/watching/listening to some of Eliezer Yudkowsky‘s stuff back then), for a long time wanted to devote my life to studying them, and actually I still would like to do that if I could somehow figure out a practical path for that in life. As a philosophy undergrad, I published an essay on the computational theory of mind in an undergrad journal and, unfortunately, that’s the closest I’ve come to making any kind of contribution to the field.
I’ve been following AI closely for a long time and I can imagine how you might have a distorted view of things if you see generative AI as having come basically out of nowhere. I started paying attention to deep learning and deep reinforcement learning around the time DeepMind showed its results with deep RL and Atari games. I really ramped up how much I started paying attention in 2017 when I started to think really seriously about self-driving cars. So, LLMs were quite a surprise for me, just as they were for many people, but they didn’t drop out of the clear blue sky. I already had an intellectual context to put them in.
If you want to read some more substantive objections to the rapid scenario, there are some in the post above. I’m not sure if you read those or just focused on the forecasting part in the preamble. The rapid scenario depends on a number of fundamental improvements in AI, including (but not limited to) vastly improved generalization, vastly improved data efficiency, the ability to learn effectively and efficiently from video data, and continual learning. These are not challenges that can be solved through scaling, full stop. And the rapid scenario cannot happen without solving them, full stop. There is more to say, but that’s a good start.
On the survey, I might update on the survey once I see it, but I need to see it first. I’d love to see it and I’ll keep an open mind when I do. There are some entirely epistemically legitimate reasons for me not to update on evidence I can’t see or confirm, especially when that’s evidence only about other people’s views and not direct evidence, and especially especially when whether it’s actually even new information to me about other people’s views depends on the details — such as who was surveyed — which I can’t see and don’t know.
There are strong concerns related to the concept of information cascades where, e.g., if you just survey the same people over and over and repackage it as new evidence, that would lead you to keep revising your credences upwards (or downwards) with no limit based on the same piece of evidence, or, e.g., people will circularly update their views — I tell you my view, you update based on that, you tell me your updated view (changed only because of me telling you my view), I update based on that, and so on and so forth, until we end up a significant way from where we started for no good reason. In case you think this is a silly hypothetical, I read a post or comment somewhere (I could dig it up) where someone who had been involved in the Bay Area rationalist community said they think this kind of circular updating actually happened. The imagery they gave was people sitting in a circle telling each other smaller and smaller numbers for the median date of AGI.
[Edited on 2025-11-14 at 07:52 UTC to add: I was able to find the report. It was easy to find. I was just looking on the wrong webpage before. The discussion of the question about the "rapid progress" scenario on page 104 and the responses on page 141 is confusing. Respondents are asked, "At the end of 2030, what percent of LEAP panelists will choose “slow progress,” “moderate progress,” or “rapid progress” as best matching the general level of AI progress?" I find that a really strange and counterintuitive way to frame the question. How is this a proxy for probability that the scenario will occur? The framing of the question is highly ambiguous and the answers are highly ambiguous and hard to interpret.
Three rationale examples are given for the rapid progress scenario and all three contradict the rapid progress scenario. How are the rationale examples selected? Was there not one example of a respondent who actually thought the rapid progress scenario would occur? I don't understand this.
This is precisely why I don't update on evidence before seeing it. The devil is in the details.
The rationale examples are useful and I'm glad they are included. They show problems both with the design of the survey and with the reasoning and evidence used by some of the respondents to come to conclusions about near-term AI progress, e.g., the famous METR time horizon graph is erroneously interpreted in a way that overlooks the crucial caveats, some of which even METR itself highlights. Instead of only measuring what METR measures, researchers should also measure something performance on a diverse, heterogeneous array of manually graded real world or realistic tasks with the same success rate as humans. The result would be entirely different, i.e., approximately a flat line at zero rather than the appearance of an exponential trend.
I'll also add asking respondents to choose only between the slow progress, moderate progress, and rapid progress scenarios is really poor survey design. All three scenarios arguably include proxies for or operationalizations of AGI, and respondents are not given the option to say no to all of them. Even the slow progress scenario says that AI "can automate basic research tasks, generate mediocre creative content, assist in vacation planning, and conduct relatively standard tasks that are currently (2025) performed by humans in homes and factories." AI can also "rarely produce novel and feasible solutions to difficult problems." And AI can "can handle roughly half of all freelance software-engineering jobs that would take an experienced human approximately 8 hours to complete in 2025", write "full-length novels", "make a 3-minute song that humans would blindly judge to be of equal quality to a song released by a current (2025) major record label", and largely substitute for "a competent human assistant".
So, respondents were given a choice between AGI, AGI, and AGI, and chose AGI. This is not a useful survey! You are not giving the respondents a chance to say no! You are baking in the result into the question!
Another serious problem with the survey is the percentage of respondents affiliated with effective altruism. On page 20, the report says 28% of respondents were affiliated with effective altruism and that was reweighted down to 12%. This is exactly the problem with information cascades and circular updating that I anticipated. I don't need a new survey of people affiliated with effective altruism to tell me what people affiliated with effective altruism believe about AI. I already know that.
Another significant problem is that only around 45% of the experts have technical expertise in AI. But now I'm just piling on.
You absolutely should not have told me to update on this survey before actually looking at it.]
Unfortunately, this is a debate where forecasts can’t be practically falsified or settled. If January 2031 rolls out and AI has still only made modest, incremental progress relative to today, the evidence is still open to interpretation as to whether a 97-98% chance the rapid scenario wouldn’t happen was more reasonable or a 99.99%+ chance it wouldn’t happen. We can’t agree on how to interpret similar evidence today. I have no reason to think it would be any easier to come to an agreement on that in January 2031.
It is an interesting question, as you say, how to update our views based on the views of other people — whether, when, why, and by how much. I was surprised to recently see a survey where around 20% of philosophers accept or lean toward a supernatural explanation of consciousness. I guess it’s possible to live in a bubble where you can miss that lots of people think so differently than you. I would personally say that the chances that consciousness is a supernatural phenomenon are less than the rapid scenario. And that survey didn't make me revise up my credence in the supernatural at all. (What about you?)
I will say that the rapid scenario is akin to a supernatural miracle in the radical discontinuity to our sense of reality it implies. It is more or less the view that we will invent God — or many gods — before the 2032 Summer Olympics in Brisbane. You should not be so quick to chide someone for saying this is less than 0.01% likely to happen. Outside of the EA/rationalist/Bay Area tech industry bubble, this kind of thing is widely seen as completely insane and ludicrous.
In my interpretation, the rapid scenario is an even higher bar than "just" inventing AGI, it implies superhuman AGI. So, for instance, a whole brain emulation wouldn't qualify. An AGI that is "merely" human-level wouldn't qualify. I can't make a Grammy-calibre album, write a Pulitzer or Booker Prize-calibre book, or make a Hollywood movie, nor can I run a company or do scientific research, and I am a general intelligence. The rapid scenario implies superhuman AGI or superintelligence, so it's less likely than "just" AGI.
Please forgive me for how long this comment is, but I suddenly felt the need to say... the following...
I'm starting to get the temptation to ask you questions like, "What probability would you put on the core metaphysical and cosmological beliefs of each of the major world religions turning out to be correct?" which is a sign this conversation is getting overly meta, overly abstract, and veering into "How many angels can dance on the head of a pin?" territory. I actually am fascinated with epistemology and want to explore some of these questions more (but in another context than this comment thread that would be more appropriate). I am a bit interested in forecasting, but not fascinated, and kind of would like to understand it better (I don't understand it very well currently). I would particularly like to understand forecasting better as it pertains to the threshold or demarcation between topics it is rigorous to forecast about, for which there is evidence of efficacy, such as elections or near-term geopolitical events, and topics for which forecasting is unrigorous, not supported by scientific evidence, and probably inappropriate, such as, "What is the probability of the core tenets of Hindu theology such as the identity of Atman and Brahman being correct?"
My personal contention is that actually a huge problem with the EA Forum (and also LessWrong, to an even worse extent) is how much time, energy, and attention gets sucked into these highly abstract meta debates. To me, it's like debating about whether you should update your probability of whether there's peanut butter in the cupboard based on my stated probability of whether there's peanut butter in the cupboard, when we could just look in the cupboard. The abstract content of that debate is actually pretty damn interesting, and I would like to take an online course on that or something, but that's the indulgent attitude of a philosophy student and not what I think practically matters here. I simply want more people to engage substantively with the object-level points I'm making, e.g. about learning from video data, generalization, data efficiency, scaling limits, and so on. That's "looking in the cupboard". I could be wrong about everything. I could be making basic mistakes. I don't know. What can I do except try to have the conversation?
By the way, when I give my probabilities for something, I am just trying to faithfully and honestly report, as a number, what I think my subjective or qualitative sense of the likelihood of something implies. I am not necessarily making an argument that anyone else should have that same probability. I just want them to talk to me about the object-level issues. The probabilities are a side thing that I need to get out of the way to talk about that. I don't intend me reporting my best guess at how my intuitions translate into numbers as an insult against anyone. This passes the reversibility test for me: if someone says they think their probability for something is 1,000x higher or lower than mine, I don't interpret that as an insult.
So, I don't think is it impolite for me to express the numbers that are my best guess. I do kind of accept that I will be less persuasive if I say a number that seems too extreme, which is why I've been kind of softballing/sandbagging what I say about this. Also, I think if someone says "significantly less than 1%" or even just "less than 1%" or "1%", that's enough to motivate the discussion of object-level topics and to move on from the guessing probabilities portion of the conversation. So, it's kind of irrelevant whether I say less than 1 in 1,000, less than 1 in 10,000, less than than 1 in 100,000, or less than 1 in 1 million. Yes, I get that these are very different probabilities (each one an order of magnitude lower than the last!), but from the perspective of just hurrying along to the object-level discussion, it doesn't really make a difference.
I almost would be willing to accept I should sandbag my actual probability even more for the sake of diplomacy and persuasion, and just say "less than 1%" or something like that. But that seems a little bit morally corrupt — maybe "morally corrupt" is too strong, but, hey, I'd rather just be transparent and honest rather than water things down to be more persuasive to people who are very far away from me on this topic. (The question of how integrate considerations about both diplomacy and frankness into one's communications is another fascinating topic, but also another diversion away from the object-level issues pertaining to the prospects of near-term AGI.)
Some people in this community sometimes like to pretend they don't have feelings and are just calculating machines running through numbers, but the emotion is betrayed anyway. The undercurrent of this conversation is that some people take offense at my views or find them irritating, and I have an incentive to placate them if I want to engage them in conversation. I accept that that is true. I am no diplomat or mediator, and I don't feel particularly competent at persuasion.
My honest motivation for engaging in these debates is mostly sheer boredom, curiosity, and a desire for intellectual enrichment and activity. Yeah, yeah, there is some plausible social benefit or moral reason to do this by course correcting effective altruism, but I'm kind of 50/50 on my p(doom) for effective altruism anyway, and I think the chances are slim that I'm going to make a dent in that. So, if it were just a grinding chore, I wouldn't do it.
Anyway, all this is to say: please just talk to me about the object-level issues, and try to keep the rest of it (e.g. getting into the weeds of forecasting, open questions in epistemology that philosophers and other experts don't agree on, abstract meta debates) low, and only bring it up when it's really, really important. (Not just you, personally, David, this is my general request.) I'm dying to talk about the object-level issues, and somehow I keep ending up talking about this meta stuff. (I am easily distractible and will talk forever about all sorts of topics, even topics that don't matter and don't relate to the issue I originally wanted to talk about, so it's my fault too.)
Executive summary: The author argues that artificial general intelligence (AGI) is extremely unlikely to emerge before 2032 (less than 0.1% chance), because current AI systems learn far more slowly and inefficiently than humans; scaling up data and compute cannot overcome these fundamental limits, and true general intelligence requires fast, flexible learning and generalization, not frozen skills trained on static datasets.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
This summary seems mostly correct and maybe I'd give it like a B or B+. You can read this and decide whether you want to dig into the whole post.
It's interesting to notice the details that SummaryBot gets wrong — there aren't "billions" of LLM users (and I didn't say there were).
SummaryBot also sort of improvises the objection about "static datasets", which is not something I explicitly raised. My approach in the post was actually just to say, okay, let's assume AI systems could continually learn from new data or experience coming in in real time. In that case, their data efficiency would be far too low and their generalization would be far too poor to make them actually competent (in the way humans are competent) at most of the tasks or occupations that humans do that we might want to automate or might want to test AI's capabilities against. It's kind of funny that SummaryBot gets its hand on the ball and adds its own ideas to the mix.
I think it is bad faith to pretend that those who argue for near-term AGI have no idea about any of this when all the well-known cases for near-term AGI (including both AI 2027 and IABIED) name continual learning as the major breakthrough required.
Can you provide citations? I wasn’t quickly able to find what you’re referring to.
I tried to search for the exact phrase “continual learning” in the book you mentioned — If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares, for the benefit of other readers of these comments — and got no results. I also did a Google search for “site:ai-2027.com continual learning” and got no results. This isn’t a foolproof method since it relies on that exact phrase being used.
In any case, a specific quote from AI 2027 or the book would be helpful.
As briefly discussed in the post, I think a major problem with AI 2027’s way of thinking about things is the notion that AI will be able to pull itself up by its bootstraps by conducting AI research in the very near future (within 2 years). That is impossible given AI systems’ current capabilities and would require some kind of fundamental research breakthrough or some other major increase in capabilities (e.g. through scaling) very soon.
Importantly, the thesis of this post is not just that continual learning would be required for AGI, but vastly increased data efficiency. You can assume, for the sake of argument, AI systems will be able to continually learn, but if they learn more than 2,500x slowly than humans (or whatever it is), that precludes AGI. Data efficiency and generalization are even more important concepts for the argument of this post than continual learning.
By the way, I don’t think I claimed that nobody who argues that very near-term AGI is very likely is aware of the sort of things I discussed in my post. I just claimed that there have been "no good answers" to these kinds of objections.
It's called online learning in AI 2027 and human-like long-term memory in IABIED.
At a glance, the only mentions of "long-term memory" in If Anyone Builds It, Everyone Dies (IABIED) by Yudkowsky and Soares are in the context of a short science fiction story. It's just stipulated that a fictional AI system has it. It's only briefly mentioned, and there is no explanation of what long-term memory consists of, how it works, or how it was developed. It's similar to Data's "positronic brain" in Star Trek: The Next Generation or any number of hand-waved technological concepts in sci-fi. Do you think this is a good answer to my objection? If so, can you explain why?
Unless there are other passages from the book that I'm missing (maybe because they use a different phrasing), the mentions of "long-term memory" in the book don't seem to have the centrality to the authors' arguments or predictions that you implied. I don't see textual evidence that Yudkowsky and Soares "name continual learning as the major breakthrough required", unless you count those very brief mentions in the sci-fi story.
I think one of the main problems with AI 2027's story is the impossibility of using current AI systems for AI research, as I discussed in the post. There is a chicken-and-egg problem. LLMs are currently useless at actually doing research (as opposed to just helping with research as a search engine, in the exactly same way Google helps with research). AI systems with extremely weak generalization, extremely poor data efficiency, and without continual learning (or online learning) cannot plausibly do research well. Somehow this challenge has to be overcome, and obviously an AI that can't do research can't do the research to give itself the capabilities required to do research. (That would be an AI pulling itself up by its bootstraps. Or, in the terminology of the philosopher Daniel Dennett, it would be a skyhook.) So, it comes down to human researches to solve this challenge.
From what I've seen, when AI 2027 talks about an AI discovery/innovation being made by human researchers, the authors just kind of give their subjective best guess of how long that discovery/innovation will take to be made. This is not a new or original objection, of course, but I share the same objection as others. Any of these discoveries/innovations could take a tenth as long or ten times as long as the authors guess. So, AI 2027 isn't a scientific model (which I don't think is what the authors explicitly claim, although that's the impression some people seem to have gotten). Rather, it's an aggregation of a few people's intuitions. It doesn't really serve as a persuasive piece of argumentation for most people for mainly that reason.
It doesn't matter if they state that is a "major breakthrough required" if they don't provide sufficient evidence that this breakthrough is in any way likely to happen in the immediate future. Yarrow is provided plenty of argumentation as to why it won't happen: if you disagree you should feel free to cite actual counter-evidence rather than throwing false accusations of bad faith around.
I agree that that comment may be going too far with claiming "bad faith", but the article does have a pretty tedious undertone of having found some crazy gotcha that everyone is ignoring. (I'd agree that it gets at a crux and that some reasonable people, e.g. Karpathy, would align more with the OP here)
What's your response to the substance of the argument? From my perspective, people much more knowledgeable about AI and much more qualified than me have made the same or very similar objections, prominently in public, for some time now, and despite being fairly keyed in to these debates, I don't see people giving serious replies to these objections. I have also tried to raise these sort of objections myself and generally found a lack of serious engagement on the substance.
I actually do see a significant number of people, including some people who are prominent in debates around AGI, giving replies that indicate a misunderstanding of these sorts of objections, or indications that people haven't considered these sort of objections before, or hand-waving dismissals. But I'm still trying to find the serious replies. It's possible there has been a serious and persuasive rebuttal somewhere I missed — part of the purpose of writing a post like this is to elicit that, either from a commenter directly or from someone citing a previous rebuttal. But if you insist such a rebuttal is so obvious that these objections are tedious, I can't believe you until you make that rebuttal or cite it.
Case in point... Matrice identified something in my post that was of secondary importance — continual learning — that, in the post, I was willing to hand-wave away to focus on the things that I think are of primary importance, namely, 1) physical limits to scaling, 2) the inability to learn from video data, 3) the lack of abundant human examples for most human skills, 4) data inefficiency, and 5) poor generalization. So, first of all, Matrice did not identify one of the five points I actually raised in the post.
Second, Matrice made a citation that, when I followed up on it, did not actually say what Matrice claimed it said, and in no way answered the objection that current AI can't continually learn (which, to repeat, was not one of the main objections I made in my post anyway). It was literally just a short sci-fi story where it's simply said a fictional AI can continually learn, with no further discussion of the topic and no details beyond that. How is that a serious response to the objection about continual learning, and especially how is that a serious response to my post, when I didn't raise continual learning as one of my main objections?
So, Matrice's reply mispresented both the thesis of my post and misrepresented the work they cited as a rebuttal to it.
If there is a better response to the substance of the objections I raised in my post than this, please let me know! I'm dying to hear it!
All of those except 2) boil down to "foundation models have to learn once and for all through training on collected datasets instead of continually learning for each instantiation". See also AGI's Last Bottlenecks.
No, none of them boil down to that, and especially not (1).
I've already read the "A Definition of AGI" paper (which the blog post you linked to is based on) and it does not even mention the objections I made in this post, let alone offer a reply.
My main objection to the paper is that it makes a false inference that tests used to assess human cognitive capabilities can be used to test whether AI systems have those same capabilities. GPT-4 scored more than 100 on an IQ test in 2023, which would imply that it is an AGI if an AI that passes a test has the cognitive capabilities a human is believed to have if it passes that same test. The paper does not anticipate this objection or try to argue against it.
(Also, this is just a minor side point, but Andrej Karpathy did not actually say AGI is a decade away on Dwarkesh Patel's podcast. He said useful AI agents are a decade away. This is pretty clear in the interview or the transcript. Karpathy did not comment directly on the timeline for AGI, although it seems to be implied that AGI can come no sooner than AI agents.
Unfortunately, Dwarkesh or his editor or whoever titles his episodes, YouTube chapters, and clips has sometimes given inaccurate titles that badly misrepresent what the podcast guest actually said.)
How is "heterogeneous skills" based on private information and "adapting to changing situation in real time with very little data" not what continual learning mean?
Here’s a definition of continual learning from an IBM blog post:
Here’s another definition, from an ArXiv pre-print:
The definition of continual learning is not related to generalization, data efficiency, the availability of training data, or the physical limits to LLM scaling.
You could have a continual learning system that is equally data inefficient as current AI systems and is equally poor at generalization. Continual learning does not solve the problem of training data being unavailable. Continual learning does not help you scale up training compute or training data if compute and data are scarce or expensive, nor does the ability to continually learn mean an AI system will automatically get all the performance improvements it would have gotten from continuing scaling trends.
Yes those quotes do refer to the need for a model to develop heterogeneous skills based on private information, and to adapt to changing situations in real life with very little data. I don't see your problem.
In case it's helpful, I prompted Claude Sonnet 4.5 with extended thinking to explain three of the key concepts we're discussing and I thought it gave a pretty good answer, which you can read here. (I archived that answer here, in case that link breaks.)
I gave GPT-5 Thinking almost the same prompt (I had to add some instructions because the first response it gave was way too technical) and it gave an okay answer, which you can read here. (Archive link here.)
I tried to Google for human-written explanations of the similarities and differences first, since that's obviously preferable. But I couldn't quickly find one, probably because there's no particular reason to compare these concepts directly to each other.
No, those definitions quite clearly don’t say anything about data efficiency or generalization, or the other problems I raised.
I think you have misunderstood the concept of continual learning. It doesn’t mean what you seem to think it means. You seem to be confusing the concept of continual learning with some much more expansive concept, such as generality.
If I’m wrong, you should be able to quite easily provide citations that clearly show otherwise.
I don't think Karpathy would describe his view as involving any sort of discontinuity in AI development. If anything his views are the most central no-discontinuity straight-lines-on-graphes view (no intelligence explosion accelerating the trends, no winter decelerating the trends). And if you think the mean date for AGI is 2035 then it would take extreme confidence (on the order of variance of less than a year) to claim AGI is less than 0.1% likely by 2032!
I was only mentioning Karpathy as someone reasonable who repeatedly points out the lack of online learning and seems to have (somewhat) longer timelines because of that. This is solely based on my general impression. I agree the stated probabilities seem wildly overconfident.
I don’t know what Andrej Karpathy’s actual timeline for AGI is. In the Dwarkesh Patel interview that everyone has been citing, Karpathy says he thinks it’s a decade until we get useful AI agents, not AGI. This implies he thinks AGI is at least a decade away, but he doesn’t actually directly address when he thinks AGI will arrive.
After the interview, Karpathy made a clarification on Twitter where he said 10 years to AGI should come across to people as highly optimistic in the grand scheme of things, which maybe implies he does actually think AGI is 10 years away and will arrive at the same time as useful AI agents. However, it’s ambiguous enough I would hesitate to interpret it one way or another.
I could be wrong, but I didn’t get the impression that continual learning or online learning was Karpathy’s main reason (let alone sole reason) for thinking useful AI agents are a decade away, or for his other comments that express skepticism or pessimism — relative to people with 5-year AGI timelines — about progress in AI or AI capabilities.
Continual learning/online learning is not one of the main issues raised in my post and while I think it is an important issue, you can hand-wave away continual learning and still have problems with scaling limits, learning from video data, human examples to imitation learn from, data inefficiency, and generalization.
It’s not just Andrej Karpathy but a number of other prominent AI researchers, such as François Chollet, Yann LeCun, and Richard Sutton, who have publicly raised objections to the idea that very near-term AGI is very likely via scaling LLMs. In fact, in the preamble of my post I linked to a previous post of mine where I discuss how a survey of AI researchers found they have a median timeline for AGI of over 20 years (and possibly much, much longer than 20 years, depending how you interpret the survey), and how, in another survey, 76% of AI experts surveyed think scaling LLMs or other current techniques is unlikely or very unlikely to reach AGI. I’m not defending a fringe, minority position in the AI world, but in fact something much closer to the majority view than what you typically see on the EA Forum.