[Link Post] Interesting shallow round-up of reasons to be skeptical that transformative AI or explosive economic growth are coming soon

David Mathers🔸

[Link Post] Interesting shallow round-up of reasons to be skeptical that transformative AI or explosive economic growth are coming soon

David Mathers🔸

21 min read · Jun 28, 2023

Comments 8

Sorted by

New & upvoted

Aaron_Scher

The article doesn't seem to have a comment section so I'm putting some thoughts here.

Economic growth: I don't feel I know enough about historical economic growth to comment on how much to weigh the "the trend growth rate of GDP per capita in the world's frontier economy has never exceeded three percent per year." I'll note that I think the framing here is quite different than that of Christiano's Hyperbolic Growth, despite them looking at roughly the same data as far as I can tell.
Scaling current methods: the article seems to cherrypick the evidence pretty significantly and makes the weak claim that "Current methods may also not be enough." It is obvious that my subjective probability that current methods are enough should be <1, but I have yet to come across arguments that push that credence below say 50%.
- "Scaling compute another order of magnitude would require hundreds of billions of dollars more spending on hardware." This is straightforwardly false. The table included in the article, from the Chinchilla paper with additions, is a bit confusing because it doesn't include where we are now, and because it lists only model size rather than total training compute (FLOP). Based on Epoch's database of models, PaLM 2 is trained with about 7.34e24 FLOP, and GPT-4 is estimated at 2.10e25 (note these are not official numbers). This corresponds to being around the 280B param (9.9e24 FLOP) or 520B param (3.43e25 FLOP) rows in the table. In this range, tens of millions of dollars are being spent on compute for the biggest training runs now. It should be obvious that you can get a couple more orders of magnitude more compute before hitting hundreds of billions of dollars. In fact, the 10 Trillion param row in the table, listed at $28 billion, corresponds to a total training compute of 1.3e28 FLOP, which is more than 2 orders of magnitude above the biggest publicly-known models are estimated. I agree that cost may soon become a limiting factor, but the claim that an order of magnitude would push us into hundreds of billions is clearly wrong given that currently costs are tens of millions.
- Re cherrypicking data, I guess one of the most important points that seems to be missing from this section is the rate of algorithmic improvement. I would point to Epoch's work here.
"Constitutional AI, a state-of-the-art alignment technique that has even reached the steps of Capitol Hill, also does not aim to remove humans from the process at all: "rather than removing human supervision, in the longer term our goal is to make human supervision as efficacious as possible."" This seems to me like a misunderstanding of Constitutional AI, for which a main component is "RL from AI Feedback." Constitutional AI is all about removing humans from the loop in order to get high quality data more efficiently. There's a politics thing where developers don't want to say they're removing human supervision, and it's also true that human supervision will probably play a role in data generation in the future, but the human:total (AI+human) contribution to data ratio is surely going to go down. For example research using AIs where we used to use humans, see also Anthropic's paper Model Written Evaluations, and the AI-labeled MACHIAVELLI benchmark. More generally, I would bet the trend toward automating datasets and benchmarks will continue, even if humans remain in the loop somewhat; insofar as humans are a limiting factor, developers will try to make them less necessary, and we already have AIs that perform very similarly to human raters at some tasks.
"We are constantly surprised in our day jobs as a journalist and AI researcher by how many questions do not have good answers on the internet or in books, but where some expert has a solid answer that they had not bothered to record. And in some cases, as with a master chef or LeBron James, they may not even be capable of making legible how they do what they do." Not a disagreement, but I do wonder how much of this is a result of information being diffuse and just hard to properly find, a kind of task I expect AIs to be good at. For instance, 2025 language models equipped with search might be similarly useful to if you had a panel of relevant experts you could ask questions to.
Noting that section 3: "Even if technical AI progress continues, social and economic hurdles may limit its impact" matters for some outcomes and not for others. It matters given the authors define "transformative AI in terms of its observed economic impact." It matters for many outcomes I care about like human well-being, that are related to economic impacts. It applies less to worries around existential risk and human disempowerment, for which powerful AIs may pose risks even while not causing large economic impacts ahead of time (e.g., bioterrorism doesn't require first creating a bunch of economic growth).
- Overall I think the claim of section 3 is likely to be right. A point pushing the other direction is that there may be a regulatory race to the bottom where countries want to enable local economic growth from AI and so relax regulations, think medical tourism for all kinds of services.
"Yet as this essay has outlined, myriad hurdles stand in the way of widespread transformative impact. These hurdles should be viewed collectively. Solving a subset may not be enough." I definitely don't find the hurdles discussed here to be sufficient to make this claim. It feels like there's a motte and bailey, where the easy to defend claim is "these 3+ hurdles might exist, and we don't have enough evidence to discount any of them", and the harder to defend claim is "these hurdles disjunctively prevent transformative AI in the short term, so all of them must be conquered to get such AI." I expect this shift isn't intended by the authors, but I'm noting that I think it's a leap.
"Scenarios where AI grows to an autonomous, uncontrollable, and incomprehensible existential threat must clear the same difficult hurdles an economic transformation must." I don't think this is the case. For example, section 3 seems to not apply as I mentioned earlier. I think it's worth noting that AI safety researcher Eliezer Yudkowsky has made a similar argument to what you make in section 3, and he is also thinks existential catastrophe in the near term is likely. I think the point your making here is directionally right, however, that AI which poses existential risk is likely to be transformative in the sense you're describing. That is, it's not necessary for such AI to be economically transformative, and there are a couple other ways catastrophically-dangerous AI can bypass the hurdles you lay out, but I think it's overall a good bet that existentially dangerous AIs are also capable of being economically transformative, so the general picture of hurdles, insofar as they are real, will affect such risks as well [I could easily see myself changing my mind about this with more thought]. I welcome more discussion on this point and have some thoughts myself, but I'm tired and won't include them in this comment; happy to chat privately about where "economically transformative" and "capable of posing catastrophic risks" lie on various spectrums.

While my comment has been negative and focused on criticism, I am quite glad this article was written. Feel free to check out a piece I wrote, laying out some of my thinking around powerful AI coming soon, which is mostly orthogonal to this article. This comment was written sloppily, partially as my off-the-cuff notes while reading, sorry for any mistakes and impolite tone.

zhengdong

Hey Aaron, thanks for your thorough comment. While we still disagree (explained a bit below), I'm also quite glad to read your comment :)

Re scaling current methods: The hundreds of billions figure we quoted does require more context not in our piece; SemiAnalysis explains in a bit more detail how they get to that number (eg assuming training in 3mo instead of 2 years). We don't want to haggle over the exact scale before it becomes infeasible, though---even if we get another 2 OOM in, we wanted to emphasize with our argument that 'the current method route' 1) requires regular scientific breakthroughs of the pre-TAI sort, and 2) even if we get there doesn't guarantee capabilities that look like magic compared to what we have now, depending on how much you believe in emergence. Both would be bottlenecks. We're pretty sure that current capabilities can be economically useful with more people, more fine-tuning. Just skeptical of the sudden emergence of the exact capabilities we need for transformative growth.

On Epoch's work on algorithmic progress specifically, we think it's important to note that:

1) They do this by measuring progress on computer vision benchmarks, which isn't a good indicator of progress in either algorithms for control (physical world important for TAI) or even language---it might be cheeky to say, little algorithmic progress there; just scale ;) Computer vision is also the exact example Schaeffer et al. gives for the subfield where emergent abilities do not arise---until you induce them by intentionally crafting the evaluations.

2) That there even is a well-defined benchmark is a good sign for beating that benchmark. AI benefits from quantifiable evaluation (beating a world champion, CASP scores) when it measures what we want. But we'd say for really powerful AI we don't know what we want (see our wrong direction / philosophy hurdle), plus at some point the quantifiable metrics we do have stop measuring what we really want. (Is there really a difference between models that get 91.0 and 91.1 top-1 accuracy on ImageNet? Do people really look at MMLU over qualitative experience when they choose which language model to play with?)

3) We don't discount algorithmic progress at all! In fact we cite SemiAnalysis and the Epoch team's suggestions on where to research next. But again, these require human breakthroughs, bottlenecked on human research timescales---we don't have a step by step progress we can just follow to improve a metric to TAI, so hard-won past breakthroughs doesn't guarantee future ones happen at the same clip.

Re Constitutional AI: We agree that researchers will continue searching for ways to use human feedback more efficiently. But under our Baumol framework, the important step is going from one to zero, not n to one. And there we find it hard to believe that in high stakes situations (say, judging AI debates), that safety researchers are willing to hand over the reins. We'd also really contest the 'perform very similarly to human raters' is enough---it'd be surprising if we already have a free lunch, no information lost, way to simulate humans well enough to make better AI.

Re 2025 language models equipped with search: For this to be as useful as a panel of experts, the models need to be searching an index where what the experts know is recorded, in some sense, which 1) doesn't happen (experts are busy being experts) 2) is sometimes impossible (chef, LeBron) 3) maybe less likely in the future when an LLM is going to just hoover up your hard won expertise? I know you mentioned you don't disagree with our point here though.

Re motte and bailey: We agree that our hurdles may have overlap. But the point of our Baumol framework is that any valid hurdle, where we don't know if it's fundamentally the same problem that causes other hurdles, each has the potential to bottleneck transformative growth. And we allude to several cases where for one reason or another a promising invention did not meet expectations precisely because they could not clear them all.

Hope this clarifies our view. Not conclusive, of course, we're happy, like your piece, to also be going for intuition pumps to temper expectations.

Aaron_Scher

Thanks for your response. I'll just respond to a couple things.

Re Constitutional AI: I agree normatively that it seems bad to hand over judging AI debates to AIs^[1]. I also think this will happen. To quote from the original AI Safety via Debate paper,

Human time is expensive: We may lack enough human time to judge every debate, which we can address by training ML models to predict human reward as in Christiano et al. [2017]. Most debates can be judged by the reward predictor rather than by the humans themselves. Critically, the reward predictors do not need to be as smart as the agents by our assumption that judging debates is easier than debating, so they can be trained with less data. We can measure how closely a reward predictor matches a human by showing the same debate to both.

We'd also really contest the 'perform very similarly to human raters' is enough---it'd be surprising if we already have a free lunch, no information lost, way to simulate humans well enough to make better AI.

I also find this surprising, or at least I did the first 3 times I came across medium-quality evidence pointing this direction. I don't find it as surprising any more because I've updated my understanding of the world to "welp, I guess 2023 AIs actually are that good on some tasks." Rather than making arguments to try and convince you, I'll just link some of the evidence that I have found compelling, maybe you will too, maybe not: Model Written Evals, MACHIAVELLI benchmark, Alpaca (maybe the most significant for my thinking), this database, Constitutional AI.

I'm far from certain that this trend, of LLMs being useful for making better LLMs and for replacing human feedback, continues rather than hitting a wall in the next 2 years, but it does seem more likely than not to me, based on my read of the evidence. Some important decisions in my life rely on how soon this AI stuff is happening (for instance if we have 20+ years I should probably aim to do policy work), so I'm pretty interested in having correct views. Currently, LLMs improving the next generation of AIs via more and better training data is one of the key factors in how I'm thinking about this. If you don't find these particular evidences compelling and are able to explain why, that would be useful to me!

^{^}
I'm actually unsure here. I expect there are some times where it's fine to have no humans in the loop and other times where it's critical. It generally gives me the ick to take humans out of the loop, but I expect there are some times where I would think it's correct.

zhengdong

Makes sense that this would be a big factor in what to do with our time, and AI timelines. And we're surprised too by how AI can overperform expectations, like in the sources you cited.

We'd still say the best way of characterizing the problem of creating synthetic data is that it's a wide open problem, rather than high confidence that naive approaches using current LMs will just work. How about a general intuition instead of parsing individual sources. We wouldn't expect making the dataset bigger by just repeating the same example over and over to work. We generate data by having 'models' of the original data generators, humans. If we knew what exactly made human data 'good,' we could optimize directly for it and simplify massively (this runs into the well-defined eval problem again---we can craft datasets to beat benchmarks of course).

An analogy (a disputed one, to be fair) is Ted Chiang's lossy compression. So for every case of synthetic data working, there's also cases where it fails, like Shumailov et el. we cited. If we knew exactly what made human data 'good,' we'd argue you wouldn't see labs continue to ramp up hiring contractors specifically to generate high-quality data in expert domains, like programming.

A fun exercise---take a very small open-source dataset, train your own very small LM, and have it augment (double!) its own dataset. Try different prompts, plot n-gram distributions vs the original data. Can you get one behavior out of the next generation that looks like magic compared to the previous, or does improvement plateau? May have nitpicks with this experiment, but I don't think it's that different to what's happening at large scale.

Erich_Grunewald 🔸

Re scaling current methods: The hundreds of billions figure we quoted does require more context not in our piece; SemiAnalysis explains in a bit more detail how they get to that number (eg assuming training in 3mo instead of 2 years).

That's hundreds of billions with current hardware. (Actually, not even current hardware, but the A100 which is last-gen; the H100 should already do substantially better.) But HW price-performance currently doubles every ~2 years. Yes, Moore's Law may be slowing, but I'd be surprised if we don't get another OOM improvement in price-performance during the next decade, especially given the insatiable demand for effective compute these days.

We don't want to haggle over the exact scale before it becomes infeasible, though---even if we get another 2 OOM in, we wanted to emphasize with our argument that 'the current method route' 1) requires regular scientific breakthroughs of the pre-TAI sort, and 2) even if we get there doesn't guarantee capabilities that look like magic compared to what we have now, depending on how much you believe in emergence. Both would be bottlenecks.

Yeah, I agree things would be a lot slower without algorithmic breakthroughs. Those do seem to be happening at a pretty good pace though (not just looking at ImageNet, but also looking at ML research subjectively). I'd assume they'll keep happening at the same rate so long as the number of people (and later, possibly AIs) focused on finding them keeps growing at the same rate.

Peter Slattery 🔸

I really appreciate that you took the time to provide such a detailed response to these arguments. I want to say this pretty often when on the forum, and maybe I should do it more often!

Arjun Ramani

Thanks for posting David! Better title than ours :) Would you mind pasting the text of the article for ease of the reader?

David Mathers🔸

Done. Sorry I was slow.

Comments

More from the author

117

Is There Actually a Standard or Convincing Response to David Thorstad's Criticisms of the Value of X-Risk Reduction and of Longtermism?

David Mathers🔸·1y ago·2m read

David Mathers🔸·3y ago·10m read

Governments Might Prefer Bringing Resources Back to the Solar System Rather than Space Settlement in Order to Maintain Control, Given that Governing Interstellar Settlements Looks Almost Impossible

David Mathers🔸·3y ago·6m read

Curated and popular this week

Was Partisanship Good for the Environmental Movement?

Jeffrey Heninger·2y ago·Curated 3d ago·6m read

This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan? Summary Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions. Public Opinion...

127

Possible mistake EAs are making and shout out to Pause AI UK

Michelle_Hutchinson·4d ago·4m read

I think right now EAs might be making a significant mistake by paying insufficient attention to the political realm. As EAs we tend to figure out what’s most impactful for us to work on and focus hard. That’s great! But there are various actions that are ‘non-delegatable’ - the extent to which an individual can do the action is limited (like voting, going to a protest, making hard money contributions to particular campaigns). It might be useful if we were all more in the habit of doing variou...

102

New Video from AI in Context: The Fall and Rise of Sam Altman

ChanaMessinger, phoebe b, Aric Floyd·6d ago·3m read

New Video from AI in Context: The Fall and Rise of Sam Altman If you want to skip straight to the video, here it is! AI in Context is excited to be back with our fourth video! For those just hearing from us, we make videos for 80,000 Hours, telling stories about transformative AI...

Recent opportunities to take action

$1M AI x-risk grant round is live on grantmaking.ai - apply for funding, review applicants, or fund projects

Matt Brooks·2h ago·3m read

127

Possible mistake EAs are making and shout out to Pause AI UK

Michelle_Hutchinson·4d ago·4m read

Build a flourishing EA group at the University of Toronto

Joseph Kostousov, Sophia Wan (navarhontes)·1w ago·1m read

Aaron_Scher

The article doesn't seem to have a comment section so I'm putting some thoughts here.

Economic growth: I don't feel I know enough about historical economic growth to comment on how much to weigh the "the trend growth rate of GDP per capita in the world's frontier economy has never exceeded three percent per year." I'll note that I think the framing here is quite different than that of Christiano's Hyperbolic Growth, despite them looking at roughly the same data as far as I can tell.
Scaling current methods: the article seems to cherrypick the evidence pretty significantly and makes the weak claim that "Current methods may also not be enough." It is obvious that my subjective probability that current methods are enough should be <1, but I have yet to come across arguments that push that credence below say 50%.
- "Scaling compute another order of magnitude would require hundreds of billions of dollars more spending on hardware." This is straightforwardly false. The table included in the article, from the Chinchilla paper with additions, is a bit confusing because it doesn't include where we are now, and because it lists only model size rather than total training compute (FLOP). Based on Epoch's database of models, PaLM 2 is trained with about 7.34e24 FLOP, and GPT-4 is estimated at 2.10e25 (note these are not official numbers). This corresponds to being around the 280B param (9.9e24 FLOP) or 520B param (3.43e25 FLOP) rows in the table. In this range, tens of millions of dollars are being spent on compute for the biggest training runs now. It should be obvious that you can get a couple more orders of magnitude more compute before hitting hundreds of billions of dollars. In fact, the 10 Trillion param row in the table, listed at $28 billion, corresponds to a total training compute of 1.3e28 FLOP, which is more than 2 orders of magnitude above the biggest publicly-known models are estimated. I agree that cost may soon become a limiting factor, but the claim that an order of magnitude would push us into hundreds of billions is clearly wrong given that currently costs are tens of millions.
- Re cherrypicking data, I guess one of the most important points that seems to be missing from this section is the rate of algorithmic improvement. I would point to Epoch's work here.
"Constitutional AI, a state-of-the-art alignment technique that has even reached the steps of Capitol Hill, also does not aim to remove humans from the process at all: "rather than removing human supervision, in the longer term our goal is to make human supervision as efficacious as possible."" This seems to me like a misunderstanding of Constitutional AI, for which a main component is "RL from AI Feedback." Constitutional AI is all about removing humans from the loop in order to get high quality data more efficiently. There's a politics thing where developers don't want to say they're removing human supervision, and it's also true that human supervision will probably play a role in data generation in the future, but the human:total (AI+human) contribution to data ratio is surely going to go down. For example research using AIs where we used to use humans, see also Anthropic's paper Model Written Evaluations, and the AI-labeled MACHIAVELLI benchmark. More generally, I would bet the trend toward automating datasets and benchmarks will continue, even if humans remain in the loop somewhat; insofar as humans are a limiting factor, developers will try to make them less necessary, and we already have AIs that perform very similarly to human raters at some tasks.
"We are constantly surprised in our day jobs as a journalist and AI researcher by how many questions do not have good answers on the internet or in books, but where some expert has a solid answer that they had not bothered to record. And in some cases, as with a master chef or LeBron James, they may not even be capable of making legible how they do what they do." Not a disagreement, but I do wonder how much of this is a result of information being diffuse and just hard to properly find, a kind of task I expect AIs to be good at. For instance, 2025 language models equipped with search might be similarly useful to if you had a panel of relevant experts you could ask questions to.
Noting that section 3: "Even if technical AI progress continues, social and economic hurdles may limit its impact" matters for some outcomes and not for others. It matters given the authors define "transformative AI in terms of its observed economic impact." It matters for many outcomes I care about like human well-being, that are related to economic impacts. It applies less to worries around existential risk and human disempowerment, for which powerful AIs may pose risks even while not causing large economic impacts ahead of time (e.g., bioterrorism doesn't require first creating a bunch of economic growth).
- Overall I think the claim of section 3 is likely to be right. A point pushing the other direction is that there may be a regulatory race to the bottom where countries want to enable local economic growth from AI and so relax regulations, think medical tourism for all kinds of services.
"Yet as this essay has outlined, myriad hurdles stand in the way of widespread transformative impact. These hurdles should be viewed collectively. Solving a subset may not be enough." I definitely don't find the hurdles discussed here to be sufficient to make this claim. It feels like there's a motte and bailey, where the easy to defend claim is "these 3+ hurdles might exist, and we don't have enough evidence to discount any of them", and the harder to defend claim is "these hurdles disjunctively prevent transformative AI in the short term, so all of them must be conquered to get such AI." I expect this shift isn't intended by the authors, but I'm noting that I think it's a leap.
"Scenarios where AI grows to an autonomous, uncontrollable, and incomprehensible existential threat must clear the same difficult hurdles an economic transformation must." I don't think this is the case. For example, section 3 seems to not apply as I mentioned earlier. I think it's worth noting that AI safety researcher Eliezer Yudkowsky has made a similar argument to what you make in section 3, and he is also thinks existential catastrophe in the near term is likely. I think the point your making here is directionally right, however, that AI which poses existential risk is likely to be transformative in the sense you're describing. That is, it's not necessary for such AI to be economically transformative, and there are a couple other ways catastrophically-dangerous AI can bypass the hurdles you lay out, but I think it's overall a good bet that existentially dangerous AIs are also capable of being economically transformative, so the general picture of hurdles, insofar as they are real, will affect such risks as well [I could easily see myself changing my mind about this with more thought]. I welcome more discussion on this point and have some thoughts myself, but I'm tired and won't include them in this comment; happy to chat privately about where "economically transformative" and "capable of posing catastrophic risks" lie on various spectrums.

^{^}

I'm actually unsure here. I expect there are some times where it's fine to have no humans in the loop and other times where it's critical. It generally gives me the ick to take humans out of the loop, but I expect there are some times where I would think it's correct.