We'd also like to see this debated more. Getting expert participation is already challenging on a simple survey, but we're hoping that publishing this will prompt skeptics (and believers) to more publicly articulate their arguments.
Oh yea, I didn't mind the title at all (although I do think it's usefully more precise now :)
Agreed on additively separable utility being unrealistic. My point (which wasn't clearly spelled out) was not that GDP growth and unit production can't look dramatically. (We already see that in individual products like transistors (>> GDP) and rain dances (<< GDP).) It was that post-full-automation isn't crucially different than pre-full-automation unless you make some imo pretty extreme assumptions to distinguish them.
By "extracting this from our utility function", I just mean my vague claim that, insofar as we are uncertain about GDP growth post-full-automation, understanding better the sorts of things people and superhuman intelligences want will reduce that uncertainty more than learning about the non-extreme features of future productivity heterogeneity (although both do matter if extreme enough). But I'm being so vague here that it's hard to argue against.
My point is that the existence of a human-only good satisfying (1) and (2) is unnecessary: the very same effect can arise even given true full automation, not due to a limitation of our ability to fully automate, but due to the fact that a technological advance can encompass full automation and go beyond yielding a world where we can produce way more of everything we would ever produce without the advance, by letting us produce some goods we otherwise wouldn't have been on track to produce at all. This has not been widely appreciated.
OK but this key feature of not being on track to produce Good 2 only happens in your model specifically because you define automation to be a thing that takes Good-2 productivity from 0 to something positive. I think this is in conflict with the normal understanding of what "automation" means! Automation is usually understood to be something that increases the productivity of something that we could already produce at least a little of in principle, even if the additional efficiency means actual spending on a specific product goes from 0 to 1. And as long as we could produce a little of Good 2 pre-automation, the utility function in your model implies that the spending in the economy would eventually be dominated by Good 2 (and hence GDP growth rates would be set by the growth in productivity of Good 2) even without full automation (unless the ratio of Good-1 and Good-2 productivity is growing superexponentially in time).
What kind product would we be unable to produce without full automation, even given arbitrary time to grow? Off the top of my head I can only think of something really ad-hoc like "artisanal human paintings depicting the real-world otherwise-fully-autonomous economy".
That's basically what makes me think that "the answer is already in our utility function", which we could productively introspect on, rather than some empirical uncertainty about what products full automation will introduce.
presumably full automation will coincide with not only a big increase in productivity growth (which raises GDP growth, in the absence of a random "utility function kink") but also a big change in the direction of productivity growth, including via making new products available (which introduces the kind of "utility function kink" that has an arbitrary effect on GDP growth).
I'm not sure what the best precise math statement to make here is, but I suspect that at least for "separable" utility functions of the form u(x1,…,xN)=∑nun(xn) you need either a dramatic difference in diminishing returns for the un (e.g., log vs. linear as in your model) or you need a super dramatic difference in the post-full-automation productivity growth curves (e.g., one grows exponentially and the other grows superexponentially) that is absent pre-automation. (I don't think it's enough that the productivities grow at different rates post-automation.) So I still think we can extract this from our utility function without knowing much about the future, although maybe there's a concrete model that would show that's wrong.
The Baumol point is that among a set of already existing goods which we don’t see as very substitutable, GDP growth can be pulled down arbitrarily by the slow-growing goods. ...The point I’m making here is that even if we fully automate production, and even if the quantity of every good existing today then grows arbitrarily quickly, we might create new goods as well. Once we do so, if the production of old goods grows quickly while our production of the new goods doesn’t, GDP growth may be slow.
Not sure this is disagreement per se, but I think the surprising behavior of GDP in your model is almost entirely due to the shape of the utility function and doesn't have much to do with either (1) the distinction between existing vs new products or (2) automation. In other words, I think this is still basically Baumol, although I admit to a large extent I'm just arguing here about preferred conceptual framing rather than facts.
Consider modifying your models as follows (which I presume makes it more like a traditional Baumol model):
There is only one time period
Good 2 is always available
Productivity of Good 1 grows at rates r1 and productivity of Good 2 grows at the much slower rate r2≪r1. Specifically, the production possibility frontier at time t is f(x1,x2)=1, under constraints x1,x2≥0, where f(x1,x2):=x1e−r1t+x2e−r2t.
Using your same utility function u(x1,x2)=log(x1)+x2, production is fully devoted to Good 1 for all times t<0, and during that time GDP is growing at r1. Then at time t=0, it becomes worthwhile to start producing Good 2. For t>0, the productivity growth rate of Good 1 remains much higher than the that of Good 2 (r1≫r2), and indeed the number of units of Good 1 produced grows exponentially faster than Good 2: x1(t)=e(r1−r2)t , x2(t)=er2t−1 Nonetheless, the marginal value of Good 1 plummets due to the log in the utility function. Specifically, the relative price of Good 1 falls exponentially in time, p1(t)/p2(t)=e−(r1−r2)t where pi(t):=dfdxi(t), as does Good 1's price-weighted fraction of production:
p1(t)x1(t)p1(t)x1(t)+p2(t)x2(t)=e−r2t
GDP growth falls from r1 and exponentially asymptotes to r2 for large t.
Two points on the model:
Although production of Good 2 was 0 for t<0, this isn't because it was unavailable or impossibly expensive. Indeed, Good 1 is getting easier to produce relative to Good 2 for all times, so in some sense Good 2 is (relatively) easiest to produce in the distant past. Rather, no units of Good 2 are produced for t<0 because they just aren't valued according to the chosen utility function.
The transition from the all-Good-1-economy (t<0) to the mostly-Good-2-economy (t>0) is due to hitting the key point in the utility curve, not to any changes in productivity growth rates (which remain constant). You can throw in automation (i.e., a productivity shock) at any point, say, increasing both r1 and r2 by a constant factor γ, and you'll still have a net fall in GDP growth rates so long as γr2<r1.
Ok, so then what are the take-aways for AI? By cleanly separating the utility-function effect from shocks to productivity, I think this is reason for us to believe that the past is a reasonable guide to the future. Yes, there could be weird kinks in our utility function, but in terms of revealing kinks there's not much reason to think that AI-induced productivity gains will be importantly different than productivity gains from the past.
What quantity should we measure if not GDP?
First, we might consider this: Take the yearly output of today's economy (2B tons of steel, etc.) and, for each future date, divide the future GDP by the cost (at that time) of producing today's basket of goods. But this doesn't work: this growth rate will, for large times, be dominated by growth rate of the good in the today's basket whose production is growing slowest in the future. (In our model, that would be Good 2 growing at rate r2.) So it's still Baumol vulnerable.
Then, we might consider: compute the value of all the good produced at a future date using today's prices. For a single year this is just real GDP, but isn't the same thing as real GDP for more than 1 year because it isn't chained. And lack of chaining is why this quantity is bad: if you have a niche good whose production is skyrocketing, this growth rate would explode even if price was similarly falling and nothing else amazing was happening in the economy. For example, if 1 transistor cost ~$1k in 1949, and we now produce almost a sextillion (~1021 ) per year, that would value today's economy at about an octillion (1024) dollars. In our model, this would be Good 1 growing at rate r1, but this isn't what we want.
I think there's just no getting around the fact that the kind of growth we care about is unavoidably wrapped up in our utility function. But as long as some fraction of people want to build Jupiter brains and explore Andromeda enough that they don't devote ~all their efforts to goals that are intrinsically upper bounded, I expect AGI to lead to rapid real GDP growth (although it does likely eventually end with light-speed limits or whatever).
If growth were slow post-singularity, I think that would imply something pretty weird about human utility in this universe (or rather, the utility of the beings controlling the economy). There could of course still be crazy things happening like wild increases in energy usage at the same time, but this isn't too different than how wild the existence of nanometer-scale transistors are relative to pre-industrial civilization. If you care about those crazy things independent of GDP (which is a measure of how fast the world overall is getting what it wants), you should probably just measure them directly, e.g., energy usage, planets colonized, etc.
Nice post. The argument for simultaneity (most models train at the same time, and then are evaled at the same time, and then released at the same time) seems ultimately to be based on the assumption that the training cap grows in discrete amounts (say, a factor of 3) each year.
> * We want to prevent model training runs of size N+1 until alignment researchers have had time to study and build evals based on models of size N. > * For-profit companies will probably all want to start going as soon as they can once the training cap is lifted. > * So there will naturally be lots of simultaneous runs...
But why not smoothly raise the size limit? (Or approximate that by raising it in small steps every week or month?) The key feature of this proposal, as I see it, is that there is a fixed interval for training and eval, to prevent incentives to rush. But that doesn't require simultaneity.
I listed this example in my comment, it was incorrect by an order of magnitude, and it was a retrodiction. "I didn't look up the data on Google beforehand" does not make it a prediction.
I'm also a little surprised you think that modeling when we will have systems using similar compute as the human brain is very helpful for modeling when economic growth rates will change. (Like, for sure someone should be doing it, but I'm surprised you're concentrating on it much.) As you note, the history of automation is one of smooth adoption. And, as I think Eliezer said (roughly), there don't seem to be many cases where new tech was predicted based on when some low-level metric would exceed the analogous metric in a biological system. The key threshold for recursive feedback loops (*especially* compute-driven ones) is how well they perform on the relevant tasks, not all tasks. And the way in which machines perform tasks usually looks very different than how biological systems do it (bird vs. airplanes, etc.).
If you think that compute is the key bottleneck/driver, then I would expect you to be strongly interested in what the automation of the semiconductor industry would look like.
I like this post a lot but I will disobey Rapoport's rules and dive straight into criticism.
Historically, many AI researchers believed that creating general AI would be more about coming up with the right theories of intelligence, but over and over again, researchers eventually found that impressive results only came after the price of computing fell far enough that simple, "blind" techniques began working (Sutton 2019).
I think this is a poor way to describe a reasonable underlying point. Heavier-than-air flying machines were pursued for centuries, but airplanes appeared almost instantly (on a historic scale) after the development of engines with sufficient power density. Nonetheless, it would be confusing to say "flying is more about engine power than the right theories of flight". Both are required. Indeed, although the Wright brothers were enabled by the arrival of powerful engines, they beat out other would-be inventors (Ader, Maxim, and Langley) who emphasized engine power over flight theory. So a better version of your claim has to be something like "compute quantity drives algorithmic ability; if we independently vary compute (e.g., imagine an exogenous shock) then algorithms follow along", which (I think) is what you arguing further in the post.
But this also doesn't seem right. As you observe, algorithmic progress has been comparable to compute progress (both within and outside of AI). You list three "main explanations" for where algorithmic progress ultimately comes from and observe that only two of them explain the similar rates of progress in algorithms and compute. But both of these draw a causal path from compute to algorithms without considering the (to-me-very-natural) explanation that some third thing is driving them both at a similar rate. There are a lot of options for this third thing! Researcher-to-researcher communication timescales, the growth rate of the economy, the individual learning rate of humans, new tech adoption speed, etc. It's plausible to me that compute and algorithms are currently improving more or less as fast as they can, given their human intermediaries through one or all of these mechanisms.
The causal structure is key here, because the whole idea is to try and figure out when economic growth rates change, and the distinction I'm trying to draw becomes important exactly around the time that you are interested in: when the AI itself is substantially contributing to its own improvement. Because then those contributions could be flowing through at least three broad intermediaries: algorithms (the AI is writing its own code better), compute (the AI improves silicon lithography), or the wider economy (the AI creates useful products that generate money which can be poured into more compute and human researchers).
Of course, even if AI performance is, in principle, predictable as a function of scale, we lack data on how AIs are currently improving on the vast majority of tasks in the economy, hindering our ability to predict when AI will be widely deployed. While we hope this data will eventually become available, for now, if we want to predict important AI capabilities, we are forced to think about this problem from a more theoretical point of view.
Humans have been automating mechanical task for many centuries, and information-processing tasks for many decades. Moore's law, the growth rate of the thing (compute) that you ague drives everything else, has been stated explicitly for almost 58 years (and presumably applicable for at least a few decade before that). Why are you drawing a distinction between all the information processing that happened in the past and "AI", which you seem to be taking as a basket of things that have mostly not had a chance to be applied yet (so no data)?
If compute is the central driving force behind AI, and transformative AI (TAI) comes out of something looking like our current paradigm of deep learning, there appear to be a small set of natural parameters that can be used to estimate the arrival of TAI. These parameters are:
The total training compute required to train TAI
The average rate of growth in spending on the largest training runs, which plausibly hits a maximum value at some significant fraction of GWP
The average rate of increase in price-performance for computing hardware
The average rate of growth in algorithmic progress
This list is missing the crucial parameters that would translate the others into what we agree is most notable: economic growth. I think needs to be discussed much more in section 4 for it to be a useful summary/invitation to the models you mention.
I agree it's important to keep the weaker fraud protection on debit cards in mind. However, for the use I mentioned above, you can just lock the debit card and only unlock it when you have a cash flow problem. (Btw, if you don't use your IB debit card, you should lock it even if you aren't using it.) Debit card liability is capped at $50 and $500 if you report fraudulent transactions within 2 days and 60 days, respectively.
That said, I have most of my net worth elsewhere, so I'm less worried about tail risks than you would reasonably be if you're mostly invested through IB.
("Optimism" is an unusual way to describe it :)
We'd also like to see this debated more. Getting expert participation is already challenging on a simple survey, but we're hoping that publishing this will prompt skeptics (and believers) to more publicly articulate their arguments.
Oh yea, I didn't mind the title at all (although I do think it's usefully more precise now :)
Agreed on additively separable utility being unrealistic. My point (which wasn't clearly spelled out) was not that GDP growth and unit production can't look dramatically. (We already see that in individual products like transistors (>> GDP) and rain dances (<< GDP).) It was that post-full-automation isn't crucially different than pre-full-automation unless you make some imo pretty extreme assumptions to distinguish them.
By "extracting this from our utility function", I just mean my vague claim that, insofar as we are uncertain about GDP growth post-full-automation, understanding better the sorts of things people and superhuman intelligences want will reduce that uncertainty more than learning about the non-extreme features of future productivity heterogeneity (although both do matter if extreme enough). But I'm being so vague here that it's hard to argue against.
Thanks!
OK but this key feature of not being on track to produce Good 2 only happens in your model specifically because you define automation to be a thing that takes Good-2 productivity from 0 to something positive. I think this is in conflict with the normal understanding of what "automation" means! Automation is usually understood to be something that increases the productivity of something that we could already produce at least a little of in principle, even if the additional efficiency means actual spending on a specific product goes from 0 to 1. And as long as we could produce a little of Good 2 pre-automation, the utility function in your model implies that the spending in the economy would eventually be dominated by Good 2 (and hence GDP growth rates would be set by the growth in productivity of Good 2) even without full automation (unless the ratio of Good-1 and Good-2 productivity is growing superexponentially in time).
What kind product would we be unable to produce without full automation, even given arbitrary time to grow? Off the top of my head I can only think of something really ad-hoc like "artisanal human paintings depicting the real-world otherwise-fully-autonomous economy".
That's basically what makes me think that "the answer is already in our utility function", which we could productively introspect on, rather than some empirical uncertainty about what products full automation will introduce.
I'm not sure what the best precise math statement to make here is, but I suspect that at least for "separable" utility functions of the form u(x1,…,xN)=∑nun(xn) you need either a dramatic difference in diminishing returns for the un (e.g., log vs. linear as in your model) or you need a super dramatic difference in the post-full-automation productivity growth curves (e.g., one grows exponentially and the other grows superexponentially) that is absent pre-automation. (I don't think it's enough that the productivities grow at different rates post-automation.) So I still think we can extract this from our utility function without knowing much about the future, although maybe there's a concrete model that would show that's wrong.
Not sure this is disagreement per se, but I think the surprising behavior of GDP in your model is almost entirely due to the shape of the utility function and doesn't have much to do with either (1) the distinction between existing vs new products or (2) automation. In other words, I think this is still basically Baumol, although I admit to a large extent I'm just arguing here about preferred conceptual framing rather than facts.
Consider modifying your models as follows (which I presume makes it more like a traditional Baumol model):
Using your same utility function u(x1,x2)=log(x1)+x2, production is fully devoted to Good 1 for all times t<0, and during that time GDP is growing at r1. Then at time t=0, it becomes worthwhile to start producing Good 2. For t>0, the productivity growth rate of Good 1 remains much higher than the that of Good 2 (r1≫r2), and indeed the number of units of Good 1 produced grows exponentially faster than Good 2:
x1(t)=e(r1−r2)t , x2(t)=er2t−1
Nonetheless, the marginal value of Good 1 plummets due to the log in the utility function. Specifically, the relative price of Good 1 falls exponentially in time, p1(t)/p2(t)=e−(r1−r2)t where pi(t):=dfdxi(t), as does Good 1's price-weighted fraction of production:
p1(t)x1(t)p1(t)x1(t)+p2(t)x2(t)=e−r2t
GDP growth falls from r1 and exponentially asymptotes to r2 for large t.
Two points on the model:
Ok, so then what are the take-aways for AI? By cleanly separating the utility-function effect from shocks to productivity, I think this is reason for us to believe that the past is a reasonable guide to the future. Yes, there could be weird kinks in our utility function, but in terms of revealing kinks there's not much reason to think that AI-induced productivity gains will be importantly different than productivity gains from the past.
What quantity should we measure if not GDP?
I think there's just no getting around the fact that the kind of growth we care about is unavoidably wrapped up in our utility function. But as long as some fraction of people want to build Jupiter brains and explore Andromeda enough that they don't devote ~all their efforts to goals that are intrinsically upper bounded, I expect AGI to lead to rapid real GDP growth (although it does likely eventually end with light-speed limits or whatever).
If growth were slow post-singularity, I think that would imply something pretty weird about human utility in this universe (or rather, the utility of the beings controlling the economy). There could of course still be crazy things happening like wild increases in energy usage at the same time, but this isn't too different than how wild the existence of nanometer-scale transistors are relative to pre-industrial civilization. If you care about those crazy things independent of GDP (which is a measure of how fast the world overall is getting what it wants), you should probably just measure them directly, e.g., energy usage, planets colonized, etc.
Nice post. The argument for simultaneity (most models train at the same time, and then are evaled at the same time, and then released at the same time) seems ultimately to be based on the assumption that the training cap grows in discrete amounts (say, a factor of 3) each year.
> * We want to prevent model training runs of size N+1 until alignment researchers have had time to study and build evals based on models of size N.
> * For-profit companies will probably all want to start going as soon as they can once the training cap is lifted.
> * So there will naturally be lots of simultaneous runs...
But why not smoothly raise the size limit? (Or approximate that by raising it in small steps every week or month?) The key feature of this proposal, as I see it, is that there is a fixed interval for training and eval, to prevent incentives to rush. But that doesn't require simultaneity.
I listed this example in my comment, it was incorrect by an order of magnitude, and it was a retrodiction. "I didn't look up the data on Google beforehand" does not make it a prediction.
I'm also a little surprised you think that modeling when we will have systems using similar compute as the human brain is very helpful for modeling when economic growth rates will change. (Like, for sure someone should be doing it, but I'm surprised you're concentrating on it much.) As you note, the history of automation is one of smooth adoption. And, as I think Eliezer said (roughly), there don't seem to be many cases where new tech was predicted based on when some low-level metric would exceed the analogous metric in a biological system. The key threshold for recursive feedback loops (*especially* compute-driven ones) is how well they perform on the relevant tasks, not all tasks. And the way in which machines perform tasks usually looks very different than how biological systems do it (bird vs. airplanes, etc.).
If you think that compute is the key bottleneck/driver, then I would expect you to be strongly interested in what the automation of the semiconductor industry would look like.
I like this post a lot but I will disobey Rapoport's rules and dive straight into criticism.
I think this is a poor way to describe a reasonable underlying point. Heavier-than-air flying machines were pursued for centuries, but airplanes appeared almost instantly (on a historic scale) after the development of engines with sufficient power density. Nonetheless, it would be confusing to say "flying is more about engine power than the right theories of flight". Both are required. Indeed, although the Wright brothers were enabled by the arrival of powerful engines, they beat out other would-be inventors (Ader, Maxim, and Langley) who emphasized engine power over flight theory. So a better version of your claim has to be something like "compute quantity drives algorithmic ability; if we independently vary compute (e.g., imagine an exogenous shock) then algorithms follow along", which (I think) is what you arguing further in the post.
But this also doesn't seem right. As you observe, algorithmic progress has been comparable to compute progress (both within and outside of AI). You list three "main explanations" for where algorithmic progress ultimately comes from and observe that only two of them explain the similar rates of progress in algorithms and compute. But both of these draw a causal path from compute to algorithms without considering the (to-me-very-natural) explanation that some third thing is driving them both at a similar rate. There are a lot of options for this third thing! Researcher-to-researcher communication timescales, the growth rate of the economy, the individual learning rate of humans, new tech adoption speed, etc. It's plausible to me that compute and algorithms are currently improving more or less as fast as they can, given their human intermediaries through one or all of these mechanisms.
The causal structure is key here, because the whole idea is to try and figure out when economic growth rates change, and the distinction I'm trying to draw becomes important exactly around the time that you are interested in: when the AI itself is substantially contributing to its own improvement. Because then those contributions could be flowing through at least three broad intermediaries: algorithms (the AI is writing its own code better), compute (the AI improves silicon lithography), or the wider economy (the AI creates useful products that generate money which can be poured into more compute and human researchers).
Humans have been automating mechanical task for many centuries, and information-processing tasks for many decades. Moore's law, the growth rate of the thing (compute) that you ague drives everything else, has been stated explicitly for almost 58 years (and presumably applicable for at least a few decade before that). Why are you drawing a distinction between all the information processing that happened in the past and "AI", which you seem to be taking as a basket of things that have mostly not had a chance to be applied yet (so no data)?
This list is missing the crucial parameters that would translate the others into what we agree is most notable: economic growth. I think needs to be discussed much more in section 4 for it to be a useful summary/invitation to the models you mention.
Thanks: https://ibkr.com/referral/charles6837
I agree it's important to keep the weaker fraud protection on debit cards in mind. However, for the use I mentioned above, you can just lock the debit card and only unlock it when you have a cash flow problem. (Btw, if you don't use your IB debit card, you should lock it even if you aren't using it.) Debit card liability is capped at $50 and $500 if you report fraudulent transactions within 2 days and 60 days, respectively.
That said, I have most of my net worth elsewhere, so I'm less worried about tail risks than you would reasonably be if you're mostly invested through IB.