Got a q on LW about what I mean by AI reform, so sharing my reply here too:
Yeah, realize there's not as much description as there could be. Basically a new camp in the 3 sided AGI debate (safety, boosters, ethics). Here's a relevant excerpt:
There are already people who believe that the decisions made about this technology should not be left to those currently making them. Contrary to those who dismiss AI as a “stochastic parrot,” they take AI’s potential seriously. And contrary to classic AI safety, they recognize that there will be no purely technical “solutions” to the problems AI presents. Instead, they see governance as the essential lever. In this book, I’ll call them AI reformers.
Deep learning is incredibly powerful, but it’s being pointed at the wrong things by the wrong people for the wrong reasons. Indeed, its potential actually means it’s even more important that we change how it’s developed. The Obsoleting Project occasionally produces genuine breakthroughs—in medicine, in science—but these are the scraps that fall from the table, not the meal itself. This isn't an accident; it's just what markets reward.
Many people on the right and left would like to just press stop on AI. They hate what it’s done to the internet, they hate having it foisted upon them, and they hate the people making it. I get it. Really. (I miss being able to use em-dashes without drawing suspicion, and, sometimes, a sentence structure really isn’t just this, it’s also that!) But this understandable revulsion can blind us to the staggering potential of a reformed AI, where the immense power of deep learning is applied not to produce slop and madness, but to enable scientific discovery and unambiguously better lives—more AlphaFold and less chatbot psychosis.
Crucially, reformers recognize that AI's present harms and its potential catastrophes aren't separate problems requiring separate approaches—they're symptoms of the same underlying dynamics: competitive pressure, concentrated power, and a staggering lack of accountability. The chatbot that encourages a teenager's suicide and the hypothetical superintelligence that slips human control both emerge from organizations racing to deploy systems they don't really understand. Getting serious about one means getting serious about the other. Reformers recognize that, if it were aimed in different directions, the technology could be so much better than it is. They realize that the people advancing the Obsoleting Project—not the various people looking to change or stop that project—are the ones with the real power, at least for now. They believe that a better world is possible, and so is better AI. In short, AI reform works toward the version of AI that the Obsoleting Project sells, but doesn’t actually make.
Thanks for writing this, but apparently the waiver is not totally effective (I have this on good authority, but can't really say more right now). See this paragraph from the NYT article: "The waiver, announced by Secretary of State Marco Rubio, seemed to allow for the distribution of H.I.V. medications, but whether the waiver extended to preventive drugs or other services offered by the program, the President’s Emergency Plan for AIDS Relief, was not immediately clear."
Bengio and Hinton are the two most-cited researchers alive. Ilya Sutskever is the 3rd most cited AI researcher, and though he's not on that paper, the superalignment intro blog post from OpenAI says this, "Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue." LeCun is probably the top AI researcher who's not worried about controlling a superintelligence (4th in total citations after Sutskever).
This is obviously a semantics disagreement, but I stand by the original claim. Note that I'm not saying that all the top AI researchers are worried about x-risk.
In regards to your overall point, it does not rebuts the idea that some people have been cynically exploiting AI fears for their own gain. I mean, remember that OpenAI was founded as an AI safety organisation. The actions of Sam Altman seem entirely consistent with someone hyping X-risk in order to get funding and support for OpenAI, then pivoting to downplaying risk as soon as ditching safety gets more profit. I doubt this applies to all people or even the majority, but it does seem like it's happened at least once.
I largely agree with this and alluded to this possibility here:
If AI companies ever needed to rely on doomsday fears to lure investors and engineers, they definitely don’t anymore.
I might write a separate piece on the best evidence for the hype argument, which OpenAI I think has been the biggest winner of. My guess is that Altman actually did believe what he was saying about AI risk back in 2015. Superintelligence came out the year before, and it's not a surprising view for him to have given what else we know about him.
I'd also guess that Altman and Elon are two of the people most associated with the x-risk story, which has been the biggest driver of skepticism about it.
There's also been more recent evidence of him ditching x-risk fears now that it seems convenient. From a recent Fox News interview:
Interviewer: “A lot of people who don’t understand AI, and I would put myself in that category, have got a basic understanding, but they worry about AI becoming sentient, about it making autonomous decisions, about it telling humans you’re no longer in charge?”
Altman: “It doesn’t seem to me to be where things are heading…is it conscious or not will not be the right question, it will be how complex of a task can it do on its own?”
Interviewer: “What about when the tool gets smarter than we are? Or the tool decides to take over?”
Altman: “I think tools in many senses are already smarter than we are. I think that the internet is smarter than you or I, the internet knows a lot of things. In fact, society itself is vastly smarter and more capable than any one person. I think we’re already good at working with tools, institutions, structures, whatever you want to call it, that are vastly more capable than one person and as long as we have a reasonably level playing field where one person or one company has vastly more power than anybody else, I think we know how to deal with that.”
1. The mafia tendencies (careerist groups working together out of self-interest and not to advance science itself) are present in the West as well these days. In fact the term was first used in this way by Italian academics.
2. They're not against big breakthroughs in PRC, esp. obvious ones. The bureaucracy bases promotions, raises, etc. on metrics like publications in top journals, cititations, ... However there are very obvious wins that they will go after in a coordinated way - including AI, semiconductors, new energy tech, etc.
3. I could be described as a China hawk in that I've been pointing to a US-China competition as unavoidable for over a decade. But I think I have more realistic views about what is happening in PRC than most China hawks. I also try to focus on simple descriptive analysis rather than getting distracted by normative midwit stuff.
4. There is coordinated planning btw govt and industry in PRC to stay at the frontier in AI/AGI/ASI. They are less susceptible to "visionaries" (ie grifters) so you'll find fewer doomers or singularitarians, etc. Certainly not in the top govt positions. The quiet confidence I mentioned extends to AI, not just semiconductors and other key technologies."
Update: E-book purchased thru OR is PDF + epub (Kindle compatible)!
Yes, and I believe if you order through my publisher, you'll get it in May.
Got a q on LW about what I mean by AI reform, so sharing my reply here too:
Yeah, realize there's not as much description as there could be. Basically a new camp in the 3 sided AGI debate (safety, boosters, ethics). Here's a relevant excerpt:
There are already people who believe that the decisions made about this technology should not be left to those currently making them. Contrary to those who dismiss AI as a “stochastic parrot,” they take AI’s potential seriously. And contrary to classic AI safety, they recognize that there will be no purely technical “solutions” to the problems AI presents. Instead, they see governance as the essential lever. In this book, I’ll call them AI reformers.
Deep learning is incredibly powerful, but it’s being pointed at the wrong things by the wrong people for the wrong reasons. Indeed, its potential actually means it’s even more important that we change how it’s developed. The Obsoleting Project occasionally produces genuine breakthroughs—in medicine, in science—but these are the scraps that fall from the table, not the meal itself. This isn't an accident; it's just what markets reward.
Many people on the right and left would like to just press stop on AI. They hate what it’s done to the internet, they hate having it foisted upon them, and they hate the people making it. I get it. Really. (I miss being able to use em-dashes without drawing suspicion, and, sometimes, a sentence structure really isn’t just this, it’s also that!) But this understandable revulsion can blind us to the staggering potential of a reformed AI, where the immense power of deep learning is applied not to produce slop and madness, but to enable scientific discovery and unambiguously better lives—more AlphaFold and less chatbot psychosis.
Crucially, reformers recognize that AI's present harms and its potential catastrophes aren't separate problems requiring separate approaches—they're symptoms of the same underlying dynamics: competitive pressure, concentrated power, and a staggering lack of accountability. The chatbot that encourages a teenager's suicide and the hypothetical superintelligence that slips human control both emerge from organizations racing to deploy systems they don't really understand. Getting serious about one means getting serious about the other. Reformers recognize that, if it were aimed in different directions, the technology could be so much better than it is. They realize that the people advancing the Obsoleting Project—not the various people looking to change or stop that project—are the ones with the real power, at least for now. They believe that a better world is possible, and so is better AI. In short, AI reform works toward the version of AI that the Obsoleting Project sells, but doesn’t actually make.
Yeah I think this is a more significant walkback, and discussed it here: https://x.com/GarrisonLovely/status/1926095320997368319?t=vfuPigtomkOn5qc9Z8jCmQ&s=19
Good find.
Yeah, I asked my editor at TIME adding an update. Will edit this piece as well.
A deep research response that doesn't discuss trump 2 at all is not very useful and could even mislead someone not currently paying attention.
Thanks for writing this, but apparently the waiver is not totally effective (I have this on good authority, but can't really say more right now). See this paragraph from the NYT article: "The waiver, announced by Secretary of State Marco Rubio, seemed to allow for the distribution of H.I.V. medications, but whether the waiver extended to preventive drugs or other services offered by the program, the President’s Emergency Plan for AIDS Relief, was not immediately clear."
Thanks Sarah!
Bengio and Hinton are the two most-cited researchers alive. Ilya Sutskever is the 3rd most cited AI researcher, and though he's not on that paper, the superalignment intro blog post from OpenAI says this, "Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue." LeCun is probably the top AI researcher who's not worried about controlling a superintelligence (4th in total citations after Sutskever).
This is obviously a semantics disagreement, but I stand by the original claim. Note that I'm not saying that all the top AI researchers are worried about x-risk.
I largely agree with this and alluded to this possibility here:
I might write a separate piece on the best evidence for the hype argument, which OpenAI I think has been the biggest winner of. My guess is that Altman actually did believe what he was saying about AI risk back in 2015. Superintelligence came out the year before, and it's not a surprising view for him to have given what else we know about him.
I'd also guess that Altman and Elon are two of the people most associated with the x-risk story, which has been the biggest driver of skepticism about it.
There's also been more recent evidence of him ditching x-risk fears now that it seems convenient. From a recent Fox News interview:
Hsu clarified his position on my thread here:
"Clarifications:
1. The mafia tendencies (careerist groups working together out of self-interest and not to advance science itself) are present in the West as well these days. In fact the term was first used in this way by Italian academics.
2. They're not against big breakthroughs in PRC, esp. obvious ones. The bureaucracy bases promotions, raises, etc. on metrics like publications in top journals, cititations, ... However there are very obvious wins that they will go after in a coordinated way - including AI, semiconductors, new energy tech, etc.
3. I could be described as a China hawk in that I've been pointing to a US-China competition as unavoidable for over a decade. But I think I have more realistic views about what is happening in PRC than most China hawks. I also try to focus on simple descriptive analysis rather than getting distracted by normative midwit stuff.
4. There is coordinated planning btw govt and industry in PRC to stay at the frontier in AI/AGI/ASI. They are less susceptible to "visionaries" (ie grifters) so you'll find fewer doomers or singularitarians, etc. Certainly not in the top govt positions. The quiet confidence I mentioned extends to AI, not just semiconductors and other key technologies."