Comments
How big a deal is the congressional commission? What is the historical track record of Congress implementing the commission's top recommendation?
With hindsight, this comment from Jan Kulveit looks prescient.
More from the author
Curated and popular this week
49
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...
Recent opportunities to take action
I spent all day crying about this. An arms race is about the least safe way to approach. And we contributed to this. Many important people read Leopold's report. He promoted it quite hard. But the background work predates Leopold's involvement.
We were totally careless and self aggrandizing. I hope other people don't pay for our sins.
This sounds very much like the missile gap/bomber gap narrative, and yeah this is quite bad news if they actually adopt the commitments pushed here.
The evidence that China is racing to AGI is quite frankly very little, and I see a very dangerous arms race that could come:
https://forum.effectivealtruism.org/posts/cXBznkfoPJAjacFoT/are-you-really-in-a-race-the-cautionary-tales-of-szilard-and
I feel that. This report saddens me and I think its recommendations are very bad.
I don't feel that I contributed to this. Perhaps I could have done more to prevent it, although it's not obvious to me what I could have done.
What makes you say this? I agree that it is likely that Aschenbrenner's report was influential here, but did we make Aschenbrenner write chapter IIId of Situational Awareness the way he did?
Is there some background EA/aligned work that argues for an arms race? Because the consensus seems to be against starting a great power war.
I think a non-trivial fraction of Aschenbrenner's influence as well as intellectual growth is due to us and the core EA/AI-Safety ideas, yeah. I doubt he would have written it if the extended community didn't exist, and if he wasn't mentored by Holden, etc.
I don't disagree with this at all. But does this mean that blame can be attributed to the entire EA community? I think not.
Re mentorship/funding: I doubt that his mentors were hoping that he would accelerate the chances of an arms race conflict. As a corollary, I am sure nukes wouldn't have been developed if the physics community in the 1930s didn't exist or mentored different people or adopted better ethical norms. Even if they did the latter, it is unclear if that would have prevented the creation of the bomb.
(I found your comments under Ben West's posts insightful; if true, it highlights a divergence between the beliefs of the broader EA community and certain influential EAs in DC and AI policy circles.)
Currently, it is just a report, and I hope it stays that way.
For as long as it's existed the "AI safety" movement has been trying to convince people that superintelligent AGI is imminent and immensely powerful. You can't act all shocked pikachu that some people would ignore the danger warnings and take that as a cue to build it before someone else does. This was all quite a predictable result of your actions.
I have no idea what you are advocating for here. I have no inherent interest in trying to convince people that AGI is likely powerful, but it does seem likely true. Should I lie to people?
Many have chosen the path of keeping their beliefs to themselves. My guess is that wasn't very helpful as the "imminent and powerful" part is kind of obvious as it starts happening.
What is the predictable result here? What is the counterfactual? How does anything better happen if you don't say anything, and why are you falsely claiming that it's been consensus that it's a good idea to publicly talk about the power and capabilities of AI systems? A substantial fraction of the AI safety movement did not do this, and indeed strongly advocated against (again, I think mistakenly), so even if you assign blame, you obviously can't assign blame uniformly.
Do you see advocating for export controls as fundamentally different from an arms race? Because it seems like export controls are pretty popular among AI policy people.
I honestly don't know. When I think of an arms race, I typically think of rapid manufacturing and accumulation of "weapons."
Do you think export controls between two countries are a sufficient condition for an arms race?
I do think they are a sufficient condition to kick off an arms race. Export controls are a declaration of hostility, and they force the two countries to decouple from each other. China and the US being decoupled makes the downside of an arms race much lower and the upside much higher.
It's hard to justify export controls unless you believe that we actually are in an arms race, sooner or later. If you wanted to prevent an arms race you couldn't pick a worse policy to put your weight behind. That leads me to conclude that AI policy people who back export controls find an arms race to be acceptable.
I don't think something as strong as this, but I did think at the time that the work on export controls was bad and likely to exacerbate arms race dynamics, and continue to believe this (and the celebration of export controls as a great success of the EA policy efforts was one of the things that caused me to update on future EA-driven AI policy efforts probably being net harmful, though FTX played a bigger role).