Comments
50
Curated and popular this week
This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...
I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences.
Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts:
* Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do
* NEST is an opinionated ideas-fi...
Recent opportunities to take action
Is the recent partial lifting of US chip export controls on China (see e.g. here: https://thezvi.substack.com/p/selling-h200s-to-china-is-unwise) good or bad for humanity? I’ve seen many takes from people whose judgment I respect arguing that it is very bad, but their arguments, imho, just don’t make sense. What am I missing?
For transparency, I am neither Chinese nor American, nor am I a paid agent of them. I am not at all confident in this take, but imho someone should make it.
I see two possible scenarios: A) you are not sure how close humanity is to developing superintelligence in the Yudkowskian sense. This is what I believe, and what many smart opponents of the Trump administration’s move to ease chip controls believe. Or B) you are pretty sure that humanity is not going to develop superintelligence any time soon, let’s say in the next century. I admit that the case against the lifting of chip controls is stronger under B), though I am ultimately inclined to reject it in both scenarios.
Why is easing of chip controls, imho, a good idea if the timeline to superintelligence might be short?
If superintelligence is around the corner, here is what should be done: an immediate international pause of AI development until we figure out how to proceed.
Competitive pressures and resulting prisoner’s dilemmas have been identified as the factor that might push us toward NOT pausing even when it would be widely recognized that the likely outcome of continuing is dire.
There are various relevant forms of competition, but plausibly the most important is that between the US and China. In order to reduce competitive dynamics and thus prepare the ground for a cooperative pause, it is important to build trust between the parties and beware of steps that are hostile, especially in domains touching AI.
Controls make sense only if you are very confident that superintelligence developed in the US, or perhaps in liberal democracy more generally, is going to turn out well for humanity, and you are not so confident about superintelligence developed in China.
I do think that the US political system is, in an important sense, whose exposition would require more than an oversized quick take, better than the Chinese political system, but despite this, I am not at all confident that superintelligence developed in a liberal democracy would be good for humanity; frankly, I find such beliefs wholly unfounded.
Why is easing of chip controls a good idea if the timeline to superintelligence is long?
Disclaimer: I know following argument is perhaps too abstract and I am even less confident in it than in previous section, nevertheless.
It is a feature of the current international order that there is a strong taboo on military action of one country against another outside narrow and contested exceptions. I think this taboo is very good and very important. And yes, I know that this taboo is sometimes broken; that does not invalidate previous two sentences.
Another feature of the current international order is that some countries are what is sometimes called pariah states; I am vaguely aware that the word “pariah” has unpleasant connotations in the Indian cultural context, but I am going to use the term anyway, since I don’t have a suitable replacement, sorry. Anyway, good examples are, e.g., Russia or Afghanistan.
Being a pariah state means essentially that some hostile actions which are considered beyond the pale against a non-pariah state are considered ok against them. Notably, the taboo against direct military action is so strong that it mostly applies even to attacks against pariah states, but certain types of heavy restrictions on trade and cross-border investment are what essentially define pariah status. Another sometimes used action against a pariah state is external support, including by providing arms, to an internal opposition, but that is facultative. Many pariah states don’t have internal opposition capable of or interested in armed resistance.
In trade, pariah status tended to be a package deal. If a country is a pariah, it usually had heavy restrictions placed on its trade and investment; if it is a normal member of the international community, it could expect that it would not have certain kinds of restrictions placed on it.
Of course, the previous paragraph has to be immediately qualified in that there are many, many types of restrictions on international trade considered completely acceptable even among normal members of the international community, such as tariffs, bans on investment in critical infrastructure, or bans on selling some military or military-adjacent technologies. But there are other kinds of restraints that were usually deployed only against pariahs.
And pariah status is not a category enshrined in international law; really, making and unmaking a pariah is based on a decision of the government of the United States, since the US is the current hegemon of the international order. E.g., Cuba is, imho, a pariah state mainly because of random historical contingency plus quirks of the US electoral system. BUT breaking a taboo against militarily attacking another country is the sort of thing that tends to get you on the pariah list.
This all means that avoiding a pariah designation is a strong incentive for countries NOT to engage in military aggression.
Now, I do think that the chip controls imposed on China by the Biden administration are breaking this logic. These controls are restrictions on selling a critical component of consumer technology which is related to military stuff only in the sense that, like, steel or oil is related to military stuff. And restrictions on selling those is something which has not usually been done against non-pariah states; the Biden administration, however, had no plan to impose a whole package of pariah status on China. They made an awkward exception in an important area, which devalues the status of being non-pariah member of international community.
Thus, the imposition of chip controls reduces the incentive for China, and indirectly for everyone else, to avoid pariah status. Which means that it reduces their incentive to avoid military aggression. I think that that is very bad and it outweighs benefits of those controls even in no-near-term-superintelligence-threat world.
Hi - thanks for this comment. As someone working on export control policy, let me give you my perspective.
Firstly, an important precondition for a cooperative pause is leverage. You don't get China to agree to a mutual pause by first giving away your main strategic advantage. You get them to agree by making the alternative to be "a race they're losing", which is worse than cooperation. Export controls are thus part of what creates the conditions for being able to pause. If you equalize compute access first, China has no reason to agree to a pause because they'd be in a great position to race.
This is basic negotiation theory. You don't just disarm and hand over your weapons before the arms control treaty; you disarm as part of the treaty.
More critically, export controls are already priced in. The US has maintained semiconductor restrictions on China since October 2022, tightening them in 2023 and again in 2024, before loosening them late last year. The core diplomatic costs of these controls have already been paid. Easing controls now doesn't recoup that trust. China won't say "oh great, all is forgiven". But easing controls does give away the strategic advantage those controls purchased.
I'd also dispute that placing chip export controls on a state makes them a "pariah state". Restricting dual-use technology exports to strategic competitors is completely normal behavior among non-pariah states. Similarly to how we might restrict F-35 technology, nuclear technology, satellite components, rocket launch components, etc., to many countries.
Hi, thx for engaging, but I guess we’ll have to agree to disagree.
I do agree that chip controls would make a lot of sense as a source of leverage in negotiations over an AI pause in a world where China would be trying to build superintelligence and the US would be trying to force them to do a mutual pause. But we don’t live in that world.
The Biden administration had no intention to agree to a cooperative pause when it imposed chip controls, and the Trump administration has somehow even less than no intention to pause; Trump seems to be going all-in on AI development.
You might object that controls are good because they are sort of increasing the supply of leverage which in the future could be used to do a pause, and that is true, BUT to do a deal, both leverage and trust are needed. And imho even without controls, the US has a lot of leverage over China just by the sheer power of its global hegemony.
So, leverage over China is already relatively abundant, so to speak, but Chinese trust is what is relatively scarce (as is, of course, American trust in Chinese intentions). So, I am skeptical that steps which build more leverage over China but decrease Chinese trust in American intentions get these countries closer to cooperation.
And I disagree that lifting controls doesn’t recoup Chinese trust. It does not recoup it fully, as if controls never happened, but I do think that it builds trust relative to a situation of continuing controls.
To use a somewhat strained example with which most are surely familiar, when Trump put up his Liberation Day tariffs, prices of various financial assets fell off a cliff, but when he partially reversed the tariffs, prices partially recouped. How come, if Trump has shown that he is untrustworthy and might change his mind any minute? Well, many investors are evidently betting he won’t do that, and so far they’ve been correct.
Imho it is a fairly general principle of human relationships that for maintaining trust: never doing a trust-damaging action > rescinding a trust-damaging action > continuing a trust-damaging action.
Also, Chinese-American relations are themselves a vivid example that trust can be rebuilt: in the 50s, they were at rock bottom due to various trust-destroying actions, notably in the Korean War, where the Chinese army attacked UN forces advancing toward the Yalu River and pushed them back from what is now unfortunately North Korea. But, skipping ahead, by 2000, after a lot of trust building, the US granted China Permanent Normal Trade Relations. Things went downhill after that, but that could change again.
I also disagree that chips used to train/run LLMs are in the same category as F-35s or nuclear material; imho they are more like steel or oil, commodities obviously important for military purposes but with extremely broad civilian applications, and actions like an oil embargo are considered to be hostile acts, usually only deployed against pariah states, unlike an “F-35 embargo.”
Finally, imho you misunderstood what point I was trying to make (ok, that’s on me, I should’ve been clearer) about pariah states – it is not that chip controls make China into a pariah state. They don’t. Rather, by deploying measures usually only used against pariah states, these controls blur differences between “normal” members of an international community and pariah states, thus devaluing the usefulness of not doing things that put the country on the naughty list, primarily for China but secondarily for everyone else.
Since a thing that puts countries on the naughty list is waging an aggressive war, I do think this development is bad.
I think if your sole objective is to enact a bilateral pause, then easing export controls may be the best option, or maybe not. It's pretty unclear to me how that shakes out, I could definitely also see unilateral concessions as being quite detrimental (for reasons similar to those Peter mention in the other comment).
But I would guess most of the people you are responding to think enacting a cooperative pause is some combination of very unlikely and/or undesirable, and also that export controls help a lot in the absence of such an agreement. The main way export controls help for other plans are by giving the US more slack to (one can hope) spend on safety, and/or because superintelligence developed in the US would imo likely be safer (cf. this comment), and/or because imo US values are better, and this would likely be reflected in the AIs (cf. Claude versus Grok).
Thx for your perspective; I should be upfront that my confidence in my own case is far from ironclad. Anyway.
From the ryan_greenblatt’s article you’ve linked, I think safety plan A, based on achieving international agreement, should be tried before going for plan B, based on achieving and using a secure US lead, while in “plans” C and D it doesn’t matter whether the leading company is American or Chinese, so slowing down Chinese development is useless for those “plans” (they are in fact more like scenarios than real plans).
I do agree that the US political system is in an important sense better than the Chinese political system, but my prior is that if superintelligence is developed before, let’s say, 2050 (and that is very optimistic), it is likely to go badly completely regardless of which country it’ll come from.
I take your point about slack being potentially useful. Theoretically, I can imagine the following sequence of events: a) the US AI industry crushes Chinese competition, then b) the US government, feeling secure in the US lead, imposes sensible safety regulation on companies. If it were smart about it, it would at the same time propose an international regulatory framework that the rest of the world would be prepared to sign on to, as an alternative to unrestrained US domination. In effect, this would be tantamount to getting so much leverage over China that they would drop out of the race, and then hoping that the US government would use its advantage to push for safety, instead of using it in some other way.
However, imho this is a plan that should be pursued as a first option only in circumstances where you are really confident it’ll work, since the consequences of trying and failing are likely to be dire, as per my original post. And I am not confident it will work.
A much better order of operations would be to 1) try to negotiate with China to establish an international regulatory framework (plan A), with export control and other stuff being imposed as something that is explicitly linked to China not agreeing to that framework, in the same way sanctions on Russia are imposed explicitly because its aggression against Ukraine, and 2) only if they refuse, try to crush them (plan B).
When political will in the US to try for plan A is lacking, I think waiting until circumstances make that plan realistic while preparing the groundwork for it is a better strategy than going straight ahead for plan B.
Maybe if you are President of the United States you can first try the one thing, and then the other. But from the perspective of an individual, you have to assume there's some probability of each of these plans (and other strategies) being executed, and that everything will be really messy (e.g., different actors having different strategies in mind, even within the US). Softening export controls seems like something you could do as part of executing Plan A, but as I mentioned above, it's very unclear to me whether unilaterally doing so makes Plan A more likely to be the chosen strategy, and it does likely make Plan B and Plan C go worse.
I think you're thinking people have more control over which strategy is adopted than I think they do? Or, what circumstances do you have in mind? Because waiting seems pretty costly.
But I think maybe the cruxiest bits are (a) I think export controls seem great in Plan B/C worlds, which seem much likelier than Plan A worlds, and (b) I think unilaterally easing export controls is unlikely to substantially affect the likelihood of Plan A happening (all else equal). It seems like you disagree with both, or at least with (b)?
"But I think maybe the cruxiest bits are (a) I think export controls seem great in Plan B/C worlds, which seem much likelier than Plan A worlds, and (b) I think unilaterally easing export controls is unlikely to substantially affect the likelihood of Plan A happening (all else equal). It seems like you disagree with both, or at least with (b)?"
Yep, this is pretty close to my views. I do disagree with (b), since I am afraid that controls might poison the well for future Plan A negotiations. As for (a), I don’t get how controls help with Plan C, and I don’t think Plan B/C worlds are much more likely than Plan A, as of now. But I do agree with you that controls help with Plan B, so if you see it as the main hope, I understand why you are supporting them.
(btw. how does one do a proper blockqoute here?)