Erich_Grunewald 🔸
Bio
Anything I write here is written purely on my own behalf, and does not represent my employer's views (unless otherwise noted).
Posts 20
Comments301
On timelines, I think it's worth separating out export controls on different items:
- Controls on AI chips themselves start having effects on AI systems within a year or so probably (say 6-12 months to procure and install the chips, and 6-18 months to develop/train/post-train a model with them), or even sooner for deployment/inference, i.e. 1-2 years or so.
- Controls on semiconductor manufacturing equipment (SME) take longer to have an impact as you say, but I think not that long. SMIC (and therefore future Ascend GPUs) is clearly limited by the 2019 ban on EUV photolithography, and I would say this was apparent as early as 2023. So I think SME controls instituted now would start having an effect on chip production in the late 2020s already, and on AI systems 1-2 years after that.
Most other relevant products (e.g., HBM and EDA software) probably fall between those two in terms of how quickly controls affect downstream AI systems.
So that means policy changes in 2025 could start affecting Chinese AI models in 2027 (for chips) and around 2030 (for SME) already, which seems relevant to even short-timeline worlds. For example, Daniel Kokotajlo's median for superhuman coders is now 2029, and IIUC Eli Lifland's median is in the (early?) 2030s.
But I would go further to say that export controls now can substantially affect compute access well into the 2030s or even the 2040s. You write that
the technical barriers [to Chinese indigenization of leading-edge chip fabrication] are higher today, but not so high that intense Chinese investment can't dent it over the course of a decade. SMEE is investing in laser-induced discharge plasma tech, with rumored trial production as soon as the end of this year. SMIC is using DUV more efficiently for (lower-yield, but still effective) chip production. There's also work on Nanoimprint lithography, immersion lithography, packaging, etc. And that won't affect market shares, until it does.
I won't have time to go into great detail here, but I have researched this a fair amount and I think you are too bullish on Chinese leading-edge chip fabrication. To be clear, China can and will certainly produce AI chips, and these are decent AI chips. But they will likely produce those chips less cost-efficiently and at lower volumes due to having worse equipment, and they will have worse performance than TSMC-fabbed chips due to using older-generation processes. The lack of EUV machines, which will likely last at least another five years and plausibly well into the 2030s, alone is a very significant constraint.
On SMEE and SMIC in particular -- you write:
SMEE is investing in laser-induced discharge plasma tech, with rumored trial production as soon as the end of this year.
SMEE was established 23 years ago to produce indigenous lithography, and 23 years later it still has essentially no market share, and it still has not produced an immersion DUV machine, let alone an EUV machine, which is far more difficult. I would not be surprised if, when the indigenous Chinese immersion DUV machine does finally arrive, it is a SiCarrier (or subsidiary) product and not an SMEE product.
SMIC is using DUV more efficiently for (lower-yield, but still effective) chip production.
In what sense do you mean SMIC is using DUV more efficiently? It is using immersion DUV multi-patterning (with ASML machines) to compensate for its lack of EUV machines. But as you note this means worse yield and lower throughput. I don't see any sense in which SMIC is using DUV more efficiently; it's just using it more, in order to get around a constraint that TSMC doesn't have. In any case, multi-patterning with immersion DUV can only take you so far; there's likely a hard stop around what's vaguely called 2 nm or 1.4 nm process nodes, even if you do multi-patterning perfectly. (For reference, TSMC is starting mass production on its "2 nm" process this year.)
On the oil analogy, it seems from
The long-term winners were definitely not the groups that extracted or refined the oil, even though they made lots of money - it was the countries that consumed the oil and built industrial capacity leading up to WWII, and could then use the controlled supply of oil. ... And as far as I can tell, no-one is restricting Chinese companies from using compute right now - they don't own it, but can use the same LLMs I do.
that you think ownership of compute does not substantially influence who will have or control the most powerful AI systems? I disagree; I think it will impact both AI developers and also companies relying on access to AI models. First, AI developers -- export controls put the Chinese AI industry as a whole at a compute disadvantage, which we see in the fact that they train less compute-intensive models, for a few reasons:
- It is generally unappealing for major AI developers to merely rent GPUs they don't own, as a result of which they often build their own data centers (xAI, Google) or rely on partnerships for exclusive access (OpenAI, Anthropic). I think the main reasons for this are cost, (un)certainty, and greater control over the cluster set-up.
- Chinese companies cannot build their own data centers with export-controlled chips without smuggling, and cannot embark on these partnerships with American hyperscalers. If they want to use cutting-edge GPUs, they must either rely on smuggling (which means higher prices and smaller quantities), or renting from foreign cloud providers.
- The US likely could, if and when it wanted to, deny access of compute via the cloud to Chinese customers, at least large-scale use and at least for the large hyperscalers. So for Chinese AI developers to rely on foreign cloud compute gives the US a lot of leverage. (There are some questions around how feasible it is to circumvent KYC checks, and especially whether the US can effectively ensure these checks are done well in third countries, but I think the US could deny China most of the world's rentable cloud compute in this way.)
- Chinese privacy law makes it harder for Chinese AI developers to use foreign cloud compute, at least for some use cases. I'm not sure exactly how strong this effect is, but it seems non-negligible.
- For deployment/inference, you may want to have your compute located close to your users, as that reduces latency.
- In the event of an actual conflict over or involving AI, you can seize compute located on territory you control. I hope that doesn't happen obviously, but it's definitely a reason why as an AI developer you'd prefer to use compute located in your own country, than located in a rival country or one of the rival's allies or partners.
That's AI developers. As for the AI industry more broadly, there are barriers for Chinese companies wanting to use US models like ChatGPT or Claude, which, for example, is likely one reason why Manus moved to Singapore. So the current disparity in who owns compute and where it is located means Chinese AI developers are relatively compute-poor, and since Chinese companies rely substantially on domestic Chinese models, it seems to me like the entire Chinese AI industry is impacted by these restrictions.
Also, I disagree that oil "only mattered because it enabled economic development". In WWII especially, oil was necessary for fuel-hungry militaries to function. I think AI will also be militarily important even ignoring its effects on economic development, though maybe less so than oil.
On the other hand, I think you're wrong in saying that "the chip supply chain has unique characteristics [compared to oil,] with extreme manufacturing concentration, decades-long development cycles, and tacit knowledge that make it different" - because the same is true for crude oil extraction! What matters is who refines it, and who buys it, and what it's used for.
I think the technical barriers to developing EUV photolithography from scratch are far higher than anything needed to extract, refine, or transport oil. I also think the market concentration is far higher in the AI chip design and semiconductor industries. There's no oil equivalent to TSMC's ~90% leading-edge logic chip, NVIDIA's ~90% data center GPU, or ASML's 100% EUVL machine market shares.
Second, if we're talking about takeoff after 2035, the investments in China are going to swamp western production. (This is the command economy advantage - though I could imagine it's vulnerable to the typical failure modes where they overinvest in the wrong thing, and can't change course quickly.)
Are you sure? I would guess that the chip supply chain used by NVIDIA has more investment than the Chinese counterpart. For example, according to a SEMI report, China will spend $38bn on semiconductor manufacturing equipment in 2025, whereas the US + Taiwan + South Korea + Japan is set to spend a combined ~$70bn. I would guess it looks directionally similar for R&D investment, though the difference may be smaller there.
For moderately short, 2-6 year timelines, the timelines for chip fabs are long enough that we're mostly locked in not just to overall western dominance via chips produced in Taiwan, but because fabrication plans built today are coming online closer to 2029, and the rush to build Chinese fabrication plants is already baked in. And that's just the fabs - for the top chips, the actual chip design usually takes as long or longer than building the plant.
I was under the impression the AI chip design process is more like 1.5-2 years, and a fab is built in 2-3 years in Taiwan or 4 years for the Arizona fab. It sounds like you think differently? Whatever it is, I would guess it's roughly similar across the industry, including in China. That seems like, if my numbers are right, it leaves enough room for policy now to influence the relative compute distribution of nations 5-6 years from now.
Interesting!
While the small body size of sardines and anchovies means that many individuals must be killed to produce a given amount of food, thereby scaling up the moral weight, a meaningful moral cost calculation should extend beyond these direct first-order consequences to account for indirect higher-order consequences, especially given that all food production invariably involves some level of collateral damage, as will be discussed further on.
On the other hand, sardines really are very small, and I reckon you'd need on the order of 100x as many sardines as you'd need salmons to get the same amount of calories. I wonder how many small animals would die to produce the amount of calories of plant-based food you'd get from a sardine? I'd guess <<0.1, but I'd be interested in seeing estimates here as it seems pretty cruxy.
As for traitor, I think the only group here that can be betrayed is humanity as a whole, so as long as one believes they're doing something good for humanity I don't think it'd ever apply.
Hmm, that seems off to me? Unless you mean "severe disloyalty to some group isn't Ultimately Bad, even though it can be instrumentally bad". But to me it seems useful to have a concept of group betrayal, and to consider doing so to be generally bad, since I think group loyalty is often a useful norm that's good for humanity as a whole.
Specifically, I think group-specific trust networks are instrumentally useful for cooperating to increase human welfare. For example, scientific research can't be carried out effectively without some amount of trust among researchers, and between researchers and the public, etc. And you need some boundary for these groups that's much smaller than all humanity to enable repeated interaction, mutual monitoring, and norm enforcement. When someone is severely disloyal to one of those groups they belong to, they undermine the mutual trust that enables future cooperation, which I'd guess is ultimately often bad for the world, since humanity as a whole depends for its welfare on countless such specialised (and overlapping) communities cooperating internally.
I'm obviously not Matthew, but the OED defines them like so:
- sell-out: "a betrayal of one's principles for reasons of expedience"
- traitor: "a person who betrays [be gravely disloyal to] someone or something, such as a friend, cause, or principle"
Unless he is lying about what he believes -- which seems unlikely -- Matthew is not a sell-out, because according to him Mechanize is good or at minimum not bad for the world on his worldview. Hence, he is not betraying his own principles.
As for being a traitor, I guess the first question is, traitor of what? To EA principles? To the AI safety cause? To the EA or AI safety community? In order:
- I don't think Matthew is gravely disloyal to EA principles, as he explicitly says he endorses them and has explained how his decisions make sense on his worldview
- I don't think Matthew is gravely disloyal to the AI safety cause, as he's been openly critical of many common AI doom arguments for some time, and you can't be disloyal to a cause you never really bought into in the first place
- Whether Matthew is gravely disloyal to the EA or AI safety communities feels less obvious to me. I'm guessing a bunch of people saw Epoch as an an AI safety organisation, and by extension its employees as members of the AI safety community, even if the org and its employees did not necessarily see itself or themselves that way, and felt betrayed for that reason. But it still feels off to me to call Matthew a traitor to the EA or AI safety communities, especially given that he's been critical of common AI doom arguments. This feels more like a difference over empirical beliefs than a difference over fundamental values, and it seems wrong to me to call someone gravely disloyal to a community for drawing unorthodox but reasonable empirical conclusions and acting on those, while broadly having similar values. Like, I think people should be allowed to draw conclusions (or even change their minds) based on evidence -- and act on those conclusions -- without it being betrayal, assuming they broadly share the core EA values, and assuming they're being thoughtful about it.
(Of course, it's still possible that Mechanize is a net-negative for the world, even if Matthew personally is not a sell-out or a traitor or any other such thing.)
This is weird because other sources do point towards a productivity gap. For example, this report concludes that "European productivity has experienced a marked deceleration since the 1970s, with the productivity gap between the Euro area and the United States widening significantly since 1995, a trend further intensified by the COVID-19 pandemic".
Specifically, it looks as if, since 1995, the GDP per capita gap between the US and the eurozone has remained very similar, but this is due to a widening productivity gap being cancelled out by a shrinking employment rate gap:
This report from Banque de France has it that "the EU-US gap has narrowed in terms of hours worked per capita but has widened in terms of GDP per hours worked", and that in France at least this can be attributed to "producers and heavy users of IT technologies":
The Draghi report says 72% of the EU-US GDP per capita gap is due to productivity, and only 28% is due to labour hours:
Part of the discrepancy may be that the OWID data only goes until 2019, whereas some of these other sources report that the gap has widened significantly since COVID? But that doesn't seem to be the case in the first plot above (it still shows a widening gap before COVID).
Or maybe most of the difference is due to comparing the US to France/Germany, versus also including countries like Greece and Italy that have seen much slower productivity growth. But that doesn't explain the France data above (it still shows a gap between France and the US, even before COVID).
He reframes EA concepts in a more accessible way, such as replacing “counterfactuals” with the sports acronym “VORP” (Value Over Replacement Player).
And here I was thinking hardly a soul read my suggesting this framing ...
Nice work!
On AI chip smuggling, rather than the report you listed, which is rather outdated now, I recommend reading Countering AI Chip Smuggling Has Become a National Security Priority, which is essentially a Pareto improvement over the older one.
I also think Chris Miller's How US Export Controls Have (and Haven't) Curbed Chinese AI provides a good overview of the AI chip export controls, and it is still quite up-to-date.