Thanks for the thoughts!

I do think the second one has more potential impact if it works out, but I also worry that it's too "out there" speculative and also dependent on the AGI being persuaded by an argument (which they could just reject), rather than something that more concretely ensures alignment. I also noticed that almost no one is working on the Game Theory angle, so maybe it's neglected, or maybe the smart people all agree it's not going to work.

The first project is probably more concrete and actually uses my prior skills as an AI/ML practitioner, but also, there's a lot of people already working on Mech Int stuff. In comparison, my knowledge of Game Theory is self-taught and not very rigorous.

I'm tempted to explore both to an extent. The first one I can probably do some exploratory experiments to test the basic idea, and rule it out quickly if it doesn't work.

I more or less agree. It's not really a complaint from me. I probably was too provocative in my choice of wording earlier.

I want to clarify that I don't think ideas like the Orthogonality Thesis or Instrumental Convergence are wrong. They're strong predictive hypotheses that follow logically from very reasonable assumptions, and even the possibility that they could be correct is more than enough justification for AI safety work to be critical.

I was more just pointing out some examples of ideas that are very strongly held by the community, that happen to have been named and popularized by people like Bostrom and Yudkowsky, both of whom might be considered elites among us.

P.S. I'm always a bit surprised that the Neel Nanda of Google DeepMind has the time and desire to post so much on the EA Forums (and also Less Wrong). That probably says very good things about us, and also gives me some more hope that the folks at Google are actually serious about alignment. I really like your work, so it's an honour to be able to engage with you here (hope I'm not fanboying too much).

I mean, from the inside it would look like what you said, good ideas that are persuasive. And I'm not saying they aren't good ideas that are persuasive (I agree they are). I'm more just pointing out some examples of ideas that form core elements of our belief ecosystem, that have their source in the works of particular elites, in this case, being named and popularized by Bostrom's book Superintelligence, and then further popularized by Yudkowsky in the Sequences. In the sense that this is elitism, it's very mild, and I don't mean to imply it's a problem or anything. It's natural for the more elite among us to be in a better position to come up with the good ideas. I think also that to the extent there is deference here, it is usually well deserved, and also very tame compared to other intellectual communities.

The universe is already 13.8 billion years old. Assuming that our world is roughly representative for how long it takes for a civilization to spring up from a planet being formed (4.5 billion years), there has been about 9 billion years during which other more advanced civilizations could develop. Assuming it takes something like 100 million years to colonize an entire galaxy, one would already expect to see aliens having colonized the Milky Way, or initiated at least one of the existential risks that you describe. The fact that we are still here, is anthropic reasoning for either being alone in the galaxy somehow, that the existential risks are overblown, or, more likely, that there is already some kind of benign aliens in our neighbourhood who for whatever reason are leaving Earth alone (to our knowledge anyway), and probably protecting the galaxy from those existential risks.

(Note: I'm aware of the Grabby Aliens theory, but I still think it's quite probable that even if we are early, we are much less likely to be the very first civilization out there.)

Keep in mind, the most advanced aliens are likely BILLIONS of years ahead of us in development. They're likely unfathomably powerful. If we exist and they exist, they're probably also wise and benevolent in ways we don't understand (or else we wouldn't be here living what seem like net positive lives). Maybe there exist strong game theoretic proofs that we don't yet know for cooperation and benevolence that ensure that any rational civilization or superintelligence will have strong reasons to agree to cooperate at a distance and not initiate galaxy killing existential risks. Maybe those big voids between galaxies are where not so benign civilizations sprouted and galaxy killing existential risks occurred.

Though, it could also be that time travellers / simulators / some other sci-fi-ish entities "govern" the galaxy. Like, perhaps humans are the first civilization to develop time travel and so use their temporal supremacy to ensure the galaxy is ripe for human civilization alone, which could explain the Fermi Paradox?

All this, is of course, wild speculation. These kinds of conjectures are very hard to ground in anything else.

Anyways, I also found your post very interesting, but I'm not sure if any of these galactic level existential risks are tractable in any meaningful way at our current level of development. Maybe we should take things one step at a time?

I admit it's possibly more about optics towards both domestic and foreign audiences than necessarily a principled moral position. No doubt there's also the question of if they'd actually keep their word if faced with an existential situation.

I'll admit that a first strike is still probably in the calculus of any serious strategic considerations by China. I'm just suggesting there are political/cultural reasons why they might hesitate to take such a drastic action. There's also the historical fact that the last time someone (the Imperial Japanese) tried this on the U.S. (Pearl Harbor), it ended up being disastrous for them.

Also, given that your own estimates put U.S. intervention at about 50%, assuming Chinese planners have similar estimates, they might be thinking about rolling the dice on this with something like a limited blockade to test the waters of a U.S. response, and try to avoid getting the U.S. actively involved (similar to how Ukraine is currently playing out). An outright first strike would remove this possibility, guaranteeing U.S. involvement from the get go.

I should clarify that I don't think we should abandon deterrence entirely. My thoughts are more that we need to balance the tradeoffs and consider the strategic situation carefully. In the limit, having an obviously insurmountable defensive force to deter an attack would be ideal, but we realistically can't get there without a massive effort that will alarm China and likely accelerate the schedule for an attack. What we probably would prefer is to something along the lines of tit-for-tat increases in military strength that keep the gap from either closing or opening up more (and potentially offer the possibility of mutual reductions and de-escalation). This I think encourages China to wait for an opportunity that may (hopefully) never come.

I also think, given China's industrial capacity, that trying to outpace them is unlikely to succeed anyway. China has been building ships far faster than the U.S., much less Taiwan, is able to. The U.S. especially has aging shipyards and ships that are getting older every day, with overbudget projects like the Zumwalt and the cancelled Littoral Combat Ship showing how problematic things have become.

There are things like the Porcupine Strategy proposal to abandon offensively capable weapons like F-16s and Abrams tanks for Taiwan in favour of more defensively oriented and far cheaper weapons like man portable Stinger and Javelin missiles and lots of drones. I do think there's some merit to this idea, particularly since it wouldn't require as much buildup time (you might even be able to smuggle them in before anyone realizes it), which makes it less likely that China will see a "window of opportunity" before the buildup is complete. (Edit: I double checked and realized you do mention this idea already, so apologies for not noticing that earlier.)

In general regarding EA involvement in this though, I think a lot of thought has already been put into these concerns by the people within the NatSec establishment, such that I'm not sure what EAs can actually add to the equation. As you mentioned, it certainly isn't a neglected cause area. The gain from adding some EA affiliated money or people into this equation seems to be unlikely to me to be worth the potential alienation of China. Again, I'm thinking about wanting to coordinate on other risks and also EAs and their orgs in China, who already have a hard enough time as is. Encouraging some EAs to get involved in the NatSec stuff is already somewhat done (notably, past EA Global conferences have been held in Washington D.C.). The people who don't like EA already think we're too cozy with the establishment, and this would likely add yet more fodder to the critics.

Edit:

Just wanted to add, I do appreciate that your analysis was very thorough and probably took a lot of work to put together. Thanks for putting this together! Even if I'm somewhat critical in some parts, I think, overall, it's a thoughtful and well presented set of arguments.

Also, I mentioned the "good guys" thing in part because a lot of people I've debated about these issues with in the past (particularly on the sordid place that was Twitter), had a kind of caricatured view of China as this mindless dystopia of sorts. I also appreciate that you discussed things like the Chinese Civil War and historical causes and the KMT, which are often left out when talking about the Taiwan situation (which in news media and social media arguments is often framed in a way that makes it seem like China is just being expansionist).

I remember hearing from my dad who served in the ROC army some time around the 1970s that conscription used to be two years of service. According to Wikipedia this got reduced because they were trying to switch to an all-volunteer force in the 2010s and only restored due to fears that an invasion was becoming imminent recently.

There is actually already a new TV drama that resembles this.

I'm not sure about it being equivalent to a boy scout camp, but there are some historical reasons why the Taiwan military has relatively low morale.

The main one is that the military has strong historical ties to the Chinese Nationalist Party aka the Kuomintang (KMT) that previously ruled China and then Taiwan as the Republic of China (ROC). Taiwan is still officially the ROC, and the military forces are still technically called the "Republic of China Armed Forces" and such. Generalissimo Chiang Kai-shek basically ran these forces as his personal army, or at least, the army of the KMT party (similar to how the PLA is the army of the CCP). As such, the leadership of the Taiwan military is often considered very pro-KMT, and have close ties with the party.

Given that the current government of Taiwan is the pro-independence Democratic Progressive Party (DPP), and the KMT are in opposition, this puts the generals who feel loyal to the KMT and the Chinese nationalism they stand for, in an awkward position. The DPP has apparently tried to purge some of these generals, but the historical ties would make this difficult to achieve completely.

The KMT today is probably friendlier to the CCP than they are to the DPP, to the point that a retired general previously encouraged reunification with the mainland and the overthrow of the DPP government. This kind of "fifth column" behaviour is unique to the historical circumstances of Taiwan, and probably contributes greatly to low morale within the armed forces.

Also, unlike Ukraine, where people speak Ukrainian, which is distinct from Russian, the people of Taiwan speak a mix of Mandarin and Taiwanese (a Chinese dialect), which in particular, makes them arguably more vulnerable to Mainland Chinese propaganda, as well as infiltration by Chinese spies.