Holly -- 

I think the frustrating thing here, for you and me, is that, compared to its AI safety fiascos, EA did so much soul-searching after the Sam Bankman-Fried fiasco with the FTX fraud in 2022. We took the SBF/FTX debacle seriously as a failure of EA people, principles, judgment, mentorship, etc. We acknowledged that it hurt EA's public reputation, and we tried to identify ways to avoid making the same catastrophic mistakes again.

But as far as I've seen, EA has done very little soul-searching for its complicity in helping to launch OpenAI, and then in helping to launch Anthropic -- both of which have proven to be far, far less committed to serious AI safety ethics than they'd promised, and far less than we'd hoped. 

In my view, accelerating the development of AGI, by giving the EA seal of approval to first OpenAI and then Anthropic, has done far, far more damage to humanity's likelihood of survival than the FTX fiasco ever did. But of course so many EAs go on to get lucrative jobs at OpenAI and Anthropic, and 80,000 Hours is delighted to host such job ads, that EA as a career-advacement movement is locked into the belief that 'technical AI safety research' within 'frontier AI labs' is a far more valuable use of bright young people's talents than merely promoting grass-roots AI safety advocacy.

Let me know if that captures any of your frustration. It might help EAs understand why this double standard -- taking huge responsibility for SBF/FTX turning reckless and evil, but taking virtually no responsibility for OpenAI/Anthropic turning reckless and evil -- is so grating to you (and me). 

Holly --

Thanks for this assertive, candid, blunt, challenging post. 

You and I have, I think, reached similar views on some of the critical weaknesses of EA as it's currently led, run, funded, and defended.

All too often,  'EA discourse norms' have been overly influenced by LessWrong discourse norms, where an ivory-tower fetishization of 'rational discourse', 'finding cruxes', 'updating priors', 'avoiding ad hominems', 'steel-manning arguments', etc becomes a substitute for effective social or political action in the world as it is, given human nature as it is, and given the existential risks that we actually face.

And, recently, way too many EAs have been seduced into the Dario Amodei delusion that if the 'good guys' build Artificial Superintelligence, with good intentions, and enough effort on 'technical AI alignment', we'll all be fine. 

That's a great excuse for EAs going over to 80k Hours, which still (unbelievably, and utterly immorally) posts dozens of 'AI safety' jobs at Anthropic and OpenAI, and getting that sweet, sweet salary to live in the Bay Area, hang out with the cool kids, and pretend you're doing good. (When, in fact, you're being used as a safety-washing prop by some of the most reckless corporations on Earth.)

People respond to incentives. Even EAs. 

And if your prospects of being hired by Anthropic for a mid-6-figure salary doing corporate safety-washing depend on not making a fuss, and denouncing Pause AI, and ignoring the passion and dedication of those who believe ASI is actually an extinction risk, then it's tempting to many EAs to ignore your message, downvote your post (and probably this one), and carry on as usual, feeling virtue about donating a bit of money to saving some shrimp, or whatever.

Matt - thanks for the quick and helpful reply.

I think the main benefit of explicitly modeling ASI as being a 'new player' in the geopolitical game is that it highlights precisely the idea that the ASI will NOT just automatically be a tool used by China or the US -- but rather than it will have its own distinctive payoffs, interests, strategies, and agendas. That's the key issue that many current political leaders (e.g. AI Czar David Sacks) do not seem to understand -- if America builds an ASI, it won't be 'America's ASI', it will be the ASI's ASI, so to speak.

ASI being unaligned doesn't necessarily mean that it will kill all humans quickly -- there are many, many possible outcomes other than immediate extinction that might be in the ASI's interests. 

The more seriously we model the possible divergences of ASI interests from the interests of current nation-states, the more persuasively we can make the argument that any nation building an ASI is not just flipping a coin between 'geopolitical dominance forever' and 'human extinction forever' -- rather, it's introducing a whole new set of ASI interests that need to be taken into account.

Matt -- thanks for an insightful post. Mostly agree.

However, on your point 2 about 'technological determinism': I worry that way too many EAs have adopted this view that building ASI is 'inevitable', and that the only leverage we have over the future of AI X-risk is to join AI companies explicitly trying to build ASI, and try to steer them in benign directions that increase control and alignment.

That seems to be the strategy that 80k Hours has actively pushed for years. It certainly helps EAs find lucrative, high-prestige jobs in the Bay Area, and gives them the illusion that they're doing good. But to outsiders, it looks like little more than a self-serving jobs program for utilitarians who want a slice of that sweet, sweet, equity in AI companies -- without any of the guilt of actually working on ASI capabilities development.

And the weird thing is, this strategy only makes sense if we believe two key things: (1) ASI development is 'inevitable' -- even if virtually all of humanity agrees that it would be suicidal, and (2) ASI alignment is solvable -- such that we can keep control of ASIs, and force them to work for humans, generation after generation, forever.  

Both of these seem equally and wildly implausible. And the sooner we recognize their implausibility, the faster we can move beyond this rather cynical/self-serving 80k Hours strategy of encouraging young EAs to join the safety-washing, PR-window-dressing 'technical AI alignment' groups at frontier AI companies like OpenAI, Anthropic, DeepMind, etc.

Thanks for this analysis. I think your post deserves more attention, so I upvoted it.

We need more game-theory analyses like this, of geopolitical arms race scenarios. 

Way too often, people just assume that the US-China rivalry can be modelled simply as a one-shot Prisoner's Dilemma, in which the only equilibrium is mutual defection (from humanity's general interests) through both sides trying to build ASI as soon as possible.

As your post indicates, the relevant game theory must include incomplete and asymmetric information, possible mixed-strategy equilibria, iterated play that depends strongly on what the other player has been doing, etc. 

I would also encourage more development of game theory scenarios that explicitly model the creation of ASI as the introduction of a new player with its own rules, strategies, and payoffs. 

Building an ASI isn't just giving existing players a new tool for 'winning the game'. It's introducing a new player with its own interests (unless the ASI is 100% totally, reliably controlled & aligned with one existing player -- which is probably impossible.)

Tobias -- I take your point. Sort of. 

Just as they say 'There are no atheists in foxholes' [when facing risk of imminent death during combat], I feel that it's OK to pray (literally and/or figuratively) when facing AI extinction risk -- even if one's an atheist or agnostic. (I'd currently identify as an 'agnostic', insofar as the Simulation Hypothesis might be true). 

My X handle 'primalpoly' is polysemic, and refers partly to polyamory, but partly to polygenic traits (which I've studied extensively), and partly to some of the hundreds of other words that start with 'poly'. 

I think that given most of my posts on X over the last several years, and the people who follow me, I'm credibly an insider to the conservative right.

My new interview (48 mins) on AI risks for Bannon's War room: https://rumble.com/v6z707g-full-battleground-91925.html

This was my attempt to try out a few new arguments, metaphors, and talking points to raise awareness about AI risks among MAGA conservatives. I'd appreciate any feedback, especially from EAs who lean to the Right politically, about which points were most or least compelling.

PS the full video of my 15-minute talk was just posted today on the NatCon YouTube channel; here's the link

David -- I considered myself an atheist for several decades (partly in alignment with my work in evolutionary psychology), and would identify now as an agnostic (insofar as the Simulation Hypothesis has some slight chance of being true, and insofar as 'Simulation-Coders' aren't functionally any different from 'Gods', from our point of view).

And I'm not opposed to various kinds of reproductive tech, regenerative medicine research, polygenic screening, etc.

However, IMHO, too many atheists in the EA/Rationalist/AI Safety subculture have been too hostile or dismissive of religion to be effective in sharing the AI risk message with religious people (as I alluded to in this post). 

And, I think way too much overlap has developed between transhumanism and the e/acc cult that dismisses AI risk entirely, and/or that embraces human extinction and replacement by machine intelligences. Insofar as 'transhumanism' has morphed into contempt for humanity-as-it-is, and into a yearning for hypothetical-posthumanity-as-it-could be, then I think it's very dangerous.

Modest, gradual, genetic selection or modification of humans to make them a little healthier or smarter, generation by generation? That's fine with me. 

Radical replacement of humanity by ASIs in order to colonize the galaxy and the lightcone faster? Not fine with me.

Arepo - thanks for your comment.

To be strictly accurate, perhaps I should have said 'the more you know about AI risks and AI safety, the higher your p(doom)'. I do think that's an empirically defensible claim. Especially insofar as most of the billions of people who know nothing about AI risks have a p(doom) of zero.

And I might have added that thousands of AI devs employed by AI companies to build AGI/ASI have very strong incentives not to learn about too much about AI risks and AI safety of the sort that EAs have talked about for years, because such knowledge would cause massive cognitive dissonance, ethical self-doubt, regret (as in the case of Geoff Hinton), and/or would handicap their careers and threaten their salaries and equity stakes.