Agreed! My comment was as aimed at the absurd conclusions one makes when weighing the tradeoffs we make today against trillions of unborn humans. That logic leads to extreme positions.

These are all great suggestions! As for my objections to EA as a whole versus a subset, it reminds me a bit of a defense that folks employ whenever a larger organization is criticised. Defenses that one hears from Republicans in the US for example. "It's not all of us, just a vocal subset!" That might be true, but I think it misses the point. It's hard to soul-search and introspect as an organization or a movement if we collectively say, "not all-EA" when someone points to the enthusiasm around SBF and ideas like buying up coal mines. 

ah yes sorry, here it is: https://link.springer.com/chapter/10.1007/978-3-662-45704-7_2

Love this thoughtful response! 

Good feedback--I see the logic of your points, and don't find faults with any of them. 

On AIXR as valid and what the response would be, you're right; I emphasize the practical nature of the policy recommendation because otherwise, the argument can veer into the metaphysical. To use an analogy, if I claim there's a 10% chance another planet could collide with Earth and destroy the planet in the next decade, you might begrudgingly accept the premise to move the conversation on to the practical aspect of my forecast. Even if that were true, what would my policy intervention look like? Build interstellar lifeboats? Is that feasible in the absence of concrete evidence? 

Agree--armchair psychoanalysis isn't really useful. What is useful is understanding how heuristics and biases work on a population level. If we know that, in general, projects run over budget and take longer than expected, we can adjust our estimates. If we know experts mis-forecast x-risk, we can adjust for that too. That's far from psychoanalysis. 

I don't really know what the median view on AIXR within EA communities truly is. One thing's for certain: the public narrative around the issue highly tilts towards the "pause AI" camp and the Yudkowskys out there. 

On the common sense of X-risk--one of the neat offices that few people know of at the State Department is the Nuclear Risk Reduction Center or NRRC.  It's staffed 24/7 and has foreign language designated positions, meaning at least someone in the room speaks Russian, etc. The office is tasked with staying in touch with other nations to reduce the odds of a miscalculation and nuclear war. That makes tons of sense. Thinking about big problems that could end the world makes sense in general--disease, asteroids, etc. 

What I find troubling is the propensity to assign odds to distant right-tail events. And then to take the second step of recommending costly and questionable policy recommendations.  I don't think these are EA consensus positions, but they certainly receive outsized attention. 

Exactly, the burden of proof lies with those who make the claim. 

I hope EA is able to get back to the basics of doing the most real-world good with limited resources rather than utilitarian nonsense of saving trillions of theoretical future humans. 

These are all very reasonable positions, and one would struggle to find fault with them. 

Personally, I'm glad there are smart folks out there thinking about what sorts of risks we might face in the near future. Biologists have been talking about the next big pandemic for years. It makes sense to think these issues through. 

Where I vehemently object is on the policy side. To use the pandemic analogy, it's the difference between a research-led investigation into future pandemics and a call to ban the use of CRISPR. It's impractical and, from a policy perspective, questionable.  

The conversation around AI within EA is framed as "we need to stop AI progress before we all die." It seems tough to justify such an extreme policy position.  

This is true--it's the same tactic anti-GMO lobbies, the NRA, NIMBYs, and anti-vaxxers have used. The public as a whole doesn't need to be anti-AI, even a vocal minority will be enough to swing elections and ensure an unfavorable regulatory environment. If I had to guess, AI would end up like nuclear fission--not worth the hassle, but with no off-ramp, no way to unring the alarm bell. 

I think you put it well when you said: 

"Some people think that the kinds of risks I’m worried about are far off, farfetched or ridiculous."

If I made the claim that we had 12 months before all of humanity is wiped by an asteroid, you'd rightly ask me for evidence. Have I picked up a distant rock in space using radio telescopes? Some other tangible proof? Or is it a best-guess, since, hey, it's technically possible that we could be hit with an asteroid on any given year. Then imagine if I advocate we spend two percent of global GDP preparing for this event. 

That's where the state of AGI fear is--all scenarios depend on wild leaps of faith and successive assumptions that build on each other. 

I've attempted to put this all in one place with this post. 

 

Let's look at the history of global bans: 
- They don't work for doping in the Olympics. 
- They don't work for fissile material.
- They don't prevent luxury goods from entering North Korea.
- They don't work against cocaine or heroine.

We could go on. And those examples are much easier to implement--there's global consensus and law enforcement trying to stop the drug trade, but the economics of the sector mean an escalating war with cartels only leads to greater payoffs for new market entrants. 

Setting aside practical limitations, we ought to think carefully before weaponizing the power of central governments against private individuals. When we can identify a negative externality, we have some justification to internalize it. No one wants firms polluting rivers or scammers selling tainted milk. 

Generative AI hasn't shown externalities that would necessitate something like a global ban. 

Trucks: we know what the externalities of a poorly piloted vehicle are. So we minimize those risks by requiring competence. 

And on a morally advanced society--yes, I'm certain a majority of folks if asked would say they'd like a more moral and ethical world. But that's not the question--the question is who gets to decide what we can and cannot do? And what criteria are they using to make these decisions? Real risk, as demonstrated by data, or theoretical risk? The latter was used to halt interest in nuclear fission. Should we expect the same for generative AI? 

In my view, we ought to show humility in our ability to accurately forecast risk from any one domain beyond the five-year window. There's little evidence to suggest anyone is better than random chance at making accurate forecasts beyond a certain time horizon. 

At the core of SBF's grift was the notion that stealing money from Canadian pensioners was justified if that money was spent reducing the odds of an x-event. After all, the end of humanity in the next century would eliminate trillions of potential lives, so some short-term suffering today is easily tolerated. Simple utilitarian logic would dictate that we should sacrifice well-being today if we can prove that those resources have a positive expected value. 

I think anyone making an extraordinary claim needs equally extraordinary evidence to back it. That doesn't mean x-risk isn't theoretically possible. 

Let's put it this way--if I said there was a chance that life on Earth could be wiped out by an asteroid, few would argue against it since we know the base rate of asteroids hitting Earth is non-zero. We could argue about the odds of getting hit by an asteroid in any given year, but we wouldn't argue over the very notion. And we could similarly argue over the right level of funding to scan deep space for potential Earth-destroyers, though we wouldn't argue over the merits of the enterprise in the first place. 

That is very different than me claiming with confidence that there is a 25% chance humanity perishes from an asteroid in the next 20 years. And along with this claim I recommend we stop all infrastructure projects globally and direct resources to interstellar lifeboats. You'd rightly ask for concrete evidence in the face of this claim. 

The latter is what I hear from AGI alarmists.