Luke Kemp and I just published a paper which criticises existential risk for lacking a rigorous and safe methodology:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3995225
It could be a promising sign for epistemic health that the critiques of leading voices come from early career researchers within the community. Unfortunately, the creation of this paper has not signalled epistemic health. It has been the most emotionally draining paper we have ever written.
We lost sleep, time, friends, collaborators, and mentors because we disagreed on: whether this work should be published, whether potential EA funders would decide against funding us and the institutions we're affiliated with, and whether the authors whose work we critique would be upset.
We believe that critique is vital to academic progress. Academics should never have to worry about future career prospects just because they might disagree with funders. We take the prominent authors whose work we discuss here to be adults interested in truth and positive impact. Those who believe that this paper is meant as an attack against those scholars have fundamentally misunderstood what this paper is about and what is at stake. The responsibility of finding the right approach to existential risk is overwhelming. This is not a game. Fucking it up could end really badly.
What you see here is version 28. We have had approximately 20 + reviewers, around half of which we sought out as scholars who would be sceptical of our arguments. We believe it is time to accept that many people will disagree with several points we make, regardless of how these are phrased or nuanced. We hope you will voice your disagreement based on the arguments, not the perceived tone of this paper.
We always saw this paper as a reference point and platform to encourage greater diversity, debate, and innovation. However, the burden of proof placed on our claims was unbelievably high in comparison to papers which were considered less “political” or simply closer to orthodox views. Making the case for democracy was heavily contested, despite reams of supporting empirical and theoretical evidence. In contrast, the idea of differential technological development, or the NTI framework, have been wholesale adopted despite almost no underpinning peer-review research. I wonder how much of the ideas we critique here would have seen the light of day, if the same suspicious scrutiny was applied to more orthodox views and their authors.
We wrote this critique to help progress the field. We do not hate longtermism, utilitarianism or transhumanism,. In fact, we personally agree with some facets of each. But our personal views should barely matter. We ask of you what we have assumed to be true for all the authors that we cite in this paper: that the author is not equivalent to the arguments they present, that arguments will change, and that it doesn’t matter who said it, but instead that it was said.
The EA community prides itself on being able to invite and process criticism. However, warm welcome of criticism was certainly not our experience in writing this paper.
Many EAs we showed this paper to exemplified the ideal. They assessed the paper’s merits on the basis of its arguments rather than group membership, engaged in dialogue, disagreed respectfully, and improved our arguments with care and attention. We thank them for their support and meeting the challenge of reasoning in the midst of emotional discomfort. By others we were accused of lacking academic rigour and harbouring bad intentions.
We were told by some that our critique is invalid because the community is already very cognitively diverse and in fact welcomes criticism. They also told us that there is no TUA, and if the approach does exist then it certainly isn’t dominant. It was these same people that then tried to prevent this paper from being published. They did so largely out of fear that publishing might offend key funders who are aligned with the TUA.
These individuals—often senior scholars within the field—told us in private that they were concerned that any critique of central figures in EA would result in an inability to secure funding from EA sources, such as OpenPhilanthropy. We don't know if these concerns are warranted. Nonetheless, any field that operates under such a chilling effect is neither free nor fair. Having a handful of wealthy donors and their advisors dictate the evolution of an entire field is bad epistemics at best and corruption at worst.
The greatest predictor of how negatively a reviewer would react to the paper was their personal identification with EA. Writing a critical piece should not incur negative consequences on one’s career options, personal life, and social connections in a community that is supposedly great at inviting and accepting criticism.
Many EAs have privately thanked us for "standing in the firing line" because they found the paper valuable to read but would not dare to write it. Some tell us they have independently thought of and agreed with our arguments but would like us not to repeat their name in connection with them. This is not a good sign for any community, never mind one with such a focus on epistemics. If you believe EA is epistemically healthy, you must ask yourself why your fellow members are unwilling to express criticism publicly. We too considered publishing this anonymously. Ultimately, we decided to support a vision of a curious community in which authors should not have to fear their name being associated with a piece that disagrees with current orthodoxy. It is a risk worth taking for all of us.
The state of EA is what it is due to structural reasons and norms (see this article). Design choices have made it so, and they can be reversed and amended. EA fails not because the individuals in it are not well intentioned, good intentions just only get you so far.
EA needs to diversify funding sources by breaking up big funding bodies and by reducing each orgs’ reliance on EA funding and tech billionaire funding, it needs to produce academically credible work, set up whistle-blower protection, actively fund critical work, allow for bottom-up control over how funding is distributed, diversify academic fields represented in EA, make the leaders' forum and funding decisions transparent, stop glorifying individual thought-leaders, stop classifying everything as info hazards…amongst other structural changes. I now believe EA needs to make such structural adjustments in order to stay on the right side of history.
In light of recent events, I came back to take another look at this paper. It’s a shame that so much of the discussion ended up focusing on the community’s reaction rather than the content itself. I think the paranoid response you describe in the post was both unjust and an overreaction. None of the paper’s conclusions seems hugely damaging or unfair to me.
That said, like other commenters, I’m not wholly convinced by your arguments. You’ve asked people to be more specific about this, and I can give two specific examples.
On technological determinism
You write that people in the EA community “disregard controlling technology on the grounds of a perceived lack of tractability” (p. 17). But you think this is probably the wrong approach, since technological determinism is “derided and dismissed by scholars of science and technology studies” and “unduly curtails the available mitigation options” (p. 18).
I’m a bit skeptical of this because I know many AI safety and biosecurity workers who would be stoked to learn that it’s possible to stop the development of powerful technologies. You write that “we have historical evidence for collective action and coordination on technological progress and regress” (p. 18). In support you give the example of weather control technology. However, I don’t think the paper you cite in this section provides very strong support for the claim. You write that weather control technology hasn’t been developed because the ENMOD Convention “seems to have successfully curtailed further research”. But the cited paper also documents significant technological challenges to progress in this area. The point of the paper is to document past cycles in which “the promise of weather control soon gave way to excessive hype and pathology” (p. 24). It’s far from clear that regulation, rather than technical challenges and lack of investment incentives, played a key role in styming progress.
In any case, I don’t think one historical case study necessarily provides much insight into modern debates over progress in AI and biotechnology. There seem to be much stronger economic incentives for investing in AI and biotechnology progress, making it much harder to reach agreements to slow or halt work in this area. Again, if that’s mistaken then I know a lot of very worried people who would be happy to hear it. In fact, I’m not even sure it’s true that EAs aren’t interested in this – many people in the community have been leading calls to ban gain-of-function research, for example!
On democracy and war
Second, I’m skeptical that democratic processes have as many benefits, and as few trade-offs, as you claim. Again, you cover a lot of ground, but on the ground that I’m most familiar with I think your assertions are too strong. You write, for example, that “[democracies] make war — a significant driver of GCRs — less likely” (p. 27). Here you cite Dafoe, Oneal and Russett on democratic peace theory. But democratic peace theory only holds that wars between democracies are, on average, less likely. Whether democracies make war in general less likely is much more debatable. See pp. 173-8 of Greg Cashman’s What Causes War? for a review of the empirical work on this question. The literature is mixed at best, with results sensitive to how one codes democracy, the time period examined, and what kind of violence (e.g. wars vs. militarized disputes) one chooses to examine as the dependent variable. But most studies find that the link between democracy and war-proneness is either non-existent or “exceedingly weak” (p. 176). Cashman writes (p. 173):
So I think your statement about the relationship between democracies and war is, at best, far more contentious than it appears. And frankly this makes me more skeptical of the other confidently-phrased statements in the paper, particularly in the democracy section, that I’m less well-placed to evaluate for myself.
On existential vs. extinction risks
On the other hand, I do also want to say that I think the section of the paper that critiques existential risk typologies is good. I'm persuaded that it’s worth distinguishing extinction risks from non-extinction existential risks, given that one has to make value judgements to identify the latter. Extinction is an objective state, but people can have different views on what it means for humanity to “fail to achieve its potential”. I think it would be helpful for people to be more careful to separate extinction ethics, existential ethics, existential risks, catastrophic risks, and extinction risks, as you recommend.
Thanks, I hadn't seen those!