This post will be direct because I think directness on important topics is valuable. I sincerely hope that my directness is not read as mockery or disdain towards any group, such as people who care about AI risk or religious people, as that is not at all my intent. Rather my goal is to create space for discussion about the overlap between religion and EA.
–
A man walks up to you and says “God is coming to earth. I don’t know when exactly, maybe in 100 or 200 years, maybe more, but maybe in 20. We need to be ready, because if we are not ready then when god comes we will all die, or worse, we could have hell on earth. However, if we have prepared adequately then we will experience heaven on earth. Our descendants might even spread out over the galaxy and our civilization could last until the end of time.”
My claim is that the form of this argument is the same as the form of most arguments for large investments in AI alignment research. I would appreciate hearing if I am wrong about this. I realize when it’s presented as above it might seem glib, but I do think it accurately captures the form of the main claims.
Personally, I put very close to zero weight on arguments of this form. This is mostly due to simple base rate reasoning: humanity has seen many claims of this form and so far all of them have been wrong. I definitely would not update much based on surveys of experts or elites within the community making the claim or within adjacent communities. To me that seems pretty circular and in the case of past claims of this form I think deferring to such people would have led you astray. Regardless, I understand other people either pick different reference classes or have inside view arguments they find compelling. My goal here is not to argue about the content of these arguments, it’s to highlight these similarities in form, which I believe have not been much discussed here.
I’ve always found it interesting how EA recapitulates religious tendencies. Many of us literally pledge our devotion, we tithe, many of us eat special diets, we attend mass gatherings of believers to discuss our community’s ethical concerns, we have clear elites who produce key texts that we discuss in small groups, etc. Seen this way, maybe it is not so surprising that a segment of us wants to prepare for a messiah. It is fairly common for religious communities to produce ideas of this form.
–
I would like to thank Nathan Young for feedback on this. He is responsible for the parts of the post that you liked and not responsible for the parts that you did not like.
I wasn't really trying to say "See, messianic stories about arriving gods really work!", as to say "Look, there are a lot of stories about huge dramatic changes, AI is not more similar to the story of Christianity as it is to stories about new technologies or plagues or a foreign invasion." I think the story of European world conquest is particularly appropriate not because it resembles anyone's religious prophecies, but because it is an example where large societies were overwhelmed and destroyed by the tech+knowledge advantages of tiny groups. This is similar to AI, which would start out outnumbered by all of humanity but might have a huge intelligence + technological advantage.
Responding to your request for times when knowledge of European invasion was actionable for natives: The "Musket Wars" in New Zealand were "a series of as many as 3,000 battles and raids fought among Māori between 1807 and 1837, after Māori first obtained muskets and then engaged in an intertribal arms race in order to gain territory or seek revenge for past defeats". The bloodshed was hugely net-negative for the Māori as a whole, but individual tribes who were ahead in the arms race could expand their territory at the expense of enemy groups.
Obviously this is not a very inspiring story if we are thinking about potential arms races in AI capabilities: