The AI Messiah

"Humanity has seen many claims of this form." What exactly is your reference class here? Are you referring just to religious claims of impending apocalypse (plus EA claims about AI technology? Or are you referring more broadly to any claim of transformative near-term change?

I agree with you that claims of supernatural apocalypse have a bad track record, but such a narrow reference class doesn't (IMO) include the pretty technically-grounded concerns about AI. Meanwhile, I think that a wider reference class including other seemingly-unbelievable claims of impending transformation would include a couple of important hits. Consider:

• It's 1942. A physicist tells you, "Listen, this is a really technical subject that most people don't know about, but atomic weapons are really coming. I don't know when -- could be 10 years or 100 -- but if we don't prepare now, humanity might go extinct."

• It's January 2020 (or the beginning of any pandemic in history). A random doctor tells you "Hey, I don't know if this new disease will have 1% mortality or 10% or 0.1%. But if we don't lock down this entire province today, it could spread to the entire world and cause millions of deaths."

• It's 1519. One of your empire's scouts tells you that a bunch of white-skinned people have arrived on the eastern coast in giant boats, and a few priests think maybe it's the return of Quetzalcoatl or something. You decide that this is obviously crazy -- religious-based forecasting has a terrible track record, I mean these priests have LITERALLY been telling you for years that maybe the sun won't come up tomorrow, and they've been wrong every single time. But sure enough, soon the European invaders have slaughtered their way to your capital and destroyed your civilization.

Although the Aztec case is particularly dramatic, many non-European cultures have the experience of suddenly being invaded by a technologically superior foe powered by an exponentially self-improving economic engine -- this sounds at least as similar to AI worries as your claim that Christianity and AI worry are in the same class. There might even be more stories of sudden European invasion than predictions of religious apocalypse, which would tilt your base-rate prediction decisively towards believing that transformational changes do sometimes happen.

Thanks for writing this.

I thought about why I buy the AI risk arguments despite the low base rate, and I think the reason touches on some pretty important and nontrivial concepts.

When most people encounter a complicated argument like the ones for working on AI risk, they are in a state of epistemic learned helplessness: that is, they have heard many convincing arguments of a similar form be wrong, or many convincing arguments for both sides. The fact that an argument sounds convincing fails to be much evidence that it's true.

Epistemic learned helplessness is often good, because in real life arguments are tricky and people are taken in by false arguments. But when taken to an extreme, it becomes overly modest epistemology: the idea that you shouldn't trust your models or reasoning just because other people whose beliefs are similar on the surface level disagree. Modest epistemology would lead you to believe that there's a 1/3 chance you're currently asleep, or that the correct religion is 31.1% likely to be Christianity, 24.9% to be Islam, and 15.2% to be Hinduism.

I think that EA does have something in common with religious fundamentalists: an orientation away from modest epistemology and towards taking weird ideas seriously. (I think the number of senior EAs who used to be religious fundamentalists or take other weird ideas seriously is well above the base rate.) So why do I think I'm justified in spending my career either doing AI safety research or field-building? Because I think the community has better epistemic processes than average. 

Whether it's through calibration, smarter people, people thinking for longer or more carefully, or more encouragement of skepticism, you have to have a thinking process that results in truth more often than average, if you want to reject modest epistemology and still believe true things. From the inside, the EA/rationalist subcommunity working on AI risk is clearly better than most millenarians (you should be well-calibrated about this claim, but you can't just say "but what about from the outside?"-- that's modest epistemology). If I think about something for long enough, talk about it with my colleagues, post it on the EA forum, invite red-teaming, and so on, I expect to reach the correct conclusion eventually, or at least decide that the argument is too tricky and remain unsure (rather than end up being irreversibly convinced of the wrong conclusion). I'm very worried about this ceasing to be the case.

Taking weird ideas seriously is crucial for our impact: I think of there being a train to crazy town which multiplies our impact by >2x at every successive stop, has increasingly weird ideas at every stop, and at some point the weird ideas cease to be correct. Thus, good epistemics are also crucial for our impact.

Hey! I liked certain parts of this post and not other parts of this post. I appreciate the thoughtfulness by which you critique EA through this post.

On your first point about the AI messiah: 

I think the key distinction is that there are many reasons to believe this argument about the dangers of an AGI are correct, though. Even if many claims with a similar form are wrong, that doesn't exclude this specific claim from being right. 

"Climate scientists keep telling us about how climate change is going to be so disastrous and we need to be prepared. But humanity has seen so many claims of this form and they've all been so wrong!"

The key distinction is that there is a lot of reason to believe that AGI will be dangerous. There is also a lot of reason to support the claim that we are not prepared currently. Without addressing that chain of logic directly, I don't think I'm convinced by this argument.

On your second point about the EA religious tendencies:

Because religious communities are one of the most common communities we see, there's obviously going to be parallels that exist between religious communities and EA. 

Some of these analogies hold, others not so much. We, too, want to community build, network, and learn from each other. I'd love for you to point at specific examples of things EA do, from conferences to holding EA university groups, that are ineffective or unnecessary.

To perhaps a greater point of EA perhaps becoming too groupthink-y, which I think may be warranted:

I think a key distinction is that EA has a healthy level of debate, disagreement, and skepticism - while religions tend to demand blind faith in believing something unprovable. This ongoing debate on how to do the most good I personally find the most valuable in the community - and I hope this spirit never dies.

Keep on critiquing EA; I think such critiques are extremely valuable. Thanks for writing this.

I'm not so sure about the religious tendencies, at least not in comparison to other communities or social movements. Especially if the people who seem to be most interested in AI alignment are the ones least interested in tithing/special diets .

Also roughly half the people who are seen as leaders  don't identify as effective altruist.  It would be hard to imagine leaders in the environmentalism movement not describing themselves as environmentalists.

What if the man you’re talking to then shows that praying to the approaching god actually works?

What if the man shows that anyone who knows the appropriate rituals can pray to the god and get real benefits from doing so?

What if people have already made billions of dollars in profit from industrialized prayer?

What if the man conclusively shows that prayers to the god have been getting easier and more effective over time?

In such a case, you should treat the god’s approach very seriously. Without such proof, you should dismiss his claims. The difference is that gods are imaginary, and AI is real.

Many environmentalists make claims that sound something like this:

“Because of the mistakes and greed of mankind, a terrible disaster is coming to the earth. I don’t know when exactly, but we are heading in the wrong direction and things are getting worse; maybe in 100 or 200 years, maybe more, but maybe in 20. We need to be change our behaviour and prepare for the consequences, because if not there will be many terrible disasters and we could all die. However, if we change our ways and prepared adequately then we can live a harmonious existence; not only will disaster be averted but many of our other problems will be resolved."

Clearly this is not exactly the same as the religious story or the AI story, but it definitely has a lot of similarities. And there are definitely some environmentalists who seem very ideological (e.g. those who oppose nuclear and hydro power, or claim the world will end in 12 years). But I don't think this is a very strong argument overall for skepticism about climate change. Just because there are some bad reasons to believe something doesn't mean we should ignore the good ones, nor ignore relevant qualified experts if we lack the ability to evaluate the arguments ourselves.

Obvious point, but you could assign significant credence to this being the right take, and still think working on A.I. risk is very good in expectation, given exceptional neglectedness and how bad an A.I. takeover could be. Something feels sleazy and motivated about this line of defence to me, but I find it hard to see where it goes wrong. 

Thank you. This brings together nicely some vague concerns I had, that I didn't really know how to formulate beyond "Why are people going around with Will MacAskill's face on their shirts?".

In brief: writing that is kind, relevant to the discussion at hand, and honest.

https://forum.effectivealtruism.org/posts/yND9aGJgobm5dEXqF/guide-to-norms-on-the-forum

Great points, very much liked your directness.

I think this is worthwhile flagging, but I also wonder how useful we should expect this kind of relatively shallow pattern matching to be in terms of making decisions?

For example, sure we have gathering, but all community organisations have gathering. How similar are our gatherings to religious gatherings? Most EA meetups I've been to tend to either be social or have a talk/discussion like all other kinds of groups interested in a particular topic. There doesn't seem to be anything specifically religious about how we go about these.

I imagine that if you went through some of the other similarities you identified, then I suspect mightn't find the similarity to run very deep. I think this is especially likely if you also look at other causes generally considered non-religious, such as environmentalism or political parties in order to establish a baseline.

One can imagine, say, a Christian charitable organization where non-Christians see its work for the poor, the sick, the hungry, and those in prison as good, and don't really mind that some of the money also goes to funding theologians and building cathedrals.

Although Christianity kind of has it built in that you have to help the poor even if the Second Coming is tomorrow. The risk in EA is if people were to become erroneously convinced of a short AI timeline and conclude all the normal charitable stuff is now pointless.

Moynihan's book X-Risk makes a key subtle distinction between religious apocalypticism where God is in control and the secular notion of a risk that destroys all value that only humanity can stop. I'm uncertain how much the distinction is maintained in practice but I recommend the book.

Hmm, do you have close friends who grew up in non-Abrahamic cultures? I feel like a lot of the stuff in this post seems to presume, though does not state outright, that something like religion (or even a very specific form of religion??) is approximately a human universal. This feels very alien/surprising compared to my upbringing (which is not particularly atheist by Chinese standards, like my mom literally converted to Christianity), so I'm wondering if one of us is just confused here.

(Looping back here because I thought the post never made sense to me but it did to a lot of people. At first I attributed it to just pro- vs anti- LT tribalism, but wanted to advance the alternative hypothesis that we just have different expectations for the appeal of religiosity or how probable different ideas* map to Christianity in particular).

*I also thought the mapping of "simulation hypothesis -> Christianity" that seems popular in some circles recently to be kind of silly pattern-matching.

It does seem like an optimistic expectation that there will be an arrival of entities that are amazingly superior to us. This is not far-fetched though. Computers already surpass humans' capacities on several thought processes, and therefore have already demonstrated that they are better in some aspects of intelligence. And we've created robots that can outperform humans in virtually all physical tasks. So, the expectation is backed by evidence.

Expecting super AGI differs from expecting the arrival of a messiah-like figure in that instead of expecting a future in which an entity will come on its own and end all our suffering and improve our lives immeasurably, we are doing the work to make AI improve our lives. Also, the expectations differ in how we prepare for them. In the case of the messiah, it seems like acting moral so we can get into heaven might be vague, unchanging, and random. On the other hand, in the case of super AGI,  AI safety work is constantly changing and learning new things. However, it is still interesting that the two expectations bear a resemblance. 

Thanks for writing this. A similar observation lead me to write this post.

Reality has no requirement to conform to your heuristics of what is ‘normal’, but I think that we could use some more outside-view work on just how bizarre and unsettling this world-view is, even if it is true.

If anything, this is a claim that people have been bringing up on Twitter recently, the parallels between EA and religion. It’s certainly something we should be aware of since, having ”blind faith” in religion is something that be good, we don’t seem to actually want to do this within EA. I could explain why I think AI risk is different from messiah thing, but Rob Miles explains it well here: 

Given limited information (but information nonetheless), I think AI risk could potentially lead to serious harm or not at all, and it’s worth hedging our bets on this cause area (among others). This feels different then choosing to have blind faith in a religion, but I can see why outsiders think this. Though we can be victims of post-rationalization, I think religious folks have reasons to believe in a religion. I think some people might gravitate towards AI risk as a way to feel more meaning in their lives (or something like that), but my impression is that this is not the norm. 

At least in my case,  it’s like, “damn we have so many serious problems in the world and I want to help with them all, but I can’t. So, I’ll focus on areas of personal fit and hedge my bets even though I’m not so sure about this AI thing and donate what I can to these other serious issues.”

The major difference, at least for me, is that with religion you're expected to believe in an oncoming superpower based on little more than the supposed credibility of those giving testimony whereas with superintelligent AI we can witness a god being incubated through our labor. 

A person does not need a great amount of technical knowledge or a willingness to defer to experts to get the gist of the AI threat. A machine that can continually and autonomously improve its general problem solving ability will eventually render us potentially powerless. AI computers look like they could be those machines given their extremely rapid development as well as inherent advantages over humans (eg faster information transfer times compared to humans who require on substances like electrolytes and neurotransmitters to physically travel). It will not necessarily take a string of programming Einsteins to develop the necessary code for superintelligence as an evolutionary trial-and-error approach will eventually accomplish this.