pretty much generally agreed upon in the EA community that the development of unaligned AGI is the most pressing problem

While there is significant support for "AI as cause area #1", I know plenty of EAs that do not agree with this. Therefore, "generally agree upon" feels like a bit too strong of a wording to me. See also my post on why EAs are skeptical about AI safety

For viewpoints from professional AI researchers, see Vael Gates interviews with AI researchers on AGI risk.

I mention those pieces not to argue that AI risk is overblown, but rather to shed more light on your question.

This is not just a general public/"uninformed masses" phenomenon. It's worth noting that even among AI/ML researchers, AGI concerns and AI safety is niche. Among the people with literally the most exposure to and expertise in AI/ML systems, only a small fraction are focusing on AGI and AI safety. A comparatively larger fraction of them work on "nearterm AI ethics" (i.e. fairness, discrimination and privacy concerns for current ML systems): there is a pretty large conference on this topic area (FAccT), and I do not know if AI safety has a comparably-sized conference.

Why is this? My anecdotal experience with ML researcher friends who don't work on AI safety is that they basically see AGI as very unlikely. I am in no position to adjudicate plausibility of these arguments, but that's the little I have seen.

I think for other prominent x-risks like pandemics, climate change and nuclear war, there are “smaller scale versions” which are well known and have caused lots of harm (Ebola / COVID, extreme weather events, Hiroshima and Nagasaki). I think these make the risk from more extreme versions of these events feel realistic and scary.

For AI, I can’t think of what smaller scale versions of uncontrollable AI would look like, and I don’t think other faults with AI have caused severe enough / well known enough harm yet. So AI x-risk feels more like a fiction thing.

So based on my ideas on this, raising awareness on immediate harms from AI, (eg - surveillance uses, discrimination) could make the x-risk feel realistic too (at least the x-risk from AI controlled by malicious actors) - this could be tested in a study. But I can’t think of a smaller scale version of uncontrollable AI per se.

without mainstream support of the idea that AI is a risk, I feel like it's going to be a lot harder to get to where we want to be.

The consequences of everyone freaking out about AI are not obviously or automatically good. (I'm personally optimistic that they could be good, but people waking up to AI by default entails more racing toward powerful AI.)

A lot of it, I would guess, comes down to lack of exposure to the reasonable version of the argument.

A key bottleneck would therefore be media, as it was for climate change.

Trying to upload "media brain", here are reasons why you might not want to run a story on this topic (apologies if it seems dismissive, that is not the intent, I broadly sympathize with the thesis that AGI is at least a potential major cause area, however it may be helpful to channel where the dismissiveness comes from on a deeper level)...

-The thesis is associated with science-fiction. Taking science-fiction seriously as an adult is associated with escapism and a refusal to accept reality.

- It is seen as something only computer science "nerds", if you'll pardon the term, worry about. This makes it easy to dismiss the underlying concept as reflecting a blinkered vision of the world. The idea is that if only those people went outside and "touched grass", they would develop a different worldview - that the worldview is a product of being stuck inside a room looking at a computer screen all day. 

- There is something inherently suspicious about CS people looking at the problem of human suffering and concluding that more than helping people in tangible ways in the physical world, the single most important thing other people should be doing is to be more like them and to do more of the things that they are already doing. Of course the paper clip optimizer is going to say that increasing the supply of paper clips is the most pressing problem in the universe. In fact it's one reason why media people love to run stories about the media itself!

- In any case there appears to be no theory of change - "if only we did this, that would take care of that".  At least with climate change there was the basic idea of reducing carbon emissions. What is the reducing carbon emissions of AI X-risk? If there is nothing we can do that would make a difference, might as well forget about it. 

What might help overcome these issues?

To start with you'd need some kind of translator, an idea launderer.  Someone 

1. well-known
2. credible (so not a Hollywood actor)
3. who isn't coded as a "nerd" (someone for whom your prior guess would be low that they would care about this topic)
4. who isn't too polarizing (no eccentric billionaire or polarizing politician).

Basically, a human equivalent of the BBC. Then have them produce documentaries on the topic - some might take off.

Babbling without any pruning:

- It is very difficult to imagine such an AI
- Currently, computers and AI in video games are stupid
- the collective representations are such as Ex Machina or Terminator reinforce the idea that it is only fiction
- understanding  the orthogonality thesis requires a fine-grained epistemology to dissociate two concepts often linked in practice in everyday life
- Loss of status for academics
- It is possible that it is really too complicated for an AI of just higher level than a human-level to design a nanoconductor factory. In order to understand the risks, we have to take a step back on history, understand what an exponential curve is, and tell ourselves that a superintelligence will arrive later on

If you speak to a stranger about your worries of unaligned AI, they'll think you're insane (and watch too many sci-fi films).

Have you actually tried this? (How many times?)

Talking to lots of normies about AI risk is on my to-do list, but in the meantime there are some relevant surveys, and while it's true that people don't often bring up AI as a thing-they're-concerned-about, if you ask them about AI a lot of people seem pretty concerned that we won't be able to control it (e.g., the third of these surveys finds that 67% of respondents are concerned about "artificial intelligence becoming uncontrollable").

(Kelsey Piper has written up the second survey linked above.)

If you speak to a stranger about your worries of unaligned AI, they'll think you're insane (and watch too many sci-fi films).

I'm not so sure this is true. In my own experience, a correct explanation of the problem with unaligned AI makes sense to almost everyone with some minimum of reasoning skill. Although this is anecdotal, I would not be surprised if an actual survey among "strangers" would show this too.

Commenting on your general point, I think the reason is that most people's sense of when AGI could plausibly happen is "in the far future", which makes it psychologically unremarkable at this point.

Something like extinction (even if literal extinction is unlikely) from climate change, although possibly further off in time, might feel closer to a lot of people because climate change is already killing people.

Nitpicking here, but I do not believe that AI is the most pressing x-risk problem, as opposed to a pressing one:

It's pretty much generally agreed upon in the EA community that the development of unaligned AGI is the most pressing problem

Added 2022-08-09: The original claim was that AGI is the most pressing problem from a longtermist point of view, so I've edited this comment to clarify that I mean problem, not x-risk. To prove that AGI is the most pressing problem, one needs to prove that it's more cost-effective to work on AGI safety than to work on any other x-risk and any broad intervention to improve the future. (For clarity, a "pressing" problem is one that's cost-effective to allocate resources to at current margins.)

It's far from obvious to me that this is a dominant view: in 2021, Ben Todd said that broad interventions like improving institutional decision-making and reducing great power conflict were the largest resource gap in the EA cause portfolio.

Here's my attempt to reflect on the topic: https://forum.effectivealtruism.org/posts/PWKWEFJMpHzFC6Qvu/alignment-is-hard-communicating-that-is-harder 

I recently found a Swiss AI survey that indicates that many people do care about AI.
[This is only very weak evidence against your thesis, but might still interest you 🙂.]

The question: 
"Do you fear the emergence of an "artificial super-intelligence", and that robots will take power over humans?"

From the general population, 11% responded "Yes, very", and 37% responded "Yes, a bit". 
So, half of the respondents (that expressed any sentiment) were at least somewhat worried.

The 'opinion leaders' however are much less concerned. Only 2% have a lot of fear and 23% have a bit of fear.

A few random thoughts I have on this:

I've tried speaking to a few non-EA people (very few, countable on hand)  and I kind of agree that they think you've watched way too much sci-fi when talking about AI safety but they don't think it's too far-fetched. A specific conversation I remember having made me realize that one reason might be that a lot of people simply think that they cannot do much about it. 'Leave it to the experts' or 'I don't know anything about AI and ML' seems to be a thought that non-EA people might have on the issue, hence preventing them from actively trying to reduce the risk, if it finds a way into their list of important problems at all. There's also the part about AI safety not being a major field leading to misconceptions like the need for a compsci PhD and a lot of technical math/CS knowledge to work in AI safety when there actually exist roles that do not require such expertise. This quite obviously prevents them from changing their career to work in AI safety, but, even more so, it discourages them to read about it at all (this might also be the reason why distillation of AI alignment work is in high demand) even though we see people read about international conflicts, nuclear risk, and climate change more frequently (I'm not sure of the difference in scale but I can personally vouch for this since I had never heard of AI alignment before joining the EA community).

Great post! I'm gonna throw out two spicy takes.

Firstly, I don't think it's so much that people don't care about AI Safety, I think it's largely who cares about a threat is highly related to who it affects. Natural disasters etc affect everyone relatively (though not exactly) equally, whereas AI harms overwhelmingly affect the underpriviliged and vulnerable. People who are vastly underrepresented in both EA and in wider STEM/academia, who are less able to collate and utilise resources, who are less able to raise alarms. As a result, AI Safety is a field where many of the current and future threats are hidden from view.

Secondly, AGI Safety as a field tends to isolate itself from other areas of AI Safety as if the two aren't massively related, and goes off on kind of a majorly theoretical angle as a result. As a consequence, AGI/ASI Safety folk are seen as something of living in a fantasy world of their own making compared to lots of other areas of AI risk by both the public and people within AI. I don't personally agree with this, but it's something I hear a lot in AI research.