Anti-'FOOM' (stop trying to make your cute pet name the thing)

david_reinstein

Anti-'FOOM' (stop trying to make your cute pet name the thing)

Comments

Anti-'FOOM' (stop trying to make your cute pet name the thing) — EA Forum

The term 'foom' was coined by artificial intelligence researcher and author Eliezer Yudkowsky in his 2008 book titled "The Sequences".

Flagging that GPT-4 is making things up here. The book didn't come out until much later, maybe 2015, the term predates the book, and I'm not sure whether Yudkowsky coined it.

The earliest usage I've found is this 2008-11-19 Robin Hanson post in what was retroactively designated the "AI FOOM Debate".

Larks

I'm not exactly sure who you're arguing with, or what the criteria is for a name to be cute, but I can think of a lot of cases which seem like to me like counter-examples, where a silly name was taken seriously:

'Brexit', if I recall correctly, was originally a joking play on 'Grexit', back when the Lib Dems were the only major party supporting a referendum, but went on to be taken very seriously.
'YIMBY' is a silly name based on another silly name but has done a fairly good job building a coalition and getting policies enacted even in the face of severe public choice problems.
Refering to HB 1557 as the 'Don't Say Gay' bill has been quite effective at getting respectable people to oppose it despite the phrase itself being silly.
#MeToo and #BelieveWomen were quite successful movements for a while despite the silly names.
'RINO' has been used as an effective attack line against moderate republicans.
'Junk Bonds' were (probably?) so named by the guy who popularized them.
'Unicorn' status is a prestigious thing for a startup to aspire to.

Also, I don't understand your 'woke' example, though you say it "basically speaks for itself." I think activists have been hugely successful with the name, using it to highlight a threat and gather an international coalition of politicians, academics and voters to push back against it. A few years ago woke stuff seemed irresistible, but using the label has now taken us to a place where some people can earnestly suggest that 'woke is dying' - though I think that is an overstatement. The fact that the name is a bit silly hasn't prevented it from being a potent rhetorical device for movement building.

david_reinstein

Some pretty good examples.

But I would counter that I'm not sure all of these are as 'inside-jokey' as "FOOM".

Brexit and YIMBY, and RINO are all actual acronyms or portmanteau's, unlike FOOM.

I maintain that 'Woke' shot itself in the foot; I feel like this label really alienated people in several ways. (Sense of superiority, use of nonstandard English slang). and maybe it did catch on a bit but the silliness of the name made it more mockable. But it also

"Don't say Gay" is not the movement name.

MeToo, ok that is close but I don't see it as a trivial fun sounding 'inside reference' someone made up; it seems like a pretty meaningful rallying cry. "BelieveWomen" is pretty straightforward, not a funny word.

Junk bonds, Unicorns: these are not movements.

Larks

I think these are closer examples than you think. 'Don't Say Gay', 'Junk bonds', and 'Unicorns' aren't movements, but neither is 'FOOM' - in each case they are a cute name for a thing that people want to draw attention to and have other people take seriously. Similarly, I think you have the woke example backwards - the term has been successfully developed by anti-woke people to draw attention to the dangers of the woke movement and have people take the threat seriously (not just a silly thing on college campuses) and gather a coalition to oppose it, in the same way that we are trying to draw attention to the dangers of FOOM as a serious thing (not just a silly science fiction thing).

david_reinstein

True, foom is not the movement but it’s the serious outcome they are trying to get ppl to bw concerned about.

But I think “woke” was in fact a term developed by the ppl espousing it, not by the anti-woke. So I think this might be evidence that flippability of a silly name might be an argument against using it.

Habryka [Deactivated]

I am not really sure what this post is arguing against, or who it is arguing against. I feel like it just kind of implies that "cute pet name"s are low-status and therefore memetically unviable, which seems dubious to me (especially in the internet age people use all kinds of cute abbreviations).

I haven't seen anyone actively advocate for using "foom" and I've most recently only seen it used by people not actually very central to the EA, Rationality or AI Alignment communities. I don't think anyone is "trying to make it happen", it's just like the current most natural term we have for it. I don't think anyone is super attached to it, and you can just propose a different name and then people might go with that one, or they might not, forecasting the adoption of terms is quite hard and I haven't seen people have a lot of success with it.

david_reinstein

it's just like the current most natural term we have for it. I don't think anyone is super attached to it, and you can just propose a different name and then people might go with that one,

It doesn't seem natural at all to me. How about, e.g., "AI explosion", "Runaway AI", "AI-apocalypse", or "catastrophic AI"? instead?

Forecasting the adoption of terms is quite hard and I haven't seen people have a lot of success with it.

This is not quite about 'forecasting the adoption of terms' (not that you were explicitly making this case. It might be that FOOM does get adopted, but then used as a term to dismiss these concerns. While I think another term might be just as easily adopted but lead to greater credibility and sympathy.

RobertM

None of those obviously mean the same thing ("runaway AI" might sort of gesture at it, but it's still pretty ambiguous). Intelligence explosion is the thing it's pointing at, though I think there are still a bunch of conflated connotations that don't necessarily make sense as a single package.

I think "hard takeoff" is better if you're talking about the high-level "thing that might happen", and "recursive self improvement" is much clearer if you're talking about the usually-implied mechanism by which you expect hard takeoff.

Greg_Colbourn ⏸️

I use "AI Apocalypse" when talking about this to non-EA/LW friends and family. Didn't really explicitly think about it, it was just the most natural choice of words in the context.

david_reinstein

Will add some clarifications. I don't think that these cute pet names are low status. To me it seems almost the opposite, it feels like it borders on

if you are in the elite in-crowd you can use a silly term, counter-signaling that you are above formality
because this is 'our genius idea' we get to name it.

But maybe that's just my reaction

Habryka [Deactivated]

Hmm, yeah, I do think there is a countersignaling thing going on a bit with those terms, though I do find it hard to forecast whether that will help or harm the term overall. The modern internet and public opinion is a really weird place. Like, take the term MAGA, which sure took off and was really successful and feels quite similar to "foom".

david_reinstein

I agree MAGA is pretty cheesy, was going to mention it. OTOH it is a legit acronym. It also doesn't have the same sort of counter-signaling feel to it as FOOM, to my ears. There's no connotation that those people are elites (the opposite, obviously).

david_reinstein

I am not really sure what this post is arguing against, or who it is arguing against

Tbh reacting initially to this Twitter poll seen by over 20,000 people. But it's used fairly widely in my experience.

I haven't seen anyone actively advocate for using "foom" and I've most recently only seen it used by people not actually very central to the EA, Rationality or AI Alignment communities.

Fwiw it seems to be used many many times on this forum

It's in the title of an [e-book promoted by MIRI](The Hanson-Yudkowsky AI-Foom Debate eBook )

Yudkowski uses the term in a recent podcast interview

But, to be fair, he didn't use the term in his Time Magazine interview

titotal

It feels like "intillegence explosion" means pretty much the same thing and is strictly better in terms of readability and getting your point across.

Jeff Kaufman 🔸

Another example: 'AI-not-kill-everyone-ism

I don't think anyone using the term is happy with it, and I do think they'd welcome a better term. [1] But in evaluating the choice to use it the history matters, which is that all the previous terms keep getting widely misunderstood to mean something different.

[1] ex: "getting just the relatively small idea of ainotkilleveryoneism (need a better name; more memetics) into the general population" (LW)

Juan Gil

I think "fast takeoff" and "intelligence explosion" mean approximately the same thing as FOOM (notably "catastrophic AI" refers to a broader category of scenarios), and these terms are often used especially in more formal contexts.

I'm not concerned about this being a big problem, but do think this post is a good nudge for people who don't typically think about the effect their language has on getting buy-in for their ideas.

Greg_Colbourn ⏸️

Note that foom is in Wiktionary (with references to appearances in books by Stephen Pinker and Robin Hanson, and a paper by Brian Tomasik). How long before official dictionaries pick it up?

AI safety

Community

Building effective altruism

Effective altruism messaging

Criticism of work in effective altruism

Frontpage

Notes/basis: This is kind of a short-form post in style but I think it's important enough to put here. Obviously let me know if someone else said this better

Summary

Formal overly-intellectual academese is bad. But using your 'cute' inside joke name for things is potentially worse. It makes people cringe, sounds like you are trying to take ownership of something, and excludes people. Use a name that is approachable but serious.

The problem.

Where did the term 'FOOM' come from, to refer to AGI risk? I asked GPT4:

[!ai] AI The term 'foom' was coined by artificial intelligence researcher and author Eliezer Yudkowsky in his 2008 book titled "The Sequences". Yudkowsky used the term to refer to a hypothetical scenario where an artificial general intelligence (AGI) rapidly and exponentially improves its own intelligence, leading to an uncontrollable and potentially catastrophic outcome for humanity. The term 'foom' is a play on the word 'boom', representing the sudden and explosive nature of AGI development in this scenario.

Another example: 'AI-not-kill-everyone-ism'

Analogies to fairly successful movements:

Global warming was not called ''Roast", and the movement was not called "anti-everyone-burns-up-ism"
Nuclear holocaust was not called "mega-boom"
Anti-slavery was not called ... (OK I won't touch this one)

How well has the use of cute names worked in the past?

I can't think of any examples where they have caught on in a positive way. The closest I can think of are

"Nudge" (by Richard Thaler?)

... to describe choice-architecture interventions; - My impression is that the term 'nudge' got people to remember it but made it rather easy to dismiss - others in that space have come up with names that caught on less well I think (like "sludge"), which also induce a bit of cringe

"Woke"

I think this example basically speaks for itself.

Tea-Party movement

This goes in the opposite direction perhaps (fairly successful), but I still think it's not quite as cringeworthy as FOOM. The term 'tea party' obviously has a long history in our culture, especially the "Boston Tea Party.

What else?

I asked GPT4

when have social movements used cute 'inside joke' names to refer to the threats faced?

The suggestions are not half as cute or in-jokey as FOOM: Net Neutrality, The Umbrella Movement, Extinction Rebellion (XR), Occupy Wall Street (OWS)

I asked it to get cuter... ^[1]

Prodding it further... Climategate, Frankenfoods, Slacktivism ... also not so inside-jokey nor as cringeworthy IMO.

Prodding it for more cutesy more inside-jokey yields a few terms that barely caught on, or didn't characterize the movement or the threat as a whole.^[2]

the best it came up with was "Treehuggers: This term, originally used pejoratively to describe environmental activists, has been reclaimed by the environmental movement as a playful and positive label for those who care about protecting nature and the planet." But this goes in the opposite direction: reclaiming a term, not pushing it forward. It also suggested Ploggers, Plarn, Solartopia, and Ecotarians. All either not as cringey, or not something I've ever heard of. ↩︎
"Nurdles: In environmental activism, "nurdles" are small plastic pellets used as raw material in the production of plastic products." OK but I've never heard of these, and they are not the focus of the movement. " "Vaxxies": This playful term is a blend of "vaccine" and "selfie" ... way to celebrate and promote vaccination efforts, turning the act of getting vaccinated into a social media moment. " ↩︎