EA Forum Bot Site
EA Forum

Hide table of contents
Comment Permalink
SiebeRozendal12
1
0

"Yet they carry on doing what they're doing, despite knowing the risks. Perhaps they're motivated by curiosity, hubris, wealth, fame, status, or prestige. (Aren't we all?) Perhaps these motives overwhelm their moral qualms about what they're doing."

I'd like to add inertia/comfort to this list: people don't like to change jobs, and changing fields is much harder.

Intervention idea: offer capabilities researchers support to transition out of the field

Reply
See in context
by Katja_Grace
3 min read 20

165

AI safetyForecastingAI forecastingPublic communication on AI safetySurveys
Frontpage
Show all topics

Katja Grace, 8 March 2023

In our survey last year, we asked publishing machine learning researchers how they would divide probability over the future impacts of high-level machine intelligence between five buckets ranging from ‘extremely good (e.g. rapid growth in human flourishing)’ to ‘extremely bad (e.g. human extinction).1 The median respondent put 5% on the worst bucket. But what does the whole distribution look like? Here is every person’s answer, lined up in order of probability on that worst bucket:

(Column widths may be distorted or columns may be missing due to limitation of chosen software.)

And here’s basically that again from the 2016 survey (though it looks like sorted slightly differently when optimism was equal), so you can see how things have changed:

Distribution from 2016 survey. (Column widths may be distorted or columns may be missing due to limitation of chosen software.)

The most notable change to me is the new big black bar of doom at the end: people who think extremely bad outcomes are at least 50% have gone from 3% of the population to 9% in six years.

Here are the overall areas dedicated to different scenarios in the 2022 graph (equivalent to averages):

  • Extremely good: 24%
  • On balance good: 26%
  • More or less neutral: 18%
  • On balance bad: 17%
  • Extremely bad: 14%

That is, between them, these researchers put 31% of their credence on AI making the world markedly worse.

Some things to keep in mind in looking at these:

  • If you hear ‘median 5%’ thrown around, that refers to how the researcher right in the middle of the opinion spectrum thinks there’s a 5% chance of extremely bad outcomes. (It does not mean, ‘about 5% of people expect extremely bad outcomes’, which would be much less alarming.) Nearly half of people are at ten percent or more.
  • The question illustrated above doesn’t ask about human extinction specifically, so you might wonder if ‘extremely bad’ includes a lot of scenarios less bad than human extinction. To check, we added two more questions in 2022 explicitly about ‘human extinction or similarly permanent and severe disempowerment of the human species’. For these, the median researcher also gave 5% and 10% answers. So my guess is that a lot of the extremely bad bucket in this question is pointing at human extinction levels of disaster.
  • You might wonder whether the respondents were selected for being worried about AI risk. We tried to mitigate that possibility by usually offering money for completing the survey ($50 for those in the final round, after some experimentation), and describing the topic in very broad terms in the invitation (e.g. not mentioning AI risk). Last survey we checked in more detail—see ‘Was our sample representative?’ in the paper on the 2016 survey.

Here’s the 2022 data again, but ordered by overall optimism-to-pessimism rather than probability of extremely bad outcomes specifically:

(Column widths may be distorted or columns may be missing due to limitation of chosen software.)

For more survey takeaways, see this blog post. For all the data we have put up on it so far, see this page.

See here for more details.

Thanks to Harlan Stewart for helping make these 2022 figures, Zach Stein-Perlman for generally getting this data in order, and Nathan Young for pointing out that figures like this would be good.

Notes

165

0
0

Reactions

0
0
Mentioned in

More posts like this

Comments20


Sorted by
New & upvoted
Click to highlight new comments since:
Geoffrey Miller67
25
5

Katja - thanks for posting these survey data. 

They results are shocking. Really shocking. Appalling, really. It's worth taking a few minutes to soak in the dark implications. 

It's hard to imagine any other industry full of smart people in which researchers themselves realize that what they're doing is barely likely to have a net positive impact on the world. And in which a large proportion believe that they're likely to impose massive suffering and catastrophe on everyone -- including their own friends, families, and kids.

Yet that's where we are with the AI industry. Almost all of the ML researchers seem to understand 'We might be the baddies'. And a much higher proportion seem to understand the catastrophic risks in 2022 than in 2016. 

Yet they carry on doing what they're doing, despite knowing the risks. Perhaps they're motivated by curiosity, hubris, wealth, fame, status, or prestige. (Aren't we all?) Perhaps these motives overwhelm their moral qualms about what they're doing.

But IMHO, any person with ethical integrity who found themselves working in an industry where the consensual prediction among their peers is that their work is fairly likely to lead straight to an extinction-level catastrophe would take a step back, re-assess, and re-think whether they should really be pushing ahead. 

I know all the arguments about the inevitability of AI arms races, between companies and between nation-states. But we're not really in a geopolitical arms race. There are very few countries with the talent, money, GPU clusters, and determination to pursue advanced AI. North Korea, Iran, Russia, and other dubious nations are not going to catch up any time soon. China is falling behind, relatively speaking. 

The few major players in the American AI industry are far, far more advanced than the companies in any other country at this point. We're really just talking about a few thousand ML researchers associated with OpenAI/Microsoft, Deepmind/Google, and a handful of other companies. Almost all American. Pushing ahead, knowing the risks, knowing they're far in advance of any other country. It's insane. It's sociopathic. And I don't understand why EAs are still bending over backwards to try to stay friendly with this industry, trying to gently nudge them into taking 'alignment' more seriously, trying to portray them as working for the greater good. They are the baddies, and they increasingly know it, and we know it, and we should call them out on it.

Sorry for the feisty tone here. But sometimes moral outrage is the appropriate response to morally outrageous behavior by a dangerous industry.

Reply
SiebeRozendal12
1
0

"Yet they carry on doing what they're doing, despite knowing the risks. Perhaps they're motivated by curiosity, hubris, wealth, fame, status, or prestige. (Aren't we all?) Perhaps these motives overwhelm their moral qualms about what they're doing."

I'd like to add inertia/comfort to this list: people don't like to change jobs, and changing fields is much harder.

Intervention idea: offer capabilities researchers support to transition out of the field

Reply
anonymous64
3
1

I think this would be counterproductive. Most ML researchers don't think the research they personally work on is dangerous, or could be dangerous, or contributes to a research direction that could be dangerous, and most of them actually are right about this. There is all kinds of stuff people work on that's not on the critical path to dangerous AI.

Reply
Geoffrey Miller21
6
0

Granted, moral outrage can sometimes be counterproductive. 

However, we have no idea which specific ML work is 'on the critical path to dangerous AI'. Maybe most of it isn't. But maybe most of it is, one way or another. 

ML researchers are clever enough to tell themselves reassuring stories about how whatever they're working on is unlikely to lead straight to dangerous AI.  Just as most scientists working on nuclear weapon systems during the Cold War could tell themselves stories like 'Sure, I'm working on ICBM rockets, but at least I'm not working on ICBM guidance systesm', or 'Sure, I'm working on guidance systems, but at least I'm not working on the nuclear payloads', or 'Sure, I'm working on simulating the nuclear payload yields, but at least I'm not physically loading the enriched uranium into the warheads'. The smarter people are, the better they tend to be at motivated reasoning, and at creating plausible deniability that they played any role in increasing existential risk.

However, there's no reason for the rest of us to trust individual ML researchers' assessments of which work is dangerous, versus which is safe. Clearly a large proportion of ML researchers think that what other ML researchers are doing is potentially dangerous. And maybe we should listen to them about that. 

Reply
anonymous65
2
0

I think a better analogy than "ICBM engineering" might be "all of aeronautical engineering and also some physicists studying fluid dynamics". If you were an anti-nuclear protester and you went and yelled at an engineer who runs wind tunnel simulations to design cars, they would see this as strange and unfair. This is true even though there might be some dual use where aerodynamics simulations are also important for designing nuclear missiles.

Reply
Geoffrey Miller12
3
0

I understand your point. But I think 'dual use' problems are likely to be very common in AI, just as human intelligence and creativity often have 'dual use' problems (e.g. Leonardo da Vinci creating beautiful art and also designing sadistic siege weapons). 

Of course AI researchers, computer scientists, tech entrepreneurs, etc may see any strong regulations or moral stigma against their field as 'strange and unfair'. So what? Given the global stakes, and given the reckless approach to AI development that they've taken so far, it's not clear that EAs should give all that much weight to what they think. They do not have some inalienable right to develop technologies that are X risks to our species.

Our allegiance, IMHO, should be to humanity in general, sentient life in general, and our future descendants. Our allegiance should not be to the American tech industry - no matter how generous some of its leaders and investors have been to EA as a movement. 

Reply
somedoomer1
0
3

The graphs above indicate there is no consensus in the industry that AI is on the road to inevitable catastrophe, as the graphs themselves above indicate. The median researcher still considers the field a net positive. But let's set that aside for now and talk only about AI doomers.

Most researchers predicting bad outcomes also have high confidence that they themselves exiting the industry would do nothing to prevent bad outcomes.  Somebody will get the bomb first. The day that person gets the bomb, unless you're a 100% doomer, humanity's chances are better the more "alignment-ready" the person who has it is, and if you had to pick an entity to receive the bomb today you could easily do worse than than OpenAI.  At least they would have the good sense to turn it off before deciding to do anything else dumb with it.

The "other actors aren't here yet" argument is literally just a punt.  They won't stay behind forever. When they catch up, we'll be in the exact same boat, except with the "overwhelming tech lead by institutions nominally pursuing alignment" solutions closed out of the book, a net loss in humanity's chances of survival however thin you think that slice is.

The notion that people simply haven't "taken a step back, reassessed, and rethought their position" ascribes a breathtaking amount of NPC-quality to people in the field, as if they somehow manage to believe in an impending apocalypse and still not spend any time thinking about it. The reason that doomers don't quit the industry themselves, a few deranged cultists aside, is the same reason people have never succeeded into baiting Yudkowsky into endorsing anti-AI terrorism: the impact you have on the final outcome is somewhere between net negative and net zero, as you will inevitably be replaced with another researcher who cares even less about alignment.

The only way quitting the industry actually leads to a better outcome is if you decide that the game is already 100% lost, you only want to maximize years of ignorant bliss before the fall, and you're certain that your competence is so high and your persuasive power so low relative to replacement that you personally quitting represents a net delay on the doom clock. The same argument applies, rescaled, to individual organizations themselves backing out of the game. There's a reason very few people think you can put the genie back in the bottle.

Reply
Kinoshita Yoshikazu (pseudonym)1
0
0

Do you think it's a serious enough issue to warrant some...not very polite responses? 

Maybe it would be better if policy makers just go and shut AI research down immediately instead of trying to make reforms and regulations to soften its impact? 

Maybe this information (that AI researchers themselves are increasingly pessimistic about the outcome) could sway public opinion enough to that point? 

Reply
Geoffrey Miller4
1
0

Just as anti-AI violence would be counter-productive, in terms of creating a public backlash against the violent anti-AI activists, I would bet (with only low-to-moderate confidence) that an authoritarian government crackdown on AI would also provoke a public backlash, especially among small-government conservatives, libertarians, and anti-police liberals.

I think public sentiment would need to tip against AI first, and then more serious regulations and prohibitions could follow. So, if we're concerned about AI X-risk, we'd need to get the public to morally stigmatize AI R&D first -- which I think would not be as hard as we expect.

Reply
Greg_Colbourn ⏸️ 5
1
0

I think we're at the stage now where we should be pushing for a global moratorium on AGI research. Getting the public on board morally stigmatizing it is an important part of this (cf. certain bio research like human genetic engineering).

Reply
Kinoshita Yoshikazu (pseudonym)3
1
0

I suspect that all three political groups (maybe not the libertarians) you mentioned could be convinced to turn collectively against AI research. Afterall, governmental capacity is probably the first thing that will benefit significantly from more powerful AIs, and that could be scary enough for ordinary people or even socialists. 

Perhaps the only guaranteed opposition for pausing AI research would come from the relevant corporations themselves (they are, of course, immensely powerful. But maybe they'll accept an end of this arms race anyway), their dependents, and maybe some sections of libertarians and progressives (but I doubt there are that many of them committed to supporting AI research). 

The public opinion is probably not very positive about AI research, but also perhaps a bit apathetic about what's happening. Maybe the information in this survey, properly presented in a news article or something, could rally some public support for AI restrictions.

Reply
D0TheMath2
0
0

Public sentiment is already mostly against AI when public sentiment has an opinion. Though its not a major political issue (yet) so people may not be thinking about it. If it turns into a major political issue (there are ways of regulating AI without turning it into a major political issue, and you probably want to do so), then it will probably become 50/50 due to what politics does to everything.

Reply
Ian Turner34
9
0

How should we think about the 17% response rate to this survey? Is it possible that researchers who are more concerned about alignment are also more likely to complete the survey?

Reply
Zach Stein-Perlman5
0
0

I'd guess very slightly, yes. I'm not aware of easy ways to get much evidence, unfortunately.

Reply
technicalities6
1
0

I dream of getting a couple questions added onto a big conference's attendee application form. But probably not possible unless you're incredibly well-connected.

Reply
Katja_Grace3
0
0

Possible, but likely a smaller effect than you might think because: a) I was very ambiguous about the subject matter until they were taking the survey (e.g. did not mention AGI or risk or timelines) b) Last time (for the 2016 survey) we checked the demographics of respondents against those for a random subset of non-respondents, and they weren't very different.

Participants were also mostly offered substantial payment for taking the survey ($50 usually, for a ~15m survey), in part in the hope of making payment a larger motivator than desire to express some particular view, but I don't think payment actually made a large difference to the response rate, so probably failed have the desired effect on possible response bias.

Reply
kokotajlod29
9
0

It would be cool to give this survey to top AI forecasters and see what the corresponding graphs look like. E.g. get superforecasters-who-have-forecast-on-AI-questions, or maybe top metaculus users, etc.

Reply
Sanjay14
5
0

I would find it fascinating to see this data for the oil and gas industry. I would guess that far fewer people in that industry think that their work is causing outcomes as bad as human extinction (presumably correctly), and yet they probably face more opprobrium for their work (at least from some left-leaning corners of the population).

Reply
Keerthana Gopalakrishnan12
4
5

There are a few obvious flaws in interpreting the survey as is:

  1. sample size and sampling bias: 13% respondents amount to 700 ish people surveyed. That is fairly small. Secondly, with MIRI and AI impacts's respective logos on the front page of the survey document, it introduces a bias into who is taking the survey, very likely it is just the people familiar with lesswrong etc who has heard of these organizations. Here's why a lot of serious AI researchers don't engage with MIRI et al:
    1. MIRI hasn't shipped a SoTA in AI alignment or AI research in the last 5-6 years
    2. A quick look at the publications on their website shows they don't publish at top ML conferences ( ICML, ICLR, NeurIPS, etc). Less wrong "research" is not research, it is usually philosophy and not backed by performing experiments ( "thought experiments" don't count). 
  2. appeal to authority fallacy: a lot of people saying something doesn't make it true, so I'd advise people to not confuse between "AI research is bad" and "a proportion of surveyed people think AI research could be bad". Some moral outrage in the comment section takes some people feeling this way as evidence for defacto truths, while the reality is that they are contested claims.  
  3. modeling the future: Human beings are notoriously bad at modeling the future. Imagine if we ran a survey in Oct among EAs about FTX health. Not just that modeling the future is hard, but modeling the far future is exponentially harder and existential risk analyses are often incomplete because:
    1. New improvements in AI safety research is not accounted in these projections
    2. Multiagent dynamics of a world with multiple AIs is not modeled in catastrophic scenario projections
    3. Multiagent dynamics with governments/stakeholders is not modeled
    4. Phased deployment: accelerate, then align to use case, then accelerate again, as we are doing today is also not modeled. AI deployment is currently also accelerating alignment research because alignment is needed to build a useful product- a gaslighting chatbot is a bad product compared to a harmless, helpful one.
    5. New research produces new knowledge previously not known.

Anthropic's write up, afaik, is a nuanced take and may be a reasonable starting point for an informed and calibrated take towards AI research. 
I usually don’t engage with AI takes here because it is a huge echo chamber in here but these are my two cents!

Reply
akash 🔸2
0
0

Disagree-voting for the following reasons:

  1. 700 people — that is a substantial sample size. 
  2. I think sampling bias will likely be small/inconsequential, given the sample size. 
    1. Ideally, yes, not having any logos would be great, but if I got an email asking me, "fill out this survey, and trust me, I am from a legitimate organization," I probably wouldn't fill out the survey.
    2. "very likely it is just the people familiar with lesswrong etc who has heard of these organizations" 
      1. Hard to say if this is true
      2. It is wrong to assume that familiarity with LW would lead people to answer a certain way. However, I can imagine people who dislike LW/similar would also be keen to complete the survey.
      3. The survey questions don't seem to prime respondents one way or the other.
      4. These 700 people did publish in ICML/NeurIPS; they have some degree of legitimacy.
  3. Nitpicky, but "a proportion of ML researchers who have published in top conferences such as ICML and NeurIPS think AI research could be bad" is probably a more accurate statement. I agree that this doesn't make the statements they are commenting on = truth; however, I think their opinion has a lot of value.

I think Metaculus does a decent job at forecasting, and their forecasts are updated based on recent advances, but yes, predicting the future is hard.

Reply
More from Katja_Grace
339
Let’s think about slowing down AI
Katja_Grace
·
9
9
286
Counterarguments to the basic AI risk case
Katja_Grace
· · 41m read
23
23
176
Survey of 2,778 AI authors: six parts in pictures
Katja_Grace
·
11
11
View more
Curated and popular this week
 ·  · 6m read
 · 
I am writing this to reflect on my experience interning with the Fish Welfare Initiative, and to provide my thoughts on why more students looking to build EA experience should do something similar.  Back in October, I cold-emailed the Fish Welfare Initiative (FWI) with my resume and a short cover letter expressing interest in an unpaid in-person internship in the summer of 2025. I figured I had a better chance of getting an internship by building my own door than competing with hundreds of others to squeeze through an existing door, and the opportunity to travel to India carried strong appeal. Haven, the Executive Director of FWI, set up a call with me that mostly consisted of him listing all the challenges of living in rural India — 110° F temperatures, electricity outages, lack of entertainment… When I didn’t seem deterred, he offered me an internship.  I stayed with FWI for one month. By rotating through the different teams, I completed a wide range of tasks:  * Made ~20 visits to fish farms * Wrote a recommendation on next steps for FWI’s stunning project * Conducted data analysis in Python on the efficacy of the Alliance for Responsible Aquaculture’s corrective actions * Received training in water quality testing methods * Created charts in Tableau for a webinar presentation * Brainstormed and implemented office improvements  I wasn’t able to drive myself around in India, so I rode on the back of a coworker’s motorbike to commute. FWI provided me with my own bedroom in a company-owned flat. Sometimes Haven and I would cook together at the residence, talking for hours over a chopping board and our metal plates about war, family, or effective altruism. Other times I would eat at restaurants or street food booths with my Indian coworkers. Excluding flights, I spent less than $100 USD in total. I covered all costs, including international transportation, through the Summer in South Asia Fellowship, which provides funding for University of Michigan under
 ·  · 7m read
 · 
My new book, Altruismo racional, is now on presale. It is my attempt at presenting a compelling case for a particular strand of "classical EA"[1]: one that emphasizes caring deeply about global health and poverty, a rational approach to giving, the importance of cost-effectiveness, and the 🔸10% Pledge. In this post, I provide some context on my reasons for writing this book and what I hope to achieve. If “new EA-themed book in Spanish” was all you needed to know, feel free to skip to How you can help or preorder now. Why write a book Imagine you wake up one morning and discover the world has changed in a few peculiar ways. There has been no 10th anniversary edition of Peter Singer's The Life You Can Save—it was last edited more than a decade ago and has been out of print for years. Will MacAskill has not written Doing Good Better nor any of his pieces for The Guardian. And that’s not all. You ask around about EA and get mostly confused looks. Someone mentions a blog called “Codice Stellare Something” that later changed its name. You look it up but it's written in some foreign language that's hard to understand. “Toby who?” He seems to be associated with something called Geben Was Wir Können that you cannot pronounce, let alone remember. Welcome to Spain—or, I dare say, the Spanish-speaking world—where language friction[2] curbs the potential of most of the ways people first hear about EA. This is true for many other topics, of course. In Spain, people usually don't hear directly from those doing cutting-edge work in the English-speaking world, but rather from local explainers or commentators. Top non-fiction books like Sapiens or Antifragile are read, overwhelmingly, in translation. I have been close to some attempts to translate key EA-themed books into Spanish. The problem? Publishers are quite uninterested because only a handful of English-speaking public intellectuals have the global name recognition to guarantee sales. The Scout Mindset and What We Owe T
 ·  · 3m read
 · 
Summary * 🤰🏽 A randomized study in Kenya found that giving families a $1,000 GiveDirectly cash transfer immediately cut infant deaths by 48%. → * 🏥 Deaths dropped most for mothers living near physician-staffed health facilities and those who received cash in the weeks before they gave birth. → * 🩺 GiveDirectly is launching a new program to maximize these life-saving impacts, partnering with a Kenyan community health provider to get cash to more expectant mothers. → ---------------------------------------- In rural Kenya, giving poor families a one-time $1,000 transfer cut infant deaths nearly in half, one of the largest reductions ever recorded for a poverty program. With global aid budgets shrinking and funders under pressure to do more with less, the findings point to cash as a powerful, underused tool to reduce preventable deaths. Cash cut infant deaths by 48% The study, led by researchers at UC Berkeley and Oxford, tracked over 100,000 births and found that infant mortality fell by 48% in the years families received a one-time $1,000 transfer from GiveDirectly. It’s a striking result, reinforcing what decades of research have already shown: poverty itself is one of the biggest risk factors for a child’s survival. As the researchers note, “infant and child mortality appears highly sensitive to economic conditions.” Cash saved lives by helping new moms rest, eat, and deliver safely The biggest gains were among newborns: deaths within the first 30 days of a baby’s life fell by 63%, with drops in maternal and newborn causes of death accounting for more than half of the overall decline in infant deaths.  This drop was driven by a 45% increase in hospital deliveries and a 51% drop in work (often physically strenuous) during the third trimester and postpartum. Less work in late pregnancy coincided with fewer deaths from complications at birth.  When mothers have cash, they get care, proving that sometimes, the best way to save a child’s life is to simp
Recent opportunities in AI safety
33
Foresight Institute launches two possible future scenarios with AI
elteerkers
· · 3m read
2
2
6
AI Researcher Surveys with Similar Results to Katja Grace, 2024?
Q
AlexChalk
· · 1m read
1
1
7
Research Engineer @ Timaeus
Sara Recktenwald
· · 3m read
1
1
View more