EA Forum Bot Site
EA Forum

Take the 2025 EA Forum Survey to help inform our strategy and prioritiesTake the survey
Hide table of contents
Comment Permalink
cdkg2
0
0

The object-level argument, as I understand it, is that worries about human-level AI capabilities of the sort that could pose an existential threat are based on a misunderstanding of what is going on under the hood in neural networks. This is what Bender means when she talks about "AI Hype". See for example her paper with Koller "Climbing towards NLU" for criticisms of attributing some kinds of mental states to neural networks.

Reply
See in context
by Peter Berggren
3 min read 19

0

AI safetyExistential riskPhilosophyBounty (closed)
Frontpage

UPDATE: The phrasing of this post has caused some confusion. I would like to say that I am not at all confident as to whether the people in question have written up their object-level arguments before, and do not mean to imply either that they have or that they haven't. All I'm saying is, in my examination of their work, I have yet to find them. By offering this bounty, I do not mean to devalue the time of anyone by insisting on a new, extensive response; linking to a pre-existing object-level argument, as Torres did, would be more than sufficient. I'm mostly doing this for my own sake, so that I can familiarize myself with their object-level arguments.

I am prepared to pay out anywhere between $20 and $100 to AI ethicists of the DAIR/"Stochastic Parrots" school of thought if they provide their object-level arguments against the idea that preventing AI from killing everyone is a real and important issue. This pay will depend on their notability within AI ethics, as well as the clarity and persuasiveness of their arguments.

Conditions for the bounty

  1. The bounty must be claimed by an AI ethicist of the DAIR/"Stochastic Parrots" school of thought. Ethicists from other schools of thought (such as the "what if self-driving cars face trolley problems" school of thought) may be given bounties on a case-by-case basis, but probably not. Any member of DAIR or coauthor of the "Stochastic Parrots" paper counts for this, but people outside of these specific circles may qualify at my discretion, if I believe that their intellectual output is similar to or connected with DAIR or the "Stochastic Parrots" coauthors.
  2. The arguments provided by the claimant must be posted publicly, ideally in the comment section of this thread (or in the comment section of the corresponding LessWrong thread: https://www.lesswrong.com/posts/uTRafHCcjNfbAByyo/bounty-available-ai-ethicists-what-are-your-object-level).
  3. The arguments provided by the claimant must be object-level. This means that they must discuss concrete subjects specific to the issues at hand. This is in contrast to meta-level arguments, which focus on facts about the question (rather than about the issues it addresses), such as difficulties involved in future prediction, the cultural milieu of contemporary AI notkilleveryoneism, the framing of my questions, etc. Note that I have nothing against meta-level arguments; it's just that I've already seen plenty of meta-level arguments by AI ethicists against AI notkilleveryoneism, and I want to see some object-level arguments.
  4. The arguments provided by the claimant must be a good-faith summary of the claimant's actual object-level arguments against AI notkilleveryoneism. For example, "AI notkilleveryoneism is unimportant because paperclips are shiny" will not count, even if made by a qualifying claimant, even though it is object-level. I do not expect that I will need to invoke this condition, but I may do so at my discretion.
  5. The following AI ethicists will be presumptively considered valid claimants, and will fall into the most notable category (meaning that I will pay each of them the maximum $100 bounty assuming they follow all the terms of the bounty, unless I notice loophole abuse):
    Emily Bender
    Timnit Gebru
    Margaret Mitchell
    Melanie Mitchell

Note that there is no requirement for the arguments to change my mind, or even to be persuasive in the slightest. The only requirements are the above ones. If someone manages to abuse a loophole to get there, I will pay them the minimum bounty of $20, and then modify the rules for all future claimants to preempt this loophole.

So far, Emile Torres has already responded to the bounty (to my understanding, they believe that AI extinction risk is real, but that the field of AI notkilleveryoneism is broken beyond repair) by recommending their book as the place where their object-level arguments have been written. I will judge this as soon as I am able to check this book out from a library near me.

Note that I may need to close this bounty if I get too many claims from it, because I have a limited budget. All the more reason to get your arguments in here soon!

0

0
0

Reactions

0
0

More posts like this

Comments19


Sorted by
New & upvoted
Click to highlight new comments since:
Devin Kalish14
3
1

This is a quick PSA, Emile Torres does think “Preventing AI from killing everyone is a real and important issue”. The last time this was pointed out to you (that I’m aware of) you clarified that Torres’ disagreement was basically with longtermism. Please, pleeease clarify this in the post, it isn’t remotely how this challenge comes off and is borderline spreading misinformation, which is especially bad for important coalition building.

Reply
Peter Berggren3
0
0

I didn't mean to imply that Emile Torres didn't think that this was an extinction risk. I'm sorry that I misspoke on that part.

Reply
Devin Kalish4
1
0

Thanks for changing it.

Reply
cdkg10
5
1

Just FYI, many people in the AI ethics community find this kind of thing offensive. They have published their arguments in numerous scholarly venues and also in major newspapers and magazines and on places like Medium and Twitter. This kind of post is interpreted as "I'm too lazy to look at your work to find your arguments but I bet I can make you dance with small sums of money." Bad optics.

Reply
Geoffrey Miller10
5
1

Many people in the AI ethics community seem to find almost everything offensive. 

I've seen no evidence that they're worth engaging with on the topic of AI X-risk. They routinely caricature and demonize AI Safety researchers and EAs on social media, they seem not to have read any of the key works on AI X-risk, and their epistemic standards seem very weak.

While I admire Peter Breggren's attempt to entice them to engage in object-level objections to AI Safety research, I very much doubt that they will engage in any serious discussion of this issue.

Reply
cdkg4
0
0

Well, you can dismiss them and their argument if you want to — I personally don't find their arguments terribly convincing, and their social media presence is, as you point out, strident.

But one must be aware that to a surprising extent, they control the narrative about AI safety in academia and the mainstream media. So if one cares about making AI safety seem credible, it's worth engaging with them.

Reply
Peter Berggren1
0
0

Do they really control the narrative in the "mainstream media," though, or just a few far-left content mills that tend to get clicks by being really outrageous?

Reply
Peter Berggren3
0
0

I never denied that they have published their arguments in many places. I just can't find any such arguments that are object-level.

Reply
cdkg2
0
0

The object-level argument, as I understand it, is that worries about human-level AI capabilities of the sort that could pose an existential threat are based on a misunderstanding of what is going on under the hood in neural networks. This is what Bender means when she talks about "AI Hype". See for example her paper with Koller "Climbing towards NLU" for criticisms of attributing some kinds of mental states to neural networks.

Reply
Larks8
2
0

The paper you mentioned doesn't seem to discuss existential risk or AGI at all, so I don't see how it could represent the sort of object-level argument against existential risk that Peter is asking for.

Reply
cdkg4
2
1

Have a little imagination.

Suppose I am very worried that ghosts will steal things out of my closet. It seems like a perfectly object-level argument against my position to provide reasons for thinking that beliefs in paranormal activity are not scientifically respectable. This can be true even if the reasons provided do not mention ghosts.

People like Bender take themselves to be offering reasons for thinking that worries about AGI are not scientifically respectable. This can be true even if the reasons they provide do not mention AGI.

Note that I think Bender's arguments are bad. But I don't see what is so mysterious about them.

Reply
Peter Berggren3
1
1

It seems to me that, while the form/meaning distinction in this paper is certainly a fascinating one if your interests tend towards philosophy of language, this has very little to say about supposed inherent limitations of language models, and does not affect forecasts of existential risk.

Reply
cdkg4
0
0

Ok, let me spell it out explicitly. In a section called "Large LMs: Hype and analysis," the linked paper says that claims that LLM can "understand," "comprehend," and "know" are "gross overclaims." The paper supports this contention by pointing to evidence that "in fact, far from doing the “reasoning” ostensibly required to complete the tasks, [LLMs] were instead simply more effective at leveraging artifacts in the data than previous approaches."

Here is where the imagination comes in. Imagine that you think that all mental state attributions to artificial systems are confused in exactly this way. Imagine that you think that artificial neural nets can't reason at all. Now imagine that someone tells you that we should all be very concerned that misaligned superintelligent AI systems will destroy us.

Your response to that would be something like: it is deeply confused to think that superintelligent AI systems are something we need to worry about, and the people who are worried about them simply do not understand what is going on under the hood in machine learning models. Worries about existential risk from superintelligent AI stem from the same kind of confusion as attributing understanding to existing systems: the tendency of people who are not technically literate to anthropomorphize the systems they interact with.

Reply
Linch2
0
0

Imagine that you think that artificial neural nets can't reason at all.

Is this a real position that real living intelligent people actually hold, or is it just one of the funny contrarian philosopher beliefs that some philosophers like to around with for fun?

Reply
zchuang2
1
0

I think this is an actual position. It's the stochastic parrots argument no? Just a recent post by a cognitive scientist holds this belief.

Reply
Linch2
0
1

I don't think there were any factual claims in that article from a skim; entirely just normative claims and a few rhetorical question.

Reply
cdkg2
0
0

I think this is really the position of the stochastic parrots people, yes.

I don't think it's plausible, but I think it partly explains their relentless opposition to work on AI safety.

Reply
Peter Berggren2
2
1

As an aside, the idea that we should prioritize optics over intellectually honest exploration of the epistemic landscape is deeply harmful to effective altruism as a whole.

Reply
cdkg5
4
1

I didn't endorse that idea and, as an academic, obviously wouldn't. Also as an academic, I think paying people to explain themselves to you when you haven't first shown that you have read their work by e.g. explaining why you don't find the arguments they have already made in print convincing is not a shining exemplar of intellectually honest exploration.

Reply
More from Peter Berggren
View more
Curated and popular this week
 ·  · 1m read
 · 
This morning I was looking into Switzerland's new animal welfare labelling law. I was going through the list of abuses that are now required to be documented on labels, and one of them made me do a double-take: "Frogs: Leg removal without anaesthesia."  This confused me. Why are we talking about anaesthesia? Shouldn't the frogs be dead before having their legs removed? It turns out the answer is no; standard industry practice is to cut their legs off while they are fully conscious. They remain alive and responsive for up to 15 minutes afterward. As far as I can tell, there are zero welfare regulations in any major producing country. The scientific evidence for frog sentience is robust - they have nociceptors, opioid receptors, demonstrate pain avoidance learning, and show cognitive abilities including spatial mapping and rule-based learning.  It's hard to find data on the scale of this issue, but estimates put the order of magnitude at billions of frogs annually. I could not find any organisations working directly on frog welfare interventions.  Here are the organizations I found that come closest: * Animal Welfare Institute has documented the issue and published reports, but their focus appears more on the ecological impact and population decline rather than welfare reforms * PETA has conducted investigations and released footage, but their approach is typically to advocate for complete elimination of the practice rather than welfare improvements * Pro Wildlife, Defenders of Wildlife focus on conservation and sustainability rather than welfare standards This issue seems tractable. There is scientific research on humane euthanasia methods for amphibians, but this research is primarily for laboratory settings rather than commercial operations. The EU imports the majority of traded frog legs through just a few countries such as Indonesia and Vietnam, creating clear policy leverage points. A major retailer (Carrefour) just stopped selling frog legs after welfar
 ·  · 10m read
 · 
This is a cross post written by Andy Masley, not me. I found it really interesting and wanted to see what EAs/rationalists thought of his arguments.  This post was inspired by similar posts by Tyler Cowen and Fergus McCullough. My argument is that while most drinkers are unlikely to be harmed by alcohol, alcohol is drastically harming so many people that we should denormalize alcohol and avoid funding the alcohol industry, and the best way to do that is to stop drinking. This post is not meant to be an objective cost-benefit analysis of alcohol. I may be missing hard-to-measure benefits of alcohol for individuals and societies. My goal here is to highlight specific blindspots a lot of people have to the negative impacts of alcohol, which personally convinced me to stop drinking, but I do not want to imply that this is a fully objective analysis. It seems very hard to create a true cost-benefit analysis, so we each have to make decisions about alcohol given limited information. I’ve never had problems with alcohol. It’s been a fun part of my life and my friends’ lives. I never expected to stop drinking or to write this post. Before I read more about it, I thought of alcohol like junk food: something fun that does not harm most people, but that a few people are moderately harmed by. I thought of alcoholism, like overeating junk food, as a problem of personal responsibility: it’s the addict’s job (along with their friends, family, and doctors) to fix it, rather than the job of everyday consumers. Now I think of alcohol more like tobacco: many people use it without harming themselves, but so many people are being drastically harmed by it (especially and disproportionately the most vulnerable people in society) that everyone has a responsibility to denormalize it. You are not likely to be harmed by alcohol. The average drinker probably suffers few if any negative effects. My argument is about how our collective decision to drink affects other people. This post is not
 ·  · 5m read
 · 
Today, Forethought and I are releasing an essay series called Better Futures, here.[1] It’s been something like eight years in the making, so I’m pretty happy it’s finally out! It asks: when looking to the future, should we focus on surviving, or on flourishing? In practice at least, future-oriented altruists tend to focus on ensuring we survive (or are not permanently disempowered by some valueless AIs). But maybe we should focus on future flourishing, instead.  Why?  Well, even if we survive, we probably just get a future that’s a small fraction as good as it could have been. We could, instead, try to help guide society to be on track to a truly wonderful future.    That is, I think there’s more at stake when it comes to flourishing than when it comes to survival. So maybe that should be our main focus. The whole essay series is out today. But I’ll post summaries of each essay over the course of the next couple of weeks. And the first episode of Forethought’s video podcast is on the topic, and out now, too. The first essay is Introducing Better Futures: along with the supplement, it gives the basic case for focusing on trying to make the future wonderful, rather than just ensuring we get any ok future at all. It’s based on a simple two-factor model: that the value of the future is the product of our chance of “Surviving” and of the value of the future, if we do Survive, i.e. our “Flourishing”.  (“not-Surviving”, here, means anything that locks us into a near-0 value future in the near-term: extinction from a bio-catastrophe counts but if valueless superintelligence disempowers us without causing human extinction, that counts, too. I think this is how “existential catastrophe” is often used in practice.) The key thought is: maybe we’re closer to the “ceiling” on Survival than we are to the “ceiling” of Flourishing.  Most people (though not everyone) thinks we’re much more likely than not to Survive this century.  Metaculus puts *extinction* risk at about 4
Recent opportunities in AI safety
17
[Event] Building What the Future Needs: A curated conference in Berlin (Sep 6, 2025) for high-impact builders and researchers
Vasiliy Kondyrev, Daniel Ajudeonu, joechr+ 2 more
· · 2m read
0
0
18
Is Optimal Reflection Competitive with Extinction Risk Reduction? - Requesting Reviewers
Jordan Arel
· · 13m read
1
1
4
Some AI safety project & research ideas/questions for short and long timelines
Lloy2 🔹
· · 6m read
0
0
View more