Oh woops didn't look at parent comment, haah

I think the vast majority of people making decisions about public policy or who to vote for either aren't ethically impartial, or they're "spotlighting", as you put it. I expect the kind of bracketing I'd endorse upon reflection to look pretty different from such decision-making.

But suppose I want to know who of two candidates to vote for, and I'd like to incorporate impartial ethics into that decision. What do I do then?

That said, maybe you're thinking of this point I mentioned to you on a call

Hmm, I don't recall this; another Eli perhaps? : )

(vibesy post)

People often want to be part of something bigger than themselves. At least for a lot of people this is pre-theoretic. Personally, I've felt this since I was little: to spend my whole life satisfying the particular desires of the particular person I happened to be born into the body of, seemed pointless and uninteresting.

I knew I wanted "something bigger" even when I was young (e.g. 13 years old). Around this age my dream was to be a novelist. This isn't a kind of desire people would generally call "altruistic," nor would my younger self have called it "altruistic." But it was certainly grounded in a desire for my life to mean something to other people. Stuff like the Discworld series and Watchmen really meant something to me, and I wanted to write stuff that meant something to others in the same way.

My current dreams and worldview, after ~10 years of escalating involvement with EA, seem to me to spring from the same seed. I feel continuous with my much younger self. I want my life to mean something to others: that is the obvious yardstick. I want to be doing the most I can on that front.

The empirics were the surprising part. It turns out that the "basic shape of the world" is much more mutable than my younger self thought, and in light of this my earlier dreams seem extremely unambitious. Astonishingly, I can probably:

• save many lives over my career at minimum, by donating to GiveWell, and likely more by doing more off the beaten path things
• save <large number> of e.g. chickens from lives full of torture
• be part of a pretty small set of people seriously trying to do something about truly wild risks from new AI and bioengineering technologies

It probably matters more to others that they are not tortured, or dying of malaria, or suffering some kind of AI catastrophe, than that there is another good book for them to read, especially given there are already a lot of good novelists. The seed of the impulse is the same — wanting to be part of something bigger, wanting to live for my effect on others and not just myself. My sense of what is truly out there in the world and of what I can do about it are what's changed.

Like if you're contemplating running a fellowship program for AI interested people, and you have animals in your moral circle, you're going to have to build this botec that includes the probability an X% of the people you bring into the fellowship are not going to care about animals and likely, if they get a policy role, to pass policies that are really bad for them...

...I sort of suspect that only a handful of people are trying to do this, and I get why! I made a reasonably straightforward botec for calculating the benefits to birds of bird-safe glass, that accounted for backfire to birds, and it took a lot of research effort. If you asked me how bird-safe glass policy is going to affect AI risk after all that, I might throw my computer at you. But I think the precise probabilities approach would imply that I should.  

Just purely on the descriptive level and not the normative one —

I agree but even more strongly: in AI safety I've basically never seen a BOTEC this detailed. I think Eric Neyman's BOTEC of the cost-effectiveness of donating to congressional candidate Alex Bores is a good public example of the type of analysis common in EA-driven AI safety work: it bottoms out in pretty general goods like "government action on AI safety" and does not try to model second-order effects to the degree described here. It doesn't model even considerations like "what if AI safety legislation is passed, but that legislation backfires by increasing polarization on the issue?" let alone anything about animals.

Instead, this kind of strategic discussion tends to be qualitative, and is hashed out in huge blocks of prose and comment threads e.g. on LessWrong, or verbally.

I sort of wonder if some people in the AI community -- any maybe you, from what you've said here? -- are using precise probabilities to get to the conclusion that you want to work primarily on AI stuff, and then spotlighting to that cause area when you're analyzing at the level of interventions.

I see why you describe it this way, and this directionally this seems right. But, what we do doesn't really sound like "spotlighting" as you describe it in the post: focusing on specific moral patient groups and explicitly setting aside others.

Essentially I think the epistemic framework we use is just more anarchic and freeform than that! In AIS discourse, it feels like "but this intervention could slow down the US relative to China" or "but this intervention could backfire by increasing polarization" or "but this intervention could be bad for animals" exist at the same epistemic level, and all are considered valid points to raise.

(I do think that there is a significant body of orthodox AI safety thought which takes particular stances on each of these issues and other issues, which in a lot of contexts likely makes various points feel like not "valid" to raise. I think this is unfortunate.)

Maybe it's similar to the difference between philosophy and experimental science, where in philosophy a lot of discourse is fundamentally unstructured and qualitative, and in the experimental sciences there is much more structure because any contribution needs to be an empirical experiment, and there are specific norms and formats for those, which have certain implications for how second-order effects are or aren't considered. AI safety discourse also feels similar at times to wonk-ish policy discourse.

(Within certain well-scoped sub-areas of AI safety things are less epistemically anarchic; e.g. research into AI interpretability usually needs empirical results if it's to be taken seriously.)

I think someone using precise probabilities all the way down is building a lot more explicit models every time they consider a specific intervention. Like if you're contemplating running a fellowship program for AI interested people, and you have animals in your moral circle, you're going to have to build this botec that includes the probability an X% of the people you bring into the fellowship are not going to care about animals and likely, if they get a policy role, to pass policies that are really bad for them. And all sorts of things like that. So your output would be a bunch of hypotheses about exactly how these fellows are going to benefit AI policy, and some precise probabilities about how those policy benefits are going to help people, and possibly animals to what degree, etc. 

Hmm, I wouldn't agree that someone using precise probabilities "all the way down" is necessarily building these kind of explicit models. I wonder if the term "precise probabilities" is being understood differently in our two areas.

In the Bayesian epistemic style that EA x AI safety has, it's felt that anyone can attach precise probabilities to their beliefs with ~no additional thought, and that these probabilities are subjective things which may not be backed by any kind of explicit or even externally legible model. There's a huge focus on probabilities as betting odds, and betting odds don't require such things (diverging notably from how probabilities are used in science).

I mean, I think typically people have something to say to justify their beliefs, but this can be & often is something as high-level as "it seems good if AGI companies are required to be more transparent about their safety practices," with little in the way of explicit models about downstream effects thereof.^[1]

Apologies for not responding to some of the other threads in your post, ran out of time; looking forward to discussing in person sometime.

1. ^{^}

   While it's common for AI safety people to agree with my statement about transparency here, some may flatly disagree (i.e. disagree about sign), and others (more commonly) may disagree massively about the magnitude of the effect. There are many verbal arguments but relatively few explicit models to adjudicate these disputes.

I just remembered Matthew Barnett's 2022 post My Current Thoughts on the risks from SETI which is a serious investigation into how to mitigate this exact scenario.

That does seem right, thanks. I intended to include dictator-ish human takeover there (which seems to me to be at least as likely as misaligned AI takeover) as well, but didn't say that clearly.

Edited to "relatively amoral forces" which still isn't great but maybe a little clearer.

Enjoyed this post.

Maybe I'll speak from an AI safety perspective. The usual argument among EAs working on AI safety is:

1. the future is large and plausibly contains much goodness
2. today, we can plausibly do things to steer (in expectation) towards achieving this goodness and away from catastrophically losing it
3. the invention of powerful AI is a super important leverage point for such steering

This is also the main argument motivating me — though I retain meaningful meta-uncertainty and am also interested in more commonsense motivations for AI safety work.

A lot of the potential goodness in 1. seems to come from digital minds that humans create, since it seems that at some point these will be much quicker to replicate than humans or animals. But lots of the interventions in 2. seem to also be helpful for getting things to go better for current farmed and wild animals, e.g. because they are aimed avoiding a takeover of society by forces which don't care at all about morals. Personally I hope we use technology to lift wild animals out of their current predicament, although I have little idea what it would look like with any concreteness.

This relies on what you call the "assigning precise probabilities" approach, and indeed I rarely encounter AI safety x EA people who aren't happy assigning precise probabilities, even in the face of deep uncertainty. I really like how your post points out that this is a difference from the discourse around wild animal welfare but that it's not clear what the high-level reason for this is. I don't see a clear high-level reason either from my vantage point. Some thoughts:

• It might be interesting to move out of high-level reason zone entirely and just look at the interventions, e.g. directly compare the robustness of installing bird-safe glass in a building vs. something like developing new technical techniques to help us avoid losing control of AIs.
• What would the justification standards in wild animal welfare say about uncertainty-laden decisions that involve neither AI nor animals: e.g. as a government, deciding which policies to enact, or as a US citizen, deciding who to vote for President?

Coda: to your "why should justification standards be the same" question, I'd just want to say I'm very interested in maintaining the ideal that EAs compare and debate these things; thanks for writing this!

No one is dying of not reading Proust, but many people are leading hollower and shallower lives because the arts are so inaccessible.

Tangential to your main point, and preaching to the choir, but... why are "the arts" "inaccessible?" The Internet is a huge revolution in the democratization of art relative to most of human history, TV dramas are now much more complex and interesting than they have been in the past, A24 is pumping out tons of weird/interesting movies, way more people are making interesting music and distributing it than before.

I think (and this is a drive-by comment, I haven't read the article), the author is conflating "serious literature" — often an acquired taste that people need to get from a class or similar — with all of "the arts." I studied literature in college, read poetry and e.g. Tolstoy in my free time now, yada yada — and I think this is extremely paternalistic.

I think there's value in someone teaching you to enjoy Proust, and indeed wish more people had access to that sort of thing. But I don't think it comes anywhere close to deserving the kind of uniquely elevated position over other forms of artistic production that literature professors etc sometimes (not always) want to give it, and which I feel is on display in this quote.

In any case, the obvious thing to do is ask whether the beneficiaries would prefer more soup or more Proust.

Vince Gilligan (the Breaking Bad guy) has a new show Pluribus which is many things, but also illustrates an important principle, that being (not a spoiler I think since it happens in the first 10 minutes)...

If you are SETI and you get an extraterrestrial signal which seems to code for a DNA sequence...

DO NOT SYNTHESIZE THE DNA AND THEN INFECT A BUNCH OF RATS WITH IT JUST TO FIND OUT WHAT HAPPENS. 

Just don't. Not a complicated decision. All you have to do is go from "I am going to synthesize the space sequence" to "nope" and look at that, x-risk averted. You're a hero. Incredible work.

One note: I think it would be easy for this post to be read as "EA should be all about AGI" or "EA is only for people who are focused on AGI."

I don't think that is or should be true. I think EA should be for people who care deeply about doing good, and who embrace the principles as a way of getting there. The empirics should be up for discussion.