Quotes: Recent discussions of backfire risks in AI safety
Some thinkers in AI safety have recently pointed out various backfire effects that attempts to reduce AI x-risk can have. I think pretty much all of these effects were known before,[1]ย but it's helpful to have them front of mind. In particular, I'm skeptical that we can weigh these effects against the upsides precisely enough to say an AI x-risk intervention is positive or negative in expectation, without making an arbitrary call. (Even if our favorite intervention doesn't have these specific downsides, we should ask if we're pricing in the downsides (and upsides) we haven't yet discovered.)
(Emphasis mine, in all the quotes below.)
Holden's Oct 2025 80K interview:
Holden:ย I mean, take any project. Letโs just take something that seems really nice, like alignment research. Youโre trying to detect if the AI is scheming against you and make it not scheme against you. Maybe thatโll be good. But maybe the thing youโre doing is something that is going to get people excited, and thenย theyโre going to try it instead of doing some other approach. And then it doesnโt work, and the other approach would have worked. Well, now youโve done tremendous harm. Maybe it will work fine, but it willย give people a false sense of security, make them think the problem is solved more than it is, make them move on to other things, and then youโll have a tremendous negative impact that way.
Rob Wiblin: Maybe itโll be used by a human group to get more control, toย more reliably be able to direct an AI to do something and then do a power grab.
Holden:ย [...] Maybe it would have been great if the AIs took over the world. Maybe weโll buildย AIs that are not exactly aligned with humans; theyโre actually just much better โ theyโre kind of like our bright side, theyโre the side we wish we were. [...]
[... M]aybe alignment is just a reallyโฆ What it means is that youโre helping make sure that someone whoโs intellectually unsophisticated โ thatโs us, thatโs humans โ remains forever in control of the rest of the universe andย imposes whatever dumb ideas we have on it forevermore, instead of having our future evolve according to things that are much more sophisticated and better reasoners following their own values.
[...]
Holden: I just think AI is too multidimensional, and thereโs too many considerations pointing in opposite directions. Iโm worried about AIs taking over the world, but Iโm also worried about the wrong humans taking over the world. Andย a lot of those things tend to offset each other, and making one better can make the other worse. [...]
[T]hereโs also a lot ofย micro ways in which you could do harm. Just literally working in safety and being annoying, you might do net harm. You might just talk to the wrong person at the wrong time, get on their nerves. Iโve heard lots of stories of this. Just like, this person does great safety work, but they really annoyed this one person, and that might be the reason we all go extinct.ย
[...]
Option value in the policy world is kind of a bad concept anyway. A lot of times when youโre at a nonprofit or a company and you donโt know what to do, you try and preserve option value. But giving the government the option to go one way or the other, thatโs not a neutral intervention โ itโs just like you donโt know what theyโre going to do with that option. Giving them the option could have been bad. โฆ you donโt know whoโs going to be in power when, andย whether theyโre going to have anything like the goals that you had when you put in some power that they had. I know people have been excited at various points about giving government more power and then at other points giving government less power.
And all this stuff, I mean, this one axis youโre talking about: centralisation of power versus decentralisation.ย Most things that touch policy at all in any way will move us along that spectrum in one direction or another, so therefore have a high chance of being negative [...]ย
And thenย most things that you can do in AI at all will have some impact on policy. Even just alignment research: policy will be shaped by what weโre seeing from alignment research, how tractable it looks, what the interventions look like.
[... I]n AI, itโs easier to annoy someone and polarise them against you, becauseย whatever it is youโre trying to do, thereโs some coalition thatโs trying to do the exact opposite. In certain parts of global health and farm animal welfare, thereโs certainly people who want to prioritise it less, but it doesnโt have the same directional ambiguity.
ย
Helen Tonerโs Nov 2025 80K interview:
Helen: And I think thereโs a natural tension here as well among some people who are very concerned about existential risk from AI, really bad outcomes, and AI safety: thereโs this sense that itโs actually helpful if thereโs only a smaller number of players. Because, one, they can coordinate better โ so maybe if racing leads to riskier outcomes, if you just have two top players, they can coordinate more directly than if you have three or four or 10 โ and also a smaller number of players is going to be easier for an outside body to regulate, so if you just have a small number of companies, thatโs going to be easier to regulate.
[...] But the problem is then the โThen what?โ question of, if you do manage to avoid some of those worst-case outcomes, and then you have this incredibly powerful technology in the hands of a very small number of people, I think just historically thatโs been really bad. Itโs really bad when you have small groups that are very powerful, and typically it doesnโt result in good outcomes for the rest of the world and the rest of humanity.
[...]
Rob:ย I feel like weโre in a very difficult spot, because so many of the obvious solutions that you might have, or approaches you might take to dealing with loss of control do make the concentration of power problem worse and vice versa. So what policies you favour and disfavourย depends quite sensitively on the relative risk of these two things, the relative likelihood of things going negatively in one way versus the other way.
And at least on the loss of control thing, people disagree so much on the likelihood. People who are similarly informed, know about everything there is to know about this, go all the way from thinking itโs a 1-in-1,000 chance to itโs a 1-in-2 chance โ a 0.1% likelihood to 50% chance that we have some sort of catastrophic loss of control. And discussing it leads sometimes to some convergence, butย people just have not converged on a common sense of how likely this outcome is.
So the people who think itโs 50% likely that we have some catastrophic loss-of-control event, itโs understandable that they think, โWell, we just have to make the best of it. Unfortunately, we have to concentrate. Itโs the only way. And the concentration of power stuff is very sad and going to be a difficult issue to deal with, but we have to bear that cost.โ And people who think itโs one in 1,000 are going to say, โThis is a terrible move that youโre making, because weโre accepting much more risk,ย weโre creating much more risk than weโre actually eliminating.โ
ย
Wei Dai, โLegible vs. Illegible AI Safety Problemsโ:
Some AI safety problems areย legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to). But some problems areย illegible (obscure or hard to understand, or in a common cognitive blind spot), meaning there is a high risk that leaders and policymakers will decide to deploy or allow deployment even if they are not solved. (Of course, this is a spectrum, but I am simplifying it to a binary for ease of exposition.)
From an x-risk perspective, working on highly legible safety problems has low or even negative expected value. Similar to working on AI capabilities, it brings forward the date by which AGI/ASI will be deployed, leaving less time to solve the illegible x-safety problems.
Linkpost: Bracketing violates the sure-thing principle
I'd been meaning to turn this into a proper post, but didn't get around to it. So I've just linked to this Google doc, which shows that the "bracketing" decision rule violates a version of the sure-thing principle. I personally don't think this is actually much of a problem (see the conclusion of this post, for one), but it might be of interest.
Musings on non-consequentialist altruism under deep unawareness
(This is a reply to a comment by Magnus Vinding, which ended up seeming like it was worth a standalone Quick Take.)
ย
From Magnus:
For example, if we walk past a complete stranger who is enduring torment and is in need of urgent help, we would rightly take action to help this person, even if we cannot say whether this action reduces total suffering or otherwise improves the world overall. I think that's a reasonable practical stance, and I think the spirit of this stance applies to many ways in which we can and do benefit strangers, not just to rare emergencies.
The intuition here seems to be, "trying to actively do good in some restricted domain is morally right (e.g., virtuous), even when we're not justified in thinking this will have net-positive consequences[1]ย according to impartial altruism". Let's call this intuition Local Altruism is Right (LAR). I'm definitely sympathetic to this. I just think we should be cautious about extending LAR beyond fairly mundane "common sense" cases, especially to longtermist work.
For one, the reason most of us bothered with EA interventions was to do good "on net" in some sense. We weren't explicitly weighing up all the consequences, of course, but we didn't think we were literally ignoring some consequences โ we took ourselves to be accounting for them with some combination of coarse-grained EV reasoning, heuristics, "symmetry" principles, discounting speculative stuff, etc. So it's suspiciously convenient if, once we realize that that reason was confused, we still come to the same practical conclusions.
Second, for me the LAR intuition goes away upon reflection unless at least the following hold (caveat in footnote):[2]
Some examples:
ย
None of which is to say I have a fleshed-out theory! I'm keen to think more about what non-consequentialist altruism under unawareness might look like.
I mean to include Clifton's Option 3 as a possible operationalization of "net-positive consequences according to impartial altruism".
In the definition of LAR, "trying to actively do good" is the key phrase. I find it pretty intuitive that we don't need conditions nearly as strong as (1)+(2) below when we're asking, "Should you refrain from doing [intuitively evil thing]?"
Maybe the most promising angle is to show that it's normatively relevant that our beliefs about the more distant moral patients are (qualitatively?) less grounded in good reasons (see Clifton).
To clarify, the context of the quoted remark was that, just as we can care for those we love in the face of cluelessness, we can likewise care for and benefit strangers.
Specifically in relation to this:
we still have reason to respect other values we hold dear โ those that were never grounded purely in the impartial good in the first place. Integrity, care for those we love, and generally not being a jerk, for starters. Beyond that, my honest answer is: I donโt know.
I think the "other values we hold dear" can and should also include a strong focus on helping strangers as well, even if challenges from cluelessness and the like were to prove devastating to standard versions of consequentialism. I mostly responded to what I saw as an overly exclusive focus on those we love versus others.
I'm not sure about this, though. As I wrote in a previous comment:
The reasons to do various parochial things, or respect deontological constraints, aren't like this. They aren't grounded in something like "this thing out there in the world is horrible, and should be prevented wherever/whenever it is [or whoever causes it]".
The concern I've tried to convey in our discussion so far is: Insofar as our moral reasons for action are grounded in "this thing out there in the world is horrible, and should be prevented wherever/whenever it is [or whoever causes it]", then shining the spotlight of our active altruism on beings who happen to be salient/near to us is arbitrary. To me, active "altruism" per se[1]ย is pretty inextricable from anti-arbitrariness.
And I'm saying, suppose for a moment we're no longer trying to be actively altruistic, and instead consider normative reasons that aren't grounded in the above. Then, prioritizing those whom you actually have special relationships with isn't arbitrary in the relevant sense. Because those relationships give you a reason to prioritize them. (Of course, if we started from an impartial altruistic perspective, this reason would be dwarfed by our duty to reduce large-scale suffering overall, insofar as that's tractable! But the worry is that it's not.)
Is your position something like, "We also have special relationships with strangers who are near to us"? I might be sympathetic to that, but it seems like it needs more unpacking.
Like I said, I do share the LAR intuition in some limited contexts, and it would be pretty sad if there's no non-arbitrary way to make sense of active altruism at all. I find this situation unsettling. But I currently feel confused as to how much I honestly endorse LAR.
As opposed to deontological(-ish) prohibitions against harming strangers.
Thanks for writing this, Anthony!
I find myself wondering what counts as "speculative" vs not. Here are some guesses at sufficient conditions for speculativeness:
An effect is speculative if it is highly sensitive to:
I'm also very interested in this question, because it isn't obvious to me where to draw the line in fields like wild animal welfare. I think I know as little about nematode sentience + welfare, for example, as I do about possible far future beings.ย
Maybe one difference is that it at least feels possible in theory to get more information about nematodes, but not really possible to get more information about far future beings? Although I'm skeptical of my intuitions here, since maybe it's easier than I think to get information about far future beings and harder than I think to get information about nematode sentience.ย
vegan advocacy [...] has a lot of ambiguous effects on wild animal suffering (which don't seem more "speculative", given the fairly robust-seeming arguments Tomasik has written about)
I'd be curious to hear more about this. What are the robust-seeming arguments?ย
The "lower meat production" โ "higher net primary productivity" โ "higher wild animal suffering" connection seems robust to me. Or not that much less robust than the intended benefit, at least.
This might not be the place for a discussion of this, but I personally don't feel that the "robustness" of Tomasikian chain of reasoning you note here is similar to the "robustness" of the idea that factory farms contain a crazy amount of suffering.ย
In the first instance, the specific chain of arrows above seems quite speculative, since we really have no idea how land use would change in a world with no factory farming. Are we that confident net primary productivity will increase? I'm aware there are good arguments for it, but I'd be surprised if someone couldn't come up with good arguments against if they tried.
More importantly, I don't think that's a sufficient reasoning chain to demonstrate that wild animal effects dominate? You'd need to show that wild+farmed animal welfare on post-factory farmed land uses is lower than wild+farmed animal welfare on current land uses, and that seems very sensitive to specific claims about moral weights, weights between types of suffering, empirical information about wild animal quality of life, what it means for a life to be net-negative, etc.
Or am I misunderstanding what you mean by robustness? I've just finished reading your unawareness sequence and mostly feel clueless about everything, including what it could mean for a reasoning chain to be robust.
ย
Sorry this wasn't clear! I wasn't thinking about the choice between fully eliminating factory farming vs. the status quo. I had in mind marginal decreased demand for animal products leading to marginal decreased land use (in expectation), which I do think we have a fairly simple and well-evidenced mechanism for.
I also didn't mean to say the wild animal effects dominate, just that they're large enough to be competitive with the farmed animal effects. I agree the tradeoffs between e.g. cow or chicken suffering vs. wild insect suffering seem ambiguous. (And yep, from a non-suffering-focused perspective, it would also plausibly be ambiguous whether increased wild insect populations are bad.)
(I think when I wrote the above comment, I was thinking of pretty coarse-grained buckets of "robustness" vs "speculativeness".)
No, longtermism is not redundant
Iโm not keen on the recent trend of arguments that persuading people of longtermism is unnecessary, or even counterproductive, for encouraging them to work on certain cause areas (e.g., here, here). This is for a few reasons:
Why expect "heuristics" to be robust to unknown unknowns?
I often read/hear claims that, if we're worried that our evaluations of interventions won't hold up under unknown unknowns, we should follow (simple) heuristics. But what precisely is the argument for this? This isn't a rhetorical question โ I'm just noting my confusion and want to understand this view better.
Interested to hear more from those who endorse this view!
Out of curiosity, any particular examples of such claims?
I partly had in mind personal communications, but some public examples (and very brief summaries of my reactions, not fleshed out counterarguments):
Permissive epistemology doesn't imply precise credences / completeness / non-cluelessness
(Many thanks to Jesse Clifton and Sylvester Kollin for discussion.)
My argumentsย against precise Bayesianism andย for cluelessness appeal heavily to the premise โwe shouldnโtย arbitrarily narrow down our beliefsโ. This premise is very compelling to me (and Iโd be surprised if itโs not compelling to most others upon reflection, at least if we leave โarbitraryโ open to interpretation). I hope to get around to writing more about it eventually.
But suppose you donโt care much about avoiding arbitrariness, for any interesting interpretation of โarbitrarinessโ. Suppose you have a highly permissive epistemology. E.g., you might think that as long as your beliefs satisfy someย โcoherenceโ conditions, thatโs as far as rationality can take you, and youโre free to have whichever coherent beliefs youโre disposed to. Or you might think โbeliefsโย are nothing more or less than constraints on preferences, and itโs fine if preferences are arbitrary.
If you have such a view, does that imply your credences are precise, or your preferences are complete, or the like?
No.[1]ย You might simply introspect on what your response to your evidence, intuitions, etc. is, and find that the most honest representation of that response is imprecise/incomplete. You might try weighing up various considerations about the consequences of some pair of actions, and find that your disposition is, โI have no clue. Even after noticing that some precise number popped into my head, all things considered I have no preference either way (but I also donโt feelย indifferent, because my preferences areย insensitive to mild sweetening).โ
Of course, if you have a permissive epistemology and you donโt have such introspective reactions, thereโs nothing more I can say to you. But itโs important to acknowledge that precision, completeness, and non-cluelessness are not some privileged default for the permissivist.
Why you should consider trying SSRIs
I was initially hesitant to post this, out of some vague fear of stigma and stating the obvious, and not wanting people to pathologize my ethical views based on the fact that I take antidepressants. This is pretty silly for two reasons. First, I think that if my past self had read something like this, he could have been spared years of suffering, and there are probably several readers in his position. EAs are pretty open about mental illness anyway. Second, if anything the fact that I am SFE "despite" currently not being depressed at all (indeed quite consistently happy), thanks to SSRIs, should make readers less likely to attribute my views to a mental illness.[1]
I claim that even if you don't feel so bad as to qualify as capital-D depressed, you might feel noticeably less bad on a daily basis if you try SSRIs.[2] That has been my experience, and I can honestly say this has likely been the cheapest sustainable boost in my well-being I've ever found. Being happier has also probably made me more effective/productive, though this is harder to assess.
(Obviously, my experience is not universal, I'm probably just lucky to an extent, this is not expert medical advice, and you might either find that SSRIs are ineffective for you or that the side effects are less tolerable than I have found them. You should definitely do your own research!)
In the months (at least) prior to SSRIs, my level of depression was "mild" according to the Burns checklist. I felt rather lonely during downtime, and like a bit of a failure for not having an exciting social life. I didn't endorse the latter judgment, and felt pretty fulfilled by my altruistic work, but that dissatisfaction persisted even when I tried to reason myself out of it (or tried taking up new hobbies). This wasn't debilitating by any meansโso much so that I didn't really feel like I "deserved" to receive treatment intended for depression, and yes I realize how dumb that sounds in hindsightโbut it was something of a pall hanging over my life all the same.
SSRIs just dispelled those feelings.
Waiting so long to give these things a try was a mistake. I made that mistake out of a combination of the aforementioned suspicion that I wasn't depressed enough to need them, and overestimation of how difficult it would be to get a prescription.[3] Just because my suffering wasn't as deep as others', that didn't mean it needed to exist.
This medication isn't magic; my life isn't perfect, and I still have some other dissatisfactions I'm working on. But, for the amount of difference this has made for me, it seemed negligent not to share my experience and hopefully encourage others in an analogous position to show themselves a bit of compassion.
[1] Yes, I have seen people do this beforeโnot to me personally, but to other SFEs.
[2] This probably holds for other antidepressants too. I'm just focusing on SSRIs here because I have experience with them, and they incidentally have a worse reputation than, e.g., Wellbutrin.
[3] At least in the U.S., there are online services where you can share your symptoms with a doctor and just get a prescription at a pretty low price. For some reason, I expected a lot more awkward bureaucracy and mandatory therapy than this. I won't get specific here because I don't want to be a shill, but if you're curious, feel free to PM me.
SlateStarCodex has a long post on SSRIs and their side-effects (from 2014); including sexual side-effects. (Here is a 2016 paper which also reports on sexual side-effects.) I don't have expertise in this topic, however.
In Defense of Aiming for the Minimum
Iโm not really sympathetic to the following common sentiment: โEAs should not try to do as much good as feasible at the expense of their own well-being / the good of their close associates.โ
Itโs tautologically true that if trying to hyper-optimize comes at too much of a cost to the energy you can devote to your most important altruistic work, then trying to hyper-optimize is altruistically counterproductive. I acknowledge that this is the principle behind the sentiment above, and evidently some peopleโs effectiveness has benefited from advice like this.
But in practice, I see EAs apply this principle in ways that seem suspiciously favorable to their own well-being, or to the status quo. When you find yourself trying to justify on the grounds of impact the amounts of self-care people afford themselves when they donโt care about being effectively altruistic, you should be extremely suspicious.
Some examples, which I cite not to pick on the authors in particularโsince I think many others are making a similar mistakeโbut just because they actually wrote these claims down.
1. โAiming for the minimum of self-care is dangerousโ
I felt a bit suspicious, looking at how I spent my time. Surely that long road trip wasnโt necessary to avoid misery? Did I really need to spend several weekends in a row building a ridiculous LED laser maze, when my other side project was talking to young synthetic biologists about ethics?
I think this is just correct. If your argument is that EAs shouldnโt be totally self-effacing because some frivolities are psychologically necessary to keep rescuing people from the bottomless pit of suffering, then sure, do the things that are psychologically necessary. Iโm skeptical that โpsychologically necessaryโ actually looks similar to the amount of frivolities indulged by the average person who is as well-off as EAs generally are.
Do I live up to this standard? Hardly. That doesnโt mean I should pretend Iโm doing the right thing.
Minimization is greedy. You donโt get to celebrate that youโve gained an hour a day [from sleeping seven instead of eight hours], or done something impactful this week, because that minimizing urge is still looking at all your unclaimed time, and wondering why you arenโt using it better, too.
How important is my own celebration, though, when you really weigh it against what I could be doing with even more time? (This isnโt just abstract impact points; there are other beings whose struggles matter no less than mine do, and fewer frivolities for me could mean relief for them.)
I think where I fundamentally disagree with this post is that, for many people, I donโt think aiming for the minimum puts you close to less than the minimum. Getting to the minimum, much less below it, can be very hard, such that people who aim at it just arenโt in much danger of undershooting. If you find this is not true for yourself, then please do back off from the minimum. But remember that in the counterfactual where you hadnโt tested your limits, you probably would not have gotten close to optimal.
This post includes some saddening anecdotes about people ending up miserable because they tried to optimize all their time for altruism. I donโt want to trivialize their suffering. Yet I can conjure anecdotes in the opposite direction (and the kind of altruism I care about reduces more suffering in expectation). Several of my colleagues seem to work more than the typical job entails, and I donโt have any evidence of the quality of their work being the worse for this. Iโve found that the amount of time I can realistically devote to altruistic efforts is pretty malleable. No, Iโm not a machine; of course I have my limits. But when I gave myself permission to do altruistic things for parts of weekends, or into later hours of weekdays, well, I could. โMy happiness is not the point,โ as Julia said in this post, and while she evidently doesnโt endorse that statement, I do. That just seems to be the inevitable consequence of taking the sentience of other beings besides yourself (or your loved ones) seriously.
See also this comment:
Personally have been trying to think of my life only as a means to an end. Will my life technically might have value, I am fairly sure it is rather minuscule compared to the potential impact can make. I think it's possible, though probably difficult, to intuit this and still feel fine / not guilty, about things. โฆ I'm a bit wary on this topic that people might be a bit biased to select beliefs based on what is satisfying or which ones feel good.
I do think Tessa's point about slack has some forceโthough in a sense, this merely shifts the โminimumโ up by some robustness margin, which is unlikely to be large enough to justify the average personโs indulgences.
2. โYou have more than one goal, and thatโs fineโ
If I donate to my friendโs fundraiser for her sick uncle, Iโm pursuing a goal. But itโs the goal of โsupport my friend and our friendship,โ not my goal of โmake the world as good as possible.โ When I make a decision, itโs better if Iโm clear about which goal Iโm pursuing. I donโt have to beat myself up about this money not being used for optimizing the world โ that was never the point of that donation. That money is coming from my โpersonal satisfactionโ budget, along with money I use for things like getting coffee with friends.
It puzzles me that, as common as concerns about the utility monsterโsacrificing the well-being of the many for the super-happiness of oneโare, we seem to find it totally intuitive that one can (passively) sacrifice the well-being of the many for oneโs own rather mild comforts. (This is confounded by the act vs. omission distinction, but do you really endorse that?)
The latter conclusion is basically the implication of accepting goals other than โmake the world as good as possible.โ What makes these other goals so special, that they can demand disproportionate attention (โdisproportionateโ relative to how much actual well-being is at stake)?
Due to the writing style, itโs honestly not clear to me what exactly this post was claiming. But the author does emphatically say that devoting all of their time to the activity that helps more people per hour would be โpremature optimization.โ And they celebrate an example of a less effective thing they do because it consistently makes a few people happy.
I donโt see how the post actually defends doing the less effective thing. To the extent that you impartially care about other sentient beings, and donโt think their experiences matter any less because you have fewer warm fuzzy feelings about them, what is the justification for willingly helping fewer people?
Some reasons not to primarily argue for veganism on health/climate change grounds
I've often heard animal advocates claim that since non-vegans are generally more receptive to arguments from health benefits and reducing climate impact, we should prioritize those arguments, in order to reduce farmed animal suffering most effectively.
On its face, this is pretty reasonable, and I personally don't care intrinsically about how virtuous people's motivations for going vegan are. Suffering is suffering, no matter its sociological cause.
But there are some reasons I'm nervous about this approach, at least if it comes at the opportunity cost of moral advocacy. None of these are original to me, but I want to summarize them here since I think this is a somewhat neglected point:
Quick comment. With respect to your first point, this has always struck me as one of the better points as to why non ethical arguments should primarily avoided when it comes to making the case for veganism. However, after reading Tobias Leenaert's 'How to Create a Vegan World: A Pragmatic Approach', I've become a bit more agnostic on this notion. ย He notes a few studies from The Humane League that show that red-meat reducers/avoiders tend to eat less chicken than your standard omnivore. He also referenced a few studies from Nick Cooney's book, Veganomics, which covers some of this on p. 107-111. Combined with the overall impact non-ethical vegans could have on supply/demand for other vegan products (and their improvement in quality), I've been a bit less worried about this reason.
I think your other reasons are all extremely important and underrated, though, so still lean overall that the ethical argument should be relied on when possible :) ย
Wow, that's promising news! Thanks for sharing.
Linkpost: "Tranquilism Respects Individual Desires"
I wrote a defense of an axiology on which an experience is perfectly good to the extent that it is absent of craving for change. This defense follows in part from a reductionist view of personal identity, which is usually considered in EA circles to be in support of total symmetric utilitarianism, but I argue that this view lends support to a form of negative utilitarianism.
Some things I liked about What We Owe the Future, despite my disagreements with the treatment of value asymmetries:
The Repugnant Conclusion is worse than I thought
At the risk of belaboring the obvious to anyone who has considered this point before: The RC glosses over the exact content of happiness and suffering that are summed up to the quantities of โwelfareโ defining world A and world Z. In world A, each life with welfare 1,000,000 could, on one extreme, consist purely of (a) good experiences that sum in intensity to a level 1,000,000, or on the other, (b) good experiences summing to 1,000,000,000 minus bad experiences summing (in absolute value) to 999,000,000. Similarly, each of the lives of welfare 1 in world Z could be (a) purely level 1 good experiences, or (b) level 1,000,001 good experiences minus level 1,000,000 bad experiences.
To my intuitions, itโs pretty easy to accept the RC if our conception of worlds A and Z is the pair (a, a) from the (of course non-exhaustive) possibilities above, even more so for (b, a). However, the RC is extremely unpalatable if we consider the pair (a, b). This conclusion, which is entailed by any plausible non-negative[1] total utilitarian view, is that a world of tremendous happiness with absolutely no suffering is worse than a world of many beings each experiencing just slightly more happiness than those in the first, but along with tremendous agony.
To drive home how counterintuitive that is, we can apply the same reasoning often applied against NU views: Suppose the level 1,000,001 happiness in each being in world Z is compressed into one millisecond of some super-bliss, contained within a life of otherwise unremitting misery. There doesnโt appear to be any temporal ordering of the experiences of each life in world Z such that this conclusion isnโt morally absurd to me. (Going out with a bang sounds nice, but not nice enough to make the preceding pure misery worth it; remember this is a millisecond!) This is even accounting for the possible scope neglect involved in considering the massive number of lives in world Z. Indeed, multiplying these lives seems to make the picture more horrifying, not less.
Again, at the risk of sounding obvious: The repugnance of the RC here is that on total non-NU axiologies, weโd be forced to consider the kind of life I just sketched a โnet-positiveโ life morally speaking.[2] Worse, we're forced to consider an astronomical number of such lives better than a (comparatively small) pure utopia.
[1] โNegativeโ here includes lexical and lexical threshold views.
[2] Iโm setting aside possible defenses based on the axiological importance of duration. This is because (1) Iโm quite uncertain about that point, though I share the intuition, and (2) it seems any such defense rescues NU just as well. I.e. one can, under this principle, maintain that 1 hour of torture-level suffering is impossible to morally outweigh, but 1 millisecond isnโt.
This conclusion, which is entailed by any plausible non-negative[1] total utilitarian view, is that a world of tremendous happiness with absolutely no suffering is worse than a world of many beings each experiencing just slightly more happiness than those in the first, but along with tremendous agony.
It seems to me that you're kind of rigging this thought experiment when you define an amount of happiness that's greater than an amount of suffering, but you describe the happiness as "slight" and the suffering as "tremendous", even though the former is larger than the latter.
I don't call the happiness itself "slight," I call it "slightly more" than the suffering (edit: and also just slightly more than the happiness per person in world A). I acknowledge the happiness is tremendous. But it comes along with just barely less tremendous suffering. If that's not morally compelling to you, fine, but really the point is that there appears (to me at least) to be quite a strong moral distinction between 1,000,001 happiness minus 1,000,000 suffering, and 1 happiness.
A Parfitian Veil of Ignorance
[Edit: I would be very surprised if I were the first person to have proposed this; it probably exists somewhere else, I just don't know of a source.]
Prompted by Holdenโs discussion of the veil of ignorance as a utilitarian intuition pump (contra Rawls), I thought about an alternative to the standard veil. My intuitions about tradeoffs of massive harms for a large number of small benefitsโat least for some conceptions of "benefit"โdiverge from those in his post, when considering this version.
The standard veil of ignorance asks you to imagine being totally ignorant as to which person you will be in a population. (Assume weโre only considering fixed population sizes, so thereโs no worry that this exercise sneaks in average utilitarianism, etc.)
But the many EA fans of Parfit (or Buddha) know that this idea of a discrete person is metaphysically problematic. So we can look at another approach, inspired by empty individualism.
Imagine that when evaluating two possible worlds, you donโt know which slice of experience in each world you would be. To make things easy enough to grasp, take a โsliceโ to be just the longest amount of time necessary for a sentient being to register an experience, but not much longer. Letโs say one second.
These worlds might entail probabilities of experiences as well. So, since itโs hard to intuitively grasp probabilities as effectively as frequencies, suppose each world is โre-rolledโ a large enough times that each outcome happens at least once, in proportion to its probability. e.g., In Holdenโs example of a 1 in 100 million chance of someone dying, the experiences of that person are repeated 100 million times, and one of those experience streams is cut short by death.
So now a purely aggregative and symmetric utilitarian offers me a choice, from behind this veil of ignorance, between two worlds. Option 1 consists of a person who lives for one day with constantly neutral experiencesโno happiness, no suffering (including boredom). In option 2, that person instead spends that day relaxing on a nice beach, with a 1 in 100 million chance of ending that day by spiraling into a depression (instead of dying peacefully in their sleep).
I imagine, first, rescaling things so in #1 the person lives 100 million days of neutrality, and in #2, they live 99,999,999 peaceful beach-daysโsuspend your disbelief and imagine they never get boredโfollowed by a beach-day that ends in depression. Then I imagine I donโt know which moment of experience in either of these options Iโll be.
Choosing #1 seems pretty defensible to me, from this perspective. Several of those experience-moments in #2 are going to consist purely of misery. They wonโt be comforted by the fact that theyโre rare, or that theyโre in the context of a โpersonโ who otherwise is quite happy. Theyโll just suffer.
Iโm not saying the probabilities donโt matter. Of course they do; Iโd rather take #2 than a third option where thereโs a 1 in 100 thousand chance of depression. Iโm also pretty uncertain where I stand when we modify #1 so that the personโs life is a constant mild itch instead of neutrality. The intuition this thought experiment prompts in me is the lexical badness of at least sufficiently intense suffering, compared with happiness or other goods. And I think the reason it prompts such an intuition is that in this version of the veil of ignorance, discrete โpersonsโ donโt get to dictate what package of experiences is worth it, i.e., what happens to the multitude of experience-moments in their life. Instead, one has to take the experience-moments themselves as sovereign, and decide how to handle conflicts among their preferences. (I discuss this more here.)
I find the framing of "experience slices" definitely pushes my intuitions in the same direction.
One question I like to think about is whether I'd choose to gain either
(a) a neutral experience
orย
(b) flipping a coin and reliving all the positive experience slices of my life if heads, and reliving all the negative ones if tails
My life feels highly net positive but I'd almost certainly not take option (b). I'd guess there's likely risk aversion intuition also being snuck here too though.
Some vaguely clustered opinions on metaethics/metanormativity
I'm finding myself slightly more sympathetic to moral antirealism lately, but still afford most of my credence to a form of realism that would not be labeled "strong" or "robust." There are several complicated propositions I find plausible that are in tension:
1. I have a strong aversion to arbitrary or ad hoc elements in ethics. Practically this cashes out as things like: (1) rejecting any solutions to population ethics that violate transitivity, and (2) being fairly unpersuaded by solutions to fanaticism that round down small probabilities or cap the utility function.
2. Despite this, I do not intrinsically care about the simplicity of a moral theory, at least for some conceptions of "simplicity." It's quite common in EA and rationalist circles to dismiss simple or monistic moral theories as attempting to shoehorn the complexity of human values into one box. I grant that I might unintentionally be doing this when I respond to critiques of the moral theory that makes most sense to me, which is "simple." But from the inside I don't introspect that this is what's going on. I would be perfectly happy to add some complexity to my theory to avoid underfitting the moral data, provided this isn't so contrived as to constitute overfitting. The closest cases I can think of where I might need to do this are in population ethics and fanaticism. I simply don't see what could matter morally in the kinds of things whose intrinsic value I reject: rules, virtues, happiness, desert, ... When I think of these things, and the thought experiments meant to pump one's intuitions in their favor, I do feel their emotional force. It's simply that I am more inclined to think of them as just that: emotional, or game theoretically useful constructs that break down when you eliminate bad consequences on conscious experience. The fact that I may "care" about them doesn't mean I endorse them as relevant to making the world a better place.
3. Changing my mind on moral matters doesn't feel like "figuring out my values." I roughly know what I value. Many things I value, like a disproportionate degree of comfort for myself, are things I very much wish I didn't value, things I don't think I should value. A common response I've received is something like: "The values you don't think you 'should' have are simply ones that contradict stronger values you hold. You have meta-preferences/meta-values." Sure, but I don't think this has always been the case. Before I learned about EA, I don't think it would have been accurate to say I really did "value" impartial maximization of good across sentient beings. This was a value I had to adopt, to bring my motivations in line with my reasons. Encountering EA materials did not feel at all like "Oh, you know what, deep down this was always what I would've wanted to optimize for, I just didn't know I would've wanted it."
4. The question "what would you do if you discovered the moral truth was to do [obviously bad thing]?" doesn't make sense to me, for certain inputs of [obviously bad thing], e.g. torturing all sentient beings as much as possible. For extreme inputs of that sort, the question is similar to "what would you do if you discovered 2+2=5?" For less extreme inputs, such that it's plausible to me I simply have not thought through ethics enough that I could imagine that hypothetical but merely find it unlikely right now, the question does make sense, and I see nothing wrong with saying "yes." I suspect many antirealists do this all the time, radically changing their minds on moral questions due to considerations other than empirical discoveries, and they would not be content saying "screw the moral truth" by retaining their previous stance.
5. I do not expect that artificial superintelligence would converge on The Moral Truth by default. Even if it did, the convergence might be too slow to prevent catastrophes. But I also doubt humans will converge on this either. Both humans and AIs are limited by our access only to our "own" qualia, and indeed our own present qualia. The kind of "moral realism" I find plausible with respect to this convergence question is that convergence to moral truth could occur for a perfectly rational and fully informed agent, with unlimited computation and - most importantly - subjective access to the hypothetical future experiences of all sentient beings. These conditions are so idealized that I am probably as pessimistic about AI as any antirealist, but I'm not sure yet if they're so idealized that I functionally am an antirealist in this sense.
Poll: Is this one of your cruxes for cluelessness?
There's a cluster of responses to arguments for cluelessness I've encountered, which I'm not yet sure I understand but maybe is important. Here's my attempted summary:[1]
If you're skeptical of cluelessness, I'd find it super helpful if you'd agree-vote if you agree with the above response or something very similar to it, and disagree-vote otherwise. ETA: added a poll widget below, please use that instead (thanks to Toby Tremlett for suggesting this). (Please don't vote if you aren't skeptical of cluelessness.) And feel free to comment with some different version of the above you'd agree with, if that difference is important for you. Thanks!
Some examples of sentiments that, IIUC, this summary encapsulates (emphasis mine):
* Greaves: "I think most of us feel like weโre really just making up arbitrary numbers, but thatโs really uncomfortable because precisely which arbitrary numbers we make up seems to make a difference to what we ended up doing." See also Greaves' discussion of the "decision discomfort" involved in complex cluelessness.
* Soares: "Now, I agree that this scenario is ridiculous. And that it sucks. And I agree that picking a precise minute feels uncomfortable. And I agree that this is demanding way more precision than you are able to generate. But if you find yourself in the game, you'd best pick the minute as well as you can. When the gun is pressed against your temple, you cash out your credences."
ย
A non-trivial reason for this is that precise numbers expose ideological assumptions, and a whole of people do not like this.
It's easy to lie with numbers, but it's even easier to lie without a number.
I understand my vote to be consistent with us never in fact reaching 'principled' precise EV numbers though.ย
Is there a way to retract a vote? I accidentally voted (I'm not skeptical of cluelessness).
There should be a little x if you hover over your icon, clicking that should remove your vote.
One of the only good scenarios that I can think of where this response to cluelessness makes sense, is if a person subscribes to moral realism. And even then, the arguments for moral uncertainty seem too compelling to me. Do you know of any person that is not skeptical of cluelessness?ย
My take is that, if you really try to look at it from first principles, most will arrive at the conclusion that it is not possible to calculate the EV of any action for us humans. The rest is just cope because one is not willing to give up on ones EA identity and all of the sacrifices you have done. Sunk-cost fallacy is just too big of an obstacle.
Yes, if there is a moral theory that is objectively correct, then one should in principle be able to translate that moral theory into a framework that helps us calculate EV. But since we are not omniscient, that just does seem impossible. But in combination with the principle that we can't understand the downstream effect of our actions in the long-term, I don't understand how somebody can be skeptical of cluelessness.ย
I know this comment does not directly address your initial prompt, but I thought I'd rather post it than not. Thank you for sharing!
(You might find this discussion helpful.)