EA is about maximization, and maximization is perilous

I strongly agree with some parts of this post, in particular:

• I think integrity is extremely important, and I like that this post reinforces that.
• I think it’s a great point that EA seems like it could be very bitterly divided indeed, and appreciating that we haven’t as well as thinking about why (despite our various different beliefs) seems like a great exercise. It does seem like we should try to maintain those features.

On the other hand, I disagree with some of it -- and thought I'd push back especially given that there isn't much pushback in the comments here:

I think it’s a bad idea to embrace the core ideas of EA without limits or reservations; we as EAs need to constantly inject pluralism and moderation. That’s a deep challenge for a community to have - a constant current that we need to swim against.

I think this is misleading in that I’d guess the strongest current we face is toward greater moderation and pluralism, rather than radicalism. As a community and as individuals, some sources of pressure in a ‘moderation’ direction include:

1. As individuals, the desire to be liked by and get along with others, including people inside and outside of EA

2. As individuals that have been raised in a mainstream ethical environment (most of us), a natural pluralism and strong attraction to common sense morality

3. The desire to live a normal life full of the normal recreational, familial, and cultural stuff

4. As a community, wanting to seem less weird to the rest of the world in order to be able to attract and/or work with people who are (currently) unfamiliar with the EA community.

5. Implicit and explicit pressure from another against weirdness so that we don’t embarrass one another/hurt EA’s reputation

6. Fear of being badly wrong in a way that feels less excusable because it’s not the case that everyone else is also badly wrong in the same way

7. Whatever else is involved in the apparent phenomenon where as a community gets bigger, it often becomes less unique

We do face some sources of pressure away from pluralism and moderation, but they seem fewer and weaker to me:

1. The desire to seem hardcore that you mentioned

2. Something about a desire for interestingness/feeling interesting/specialness (possible overlap with the above)

3. Selection effects-- EA tends to attract people who are really into consistency and following arguments wherever they lead (though I'd guess this is getting weaker over time bc of the above effects).

4. Maybe other things?

I do agree that we should try hard to guard against bad maximising - but I think we also need to make sure we remember what is really important about maximising in the face of pressure not to.

Also, moral and empirical uncertainty strongly favour moderation and pluralism -- so I agree that it’s good to have reservations about EA ideas (though primarily in the same way it’s good to have reservations about a lot of ideas). I do not want to think of those ideas as separate from or in tension with the core ideas of EA. I think it would be better to think of them as an important part of the ideas of EA.

Somewhat speculating: I also wonder if the two problems you cite at the top are actually sort of a problem and a solution:

If you’re maximizing X, you’re asking for trouble by default. You risk breaking/downplaying/shortchanging lots of things that aren’t X, which may be important in ways you’re not seeing. Maximizing X conceptually means putting everything else aside for X - a terrible idea unless you’re really sure you have the right X. (This idea vaguely echoes some concerns about AI alignment, e.g., powerfully maximizing not-exactly-the-right-thing is something of a worst-case event.)

EA is about maximizing how much good we do. What does that mean? None of us really knows. EA is about maximizing a property of the world that we’re conceptually confused about, can’t reliably define or measure, and have massive disagreements about even within EA. By default, that seems like a recipe for trouble.

Maybe EA is avoiding the dangers of maximisation (insofar as we are) exactly because we are trying to maximize something we’re confused about. Since we’re confused about what ‘the good’ is, we’re constantly hedging our bets; since we’re unsure how to achieve the good, we go for robust approaches and try a variety of approaches and try not to alienate people who can help us figure out what the good is and how to make it happen. This uncertainty reduces the risks of maximisation greatly. Analogy: Stuart Russel’s strategy to make AI safe by making it unsure about its goals.

Thank you for writing this. For a while, I have been thinking of writing a post with many similar themes and maybe I still will at some point. But this post fills a large hole.

As is obligatory for me, I must mention Derek Parfit, who tends to have already well-described many ideas that resurface later.

In Reasons and Persons, Part 1 (especially Chapter 17), Derek Parfit argues that good utilitiarians should self-efface their utilitarianism. This is because people tend to have motivated reasoning, and tend to be wrong. Under utilitarianism, it is possible to justify nearly anything, provided your epistemics are reasonably bad (your epistemics would have to be very bad to justify murder under deontological theories that prohibit murder; you would have to claim that something was not in fact murder at all). Parfit suggests adopting whatever moral system seems to be most likely to produce the highest utility for that person in the long run (perhaps some theory somewhat like virtue ethics). This wasn't an original idea, and Mill said similar things.

One way to self-efface your utilitiarianism would be to say "yeah, I know, it makes sense under utilitarianism for me to keep my promises" (or whatever it may be). Parfit suggests that may not be enough, because deep down you still believe in utilitarianism; it will come creeping through (if not in you, in some proportion of people who self-efface this way). He says that you may instead need to forget that you ever believed in utilitarianism, even if you think it's correct. You need to believe a lie, and perhaps even convince everyone else of this lie.

He also draws an interesting caveat: what if the generally agreed upon virtues or rules are no longer those with the highest expected utility? If nobody believed in utilitarianism, why would they ever be changed? He responds:

This suggests that the most that could be true is that C [consequentialism] is partly self-effacing. It might be better if most people caused themselves to believe some other theory, by some process of self-deception that, to succeed, must also be forgotten. But, as a precaution, a few people should continue to believe C, and should keep convincing evidence about this self-deception. These people need not live in Government House, or have any other special status. If things went well, the few would do nothing. But if the moral theory believed by most did become disastrous, the few could then produce their evidence. When most people learnt that their moral beliefs were the result of self-deception, this would undermine these beliefs, and prevent the disaster.

This wasn't an original idea either; Parfit here is making a reference to Sidgwick's "Government House utilitarianism," which seemed to suggest only people in power should believe utilitarianism but not spread it. Parfit passingly suggests the utilitarians don't need to be the most powerful ones (and indeed Sidgwick's assertion may have been motivated by his own high position).

Sometimes I think that this is the purpose of EA. To attempt to be the "few people" to believe consequentialism in a world where commonsense morality really does need to change due to a rapidly changing world. But we should help shift commonsense morality in a better direction, not spread utilitarianism.

Maybe utilitarianism is an info hazard not worth spreading. If something is worth spreading, I suspect it's virtues.

Which virtues? Some have suggestions.

I'm really glad you wrote this post. Hearing critiques from prominent EAs promotes a valuable community norm of self-reflection and not just accepting EA as is, in my opinion. 

A few thoughts:

1. It's important to emphasize how much maximization can be normalized in EA subpockets. You touch on this in your post: "And I’m nervous about what I perceive as dynamics in some circles where people seem to “show off” how little moderation they accept - how self-sacrificing, “weird,” extreme, etc. they’re willing to be in the pursuit of EA goals"). I agree, and I think this is relevant to growing EA Hubs and cause-area silos. If you move to an EA hub that predominantly maximizes along one belief (e.g.,  AI safety in Berkeley), very natural human tendencies will draw you to also maximize along that belief. Maximizing will win you social approval and dissenting is hard, especially if you're still young and impressionable (like meee). If you agree with this post's reasoning, I think you should take active steps to correct for a social bias toward hard-core maximizing (see 2).
2. If you're going to maximize along some belief, you should seriously engage with the best arguments for why you're wrong. Scout mindset baby. Forming true beliefs about a complicated world is hard and motivated reasoning is easy. 
3. Maximizing some things is still pretty cool. I think some readers (of the post and my comment) could come away with a mistaken impression that more moderation in all aspects of EA is always a good thing. I think it's more nuanced than that: Most people who have done great things in the past have maximized much harder than their peers. I agree one should be cautious of maximizing things we are"conceptually confused about, can’t reliably define or measure, and have massive disagreements about even within EA." But maximizing some things that are good across a variety of plausibly true beliefs can be pretty awesome for making progress on your goals (e.g., maximizing early-career success and learning). And even if the extreme of maximization is bad, more maximization might be directionally good, depending on how much you're currently maximizing. We also live in a world that may end within the next 100 years, so you have permission to be desperate.

Mmh, not sure how many of these problems can be solved by "just maximizing correctly", or, as I like to call it, "better implementing utilitarianism". E.g., some of the problems you gesture at seem like they could be solved by better evaluation and better expected value calculations. 

Arguendo, once our expected value calculations become good enough, we should just switch to them, even if there is some "valley of naïve utilitarianism".

And I’m nervous about what I perceive as dynamics in some circles where people seem to “show off” how little moderation they accept - how self-sacrificing, “weird,” extreme, etc. they’re willing to be in the pursuit of EA goals. I think this dynamic is positive at times and fine in moderation, but I do think it risks spiraling into a problem.

There seems to be an important trade-off here, where this is a valuable signal that the person "showing off" is aligned with your values and it's actually pretty useful to know that (especially since current gradients often push in favor of people who are not aligned paying lip service to EA ideas in order to gain money/status/power for themselves). 

The balance of how much we should ask or expect of this category of sacrifice seems like one that we should put lots of time as a community into thinking about, especially when we're trying to grow quickly and are unusually willing to provide resources to people.

"My view is that - for the most part - people who identify as EAs tend to have unusually high integrity. But my guess is that this is more despite utilitarianism than because of it."

This seems unlikely to me. I think utilitarianism broadly encourages pro-social/cooperative behaviors, especially because utilitarianism encourages caring about collective success rather than individual success. Having a positive community and trust helps achieve these outcomes. If you have universalist moralities, it's harder for defection to make sense.

Broadly, I think that worries about utilitarianism/consequentialism should lead to negative outcomes are often self defeating, because the utilitarians/consequentiality see the negative outcomes themselves.  If you went around killing people for their organs, the consequences would obviously be negative; it's the same for going around lying or being an asshole toe people all the time.

I disagree with the following:

But I doubt you can make a case that’s robustly compelling and is widely agreed upon, enough to prevent the dynamics I worry about above.

"I doubt you can make a case that's robustly compelling..."

Systemic cascading effects and path dependency might be very coherent consequentialist frameworks & catchphrases to resolve a lot of your epistemic concerns (and this is something I want to explore further).

Naive consequentialism might incentivize you to lie to "do whatever it takes to do good", but the impacts of lying can cascade and affect the bedrock institutional culture and systems of a movement. On aggregate, these cascading (second-order) effects will make it more difficult for people to trust each other and work together in honest ways, making the moral calculus not worth it.

Furthermore, this might have a path-dependent effect, analogous to a significant/persistent/contingent effect, where choosing this path encodes certain values in the institution and makes it harder for other community values to arise in the future.

This similarly generalizes to most "overoptimization becomes illogical" problems. Naive consequentialism & low-integrity epistemics rarely make sense in the long run anyways, so it's just a matter of dispelling simplified, naive models of reality and coherently phrasing the importance of epistemics, diversity, and plurality through a consequentialist lens.

"...and is widely agreed upon."

Still relatively new to the community, so I might have the wrong view on this - but I'm always remarkably surprised by how openly EAs are willing to discuss flaws in the community & are concerned about solid epistemics within the community. 

E.g. recently just posted a submission to the EA criticism contest - and it's difficult for me to imagine any other subgroup which pours $100k into a contest seriously considering and rewarding internal & external criticism about its most fundamental values and community.

This is a really great post Holden; thank you for writing it.

And I’m nervous about what I perceive as dynamics in some circles where people seem to “show off” how little moderation they accept - how self-sacrificing, “weird,” extreme, etc. they’re willing to be in the pursuit of EA goals. I think this dynamic is positive at times and fine in moderation, but I do think it risks spiraling into a problem.

As a somewhat outside observer, it seems a larger number of EAs, including many of those who drive the zeitgeist of this forum, are orienting their entire lives around EA (working directly + EA dominated social life + dating within EA + consumption of media through twitter/podcasts  largely consisting of EA curation). I think this is a serious concern for many reasons,  but one important one is that I suspect an insular community is more likely to produce behaviours like those described in your post.

One issue I have with these arguments for pluralism and for sometimes obeying something like common sense morality for its own sake and independent of utilitarian justification is that common sense morality is crazy/impossible to follow in almost all normal decision situations if you think  it's implications through properly.

One argument for this is MacAskill's argument that deontology required paralysis. Every time you leave the house, you foreseeably cause someone to die by changing the flow of traffic. Cars also contribute to air pollution which foreseeably kills people, suggesting that emitting any amount of pollution is impermissible. This violates nonconsequentialist side-constraints. I don't understand how you can give some weight to this type of view. 

 This is not the point that we should follow utilitarian morality when the stakes are high from a utilitarian point of view. 

I agree, and I think this analogy works: a company that sets out with the mission to "maximize shareholder value" probably won't maximize shareholder value. Most of the world's most valuable companies (that actually appear to have succeeded in maximizing shareholder value) have (fairly credibly) defined more meaningful missions. 

And I’m nervous about what I perceive as dynamics in some circles where people seem to “show off” how little moderation they accept - how self-sacrificing, “weird,” extreme, etc. they’re willing to be in the pursuit of EA goals.

There is an episode in the life of the Buddha where he believes that it would be best for him to eat very little. He does so and:

When I wanted to touch the skin of my belly I felt my backbone, and when I wanted to touch my backbone I felt the skin of my belly; in fact the skin of my belly stuck to my backbone because of eating so little

But it does not work for him:

And yet by means of this severe, harsh practice I had achieved no special knowledge and insight beyond the capacity of human beings and worthy of the noble ones: might there in fact be another path to awakening? Then it occurred to me that I remembered that when I was sitting in the cool shade of a rose apple tree while my Sakyan father was engaged in work, I had spent time having attained the joy and happiness of the first absorption, which is accompanied by thinking and examining, and is born of seclusion: might this in fact be the path to awakening?

He concludes:

‘It occurred to me that it would not be easy to achieve this happiness with a body that had become so extremely emaciated, so I thought I should take some solid food, some rice gruel. Then I took solid food, some rice gruel. Now at that time there were five monks with me, who thought: “The ascetic Gotama [the Buddha] will inform us of any Truth he achieves.” But after I had taken some solid food, some rice gruel, those five monks lost their enthusiasm for me and left, thinking: “The ascetic Gotama is one for excess. He has given up the struggle and reverted to a life of excess.”

While it's not common in my experience for EAs to intentionally deprive themselves of food (at least not nowadays), they do sometimes tend to deprive themselves of time for things other than impact. Perhaps they shouldn't. 

And be wary of being one of those monks. 

(From The Dialogue with Prince Bodhi)

Thank you very much for writing this post.

Thanks for writing this, Holden!  I agree that potential harms from the naive (mis-)application of maximizing consequentialism is a risk that's important to bear in mind, and to ward against.  It's an interesting question whether this is best done by (i) raising concerns about maximizing in principle, or (ii) stressing the instrumental reasons why maximizers should be co-operative and pluralistic.

I strongly prefer the latter strategy, myself.  It's something we take care to stress on utilitarianism.net (following the example of historical utilitarians from J.S. Mill to R.M. Hare, who have always urged the importance of wise rules of thumb to temper the risks of miscalculation).  A newer move in this vicinity is to bring in moral uncertainty as an additional reason to avoid fanaticism, even if utilitarianism is correct and one could somehow be confident that violating commonsense norms was actually utility-maximizing on this occasion, unlike all the other times that following crude calculations unwittingly leads to disaster. (I'm excited that we have a guest essay in the works by a leading philosopher that will explore the moral uncertainty argument in more detail.)

One reason why I opt for option (ii) is honesty: I really think these principles are right, in principle!  We should be careful not to misapply them.  But I don't think that practical point does anything to cast doubt on the principles as a matter of principle.  (Others may disagree, of course, which is fine: route (i) might then be an available option for them!)

Another reason to favour (ii) is the risk of otherwise shoring up harmful anti-consequentialist views. I think encouraging more people to think in a more utilitarian way (at least on current margins, for most people--there could always be exceptions, of course) is on average very good.  I've even argued on this basis that non-consequentialism may be self-effacing.

That said, some sort of loosely utilitarian-leaning meta-pluralism (of the sort Will MacAskill has been endorsing in recent interviews) may well be optimal.  (It also seems more reasonable than dogmatic certainty in any one ethical approach.)

One larger point to make here is that there may be no one true morality, ie that moral realism is false, and insofar as such a thing is correct, then every morality becomes essentially ideological, and there's no special claim for anyone, including EA to have any morality as truth. This would suggest very different actions from today.

This is also a reason to be wary of moral progress theories.

Also, this is why the future could be lovecraftian in it's morality, assuming the AI Alignment project goes well, in both it's positive and negative senses, since extremely high-end technology like genetic editing and digital people will warp our moral conceptions of personal identity, species, happiness and more.

Finally, I think that honesty as a morality should bend to the risks of information hazards, where it's better off if society or individuals didn't know something about it, where the misleading explanation is better than the truth, also known as strategic lies. AI safety is cursed here, but some other fields might have this problem.

Over the long-term for long-termism, extremes, not moderates win in the long-term.

PS: I actually think the politicization of the movement, alongside PR scandals, is plausibly a long-term threat to the movement.

>Maximizing X conceptually means putting everything else aside for X - a terrible idea unless you’re >really sure you have the right X.

Disagree. This sounds like a rejection of expected value. You only "put everything else aside" if you are sure that "everything else" doesn't matter. If you aren't sure then put some weight into "everything else". If you have any conception of better and worse using EV is tautological.

I have no issues with a movement stating it's normative framework and maximizing based off of that. If I don't (fully) agree with it's framework then I won't (fully) support it. To be clear though EA absolutely isn't stating its normative framework. 

If your ethical framework causes you to act as a "low integrity" or "bad person" and you see these things as very bad then use a different framework. If other people don't think low integrity is a bad thing then that's just what they think. If you want to stop them you need to make a deal with them or use force or convince them why that isn't actually rational under their framework. 

>We’d have a bitterly divided community, with clusters having diametrically opposed goals.

I see this as inevitable. If people have different values they can't live in agreement forever. There is no reason to believe that we all have the same values. We should of course engage in moral trades to fix common ground while there is common ground to be improved. At some point there isn't though, and deciding not to maximize because you will be fighting with others is just a losers mentality. Whoever does maximize will imprint their values into the world more. 

Agree with this post! It's nice to see these concerns written down.

Deeply agree that "the thing we're trying to maximize" is itself confusing/mysterious/partially unknown, and there is something slightly ridiculous and worrying about running around trying really hard to maximize it without knowing much about what it is. (Like, we hardly even have the beginnings of a science of conscious experience, yet conscious experience is the thing we're trying to affect.).

And I don't think this is just a vague philosophical concern — I really do think that we're pretty terrible at understanding the experience of many different peoples across time/which combinations of experiences are good/bad and how to actually facilitate various experiences.

People seem to have really overconfident views about what counts as improvement in value, i.e., many people by default seem to think that GDP going up and Our World in Data numbers going up semi-automatically means that that things are largely improving. The real picture might be much more mixed — I think it'd be possible to have those numbers go way up while things simultaneously get worse for large tracts (or even a majority of) people. People are complicated, value is really complex and hard to understand, but often people act as if these things are mostly understood for intents and purposes. I think they're mostly not understood.

More charitably, the situation could be described as, "we don't know exactly what we're trying to maximize, but it's something in the vicinity of 'over there', and it seems like it would be bad if e.g. AI ran amok, since that would be quite likely to destroy whatever it is that actually is important to maximize". I think this is a fine line of reasoning, but I think it's really critical to be very consciously alert to the fact that we only have a vague idea of what we're trying to maximize.

One potential approach could be to choose a maximand which is pluralistic. In other words:

• Seek a large vector of seemingly-important things (e.g. include many many detailed aspects of human experience. Could even include things which you might not care about fundamentally but are important instrumental proxies for things that you do care about fundamentally, e.g. civic engagement, the strength of various prosocial norms, ...)
• Choose a value function over that vector which has a particular kind of shape: it goes way way down if even one or two elements of the vector end up close to zero. I.e., don't treat things in the vector as substitutable with one another; having 1000x of item A isn't necessarily enough to make up for having 0 of item B. To give a general idea: something like the product of the sqrts of the items in the vector.
• Maintain a ton of uncertainty about what elements should be included in the vector, generally seek to be adding things, and try to run processes that pull deep information in from a TON of different people/perspectives about what should be in the vector. (Stuff like Polis can be kind of a way to do this; could likely be taken much further though. ML can probably help here. Consider applying techniques from next-gen governance — https://forum.effectivealtruism.org/posts/ue9qrxXPLfGxNssvX/cause-exploration-governance-design-and-formation)
• Treat the question of "which items should be in the vector?" and "what value function should we run over the vector?" as open questions that need to be continually revisited. Answering that question is a whole-of-society project across all time!

I strongly agree with this post.

Thinking consequentially, in terms of expected value and utility functions, will make you tend to focus on the first-order consequences of your actions and lead to a blind-spot for things that are fuzzy and not easily quantifiable, e.g. having loyal friends or being considered a trustworthy person. 

I think that especially in the realm of human relationships the value of virtues such as trust, honesty, loyalty, honor is tremendous - even if these virtues may often imply actions with first-order consequences that have 'negative expected value' (e.g. helping a friend clean the kitchen when you could be working on AI alignment). 

This is why I try to embrace deontological frameworks and heuristics in day-to-day life and in such things as social relationships, friendships, co-living etc.: Even if the upside of that is hard to quantify, I am convinced that the value of the higher-order consequences of it far outweigh the 'first-order inconvenience/downside'.

Thanks for writing this. My comment is on how the ideology here overlaps with practical community building efforts.

I think avoiding being extremely hard core in the ways described here have a lot of synergy with making EA a Big Tent movement. Luke argues that most folks who find EA take a gradual approach to becoming engaged and making bigger commitments. EA culture ought to leave room for different levels of commitment in ideology and in personal commitment.

There's also the possibility that a maximum doesn't exist.

Suppose you had a one-shot utility machine, where you simply punch in a number, and the machine will generate that many utils then self-destruct. The machine has no limit in the number of utils it can generate. How many utils do you select?

"Maximise utility" has no answer to this, because there is no maximum.

In real life, we have a practically infinite number actions available to us. There might be a sense in which due to field quantisation and finite negentropy there are technically finite actions available, but certainly there are more actions available than we could ever enumerate, let alone estimate the expected utility for.

In practice, it seems like the best way to actually maximise value is just to do lots of experimental utility-generating projects, and greedily look for low-effort, high-reward strategies.

This problem of consequentialism applied to real human problems will always be there as long as what is "good" for others is not defined by those others, for themselves. Its impossible to determine what is or isn't good for another person, without abstracting their agency away - which means whatever conclusion you come to about what is "good" for them, will always be flawed. 

There are a lot of things we can say with certainty are nearly universally understood as "good" - like being able to live, for instance. It means EA isn't in conflict with the right direction, for most, at this moment in time, because its giving and "good" work is largely focused on these sort of lower level, fundamental human and animal welfare problems. As you progress however,  which is all basically theoretical at this point,  this is where what is "good" becomes more and more grey and you encounter the problems created by having ignored the agency of those others. 

As you get closer to a potential maximum benefit, I'd suggest you'd realize that the only real "good" or benefit an altruist can pursue is the maximization of agency for others. People with agency choose life, they choose better living, they choose personal progress, they can better see suffering, they choose to acknowledge the agency of others and if this sort of agency as good is the motivation of EAs, then you are far less likely to have to deal with fanaticism issues. When was there ever a harmful fanatic who's MO was "I want to give everyone else all the power"? 

I agree with this concept objectively. As in many things in life, the truth is probably complicated - even if we can simplify it with an analogy. Depth and quality matter as much as quantity.

When you're looking at "downstream" vs "upstream" solutions (immediate consequences vs root issues), the lens of maximization is going to have a different impact. The efficiency of energy spent, the effort and markers identified and measured; all would be different depending on what you are paying attention to. It's difficult, but being able to hold space for diverse and even conflicting opinions (within the mindset of "yes &") is something I see a lot of beuracracies struggle to do. In IFS (a therapy that treats aspects our inner selves as "parts"); finding ways to validate concerns can ultimately lead to better consensus and creative ways of resovling tension - if you apply this concept to an org, arguing internally will maintain tension whereas vaildation may lead to precious refinements/insights that shift the goal slightly.

In short: Layers of mindfulness, openness to conflicting perspectives, and discussion about the positive impact qualities may help embody "good maximization" without becoming too narrow minded or solution jumping. 

"Certainty is the antidote to learning" - High Performance Habits 

That came to my mind when reading this post (which I really liked (so much that I don't have much else to add:)).