I really liked the piece. It resonated with my experiences in EA. I don't know that I agree with the mechanisms Sasha proposes, but I buy a lot of the observations they're meant to explain.
I asked Sasha for his permission to post this (and heavily quote it). He said that he hopes it comes off as more than a criticism of EA/rationality specifically--it's more a "general nerd social patterns" thing. I only quoted parts very related to EA, which doesn't help assuage his worry :(
So, I’ve noticed that a significant number of my friends in the Rationalist and Effective Altruist communities seem to stumble into pits of despair, generally when they structure their lives too rigidly around the in-group’s principles. Like, the Rationalists become miserable by trying to govern their entire lives through nothing but rationality, and the EAs feel bad by holding themselves to an impossible standard of ethics. [...]
This is a real shame, because these are some of the most productive, original, intelligent, charming, strange people in my world. When they’re depressed, they do their important work less effectively. And when they burn out, it is a major loss. And the burnouts can be dramatic—like, quit your life, do nothing for years, have no grounding principles whatsoever, eventually hallucinogen yourself back into self-acceptance. (That’s a reasonable way to spend time, but there’s probably a healthier middle way that doesn’t involve total personal collapse.) [...]
Today I realized that it’s generally much simpler than I thought previously. Most of it is just toxic social norms. These groups develop toxic social norms. In the Rationalist community, one toxic norm is something like, “you must reject beliefs that you can’t justify, sentiments that don’t seem rational, and woo things.” In the EA community, one toxic norm is something like, “don’t ever indulge in Epicurean style, and never, ever stop thinking about your impact on the world.”
Generally, toxic social norms don’t develop intentionally, nobody wants them to happen, they’re not written down, and nobody enforces them explicitly. (The intentional development of toxic social norms is otherwise known as founding a cult.) What happens is that there are positive social norms, like, “talking about epistemics and being curious about beliefs is cool,” or “try to be intentional about the positive impact you can have on the world.” These norms are great! But then, the group acts like a group, which is to say, people confer status depending on level of apparent adherence to values. This leads insecure people who completely depend on the group to over-identify with the set of values, to the extent that even slightly contrary actions become forbidden. Not forbidden in the like “we’ll arrest you” way, but in the like “everyone in the room immediately looks at you like you’re being rude if you talk about spirituality” way.
And then the second, more sinister stage occurs—the point at which these toxic norms are internalized such that they apply to you when you’re in a room alone. As Wittgenstein noted, it’s hard to tell where aesthetics end and ethics begin; it can start to feel unethical, like, dirty, to perform behaviors your peers would think distasteful. Toxic norms eventually pervade completely, to the point where you don’t even want to think bad thoughts.
Sometimes—often—these forbidden thoughts/actions aren’t even contrary to the explicit values. They just don’t fit in with the implied group aesthetic, which is often a much stricter, more menacing guideline, all the more so because it’s a collective unwritten fiction. “Rationality is cool” becomes “rationality is the best framework” becomes “Rationalist and Rationalist-flavored stuff is a better use of your time than anything else” becomes “it’s uncool if you want to spend a lot of time doing stuff that has nothing to do with testable beliefs, or our favorite issues.” This is all unintentional and implicit. No Rationalist has ever said, to my knowledge, that you shouldn’t write poetry, but a few Rationalists have told me that they feel like they shouldn’t make weird art because it’s dumb and un-Rationalist to do so—they feel they ought to produce useful thoughts instead, even though their hearts are trying to steer them somewhere else. I point out to them that Scott Alexander wrote a fantasy novel for fun, but somehow this isn’t persuasive enough.
Here, I should probably stop and define toxic norms. I think a toxic norm is any rule where following it makes you feel like large parts of you are bad. The EA version is thinking that you’re evil if your soul/body/emotions are crying out for you to relax, slack off a bit, and spend money on yourself, because you ought to be spending every possible moment working on human flourishing. I’ve heard tales of people struggling with their decision to buy a tasty snack rather than donate $5 to charity, and, more worryingly, people feeling guilty that they want to have children, since that would distract them from the work of Improving Humanity. This obviously leads to burnout and self-loathing. Meanwhile, the Rationalist version is thinking that you’re stupid and not worth talking to if you yearn for the spiritual/aesthetic/woo/non-justifiable, or if you haven’t been able to come to grips with your issues through rational means. This leads to emotional damage being ignored, intuition being dismissed, and systematizing being preferred inappropriately above all other modes of thinking and feeling.
One sign of toxic social norms is if your behavior does deviate from the standard, you feel that the only way of saving face is through explaining your behavior via the group values. Like, if you watch the Bachelor all the time, and one of your smart peers finds out about that, you might find yourself hastily explaining that the series is enjoyable to you as an applied experiment in evolutionary psychology, when, in fact, you just like social drama because watching humans freak out is fun. I will never forget hearing a Rationalist friend ask a non-Rationalist friend whether he loved riding motorcycles because it was an experiment in social status, rather than, y’know, vroom vroom fun thing go fast.
I’m not mentioning these communities because I think they’re extra toxic or anything, by the way. They’re probably less toxic than the average group, and a lot of their principles are great. Any set of principles, if followed too strictly and assigned too much social value, can become a weird evil super-ego that creeps into the deepest crevices of your psyche. (One must imagine the serene Yogi seized with crippling shame over their perfectly normal road rage.) These groups are just the ones I’m most familiar with right now, and thus the places where I see these patterns most often. [...]
Also, these norms aren’t toxic for everyone! There are a few people who are, in fact, happiest when they’re entirely, or almost entirely, devoted to the fancy intellectual principles of a specialized group. But this is not most people. And this can actually compound the problem! If there are people in the group who are perfect examples of the desired behavior, they can be positive exemplars, but also negative exemplars—constant reminders that you are falling short. (Also, certain group leaders can quietly, and probably unintentionally, inflect the norms in a subtle way, thus accentuating the degree to which they are seen as exemplary, and the degree to which others are seen as inferior.)
This is, perhaps, an inevitable danger for nerdy people. For lots of intellectual weird people that don’t fit in, their first social stage is rejection from society in general, and then, later on, their second social stage is finding understanding in a tightly-knit subculture. And they cling to this subculture like a life-raft and are willing—happy, even—to initially reject any parts of themselves that don’t fit within this new community. And their new peers, unintentionally, facilitate this rejection. They don’t feel that this is toxic, because they feel like they’ve already seen what social toxicity is: it’s the prime normie directive that we learn in school: don’t be weird, ever. [...]
And the people being deferred to—the senior members of the group—don’t want this dynamic at all, but they don’t necessarily notice that it’s happening, because the outward manifestation of this is people being really impressed by you. Like, if you’re big in the EA scene, and a young freshly minted EA can’t stop talking about how excited they are to do good, and how inspired they are by your virtuousness, there’s maybe no obvious sign that they’ve started rejecting every part of themself that is not congruent to this new identity. You would have no reason to worry about that. You would probably just feel good, and glad that your principles are so convincing. So it’s hard to even see this issue sometimes, let alone figure out how to solve it. (Although I’ve heard from Rationalist luminary Aella that some Rationalists are, indeed, beginning to take it seriously, which is great.)
I don’t know whether all of this can be avoided entirely. Part of it is just growing up. It’s regular Kegan Stage 4 stuff. You conceive of who you are by seeing the world through some epistemic/moral lens, usually the one relied upon by the group who abuses you least. Eventually, you notice the flaws in that lens, and then you become your own thing, clearly related to the group, but distinct from it, not easily captured by any label or list of properties.
This isn’t a toxic social norm. This is the point of rationality, is it not?
Ah. In one sense, a core part of rationality is indeed rejecting beliefs you can't justify. Similarly, a core part of EA is thinking carefully about your impact. However, I think one claim you could make here is that naively, intensely optimising these things will not actually win (e.g. lead to the formation of accurate beliefs; save the world). Specifically:
Sure, but that isn’t what the quoted text is saying. Trusting your gut or following social norms are not even on the same level as woo, or adopting beliefs with no justification.
If the harmful social norms Sasha actually had in mind were not trusting your gut & violating social norms with no gain, then I’d agree these actions are bad, and possibly a result of social norms in the rationality community. Another alternative is that the community’s made up of a bunch of socially awkward nerds, who are known for their social ineptness and inability to trust their gut.
But as it stands, this doesn’t seem to be what’s being argued, as the quoted text is tangential to what you said at best.
There are a few different ways of interpreting the quote, but there's a concept of public positions and private guts. Public positions are ones that you can justify in public if pressed on, while private guts are illegible intuitions you hold which may nonetheless be correct - e.g. an expert mathematician may have a strong intuition that a particular proof or claim is correct, which they will then eventually translate to a publicly-verifiable proof.
As an another example, in the recent dialog on AGI alignment, Yudkowsky frequently referenced having strong intuitions about how minds work that come from studying specific things in detail (and from having "done the homework"), but which he does not know how to straightforwardly translate into a publicly justifiable argument.
Private guts are very important and arguably the thing that mostly guides people's behavior, but they are often also ones that the person can't justify. If a person felt like they should reject any beliefs they couldn't justify, they would quickly become incapable of doing anything at all.
Separately, there are also lots of different claims that seem (or even are) irrational but are pointing to true facts about the world.
If this is what the line was saying, I agree. But it’s not, and having intuitions & a track record (or some reason to believe) those intuitions correlate with reality, and useful but known to be not true models of the world is a far cry from having unjustified beliefs & believing in woo, and the lack of these is what the post actually claims is the toxic social norm in rationality.
What makes you think it isn't? To me it seems both like a reasonable interpretation of the quote (private guts are precisely the kinds of positions you can't necessarily justify, and it's talking about having beliefs you can't justify) as well as a dynamic that feels like one that I recognize as one that has been occasionally present in the community. Fortunately posts like the one about private guts have helped push back against it.
Even if this interpretation wasn't actually the author's intent, choosing to steelman the claim in that way turns the essay into a pretty solid one, so we might as well engage with the strongest interpretation of it.
Because it also mentions woo, so I think it’s talking about a broader class if unjustified beliefs than you think.
I agree, but in that case you should say make it clear how your interpretation differs from the author’s. If you don’t, then it looks like a motte-bailey is happening (where the bailey is “rationalists should be more accepting of woo & other unjustified beliefs”, and the bailey is “oh no! I/they really just mean you shouldn’t completely ignore gut judgements, and occasionally models can be wrong in known ways but still useful”), or you may miss out on reasons the post-as-is doesn’t require your reformulation to be correct.
My earlier comment mentioned that "there are also lots of different claims that seem (or even are) irrational but are pointing to true facts about the world." That was intended to touch upon "woo"; e.g. meditation used to be, and to some extent still is, considered "woo", but there nonetheless seem to be reasonable grounds to think that there's nonetheless something of value to be found in meditation (despite there also being various crazy claims around it).
My above link mentions a few other examples (out-of-body experiences, folk traditions, "Ki" in martial arts) that have claims around them that are false if taken as the literal truth, but are still pointing to some true aspect of the world. Notably, a policy of "reject all woo things" could easily be taken to imply rejecting all such things as superstition that's not worth looking at, thus missing out on the parts of the woo that were actually valuable.
IME, the more I look into them, the more I come to find that "woo" things that I'd previously rejected as not worth looking at because of them being obviously woo and false, are actually pointing to significantly valuable things. (Even if there is also quite a lot of nonsense floating around those same topics.)
That's fair.