Rationalist Epistemics and the Sequences (Effective Altruism Definitions Sequence)

Effective altruists are heavily influenced by the rationalist^[1] community, which formed around AI researcher Eliezer Yudkowsky’s series of blog posts The Sequences. In particular, the rationality community has shaped effective altruists’ ways of reasoning.

Here I’m providing an opinionated history of the rationalist movement. I was here for all of it: I’ve been reading LessWrong since it was part of Robin Hanson’s blog Overcoming Bias. But I have a particular idiosyncratic set of experiences; other people might have different experiences. And history always involves imposing a coherent narrative on a disorganized series of events; I emphasize some things, deemphasize others, and leave others out entirely. I hope I offer some enlightenment anyway.

I split this post into two parts because it got long.^[2] The first part covers what we got the Sequences themselves; the second part covers later developments in the community.

Bayesianism

In between when I conceived of this series and when I started writing this post, dynomight wrote a really good article that said much of what I would have said in this section. So go read that one.

For those of you who refuse to click links, here’s the short version: philosophical Bayesianism^[3] is the idea that the probability of an event is a fact about an individual’s state of mind. For example, when I say “there is a 50% chance that this coin will come up heads,” what I mean is something like “I expect that, when I make statements that I am as certain of as I’m certain of this coin coming up heads, I will be right 50% of the time.”

Philosophical Bayesianism means that you can make statements like “there is a 55% chance that Donald Trump will win the election,” “there is a 10% chance of nuclear war this century,” or “there is a 95% chance my husband will come home from the store with the quantity of tomatoes requested.”^[4] Under frequentism, the other popular theory of how probability works, these statements are meaningless: you can flip a coin thousands of times to find out how often it comes up heads or tails, but you can’t rerun the next century thousands of times to find out how often there is a nuclear war.

From an effective altruist or rationalist perspective, this kind of prediction is, in a certain sense, the thing that knowledge is. When I say that I understand physics, what I mean is that I can make accurate predictions about physical systems, and also about the contents of physics textbooks. When I say that I understand my husband, what I mean is that I can make accurate predictions about whether he will pick up tomatoes, what kind of movies he likes, his feelings about Donald Trump, whether he wants to take our child camping, etc. When I say that I understand Shakespeare, what I mean is—I don’t know—that if you showed me an actual Shakespearean sonnet and a ChatGPT Shakespearean sonnet, I could pick out which of them Shakespeare wrote.

For this reason, effective altruists and rationalists are interested (probably out of proportion to their importance) in prediction markets and other ways of aggregating predictions. They tend to give a lot of status to so-called “superforecasters”, who have a track record of making accurate predictions. For a particularly glaring example, Peter Wildeford’s Substack about page gives pride of place to his status as a top 1% forecaster, but omits such minor details as the fact that he cofounded effective altruist think tank Rethink Priorities.

Effective altruists also believe in reasoning under uncertainty. You are never perfectly certain of any statement, because there is always some information that could convince you otherwise. If you took two objects and two objects and counted them and got five, over and over again, at some point you would be convinced that 2+2=5.^[5] Most people want to say that something either is true or is not true. Rationalists and effective altruists say things like “I have a 60% probability that this is true, but I don’t know much about the area so I expect my probabilities to bounce around a lot as I learn more.”

For this reason, effective altruists think about decisions in terms of expected value. Imagine that someone offered to give you $10 if two coins of your choosing came up heads, in exchange for you paying them $1 if they didn’t. You will probably earn money (that is, the bet has positive expected value). That’s true even though you believe that the coins probably won’t both come up heads. Here are some examples of effective altruist reasoning using expected value:

• “I think there’s a 40% chance of artificial general intelligence in the next five years, and otherwise it will take decades. I’m focusing all my work on the five-years scenario—if it takes longer, we have plenty of time to figure this stuff out.”
• “I think there’s maybe a 5% chance that shrimp feel pain, but if they feel pain we’re torturing them. No shrimp scampi for me, please.”
• “Oh, there’s a 1% chance my startup will succeed, but if it works I’ll be a billionaire and if it doesn’t then ‘founder of failed startup’ looks great on a resume.”
• “I almost certainly won’t want to do comms as a career, but I’m going to take this weekend and write some blog posts to see if I hate it less than I’d expect. Value of information!”
• “I have prepared a list of two hundred strategies to cure my depression that won’t work and I’m going to try every last one of them.”

Cognitive Biases

The late 2000s, when the Sequences were written, was a heady time.

If you signed your name at the top of a form rather than the bottom, you were less likely to lie. You walked more slowly if you’d recently read words like “bingo” and “Florida.” If you sat with your legs on a desk, you’d feel more powerful and confident. You thought you were a rational person who made decisions for good reasons, but psychology had shown, incontrovertibly, that no such thing was true. You were a leaf on the wind, blown around by small details of circumstance over which you had no control.

The Sequences proposed that you could take control back. If you were aware of priming effects, maybe you could walk quickly even if you had recently read about Miami. At the very least, you could use it to your advantage: if your girlfriend is always complaining that you outpace her on casual walks, try thinking “old people” very hard before you go out. With your knowledge of the latest findings from psychology, you could become a rational superbeing!

…yeah. Well. That worked out well.

The replication crisis started to hit almost immediately after Eliezer Yudkowsky finished writing the Sequences. Psychology finding after psychology finding collapsed. At best, they came from a psychologist dredging the dataset for an accidental yet publishable correlation; at worst, they were outright fraudulent. And the Sequences clearly failed to make anyone a rational superbeing, or even noticeably more successful. A young Scott Alexander pointed this fact out as early as 2009.

For many years, some people continued to mess around with the one-easy-trick-to-become-a-rational-superbeing model of rationality, particularly in the workshops and curricula developed by the organization CFAR. But it became increasingly clear that, as Scott Alexander put it, extreme rationality is not that great. By the time I moved to the Bay in 2014, CFAR attendees routinely remarked that the primary benefit of CFAR was socializing in the alumni group.

Dispirited by the failure of mainstream rationality, some people developed stranger and stranger ideas, looking for the magic bullet. For example, many so-called “post-rationalists” asked whether perhaps all along the secret was balancing your chakras. Other people’s quest for the secrets of rationality resulted in tragedy. Over time, the approach of the mainstream of the rationality community shifted—a process I’ll talk about more in the next post.

So where does that leave the overall project of Trying To Think More Better?

Many of the famous One Easy Tricks, Backed By Science turned out to be laughable nonsense. But rationalists continue to embrace the concept of cognitive biases. Human thinking is distorted—just not by reading the word “bingo.” Instead, our thinking is distorted by:

• Tending to believe positive things about people we like and negative things about people we dislike.
• Attributing other people’s behavior to their fundamental personality and character, rather than their circumstances.
• Tending to believe things that make us feel good about ourselves (or, if depressed, things that make us feel bad about ourselves).
• Favoring members of groups that we’re also a member of.
• Automatically doing what we always do without thinking about it.
• Thinking of past events as more predictable than they really are.
• Dismissing “weird” ideas out of hand.
• Not knowing how to do math.
• Getting in a bad mood when we’re hungry or tired.

In a surprising number of cases, the magic bullet to rationality is having a snack.

None of these are as exciting as power poses or priming. Indeed, none of them are new to the rationalist movement; philosophers and moralists have warned against some of them for literal millennia. But you can in fact improve your thinking through being aware of and taking steps against these biases.

You can ask yourself “is it possible this person’s behavior is a reaction to their particular situation that doesn’t say that much about who they are as a person?” or “do I just hate this idea because it’s weird?” or “am I giving special treatment to people I like?” You can write down your thoughts and then consider the evidence for and against them: this is called “cognitive behavioral therapy” when you do it with a therapist, but it really does work in a surprising number of cases. You can recognize and plan around your irrationalities: for example, if you know you’ll always check your phone even when you really want to be focused on your friend, you can leave your phone at home.

Many rationalist and effective altruist organizations hire through blinded work tests: they give you a three-hour task to complete on your own that matches up with what you’ll be doing at work, and which is graded by someone who doesn’t know who you are. In my opinion, work tests are popular because people are concerned about cognitive biases. In interviews, people will favor their friends, or physically attractive people, or people like them. In a blinded work test, they can only make decisions based on the work. This approach to cognitive biases is less flashy than some of the stuff in the Sequences, but I think it is real and helpful.

The Sequences are long, idiosyncratic, and full of claims about both rationality and AI that turned out to be hilariously false; a cursory glance at Eliezer Yudkowsky’s X account will disabuse the reader of the notion that he’s an exceptionally rational person.^[6] But the classic posts—I talked about some of them here but also want to link to A Human's Guide to Words, much of Mysterious Answers, and Letting Go—were and are very influential on effective altruist thought, as well as my own. Check them out.

1. ^{^}

   In the sense of “rational”, not in the sense of “reasoning from your armchair instead of checking things empirically”. Rationalists are actually quite empiricist.

2. ^{^}

   Fortunately, this was the only such split and this series didn’t become Homestuckian in nature.

3. ^{^}

   The term comes from the theorem Bayes’ Theorem, a particular mathematical theorem the details of which are almost entirely irrelevant to this discussion.

^{^}

He did.

^{^}

And also be very, very confused about the nature of the universe.

^{^}

For one thing he has an X account.

Show all footnotes