Hide table of contents

This post offers some reasons why, on consequentialist grounds, a truth-seeking person with a primary goal of having a positive impact on the world might prioritize upholding certain norms that are widely held in their broader social and professional communities, even if the instrumental value of those norms isn’t obvious in the context in which they are applied.[1]

One of my goals in writing this post is to help explain a subset of the reasons why I think some people in the EA community believe Nick Bostrom’s email and apology and some statements in the subsequent conversation have been inadequate or harmful: The original email, recent apology, and some subsequent comments have violated norms that are widely followed by people who aim to have a positive impact on the world, including a norm against asserting that there are meaningful genetic differences between groups of people in traits like intelligence.  My impression is that some people engaging here may not be aware of the reasons why this norm exists or the potential consequences of violating it, while other people may assume that others know and take these reasons and consequences for granted.  While this post does not address the object-level reasons why that norm exists,[2] I hope that it helps explain some of the potential consequences of violating that norm and others like it.

I acknowledge that some people in the EA community prioritize a different norm—specifically, a norm in favor of communicating one’s own beliefs as accurately and directly as possible, sometimes called “epistemic integrity.”  I know many people sincerely believe that this norm is exceptionally important to the project of achieving a positive impact on the world, and I have tried to live up to it in writing this post.  I hope this post helps explain why others might believe that achieving a positive impact on the world (and even accurately communicating one’s own beliefs) sometimes requires giving priority to other norms.

Norms play an important role in coordination

Many ways of making a positive impact on the world depend on coordinating with other people.  One thing that is often important but hard to do when coordinating with other people is communicating accurate information about each other’s goals and beliefs.  

One way of communicating your goals and beliefs is to state them directly.  But norms also play an important role in communicating goals and beliefs.  When you follow a norm, you are providing evidence that your goals and beliefs are like those of other people who follow that norm.  Conversely, when you violate a norm, people might infer that your goals and beliefs are not like those of people who do follow the norm.[3]

People who aim to have a positive impact on the world might uphold certain norms as one way of communicating about their goals and beliefs.  They might also draw negative inferences from norm-breaking behavior by others.

Examples of inferences others might draw from norm-breaking behavior

If a person violates a norm, sincere and truth-seeking people might conclude that they don’t know about the norm itself, from which they might infer:

  • The person isn’t aware of important social or historical information underlying the norm, which might affect the person's ability to have a positive impact on the world.
  • The person doesn't interact very much with people who could have taught them about the norm, which might signal that they lack social ties with (or perhaps even harbor prejudices against) a group of people who might be able to help them have a positive impact on the world.

Other sincere and truth-seeking people might conclude that a person who violates a norm does know about the norm, but has chosen not to follow it, from which they might infer:

  • The person might not be trustworthy because they have shown that they are willing to violate an important norm; they might therefore be willing to violate other important norms like telling the truth and keeping promises.
  • The person might be unpredictable and difficult to coordinate with because they have shown they are willing to make a “one-off,” context-specific decision that deviates from a general rule of behavior that many other people follow.
  • The person might have other dangerous, difficult-to-observe traits that are more common among people who violate the norm than they are among people who consistently follow it.  (For example, the person might harbor racial animus or misogynistic beliefs.)
  • The person’s value system might be different in important ways because they seem to place higher priority on the first-order consequences of the action they are taking than on the higher-order consequences that an important norm is intended to prevent.

Norms can also be valuable when following “single-player” strategies

Some norms function like ethical injunctions and serve to protect people from predictable mistakes that people tend to make when reasoning from first principles.  People who aim to form true beliefs about how to have a positive impact on the world might prioritize upholding these norms even when following a single-player strategy and not trying to coordinate with others.

As one example, a person might decide to prioritize upholding social norms about how to allocate their time between work responsibilities, family responsibilities, and leisure.  Such a person might hypothesize that social norms on that topic encode valuable information about the long-term consequences of working too much (or too little).  They might also reason that they do not have clear guiding principles or “stopping points” to protect them from making mistakes if they deviate from the norms and try to allocate their time from first principles.

As another example, a person might decide to uphold social norms against trying to reason about a certain subject from first principles.  Such a person might hypothesize that social norms encode valuable information about the risk of being misled by poor-quality evidence about that subject or the risk that inquiries into the subject could be misused to cause serious harm.  They might also take the social norm as evidence that, on that subject, a different style of reasoning might be more productive than trying to proceed from first principles.

A final word on strategies for upholding norms with compassion

I have offered a handful of reasons why people who aim to have a positive impact on the world should uphold certain norms.  But sometimes people worry that they will lose social or professional status or suffer other sanctions if they accidentally violate a norm.  This dynamic causes some people to experience anxiety about the whole topic of social and professional norms, even if they are trying in good faith to have a positive impact on the world.  

Responding to norm violations with compassion can help uphold important norms without contributing unnecessarily to this kind of anxiety and worry.  Toward that end, here are some things you could consider doing if you notice that someone has violated a norm that you think is important:

  • Take the person who violated the norm aside and talk with them in private (e.g., through an email or direct message) about the norm.  Find out how much they know about the norm, including the reasons why it exists.  Point out the potential consequences of violating important norms for the person’s ability to achieve their goals.
  • Make a positive case for the norm by explaining the purposes it serves, either privately to the person who violated the norm or publicly to a relevant community.[4]
  • For serious violations, ask for an apology that demonstrates that the person who violated the norm understands the reasons why their actions might have been harmful and gives readers reasons to think the person who violated the norm won’t do so again.[5]
  1. ^

    Many people also endorse deontological arguments in favor of upholding certain norms, but this post does not focus on those arguments.

  2. ^

    If you are interested in learning more about this difficult subject, consider visiting this page from the U.S. National Human Genome Research Institute.

  3. ^

    Sometimes, people want to signal that their goals and beliefs are not like those of others who follow a certain norm.  However, before deciding to break a norm as a way of sending that signal, it is important to understand the goals and beliefs that others actually associate with the norm; they might not turn out to be the goals and beliefs you expect or intend to be communicating about.

  4. ^

    One goal of this post is to make this kind of positive, public case.

  5. ^

    I also liked the advice in this article about heartfelt apologies that titotal recently shared in another post.





More posts like this

Sorted by Click to highlight new comments since: Today at 3:04 AM

One more reason in favour of following most social norms is that breaking certain norms risks attracting people who enjoy breaking social norms for the wrong reasons, which can have a snowball effect on a community. 

As an example, if a community likes to make edgy misogynistic jokes, you will attract people who enjoy being edgy and misogynistic, and drive away women and people who don’t like edgy humour. This shifts the community norms towards edginess, making it more likely for more and worse misogynistic jokes to be made, that drive off kinder people, etc.  This edgy death spiral can end up with an entire community full of full-on misogynists. In my many years on the internet, I have seen this kind of pattern play out over and over again in different communities.  

If we want EA to be kind, empathetic, and truthful (As I obviously do), then we need to act in ways that are kind, empathetic and truthful, and hold ourselves and our leaders to those standards. 

Regarding: "a norm against asserting that there are meaningful genetic differences between groups of people in traits like intelligence".

I agree that the original email violated this norm, but I don't believe that his apology violated this norm. Now, I know you just said that this norm had been violated without specifically saying that both documents violated these norms, but the way this was written made it sound like both documents violated the norm, although this might not be what you meant.

I won't be surprised if you reply that his apology violated a second, very closely related norm and I think that would a very reasonable thing to point out.

However, I want to emphasise that precision here matters and that making progress on these kinds of challenging issues requires us to set this kind of level of precision as the benchmark. If we don't do this, then it's very easy for a number of these little, tiny issues to stack on top of each other and for us to end up believing things that are very far away from reality.

Curated and popular this week
Relevant opportunities