I think welcoming/unwelcoming is one of those things that most people initially assess almost immediately upon contact with a community. Yes, people who stay in the community will update their perceptions over time, but I have definitely been to enough meetups, dances, and general social gatherings to have a sense after one interaction with a community of whether I feel welcome and to have noticed that this affects my probability of returning. It even affects my probability of returning if I go with a friend, or know a subset of people there; being welcomed by one or a few people is not necessarily enough if the community as a whole doesn't feel welcoming, even if those people are deeply connected in the community and think I will want to return.

I also think that what it means for long-time community members to feel like a group is welcoming is pretty unclear, because they themselves actually do not need to be made welcome there. I would be much more interested in thinking about whether EA is welcoming based on how welcoming newcomers (say, people who have participated in local or online EA groups at least once, but for no longer than a year) think it is, what proportion of newcomers return or bring friends, etc.

Thanks for being so open to feedback and non-defensive on this and thanks especially for updating the body of the post. I think there's a big problem where people change their minds due to feedback but it never gets propagated out to readers b/c it's buried in a comment section. With preliminaries out of the way. . .

This wasn't the tone I was going for. On my reading of the post, it's pretty hedge-y, with the exception of the title. Can you help me out by pointing out some ways that I seem overconfident in the empirical view?

Looking back at it, I do think the headline really primed me here. But I think the other things were: "the NPS score coupled with the referral evidence suggests that as a brand EA has a very dedicated fanbase that is willing to promote the brand." I see how this could feel like a nitpick because you said "suggests" and not "proves" but I think somebody reading the post quickly, glossing over the actual numbers, and trusting your description of their implications (which I think is pretty common/reasonable) would take this to mean that the NPS score unambiguously is evidence in this direction and the uncertainty is due to the fact that it's just one form of evidence. I'm not sure exactly what I would've said instead but I would've said something more moderate when describing data that's less than excellent on it's own terms. Taking out the phrase "very dedicated" would've made a pretty big difference, I think. I think it would've made a big difference if there was some explicit discussion of the fact that the NPS score was good-but-not-great and could reasonably have been better. Putting the data in there for comparison definitely helps but if I'm reading something about metrics I've never seen before, I kind of expect the writer to do the work for me and tell me how to interpret the comparison. If there's some data and then the author says it suggests "very dedicated fanbase" I'm likely to assume that the score EA got is relatively close to the score Apple got. If it's not, that seems like an important enough fact to grapple with instead of just present. *The other places you implicitly described what the data mean are "highly loyal community willing to promote EA" and "high brand loyalty." Combining all of this, I think the post really reads like EA killed it on the NPS front.

I don't see why this is a reasonable conclusion to reach. I think the welcoming/unwelcoming distinction is a claim about the experience of being in the community and interacting with EAs. Since new people haven't had a chance to interact with EAs, it would be surprising if they found the community unwelcoming. It could be that the EA brand prevents people from wanting to join the community in the first place. That seems like a hypothesis worth testing, but it doesn't seem to me like a claim about how welcoming the community is. Fair points but I’m not convinced.

I’d guess that there’s a lot of people who have had enough contact with the EA community to have been affected by its welcomingness or lack thereof but who wouldn’t have counted as applying to EAG b/c of someone else in EA according to your metric. People with relatively weak connections to EA are likely to be most affected by welcomingness so it seems possible that the relevant margin here is whether people who fall into the non-referral group feel welcome to apply. I think a major mechanism through which welcomingness has effects is welcomingness -> experience of people interacting with the community -> EA’s reputation/brand among people outside the community. So I’d actually expect welcomingness to have a big effect on whether EA has a brand that gets people to want to join the community in the first place. For a fairly big, somewhat outward facing event like EAG where the pool of potential non-EA applicants is so large compared to the pool of potential EA applicants, it seems possible that this mechanism through which welcomingness decreases proportion of applicants coming through referrals could dominate your proposed mechanism through which welcomingness increases the proportion of applicants coming through referrals. JTBC, I don’t have a net take on the above. My main point is just that the direction is ambiguous so I don’t think the data says much about welcomingness.

Agree 100%.

I think the founder effect explanation is definitely a big part of the story in terms of the demographic makeup. However, that does not mean being more welcoming won't help. As a white founder myself, I have learned the hard way that racial diversity in particular must be actively cultivated, and you can get yourself very deep into a hole after a short time because people start to notice the demographic makeup and make judgments and inferences about the community based on it. I was pleased to see more racial diversity at EA Global this year than I expected (although still very few black and Latino participants), but one thing I couldn't help but notice is that there were no non-white speakers at ANY of the sessions I attended over two days. That's the sort of thing that can be perceived as unwelcoming for someone who has made the commitment to attend a conference and has already spent a bunch of time being one of the few people in a room who looks a certain way, and it also shows that it's not just about who expresses interest in EA in the first place.

I think the point is that for NPS, we're interested in what all effective altruists think, since they're the users of the product. But EAG attendees are not likely to be typical effective altruists: they will probably be more committed, and more positive about EA than a typical EA is.

To continue the Apple analogy, it's a bit like basing your NPS score not on everyone that buys a laptop, but on the people who comment most on Apple product forums: these people won't be typical of Apple's consumers.

My current belief is something like "EA is unwelcoming to people not in the standard EA demographic". So based on that:

Weak evidence:

1. EA demographics have been moving towards "normal" (eg. less gender bias, more racial diversity, but probably still mostly relatively rich people). A priori, I would expect this to happen anyway, but at a pretty gradual rate, something like 1-5 percentage points per year for gender bias.

Moderate evidence:

1. Data that shows that people not in the standard EA demographic are just as likely to be committed EAs after a medium/high interaction that involved meeting other EAs in the real world (eg. physical meetups or EA Global).

2. Data that shows that people not in the standard EA demographic are just as likely to be committed EAs after a low/medium interaction (eg. a Giving Game). A priori, I would expect that they are just as likely to want to learn more, but are less likely to continue on the path after learning more and engaging with the community.

3. Something similar to the above two, like Ajeya's suggestion.

Strong evidence:

1. The majority of other local EA group leaders disagree with me.

For this issue specifically, I trust observations made by local group organizers more than I would trust large scale observational/correlational data, just because I can imagine so many different factors at play here that even if the data supported the hypothesis that EA as a whole is welcoming, I would still expect there to be several subfactors where we could and should still improve. (Though I could be convinced that it may not be worthwhile to do so.)

You could also ask these questions of EAG attendees who had relatively little contact with the community before attending.