EA isn't immune to money, status and power

There are many ways in which EA could have handled it better and I agree with your headline: "EA isn't immune to money, status and power".

At the same time, I really want to push back against the simplistic narrative that Wytham Abbey was obviously a silly and ineffective thing to buy^[1]. Lots of people were skeptical that £14.9 million pounds could possibly be well spent buying a historic building when it could have saved like 4000 lives based on GiveWell figures. I think it should be quite clear today that this was way too dismissive.

EA correctly called AI as one of the largest issues of our time early and, by taking this gamble, has managed to achieve dramatic influence over the way this is playing out. This grant was a bet on purchasing a nice venue for retreats could allow EA ideas to become more influential. Given how influential EA ideas have become subsequently, it feels plausible that this kind of bet could have worked out. There's definitely a part of me that views this way of influencing the world as shallow, but, at the end of the day, the world is how it is, and you can have a lot of impact if you can bring the right people together.

Now, it may turn out that the influence of EA on AI is actually net-negative. I put a decent amount of weight on this possibility. At the same time, I mostly don't think the folks criticising the castle should get to claim this as a victory^[2]. On the contrary, I consider the whole castle brouhaha to have been a distraction from more substantial issues in terms of integrity.

Many people didn't necessarily object to the castle in and of itself, but more as a PR risk. I think Will MacAskill does a great job of explaining why the "PR mindset" was likely a mistake: "Post-FTX, I think core EA adopted a “PR mentality” that (i) has been a failure on its own terms and (ii) is corrosive to EA’s soul".

It wasn't obvious at the time, but it seems pretty clear to me now that a lot of EA's post-FTX reputational issues were actually the result of EA-on-EA fire and then various folks or groups being unwilling to sufficiently back themselves (if you're not willing to back yourself, why would anyone else back you?) rather than anything that happened. Honestly, I think that a lot of the infighting that occurred after FTX was low integrity behaviour, in particular, people taking advantage of the moment to litigate their grievances. Now if this is true, and I think it is, it creates a bit of a bind. On one hand, criticising the motives of critics most typically ends up sliding into a group becoming more insular and closed-minded. On the other hand, giving bad faith critics free reign without ever calling out their bullshit can be disasterous too, as the near-collapse of EA demonstrated. After all, who wants to be part of a community if there's an active effort to tar the community's reputation, the community itself is full of drama and almost no-one is sticking up for people who are fighting their hardest to make the world a better place?

Just to be clear, there are extremely strong reasons be wary of pushing back against criticism. It's very common for this to start off reasonably and then escalate. At the same time, reversed stupidity is not intelligence. Just because the default is for communities to mess up their epistemics by being overly reactive against critics, doesn't mean that you can swing all the way to the opposite end and everything will be fine and gandy. For a long time, it worked rather well, but then the community obtained money and power and the wrong people flooded in and then when things blew up, they took the opportunity to litigate their petty grievances and the EA community that had previously been able to operate on an assumption of good faith was mostly caught flatfooted and let things play out by default, when we needed to be defending ourselves. (Lucky for us most of the people with the wrong motivations left shortly after).

You linked to your comment here as an example of EA doing poorly on conflicts of interest. Honestly, I think a) you're wrong and that Claire had an extremely reasonable explanation b) your criticism focused on prioritised non-mandatory process over the deeper principles at stake c) I honestly think you should apologise to her and admit you were wrong d) maybe there's enough uncertainty that an apology wouldn't be justified, but I still think it's at least somewhat arrogant for you to be acting as though you were clearly in the right here.

Just to be clear, I'm not trying to put you in the crowd of "wrong people" or "bad faith critics", you seem pretty good faith in this post and I can't remember if I've read any of your other comments or posts so I'm going off this post.

But from what I've read, I suspect that you want the community to focus a bit too much on process over real virtue or integrity. Just to be clear, I'm not trying to dismiss the importance of rules, you can follow almost all the rules, but then let one slide and create major legal trouble for yourself and your organisation. Additionally, there are certain norms that are important, not so much because of legal reasons, but because not following them would be a serious breach of trust^[3].

My position is simply that I don't think we should take this so far that we let processes colonise our minds as to what true integrity looks like. If money, status and power draw the wrong people to the community then adding just a tiny bit of extra process around the margins isn't going to save you (but it may distract you).

Don't get me wrong, I think what you're doing here is great (honorable even)!

Accordingly, I am deliberately selecting Board members with a specific brief to be ethical watchdogs within the organisation. I have approached people I believe to be unusually high integrity, and unusually willing to challenge or confront me if necessary.

Second, I am asking some people to become personal ethics advisors to me, completely separate from the Board. This is because I think the bar for someone to approach a Board member with feedback is quite high. I think that external ethics advisors might be able to receive more feedback, including “epistemically rigorous gossip”, than charity directors. They might also be perceived as more impartial.

So where do I disagree:

a) "These are easier to mitigate, because we can choose to change our own behaviours more easily than we can gatekeep the community" — maybe we should avoid solutions closer to gatekeeping because they would be infeasible or they would backfire, but I certainly wouldn't make a decision here merely on the basis of one path being "easier".
b) "If you prefer to appoint people you trust personally and who share EA values to positions of responsibility, they will be conflicted or appear to be conflicted more often than in bigger communities" — We should avoid these conflicts when we can; own it when we can't or when the cost would be too high. Again, we need to comply with all relevant laws, mistakes here can be very costly, but it's worth understanding that laws are designed for a low trust environment (ie. all of society). We can't let this completely colonise our mindset. Some people want us to shift much further towards low trust norms, I feel this is defeatist and so I oppose.

1. ^{^}

   It may have been a bad investment if it were underutilised or if it didn't deliver significant value over hiring external venues.

2. ^{^}

   I said "mostly" because this depends on the specifics of what they said.

3. ^{^}

   I haven't thought deeply about this yet, but I wonder whether thinking about whether an action would be a "breach of trust" with the public would be a healthier mindset than PR mindset but I haven't thought deeply about it. Arguably, the recommended PR response to many situations would be at least a minor breach of trust because you're optimising your interaction with the person to obtain an outcome rather than genuinely engaging with them.

Thank you for writing this up, Jack. I think this is a very important topic. As a board member and organizational leader, I've thought a lot about these things, and I've been meaning to write more about this for a long time, but I've struggled with the nuances. I'll quickly try here. 

There are robust measures like an independent board with the ability and standing to evaluate and even fire the CEO (not only based on written rules, but also on a holistic picture of their conduct). I often come back to "Nonprofit Boards are Weird", which I think sums up the responsibility quite well. One concrete thing that I've been missing, and that has been on my list of ideas, has been to find ways for board members to upskill better and learn from the ones who have dealt with challenging and delicate situations. When I was recently advising on a round of board member hirings, I noticed how hard it is to find a group of people who don't have conflicts of interest, who have a diverse set of points of view, and who have also experienced the kind of situations where they didn't just have to tick boxes and follow scripts. 

I'm myself very grateful to have a board that took my evaluation seriously and interviewed founders, former colleagues, and staff to get a complete picture before confirming me and my role.

For me as a leader, an important aspect is having a community of peers - leaders who meet weekly and discuss concerns around integrity and who care deeply about this topic. I found these people through the Integral Altruism community, and through that, I also attended a retreat called Integrity Institutions this year. Part of our discussions there, with value-driven people who have gone into politics and organizations that are not mission-driven, has been that it is easy to be corrupted in different ways and get sucked into certain forms of status-seeking. This doesn't have to mean money or direct power-seeking, but more subtle forms of being motivated by the number of direct reports, the amount of funding you raise, or the kind of people you can hire in your organization, instead of thinking about impact first principles. It's easy not to notice how these factors influence you, so having time and a group of people who can help you reflect has been very important for me. 

"Don’t trust common-sense morality fully - but trust your deviations from it even less." (Toby Ord at EAG Bay Area 23) seems like one thing that points in the direction that I find useful. Developing this common-sense morality could involve metacognition, learning from wisdom traditions, and being able to take different points of view and life experiences. I hope we can find more people who have dealt with the kinds of situations we sometimes struggle with and bring them in as advisors, board members, and leaders.

This is a piece worth a lot of reflection, and I really admire the steps you’re taking!

I worry also about culture and individual role modelling might not be enough to prevent the worse forms of status-seeking & greed. I don’t think that a norm of whistleblowing policies and independent boards would’ve stopped SBF—I mean, we know that Tara Mac Aulay quit Alameda way back in 2018 and warned others about it.

I think people with power in this space (funders, evaluators, incubators, implementers) need to be on the lookout for charlatans. We should narrow down which inputs a bad actor could manipulate to access our power: For a funder, this is in evaluation, due diligence, etc. For an implementer, this might be in your hiring process. Validate your assumptions and lock these things down. ‘Assume good faith’ is a great rule to have when your movement is small—‘Trust, but verify’ needs to replace it.

We also need to acknowledge—and I don’t believe this was done well after the FTX collapse—that bad actors do not have to have ill intent. I think it might be just as harmful to exaggerate the impact of your programme, or to fail to run M&E on it appropriately, whether you are deliberately concealing and defrauding donors, or just being conveniently neglectful. There’s a selection bias here with a long tail; for every outright fraudster losing billions, there can be hundreds of little exaggerations and not-being-completely-upfronts costing millions each.

When I feel the tug of these incentives myself, I’m often a little surprised that there aren’t a lot of checks and balances on these kinds of deceptions within EA. I hope that with some amount of the money that comes into the space, that we set much higher standards for ourselves and independently hold others to account.

Thanks Jack, I think this is a really important topic and appreciate you writing about it. Some quick thoughts below - I'm sure I'll have more to say over time.

At an organisational level: I think that your suggestions around governance and policies are a good and obvious place to start. For example, at EAA, we already have conflict of interest and whistleblower policies, and an independent board, who I meet with very regularly. I'm not sure all of these policies are online, but I will aim to publish them on our new website, up in mid-November.

On a personal level: I have thought a lot about the narrative of "power corrupts" and thinking about how this might ever apply to me, and what warning signs I should look for in myself, or others around me that should be obvious warning signs. Some very quick examples: clear motivated reasoning, evasiveness when asked direct questions, not taking the concerns of people who are less "powerful" into consideration when they are a relevant stakeholder etc. I plan to build this into my ongoing reflections of my leadership. It's important to me to use my power and influence responsibly, and in ways that are aligned with my values. 

I think external accountability is much stronger than personal accountability though, and am grateful to be working with my board of directors to ensure strong governance and accountability.

Really interesting piece, Jack. Thanks for sticking your neck out there and being willing to tackle the tough questions. Having spent time thinking about EA governance (thanks for the link above), I think things have hugely improved, particularly at the larger orgs. The challenge in my view is the proliferation of small 1-3 person organisations with little or no trustee board and little follow-up from funders. I firmly believe everybody needs a boss to deliver to their full potential. This is based on my own personal experience (I get more done when someone is chasing me for quality output) and conversations with friends within EA. The model that some funders have of granting money and then never following up is not good for effectiveness, even before you consider potential fraud.

Thank you for raising this issue, Jack, I share the concern regarding governance and the right accountability mechanisms, particularly as our community is expected to have a lot more money at its disposal. I liked how you not only diagnosed the problem but also shared your own actionable mitigation mechanisms. I have two questions on them and one suggestion:

1. There are many different interpretations of what a non-profit board should do, but at its very core it has the power to change the CEO. Given how founder-driven philanthropic advisories tend to be, do you think that Ultra’s board would realistically have that power of you? If not, what other mechanisms would it have to push back, especially when you disagree?
2. I like the concept of personal advisors and I’m curious how you expect it to work in practice:

• How will they gather information about your decisions and actions? Is it self-reported, or is there a structure or cadence for proactively getting external feedback
• Beyond your personal respect for the advisors, what mechanisms will they have to influence your actions if you disagree? Also, if you have more than one advisor and they disagree with each other, how is that resolved?

Suggestion: EA, as any group, is likely to have shared biases and blind spots. To reduce this risk, I’d suggest including non-EA people both on your board and among your ethics advisors.

I recognise these are ‘what-ifs’ and you might be still figuring out the answers yourself. I’m interested in the operational details because I would love to see similar steps taken across our community so that we can learn together how to bring more accountability to the community. Really glad to see you bringing this discussion forward!

Thanks for writing this Jack, feels like the right time to be having this conversation.

The thing that concerns me most isn’t donors with an obviously different agenda, but donors who share most of our values gradually pulling priorities towards theirs, because it’s much harder to spot.

Imagine someone who seems aligned with our mission gives $100m. If, down the line, they want to be part of a strategy conversation, we’re very unlikely to say no.

However “unrestricted” a donation is, large gifts rarely come without strings, so I think this needs an organisational defence, not just personal integrity. I’d add a fourth policy to your conflict of interest, whistleblowing and compensation policies: a published gift acceptance policy. It would say what conditions we’ll accept, what access a donor does and doesn’t get, and who decides. It’s much easier to write that now than mid-negotiation with a nine-figure donor. It also turns saying no from a personal act of courage into “that’s our policy”, which is far easier to do.

I think it is simultaneously true that:

1. EAs are unusually trustworthy, virtuous, high-integrity people, who are unusually unlikely to be (as you rightly note) "motivated by personal financial gain".
2. Despite 1, there will no doubt be some existing bad actors within EA, and the influx of money may attrat more.
3. More importantly, it is still beneficial to have sound governance even when the people involved are well-intentioned and virtuous. Partly for optics, and partly so that it is someone's job to spot and correct mistakes.

I definitely don't want to lose sight of 1, and how wonderful that is. But I agree that as EA grows in money and power, good internal governance will become more important.

One minor point of hope is that as more funding comes online, from more different sources, orgs may well feel more able to criticise any single funder (e.g. cG) when previously that would have been even more costly. I'm not sure how large that effect is. I also expect that as money becomes less scarce, founders and org-builders will increase more in status, as they will be (seen as) more of the limiting factors.

As a newcomer to EA I am not able to give much input, but as a psychologist I can tell a few things about money, power and status.

The most important is what you described: we are fallible. We care about social status because is inherent in our DNA as a means to survive within the tribe. And rather than deny it, the best we can do is admit it and embrace that side of humanity. Because then we will be able to control it, instead of being controlled by greed.

I truly believe you can pursue selfish and altruistic goals at the same time. The issue is what you do when both are in conflict, how can you reconcile those motivations.

Maybe, doing most good requires also caring about your selfish motivations instead of fighting against them along the way (so that part within ourselves will not sabotage us). Being honest with yourself in the first place, and practice pure honesty and transparency with those outside. 

Pursuing money, status and power is not the problem. Pursuing them secretly using EA or other people as a means, is the big concern. 

Great write-up, Jack. Appreciate you taking the time to post this 'public good' when it takes time away from other things. 

On status, I feel funny about that yellow GWWC diamond. I feel a bit uncomfortable about it sometimes since it's like purchasing moral status. It's like purchasing a blue verified tick on X, I guess - (as I understand) before you had to be of actually a certain status to have people try to impersonate you, and you would earn that blue tick. But now I'm pretty sure you can just pay for it. FWIW I am on a trial pledge but feel like my motivations aren't right if I were to put the blue diamond everywhere, so I am totally susceptible to this thinking. Also this leads to some people's first impression of EA being very money and donation-centric, which for me as a layperson I would associate with cult behaviour. 

 I'm sure these feelings are valid but EA being so rational will counter these with "oh well but money was donated to a good cause at the end of the day"