[Link] How effective altruists ignored risk

Reflecting on the piece as a whole, I think there are some very legitimate concerns being brought up, and I think that Cremer mostly comes across well (as she has consistently imo, regardless of whether you agree or disagree with her specific ideas for reform) - with the exception of a few potshots laced into the piece^[1]. I think it would be a shame if she were to completely disavow engaging with the community, so I hope where this is disagreement we can be constructive, and where there is agreement we can actually act rather than just talk about it.

Some specific points from the article:

• She does not think that longtermism or utilitarianism was the prime driver behind SBF's actions, so please update towards her not hating longtermism. Where she is against it is because it's difficult to have good epistemic feedback loops to deciding whether our ideas and actions actually are doing the most good (or even just doing good better):

"Futurism gives rationalization air to breathe because it decouples arguments from verification."^[2]

• Another underlying approach is to be wary of the risks of optimisation which shouldn't be too controversial? It reminds me of Galef's 'Straw Vulcan' - relentlessly optimising towards your current idea of The Good doesn't seem like a plausibly optimal strategy to me. It sounds very parsimonious with the 'Moral Uncertainty' approach.

"a small error between a measure of that which is good to do and that which is actually good to do suddenly makes a big difference fast if you’re encouraged to optimize for the proxy. It’s the difference between recklessly sprinting or cautiously stepping in the wrong direction. Going slow is a feature, not a bug."

• One main thrust around the piece is her concern with the institutional design of the EA space: 

"Institutional designs must shepherd safe collective risk-taking and help navigate decision-making under uncertainty..."

In what direction would she like EA to move? In her own words:

"EA should offer itself as the testing ground for real innovation in institutional decision-making."

We have a whole cause area about that! My prior is that it hasn't had as much sunlight as other EA cause areas though.

There are some fairly upsetting quotes about people who have contacted her because they don't feel like they can voice those doubts openly. I wish we could find a way to remove that fear asap.

"It increasingly looks like a weird ideological cartel where, if you don’t agree with the power holders, you’re wasting your time trying to get anything done.”

Summary:

On a second reading, there were a few more potshots than I initially remembered, but I suppose this a Vox article and not an actually set of reform proposal -  something more like that can probably be found in the Democratising Risk article itself.    

But I genuinely think that there's a lot of value here for us to learn from. And I hope that we can operationalise some ways to improve our own community's institutions so that the EA at the end of 2023 looks much healthier than the one right now.

1. ^{^}

   In particular, the shot at Cold Takes being "incomprehensible" didn't sit right with me - Holden's blog is a really clear presentation of the idea that misaligned AI can have significant effects on the long-run future, regardless of whether you agree with it or not.

2. ^{^}

   I think this is similar to criticism that Vaden Masrani made of the philosophy underlying longtermism.

The quality of reasoning in the text seems somewhat troublesome. Using two paragraphs as example 

On Halloween this past year, I was hanging out with a few EAs. Half in jest, someone declared that the best EA Halloween costume would clearly be a crypto-crash — and everyone laughed wholeheartedly. Most of them didn’t know what they were dealing with or what was coming. I often call this epistemic risk: the risk that stems from ignorance and obliviousness, the catastrophe that could have been avoided, the damage that could have been abated, by simply knowing more. Epistemic risks contribute ubiquitously to our lives: We risk missing the bus if we don’t know the time, we risk infecting granny if we don’t know we carry a virus. Epistemic risk is why we fight coordinated disinformation campaigns and is the reason countries spy on each other.

Still, it is a bit ironic for EAs to have chosen ignorance over due diligence. Here are people who (smugly at times) advocated for precaution and preparedness, who made it their obsession to think about tail risks, and who doggedly try to predict the future with mathematical precision. And yet, here they were, sharing a bed with a gambler against whom it was apparently easy to find allegations of shady conduct. The affiliation was a gamble that ended up putting their beloved brand and philosophy at risk of extinction.

It appears that a chunk of  Zoe's epistemic risk bears a striking resemblance to financial risk. For instance, if one simply knew more about tomorrow's stock prices, they could sidestep all stock market losses and potentially become stupendously rich.

This highlights the fact that gaining knowledge in certain domains can be difficult task, with big hedge funds splashing billions and hiring some of the brightest minds just to gain a slight edge in simply knowing a bit more about asset prices. It extends to having more info about which companies may go belly up or engage in fraud.

Acquiring more knowledge comes at a cost. Processing knowledge comes at cost. Choosing ignorance is mostly not a result of recklessness or EA institutional design but a practical choice given the resources required to process information. It's actually rational for everyone to ignore most information most of the time (this is standard econ, check rational inattention and extensive literature on the topic).

One real question in this space is if EAs have allocated their attention wisely. The answer seems to be "mostly yes." In case of FTX, heavyweights like Temasek, Sequoia Capital, and SoftBank with billions on the line did their due diligence but still missed what was happening. Expecting EAs to be better evaluators of FTX's health than established hedge funds is somewhat odd. EAs, like everyone else, face the challenge of allocating attention and their expertise lies in "using money for good" rather than "evaluating the health of big financial institutions". For the typical FTX grant recipient to assume they need to be smarter than Sequoia or SoftBank about FTX would likely not be a sound decision.

Some questions for CEA:

But this changed fast. In 2019, I was leaked a document circulating at the Centre for Effective Altruism, the central coordinating body of the EA movement. Some people in leadership positions were testing a new measure of value to apply to people: a metric called PELTIV, which stood for “Potential Expected Long-Term Instrumental Value.” It was to be used by CEA staff to score attendees of EA conferences, to generate a “database for tracking leads” and identify individuals who were likely to develop high “dedication” to EA — a list that was to be shared across CEA and the career consultancy 80,000 Hours. There were two separate tables, one to assess people who might donate money and one for people who might directly work for EA.

Individuals were to be assessed along dimensions such as “integrity” or “strategic judgment” and “acting on own direction,” but also on “being value-aligned,” “IQ,” and “conscientiousness.” Real names, people I knew, were listed as test cases, and attached to them was a dollar sign (with an exchange rate of 13 PELTIV points = 1,000 “pledge equivalents” = 3 million “aligned dollars”).

What I saw was clearly a draft. Under a table titled “crappy uncalibrated talent table,” someone had tried to assign relative scores to these dimensions. For example, a candidate with a normal IQ of 100 would be subtracted PELTIV points, because points could only be earned above an IQ of 120. Low PELTIV value was assigned to applicants who worked to reduce global poverty or mitigate climate change, while the highest value was assigned to those who directly worked for EA organizations or on artificial intelligence.

The list showed just how much what it means to be “a good EA” has changed over the years. Early EAs were competing for status by counting the number of mosquito nets they had funded out of their own pocket; later EAs competed on the number of machine learning papers they co-authored at big AI labs.

When I confronted the instigator of PELTIV, I was told the measure was ultimately discarded. Upon my request for transparency and a public apology, he agreed the EA community should be informed about the experiment. They never were. Other metrics such as “highly engaged EA” appear to have taken its place.

1. On CEA gathering information from EA conference attendees:
   1. Can someone from CEA clarify what information, if any, is currently being gathered on EA members
   2. which of these, if any, is being used for assessing individuals,
   3. for what purpose (e.g. for EAGs, other CEA opportunities, "identifying individuals who were likely to develop high dedication to EA"), and
   4. which organizations these are shared with, if relevant?
2. Given CEA had a leadership change in 2019, the same year as the most recent leadership change, can someone from CEA clarify the timing of this measure of value (i.e. was this under Larissa Hesketh-Rowe or Max Dalton as CEO)?
3. Can someone from CEA also justify the reasoning behind these two claims in particular, and the extent to which this represents the views of CEA leadership at present?

For example, a candidate with a normal IQ of 100 would be subtracted PELTIV points, because points could only be earned above an IQ of 120.

Low PELTIV value was assigned to applicants who worked to reduce global poverty or mitigate climate change, while the highest value was assigned to those who directly worked for EA organizations or on artificial intelligence.

Will also note the apparent consistency with the previous case of CEA over-emphasizing longtermism and AI over global health and animal welfare in earlier versions of the handbook, despite claiming to not take an organizational stance on any cause areas specifically.