Building EABuilding effective altruism

Today I had to turn off my VPN to pass the Vercel Security Check and access the EA Forum. I think this is bad, especially given the state of internet freedom around the world. As in many places, the internet in my country is subject to frequent bandwidth throttling, ad hoc censorship, and mass surveillance.

In the late 2000s and early 2010s, GiveWell would look at meta-analyses of randomized controlled trials and get lots of expert input on whether they were interpreting them correctly. This was the sort of approach that was supposed to typify effective altruism.

Fast forward to the mid-2020s and now subjective intuition plays an unreasonably large role in how effective altruism attempts to answer empirical, quantitative questions of a scientific or technical nature. What happened to effective altruism?

Worse, it's become a recurring problem that subjective intuitions are deceptively dressed up as objective empirical evidence. It's bad enough to leave behind more rigorous, scientific approaches in favour of subjective intuition. It's even worse to mislead people — either willingly or through carelessness — into thinking you're reporting objective, measured data when you're really just reporting your subjective guesses.

This shift away from science toward intuition feels even more troubling when you connect it to the hostility expressed by some members of the EA community toward academia, journalism, and liberal democracy. There is a sort of dark, conspiratorial, fringe worldview you get when you combine these troubling tendencies. When considering this, it should be remembered that there have been violent killings and threats of killings at the peripheries of the EA community. This does not normally happen with mainstream social movements of this size. What is fuelling these extremist acts?

More and more, I appreciate just how important are the intellectual cornerstones of modern, Enlightenment society like science and liberalism, and the institutions that support them, like academia and journalism. Extremism and darkness are only a few short steps away from the light. The barrier between them is as thin as air.

May those who carry the light win, and may those lost in the darkness find their way back.

This is a linkpost for https://heresjonny.substack.com/p/paul-christianos-barista. 

When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul^[1]. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.

Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse^[2]. I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it^[3]. I figured that all these super smart, status-y types had it under control already.

                                          The EMH is trivially false across all domains

In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety^[4]. I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.

I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:

Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.

And then of course something deeply upsetting happened. I realised that I was not following my own advice.

So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.

Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.

I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:

experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius

I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.

Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.

1. ^{^}

   Although Michael Burry seems to be making it work for him

2. ^{^}

   I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté.

3. ^{^}

   Despite being motivated, I do think there’s some legitimacy to the claim regardless.

^{^}

It was my sixth EAG, and the other five had all been dominated by the topic.

Show all footnotes

I wrote a case study on the School for Moral Ambition's storytelling. I came to SMA from EA, and many of its ideas sounded familiar. What surprised me was how far a new organization got with them. It seemed to reach people EA rarely reaches, with much less pushback than EA tends to get. That's an impression I can't measure.

I think SMA rewrites the usual nonprofit cast. The reader is the hero, a system is the villain, and history is the judge. Every cause comes with a named opponent, like the tobacco industry or tax rules that favor billionaires, and the reader stays off the list of suspects. Its food explainer goes after intensive animal agriculture, and veganism doesn't come up.

Is an easier version of these ideas a fair price for reaching people EA never reached? I'm unsure. Pushback welcome, and suggestions for who to look at next.

https://annapit.substack.com/p/hero-villain-judge-how-the-school?r=2fkhlo

I've noticed a couple of times in workshops, retreats and conferences that folks quite new to their jobs in EA orgs find themselves in a position where they are talking about their org's work and mission statement and get questioned on their theory of change, and while the first two make complete sense, the last one often gets met with "I've just started my role" or "I've only been at X for Y weeks".

This is perfectly reasonable, but I wonder if this has been noticed by other people, and this overall makes me feel we need to see more experienced members of an org dedicate more time to being public-facing in said events!

EA often gets criticized for taking small probabilities too seriously. But people take actions with small probabilities of big success all the time! Asking a girl out? Small chance she'll become your wife. Trying to affect political chance? Fat chance of that. Launching a startup? Good luck buddy. The difference between EA and the standard way of acting is that standard psychology deals with the problem of small probabilities by being highly overconfident. Of course _my_ startup/political movement/YouTube channel is going to make it big. You cannot maintain high agency while ignoring small probability options. 

The EA approach is simply more honest. 

“We want lab employees to show up”

Can someone explain to me the over-blown importance of lab employees? That we need them to show up to our events or community or snag a conversation with them at an EAG = some sort of impact. (I.e I speculate lab employees showing up to EA SF would mean we get more funding or something) 

My understanding is that many employees who would show up are already AIS-pilled and doing their best. What more do you want from them? To leave, to unionize, to influence them in some way? 

I’m sure I’m missing something here 🤷‍♀️ 

With all the recent media attention on EA, I think what's really unclear to outsiders is the relationship between EA and Anthropic (rightfully so!). To most people it doesn't make sense that a community so focused on AI safety can also be linked to one of the leading companies. I'd personally be very interested to see a survey of what EAs think of Anthropic, but maybe that's a minefield.. Even a good explainer meant for the public could go a long way.