On the way out of EA.
🔸 10% Pledger.
Likes pluralist conceptions of the good.
Dislikes Bay Culture being in control of the future.
Hey Ollie, thanks for your feedback! It helped me understand some of the downvotes the post was getting which I was a bit confused by. I think you and perhaps others are interpreting the post as "Here are some case studies that show EA is in decline", but that's not what I was trying to write, it was more "EA is in decline, what historical cases can inform us about this?" I'm not really arguing for "Is EA in decline?" in the post, in fact I'm just assuming it and punting the empirical evidence for another time, since I was interested in bringing out the historical cases rather than EAs current state. So the tweets are meant to be indicative of mood/sentiment but not load bearing proof. I do see that the rhetorical flourish in the intro might have given a misleading impression, so I will edit that to make the point of the piece more clear.
As for why, I mean, it does just seem fairly obvious to me, but social movements have fuzzy boundaries and decline doesn't have to be consistent. Nevertheless answering this question was a post I was planning on write and the evidence seemed fairly damning to me - for instance:
If you search "Effective Altruism" on almost any major social media site (X, Bluesky, Reddit, etc) I suspect the general sentiment toward EA will be strongly negative and probably worse than it was preFTX and staying that way. There might be some counter evidence. Some metrics might have improved, and I know some surveys are showing mixed or positive things. I think Habryka's point that reputation is evaluated lazily rings true to me even if I disagree on specifics.
But again, the above is my memory of a draft, and I'm not sure I'll ever finish that post. I think hard data on a well formed version of the question would be good, but once again it's not the question I was trying to get at with this post.
Some Positions In EA Leadership Should Be Elected
If we take this literally, then 'some' means 'any at all', and I think that the amount of democratic input is >0 so this should be 100% yes.
Having said that, I think the bar is kind-of acting as a 'how much do you support democratising EA', and in that sense while I do support Cremer-style reforms, I think they're best introduced at a small scale to get a track record.
I wish this post - and others like it - had more specific details when it comes to these kind of criticisms, and had a more specific statement of what they are really taking issue with, because otherwise it sort of comes across as "I wish EA paid more attention to my object-level concerns" which approximately ~everyone believes.
If the post it's just meant to represent your opinions thats perfectly fine, but I don't really think it changed my mind on its own merits. I also just don't like withholding private evidence, I know there are often good reasons for it, but it means I just can't give much credence to it tbqh. I know it's a quick take but still, it left me lacking in actually evaluating.
I general this discussion reminds me a bit of my response to another criticism of EA/Actions by the EA Community, and I think what you view as 'sub-optimal' actions are instead best explained by people having:
And that, with these constraints accepted, many people are not actually acting sub-optimally given their information and values (they may later with hindsight regret their actions or admit they were wrong, of course)
Some specific counterexamples:
Trying to read between the lines - I think you think the AGI is going to be a super big deal soon, and that its political consequences might be the most important and consequently political actions might be the most important ones to take? But OpenPhil and 80K haven't been on the ball with political stuff and now you're disillusioned? And people in other cause areas that aren't AI policy should presumably stop donating there and working their and pivot? I don't know, it doesn't feel accurate to me,[1] but like I can't get a more accurate picture because you just didn't provide any specifics for me to tune my mental model on 🤷♂️
Even having said all of this, I do think working on mitigating concentration-of-power-risks is a really promising direction for impact and wish you the best in pursuing it :)
As in, i don't think you'd endorse this as a fair or accurate description of your views
Note - this was written kinda quickly, so might be a bit less tactful than I would write if I had more time.
Making a quick reply here after binge listening to three Epoch-related podcasts in the last week, and I basically think my original perspective was vindicated. It was kinda interesting to see which points were repeated or phrased a different way - would recommend if your interested in the topic.
Basically, the Mechanize cofounders don't agree at all with 'AI Safety Classic', I am very confident that they don't buy the arguments at all, that they don't identify with the community, and somewhat confident that they don't respect the community or its intellectual output that much.
Given that their views are: a) AI will be a big deal soon (~a few decades), b) returns to AI will be very large, c) Alignment concerns/AI risks are overrated, and d) Other people/institutions aren't on the ball, then starting an AI Start-up seems to make sense.
What is interesting to note, and one I might look into in the future, is just how much these differences in expectation of AI depend on differences in worldview, rather than differences in technical understanding of ML or understanding of how the systems work on a technical level.
So why are people upset?
Probably some mix of it. I personally remain not that upset because a) I didn't really class Epoch as 'part of the community', b) I'm not really sure I'm 'part of the community' either and c) my views are at least somewhat similar to the Epoch set above, though maybe not as far in their direction, so I'm not as concerned object-level either.
I'm not sure I feel as concerned about this as others. tl;dr - They have different beliefs from Safety-concerned EAs, and their actions are a reflection of those beliefs.
It seems broadly bad that the alumni from a safety-focused AI org
Was Epoch ever a 'safety-focused' org? I thought they were trying to understand what's happening with AI, not taking a position on Safety per se.
...have left to form a company which accelerates AI timelines
I think Matthew and Tamay think this is positive, since they think AI is positive. As they say, they think explosive growth can be translated into abundance. They don't think that the case for AI risk is strong, or significant, especially given the opportunity cost they see from leaving abundance on the table.
Also important to note is what Epoch boss Jaime says in this very comment thread.
As I learned more and the situation unfolded I have become more skeptical of AI Risk.
The same thing seems to be happening with me, for what it's worth.
People seem to think that there is an 'EA Orthodoxy' on this stuff, but there either isn't as much as people think, or people who disagree with it are no longer EAs. I really don't think it makes sense to clamp down on 'doing anything to progress AI' as being a hill for EA to die on.
Note: I'm writing this for the audience as much as a direct response
The use of Evolution to justify this metaphor is not really justified. I think Quintin Pope's Evolution provides no evidence for the sharp left turn (which won a prize in an OpenPhil Worldview contest) convincingly argues against it. Zvi wrote a response from the "LW Orthodox" camp that wasn't convincing and Quintin responds against it here.
On "Inner vs Outer" framings for misalignment is also kinda confusing and not that easy to understand when put under scrutiny. Alex Turner points this out here, and even BlueDot have a whole "Criticisms of the inner/outer alignment breakdown" in their intro which to me gives the game away by saying "they're useful because people in the field use them", not because their useful as a concept itself.
Finally, a lot of these concerns revolve around the idea of their being set, fixed, 'internal goals' that these models have, and represent internally, but are themselves immune from change, or can hide from humans, etc. This kind of strong 'Goal Realism' is a key part of the case for 'Deception' style arguments, whereas I think Belrose & Pope show an alternative way to view how AIs work is 'Goal Reductionism', in which framing the issues imagined don't seem certain any more, as AIs are better understood as having 'contextually-activated heuristics' rather than Terminal Goals. For more along these lines, you can read up on Shard Theory.
I've become a lot more convinced about these criticisms of "Alignment Classic" by diving into them. Of course, people don't have to agree with me (or the authors), but I'd highly encourage EAs reading the comments on this post to realise Alignment Orthodoxy is not uncontested, and is not settled, and if you see people making strong cases based on arguments and analogies that seem not solid to you, you're probably right, and you should look to decide for yourself rather than accepting that the truth has already been found on these issues.[1]
And this goes for my comments too
I'm glad someone wrote this up, but I actually don't see much evaluation here from you, apart from "it's too early to say", but then Zhou Enlai pointed out that you could say that about the French Revolution,[1] and I think we can probably say some things. I generally have you mapped to the "right-wing Rationalist" subgroup Arjun,[2] so it'd be actually interested to get your opinion instead of trying to read between the lines on what you may or may not believe. I think there was a pretty strong swing in Silicon Valley / Tech Twitter & TPOT / Broader Rationalism towards Trump, and I think this isn't turning out well, so I'd actually be interested to see people saying what they actually think - be that "I made a huge mistake", "It was a bad gamble but Harris would've been worse" or even "This is exactly what I want"
Hey Cullen, thanks for responding! So I think there are object-level and meta-level thoughts here, and I was just using Jeremy as a stand-in for the polarisation of Open Source vs AI Safety more generally.
Object Level - I don't want to spend too long here as it's not the direct focus of Richard's OP. Some points:
Meta Level -
Again, not saying that this is referring to you in particular
I responded well to Richard's call for More Co-operative AI Safety Strategies, and I like the call toward more sociopolitical thinking, since the Alignment problem really is a sociological one at heart (always has been). Things which help the community think along these lines are good imo, and I hope to share some of my own writing on this topic in the future.
Whether or not I agree with Richard's personal politics or not is kinda beside the point to this as a message. Richard's allowed to have his own views on things and other people are allowed to criticse this (I think David Mathers' comment is directionally where I lean too). I will say that not appreciating arguments from open-source advocates, who are very concerned about the concentration of power from powerful AI, has lead to a completely unnecessary polarisation against the AI Safety community from it. I think, while some tensions do exist, it wasn't inevitable that it'd get as bad as it is now, and in the end it was a particularly self-defeating one. Again, by doing the kind of thinking Richard is advocating for (you don't have to co-sign with his solutions, he's even calling for criticism in the post!), we can hopefully avoid these failures in the future.
On the bounties, the one that really interests me is the OpenAI board one. I feel like I've been living in a bizarro-world with EAs/AI Safety People ever since it happened because it seemed such a collosal failure, either of legitimacy or strategy (most likely both), and it's a key example of the "un-cooperative strategy" that Richard is concerned about imo. The combination of extreme action and ~0 justification either externally or internally remains completely bemusing to me and was big wake-up call for my own perception of 'AI Safety' as a brand. I don't think people can underestimate the second-impact effect this bad on both 'AI Safety' and EA, coming about a year after FTX.
No worries Ollie, thanks for the feedback :)
As I said, those bullet points were a memory of a draft so I don't have the hard data to share on hand. But when dealing with social movements it's always going to be somewhat vibesy - data will necessarily be observational and we can't travel back in time and run RCTs on whether SBF commits fraud or not. And the case studies do show that declines can go on for a very long time post major crisis. It's rare for movements to disappear overnight (The Levellers come closest of all the cases I found to that)
Fwiw I think that the general evidence does point to "EA is in decline" broadly understood, and that should be considered the null hypothesis at this point. I'd feel pretty gaslit if someone said EA was going swimmingly and unaffected by the tribulations of the last couple of years, perhaps less so if they think there's been a bounce back after an initial decline but, you know, I'd want to see the data for that.
But as I said, it's really not the (main) point of the post! I'd love to add my points to a post where someone did try and do a deep dive into that question.