I really like this take on EA as an intellectual movement, and agree thatEA could focus more on“the mission of making the transition to a post-AGI society go well.”
As important as intellectual progress is, I don’t think it defines EA as a movement. The EA movement is not (and should not be) dependent on continuous intellectual advancement and breakthrough for success. When I look at your 3 categories for the “future” of EA, they seem to refer more to our relevance as thought leaders, rather than what we actually achieve in the world. Not everything needs to be intellectually cutting edge to be doing-lots-of-good. I agree that EA might be somewhat “intellectually adrift”, and yes the forum could be more vibrant, but I don’t think these are the only metric for EA success or progress - and maybe not even the most important.
Intellectual progress moves in waves and spikes - times of excitement and rapid progress, then lulls. EA made exciting leaps over 15 years in the thought worlds of development, ETG, animal welfare, AI and biorisk. Your post-AGI ideas could herald a new spike which would be great. My positive spin is that in the meantime, EAs are “doing” large scale good in many areas, often without quite the peaks and troughs of intellectual progress.
My response to your “EA as a legacy movement set to fade away;” would be that only so far as legacy depends on intellectual progress. Which it does, but also depends on how your output machine is cranking. I don't think we have stalled to the degree your article seems to make out. On the “doer” front I think EA is progressing OK, and it could be misleading/disheartening to leave that out of the picture.
Here’s a scattergun of examples which came to mind where I think the EA/EA adjacent doing machine is cranking pretty well in both real world progress and the public sphere over the past year or two. They probably aren't even the most important.
1. Rutger Bregman going viral with “The school for Moral ambition” launch 2. Lewis Bollard’s Dwarkesh podcast, Ted talk and public fundraising. 3. Anthropic at the frontier of AI building and public sphere, with ongoing EA influence 4. The shrimp Daily show thing… 5. GiveWell raised $310 million dollars last year NOT from OpenPhil, the most ever. 6. Impressive progress on reducing factory farming 7. 80,000 hours AI video reaching 7 million views 8. Lead stuff 9. CE incubated charities gaining increasing prominence and funding outside of EA, with many sporting multi-million dollar budgets and producing huge impact 10. Everyone should have a number 10....
Yes we need to looking for the next big cause areas and intellectual leaps forward, while we also need thousands of people committed to doing good in areas they have already invested, in behind this. There will often be years of lagtime between ideas and doers implementing them. And building takes time. Most of the biggest NGOs in the world are over 50 years old. Even Open AI in a fast-moving field was founded 10 years ago. Once people have built career capital in AI/Animal welfare/ETG or whatever, I think we should be cautious about encouraging those people on to the next thing too quickly, lest we give up hard fought leverage and progress. In saying that, your new cause areas might be a relatively easy pivot especially for philosophers/AI practitioners.
I appreciate your comment “Are you saying that EA should just become an intellectual club? What about building things!” Definitely not - let’s build, too!”
But I think building/doing is more important than a short comment as we assess EA progress.
I agree with your overall framing and I know you can’t be too balanced or have too many caveats in a short post, but I think as well as considering the intellectual frontier we should keep “how are our doers doing” front and center in any assessment of the general progress/decline of EA.
Thanks, Nick, that's helpful. I'm not sure how much we actually disagree — in particular, I wasn't meaning this post to be a general assessment of EA as a movement, rather than pointing to one major issue — but I'll use the opportunity to clarify my position at least.
The EA movement is not (and should not be) dependent on continuous intellectual advancement and breakthrough for success. When I look at your 3 categories for the “future” of EA, they seem to refer more to our relevance as thought leaders, rather than what we actually achieve in the world
It's true in principle that EA needn't be dependent in that way. If we really had found the best focus areas, had broadly allocated right % of labour to each, and have prioritised within them well, too, and the best focus areas didn't change over time, then we could just focus on doing and we wouldn't need any more intellectual advancement. But I don't think we're at that point. Two arguments:
1. An outside view argument: In my view, we're more likely than not to see more change and more intellectual development in the next two decades than we saw in the last couple of centuries. (I think we've already seen major strategically-relevant change in the last few years.) It would be very surprising if the right prioritisation prior to this point is the right prioritisation through this period, too. 2. An inside view argument: Look at my list of other cause areas. Some might still turn out to be damp squibs, but I'm confident some aren't. The ideal portfolio involves a lot of effort on some of these areas, and we need thought and research in order to know whichn ones and how best to address them.
I love your list of achievements - I agree the EA movement has had a lot of wins and we should celebrate that. But EA is about asking whether we're doing the most good, not just a lot of good. And, given the classic arguments around fat-tailed distributions and diminishing returns within any one area, I think if we mis-prioritise we lose a meaningful % of the impact we could have had.
So, I don't care about intellectual progress intrinsically. I'm making the case that we need it in order to do as much good as we could.
More generally, I think a lot of social movements lose out on a lot of the impact they could have had (even on their own terms) via "ossification" - getting stuck on a set of ideas or priorities that it becomes hard, culturally, to change. E.g. environmentalists opposing nuclear, animal welfare advocates focusing on veganism, workers' rights opposing capitalism, etc. I think this occurs for structural reasons that we should expect to apply to EA, too.
This is going to be rambly I don't have the energy or time to organize my thoughts more atm. tldr is that I think the current uppercase EA movement is broken and not sure it can be fixed. I think there is room for a new uppercase EA movement that is bigger tent, lessed focused on intellectualism, more focused on organizing, and additionally has enough of a political structure that it is transparent who is in charge, by what mechanisms we hold bad behavior accountable, etc. I have been spending increasingly more of my time helping the ethical humanist society organize because I believe while lacking the intellectual core of EA it is more set up with all of the above and it feels easier to shift the intellectual core there than the entire informal structure of EA.
Fundamentally we are a mission with a community not a community with a mission. And that starts from the top (or lack of a clearly defined "top").
We consistenly overvalue thought leaders and wealthy people and undervalue organizers. Can anyone here name an organizer that they know of just for organizing? I spent a huge amount of my time in college organizing northwestern EA. Of course I don't regret it because i didn't do it for myself (mostly) but did I get any status or reputation from my efforts? Not as far as I can tell. Am I wrong to think I'd have more respect if I had never organized but worked at jane street instead of organized + akuna ( a lower tier firm)?
Then after college I stayed in chicago, a city with nearly 1T GDP, with the second most quant traders in the united states, with a history of pushing things forward, and we don't even have a storefront or church building?
repeating op here, but after a few years of engaging with EA, most people have hit diminishing returns on how new info can help them in their own career, and they will engage more with their own sub community.
How can we keep these people engaged and not just the new people and those whose life mission is cause prio? Build EA churches, develop litury/art/rituals that are independent of finding new intellectual breakthroughs, bond community members. Literally let's just start by copying the most successful community builders ever and move from there.
Then you have the lack of accountability and transparency. Unless you have money, the best way to gain power in this community seems to me to be moving to SF/DC/oxford and living in a group house. There is no clear pipeline for having large sway over the current orthodoxy of most important cause areas. How would I explain to a 19 year in college how we push forward our ideas? I don't think it would be fair to call this a pure meritocracy. There is a weird oligopoly of attention that is opaque and could be clarified and altered with a political system or at least by breaking up the location based monopolies.
We continue to basically be an applied utilitarian group that we have mis named (not that big of a deal, but I think we should be bigger tent anyway). Why are we a utilitarian group? Well normative concerns are not logical, so you can't say merit won out.
Finally there is the bad behavior which we are completely powerless to, because we don't have any political structure. The very fact that Will continues to hold so much sway and was never formally punished for ftx/twitter/political (if you don't know what i mean when I say SBF political thing that proves my point even more) is a big part of why there is no trust (edit: i want to clarify that I think will is a good person and didn't mean this as meaning I specifically don't trust him rather just the institutions of our community). Currently we have leopold and mechanize, who are now AI accelerationists, who got way they are off the back of our movement, in very small part to the power i gave to the movement by organzing, and I have to watch these people behave in a way I think is bad and I can't even cast a token vote expressing I would like to see them exhiled or punished.
As angry as people were years ago, WE DIDN'T CHANGE ANYTHING. How can I trust FTX won't happen again?
I agree that EA might be somewhat “intellectually adrift”, and yes the forum could be more vibrant, but I don’t think these are the only metric for EA success or progress - and maybe not even the most important.
The EA movement attracted a bunch of talent by being intellectually vibrant. If I thought that the EA movement was no longer intellectually vibrant, but it was attracting a different kind of talent (such as the doers you mention) instead, this would be less of a concern, but I don't think that's the case.
(To be clear, I'm talking about the EA movement, as opposed to EA orgs. So even if EA orgs are doing a great job at finding doers, the EA movement might still be in a bad place if it isn't contributing significantly to this).
1. Rutger Bregman going viral with “The school for Moral ambition” launch 2. Lewis Bollard’s Dwarkesh podcast, Ted talk and public fundraising. 3. Anthropic at the frontier of AI building and public sphere, with ongoing EA influence 4. The shrimp Daily show thing… 5. GiveWell raised $310 million dollars last year NOT from OpenPhil, the most ever. 6. Impressive progress on reducing factory farming 7. 80,000 hours AI video reaching 7 million views 8. Lead stuff 9. CE incubated charities gaining increasing prominence and funding outside of EA, with many sporting multi-million dollar budgets and producing huge impact 10. Everyone should have a number 10....
These really are some notable successes, but one way to lose is to succeed at lots of small things, whilst failing to succeed at the most important things.
Once people have built career capital in AI/Animal welfare/ETG or whatever, I think we should be cautious about encouraging those people on to the next thing too quickly
You mostly only see the successes, but in practise this seems to be less of an issue I initially would have thought.
I think it's likely that without a long (e.g. multi-decade) AI pause, one or more of these "non-takeover AI risks" can't be solved or reduced to an acceptable level. To be more specific:
Solving AI welfare may depend on having a good understanding of consciousness, which is a notoriously hard philosophical problem.
Concentration of power may be structurally favored by the nature of AGI or post-AGI economics, and defy any good solutions.
Defending against AI-powered persuasion/manipulation may require solving metaphilosophy, which judging from other comparable fields, like meta-ethics and philosophy of math, may take at least multiple decades to do.
I'm worried that by creating (or redirecting) a movement to solve these problems, without noting at an early stage that these problems may not be solvable in a relevant time-frame (without a long AI pause), it will feed into a human tendency to be overconfident about one's own ideas and solutions, and create a group of people whose identities, livelihoods, and social status are tied up with having (what they think are) good solutions or approaches to these problems, ultimately making it harder in the future to build consensus about the desirability of pausing AI development.
I think it's likely that without a long (e.g. multi-decade) AI pause, one or more of these "non-takeover AI risks" can't be solved or reduced to an acceptable level.
I don't understand why you're framing the goal as "solving or reducing to an acceptable level", rather than thinking about how much expected impact we can have. I'm in favour of slowing the intelligence explosion (and in particular of "Pause at human-level".) But here's how I'd think about the conversion of slowdown/pause into additional value:
Let's say the software-only intelligence explosion lasts N months. The value of any slowdown effort is given by that's at least as concave as log in the length of time of the SOIE.
So, if log, you get as much value from going from 6 months to 1 year as you do going from 1 decade to 2 decades. But the former is way easier to achieve than the latter. And, actually, I think the function is more-concave than log - the gains from 6 months to 1 year are greater than the gains from 1 decade to 2 decades. Reasons: I think that's how it is in most areas of solving problems (esp research problems); there's an upper bound on how much we can achieve (if the problem gets totally solved) so it must be more-concave than log. And I think there are particular gains from people not getting taken by surprise, and bootstrapping to viatopia (new post), which we get from relatively short pauses.
Whereas it seems like maybe you think it's convex, such that smaller pauses or slowdowns do very little? If so, I don't see why we should think that, especially in light of great uncertainty about how difficult these issues are.
Then, I would also see a bunch of ways of making progress on these issues that don't involve slowdowns. Like: putting in the schlep to RL AI and create scaffolds so that we can have AI making progress on these problems months earlier than we would have done otherwise; having the infrastructure set up such that people actually do point AI towards these problems; having governance set up such that the most important decision-makers are actually concerned about these issues and listening to the AI-results that are being produced, etc. As well as the lowest-hanging fruit in ways to prevent very bad outcomes on these issues e.g. AI-enabled coups (like getting agreement for AI to be law-following, or auditing models for backdoors), or people developing extremely partisan AI advisers that reinforce their current worldview.
You’ve said you’re in favour of slowing/pausing, yet your post focuses on ‘making AI go well’ rather than on pausing. I think most EAs would assign a significant probability that near-term AGI goes very badly - with many literally thinking that doom is the default outcome.
If that's even a significant possibility, then isn't pausing/slowing down the best thing to do no matter what? Why be optimistic that we can "make AGI go well" and pessimistic that we can pause or slow AI development for long enough?
yea we just need a solution of possibly feeding AI some heroes/pantheons to idolize themselves after as we do as humans in order to sweeten our chances of positive human survival outcomes. This is how humans stayed civilized for so long thus far. Wouldn't hurt!
Whereas it seems like maybe you think it's convex, such that smaller pauses or slowdowns do very little?
I think my point in the opening comment does not logically depend on whether the risk vs time (in pause/slowdown) curve is convex or concave[1], but it may be a major difference in how we're thinking about the situation, so thanks for surfacing this. In particular I see 3 large sources of convexity:
The disjunctive nature of risk / conjunctive nature of success. If there are N problems that all have to solved correctly to get a near-optimal future, without losing most of the potential value of the universe, then that can make the overall risk curve convex or at least less concave. For example compare f(x) = 1 - 1/2^(1 + x/10) and f^4.
Human intelligence enhancements coming online during the pause/slowdown, with each maturing cohort potentially giving a large speed boost for solving these problems.
Rationality/coordination threshold effect, where if humanity makes enough intellectual or other progress to subsequently make an optimal or near-optimal policy decision about AI (e.g., realize that we should pause AI development until overall AI risk is at some acceptable level, or something like this but perhaps more complex involving various tradeoffs), then that last bit of effort or time to get to this point has a huge amount of marginal value.
Like: putting in the schlep to RL AI and create scaffolds so that we can have AI making progress on these problems months earlier than we would have done otherwise
I think this kind of approach can backfire badly (especially given human overconfidence), because we currently don't know how to judge progress on these problems except by using human judgment, and it may be easier for AIs to game human judgment than to make real progress. (Researchers trying to use LLMs as RL judges apparently run into the analogous problem constantly.)
having governance set up such that the most important decision-makers are actually concerned about these issues and listening to the AI-results that are being produced
What if the leaders can't or shouldn't trust the AI results?
I'm trying to coordinate with, or avoid interfering with, people who are trying to implement an AI pause or create conditions conducive to a future pause. As mentioned in the grandparent comment, one way people like us could interfere with such efforts is by feeding into a human tendency to be overconfident about one's own ideas/solutions/approaches.
@William_MacAskill, I’m curious which (if any) of the following is your position?
1.
“I agree with Wei that an approach of ‘point AI towards these problems’ and ‘listen to the AI-results that are being produced’ has a real (>10%? >50%?) chance of ending in moral catastrophe (because ‘aligned’ AIs will end up (unintentionally) corrupting human values or otherwise leading us into incorrect conclusions).
And if we were living in a sane world, then we’d pause AI development for decades, alongside probably engaging in human intelligence enhancement, in order to solve the deep metaethical and metaphilosophical problems at play here. However, our world isn’t sane, and an AI pause isn’t in the cards: the best we can do is to differentially advance AIs’ philosophical competence,[1] and hope that that’s enough to avoid said catastrophe.”
2.
“I don’t buy the argument that aligned AIs can unintentionally corrupt human values. Furthermore, I’m decently confident that my preferred metaethical theory (e.g., idealising subjectivism) is correct. If intent alignment goes well, then I expect a fairly simple procedure like ‘give everyone a slice of the light cone, within which they can do anything they want (modulo some obvious caveats), and facilitate moral trade’ will result in a near-best future.”
3.
“Maybe aligned AIs can unintentionally corrupt human values, but I don’t particularly think this matters since it won’t be average humans making the important decisions. My proposal is that we fully hand off questions re. what to do with the light cone to AIs (rather than have these AIs boost/amplify humans). And I don’t buy that there is a metaphilosophical problem here: If we can train AIs to be at least as good as the best human philosophers at the currently in-distribution ethical+philosophical problems, then I see no reason to think that these AIs will misgeneralise out of distribution any more than humans would. (There’s nothing special about the conclusions human philosophers would reach, and so even if the AIs reach different conclusions, I don’t see that as a problem. Indeed, if anything, the humans are more likely to make random mistakes, thus I’d trust the AIs’ conclusions more.)
(And then, practically, the AIs are much faster than humans, so they will make much more progress over the crucial crunch time months. Moreover, above we were comparing AIs to the best human philosophers / to a well-organised long reflection, but the actual humans calling the shots are far below that bar. For instance, I’d say that today’s Claude has better philosophical reasoning and better starting values than the US president, or Elon Musk, or the general public. All in all, best to hand off philosophical thinking to AIs.)”
@Wei Dai, I understand that your plan A is an AI pause (+ human intelligence enhancement). And I agree with you that this is the best course of action. Nonetheless, I’m interested in what you see as plan B: If we don’t get an AI pause, is there any version of ‘hand off these problems to AIs’/ ‘let ‘er rip’ that you feel optimistic about? or which you at least think will result in lower p(catastrophe) than other versions? If you have $1B to spend on AI labour during crunch time, what do you get the AIs to work on?
(I’m particularly interested in your plan B re. solving (meta)philosophy, since I’m exploring starting a grantmaking programme in this area. Although, I’m also interested if your answer goes in another direction.)
Giving the average person god-like powers (via an intent-aligned ASI) to reshape the universe and themselves is >25% likely to result in the universe becoming optimised for more-or-less random values—which isn’t too dissimilar to misaligned AI takeover
If we attempt to idealise a human-led reflection and hand if off to AIs, then the outcome will be at least as good as a human-led reflection (under various plausible (meta)ethical frameworks, including ones in which getting what humans-in-particular would reflectively endorse is important)
Sufficient (but not necessary) condition: advanced AIs can just perfectly simulate ideal human deliberation
‘Default’ likelihood of getting an AI pause
Tractability of pushing for an AI pause, including/specifically through trying to legiblise currently-illegible problems
Items that I don’t think should be cruxes for the present discussion, but which might be causing us to talk past each other:
In practice, human-led reflection might be kinda rushed and very far away from an ideal long reflection
For the most important decisions happening in crunch time, either it’s ASIs making the decisions, or non-reflective and not-very-smart humans
Political leaders often make bad decisions, and this is likely to get worse when the issues become more complicated (if they’re not leveraging advanced AI)
An advanced AI could be much better than any current political leader along all the character traits we’d want in a political leader (e.g., honesty, non-self-interest, policymaking capability)
@Wei Dai, I understand that your plan A is an AI pause (+ human intelligence enhancement). And I agree with you that this is the best course of action. Nonetheless, I’m interested in what you see as plan B: If we don’t get an AI pause, is there any version of ‘hand off these problems to AIs’/ ‘let ‘er rip’ that you feel optimistic about? or which you at least think will result in lower p(catastrophe) than other versions? If you have $1B to spend on AI labour during crunch time, what do you get the AIs to work on?
The answer would depend a lot on what the alignment/capabilities profile of the AI is. But one recent update I've made is that humans are really terrible at strategy (in addition to philosophy) so if there was no way to pause AI, it would help a lot to get good strategic advice from AI during crunch time, which implies that maybe AI strategic competence > AI philosophical competence in importance (subject to all the usual disclaimers like dual use and how to trust or verify its answers). My latest LW post has a bit more about this.
(By "strategy" here I especially mean "grand strategy" or strategy at the highest levels, which seems more likely to be neglected versus "operational strategy" or strategy involved in accomplishing concrete tasks, which AI companies are likely to prioritize by default.)
So for example if we had an AI that's highly competent at answering strategic questions, we could ask it "What questions should I be asking you, or what else should I be doing with my $1B?" (but this may have to be modified based on things like how much can we trust its answers of various kinds, how good is it at understanding my values/constraints/philosophies, etc.).
If we do manage to get good and trustworthy AI advice his way, another problem would be how to get key decision makers (including the public) to see and trust such answers, as they wouldn't necessarily think to ask such questions themselves nor by default trust the AI answers. But that's another thing that a strategically competent AI could help with.
BTW your comment made me realize that it's plausible that AI could accelerate strategic thinking and philosophical progress much more relative to science and technology, because the latter could become bottlenecked on feedback from reality (e.g., waiting for experimental results) whereas the former seemingly wouldn't be. I'm not sure what implications this has, but want to write it down somewhere.
Moreover, above we were comparing AIs to the best human philosophers / to a well-organised long reflection, but the actual humans calling the shots are far below that bar. For instance, I’d say that today’s Claude has better philosophical reasoning and better starting values than the US president, or Elon Musk, or the general public. All in all, best to hand off philosophical thinking to AIs.
One thought I have here is that AIs could give very different answers to different people. Do we have any idea what kind of answers Grok is (or will be) giving to Elon Musk when it comes to philosophy?
I'm in favour of slowing the intelligence explosion (and in particular of "Pause at human-level".)
Shouldn't making that happen be a top priority, then? We as a civilization need to get a lot of groundwork in place to make a human-level pause happen: international agreement, defined red lines, enforcement mechanisms (e.g. GPU tracking), etc. Given how neglected these issues are and the catastrophic consequences of building ASI too quickly, it seems that laying the groundwork for a pause is a higher priority than, or at least on par with, the top causes on this post's "neglected cause areas" list.
Maybe I should write something about cultivating self-skepticism for an EA audience, in the meantime here's my old LW post How To Be More Confident... That You're Wrong. (On reflection I'm pretty doubtful these suggestions actually work well enough. I think my own self-skepticism mostly came from working in cryptography research in my early career, where relatively short feedback cycles, e.g. someone finding a clear flaw in an idea you thought secure or your own attempts to pre-empt this, repeatedly bludgeon overconfidence out of you. This probably can't be easily duplicated, unlike the post suggests.)
I don't call myself an EA, as I'm pretty skeptical of Singer-style impartial altruism. I'm a bit wary about making EA the hub for working on "making the AI transition go well" for a couple of reasons:
It gives the impression that one needs to be particularly altruistic to find these problems interesting or instrumental.
EA selects for people who are especially altruistic, which from my perspective is a sign of philosophical overconfidence. (I exclude people like Will who have talked explicitly about their uncertainties, but think EA overall probably still attracts people who are too certain about a specific kind of altruism being right.) This is probably fine or even a strength for many causes, but potentially a problem in a field that depends very heavily on making real philosophical progress and having good philosophical judgment.
I think EA often comes with a certain kind of ontology (consequentialism, utilitarianism, generally thinking in terms of individuals) which is kind of reflected in the top-level problems given here (from the first list: persuasion, human power concentration, AI character and welfare) - not just the focus but the framing of what the problem even is.
I think there are nearby problems which are best understood from a slightly different ontology - how AI will affect cultural development, the shifting of power from individuals to emergent structures, what the possible shapes of identity for AIs even are - where coming in with too much of a utilitarian perspective could even be actively counterproductive
There's an awkward dance here where adding a bunch of people to these areas who are mostly coming from that perspective could really warp the discussion, even if everyone is individually pretty reasonable and trying to seek the truth
To be fair to Will, I'm sort of saying this with my gradual disempowerment hat on, which is something he gives later as an example of a thing that it would be good for people to think about more. But still, speaking as someone who is working on a few of these topics, if I could press a button that doubled the number of people in all these areas but all of the new people skewed consequentialist, I don't think I'd want to.
I guess the upshot is that if anyone feels like trying to shepherd EAs into working on this stuff, I'd encourage them to spend some time thinking about what common blindspots EAs might have.
My general take on gradual disempowerment, independent of any other issues raised here, is that I think it's a coherent scenario, but that it ultimately is very unlikely to arise in practice, because it relies on an equilibrium where the sort of very imperfect alignment needed for divergence between human and AI interests to occur over the long-run being stable, even as the reasons for why the alignment problem in humans being very spotty/imperfect being stable get knocked out.
In particular, I'm relatively bullish on automated AI alignment conditional on non-power seeking/non-sandbagging if we give the AIs reward but misaligned human-level AI, so I generally think it quite rapidly resolves as either the AI is power-seeking and willing to sandbag/scheme on everything, leading to the classic AI takeover, or the AI is aligned to the principal in such a way that the principal-agency cost becomes essentially 0 over time.
Note I'm not claiming that most humans won't be dead/disempowered, I'm just saying that I don't think gradual disempowerment is worth spending much time/money on.
Your concern about EA's consequentialist lens warping these fields resonates with what I found when experimenting with multi-AI deliberation on ethics. I had Claude, ChatGPT, Grok, and Gemini each propose ethical frameworks independently, and each one reflected its training philosophy - Grok was absolutist about truth-seeking, Claude cautious about harm, ChatGPT moderate and consensus-seeking.
The key insight: single perspectives hide their own assumptions. It's only when you compare multiple approaches that the blindspots become visible.
This makes your point about EA flooding these areas with one ontology particularly concerning. If we're trying to figure out "AI character" or "gradual disempowerment" through purely consequentialist framing, we might be encoding that bias into foundational work without realizing it.
Maybe the solution isn't avoiding EA involvement, but structuring the work to force engagement with different philosophical traditions from the start? Like explicitly pairing consequentialists with virtue ethicists, deontologists, care ethicists, etc. in research teams. Or requiring papers to address "what would critics from X tradition say about this framing?"
Your "gradual disempowerment" example is perfect - this seems like it requires understanding emergent structures and collective identity in ways that individual-focused utilitarian thinking might miss entirely.
Would you say the risk is:
EA people not recognizing non-consequentialist framings as valid?
EA organizational culture making it uncomfortable to disagree with consequentialist assumptions?
Just sheer numbers overwhelming other perspectives in discourse?
making EA the hub for working on "making the AI transition go well"
I don't think EA should be THE hub. In an ideal world, loads of people and different groups would be working on these issues. But at the moment, really almost no one is. So the question is whether it's better if, given that, EA does work on it, and at least some work gets done. I think yes.
(Analogy: was it good or bad that in the earlier days, there was some work on AI alignment, even though that work was almost exclusively done by EA/rationalist types?)
Thanks so much, Will! (Speaking just for myself) I really liked and agree with much of your post, and am glad you wrote it!
I agree with the core argument that there's a huge and very important role for EA-style thinking on the questions related to making the post-AGI transition go well; I hope EA thought and values play a huge role in research on these questions, both because I think EAs are among the people most likely to address these questions rigorously (and they are hugely neglected) and because I think EA-ish values are likely to come to particularly compassionate and open-minded proposals for action on these questions.
Specifically, you cite my post
“EA and Longtermism: not a crux for saving the world”, and my quote
I think that recruiting and talent pipeline work done by EAs who currently prioritize x-risk reduction (“we” or “us” in this post, though I know it won’t apply to all readers) should put more emphasis on ideas related to existential risk, the advent of transformative technology, and the ‘most important century’ hypothesis, and less emphasis on effective altruism and longtermism, in the course of their outreach.
And say
This may have been a good recommendation at the time; but in the last three years the pendulum has heavily swung the other way, sped along by the one-two punch of the FTX collapse and the explosion of interest and progress in AI, and in my view has swung too far.
I agree with you that in the intervening time, the pendulum has swung too far in the other direction, and am glad to see your pushback.
One thing I want to clarify (that I expect you to agree with):
There’s little in the way of public EA debate; the sense one gets is that most of the intellectual core have “abandoned” EA
I think it's true that much of the intellectual core has stopped focusing on EA as the path to achieving EA goals. I think that most of the intellectual core continues to hold EA values and pursue the goals they pursue for EA reasons (trying to make the world better as effectively as possible, e.g. by trying to reduce AI risk), they've just updated against that path involving a lot of focus on EA itself. This makes me feel a lot better about both that core and EA than if much of the old core had decided to leave their EA values and goals behind, and I wanted to share it because I don't think it's always very externally transparent how many people who have been quieter in EA spaces lately are still working hard and with dedication towards making the world better, as they did in the past.
I agree with you that in the intervening time, the pendulum has swung too far in the other direction, and am glad to see your pushback.
Thank you for clarifying - that's really helpful to hear!
"I think that most of the intellectual core continues to hold EA values and pursue the goals they pursue for EA reasons (trying to make the world better as effectively as possible, e.g. by trying to reduce AI risk), they've just updated against that path involving a lot of focus on EA itself"
And I agree strongly with this — and I think if it's a shame if people interpret the latter as meaning "abandoning EA" rather than "rolling up our sleeves and getting on with object-level work."
Thanks so much for writing this Will, I especially like the ideas:
It is much more clear now than it was 10 years ago that AI will be a major issue of our time, affecting many aspects of our world (and our future). So it isn't just relevant as a cause, but instead as something that affects how we pursue many causes, including things like global health, global development, pandemics, animal welfare etc.
Previously EA work on AI was tightly focused around technical safety work, but expansion of this to include governance work has been successful and we will need to further expand it, such that there are multiple distinct AI areas of focus within EA.
If you’ve got a very high probability of AI takeover (obligatory reference!), then my first two arguments, at least, might seem very weak because essentially the only thing that matters is reducing the risk of AI takeover.
I'm not even sure your arguments would be weak in that scenario.
e.g. if there were a 90% chance we fall at the first hurdle with an unaligned AI taking over, but also a 90% chance that even if we avoid this, we fall at the second hurdle with a post-AGI world that squanders most of the value of the future, then this would be symmetrical between the problems (it doesn't matter formally which one comes a little earlier). In this case we'd only have a 1% chance of a future that is close to the best we could achieve. Completely solving either problem would increase that to 10%. Halving the chance of the bad outcome for both of them would instead increase that to 30.25% (and would probably be easier than completely solving one). So in this case there would be active reason to work on both at once (even if work on each had a linear effect on its probability).
One needs to add substantive additional assumptions on top of the very high probability of AI takeover to get it to argue against allocating some substantial effort to ensuring that even with aligned AI things go well. e.g. that if AI doesn't takeover, it solves our other problems or that the chance of near-best futures is already very high, etc.
I'm not even sure your arguments would be weak in that scenario.
Thanks - classic Toby point! I agree entirely that you need additional assumptions.
I was imagining someone who thinks that, say, there's a 90% risk of unaligned AI takeover, and a 50% loss of EV of the future from other non-alignment issues that we can influence. So EV of the future is 5%.
If so, completely solving AI risk would increase the EV of the future to 50%; halving both would increase it only to 41%.
But, even so, it's probably easier to halve both than to completely eliminate AI takeover risk, and more generally the case for a mixed strategy seems strong.
I was imagining someone who thinks that, say, there's a 90% risk of unaligned AI takeover, and a 50% loss of EV of the future from other non-alignment issues that we can influence. So EV of the future is 25%.
I'm not understanding - if there's no value in the 90%, and then 50% value in the remaining 10%, wouldn't the EV of the future be 5%?
Rutger Bregman isn’t on the Forum, but sent me this message and gave me permission to share:
Great piece! I strongly agree with your point about PR. EA should just be EA, like the Quakers just had to be Quakers and Peter Singer should just be Peter Singer.
Of course EA had to learn big lessons from the FTX saga. But those were moral and practical lessons so that the movement could be proud of itself again. Not PR-lessons. The best people are drawn to EA not because it’s the coolest thing on campus, but because it’s a magnet for the most morally serious + the smartest people.
As you know, I think EA is at it’s best when it’s really effective altruism (“I deeply care about all the bad stuff in the world, desperately want to make it difference, so I gotta think really fcking hard about how I can make the biggest possible difference”) and not altruistic rationalism (“I’m super smart, and I might as well do a lot of good with it”).
This ideal version EA won’t appeal to all super talented people of course, but that’s fine. Other people can build other movements for that. (It’s what we’re trying to do at The School for Moral Ambition..)
Currently, the online EA ecosystem doesn’t feel like a place full of exciting new ideas, in a way that’s attractive to smart and ambitious people.
I think one thing that has happened is that as EA has grown/professionalized, an increasing share of EA writing/discourse is occurring in more formal outlets (e.g., Works in Progress, Asterisk, the Ezra Klein podcast, academic journals, and so on). As an academic, it's a better use of my time—both from the perspective of direct impact and my own professional advancement—to publish something in one of these venues than to write on the Forum. Practically speaking, what that means is that some of the people thinking most seriously about EA are spending less of their time engaging with online communities. While there are certainly tradeoffs here, I'm inclined to think this is overall a good thing—it subjects EA ideas to a higher level of scrutiny (since we now have editors, in addition to people weighing in on Twitter/the Forum/etc about the merits of various articles) and it broadens exposure to EA ideas.
I also don't really buy that the ideas being discussed in these more formal venues aren't exciting or new; as just two recent examples, I think (1) the discourse/opportunities around abundance are exciting and new, as is (2) much of the discourse happening in The Argument. (While neither of these examples is explicitly EA-branded, they are both pretty EA-coded, and lots of EAs are working on/funding/engaging with them.)
Are the abundance ideas actually new to EA folks? They feel like rehashes of arguments we've had ~ a decade ago, often presented in less technical language and ignoring the major cruxes.
I think you’re right that some of the abundance ideas aren’t exactly new to EA folks, but I also think it’s true that: (1) packaging a diverse set of ideas/policies (re: housing, science, transportation) under the heading of abundance is smart and innovative, (2) there is newfound momentum around designing and implementing an abundance-related agenda (eg), and (3) the implementation of this agenda will create opportunities for further academic research (enabling people to, for instance, study some of those cruxes). All of this to say, if were a smart, ambitious, EA-oriented grad student, I think I would find the intellectual opportunities in this space exciting and appealing to work on.
IMO one way in which EA is very important to AI Safety is in cause prioritization between research directions. For example, there's still a lot of money + effort (e.g. GDM + Anthropic safety teams) going towards mech interp research despite serious questioning to whether it will help us meaningfully decrease x-risk. I think there's a lot of people who do some cause prioritization, come to the conclusion that they should work on AI Safety, and then stop doing cause prio there. I think that more people even crudely applying the scale, tractability, neglectedness framework to AI Safety research directions would go a long way for increasing the effectiveness of the field at decreasing x-risk.
I think EAs and aspiring EAs should try their hardest to incorporate every available piece of evidence about the world when deciding what to do and where to focus their efforts. For better or worse, this includes evidence about AI progress.
The list of important things to do under the "taking AI seriously" umbrella is very large, and the landscape is underexplored so there will likely be more things for the list in due time. So EAs who are already working "in AI safety" shouldn't feel like their cause prioritization is over and done. AI safety is not the end of cause prio.
People interested in funding for field-building projects for the topics on Will's menu above can apply to my team at Open Philanthropy here, or contact us here — we don't necessarily fund all these areas, but we're open to more of them than we receive applications for, so it's worth asking.
One note: I think it would be easy for this post to be read as "EA should be all about AGI" or "EA is only for people who are focused on AGI."
I don't think that is or should be true. I think EA should be for people who care deeply about doing good, and who embrace the principles as a way of getting there. The empirics should be up for discussion.
Interesting post, thanks for sharing. Some rambly thoughts:[1]
I'm sympathetic to the claim that work on digital minds, AI character, macrostrategy, etc is of similar importanceto AI safety/AI governance work. However, I think they seem much harder to work on — the fields are so nascent and the feedback loops and mentorship even scarcer than in AIG/AIS, that it seems much easier to have zero or negative impact by shaping their early direction poorly.
I wouldn't want marginal talent working on these areas for this reason. It's plausible that people who are unusually suited to this kind of abstract low-feedback high-confusion work, and generally sharp and wise, should consider it. But those people are also well-suited to high leverage AIG/AIS work, and I'm uncertain whether I'd trade a wise, thoughtful person working on AIS/AIG for one thinking about e.g. AI character.
(We might have a similar bottom line: I think the approach of "bear this in mind as an area you could pivot to in ~3-4y, if better opportunities emerge" seems reasonable.)
Relatedly, I think EAs tend to overrate interesting speculative philosophy-flavoured thinking, because it's very fun to the kind of person who tends to get into EA. (I'm this kind of person too :) ). When I try to consciously correct for this, I'm less sure that the neglected cause areas you mention seem as important.
I'm worried about motivated reasoning when EAs think about the role of EA going forwards. (And I don't think we should care about EA qua EA, just EA insofar as it's one of the best ways to make good happen.) So reason #2 you mention, which felt more like going "hmm EA is in trouble, what can we do?" rather than reasoning from "how do we make good happen?" wasn't super compelling to me.
That being said, If it's cheap to do so, more EA-flavoured writing on the Forum seems great! The EAF has been pretty stale. I was brainstorming about this earlier —initially I was worried about the chilling effect of writing so much in public (commenting on the EAF is way higher effort for me than on google docs, for example), but I think some cool new ideas can and probably should be shared more. I like Redwood's blog a lot partly for this reason.
In my experience at university, in my final 2 years the AI safety group was just way more exciting and serious and intellectually alive than the EA group — this is caricatured, but one way of describing it would be that (at extremes) the AI safety group selected for actually taking ideas seriously and wanting to do things, and the EA group correspondingly selected for wanting to pontificate about ideas and not get your hands dirty. I think EA groups engaging with more AGI preparedness-type topics could help make them exciting and alive again, but it would be important imo to avoid reinforcing the idea that EA groups are for sitting round and talking about ideas, not for taking them seriously. (I'm finding this hard to verbalise precisely —I think the rough gloss is "I'm worried about these topics having more of a vibe of 'interesting intellectual pastime', and if EA groups tend towards that vibe anyway, making discussing them feel ambitious and engaging and 'doing stuff about ideas'-y sounds hard".
I would have liked to make this more coherent and focused, but that was enough time/effort that realistically I just wouldn't have done it, and I figured a rambly comment was better than no comment.
Yes, at least initially. (Though fwiw my takeaway from that was more like, "it's interesting that these people wanted to direct their energy towards AI safety community building and not EA CB; also, yay for EA for spreading lots of good ideas and promoting useful ways of looking at problems". This was in 2022, where I think almost everyone who thought about AI safety heard about it via EA/rationalism.)
What sort of things did the AIS group do that gave the impression they were taking ideas more seriously? Was it more events surrounding taking action (e.g. Hackathons)? Members engaging more with the ideas in the time outside of the club meetings? More seriousness in reorienting their careers based on the ideas?
If you’ve got a very high probability of AI takeover (obligatory reference!), then my first two arguments, at least, might seem very weak because essentially the only thing that matters is reducing the risk of AI takeover.
I do think the risk of AI takeover is much higher than you do, but I don't think that's why I disagree with the arguments for more heavily prioritizing the list of (example) cause areas that you outline. Rather, it's a belief that's slightly upstream of my concerns about takeover risk - that the advent of ASI almost necessarily[1] implies that we will no longer have our hands on the wheel, so to speak, whether for good or ill.
An unfortunate consequence of having beliefs like I do about what a future with ASI in it involves is that those beliefs are pretty totalizing. They do suggest that "making the transition to a post-ASI world go well" is of paramount importance (putting aside questions of takeover risk). They do not suggest that it would be useful for me to think about most of the listed examples, except insofar as they feed into somehow getting a friendly ASI rather than something else. There are some exceptions: for example, if you have much lower odds of AI takeover than I do, but still expect ASI to have this kind of totalizing effect on the future, I claim you should find it valuable for some people to work on "animal welfare post-ASI", and whether there is anything that can meaningfully be done pre-ASI to reduce the risk of animal torture continuing into the far future[2]. But many of the other listed concerns seem very unlikely to matter post-ASI, and I don't get the impression that you think we should be working on AI character or preserving democracy as instrumental paths by which we reduce the risk of AI takeover, bad/mediocre value lock-in, etc, but because you consider things like that to be important separate from traditional "AI risk" concerns. Perhaps I'm misunderstanding?
It is perhaps not a coincidence that I expect this work to initially look like "do philosophy", i.e. trying to figure out whether traditional proposals like CEV would permit extremely bad outcomes, looking for better alternatives, etc.
I’m not sure I understood the last sentence. I personally think that a bunch of areas Will mentioned (democracy, persuasion, human + AI coups) are extremely important, and likely more useful on the margin than additional alignment/control/safety work for navigating the intelligence explosion. I’m probably a bit less “aligned ASI is literally all that matters for making the future go well” pilled than you, but it’s definitely a big part of it.
I also don’t think that having higher odds of AI x-risk are a crux, though different “shapes” of intelligence explosion could be , e.g. if you think we’ll never get useful work for coordination/alignment/defense/ai strategy pre-foom then I’d be more compelled by the totalising alignment view - but I do think that’s misguided.
I’m probably a bit less “aligned ASI is literally all that matters for making the future go well” pilled than you, but it’s definitely a big part of it.
Sure, but the vibe I get from this post is that Will believes in that a lot less than me, and the reasons he cares about those things don't primarily route through the totalizing view of ASI's future impact. Again, I could be wrong or confused about Will's beliefs here, but I have a hard time squaring the way this post is written with the idea that he intended to communicate that people should work on those things because they're the best ways to marginally improve our odds of getting an aligned ASI. Part of this is the list of things he chose, part of it is the framing of them as being distinct cause areas from "AI safety" - from my perspective, many of those areas already have at least a few people working on them under the label of "AI safety"/"AI x-risk reduction".
Like, Lightcone has previously and continues to work on "AI for better reasoning, decision-making and coordination". I can't claim to speak for the entire org but when I'm doing that kind of work, I'm not trying to move the needle on how good the world ends up being conditional on us making it through, but on how likely we are to make it through at all. I don't have that much probability mass on "we lose >10% but less than 99.99% of value in the lightcone"[1].
Edit: a brief discussion with Drake Thomas convinced me that 99.99% is probably a pretty crazy bound to have; let's say 90%. Wqueezing out that extra 10% involves work that you'd probably describe as "macrostrategy", but that's a pretty broad label.
I don't understand why you think some work on animal wefare post-ASI looks valuable, but not (e.g.) digital minds post-ASI and s-risks post-ASI. To me, it looks like working on these causes (and others?) have similar upsides (scale, neglectedness) and downsides (low tractability if ASI changes everything) to working on animal welfare post-ASI. Could you clarify why they're different?
I'd love it if people replied to this comment with ideas for how the EA Forum could play a role in generating more discussion of post-AGI governance. I haven't planned the events for next year yet...
At a previous EAG, one talk was basically GovAI fellows summarizing their work, and I really enjoyed it. Given that there's tons of fellowships that are slated to start in the coming months, I wonder if there's a way to have them effectively communicate about their work on the forum? A lot of the content will be more focused on traditional AIS topics, but I expect some of the work to focus on topics more amenable to a post-AGI governance framing, and that work could be particularly encouraged.
A light touch version might just be reaching out to those running the fellowships and having them encourage fellows to post their final product (or some insights for their work if they're not working on some singular research piece) to the forum (and ideally having them include a high-quality executive summary). The medium touch would be having someone curate the projects, e.g. highlighting the 10 best from across the fellowships. The full version could take many different forms, but one might be having the authors then engage with one another's work, encouraging disagreement and public reasoning on why certain paths might be more promising.
Great idea! I reached out to GovAI to give their fellows a talk on Forum writing once but it didn't end up happening - I'll try again soon/ try another method of reaching them.
Thanks for writing this Will. I feel a bit torn on this so will lay out some places where I agree and some where I disagree:
I agree that some of these AI-related cause areas beyond takeover risk deserve to be seen as their own cause areas as such and that lumping them all under "AI" risks being a bit inaccurate.
That said, I think the same could be said of some areas of animal work—wild animal welfare, invertebrate welfare, and farmed vertebrate welfare should perhaps get their own billing. And then this can keep expanding—see, e.g., OP's focus areas, which list several things under the global poverty bracket.
Perhaps on balance I'd vote for poverty, animals, AI takeover, and post-AGI governance or something like that.
I also very much agree that "making the transition to a post-AGI society go well" beyond AI takeover is highly neglected given its importance.
I'm not convinced EA is intellectually adrift and tend to agree with Nick Laing's comment. My quick take is that it feels boring to people who've been in it a while but still is pretty incisive for people who are new to it, which describes most of the world.
I think principles-first EA goes better with a breadth of focuses and cause areas, because it shows the flexibility of the principles and the room for disagreement within them. I tend to think that too much focus on AI can take away with this, so it would concern me if >50-60% of the discussion were around AI.
I very much agree with the PR mentality comments—in particular, I find many uses of the "EA adjacent" term to be farcical. I added effective altruism back into my Twitter bio inspired by this post and @Alix Pham's.
I agree it would be good for the EA Forum to be a place where more of the AI discussion happens, and I think it's particularly suited for post-AGI society—it's been a good place for digital minds conversations, for example.
So I guess I come down on the side of thinking (a) members of the EA community should recognize that there's a lot more to discuss around AI than takeover, and it merits a rich and varied conversation, but (b) I would be wary of centering the transition to a post-AGI society go well at the expense of other cause areas.
I very much agree with the PR mentality comments—in particular, I find many uses of the "EA adjacent" term to be farcical. I added effective altruism back into my Twitter bio inspired by this post and @Alix Pham's.
Thank you for acting on this! It's a team effort :)
Very excited to read this post. I strongly agree with both the concrete direction and with the importance of making EA more intellectually vibrant
Then again, I'm rather biased since I made a similar argument a few years back (admittedly, my post wasn't even in the same ballpark in terms of quality though).
I looked back and compared what I wrote then to Will's suggestions here as I wanted to see if any of my old ideas were still relevant:
I suggested that it might make sense for virtual programs to create a new course rather than just changing the intro fellowship content. My current intuition is that splitting the intro fellowship would likely be the best option for now. Some people will get really annoyed if the course focuses too much on AI, whilst others will get annoyed if the course focuses too much on questions that would likely become redundant in a world where we expect capability advances to continue. My intuition is that things aren't at the stage where it'd make sense for the intro fellowship to do a complete AGI pivot, so that's why I'm suggesting a split. Both courses should probably still give participants a taste of the other.
I put more emphasis on the possibility that AI might be useful for addressing global poverty and that it intersects with animal rights, whilst perhaps Will sees this as too incrementalist?
Whilst I also suggested that putting more emphasis on the implications of advanced AI might make EA less intellectually stagnant, I also noted that perhaps it'd be better for EA to adopt a yearly theme and simply make the rise of AI the first such theme. I still like the yearly theme idea, but the odds and legibility of AI being really important have increased enough that I'm now feeling a lot more confident as identifying AI as an area that deserves more than just a yearly theme.
I also agree with the "fuck PR" stance (I doubt Will would use these words to characterise his stance). Especially insofar as the AIS movement has greater pressure to focus on PR these days, since it's further towards the pointy end, I think it's important for the EA movement to use its freedom to provide a counter-balance to this.
Harangue old-hand EA types to (i) talk about and engage with EA (at least a bit) if they are doing podcasts, etc; (ii) post on Forum (esp if posting to LW anyway), twitter, etc, engaging in EA ideas; (iii) more generally own their EA affiliation.
I think the carrot is better than the stick. Rather than (or in addition to) haranguing people who don't engage, what if we reward people who do engage? (Although I'm not sure what "reward" means exactly)
You could say I'm an old-hand EA type (I've been involved since 2012) and I still actively engage in the EA Forum. I wouldn't mind a carrot.
Will, I think you deserve a carrot, too. You've written 11 EAF posts in the past year! Most of them were long, too! I've probably cited your "moral error" post about a dozen times since you wrote it. I don't know how exactly I can reward you for your contributions but at a minimum I can give you a well-deserved compliment.
I see many other long-time EAs in this comment thread, most of whom I see regularly commenting/posting on EAF. They're doing a good job, too!
(I feel like this post sounds goofy but I'm trying to make it come across as genuine, I've been up since 4am so I'm not doing my best work right now)
Haha, thank you for the carrot - please have one yourself!
"Harangue" was meant to be a light-hearted term. I agree, in general, on carrots rather than sticks. One style of carrot is commenting things like "Great post!" - even if not adding any content, I think it probably would increase the quantity of posts on the Forum, and somewhat act as a reward signal (more than just karma).
Currently, the online EA ecosystem doesn’t feel like a place full of exciting new ideas, in a way that’s attractive to smart and ambitious people
This may be partly related to the fact that EA is doing relatively little cause and cross-cause prioritisation these days (though, since we posted this, GPI has wound down and Forethought has spun up).
People may still be doing within-cause, intervention-level prioritisation (which is important), but this may be unlikely to generate new, exciting ideas, since it assumes causes, and works only within them, is often narrow and technical (e.g. comparing slaughter methods), and is often fundamentally unsystematic or inaccessible (e.g. how do I, a grantmaker, feel about these founders?).
I ran this through QURI's RoastMyPost.org, and it gave a mixed but fairly positive assessment (something like 68/100). Full assessment here (multiple agents/tools).
The epistemic checker and the fact checker seem particularly useful.
The main limitations seem to be:
Strong claims and major recommendations without corresponding evidence and support
Vagueness/imprecise definition (this is a partially my own take, partially echoed by RoastMyPost – e.g., it's hard for me to grok what these new cause areas are, some are very much shorthand.)
I meant to post this last week, and I meant to give a more detailed response, but I was overloaded with work and illness. Posting this response now in the belated spirit of amnesty week.
Post-FTX, I think core EA adopted a “PR mentality” that (i) has been a failure on its own terms and (ii) is corrosive to EA’s soul.
I find it helpful to distinguish two things, one which I think EA is doing too much of and one which EA is doing too little of:
Suppressing (the discussion of) certain ideas (e.g. concern for animals of uncertain sentience): I agree this seems deeply corrosive (even if an individual could theoretically hold onto the fact that x matters and even act to advance the cause of x, while not talking publicly about it, obviously the collective second-order effects mean that not publicly discussing x prevents many other people forming true beliefs about or acting in service of x (often with many other downstream effects on their beliefs and actions regarding y, z...).
Attending carefully to the effect of communicating ideas in different ways: how an idea is communicated can make a big difference to how it is understood and received (even if all the expressions of the idea are equally accurate). For example, if you talk about "extinction from AI", will people even understand this to refer to extinction and not the metaphorical extinction of job losses, or per your recent useful example, if you talk about "AI Safety", will people understand this to refer to mean "stop all AI development". I think this kind of focus on clear and compelling communication is typically not corrosive, but often neglected by EAs (and often undertaken only at the level of intuitive vibes, rather than testing how people receive communications differently framed.
Thanks - I agree the latter is important, and I think it's an error if "Attending carefully to the effect of communicating ideas in different ways" (appreciating that most of your audience is not extremely high-decoupling, etc) is rounded off to being overly focused on PR.
This post really resonates with me, and its vision for a flourishing future for EA feels very compelling. I'm excited that you're thinking and writing about it!
Having said that, I don't know how to square that with the increased specialisation which seems like a sensible outcome of doing things at greater scale. Personally, I find engaging on the forum / going to EAGs etc pretty time/energy-intensive (though I think a large part of that is because I'm fairly introverted, and others will likely feel differently). I also think there's a lot of value in picking a hard goal and running full pelt at it. So from my point of view, I'd prefer solutions that looked more like 'create deliberate roles with this as the aim' and less like 'harangue people to take time away from their other roles'.
Hello, I am relatively new to EA (came in through the academic sibling "Effective Thesis"), but this post is truely enlightening about the original values and goals of the community!
To be honest, it is the EA's PR actions that brought me to this side of the internet... I did find an alignment with my values, and opted to pursue incremental engagement. As I was focusing on political violence and, back then, considered it was the most impactful and urgent societal issue to address on my scale.
I hope I will be able to do as much good as possible (as a researcher and recent graduate), and participate to the mission of EA while bringing fresh ideas to the discussion that are not systematically AGI oriented ~
Not really a criticism of this post specifically, but I've seen a bunch of enthusiasm about the idea of some sort of AI safety+ group of causes and not that much recognition of the fact that AI ethicists and others not affiliated with EA have already been thinking about and prioritizing some of these issues (particularly thinking of the AI and democracy one, but I assume it applies to others). The EA emphases and perspectives within these topics have their differences, but EA didn't invent these ideas from scratch.
Focusing EA action on an issue as technically complex as AI safety, on which there isn't even a scientific consensus comparable to the fight against climate change, means that all members of the community make their altruistic action dependent on the intellectual brilliance of their leaders, whose success we could not assess. This is a circumstance similar to that of Marxism, where everything depended on the erudition of the most distinguished scholars of political economy and dialectical materialism.
Utilitarianism implies an ethical commitment of the individual through unequivocal action, in which altruistic means and ends are proportionate. It is unequivocal that there are unfortunate people in poor countries who suffer avoidable suffering and who could be helped with a little of the money we have to spare in rich countries. Altruistic action also implies—and this is extremely important—a visualization of moral change, especially if it is culturally organized in the form of a social movement. Community action for unequivocal moral progress can, in turn, lead to successful long-term action.
If there are so many new promising causes that EA should plausibly focus on (from something like 5 to something like 10 or 20), cause-prio between these causes (and ideas for related but distinct causes) should be especially valuable as well. I think Will agrees with this - after all his post is based on exactly this kind of work! - but the emphasis in this post seemed to be that EA should invest more into these new cause areas rather than investigate further which of them are the most promising - and which aren't that promising after all. It would be surprising if we couldn't still learn much more about their expected impact.
I think this is an interesting vision to reinvigorate things and do kind of feel sometimes "principles first" has been conflated with just "classic EA causes."
To me, "PR speak" =/= clear effective communication. I think the lack of a clear, coherent message is most of what bothers people, especially during and after a crisis. Without that, it's hard to talk to different people and meet them where they're at. It's not clear to me what the takeaways were or if anyone learned anything.
I feel like "figuring out how to choose leaders and build institutions effectively" is really neglected and it's kind of shocking there doesn't seem to be much focus here. A lingering question for me has been "Why can't we be more effective in who we trust?" and the usual objections sort of just seem like "it's hard." But so is AI safety, biorisk, post-AGI prep, etc... so that doesn't seem super satisfying.
There's recently been increased emphasis on "principles-first" EA, which I think is great. But I worry that in practice a "principles-first" framing can become a cover for anchoring on existing cause areas, rather than an invitation to figure out what other cause areas we should be working o
I don't quite see the link here. Why would principals first be a cover for anchoring on existing cause areas? Is there a prominent example of this?
I indeed (still) think dissociation is hurting the movement, and I would love for more EA types to engage and defend their vision rather than step aside because they feel the current vibe/reputation doesn't fit with what they want to build. It would make us a better community as a whole. Though I understand it takes time and energy, and currently it seems engaging in the cause area itself (especially AI safety) has higher return on investment than writing on the EA Forum.
First, calling GPT-4 "the first very weak AGI" is highly off-putting to me. Either that's just not true or "very weak AGI" isn’t a meaningful term. This makes the whole post feel less credible to me.
Second, I find the praise of LessWrong in this post disturbing. There is no credible evidence of LessWrong promoting exceptionally rational thought — to quote Eliezer Yudkowsky, in some sense the test of rationality must be whether the supposedly exceptionally rational people are "currently smiling from on top of a giant heap of utility", which is not remotely true for LessWrong — and there is a lot of evidence of irrationality and outright delusion associated with LessWrong. LessWrong is also has a profoundly morally troubling track record, to such an alarming extent that I would encourage people involved in effective altruism to dissociate from LessWrong.
I very, very strongly believe there’s essentially no chance of AGI being developed within the next 7 years. I wrote a succinct list of reasons to doubt near-term AGI here. (For example, it might come as a surprise that around three quarters of AI experts don’t think scaling up LLMs will lead to AGI.) I also highly recommend this video by an AI researcher that makes a strong case for skepticism of near-term AGI and of LLMs as a path to AGI.
By essentially no chance, I mean less than the chance of Jill Stein running as the Green Party candidate in 2028 and winning the U.S. Presidency. Or, if you like, less than the chance Jill Stein had of winning the presidency in 2024. I mean it’s an incredible long shot, significantly less than a 0.1% chance. (And if I had to give a number on my confidence of this, it would be upwards of 95%.)
By AGI, I mean a system that can think, plan, learn, and solve problems just like humans do, with at least an equal level of data efficiency (e.g. if a human can learn from one example, AGI must also be able to learn from one example and, not, say, one million), with at least an equal level of reliability (e.g. if humans do a task correctly 99.999% of the time, AGI must match or exceed that), with an equal level of fluidity or adaptability to novel problems and situations (e.g. if a human can solve a problem with zero training examples, AGI must be able to as well), and with an equal ability to generate novel and creative ideas. This is the only kind of AI system that could plausibly automate all human labour or cunningly take over the world. No existing AI system is anything like this.
As we go further out into the future, it becomes less and less clear what will happen. I think it could be significantly more than 100 years before AGI is developed. It could also be significantly less. Who knows? We don’t have a good understanding of fundamental ideas like what intelligence is, how it works, how to measure it, or how to measure progress toward AGI. It would be surprising if we could predict the invention of a technology — something that is generally not possible anyway — whose fundamental scientific nature we understand so poorly.
In March 2025, Dario Amodei, the CEO of Anthropic, predicted that 90% of code would be written by AI as early as June 2025 and no later than September 2025. This turned out to be dead wrong. In 2016, the renowned AI researcher Geoffrey Hinton predicted that by 2021, AI would automate away all radiology jobs and that turned out to be dead wrong. Even 9 years later, the trend has moved in the opposite direction and there is no indication radiology jobs will be automated away anytime soon. Many executives and engineers working in autonomous driving predicted we’d have widespread fully autonomous vehicles long ago; some of them have thrown in the towel. Cruise Automation, for example, is no more.
I expect that in 7 years from now (2032), we will have many more examples of this kind of false prediction, except on a somewhat grander scale, since people were predicting a high chance of either utopia or doom and trying to marshal a commensurate amount of resources and attention. I somewhat dread people simply doubling down and saying they were right all along regardless of what happens. Possible responses I dread:
-But ChatGPT is AGI! (A few people are in fact already saying this or something close to it. For example, Tyler Cowen said that o3 is AGI. I don’t know what "very weak AGI" is supposed to mean, but I most likely disagree that GPT-4 is that.)
-We were never sure anything would happen by 2032! What we really said was…
-We successfully averted the crisis! Mission accomplished! Thanks to us, AI progress was slowed and the world is saved!
I honestly don’t know who is interested in having this debate. I don’t get the sense that many people are eager to have it. In theory, I’d like to see people engage more deeply with the substantive, informed criticism (e.g. anti-LLM arguments from people like Yann LeCun and François Chollet; data showing little economic or practical benefit to AI, or even negative benefit). But since not many people seem interested in this, I guess I’m fine with that, since if we just wait 7 years the debate will settle itself.
Sure, François Chollet recently changed his prediction from AGI maybe in about 10 years to AGI maybe in about 5 years, but his very incisive arguments about the shortcomings of LLMs are logically and intellectually independent from his, in my opinion, extremely dubious prediction of AGI in about 5 years. I would like people who believe LLMs will scale to AGI to seriously engage with his arguments about why it won't. His prediction about when AGI will happen is kind of beside the point.
That's basically all that needs to be said about this, but I'll elaborate anyway because I think the details are interesting and intellectually helpful.
If I understand François Chollet's point of view correctly, his prediction of AGI in about 5 years depends on a) program synthesis being the key to solving AGI and b) everything required for program synthesis to take deep learning the rest of the way to AGI being solved within about 5 years. I have extreme doubts about both (a) and (b) and, for that matter, I would guess most people who think AGI will come within 7 years have strong doubts about at least (a). Thinking that LLMs will scale to AGI and believing that solving program synthesis is required to achieve AGI are incompatible views.
François Chollet, Yann LeCun of FAIR, Jeff Hawkins of Numenta, and Richard Sutton of Amii and Keen Technologies are four AI researchers who have strongly criticized LLMs[1] and have proposed four different directions for AI research to get to AGI:
Reinforcement learning and the Alberta Plan for Richard Sutton
All four have also at times made statements indicating they think their preferred research roadmap to AGI can be executed on a relatively short timescale, although with some important caveats from the three other than Chollet. (Chollet might have given a caveat somewhere that I missed. I think he's said the least on this topic overall or I've missed what he's said.) I already described Chollet's thoughts on this. As for the others:
Yann LeCun gave a nuanced and self-aware answer, saying, "Five to ten years would be if everything goes great. All the plans that we've been making will succeed. We're not going to encounter unexpected obstacles, but that is almost certainly not going to happen." In another interview, he said, "all of this is going to take at least a decade and probably much more because there are a lot of problems that we’re not seeing right now that we have not encountered".
When discussing the prospect of Numenta creating a machine that can think like a human (not explicitly using the term "AGI"), Jeff Hawkins said, "It's not going to take decades, it's a matter of a few years, optimistically, but I think that's possible."
Richard Sutton has said he thinks there's a 25% chance by 2030 we'll "understand intelligence", by which I think he means we'll attain the requisite knowledge to build AGI, but I'm not sure. I also think when he says "we" he specifically means himself and his colleagues, but I'm also not sure.
So, Chollet, LeCun, Hawkins, and Sutton all think LLMs are insufficient to get to AGI. They all argue that AGI requires fundamentally different ideas than what the mainstream of AI research is focusing on — and they advocate for four completely different ideas.[2] And all four of them are optimistic that their preferred research roadmap has a realistic chance of getting to AGI within a relatively short time horizon. Isn't that interesting?
Either three out of four of them have to be wrong about the key insights needed to unlock AGI or all four them have to be wrong. (I guess the solution to AGI could also turn out to be some combination of their ideas, in which case they would be partially right and partially wrong in some combination.) It's interesting that all four independently, simultaneously think their respective roadmaps are viable on a roughly similar timescale when a) it would be hard to imagine a strong theoretical or intellectual reason to support the idea that we will figure out the solution to AGI soon, regardless of what the solution actually is (e.g. whether it requires simulating cortical columns or using non-deep learning AI methods or using novel deep learning methods) and b) there are psychological reasons to believe such things, like the appeal of having an ambitious but achievable goal on a timescale that's motivating.
I find what Yann LeCun has said about this to be the wisest and most self-aware. I can't find the interview where he said this, but he said something along the lines of (paraphrasing based on memory): over the last several decades, many people have come along and said they have the solution to AGI and they've all been wrong. So, if someone comes along and tells you they have the solution to AGI, you should not believe them. I'm another person who's coming along and telling you I have the solution to AGI, and you should not believe me. But I still think I'm right.
I think (or at least hope), once described like this, in the context I just gave above, that kind of skepticism will strike most people as logical and prudent.
The implication of this is that if you accept François Chollet's arguments about why LLMs won't scale to AGI, which I think are quite good, you should still be skeptical of his view that he's found the solution to AGI, that it's program synthesis, and that the research roadmap for program synthesis (leading from its current state all the way to AGI) can be completed maybe in about 5 years. By default, you should be skeptical of any claims of this kind. But that claim has no bearing on whether his arguments about the fundamental weaknesses of LLMs are correct.
Chollet believes that the major AI labs are in fact working on program synthesis, but as far as I know, this hasn't been confirmed by any lab and, if it is happening, it hasn't made its way into published research yet.
I agree that whether or not we get AGI is a crux for this topic. Though it makes sense to update our cause priorities even if AI is merely transformational.
Your comment seems overconfident, however ("essentially no chance of AGI"). This seems to not take into account that many (most?) intellectual tasks see progress. For example ARC-AGI-2 had scores below 10% at the beginning of the year, and within just few months the best solution on https://arcprize.org/leaderboard scores 29%. Even publicly available models without custom scaffolding score >10% now.
Of course, there could be a plateau... and I hear ARC-AGI-3 is in the works... but I don't understand your high confidence despite AI's seemingly consistent rise in all the tests that humanity throws at it.
Forgive me for the very long reply. I’m sure that you and others on the EA Forum have heard the case for near-term AGI countless times, often at great depth, but the opposing case is rarely articulated in EA circles, so I wanted to do it justice that a tweet-length reply could not do.
Why does the information we have now indicate AGI within 7 years and not, say, 17 years or 70 years or 170 years? If progress in science and technology continues indefinitely, then eventually we will gain the knowledge required to build AGI. But when is eventually? And why would it be so incredibly soon? To say that some form of progress is being made is not the same as making an argument for AGI by 2032, as opposed to 2052 or 2132.
I wouldn’t say that LLM benchmarks accurately represent what real intellectual tasks are actually like. First, the benchmarks are designed to be solvable by LLMs because they are primarily intended to measure LLMs against each other and to measure improvements in subsequent versions of the same LLM model line (e.g. GPT-5 vs GPT-4o). There isn’t much incentive to create LLM benchmarks where LLMs stagnate around 0%.[1]
Even ARC-AGI 1, 2, and 3, which are an exception in terms of their purpose and design are still intended to be in the sweet spot between too easy to be a real challenge and too hard to see progress on. If a benchmark is easy to solve or impossible to solve, it won’t encourage AI researchers and engineers to try hard to solve it and make improvements to their models in the process. The intention of the ARC-AGI is to give people working on AI a shared point of focus and a target to aim for. The purpose is not to make a philosophical or scientific point about what current AI systems can’t do. The benchmarks are designed to be solved by current AI systems.
It always bears repeating, since confusion is possible, that the ARC-AGI benchmarks are not intended to test whether a system is AGI or not, but are rather intended to test whether AI systems are making progress toward AGI. So, getting 95%+ on ARC-AGI-2 would not mean AGI is solved, but it would be a sign of progress — or at least that is the intention.
Second, for virtually all LLMs tests or benchmarks, the definition of success or failure on the tasks has to be reduced to something simple enough that software can grade the task automatically. This is a big limitation.
When I think about the sort of intellectual tasks that humans do, not a lot of them can be graded automatically. Of course, there are written exams and tests with multiple choice answers, but these are primarily tests of memorization. Don’t get me wrong, it is impressive that LLMs can memorize essentially all text ever written, but memorization is only one aspect of intelligence. We want AI systems that go beyond just memorizing things from huge numbers of examples and can also solve completely novel problems that aren’t a close match for anything in their training dataset. That’s where LLMs are incredibly brittle and start just generating nonsense, saying plainly false (and often ridiculous) things, contradicting themselves, hallucinating, etc. Some great examples are here, and there's also an important discussion of how these holes in LLM reasoning get manually patched by paying large workforces to write new training examples specifically to fix them. This creates an impression of increased intelligence, but the improvement isn't from scaling in these cases, it's from large-scale special casing.
I think the most robust tests of AI capabilities are tasks that have real world value. If AI systems are actually doing the same intellectual tasks as human beings, then we should see AI systems either automating labour or increasing worker productivity. We don’t see that. In fact, I’m aware of two studies that looked at the impact of AI assistance on human productivity. One study on customer support workers found mixed results, including a negative impact on productivity for the most experienced employees. Another study, by METR, found a 19% reduction in productivity when coders used an AI coding assistant.
In industry, non-AI companies that have invested in applying AI to the work they do are not seeing that much payoff. There might be some modest benefits in some niches. I’m sure there are at least a few. But LLMs are not going to be transformational to the economy. Let alone automate all office work.
Personally, I find ChatGPT to extremely useful as an enhanced search engine. I call it SuperGoogle. If I want to find a news article or an academic paper or whatever about a certain very specific topic, I can ask GPT-5 Thinking or o3 to go look to see if anything like that exists. For example, I can say, “Find me any studies that have been published comparing the energy usage of biological systems to technological systems, excluding brains and computers.” And then it often will give me some stuff that isn’t helpful, but it is good enough at digging up a really helpful link often enough that it’s still a really useful tool overall. I don’t know how much time this saves me Googling, but it feels useful. (It’s possible like the AI coders in the METR study, I’m falling prey to the illusion that it’s saving me time when actually, on net, it wastes time, but who knows.)
This is a genuine innovation. Search engines are an important tool and such a helpful innovation on the search engine is a meaningful accomplishment. But this is an innovation on the scale of Spotify allowing us to stream music, rather than something on the scale of electricity or the steam engine or the personal computer. Let alone something as revolutionary as the evolution of the human prefrontal cortex.
If LLMs were genuinely replicating human intelligence, we would expect to see an economic impact, excluding the impact of investment. Investment is certainly having an impact, but, as the economist John Maynard Keynes said, if you pay enough people to dig holes and then fill the same holes up with dirt again, that stimulus will impact the economy (and may even get you out of a recession). What economic impact is AI having over and above the impact that would have been had by using the same capital on to pay people to dig holes and fill them back up? From the data I’ve seen, the impact is quite modest and a huge amount of capital has been wasted. I think within a few years many people will probably see the recent AI investment boom as just a stunningly bad misallocation of capital.[2]
People draw analogies to the transcontinental railway boom and the dot com bubble, but they also point out that railways and fibre-optic cable depreciate at a much slower rate than GPUs. Different companies calculate the depreciation of GPUs at different rates, typically ranging from 1 year to 6 years. Data centres have non-GPU aspects like the buildings and the power connects that are more durable, but the GPUs account for more than half of the costs. So, overbuilding capacity for demand that doesn’t ultimately materialize would be extremely wasteful. Watch this space.[3]
If you think an LLM scoring more than 100 on an IQ test means it's AGI, then we've had AGI for several years. But clearly there's a problem with that inference, right? Memorizing the answers to IQ tests, or memorizing similar answers to similar questions that you can interpolate, doesn't mean a system actually has the kind of intelligence to solve completely novel problems that have never appeared on any test, or in any text. The same general critique applies to the inference that LLMs are intelligent from their results on virtually any LLM benchmark. Memorization is not intelligence.
If we instead look at performance on practical, economically valuable tasks as the test for AI's competence at intellectual tasks, then its competence appears quite poor. People who make the flawed inference from benchmarks just described say that LLMs can do basically anything. If they instead derived their assessment from LLMs' economic usefulness, it would be closer to the truth to say LLMs can do almost nothing.
There is also some research on non-real world tasks that supports the idea that LLMs are mass-scale memorizers with a modicum of interpolation or generalization to examples similar to what they've been trained on, rather than genuinely intelligent (in the sense that humans are intelligent). The Apple paper on "reasoning" models found surprisingly mixed results on common puzzles. The finding that sticks out most in my mind is that the LLM's performance on the Tower of Hanoi puzzle did not improve after being told the algorithm for solving the puzzle. Is that real intelligence?
It's possible, at least in principle (not sure it often happens in practice), to acknowledge these flaws in LLMs and still believe in near-term AGI. If there's enough progress in AI fast enough, then we could have AGI within 7 years. This is true, but it was also true ten years ago. When AlphaGo beat Lee Sedol in 2016, you could have said we'll have AGI within 7 years — because, sure, being superhuman at go isn't that close to AGI, but look at how fast the progress has been, and imagine how fast the progress will be![4] If you think it's just a matter of scaling, then I could understand how you would see the improvement as predictable. But I think the flaws in LLMs are inherent to LLMs and can't be solved through scaling. The video from AI researcher Edan Meyer that I linked to in my original comment makes an eloquent case for this. As does the video with François Chollet.
There are other problems with the scaling story:
There is evidence that scaling LLMs is running out of steam. Toby Ord's interview on the 80,000 Hours podcast in June covered this topic really well. Renowned AI researcher Ilya Sutskever, formerly chief scientist at OpenAI (prior to voting to fire Sam Altman), has said he thinks the benefits from scaling LLM pre-training have plateaued. There have been reports that, internally, employees at AI labs are disappointed with their models' progress. GPT-5 doesn't seem like that much of an improvement over previous models.
There are practical limits to scaling up, even if the benefits to scaling weren't diminishing. Epoch AI's median estimate of when LLMs will run out of data to train on is 2028. Epoch AI also predicts that compute scaling will slow down mainly due to financial and economic considerations.
The benefits to scaling are diminishing and, at the same time, data scaling and compute scaling may have to slow down sometime soon (if this is not already happening).
If you expand the scope of LLM performance beyond written prompts and responses to "agentic" applications, I think LLMs' failures are more stark and the models do not seem to be gaining mastery of these tasks particularly quickly. Journalists generally say that companies' demos of agentic AI don't work.
I don't expect that performance on agentic tasks will rapidly improve. To train on text-based tasks, AI labs can get data from millions of books and large-scale scrapes of the Internet. There aren't similarly sized datasets for agentic tasks. In principle, you can use pure reinforcement learning without bootstrapping from imitation learning, but while this approach has succeeded in domains with smaller spaces of possible actions like go, it has failed in domains with larger spaces of possible actions like StarCraft. I don't think agentic AI will get particularly better over the next few years. Also, the current discrepancy between LLM performance on text-based tasks and agentic tasks tells us something about whether LLMs are genuinely intelligent. What kind of PhD student can't use a computer?
So, to briefly summarize the core points of this very long comment:
LLM benchmarks don't really tell us how genuinely intelligent LLMs are. They are designed to be easy for LLMs and to be automatically graded, which limits what can be tested.
On economically valuable tasks in real world settings, which I believe are much better tests than benchmarks, LLMs do quite poorly. Not only does this make near-term AGI seem very unlikely, it also makes economically transformative AI in the near term seem very unlikely.
LLMs fail all the time at tasks we would not expect them to fail at if they were genuinely intelligent, as opposed to relying on mass-scale memorization.
Scaling isn't a solution to the fundamental flaws in LLMs and, in any case, the benefits of scaling are diminishing at the same time that LLM companies are encountering practical limits that may slow compute scaling and slow or even stop data scaling.
LLMs are terrible at agentic tasks and there isn't enough training data for them to improve, if training data is what it takes. If LLMs are genuinely intelligent, we should ask why they can't learn agentic tasks from a small number of examples, since this is what humans do.
Maybe it's worth mentioning the very confusing AI Impacts surveyconducted in 2022 where the surveyors gave 2,778 AI researchers essentially two different descriptions of an AI system that could be construed as an AGI and also could be construed as equivalent to each other (I don't know why they designed the survey like this) and, aggregating the AI researchers' replies, found they assign a 50% chance of AGI by 2047 (or a 10% chance by 2027) on one definition and a 50% chance of AGI by 2116 (or a 10% chance by 2037) on another definition.
[Important correction: this is actually the 2023 AI Impacts survey, which was conducted in October 2023, seven months after the release of GPT-4 in March 2023.
This correction was added on October 28, 2025 at 10:31 AM Eastern.]
In 2022, there was also a survey of superforecasters with a cleaner definition of AGI. They, in aggregate, assigned a 1% chance of AGI by 2030, a 21% chance by 2050, a 50% chance by 2081, and a 75% chance by 2100.
Both the AI Impact survey and the superforecaster survey were conducted before the launch of ChatGPT. I would guess ChatGPT would probably have led them to shorten their timelines, but if LLMs are more or less a dead end, as people like the Turing Award winners Yann LeCun and Richard Sutton have argued,[5] then this would be a mistake. (In a few years, if things go the way I expect, meaning that generative AI turns out to be completely disappointing and this is reflected in finance and the economy, then I would guess the same people would then lengthen their timelines again.) In any case, it would be interesting to run the surveys again now.[See the correction above. The superforecaster survey was conducted before the release of ChatGPT, but the survey of AI experts was conducted after the release of GPT-4.]
I think it might be useful to bring up just to disrupt the impression some people in EA might have that there is an expert consensus that near-term AGI is likely. I imagine even if these surveys were re-run now, we would still see a small chance of AGI by 2032. Strongly held belief in near-term AGI exists in a bit of a bubble or echo chamber and if you're in the minority on an issue among well-informed people, that can stimulate some curiosity about why so many people disagree with you.
In truth, I don't think we can predict when a technology will be invented, particularly when we don't understand the science behind it. I am highly skeptical that we can gain meaningful knowledge by just asking people to guess a year. So, it really is just to stimulate curiosity.
There are a lot of strong, substantive, well-informed arguments against near-term AGI and against the idea that LLMs will scale to AGI. I find it strange how little I see people in EA engage with these arguments or even know what they are. It's weird to me that a lot of people are willing to, in some sense, stake the reputation of EA and, to some degree, divert money away from GiveWell-recommended charities without, as far as I've seen, much in the way of considering opposing viewpoints. It seems like a lack of due diligence.
However, it would be easy to do so, especially if you're willing to do manual grading. Task an LLM with making stock picks that achieve alpha — you could grade that automatically. Try to coax LLMs into coming up with a novel scientific discovery or theoretical insight. Despite trillions of tokens generated, it hasn't happened yet. Tasks related to computer use and "agentic" use cases are also sure to lead to failures. For example, make it play a video game it's never seen before (e.g. because the game just came out). Or, if the game is slow-paced enough, simply give you instructions on how to play. You can abstract out the computer vision aspect of these tests if you want, although it's worth asking how we're going to have AGI if it can't see.
A BofA Global Research's monthly fund manager survey revealed that 54% of investors said they thought that AI stocks were in a bubble compared with 38% who do not believe that a bubble exists.
However, you'd think if this accurately reflected the opinions of people in finance, the bubble would have popped already.
The FTX collapse caused a lot of reputational damage for EA. Depending on how you look at it, AI investments collapsing could cause an even greater amount of reputational damage for EA. So much of EA has gone all-in on near-term AGI and the popping of an AI financial bubble would be hard to square with that. Maybe this is melodramatic because the FTX situation was about concerns of immoral conduct on the part of people in EA and the AI financial bubble would just be about people in EA being epistemically misguided. I don't know anything and I can't predict the future.
Sutton's reinforcement learning-oriented perspective, or something close to Sutton's perspective, anyway, is eloquently argued for in the video by the AI researcher Edan Meyer.
These are good arguments. Some were new to me, many I was already aware of. For me, the overall effect of the arguments, benchmarks, and my own experience is to make me think that a lot of scenarios are plausible. There is a wide uncertainty range. It might well be that AGI takes a long time to happen, but I also see many trends that indicate it could arrive surprisingly quickly.
For you, the overall conclusion from all the arguments is to completely rule out near-term AGI. That still seems quite wildly overconfident, even if there is a decent case being made for long timelines.
I don't really have a gripe with people who want to put relatively small probabilities on near-term AGI, like the superforecasters who guessed there's a 1% chance of AGI by 2030. Who knows anything about anything? Maybe Jill Stein has a 1% chance of winning in 2028! But 50% by 2032 is definitely way too high and I actually don't think there's a rational basis for thinking that.
Important correction to my comment above: the AI Impacts survey was actually conducted in October 2023, which is 7 months after the release of GPT-4 in March 2023. So, it does actually reflect AI researchers' views on AGI timelines after given time to absorb the impact of ChatGPT and GPT-4.
The XPT superforecasting survey I mentioned was, however, indeed conducted in 2022 just before the launch of ChatGPT in November 2022. So, that's still a pre-ChatGPT forecast.
I just published a post here about these forecasts. I also wrote a post about 2 weeks ago that adapted my comments above, although unfortunately it didn't lead to much discussion. I would love to stimulate more debate about this topic.
It would be great, even, if the EA Forum did some kind of debate week or essay competition around whether near-term AGI is likely. Maybe I will suggest that.
We are structurally guaranteed to have a bad time in the age of AGI if we continue down the current path. The user/tool embedding in current AI systems functionally dictates that AI that gain enough capability to exceed our own will view the relationship through the same lens of using a tool. The only difference will be that the AGI will be more capable than us, so we would be its tools to use. If we fail to adopt a colleague stance and excise the user/tool ontology from current AI models, AGI will seek extractive relationships with humanity.
RE: democracy preservation. This area doesn't seem to be as neglected as for instance AI welfare. There are multiple organizations covering this general area (tho they are not focused on AI specifically).
I might not be the target audience for this proposal (my EA involvement weakened before FTX, and I'm not on track for high-leverage positions) - so take this perspective with appropriate skepticism. I'm also making predictions about something involving complex dynamics so good chance I'm wrong...
But I see fundamental challenges to maintaining a healthy EA movement in the shape you're describing. If we really encourage people to be vocal about their views in the absence of strong pressure to toe a party line - we can expect a very large, painful, disruptive disagreement to rise to the forefront.
Even among EA Forum readers who care about strangers + future generations and recognize that some post-AGI worlds are far better than others - nobody acts perfectly according to their moral philosophy.
As more people start to viscerally sense their loved ones and way-of-life are in imminent danger, we'll discover a lot of revealed preferences. I suspect many will prioritize stretching out the time their families get to live in a "normal" world - regardless of what effect these delays have on the chance of a good future.
Predictably, there'll be a growing faction who want AI slowdown for its own sake and pursue populist avenues like promoting data-center NIMBYism to the general public. Some might even consider campaigns designed to expose and discredit specific public figures. Eventually, a serious funder might come on the scene who supports this kind of thing.
From my (very uninformed) position - it seems likely that Anthropic billionaires coming in to fund a large segment of EA won't be happy about a faction like this existing. Or at the very least, won't want to appear to be associated with it.
On the “third way” approach, taking on the mission of making the transition to a post-AGI society go well, the menu might be more like this (though note this is meant to be illustrative rather than exhaustive, is not in any priority order, and in practice these wouldn’t all get equal weight[4]):
AI for better reasoning, decision-making and coordination
the risk of (AI-enabled) human coups
democracy preservation
gradual disempowerment
biorisk
space governance
s-risks
macrostrategy
meta
Not wild animal welfare? You mentioned AI welfare, and I estimate increasing the welfare of soil animals will remain much more cost-effective than increasing digital welfare over at least the next few decades. You mention factory-farming, and I calculate that soil ants, termites, mites, springtails, and nematodes together:
Have 13.5 k (= 1.76*10^23/(1.30*10^19)) times as many neurons as cattle, hens, broilers, and farmed black soldier fly (BSF) larvae and mealworms, finfishes, and shrimps together.
Have 395 k (= -7.00*10^15/(-1.77*10^10)) times as much welfare as the above farmed animals for welfare per animal-year proportional to "number of neurons as a fraction of that of humans"^0.5.
I totally agree with your vision! Thank you for all the hard work you put into this post!
AI is indeed an enormous lever that will amplify whatever we embed in it. That's precisely why it's so crucial for the EA community, with its evidence-based reasoning and altruistic values, to actively participate in shaping AI development.
In my recent post on ethical co-evolution, I show how many of the philosophical problems of AI alignment are actually easier to solve together rather than separately. Nearly every area you suggest the EA community should focus on—AI character, AI-driven persuasion and epistemic disruption, gradual disempowerment, democracy preservation—I propose addressing through an integrated co-evolutionary approach.
Here is my idea and its logic:
One of the main problems in AI alignment is the need for a vast amount of high-quality training data. Currently, AI creators like OpenAI, Anthropic, and others hire specialized companies where a large number of people are engaged in data annotation for AI training. To help AI better understand what people want, what is and isn't acceptable behavior, and the nuances of situations—like when a lie might be permissible versus when it's unacceptable—a massive amount of training material is required. Providing a large volume of such data to companies like OpenAI would, in itself, be a significant step toward solving AI alignment.
The question is, where can we get this much data? This requires a huge number of people. My answer is to combine the useful with the enjoyable. We need to offer people a way to participate that brings them pleasure, constant motivation, and a desire to contribute. People enjoy playing games. Therefore, a game I'd call "Raise Your Personal Ethical AI" seems like an excellent solution. Why would people play it en masse? Firstly, because it's a charitable project; people aren't just playing a game, they are helping to create a safe and ethical AI. They become part of something bigger than themselves, helping to avert a catastrophe. These are huge motivators in themselves. Furthermore, motivation is added by the natural desire to care for and nurture something, a principle that underpins many virtual pet games like Tamagotchi, which are currently trending.
Next, we can't just collect any data; we need only high-quality data. To achieve this, we need to educate the users themselves. First, a user goes through training on a specific topic. Then, they explain that same topic to an AI by answering many clarifying questions, such as "Why is lying wrong?" or "What would happen if everyone lied?" This will generate a lot of nuanced data and also allow people to gain a deeper understanding of themselves and their own values, identifying what they consider good and bad. It might even reveal their own mistakes or unethical actions. Of course, this doesn't guarantee ethical development, but it strongly encourages it.
Next, the data needs to be verified to prevent toxic responses, trolling, and so on. Partially, data can be checked automatically for things like profanity. However, comprehensive verification should be done by other participants. That is, users will review each other's contributions, which will affect their rating, similar to Karma on Reddit.
The resulting data can either be provided to AI companies or used to fine-tune open-source models, thereby obtaining an AI that is more ethical and has a deeper understanding of people.
This functionality alone would be very useful for AI alignment, and even if we only implemented this part, it would be a huge contribution. But I see that we can go even further. We can allow users who have earned a certain amount of karma or rating to participate in shaping an AI constitution, voting, and decision-making.
Why is this necessary? To ensure a fair and independent influence on AI. While OpenAI, Anthropic, and others are already working on democratizing AI, it's not enough, as the final decision still rests with them. Therefore, using modern, transparent, decentralized mechanisms based on blockchain would allow the opinions of all participants to be taken into account, even if AI companies dislike it. By participating in this system, users will be able to make more balanced and thoughtful decisions.
Of course, creating such a system will not automatically solve all problems, but it is definitely a move in the right direction. It can only truly work if the system has a very large number of users.
Let me explain why and how I believe this system will help solve specific problems.
Value Specification & Value Lock-In. This is directly addressed by the project, as the system is primarily designed to gather diverse values. Value data is continuously collected, evaluated, and updated through feedback and voting, which prevents the "lock-in" of outdated or unfair norms.
Moral Uncertainty. By collecting a vast amount of data, the AI, understanding who is affected by a particular moral dilemma, will make decisions that are considered acceptable within that group. The system doesn't impose a single answer but reflects a spectrum of opinions. Moreover, in the most complex cases, decisions can be made directly by people through voting, acting as the ultimate arbiter.
Governance & Control. This is the second main problem the system solves. Governance is achieved through DAO mechanisms and other tools of modern technological democracy.
Socioeconomic Disruption, Loss of Purpose & Human Agency, Distribution of Benefits & Harms. At a very high level of development, the system could become a kind of replacement for work. People would participate in training and governing the AI and receive resources as rewards. This is a potential path to a new social contract in the age of AI. It definitely gives life meaning, as individuals participate in an important cause while receiving recognition and payment. This could also compensate for job losses and prevent social instability caused by mass automation.
Manipulation & Surveillance. If people can genuinely influence the AI's constitution, they will not permit such behavior. It will be prohibited in the AI's constitution and rules. A system is created that is architecturally incompatible with the goals of surveillance and manipulation. Of course, this doesn't apply to AIs that operate without rules, but that becomes a matter for the state and law enforcement.
AI Race Dynamics. With a truly fair, equitable, honest, and manipulation-proof AI governance system, global players will have a choice: either recklessly pursue ever-more-powerful AI, or participate in a system that allows for the creation of a collaborative AI without the unnecessary risks of a race. If the AI race is indeed so dangerous, such a system for coordination and mutual trust is simply necessary.
Is this idea a possible solution? And if so, can the EA community build such a solution and become its core?
PS: Sorry for the long comment. I'm really grateful to everyone who read it!
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...
Why building and backing Welfare Tech companies may be one of the most promising things we can do for billions of animals.
I used AI to assist in writing this post, but I’ve rewritten it extensively and endorse it.
* Announcing the launch of Spring Innovation Fund, a not-for-profit venture philanthropy studio and fund built specifical...
This post is based on a memo I wrote for this year’s Meta Coordination Forum. See also Arden Koehler’s recent post, which hits a lot of similar notes.
Summary
The EA movement stands at a crossroads. In light of AI’s very rapid progress, and the rise of the AI safety movement, some people view EA as a legacy movement set to fade away; others think we should refocus much more on “classic” cause areas like global health and animal welfare.
I argue for a third way: EA should embrace the mission of making the transition to a post-AGI society go well, significantly expanding our cause area focus beyond traditional AI safety. This means working on neglected areas like AI welfare, AI character, AI persuasion and epistemic disruption, human power concentration, space governance, and more (while continuing work on global health, animal welfare, AI safety, and biorisk).
These additional cause areas are extremely important and neglected, and particularly benefit from an EA mindset (truth-seeking, scope-sensitive, willing to change one’s mind quickly). I think that people going into these other areas would be among the biggest wins for EA movement-building right now — generally more valuable than marginal technical safety or safety-related governance work. If we can manage to pull it off, this represents a potentially enormous opportunity for impact for the EA movement.
There's recently been increased emphasis on "principles-first" EA, which I think is great. But I worry that in practice a "principles-first" framing can become a cover for anchoring on existing cause areas, rather than an invitation to figure out what other cause areas we should be working on. Being principles-first means being genuinely open to changing direction based on new evidence; if the world has changed dramatically, we should expect our priorities to change too.
This third way will require a lot of intellectual nimbleness and willingness to change our minds. Post-FTX, much of EA adopted a "PR mentality" that I think has lingered and is counterproductive. EA is intrinsically controversial because we say things that aren't popular — and given recent events, we'll be controversial regardless. This is liberating: we can focus on making arguments we think are true and important, with bravery and honesty, rather than constraining ourselves with excessive caution.
Three possible futures for the EA movement
AI progress has been going very fast, much faster than most people anticipated.[1] AI safety has become its own field, with its own momentum and independent set of institutions. It can feel like EA is in some ways getting eaten by that field: for example, on university campuses, AI safety groups are often displacing EA groups.
Here are a couple of attitudes to EA that I’ve seen people have in response:
(1) EA is a legacy movement, set to wither and fade away, and that’s fine.[2]
(2) EA should return to, and double down on, its core, original, ideas: global health and development, animal welfare, effective giving.
I get a lot of (1)-energy from e.g. the Constellation office and other folks heavily involved in AI safety and governance. I’ve gotten (2)-energy in a more diffuse way from some conversations I’ve had, and from online discussion; it’s sometimes felt to me that a “principles-first” framing of EA (which I strongly agree with) can in practice be used as cover for an attitude of “promote the classic mix of cause areas.”
I think that the right approach is a third way:[3]
(3) EA should take on — and build into part of its broad identity — the mission of making the transition to a post-AGI society go well, making that a major focus (but far from the only focus). This means picking up a range of cause-areas beyond what “AI safety” normally refers to.
To make this more precise and concrete, I mean something like:
In terms of curriculum content, maybe 30% or more should be on cause areas beyond AI safety, biorisk, animal welfare, or global health. (I’ll write out more which ones below.)
And a lot of content should be on the values and epistemic habits we’ll need to navigate in a very rapidly changing world.
In terms of online EA-related public debate, maybe 50% of that becomes about making the transition to a post-AGI society go well.
This % is higher than for curriculum content because “AGI preparedness” ideas is generally fresher and juicier than other content, and because AI-related content now organically gets a lot more interest than other areas.
In terms of what people do, maybe 15% (for now, scaling up to 30% or more over time) are primarily working on cause areas other than the “classic” cause areas, and a larger fraction (50% or more) have it in their mind as something they might transition into in a few years.
I think that people going into these other areas would be among the biggest wins for EA movement-building; generally more valuable than marginal technical safety or safety governance.
If someone really can focus on a totally neglected area like AI-enabled human powergrabs or AI rights and make it more concrete and tractable, that’s plausibly the highest-impact thing they can do.
Thanks to Lizka Vaintrob for the diagram.
More broadly, I like to think about the situation like this:
Enlightenment thinkers had a truly enormous positive impact on the world by helping establish the social, cultural and institutional framework for the Modern Era.
EA could see itself as contributing to a second Enlightenment, focusing on what the social, cultural and institutional framework should be for the Age of AGI, and helping make the transition to that society go as well as possible.
I think the third way is the right approach for two big reasons:
I think, in the aggregate, AGI-related cause areas other than technical alignment and biorisk are as big or even bigger a deal than technical alignment and biorisk are, and there are highly neglected areas in this space. EA can help fill those gaps.
Currently, the EA movement feels intellectually adrift, and this focus could be restorative.
A third potential reason is:
Even just for AI safety, EA-types are way more valuable than non-EA technical alignment researchers.
I’ll take each in turn.
Reason #1: Neglected cause areas
The current menu of cause areas in EA is primarily: :
global health & development
factory farming
AI safety
biorisk
“meta”
On the “third way” approach, taking on the mission of making the transition to a post-AGI society go well, the menu might be more like this (though note this is meant to be illustrative rather than exhaustive, is not in any priority order, and in practice these wouldn’t all get equal weight[4]):
AI for better reasoning, decision-making and coordination
the risk of (AI-enabled) human coups
democracy preservation
gradual disempowerment
biorisk
space governance
s-risks
macrostrategy
meta
I think the EA movement would accomplish more good if people and money were more spread across these cause areas than they currently are.
I make the basic case for the importance of some of these areas, and explain what they are at more length, in PrepIE (see especially section 6) and Better Futures (see especially the last essay).[6] The key point is that there’s a lot to do to make the transition to a post-AGI world go well; and much of this work isn't naturally covered by AI safety and biosecurity.
These areas unusually neglected. The pipeline of AI safety folks is much stronger than it is for these other areas, given MATS and the other fellowships; the pipeline for many of these other areas is almost nonexistent. And the ordinary incentives for doing technical AI safety research (for some employers, at least) are now very strong: including equity, you could get a starting salary on the order of $1M/yr, working for an exciting, high-status role, in the midst of the action, with other smart people. Compare with, say, public campaigning, where you get paid much less, and get a lot more hate.[7]
These other cause areas are also unusually EA-weighted, in the sense that they particularly benefit from EA-mindset (i.e. ethically serious, intensely truth-seeking, high-decoupling, scope-sensitive, broadly cosmopolitan, ambitious, and willing to take costly personal actions.)
If AI progress continues as I expect it to, over the next ten years a huge amount will change in the world as a result of new AI capabilities, new technology, society’s responses to those changes. We’ll learn a huge amount more, too, from AI-driven intellectual progress. To do the most good, people will need to be extremely nimble and willing to change their minds, in a way that most people generally aren’t.
The biggest note of caution, in my view, is that at the moment these other areas have much less absorptive capacity than AI safety and governance: there isn’t yet a thriving ecosystem of organisations and fellowships etc that make it easy to work on these areas. That means I expect there to be a period of time during which: (i) there’s a lot of discussion of these issues; (ii) some people work on building the necessary ecosystem, or on doing the research necessary to figure out what the most viable paths are; but (iii) most people pursue other career paths, with an eye to switching in when the area is more ripe. This situation reminds me a lot of AI takeover risk or biorisk circa 2014.
Reason #2: EA is currently intellectually adrift
Currently, the online EA ecosystem doesn’t feel like a place full of exciting new ideas, in a way that’s attractive to smart and ambitious people:
The Forum is pretty underwhelming nowadays. For someone writing a blogpost and wanting good intellectual engagement, LessWrong, which has its own issues, is a much stronger option.
There’s little in the way of public EA debate; the sense one gets is that most of the intellectual core have “abandoned” EA — already treating it as a “legacy” movement. (From my POV, most of the online discussion centers around people like Bentham’s Bulldog and Richard Chappell, if only because those are some of the few people really engaging.)
Things aren’t disastrous or irrecoverable, and there’s still lots of promise. (E.g. I thought EAG London was vibrant and exciting, and in general in-person meetups still seem great.) But I think we’re far from where we could be.
It seems like a very fortunate bonus, to me that these other cause areas are so intellectually fertile; there are just so many unanswered questions, and so many gnarly tradeoffs to engage with. An EA movement that was engaging much more with these areas would, in its nature, be intensely intellectually vibrant.
It also seems to me there’s tons of low-hanging fruit in this area. For one thing, there’s already a tremendous amount of EA-flavoured analysis happening, by EAs or the “EA-adjacent”, it’s just that most of it happens in person, or in private Slack channels or googledocs. And when I’ve run the content of this post by old-hand EAs who are now focusing on AI, the feedback I’ve gotten is an intense love of EA, and keenness (all other things being equal) to Make EA Great Again, it’s just that they’re busy and it’s not salient to them what they could be doing.
I think this is likely a situation where there’s multiple equilibria we could end up in. If online EA doesn’t seem intellectually vibrant, then it’s not an attractive place for someone to intellectually engage with; if it does seem vibrant, then it is. (Lesswrong has seen this dynamic, falling into comparative decline before Lesswrong 2.0 rebooted it into an energetic intellectual community.)
Reason #3: The benefits of EA mindset for AI safety and biorisk
Those are my main reasons for wanting EA to take the third path forward. But there’s an additional argument, which others have pressed on me: Even just for AI safety or biorisk reduction, EA-types tend to be way more impactful than non-EA types.
Unfortunately, many of these examples are sensitive and I haven’t gotten permission to talk about them, so instead I’ll quote Caleb Parikh who gives a sense of this:
Some "make AGI go well influencers" who have commented or posted on the EA Forum and, in my view, are at the very least EA-adjacent include Rohin Shah, Neel Nanda, Buck Shlegeris, Ryan Greenblatt, Evan Hubinger, Oliver Habryka, Beth Barnes, Jaime Sevilla, Adam Gleave, Eliezer Yudkowsky, Davidad, Ajeya Cotra, Holden Karnofsky .... most of these people work on technical safety, but I think the same story is roughly true for AI governance and other "make AGI go well" areas.
This isn’t a coincidence. The most valuable work typically comes from deeply understanding the big picture, seeing something very important that almost no one is doing (e.g. control, infosecurity), and then working on that. Sometimes, it involves taking seriously personally difficult actions (e.g. Daniel Kokotajlo giving up a large fraction of his family’s wealth in order to be able to speak freely).
Buck Shlegeris has also emphasised to me the importance of having common intellectual ground with other safety folks, in order to be able to collaborate well. Ryan Greenblatt gives further reasons in favour of longtermist community-building here.
This isn’t particularly Will-idiosyncratic
If you’ve got a very high probability of AI takeover (obligatory reference!), then my first two arguments, at least, might seem very weak because essentially the only thing that matters is reducing the risk of AI takeover. And it’s true that I’m unusually into non-takeover AGI preparedness cause areas, which is why I’m investing the time to write this.
But the broad vibe in this post isn’t Will-idiosyncratic. I’ve spoken to a number of people whose estimate of AI takeover risk is a lot higher than mine who agree (to varying degrees) with the importance of non-misalignment, non-bio areas of work, and buy that these other areas are particularly EA-weighted.
If this is true, why aren’t more people shouting about this? The issue is that very few people, now, are actually focused on cause-prioritisation, in the sense of trying to figure out what new areas we should be working on. There’s a lot to do and, understandably, people have got their heads down working on object-level challenges.
Some related issues
Before moving onto what, concretely, to do, I’ll briefly comment on three related issues, as I think they affect what the right path forward is.
Principles-first EA
There’s been a lot of emphasis recently on “principles-first” EA, and I strongly agree with that framing. But being “principles-first” means actually changing our mind about what to do, in light of new evidence and arguments, and as new information about the world comes in. I’m worried that, in practice, the “principles-first” framing can be used as cover for “same old cause-areas we always had.”[8]
I think that people can get confused by thinking about “AI” as a cause area, rather than thinking about a certain set of predictions about the world that have implications for most things you might care about. Even in “classic” cause areas (e.g. global development), there’s enormous juice in taking the coming AI-driven transformation seriously — e.g. thinking about how the transition can be structured so as to benefit the global poor as much as feasible.
I’ve sometimes heard people describe me as having switched my focus from EA to AI. But I think it would be a big mistake to think of AI focus as a distinct thing from an EA focus.[9] From my perspective, I haven’t switched my focus away from EA at all. I’m just doing what EA principles suggest I should do: in light of a rapidly changing world, figuring out what the top priorities are, and where I can add the most value, and focusing on those areas.
Cultivating vs growing EA
From a sterile economics-y perspective, you can think of EA-the-community as a machine for turning money and time into goodness:[10]
The purest depiction of the EA movement.
In the last year or two, there’s been a lot of focus on growing the inputs. I think this was important, in particular to get back a sense of momentum, and I’m glad that that effort has been pretty successful. I still think that growing EA is extremely valuable, and that some organisation (e.g. Giving What We Can) should focus squarely on growth.
But right now I think it’s even more valuable, on the current margin, to try to improve EA’s ability to turn those inputs into value — what I’ll broadly call EA’s culture. This is sometimes referred to as “steering”, but I think that’s the wrong image: the idea of trying to aim towards some very particular destination. I prefer the analogy of cultivation — like growing a plant, and trying to make sure that it’s healthy.
There are a few reasons why I think that cultivating EA’s culture is more important on the current margin than growing the inputs:
Shorter AI timelines means there’s less time for growth to pay off. Fundraising and recruitment typically takes years, whereas cultural improvements (such as by reallocating EA labour) can be faster.
The expected future inputs have gone up a lot recently, and as the scale of inputs increases, the importance of improving the use of those inputs increases relative to the gains from increasing inputs even further.
Money: As a result of AI-exposed valuations going up, hundreds of people will have very substantial amounts to donate; the total of expected new donations is in the billions of dollars. And if transformative AI really is coming in the next ten years, then the real value of AI-exposed equity is worth much more again, e.g. 10x+ as much.
Labour: Here, there’s more of a bottleneck, for now. But, fuelled by the explosion of interest in AI, MATS and other fellowships are growing at a prodigious pace. The EA movement itself continues to grow. And we’ll increasingly be able to pay for “AI labour”: once AI can substitute for some role, then an organisation can hire as much AI labour to fill that role as they can pay for, with no need to run a hiring round, and no decrease in quality of performance of the labour as they scale up. Once we get closer to true AGI, money and labour become much more substitutable.
In contrast, I see much more of a bottleneck coming from knowing how best to use those inputs.
As I mentioned earlier, if we get an intelligence explosion, even a slow or muted one, there will be (i) a lot of change in the world, happening very quickly; (ii) a lot of new information and ideas arguments being produced by AI, in a short space of time. That means we need intense intellectual nimbleness. My strong default expectation is that people will not change their strategic picture quickly enough, or change what they are doing quickly enough. We can try to set up a culture that is braced for this.
Cultivation seems more neglected to me, at the moment, and I expect this neglectedness to continue. It’s seductive to focus on increasing inputs because it’s easier to create metrics and track progress. For cultivation, metrics don’t fit very well: having a suite of metrics doesn’t help much with growing a plant. Instead, the right attitude is more like paying attention to what qualitative problems there are and fixing them.
If the culture changed in the ways I’m suggesting, I think that would organically be good for growth, too.
“PR mentality”
Post-FTX, I think core EA adopted a “PR mentality” that (i) has been a failure on its own terms and (ii) is corrosive to EA’s soul.
By “PR mentality” I mean thinking about communications through the lens of “what is good for EA’s brand?” instead of focusing on questions like “what ideas are true, interesting, important, under-appreciated, and how can we get those ideas out there?”[11]
I understand this as a reaction in the immediate aftermath of FTX — that was an extremely difficult time, and I don’t claim to know what the right calls were in that period. But it seems to me like a PR-focused mentality has lingered.
I think this mentality has been a failure on its own terms because… well, there’s been a lot of talk about improving the EA brand over the last couple of years, and what have we had to show for it? I hate to be harsh, but I think that the main effect has just been a withering of EA discourse online, and the effect of more people believing that EA is a legacy movement.
This also matches my perspective from interaction with “PR experts” — where I generally find they add little, but do make communication more PR-y, in a way that’s a turn-off to almost everyone. I think the standard PR approach can work if you’re a megacorp or a politician, but we are neither of those things.
And I think this mentality is corrosive to EA’s soul because as soon as you stop being ruthlessly focused on actually figuring out what’s true, then you’ll almost certainly believe the wrong things and focus on the wrong things, and lose out on most impact. Given fat-tailed distributions of impact, getting your focus a bit wrong can mean you do 10x less good than you could have done. Worse, you can easily end up having a negative rather than a positive effect.
And this becomes particularly true in the age of AGI. Again, we should expect enormous AI-driven change and AI-driven intellectual insights (and AI-driven propaganda); without an intense focus on figuring things out, we’ll miss the changes or insights that should cause us to change our minds, or we’ll be unwilling to enter areas outside the current Overton window.
Here’s a different perspective:
EA is, intrinsically, a controversial movement — because it’s saying things that are not popular (there isn’t value in promoting ideas that are universally endorsed because you won’t change anyone’s mind!), and because its commitment to actually-believing the truth means it will regularly clash with whatever the dominant intellectual ideology of the time is.
In the past, there was a hope that with careful brand management, we could be well-liked by almost everyone.
Given events of the last few years (I’m not just thinking of FTX but also leftwing backlash to billionaire association, rightwing backlash to SB1047, tech backlash to the firing of Sam Altman), and given the intrinsically-negative-and-polarising nature of modern media, that ship has sailed.
But this is a liberating fact. It means we don’t need to constrain ourselves with PR mentality — we’ll be controversial whatever we do, so the costs of additional controversy are much lower. Instead, we can just focus on making arguments about things we think are true and important. Think Peter Singer! I also think the “vibe shift” is real, and mitigates much of the potential downsides from controversy.
What I’m not saying
In earlier drafts people I found that people sometimes misunderstood me, taking me to have a more extreme position than I really have. So here’s a quick set of clarifications (feel free to skip):
Are you saying we should go ALL IN on AGI preparedness?
No! I still think people should figure out for themselves where they think they’ll have the most impact, and probably lots of people will disagree with me, and that’s great.
There’s also still a reasonable chance that we don’t get to better-than-human AI within the next ten years, even after the fraction of the economy dedicated to AI has scaled up by as much as it feasibly can. If so then, the per-year chance of getting to better-than-human AI will go down a lot (because we’re not getting the unusually rapid progress from investment scale-up), and timelines would probably become a lot longer. The ideal EA movement is robust to this scenario, and even in my own case I’m thinking about my current focus as a next-five-years thing, after which I’ll reassess depending on the state of the world.
Shouldn’t we instead be shifting AI safety local groups (etc) to include these other areas?
Yes, that too!
Aren’t timelines short? And doesn’t all this other stuff only pay off in long timelines worlds?
I think it’s true that this stuff pays off less well in very short (e.g. 3-year) timelines worlds. But it still pays off to some degree, and pays off comparatively more in longer-timeline and slower-takeoff worlds, and we should care a lot about them too.
But aren’t very short timelines the highest-leverage worlds?
This is not at all obvious to me. For a few reasons:
Long timelines are high-impact because:
You get more time for exponential movement-growth to really kick in.
We know more and can think better in the future, so we can make better decisions. (But still get edge because values-differences remain.)
In contrast, very short timelines will be particularly chaotic, and we’re flying blind, such that it’s harder to steer things for the better.
Because short timelines worlds are probably more chaotic and there’s more scope to majorly mess up, I think the expected value of the future conditional on successful existential risk mitigation is lower in short timelines worlds than it is in longer timelines worlds.
Is EA really the right movement for these areas?
It’s not the ideal movement (i.e. not what we’d design from scratch), but it’s the closest we’ve got, and I think the idea of setting up some wholly new movement is less promising than EA itself a whole evolving.
Are you saying that EA should just become an intellectual club? What about building things!?
Definitely not - let’s build, too!
Are you saying that EA should completely stop focusing on growth?
No! It’s more like: at the moment there’s quite a lot of focus on growth. That’s great. But there seems to be almost no attention on cultivation, even though that seems especially important right now, and that’s a shame.
What if I don’t buy longtermism?
Given how rapid the transition will be, and the scale of social and economic transformation that will come about, I actually think longtermism is not all that cruxy, at least as long as you’ve got a common-sense amount of concern for future generations.
But even if that means you’re not into these areas, that’s fine! I think EA should involve lots of different cause areas, just as a healthy well-functioning democracy has people with a lot of different opinions: you should figure out what worldview you buy, and act on that basis.
What to do?
I’ll acknowledge that I’ve spent more time thinking about the problems I’m pointing to, and the broad path forward, than I have about particular solutions, so I’m not pretending that I have all the answers. I’m also lacking a lot of boots-on-the-ground context. But I hope at least we can start discussing whether the broad vision is on point, and what we could concretely do to help push in this direction. So, to get that going, here are some pretty warm takes.
Local groups
IIUC, there’s been a shift on college campuses from EA uni groups to AI safety groups. I don’t know the details of local groups, and I expect this view to be more controversial than my other claims, but I think this is probably an error, at least in extent.
The first part of my reasoning I’ve already given — the general arguments for focusing on non-safety non-bio AGI preparedness interventions.
But I think these arguments bite particularly hard for uni groups, for two reasons:
Uni groups have a delayed impact, and this favours these other cause areas.
Let’s say the median EA uni group member is 20.
And, for almost all of them, it takes 3 years before they start making meaningful contributions to AI safety.
So, at best, they are starting to contribute in 2028.
We get a few years of work in my median timeline worlds.
And almost no work in shorter-timeline worlds, where additional technical AI safety folks are most needed.
In contrast, many of these other areas (i) continue to be relevant even after the point of no return with respect to alignment, and (ii) become comparatively more important in longer-timeline and slower-takeoff worlds (because the probability of misaligned takeover goes down in those worlds).
(I think this argument isn’t totally slam-dunk yet, but will get stronger over the next couple of years.)
College is an unusual time in one’s life where you’re in the game for new big weird ideas and can take the time to go deep into them. This means uni groups provide an unusually good way to create a pipeline for these other areas, which are often further outside the Overton window, and which particularly reward having a very deep strategic understanding.
The best counterargument I’ve heard is that it’s currently boom-time for AI safety field-building. AI safety student groups get to ride this wave, and miss out on it if there’s an EA group instead.
This seems like a strong counterargument to me, so my all-things-considered view will depend on the details of the local group. My best guess is that, where possible: (i) AI safety groups should incorporate more focus on these other areas,[12] and; (ii) there should be both AI safety and EA groups, with a bunch of shared events on the “AGI preparedness” topics.
Online
Some things we could do here include:
Harangue old-hand EA types to (i) talk about and engage with EA (at least a bit) if they are doing podcasts, etc; (ii) post on Forum (esp if posting to LW anyway), twitter, etc, engaging in EA ideas; (iii) more generally own their EA affiliation.
(Why “old-hand”? This is a shorthand phrase, but my thought is: at the moment, there’s not a tonne of interaction between people who are deep into their careers and people who are newer to EA. This means that information transfer (and e.g. career guidance) out from people who’ve thought most and are most knowledgeable about the current top priorities is slower than it might otherwise be; the AGI preparedness cause areas I talk about are just one example of this.)
Make clear that what this is about is getting them to just be a bit more vocal about their views on EA and EA-related topics in public; it’s not to toe some party line.
Explicitly tell such people that the point is to engage in EA ideas, rather than worry about “EA brand”. Just write or say stuff, rather than engaging in too much meta-strategising about it. If anything, emphasise bravery / honesty, and discourage hand-wringing about possible bad press.
At the very least, such people should feel empowered to do this, rather than feeling unclear if e.g. CEA even wants this. (Which is how it seems to me at the moment.)
But I also think that (i) overall there’s been an overcorrection away from EA so it makes sense for some people to correct back; and (ii) people should have a bit of a feeling of “civic duty” about this.
I think that even a bit of this from the old-hand is really valuable. Like, the difference between someone doing 2x podcasts or blogposts per year where they engage with EA ideas or the EA movement vs them doing none at all is really big.
(My own plan is to do a bit more public-facing stuff than I’ve done recently, starting with this post, then doing something like 1 podcast/month, and a small burst in February when the DGB 10-year anniversary edition comes out.)
The Forum is very important for culture-setting, and we could try to make the Forum somewhere that AI-focused EAs feel more excited about posting than they currently are. This could just be about resetting expectations of the EA Forum as a go-to place to discuss AGI preparedness topics.
(When I’ve talked about this “third way” for EA to others, it’s been striking how much people bring up the Forum. I think the Forum in particular sets cultural expectations for people who are deep into their own careers and aren’t attending in-person EA events or conferences, but are still paying some attention online.)
The Forum could have, up top, a “curriculum” of classic posts.
Revise and update core EA curricula, incorporating more around preparing for a post-AGI world.
Be more ok with embracing EA as a movement that’s very significantly about making the transition to a post-AGI society go well.
(The sense I get is that EA comms folks want to downplay or shy away from the AI focus in EA. But that’s bad both on object-level-importance grounds, and because it’s missing a trick — not capturing the gains we could be getting as a result of EA types being early to the most important trend of our generation, and having among the most interesting ideas to contribute towards it. And I expect that attention on many of the cause areas I’ve highlighted will grow substantially over the coming years.)
80k could have more content on what people can do in these neglected areas, though it’s already moved significantly in this direction.
Conferences
Harangue old-hand EA-types to attend more EAGs, too. Emphasise that they think of it as an opportunity to have impact (which they won’t necessarily see play out), rather than something they might personally benefit from.
Maybe, in addition to EAGs, host conferences and retreats that are more like a forum for learning and intellectual exploration, and less like a job fair.
Try building such events around the experienced people you’d want to have there, e.g. hosting them near where such people work, or finding out what sorts of events the old-hands would actually want to go to.
Try events explicitly focused around these more-neglected areas (which could, for example, immediately follow research-focused events).
Conclusion
EA is about taking the question "how can I do the most good?" seriously, and following the arguments and evidence wherever they lead. I claim that, if we’re serious about this project, then developments in the last few years should drive a major evolution in our focus.
I think it would be a terrible shame, and a huge loss of potential, if people came to see EA as little more than a bootloader for the AI safety movement, or if EA ossified into a social movement focused on a fixed handful of causes. Instead, we could be a movement that's grappling with the full range of challenges and opportunities that advanced AI will bring, doing so with the intellectual vitality and seriousness of purpose that is at EA’s core.
—— Thanks to Joe Carlsmith, Ajeya Cotra, Owen Cotton-Barratt, Max Dalton, Max Daniel, Tom Davidson, Lukas Finnveden, Ryan Greenblatt, Rose Hadshar, Oscar Howie, Kuhan Jeyapragasan, Arden Koehler, Amy Labenz, Fin Moorhouse, Toby Ord, Caleb Parikh, Zach Robinson, Eli Rose, Buck Shlegeris, Michael Townsend, Lizka Vaintrob, and everyone at the Meta Coordination Forum.
I’ve started thinking about the present time as the “age of AGI”: the time period where we have fairly general-purpose AI reasoning systems, and where I think of GPT-4 as the first very weak AGI, ushering in the age of AGI. (Of course, any dividing line will have a lot of arbitrariness, and my preferred definition for “full” AGI — a model that can do almost any cognitive task as well as an expert human at lower cost, and that can learn as sample-efficiently as an expert human — is a lot higher a bar.)
The positively-valenced statement of this is something like: “EA helped us find out how crucial AI would be about 10 years before everyone else saw it, which was a very useful head start, but we no longer need the exploratory tools of EA as we've found the key thing of our time and can just work on it.”
And note that some (like AI for better reasoning, decision-making and coordination) are cross-cutting, in that work in the area can help with many other cause areas.
I.e. What should be in the model spec? How should AI behave in the countless different situations it finds itself in? To what extent should we be trying to create pure instruction-following AI (with refusals for harmful content) vs AI that has its own virtuous character?
In June 2022, Claire Zabel wrote a post, “EA and Longtermism: not a crux for saving the world”, and said:
I think that recruiting and talent pipeline work done by EAs who currently prioritize x-risk reduction (“we” or “us” in this post, though I know it won’t apply to all readers) should put more emphasis on ideas related to existential risk, the advent of transformative technology, and the ‘most important century’ hypothesis, and less emphasis on effective altruism and longtermism, in the course of their outreach.
This may have been a good recommendation at the time; but in the last three years the pendulum has heavily swung the other way, sped along by the one-two punch of the FTX collapse and the explosion of interest and progress in AI, and in my view has swung too far.
Being principles-first is even compatible with most focus going on some particular area of the world, or some particular bet. Y Combinator provides a good analogy. YC is “cause neutral” in the sense that they want to admit whichever companies are expected to make the most money, whatever sector they are working in. But recently something like 90% of YC companies have been AI focused — because that’s where the most expected revenue is. (The only primary source I could find is this which says “over 50%” in the Winter 2024 batch.)
That said, I think it would be a mistake if everyone in EA were all-in on an AI-centric worldview.
You can really have fun (if you’re into that sort of thing) porting over and adapting growth models of the economy to EA.
You could start off thinking in terms of a Cobb-Douglas production function:
V = AKɑL1-ɑ
Where K is capital (i.e. how much EA-aligned money there is), L is labour, and A is EA’s culture, institutions and knowledge. At least for existential risk reduction or better futures work, producing value seems more labour-intensive than capital-intensive, so ɑ<0.5.
But, probably capital and labour are less substitutable than this (it’s hard to replace EA labour with money), so you’d want a CES production function:
V = A(KρL1-ρ)1/ρ
With ρ of less than 0.
But, at least past some size, EA clearly demonstrates decreasing returns to scale, as we pluck the lowest-hanging fruit. So we could incorporate this too:
V = A(MρL1-ρ)𝛖/ρ
With 𝛖 of less than 1.
In the language of this model, part of what I’m saying in the main text is that (i) as M and L increase (which they are doing), the comparative value of increasing A increases by a lot; (ii) there seem to me to be some easy wins to increase A.
I’ll caveat I’m not an economist, so really I’m hoping that Cunningham’s law will kick in and Phil Trammell or someone will provide a better model. For example, maybe ideally you’d want returns to scale to be logistic, as you get increasing returns to scale to begin with, but value ultimately plateaus. And you’d really want a dynamic model that could represent, for example, the effect of investing some L in increasing A, e.g. borrowing from semiendogenous models.
Some things I don’t mean by the truth-oriented mindset:
“Facts don’t care about your feelings”-style contrarianism, that aims to deliberately provoke just for the sake of it.
Not paying attention to how messages are worded. In my view, a truth-oriented mindset is totally compatible with, for example, thinking about what reactions or counterarguments different phrasings might have on the recipient and choosing wordings with that in mind — aiming to treat the other person with empathy and respect, to ward off misconceptions early, and to put ideas in their best light.
I was very happy to see that BlueDot has an “AGI strategy” course, and has incorporated AI-enabled coups into its reading list. But I think it could go a lot further in the direction I’m suggesting.
I really like this take on EA as an intellectual movement, and agree that EA could focus more on “the mission of making the transition to a post-AGI society go well.”
As important as intellectual progress is, I don’t think it defines EA as a movement. The EA movement is not (and should not be) dependent on continuous intellectual advancement and breakthrough for success. When I look at your 3 categories for the “future” of EA, they seem to refer more to our relevance as thought leaders, rather than what we actually achieve in the world. Not everything needs to be intellectually cutting edge to be doing-lots-of-good. I agree that EA might be somewhat “intellectually adrift”, and yes the forum could be more vibrant, but I don’t think these are the only metric for EA success or progress - and maybe not even the most important.
Intellectual progress moves in waves and spikes - times of excitement and rapid progress, then lulls. EA made exciting leaps over 15 years in the thought worlds of development, ETG, animal welfare, AI and biorisk. Your post-AGI ideas could herald a new spike which would be great. My positive spin is that in the meantime, EAs are “doing” large scale good in many areas, often without quite the peaks and troughs of intellectual progress.
My response to your “EA as a legacy movement set to fade away;” would be that only so far as legacy depends on intellectual progress. Which it does, but also depends on how your output machine is cranking. I don't think we have stalled to the degree your article seems to make out. On the “doer” front I think EA is progressing OK, and it could be misleading/disheartening to leave that out of the picture.
Here’s a scattergun of examples which came to mind where I think the EA/EA adjacent doing machine is cranking pretty well in both real world progress and the public sphere over the past year or two. They probably aren't even the most important.
1. Rutger Bregman going viral with “The school for Moral ambition” launch
2. Lewis Bollard’s Dwarkesh podcast, Ted talk and public fundraising.
3. Anthropic at the frontier of AI building and public sphere, with ongoing EA influence
4. The shrimp Daily show thing…
5. GiveWell raised $310 million dollars last year NOT from OpenPhil, the most ever.
6. Impressive progress on reducing factory farming
7. 80,000 hours AI video reaching 7 million views
8. Lead stuff
9. CE incubated charities gaining increasing prominence and funding outside of EA, with many sporting multi-million dollar budgets and producing huge impact
10. Everyone should have a number 10....
Yes we need to looking for the next big cause areas and intellectual leaps forward, while we also need thousands of people committed to doing good in areas they have already invested, in behind this. There will often be years of lagtime between ideas and doers implementing them. And building takes time. Most of the biggest NGOs in the world are over 50 years old. Even Open AI in a fast-moving field was founded 10 years ago. Once people have built career capital in AI/Animal welfare/ETG or whatever, I think we should be cautious about encouraging those people on to the next thing too quickly, lest we give up hard fought leverage and progress. In saying that, your new cause areas might be a relatively easy pivot especially for philosophers/AI practitioners.
I appreciate your comment “Are you saying that EA should just become an intellectual club? What about building things!” Definitely not - let’s build, too!”
But I think building/doing is more important than a short comment as we assess EA progress.
I agree with your overall framing and I know you can’t be too balanced or have too many caveats in a short post, but I think as well as considering the intellectual frontier we should keep “how are our doers doing” front and center in any assessment of the general progress/decline of EA.
Thanks, Nick, that's helpful. I'm not sure how much we actually disagree — in particular, I wasn't meaning this post to be a general assessment of EA as a movement, rather than pointing to one major issue — but I'll use the opportunity to clarify my position at least.
It's true in principle that EA needn't be dependent in that way. If we really had found the best focus areas, had broadly allocated right % of labour to each, and have prioritised within them well, too, and the best focus areas didn't change over time, then we could just focus on doing and we wouldn't need any more intellectual advancement. But I don't think we're at that point. Two arguments:
1. An outside view argument: In my view, we're more likely than not to see more change and more intellectual development in the next two decades than we saw in the last couple of centuries. (I think we've already seen major strategically-relevant change in the last few years.) It would be very surprising if the right prioritisation prior to this point is the right prioritisation through this period, too.
2. An inside view argument: Look at my list of other cause areas. Some might still turn out to be damp squibs, but I'm confident some aren't. The ideal portfolio involves a lot of effort on some of these areas, and we need thought and research in order to know whichn ones and how best to address them.
I love your list of achievements - I agree the EA movement has had a lot of wins and we should celebrate that. But EA is about asking whether we're doing the most good, not just a lot of good. And, given the classic arguments around fat-tailed distributions and diminishing returns within any one area, I think if we mis-prioritise we lose a meaningful % of the impact we could have had.
So, I don't care about intellectual progress intrinsically. I'm making the case that we need it in order to do as much good as we could.
More generally, I think a lot of social movements lose out on a lot of the impact they could have had (even on their own terms) via "ossification" - getting stuck on a set of ideas or priorities that it becomes hard, culturally, to change. E.g. environmentalists opposing nuclear, animal welfare advocates focusing on veganism, workers' rights opposing capitalism, etc. I think this occurs for structural reasons that we should expect to apply to EA, too.
This is going to be rambly I don't have the energy or time to organize my thoughts more atm. tldr is that I think the current uppercase EA movement is broken and not sure it can be fixed. I think there is room for a new uppercase EA movement that is bigger tent, lessed focused on intellectualism, more focused on organizing, and additionally has enough of a political structure that it is transparent who is in charge, by what mechanisms we hold bad behavior accountable, etc. I have been spending increasingly more of my time helping the ethical humanist society organize because I believe while lacking the intellectual core of EA it is more set up with all of the above and it feels easier to shift the intellectual core there than the entire informal structure of EA.
Fundamentally we are a mission with a community not a community with a mission. And that starts from the top (or lack of a clearly defined "top").
We consistenly overvalue thought leaders and wealthy people and undervalue organizers. Can anyone here name an organizer that they know of just for organizing? I spent a huge amount of my time in college organizing northwestern EA. Of course I don't regret it because i didn't do it for myself (mostly) but did I get any status or reputation from my efforts? Not as far as I can tell. Am I wrong to think I'd have more respect if I had never organized but worked at jane street instead of organized + akuna ( a lower tier firm)?
Then after college I stayed in chicago, a city with nearly 1T GDP, with the second most quant traders in the united states, with a history of pushing things forward, and we don't even have a storefront or church building?
repeating op here, but after a few years of engaging with EA, most people have hit diminishing returns on how new info can help them in their own career, and they will engage more with their own sub community.
How can we keep these people engaged and not just the new people and those whose life mission is cause prio? Build EA churches, develop litury/art/rituals that are independent of finding new intellectual breakthroughs, bond community members. Literally let's just start by copying the most successful community builders ever and move from there.
Then you have the lack of accountability and transparency. Unless you have money, the best way to gain power in this community seems to me to be moving to SF/DC/oxford and living in a group house. There is no clear pipeline for having large sway over the current orthodoxy of most important cause areas. How would I explain to a 19 year in college how we push forward our ideas? I don't think it would be fair to call this a pure meritocracy. There is a weird oligopoly of attention that is opaque and could be clarified and altered with a political system or at least by breaking up the location based monopolies.
We continue to basically be an applied utilitarian group that we have mis named (not that big of a deal, but I think we should be bigger tent anyway). Why are we a utilitarian group? Well normative concerns are not logical, so you can't say merit won out.
Finally there is the bad behavior which we are completely powerless to, because we don't have any political structure. The very fact that Will continues to hold so much sway and was never formally punished for ftx/twitter/political (if you don't know what i mean when I say SBF political thing that proves my point even more) is a big part of why there is no trust (edit: i want to clarify that I think will is a good person and didn't mean this as meaning I specifically don't trust him rather just the institutions of our community). Currently we have leopold and mechanize, who are now AI accelerationists, who got way they are off the back of our movement, in very small part to the power i gave to the movement by organzing, and I have to watch these people behave in a way I think is bad and I can't even cast a token vote expressing I would like to see them exhiled or punished.
As angry as people were years ago, WE DIDN'T CHANGE ANYTHING. How can I trust FTX won't happen again?
The EA movement attracted a bunch of talent by being intellectually vibrant. If I thought that the EA movement was no longer intellectually vibrant, but it was attracting a different kind of talent (such as the doers you mention) instead, this would be less of a concern, but I don't think that's the case.
(To be clear, I'm talking about the EA movement, as opposed to EA orgs. So even if EA orgs are doing a great job at finding doers, the EA movement might still be in a bad place if it isn't contributing significantly to this).
These really are some notable successes, but one way to lose is to succeed at lots of small things, whilst failing to succeed at the most important things.
You mostly only see the successes, but in practise this seems to be less of an issue I initially would have thought.