Thank you for doing this analysis!

Would you say this analysis is limited to safety from misalignment related risks, or any (potentially catastrophic) risks from AI, including misuse, gradual disempoerment, etc.?

Far-future effects are the most important determinant of what we ought to do

I agree it's insanely hard to know what will affect the far future, and how. But I think we should still try, often by using heuristics (one I'm currently fond of is "what kinds of actions seem to put us on a good trajectory, e.g. to be doing well in 100 years?")

I think that in cases where we do have reason to think an action will affect the long run future broadly and positively in expectation (i.e. even if we're uncertain) that's an extremely strong reason -- and usually an overriding one -- to favour it over one that looks worse for the long-run future. I think that's sufficient for agreement with the statement. 

I feel unsure I'd be trying hard to do good at all, let alone actually doing things I think have a lot of ex ante value. I wasn't on track when I heard of EA to dedicate much of my resources to positive impact. But hard to be certain ofc! + not sure doing good now, since what I work on has a lot of uncertainty on the impacts (& even sign). 

(trying to hit like 80% agree but seem to be missing it) 

Hey Matt,

1. I share several of the worries articulated in this post.
2. I think you're wrong about how you characterise 80k's strategic shift here, and want to try to correct the record on that point. I'm also going to give some concrete examples of things I'm currently doing, to illustrate a bit what I mean, & also include a few more personal comments.

(Context: I run the 80k web programme.)

if you glorify some relatively-value-neutral conception of AI safety as the summum bonum of what is or used to be EA, there is just a good chance that you will lose the plot and end up not pursuing the the actual highest good, the good itself.

Well put. And I agree that there are some concerning signs in this direction (though I've also had countervailing, inspiring experiences of AIS-focused people questioning whether some prevailing view about what to do in AIS is actually best for the world.)

I'd also love to see more cause prioritisation research. And it's gonna be hard to both stay open enough to changing our minds about how to make the world better & to pursue our chosen means with enough focus to be effective. I think this challenge is fairly central to EA. 

On 80k's strategic shift:

You wrote:

Perhaps the clearest and most predictive embodiment of the trend is 80,000 Hours’ new strategic focus on AI. 80k was always fundamentally about providing thorough, practical cause/intervention prioritization and that exercise can be fairly regarded as the core of EA. They’re now effectively saying the analysis is done: doing the most good means steering AI development, so we’ll now focus only on the particulars of what to do in AI. Thanks, we'll take it from here indeed.

How do we see the relationship between focusing on helping AGI go well and doing the most good?

It has always been the case that people and organisations need to find some intermediary outcome that comes before the good to point at strategically, some proxy for impact. Strategy is always about figuring out what's gonna be the biggest/most cost-effective causal factor for that (i.e. means), & therefore the best proxy to pursue.

We used to focus on career changes not necessarily inside one specific cause area but it was still a proxy for the good. Now our proxy for the good is helping people work on making AGI go well, but our relationship to the good is the same as it was before: trying our best to point at it, trying to figure out the best means for doing so.

EA values & ideas are still a really important part of the strategy.

We wrote this in our post on the shift:

As mentioned, we’re still using EA values (e.g. those listed here and here) to determine what to prioritise, including in making this strategic shift.

And we still think it’s important for people to use EA values and ideas as they’re thinking about and pursuing high-impact careers. Some particular examples which feel salient to us:

• Scope sensitivity and thinking on the margin seem important for having an impact in any area, including helping AGI go well.
• We think there are some roles / areas of work where it’s especially important to continually use EA-style ideas and be steadfastly pointed at having a positive impact in order for it to be good to work in the area. For example, in roles where it’s possible to do a large amount of accidental harm, like working at an AI company, or roles where you have a lot of influence in steering an organisation's direction.
• There are also a variety of areas where EA-style thinking about issues like moral patienthood, neglectedness, leverage, etc. are still incredibly useful – e.g. grand challenges humanity may face due to explosive progress from transformatively powerful AI.

Though one might understandably worry that was paying lip service, just to reassure people. Let me talk about some internal recent goings-on off the top of my head, which hopefully do something to show we mean it:

1. Our internal doc on web programme strategy (i.e. the strategy for the programme I run) currently says that in order for our audience to actually have much more impact with their careers, engagement with the site ideally causes movement along at least 3[1] dimensions:

A. Awareness of transformative AI potential 
B. EA-mindest (i.e. using ideas like impartiality, scope sensitivity, and thinking on the margin) 
C. Career position

This makes re-designing the user flow post-strategic-shift a difficult balancing act/full of tradeoffs. How do we both quickly introduce people to AI being a big deal & urgent, and communicate EA ideas, plus help people shift their careers? Which do we do first?

We're going to lose some simplicity (and some people, who don't want to hear it) trying to do all this, and it will be reflected in the site being more complex than a strategy like "maximize for engagement or respectability" or "maximize for getting one idea across effectively" would recommend.

My view is that it's worth it, because there is a danger of people just jumping into jobs that have "AI" or even "AI security/safety" in the name, without grappling with tough questions around what it actually means to help AGI go well or prioritising between options based on expected impact.

(On the term "EA mindset" -- it's really just a nickname; the thing I think we should care about is the focus on impact/use of the ideas.)

2. Our CEO (Niel Bowerman) spent several weeks recently with his top proactive priority helping figure out the top priorities within making AGI go well – i.e. which is more pressing (in the sense of where can additional talented people do the most marginal good) between issues like AI-enabled human coups, getting things right with rights and welfare of digital minds, and catastrophic misalignment. We argued about questions like "how big is the spread between issues within making AGI go well?" and "to what extent is AI rights and welfare an issue human has to get right before AI becomes incredibly powerful, due to potential lock-in effects of bad discourse or policies?"

So, we agree with this:

exactly what people end up doing within “AI safety” matters enormously from the EA perspective. [...] We actually care about the overall moral value of the long run future. Making AI less racist or preventing its use in petty scams doesn’t really cut it in those terms.

In other words, the analysis, as you say, is not done. It's gonna be hecka hard to figure out "the particulars of what to do with AI." And we do not "have it from here" – we need people thinking critically about this going forward so they stand the best chance of actually helping AGI go well, rather than just having a career in "something something AI."

[1](I'm currently debating whether we should add a 4th: tactical sophistication about AI.)

Anecdote: I'm one of those people -- would say I'd barely heard of ea / basically didn't know what it was, before a friend who already knew of it suggested I come to an EA global (I think at the time one got a free t-shirt for referring friends). We were both philosophy students & I studied ethics, so I think he thought I might be intersted even though we'd never talked about EA.

Thanks as always for this valuable data! 

Given 80k is a large and growing source of people hearing about and getting involved in EA, some people reading this might be worried that 80k will stop contributing to EA's growth, given our new strategic focus on helping people work on safely navigating the transition to a world with AGI. 

tl;dr I don't think it will stop, and might continue as before, though it's possible it will be reduced some.

More:

I am not sure whether 80k's contribution to building ea in terms of sheer numbers of people getting involved is likely to go down due to this focus vs. what it would otherwise be if we simply continued to scale our programmes as they currently are without this change in direction. 

My personal guess at this time is that it will reduce at least slightly.

Why would it? 

• We will be more focused on helping people work on helping AGI go well - that means that e.g. university groups might be hesitant to recommend us to members who are not interested in AIS as a cause area
• At a prosaic level, some projects that would have been particularly useful for building EA vs. helping with AGI in a more targeted way are going to be de-prioritised - e.g. I personally dropped a project I began of updating our "building ea" problem profile in order to focus more on AGI targeted things
• Our framings will probably change. It's possible that the framings we use more going forward will emphasise EA style thinking a little less than our current ones, though this is something we're actively unsure of.
• We might sometimes link off to the AI safety community in places where we might have linked off to EA before (though it is much less developed, so we're not sure).

However, I do expect us to continue to significantly contribute to building EA – and we might even continue to do so at a similar level vs. before. This is for a few reasons: 

1. We still think EA values are important, so still plan to talk about them a lot. E.g. we will talk about *why* we're especially concerned about AGI using EA-style reasoning, emphasise the importance of impartiality and scope sensitivity, etc.
2. We don't currently have any plans for reducing our links to the ea community – e.g. we don't plan to stop linking to the EA forum, or stop using our newsletter to notify people about EAGs.
3. We still plan to list meta EA jobs on our job board, put advisees in touch with people from the EA community when it makes sense, and by default keep our library of content online
4. We're not sure whether, in terms of numbers, the changes we're making will cause our audience to grow or shrink. On the one hand, it's a more narrow focus, so will appeal less to people who aren't interested in AI. On the other, we are hoping to appeal more to AI-interested people, as well as older people, who might not have been as interested in our previous framings.

This will probably lead directly and indirectly to a big chunk of our audience continuing to get involved in EA due to engaging with us. This is valuable according to our new focus, because we think that getting involved in EA is often useful for being able to contribute positively to things going well with AGI. 

To be clear, we also think EA growing is valuable for other reasons (we still think other cause areas matter, of course!). But it's actually never been an organisational target[1] of ours to build EA (or at least it hasn't since I joined the org 5 years ago); growing EA has always been something we cause as a side effect of helping people pursue high impact careers (because, as above, we've long thought that getting involved in EA is one useful step for pursuing a high impact career!) 

Note on all the above: the implications of our new strategic focus for our programmes are still being worked out, so it's possible that some of this will change.

Also relevant: FAQ on the relationship between 80k & EA (from 2023 but I still agree with it)

[1] Except to the extent that helping people into careers building EA constitutes helping them pursue a high impact career - & it is one of many ways of doing that (along with all the other careers we recommend on the site, plus others). We do also sometimes use our impact on the growth of EA as one proxy for our total impact, because the data is available, and we think it's often a useful step to having an impactful career, & it's quite hard to gather data on people we've helped pursue high impact careers more directly.

Hey Geoffrey,

Niel gave a response to a similar comment below -- I'll just add a few things from my POV:

• I'd guess that pausing (incl. for a long time) or slowing downAGI development would be good for helping AGI go well if it could be done by everyone / enforced / etc- so figuring out how to do that would be in scope re this more narrow focus. SO e.g. figuring out how an indefinite pause could work (maybe in a COVID-crisis like world where the overton window shifts?) seems helpful
• I (& others at 80k) are just a lot less pessimistic vis a vis the prospects for AGI going well / not causing an existential catastrophe. So we just disagree about the premise that "there is actually NO credible path for 'helping the transition to AGI go well'". In my case maybe because I don't believe your (2) is necessary (tho various other governance things probably are) & I think your (1) isn't that unlikely to happen (tho very far from guaranteed!)
• I'm at the same time more pessimistic about everyone the world stopping development toward this hugely commercially exciting technology, so feel like trying for that would be a bad strategy.

We don’t have anything written/official on this particular issue I don't think (though we have covered other mental health topics here), though this is one reason why we don’t think it’s the case that everyone should work on AIS/trying to help things go well with AGI, such that even though we want to encourage more people to consider it, we don’t blanket recommend it to everyone. We wrote a little bit here about an issue that seems related - what to do if you find the case for an issue intellectually compelling but don't feel motivated by it.

Hi Romain,

Thanks for raising these points (and also for your translation!)

We are currently planning to retain our cause-neutral (& cause-opinionated), impactful careers branding, though we do want to update the site to communicate much more clearly and urgently our new focus on helping things go well with AGI, which will affect our brand.

How to navigate the kinds of tradeoffs you are pointing to is something we will be thinking about more as we propagate through this shift in focus through to our most public-facing programmes. We don't have answers just yet on what that will look like, but do plan to take into account feedback from users on different framings to try to help things resonate as well as we can, e.g. via A/B tests and user interviews.

Thanks for the feedback here. I mostly want to just echo Niel's reply, which basically says what I would have wanted to. But I also want to add for transparency/accountability's sake that I reviewed this post before we published it with the aim of helping it communicate the shift well – I focused mostly on helping it communicate clearly and succinctly, which I do think is really important, but I think your feedback makes sense, and I wish that I'd also done more to help it demonstrate the thought we've put into the tradeoffs involved and awareness of the costs. For what it's worth, & we don't have dedicated comms staff at 80k - helping with comms is currently part of my role, which is to lead our web programme.