That's a really great point, Dylan!

Yeah, I agree with Chana.

I think that, if there is some cause area that is distinctly more important than all other cause areas, it would make a lot of sense for 80k to simply shift to focusing on that cause area.

Like, if 95% of virologists thought an engineered pandemic could wipe out humanity in the next ten years and 80k shifted to just focusing on preventing engineered pandemics, I wouldn't be saying that that is a "loss" for EA.

That said, in general, it seems very hard to compare cause areas because of the epistemic limitations we face in predicting the consequences of our actions, in understanding our world, and in determining how to act under different kinds of uncertainty. As such, I generally would prefer 80,000 Hours to be more cause neutral.

Yeah, I definitely wonder if you're right about this. As someone who's unsure about AI, I'd much prefer people to come to AI by being curious about EA first rather than having it be the main idea they're exposed to because then that's a strong indicator that AI is something worth being seriously worried about.

It also seems like 80k's AI-focused guide could have been more of a spun-out project or just one part of their site rather than being the whole thing and that would have had a similar/higher level of impact because of the selection effect you mentioned.

Hi Oscar,

I do think that taking the future into account when making decisions is important and that this process can inevitably come down to a lot of subjective speculations. For instance, when comparing the importance of alternative proteins to other approaches for helping animals, the best we can really do is making subjective predictions.

That said, I think that Forethought goes too far with its speculation by attempting to approach a vast space of future possibilities with minimal revision and external dispute about its ideas. To me, it seems like, if any of the ideas discussed by the authors were approached by different people, very different conclusions and visions of the future would be come to because the organization's approach to outlining the future is methodologically flawed. For this reason, it doesn't seem like virtually any of the research would pass peer review.

If I have time, it would be very interesting to critique the piece and hear your thoughts!

Also, thanks for the cordial response given my criticism of your piece. I wasn't expecting that you'd see my comment.

I'd like to see an AMA with Lewis Bollard.

Thanks for the thoughtful response!

Hi Chana,

Thanks for taking time out of your day to respond to my post, and I appreciate your honesty.

To be clear, I have spent a significant amount of time looking into AI. I have read most of 80k's AI content and seen many of your podcasts and all of the AI In Context videos (as well as a lot of other AI content), and I still find myself feeling that I would need to research the topic a lot more to feel convinced about risks from AI takeover. Given this, it's not that I think that AI takeover won't happen as much as I don't have a strong view either way, and I feel I need to research the topic more.

Given this, I was writing the post with the assumption that AI takeover should not be EA's dominant focus, and I definitely understand your criticism that I haven't sufficiently made the argument for that assumption.

I want to make clear that I'm very sympathetic to concerns about AI takeover, and I don't want people at 80k to feel that I'm dismissing their work.

As someone with a more vanilla EA worldview, my intent in writing this was to process my frustration for losing a meta EA org, not to invalidate the work of others.

I have added an edit to the top of the post to make these caveats and my intent clear. I apologize for not initially including them. Thanks for responding so thoughtfully.

That said, I do think many of the problem profiles really are very esoteric in that, if we were to re-play the 21st century, it's unlikely most of these concerns would be discovered/created.

For instance, Forethought's work, which inspired the "AI-enhanced decision making profile" and, to some extent, the "Space governance" and "Extreme power concentration" profiles is very bizarre and only makes sense in context to the mythology created by FHI. For instance, one of their recent articles is "A nightwatchman on every probe: Superintelligent surveillance to prevent galactic anarchy." To me, this is a very fringe idea that has many assumptions about the future and the extent to which we can influence it baked into it. I think that much of Forethought's work is a concerning departure from the very evidence-based roots of the movement and the significantly researched concerns about AI, and I think their work is indicative of an extreme echo chamber.

Given this, I do think it's problematic for 80,000 Hours to highlight their work given how fringe and speculative it is.

It feels like EA has, at its edges, a tendency to push towards increasingly unusual topics, but it's important to not let that be motivated by social factors rather than evidential ones. (For instance, there is actually strong evidence that shrimp are sentient. The same cannot be said for whether we need to start coming up with considerations for what principles should be baked into superintelligent interstellar probes.)

Also, despite spending a couple of hours looking for it, I have been unable to find evidence that engineered pandemics are a mainstream concern among biosecurity experts, which makes me pretty skeptical that we should be concerned about them.

Yeah, I'm definitely sympathetic to people's concerns about AI as you are, although I guess the concern is so speculative and extraordinary that I find it difficult to believe it should be almost every visitor to 80k's primary career choice.

I'm very interested in preserving EA as a movement, but I don't have very strong opinions about AI. That's why I wrote the post in this way.

The idea that AI will disempower humanity is such an extraordinary claim that I think I would need to spend a very long time researching it before coming to any strong conclusions about it. I've read a lot of "AI safety" texts, but some of the most popular ones seem more like sci-fi world-building than arguments so I feel I have good reason to be skeptical of the topic.

Given this, I'd, currently, rather spend my time looking into other topics. After all, researching AI comes with a large opportunity cost.

Yeah, I definitely wonder that too.

I discovered EA at 22 so I generally feel like EA needs to do a better job at making the general population aware of its ideas.