It's unclear what the first O in "TONOP..." stands for, but considering how memorable and widely-adopted this terminology has become, it would be unwise to change this now.
acknowledges the value of x-risk reduction in general from a non-longtermist perspective
clarifies that it is making a point about the marginal altruistic value of x-risk vs AW or GHW work and points to a post making this argument in more detail
Your response merely reiterates that x-risk prevention has substantial altruistic (and non-altruistic) value. This isn't responsive to the claim about whether, under non-longtermist assumptions, that value is greater on the margin than AW or GHW work.
So even though I actually agree with the claims in your comment, I downvoted it (along with this one complaining about the downvotes) for being off-topic and not embodying the type of discourse I think the EA Forum should strive for.
I take your claim in the post not to be "the fact that an offer is +EV is one strong reason to be in favor of it," but rather "you ought to take the cosmic coin flip, risking the universe, just because it is +EV." (Because being +EV definitionally means the good scenario is super unbelievably good, much better than most people considering the thought experiment are probably imagining.)
But even within the thought experiment, abstracting away all empirical uncertainties, I have enough philosophical uncertainty about EV maximization that I don't want to take the bet.
Again, just giving my impressions from interacting with AI safety people: it doesn't seem to me like I get this impression by drawing a larger circle -- I don't recall hearing the types of arguments you allude to even from people I consider "core" to AI safety. I think it would help me understand if you were able to provide some examples? (Although like I said, I found examples either way hard to search for, so I understand if you don't have any available.)
I still disagree about the Dial post: at the end Zvi says
Seeing highly intelligent thinkers who are otherwise natural partners and allies making a variety of obvious nonsense arguments, in ways that seem immune to correction, in ways that seem designed to prevent humanity from taking action to prevent its own extinction, is extremely frustrating. Even more frustrating is not knowing why it is happening, and responding in unproductive ways.
So my read is that he wants to explain and understand the position as well as possible, so that he can cooperate as effectively as possible with people who take the Dial position. He also agrees on lots of object-level points with the people he's arguing against. But ultimately actually using the Dial as an argument is "obvious nonsense," for the same reason the Technology Bucket Error is an error.
I agree that there's been a phenomenon of people suddenly all agreeing that all of SBF's opinions were wrong post-FTX collapse. So I appreciate the effort to make the case for taking the deal, and to portray the choice as not completely obvious.
To the extent that you're hoping to save "maximizing utility via maximizing expected value," I think it's still an uphill battle. I like Beckstead and Thomas's "A paradox for tiny probabilities and enormous values" on this, which runs essentially the same thought experiment as "flip the coin many times," except with the coin weighted to 99.9% heads (and only your own life in play, not the universe). They point out that both positions, "timidity" and "recklessness", have implausible conclusions.
I'm ultimately quite philosophically troubled by this "concentrating all the value into narrow regions of probability space" feature of EV maximization as a result (but I don't have a better alternative on hand!). This makes me, in particular, not confident enough in EV-maximization to wager the universe on it. So while I'm more sympathetic than most to the position that the coin flip might be justifiable, I'm still pretty far from wanting to bite that bullet.
I agree with the main ideas of this post. But I want to flag that, as someone who's engaged with the AI safety community (outside of the Bay Area) for several years, I don't recognize this depiction. In my experience, it's very common to say "even a [1, 5, 10]% chance of AI x-risk justifies taking this very seriously."
I don't have time to track down many examples, but just to illustrate:
Neel Nanda's "Simplify EA Pitches to 'Holy Shit, X-Risk'": "If you believe the key claims of "there is a >=1% chance of AI causing x-risk and >=0.1% chance of bio causing x-risk in my lifetime" this is enough to justify the core action relevant points of EA."
Toby Ord estimated AI x-risk at 10% in The Precipice.
Scott Alexander is sympathetic to libertarianism and put AI x-risk at 20-25% in 2023.
To address another claim: "The Dial of Progress" by Zvi, a core LessWrong contributor, makes the case that technology is not always good (similar to the "Technology Bucket Error" post) and the comments overwhelmingly seem to agree.
I'm sure someone has said that 10-25% x-risk would not be worth addressing due to libertarianism -- but I don't believe I've heard this argument, and wasn't able to find someone making it after a few minutes of searching. (But it's a hard thing to search for.)
I don't doubt that Holly is accurately reporting her experiences, and she's almost certainly engaged more widely than I have with people in AI safety. I wouldn't be surprised if there are people saying these things in the Bay Area. But I don't have the impression that they represent the mainstream of the AI safety community in any forum I'm familiar with (Twitter, LW, and definitely EA).
While I agree with many points in this post, I think it would be stronger if it engaged more with the existing discussion within EA on mental health, on the Forum and elsewhere.
The first three sections of the career guide’s "How to be successful in any job" chapter emphasize self-care, with the second section stating that dealing with poor mental health should be the top priority
Several programs have popped up over the years offering to provide or connect EAs with mental health services. Currently Rethink Wellbeing is active and provides CBT and IFS-based peer-facilitated programs. (There may be others I’m forgetting.)
A few of these seem to me like the sort of thing the suggestions were asking for, e.g. "a few podcast episodes that could succinctly demonstrate the way therapy might explore common blind spots held by EAs, providing a rapid update with less resistance than other methods."
I've personally experienced mental health challenges due to EA, so I'm certainly not saying the problems are all solved, or that the resources above cover everything. Publishing one podcast doesn't solve a community-wide problem. But parts of this post read to me as suggesting these resources and discussions don't exist, so I want to provide an alternate perspective.
What makes the agriculture transition stand out as the "point of no return"? Agriculture was independently invented several times, so I'd expect the hunter-gatherer -> agriculture transition to be more convergent than agriculture -> AGI.
Sorry you're right, you didn't say this -- I misread that part of your comment.
I still think your framing misses something important: the logic "50% of people are women so I think women’s suffrage had a pretty strong support base" applies at all points in time, so it doesn't explain why suffrage was so unpopular for so long. Or to put it another way, for some reason the popularity and political influence of the suffrage movement increased dramatically without the percentage of women increasing, so I'm not sure the percentage of people who are women is relevant in the way you're implying.
The idea that you can go regulating without considering public support/resistance is silly
On the other hand I didn't say this! The degree of public support is certainly relevant. But I'm not sure what your practical takeaway or recommendation is in the case of an unpopular movement.
For example you point out abolition as an example where resistance caused massive additional costs (including the Civil War in the US). I could see points 1, 3, 7, and possibly 8 all being part of a "Ways I see the Quaker shift to abolitionism backfiring" post. They could indeed be fair points that Quakers / other abolitionists should have considered, in some way -- but I'm not sure what that post would have actually wanted abolitionists to do differently, and I'm not sure what your post wants EAs to do differently.
Maybe you just intend to be pointing out possible problems, without concluding one way or another whether the GH -> AW shift is overall good or bad. But I get a strong sense from reading it that you think it's overall bad, and if that's the case I don't know what the practical upshots are.
I'm not super knowledgeable about women's suffrage, but
It was not universally supported by women (See e.g. here; I couldn't quickly find stats but I'd be interested).
Surely the relevant support base in this case is those who had political power, and the whole point is that women didn't. So "50% support" seems misleading in that sense.
I could similarly say ">99.999% of animals are nonhumans, so nonhuman animal welfare has an extremely large support base." But that's not the relevant support base for the discussion at hand.
Can't wait to meet other TONOPWAPSATIAWCDOTOWMAANBists[1] at CGoLMIWMAKEOBDAPoASMC this year!
The Network Of People Who All Pursue Systematic Altruism Together In A Way Causally Downstream Of Toby Ord, Will MacAskill, And Nick Bostrom[2]
It's unclear what the first O in "TONOP..." stands for, but considering how memorable and widely-adopted this terminology has become, it would be unwise to change this now.
The comment you replied to
Your response merely reiterates that x-risk prevention has substantial altruistic (and non-altruistic) value. This isn't responsive to the claim about whether, under non-longtermist assumptions, that value is greater on the margin than AW or GHW work.
So even though I actually agree with the claims in your comment, I downvoted it (along with this one complaining about the downvotes) for being off-topic and not embodying the type of discourse I think the EA Forum should strive for.
I take your claim in the post not to be "the fact that an offer is +EV is one strong reason to be in favor of it," but rather "you ought to take the cosmic coin flip, risking the universe, just because it is +EV." (Because being +EV definitionally means the good scenario is super unbelievably good, much better than most people considering the thought experiment are probably imagining.)
But even within the thought experiment, abstracting away all empirical uncertainties, I have enough philosophical uncertainty about EV maximization that I don't want to take the bet.
Again, just giving my impressions from interacting with AI safety people: it doesn't seem to me like I get this impression by drawing a larger circle -- I don't recall hearing the types of arguments you allude to even from people I consider "core" to AI safety. I think it would help me understand if you were able to provide some examples? (Although like I said, I found examples either way hard to search for, so I understand if you don't have any available.)
I still disagree about the Dial post: at the end Zvi says
So my read is that he wants to explain and understand the position as well as possible, so that he can cooperate as effectively as possible with people who take the Dial position. He also agrees on lots of object-level points with the people he's arguing against. But ultimately actually using the Dial as an argument is "obvious nonsense," for the same reason the Technology Bucket Error is an error.
I agree that there's been a phenomenon of people suddenly all agreeing that all of SBF's opinions were wrong post-FTX collapse. So I appreciate the effort to make the case for taking the deal, and to portray the choice as not completely obvious.
To the extent that you're hoping to save "maximizing utility via maximizing expected value," I think it's still an uphill battle. I like Beckstead and Thomas's "A paradox for tiny probabilities and enormous values" on this, which runs essentially the same thought experiment as "flip the coin many times," except with the coin weighted to 99.9% heads (and only your own life in play, not the universe). They point out that both positions, "timidity" and "recklessness", have implausible conclusions.
I'm ultimately quite philosophically troubled by this "concentrating all the value into narrow regions of probability space" feature of EV maximization as a result (but I don't have a better alternative on hand!). This makes me, in particular, not confident enough in EV-maximization to wager the universe on it. So while I'm more sympathetic than most to the position that the coin flip might be justifiable, I'm still pretty far from wanting to bite that bullet.
I agree with the main ideas of this post. But I want to flag that, as someone who's engaged with the AI safety community (outside of the Bay Area) for several years, I don't recognize this depiction. In my experience, it's very common to say "even a [1, 5, 10]% chance of AI x-risk justifies taking this very seriously."
I don't have time to track down many examples, but just to illustrate:
To address another claim: "The Dial of Progress" by Zvi, a core LessWrong contributor, makes the case that technology is not always good (similar to the "Technology Bucket Error" post) and the comments overwhelmingly seem to agree.
I'm sure someone has said that 10-25% x-risk would not be worth addressing due to libertarianism -- but I don't believe I've heard this argument, and wasn't able to find someone making it after a few minutes of searching. (But it's a hard thing to search for.)
I don't doubt that Holly is accurately reporting her experiences, and she's almost certainly engaged more widely than I have with people in AI safety. I wouldn't be surprised if there are people saying these things in the Bay Area. But I don't have the impression that they represent the mainstream of the AI safety community in any forum I'm familiar with (Twitter, LW, and definitely EA).
While I agree with many points in this post, I think it would be stronger if it engaged more with the existing discussion within EA on mental health, on the Forum and elsewhere.
For example:
A few of these seem to me like the sort of thing the suggestions were asking for, e.g. "a few podcast episodes that could succinctly demonstrate the way therapy might explore common blind spots held by EAs, providing a rapid update with less resistance than other methods."
I've personally experienced mental health challenges due to EA, so I'm certainly not saying the problems are all solved, or that the resources above cover everything. Publishing one podcast doesn't solve a community-wide problem. But parts of this post read to me as suggesting these resources and discussions don't exist, so I want to provide an alternate perspective.
What makes the agriculture transition stand out as the "point of no return"? Agriculture was independently invented several times, so I'd expect the hunter-gatherer -> agriculture transition to be more convergent than agriculture -> AGI.
Sorry you're right, you didn't say this -- I misread that part of your comment.
I still think your framing misses something important: the logic "50% of people are women so I think women’s suffrage had a pretty strong support base" applies at all points in time, so it doesn't explain why suffrage was so unpopular for so long. Or to put it another way, for some reason the popularity and political influence of the suffrage movement increased dramatically without the percentage of women increasing, so I'm not sure the percentage of people who are women is relevant in the way you're implying.
On the other hand I didn't say this! The degree of public support is certainly relevant. But I'm not sure what your practical takeaway or recommendation is in the case of an unpopular movement.
For example you point out abolition as an example where resistance caused massive additional costs (including the Civil War in the US). I could see points 1, 3, 7, and possibly 8 all being part of a "Ways I see the Quaker shift to abolitionism backfiring" post. They could indeed be fair points that Quakers / other abolitionists should have considered, in some way -- but I'm not sure what that post would have actually wanted abolitionists to do differently, and I'm not sure what your post wants EAs to do differently.
Maybe you just intend to be pointing out possible problems, without concluding one way or another whether the GH -> AW shift is overall good or bad. But I get a strong sense from reading it that you think it's overall bad, and if that's the case I don't know what the practical upshots are.
I'm not super knowledgeable about women's suffrage, but
I could similarly say ">99.999% of animals are nonhumans, so nonhuman animal welfare has an extremely large support base." But that's not the relevant support base for the discussion at hand.