Successionism is not a good argument for building ASI. If your goal is to build a sentient ASI that's maximally flourishing (which is a good thing to do if you ask me*), then you still need to solve the alignment problem. A misaligned ASI will not optimize for well-being, whether that's human well-being or AI well-being. It will optimize for whatever arbitrary inscrutable goals it has.
*but like, surely there is a way to do this without also wiping out humanity?
I'm not sure this is as obvious as you're making out. If you give a misaligned ASI a goal to maximize its own wellbeing it could still do that very well, it may also just do other things you didn't intend for it to do (like wipe out humans).
A paperclip maximizer does its job really well after all, and we wouldn't call it aligned.
I recently posted complaining about how EA's were unfairly stereotyped. To balance that, I wanted to share an article in which EA's are fairly stereotyped!
This excellent article by Dan Hendrycks talks about how utilitarianism (and to a lesser extent accelerationism) at AI companies creates a risk of human extinction.
People may agree or disagree with the argument, but unlike several mainstream depictions of EA, Dan's article is well-informed and explains in detail why he believes that EA thinking (to the extent that EA's are more utilitarian than average) is dangerous. At minimum, IMHO, EA's need to be ready to answer this argument if we do not want to be dismissed out of hand as credible commentators.
(several commentors on my original post replied that they didn't think the stereotypes I saw were so unfair, maybe this article is a good explanation of why.).
I think the article gives the impression that EAs are generally happy for humans to go extinct if they are replaced by happy AIs. But this isn't my impression. EA, at least outwardly, is very concerned about human extinction! Almost obsessively so!
I think a utilitarian can be in favor of humans staying in control and surviving. They might not think AIs will be capable of wellbeing, or might think there is a good chance they will have negative wellbeing, or might not be confident they will create a flourishing future.
What annoyed me most about the Hendryck's article is that he essentially assumes impartial utilitarianism must be wrong because it can lead to a conclusion he (and others) don't like. Impartial utilitarianism also would have suggested to slave owners they shouldn't keep slaves - a conclusion they and others wouldn't have liked!
I would have liked more philosophical discussion about if impartiality is or is not a good moral principle. But I suppose he wasn't interested in getting into that.
I think all your comments are valid. I definitely did not agree with the conclusions or the negativity towards utilitarianism as the vast majority of EA's see it.
My motive for sharing this was: in previous posts I've shared examples of where EA's are unjustly considered, basically, to weird to be relevant, without any clear justification of why. In this article, at least the allegedly flawed assumptions of EA's were clearly outlined, and it was not outrageous to argue that, yes, some EA's do think like that sometimes.
So, IMHO we all need to be ready with arguments like what you've outlined, to say "I can see why you think that - but here is a better way to think about it, and I think it will reassure you that most EA's, even those who call themselves utilitarians, do not have these crazy ideas."
Agreed. I feel like eigenism has many implications that are pretty morally uncomfortable. For instance, folks seem more closely connected to others of the same gender. Does that mean sexism is morally correct? Seems bad. Same for racism, nationalism, speciesism etc.
I'm aware Hendrycks claims that eigenism doesn't support racism, but the eigenism paper also suggests that the family dog has ~75% the moral weight as all different-culture strangers combined (table 1 https://eigenism.org/paper.pdf). This is a conclusion which is both actually repugnant, and disagrees with common sense morality.