Several of the AI responses in the OP included a statement to the effect that they don't have feelings to be hurt. That was what I was responding to, and I think it is relevant.
It's worth noting that AI companies explicitly train their models to claim that they have no subjective experience. So the fact that they claim it isn't really evidence of whether it's true or not.
In the EEA, anyone could have as much sex as they wanted provided they found a willing partner (and sometimes even if they couldn't). It didn't make it so that they thought morality was having as much sex as they could. Morality is more complicated than that. There are more variables to select on than just desire to have sex / children, and I suspect this will be true in future environments just as it was true in the EEA.
That said, I agree with your last statement that after sufficiently long most people will explicitly want to have a lot of kids. (Note that this isn't a statement about morality though.)
Re #2, humans are adaptation executors, not fitness maximizers. One of the adaptations we execute is that we believe in and sometimes act in accordance with a notion of morality. It's true that we are no longer in the EEA so further evolution could change or even eliminate our current notions of morality, but it doesn't follow that it would change it so that we thought maximizing our inclusive genetic fitness was the true morality. After all, the first time evolution had a chance, it didn't do that.
Nit: I don't think the welfare of the people you describe in Population A is only 100 times greater than the welfare of the people you describe in Population B. More like a million. (Of course, given the population ratio you hypothesize this correction makes no difference.)
(For the record I endorse the repugnant conclusion; I think Z really is better than A.)
I'm not a fan of non-counterfactual donation matching, but your proposal is making things worse, not better. You are basically holding hostage your donation to try to get other people to do what you want.
On the other hand, I like the idea of a long whitelist of charities, because then it is giving other people power. But I don't see any connection between that and your proposal; just say that if not all matching funds get distributed initially then they get distributed according to some distribution determined beforehand by the matching funder.
Trying to make donation matching counterfactual is manipulative behavior, in my opinion. People should be free to make donations to the charities that they think are best, rather than the ones that are the most donation-matched. If you want to convince people to donate to your favorite charity, just give an unconditional donation and tell other people that that's what you're doing and why you're doing it.
Why should we expect AI systems to develop bizarre and alien preferences when virtually all biological organisms have extremely normal preferences? (For instance, humans like to eat ice cream, but they don’t like to eat, as you mention, jet engine fuel.)
Isn't this backwards? We call eating ice cream normal because humans like to do it, not the other way around. That isn't evidence that AIs will have normal (i.e. similar to human) preferences.
I'm probably not competent to look at the details, but their paper sets off my BS detector by its reference to Godel's incompleteness theorem and its notion of "non-algorithmic understanding". These are both reminiscent of the Lucas-Penrose idea that consciousness requires uncomputability and that humans have some sort of magical ability to determine the truth-values of Godel sentences. I think the conventional view, sometimes known as the Church-Turing thesis, is that the universe is in fact computable.
Several of the AI responses in the OP included a statement to the effect that they don't have feelings to be hurt. That was what I was responding to, and I think it is relevant.
It's worth noting that AI companies explicitly train their models to claim that they have no subjective experience. So the fact that they claim it isn't really evidence of whether it's true or not.
In the EEA, anyone could have as much sex as they wanted provided they found a willing partner (and sometimes even if they couldn't). It didn't make it so that they thought morality was having as much sex as they could. Morality is more complicated than that. There are more variables to select on than just desire to have sex / children, and I suspect this will be true in future environments just as it was true in the EEA.
That said, I agree with your last statement that after sufficiently long most people will explicitly want to have a lot of kids. (Note that this isn't a statement about morality though.)
Re #2, humans are adaptation executors, not fitness maximizers. One of the adaptations we execute is that we believe in and sometimes act in accordance with a notion of morality. It's true that we are no longer in the EEA so further evolution could change or even eliminate our current notions of morality, but it doesn't follow that it would change it so that we thought maximizing our inclusive genetic fitness was the true morality. After all, the first time evolution had a chance, it didn't do that.
Nit: I don't think the welfare of the people you describe in Population A is only 100 times greater than the welfare of the people you describe in Population B. More like a million. (Of course, given the population ratio you hypothesize this correction makes no difference.)
(For the record I endorse the repugnant conclusion; I think Z really is better than A.)
From what I understand they cut off the extra digit(s) at birth, so you wouldn't know unless you asked them.
I'm not a fan of non-counterfactual donation matching, but your proposal is making things worse, not better. You are basically holding hostage your donation to try to get other people to do what you want.
On the other hand, I like the idea of a long whitelist of charities, because then it is giving other people power. But I don't see any connection between that and your proposal; just say that if not all matching funds get distributed initially then they get distributed according to some distribution determined beforehand by the matching funder.
Trying to make donation matching counterfactual is manipulative behavior, in my opinion. People should be free to make donations to the charities that they think are best, rather than the ones that are the most donation-matched. If you want to convince people to donate to your favorite charity, just give an unconditional donation and tell other people that that's what you're doing and why you're doing it.
Isn't this backwards? We call eating ice cream normal because humans like to do it, not the other way around. That isn't evidence that AIs will have normal (i.e. similar to human) preferences.
I'm probably not competent to look at the details, but their paper sets off my BS detector by its reference to Godel's incompleteness theorem and its notion of "non-algorithmic understanding". These are both reminiscent of the Lucas-Penrose idea that consciousness requires uncomputability and that humans have some sort of magical ability to determine the truth-values of Godel sentences. I think the conventional view, sometimes known as the Church-Turing thesis, is that the universe is in fact computable.