Thanks again for your comment. Two quick related points:
People at the places you mention are definitely already doing interesting work relevant to ideal theory, e.g., regarding institutional design. Something distinctive about AI ideal governance research that I do think is less common is consideration of the normative components of AI governance issues.
On reflection, your comments and examples have convinced me that in the original post I didn't take the 'weirdness problem' seriously enough. Although I'd guess we might still have a slight disagreement about the scope (and possibly the implications) of the problem, I certainly see that it is particularly salient for longtermists at the moment given the debates around the publication of Will MacAskill's new book. As an example of ideal governance research that considers longer-term issues (including some 'weird' ones) in an analytical manner, Nick Bostrom, Allan Dafoe and Carrick Flynn's paper on 'Public Policy and Superintelligent AI' may be of interest.
Thanks for this comment John. Briefly, one related thought I have is that weirdness is an important concern but that not all AI ideal governance theories are necessarily so weird that they're self-defeating. I'm less concerned that theories that consider what institutions/norms should look like in the nearer future (such as around the future of work) are too weird, for example.
Broadly, I think your comment reinforces an important concern, and that further research on this topic would benefit from being mindful of the purpose it is trying to serve and its intended audience.
Thank you! A few quick thoughts on some great points:
Other than Karnofsky's piece, I didn't find too much empirical research trying to understand why people find utopias appealing, but I share the intuition that a lack of scarcity/great wealth is often a plausible reason. There's an interesting open empirical question about how that relates with people's views about freedom, and a related normative one about our intuitions regarding the importance of freedom more generally.
Agree that it's very hard to construct pluralism-respecting utopias and even harder to work out ways to get there. In the post, main aim was to question the idea that the former is impossible.
Definitely think the question of how useful AI ideal governance theories can be is next step of discussion after establishing ways in which they can be helpful. I don't have too many abstract thoughts on how many people should be working in this sub-field (this may depend on what problem is trying to be solved at a given time) - mainly wanted to establish that, e.g., people interested in policy, shouldn't see it as totally irrelevant approach.
Thanks for posting this! I think it is often really useful to look at previous historical movements to learn lessons and contextualise modern concerns.
I really enjoyed that review, and found two potential further similarities between the Fabians and EA. Firstly, the Fabians had a number of strategies to have an impact. Those mentioned in the review include pamphleting, running political campaigns and even setting up universities. Secondly, the Fabians are said to have been a politically diverse group, or a 'big tent'. I think that at its best, EA can learn from a variety of different traditions to try to find the most effective ways to do good.
Thanks again for your comment. Two quick related points:
People at the places you mention are definitely already doing interesting work relevant to ideal theory, e.g., regarding institutional design. Something distinctive about AI ideal governance research that I do think is less common is consideration of the normative components of AI governance issues.
On reflection, your comments and examples have convinced me that in the original post I didn't take the 'weirdness problem' seriously enough. Although I'd guess we might still have a slight disagreement about the scope (and possibly the implications) of the problem, I certainly see that it is particularly salient for longtermists at the moment given the debates around the publication of Will MacAskill's new book. As an example of ideal governance research that considers longer-term issues (including some 'weird' ones) in an analytical manner, Nick Bostrom, Allan Dafoe and Carrick Flynn's paper on 'Public Policy and Superintelligent AI' may be of interest.
Thanks for this comment John. Briefly, one related thought I have is that weirdness is an important concern but that not all AI ideal governance theories are necessarily so weird that they're self-defeating. I'm less concerned that theories that consider what institutions/norms should look like in the nearer future (such as around the future of work) are too weird, for example.
Broadly, I think your comment reinforces an important concern, and that further research on this topic would benefit from being mindful of the purpose it is trying to serve and its intended audience.
Thank you! A few quick thoughts on some great points:
Other than Karnofsky's piece, I didn't find too much empirical research trying to understand why people find utopias appealing, but I share the intuition that a lack of scarcity/great wealth is often a plausible reason. There's an interesting open empirical question about how that relates with people's views about freedom, and a related normative one about our intuitions regarding the importance of freedom more generally.
Agree that it's very hard to construct pluralism-respecting utopias and even harder to work out ways to get there. In the post, main aim was to question the idea that the former is impossible.
Definitely think the question of how useful AI ideal governance theories can be is next step of discussion after establishing ways in which they can be helpful. I don't have too many abstract thoughts on how many people should be working in this sub-field (this may depend on what problem is trying to be solved at a given time) - mainly wanted to establish that, e.g., people interested in policy, shouldn't see it as totally irrelevant approach.
Thanks for posting this! I think it is often really useful to look at previous historical movements to learn lessons and contextualise modern concerns.
I really enjoyed that review, and found two potential further similarities between the Fabians and EA. Firstly, the Fabians had a number of strategies to have an impact. Those mentioned in the review include pamphleting, running political campaigns and even setting up universities. Secondly, the Fabians are said to have been a politically diverse group, or a 'big tent'. I think that at its best, EA can learn from a variety of different traditions to try to find the most effective ways to do good.