Pros:
- Reduces company incentive to dangerously race to build AI (also increases government incentive to do that, but governments don't really respond to incentives in the same way)
- Alleviates economic disenfranchisement due to AI displacing human workers (I'm more concerned about x-risk but this still matters more than zero)
Cons:
- Right now the general public hates AI. Putting everyone in a position to make money off AI might change that, and make it harder to get the safety regulations we actually need (this argument sounds right to me, but it also sounds suspiciously like accelerationism)
- Something about government overreach stifling capitalism (this is a real concern but it's dwarfed by x-risk concerns)
Neither-pro-nor-con:
- Has basically nothing to do with x-risk
(I'm deliberately sidelining the details of how exactly to establish an AI SWF because I don't think it is important this early in the conversation.)
On the pro side, a SWF is a good answer to intelligence curse issues. It may also give an higher chance of AI risks ("near-term" or "long-term") being weighted into AIcos' decisions (particularly those who are supposed to be PBCs), for the same reason the LTFF and OpenAI Foundation are beneficial and dilution through IPOs might be dangerous and lead them to act more like any publicly listed company.
A more subtle pro would be that with the more immediate and intense social fear of corporate domination subsiding, this would leave room for the more complicated issues, including existential risk, to enter the public debate.
On the anti side, there has been an argument floating around that the government owning AI stocks might discourage it from taking measures that might hurt AI profits.
I think this argument is very weak, the economy is already quite dependent on AI companies doing well, and oil SWFs don't seem to have been an issue for Norway and even the Gulf taking the green transition seriously.
I'm not even sure e.g. a pause would be a bad thing economically for AI companies? Training increasingly large models is the whole reason they can't make a profit right now.
The more interesting anti argument is that, badly constructed (in particular, if the influence of the "unitary executive" is stronger than the influence of the bipartisan, deliberative congress), this could serve as a move toward greater centralization into a small group, which would neutralize most of the pros I discussed earlier. This part is certainly more worrisome considering the current levels of corruption going on.