What would it mean for instrumental convergence to be "false" in this context?
I don't think I can imagine such a world.
What would it mean for instrumental convergence to be "false" in this context?
I don't think I can imagine such a world.
I wanted to keep it abstract to reduce the risk of priming people, but something like 'doesn't give reason to suppose any particular of the supposedly convergent behaviours are strongly linked in nontrivial ways or likely to dominate in any particular AI system we should expect to encounter in the near future'.
I'm working on an essay arguing that its conclusions basically aren't well-supported in this respect. See also the series Vasco posted in one of the other comments to this poll.
Yesterday I read (some of) Scott Alexander's recent open letter to Steven Pinker. In it, he lists 3 mecanisms that I suppose he considers to be central to the credibility of AI-induced catastrophic scenarios: it starts here.
(they're instrumental convergence, misgeneralization, and reward-hacking)
Instrumental convergence is the thesis that sufficiently intelligent agents with a high proportion of goals will tend to pursue certain similar instrumental goals, such as resource- and power-seeking, self-preservation, and recursive self-improvement.
I'm writing on the subject, and want to get a sense of how important the community thinks this is in the context of AI safety. So this needs two polls for comparison.
Defining 'catastrophe' as 'a >50% drop in human population in under a decade' and 'AI-caused' as 'some event clearly tracing to the action of one or more AIs alone or of humans using AI' (so e.g. an artificially created pandemic would count).
(I couldn't see a way to get a legend on the poll, but there are 20 values available, so they're implicitly increments of 5%)
Please give any context that you think matters.
Ty!
Hello. You may be interested in David Thorstad's series Instrumental convergence and power seeking.