There seem to be two main framings emerging from recent AGI x-risk discussion: default doom, given AGI, and default we're fine, given AGI.
I'm interested in what people who have low p(doom|AGI) think are the reasons that things will basically be fine once we have AGI (or TAI, PASTA, ASI). What mechanisms are at play? How is alignment solved so that there are 0 failure modes? Can we survive despite imperfect alignment? How? Is alignment moot? Will physical limits be reached before there is too much danger?
If you have high enough p(doom|AGI) to be very concerned, but you're still only at ~1-10%, what is happening in the other 90-99%?
Added 22Apr: I'm also interested in detailed scenarios and stories, spelling out how things go right post-AGI. There are plenty of stories and scenarios illustrating doom. Where are the similar stories illustrating how things go right? There is the FLI World Building Contest, but that took place in the pre-GPT-4+AutoGPT era. The winning entry has everyone acting far too sensibly in terms of self-regulation and restraint. I think we can now say, given the fervour over AutoGPT, that this will not happen, with high likelihood.
People will continue to prefer controllable to uncontrollable AI and continue to make at least a commonsense level of investment in controllability; that is, they invest as much as naively warranted by recent experience and short term expectations, which is less than warranted by a sophisticated assessment of uncertainty about misalignment, though the two may converge as “recent experience” involves more and more capable AIs. I think this minimal level of investment in control is very likely (99%+).
Next, the proposed sudden/surprising phase transition that breaks controllability properties never materialises so that commonsense investment turns out to be enough for an OK outcome. I think this is about 65%.
Next, AI-enhanced human politics also manages to generate an OK outcome. About 70%.
That’s 45%, but the bar is perhaps higher than you have in mind (I’m also counting non misalignment paths to bad outcomes). There are also worlds where the problem is harder but more is invested & still ends up being enough. Not sure how much weight goes there, somewhat less.
Don’t take these probabilities too seriously.
That’s what I meant by “phase transition”