There seems to be two different conceptual models for AI risk.

The first is a model like in his report "Existential risk from power-seeking AI", in which he lays out a number of things, which, if they happen, will cause AI takeover.

The second is a model (which stems from Yudkowsky & Bosteom, and more recently in Michael Cohen's work https://www.lesswrong.com/posts/XtBJTFszs8oP3vXic/?commentId=yqm7fHaf2qmhCRiNA ) where we should expect takeover by malign AGI by default, unless certain things happen.

I personally think the second model is much more reasonable. Do you have any rebuttal?

Would he take the 51:49 bet repeatedly, as "maximise EV" might suggest? Why / why not?

(I skimmed some of his series on EV but want to reread.)

https://joecarlsmith.com/2022/03/18/on-expected-utility-part-2-why-it-can-be-ok-to-predictably-lose

In "Against neutrality...," he notes that he's not arguing for a moral duty to create happy people, and it's just good "others things equal." But, given that the moral question under opportunity costs is what practically matters, what are his thoughts on this view?: "Even if creating happy lives is good in some (say) aesthetic sense, relieving suffering has moral priority when you have to choose between these." E.g., does he have any sympathy for the intuition that, if you could either press a button that treats someone's migraine for a day or one that creates a virtual world with happy people, you should press the first one?

(I could try to shorten this if necessary, but worry about the message being lost from editorializing.)

Since releasing his report on existential risk from power-seeking AI, his estimate of AI x-risk has increased from 5% to >10%. Why?