P(doom|AGI) is high: why the default outcome of AGI is doom

Ay thanks, sorry I’m late back to you. I’ll respond to various parts in turn.

I don't find Carlsmith et al's estimates convincing because they are starting with a conjunctive frame and applying conjunctive reasoning. ​​They are assuming we're fine by default (why?), and then building up a list of factors that need to go wrong for doom to happen.

My initial interpretation of this passage is: you seem to be saying that conjunctive/disjunctive arguments are presented against a mainline model (say, one of doom/hope). In presenting a ‘conjunctive’ argument, Carlsmith belies a mainline model of hope. However, you doubt the mainline model of hope, and so his argument is unconvincing. If that reading is correct, then my view is that the mainline model of doom has not been successfully argued for. What do you take to be the best argument for a ‘mainline model’ of doom? If I’m correct in interpreting the passage below as an argument for a ‘mainline model’ of doom, then it strikes me as unconvincing:

Any one of a vast array of things can cause doom. Just the 4 broad categories mentioned at the start of the OP (subfields of Alignment) and the fact that "any given [alignment] approach that might show some promise on one or two of these still leaves the others unsolved." is enough to provide a disjunctive frame!

Under your framing, I don’t think that you’ve come anywhere close to providing an argument for your preferred disjunctive framing. On my way of viewing things, an argument for a disjunctive framing shows that “failure on intent alignment (with success in the other areas) leads to a high P(Doom | AGI), failure on outer alignment alignment (with success in the other areas) leads to a high P(Doom | AGI), etc …”. I think that you have not shown this for any of the disjuncts, and an argument for a disjunctive frame requires showing this for all of the disjuncts.

Nate’s Framing

I claimed that an argument for (my slight alteration of) Nate’s framing was likely to rely on the conjunction of many assumptions, and you (very reasonably) asked me to spell them out. To recap, here’s the framing:

For humanity to be dead by 2070, only one of the following needs to be true:

1. Humanity has < 20 years to prepare for AGI
2. The technical challenge of alignment isn’t “pretty easy”
3. Research culture isn’t alignment-conscious in a competent way.

For this to be a disjunctive argument for doom, all of the following need to be true:

1. If humanity has < 20 years to prepare for AGI, then doom is highly likely. 
2. Etc … 

That is, the first point requires an argument which shows the following: 

A Conjunctive Case for the Disjunctive Case for Doom:^[1]

1. Even if we have a competent alignment-research culture, and 
2. Even if the technical challenge of alignment is also pretty easy, nevertheless 
3. Humanity is likely to go extinct if it has <20 years to prepare for AGI. 

If I try to spell out the arguments for this framing, things start to look pretty messy. If technical alignment were “pretty easy”, and tackled by a culture which competently pursued alignment research, then I don’t feel >90% confident in doom. The claim “if humanity has < 20 years to prepare for AGI, then doom is highly likely” requires (non-exhaustively) the following assumptions:

1. Obviously, the argument directly entails the following: Groups of competent alignment researchers would fail to make ‘sufficient progress’ on alignment within <20 years, even if the technical challenge of alignment is “pretty easy”.
   1. There have to be some premises here which help make sense of why this would be true. What’s the bar for a competent ‘alignment culture’? 
   2. If the bar is low, then the claim does not seem obviously true. If the bar for ‘competent alignment-research culture’ is very high, then I think you’ll need an assumption like the one below.
2. With extremely high probability, the default expectation should be that the values of future AIs are unlikely to care about continued human survival, or the survival of anything we’d find valuable.
   1. I will note that this assumption seems required to motivate the disjunctive framing above, rather than following from the framing above.
   2. The arguments I know of for claims like this do seem to rely on strong claims about the sort of ‘plan search’ algorithms we’d expect future AIs to instantiate. For example, Rob claims that we’re on track to produce systems which approximate ‘randomly sample from the space of simplicity-weighted plans’. See discussion here.
   3. As Paul notes, “there are just a lot of plausible ways to care a little bit (one way or the other!) about a civilization that created you, that you've interacted with, and which was prima facie plausibly an important actor in the world.”
3. By default, the values of future AIs are likely to include broadly-scoped goals, which will involve rapacious influence-seeking.
   1. I agree that there are instrumentally convergent goals, which include some degree of power/influence-seeking. But I don’t think instrumental convergence alone gets you to ‘doom with >50%’. 
   2. It’s not enough to have a moderate desire for influence. I think it’s plausible that the default path involves systems who do ‘normal-ish human activities’ in pursuit of more local goals. I quote a story from Katja Grace in my shortform here.

So far, I’ve discussed just one disjunct, but I can imagine outlining similar assumptions for the other disjuncts. For instance: if we have >20 years to conduct AI alignment research conditional on the problem not being super hard, why can’t there be a decent chance that a not-super-competent research community solves the problem? Again, I find it hard to motivate the case for a claim like that without already assuming a mainline model of doom. 

I’m not saying there aren’t interesting arguments here, but I think that arguments of this type mostly assume a mainline model of doom (or the adequacy of a ‘disjunctive framing’), rather than providing independent arguments for a mainline model of doom. 

Future Responses  

This blog is ~1k words. Can you write a similar length blog for the other side, rebutting all my points?

I think so! But I’m unclear what, exactly, your arguments are meant to be. Also, I would personally find it much easier to engage with arguments in premise-conclusion format. Otherwise, I feel like I have to spend a lot of work trying to understand the logical structure of your argument, which requires a decent chunk of time-investment. 

Still, I’m happy to chat over DM if you think that discussing this further would be profitable. Here’s my attempt to summarize your current view of things.  

We’re on a doomed path, and I’d like to see arguments which could allow me to justifiably believe that there are paths which will steer us away from the default attractor state of doom. The technical problem of alignment has many component pieces, and it seems like failure to solve any one of the many component pieces is likely sufficient for doom. Moreover, the problems for each piece of the alignment puzzle look ~independent.

1. ^{^}

   Suggestions for better argument names are not being taken at this time.

Here's a quick attempt at a subset of conjunctive assumptions in Nate's framing:

- The functional ceiling for AGI is sufficiently above the current level of human civilization to eliminate it
- There is a sharp cutoff between non-AGI AI and AGI, such that early kind-of-AGI doesn't send up enough warning signals to cause a drastic change in trajectory.
- Early AGIs don't result in a multi-polar world where superhuman-but-not-godlike agents can't actually quickly and recursively self-improve, in part because none of them wants any of the others to take over - and without being able to grow stronger, humanity remains a viable player.

Yes, I completely agree that this is nowhere near good enough. It would make me very nervous indeed to end up in that situation.

The thing I was trying to push back against was the idea that what I thought you were claiming: that we're effectively dead if we end up in this situation.

It doesn't seem to me that titotal is assuming MI is solved; having direct access to the brain doesn't give you full insight into someone's thoughts either, because neuroscience is basically a pile of unsolved problems with growing-but-still-very-incomplete-picture of low-level and high-level details. We don't even have a consensus on how memory is physically implemented.

Nonetheless, if you had a bunch of invasive probes feeding you gigabytes/sec of live data from the brain of the genius general of the opposing army, it would be extremely likely to be useful information.

A really interesting thing is that, at the moment, this appears in practice to be a very-asymmetrical advantage. The high-level reasoning processes that GPT-4 implements don't seem to be able to introspect about fine-grained details, like "how many tokens are in a given string". The information is obviously and straightforwardly part of the model, but absent external help the model doesn't seem to bridge the gap between low-level implementation details and high-level reasoning abilities - like us.

Regarding the whether they have the same practical implications, I guess I agree that if everyone had a 90% credence in catastrophe, that would be better than them having 50% credence or 10%.

Inasmuch as you're right that the major players have a 10% credence of catastrophe, we should either push to raise it or to advocate for more caution given the stakes.

My worry is that they don't actually have that 10% credence, despite maybe saying they do, and that coming across as more extreme might stop them from listening.

I think you might be right that if we can make the case for 90%, we should make it. But I worry we can't.

Ah I think I see the misunderstanding.

I thought you were invoking "Murphy's Law" as a general principle that should generally be relied upon - I thought you were saying that in general, a security mindset should be used.

But I think you're saying that in the specific case of AGI misalignment, there is a particular reason to apply a security mindset, or to expect Murphy's Law to hold.

Here are three things I think you might be trying to say:

1. As AI systems get more and more powerful, if there are any problems with your technical setup (training procedures, oversight procedures, etc.), then if those AI systems are misaligned, they will be sure to exploit those vulnerabilities.
2. Any training setup that could lead to misaligned AI will lead to misaligned AI. That is, unless your technical setup for creating AI is watertight, then it is highly likely that you end up with misaligned AI.
3. Unless the societal process you use to decide what technical setup gets used to create AGI is watertight, then it's very likely to choose a technical setup that will lead to misaligned AI.

I would agree with 1 - that once you have created sufficiently powerful misaligned AI systems, catastrophe is highly likely.

But I don't understand the reason to think that 2 and especially 3 are both true. That's why I'm not confident in catastrophe: I think it's plausible that we end up using a training method that ends up creating aligned AI even though the way we chose that training method wasn't watertight.

You seem to think that it's very likely that we won't end up with a good enough training setup, but I don't understand why.

Looking forward to your reply!

Interesting perspective, although I'm not sure how much we actually disagree. "Complicated and open", to me reads as "difficult"

Is there a rephrasing of the initial statement you would endorse that makes this clearer? I'd suggest "If you apply a security mindset (Murphy’s Law) to the problem of AI alignment, it should quickly become apparent that we do not currently possess the means to ensure that any given AI is safe."