Is the threat model for stages one and two the same as the one in my post on whether one country can outgrow the rest of the world?

That post just looks at the basic economic forces driving super-exponential growth and argues that they would lead an actor that starts with more than half of resources, and doesn't trade, to gain a larger and larger fraction of world output. 

I can't tell from this post whether there are dynamics that are specific to settling the solar system in Stage II that feed into the first-mover advantage or whether it's just simple super-exponential growth. 

My current guess is that there are so many orders of magnitude for growth on Earth that super-exp growth would lead to a decisive strategic advantage without even going to space. If that's right (which it might not be), then it's unclear that stage two adds that much. 

Perhaps many minds end up at a shared notion of what they’re aiming for, via acausal trade (getting to some grand bargain), or evidential cooperation in large worlds

This isn't convergence. It's where ppl DIDN"T converge but make a win-win compromise deal. If ppl all converge, there's no role for acausal trade/ECL 

It seems like we can predictably make moral progress by reflecting; i.e. coming to answers + arguments that would be persuasive to our former selves
I think I’m more likely to update towards the positions of smart people who’ve thought long and hard about a topic than the converse (and this is more true the smarter they are, the more they’re already aware of all the considerations I know, and the more they’ve thought about it)
If I imagine handing people the keys to the universe, I mostly want to know “will they put serious effort into working out what the right thing is and then doing it” rather than their current moral views

I guess i feel this shows that there's some shared object-level moral assumptions and some shared methodology for making progress among humans. 

But doesn't show that the overlap is strong enough for convergence. 

And you could explain the overlap by conformism, in which case i wouldn't agree with societies i haven't interacted with

I think my biggest uncertainty about this is:

If there were a catastrophic setback of this kind, and civilisation tried hard to save and maintain the weights of superintelligent AI (which they presumably would), how likely are they to succeed? 

My hunch is that they very likely could succeed. E.g. in the first cpl of decades they'd have continued access to superintelligent AI advice (and maybe robotics) from pre-existing hardware. They could use that to bootstrap to longer periods of time. E.g. saving the weights on hard drives rather than SSDs, and then later transferring them to a more secure, long lasting format. Then figure out the minimal-effort-version of compute maintenance and/or production needed to keep running some superintelligences indefinitely

Really like this post!

I'm wondering whether human-level AI and robotics will significantly decrease civilisation's susceptibility to catastrophic setbacks?

AI systems and robots can't be destroyed by pandemics. They don't depend on agriculture -- just mining and some form of energy production. And a very small number of systems could hold tacit expertise for ~all domains. 

Seems like this this might reduce the risk by a lot, such that the 10% numbers you're quoting are too high. E.g. you're assigning 10% to a bio-driven set-back. But i'd have thought that would have to happen before we get human-level robotics?

I also work at Forethought!

I agree with a lot of this, but wanted to flag that I would be very excited for ppl doing blue skies research to apply and want Forethought to be a place that's good for that. We want to work on high impact research and understand that sometimes mean doing things where it's unclear up front if it will bear fruit.

Thanks, good Q.

I'm saying that if there is such a new paradigm then we could have >10 years worth of AI progress at rates of 2020-5, and >10 OOMs of effective compute growth according to the old paradigm. But that, perhaps, within the new paradigm these gains are achieved while the efficiency of AI algs only increases slightly. E.g. a new paradigm where each doubling of compute increases capabilities as much as 1000X does today. Then, measured within the new paradigm, the algorithmic progress might seem like just a couple of OOMs so 'effective compute' isn't rising fast, but relative to the old paradigm progress (and effective compute growth) is massive.

Fwiw, this X thread discusses the units I'm using for "year of Ai progress", and Eli Lifland gives a reasonable alternative. Either work as a way to understand the framework.

Can you give more examples of where ppl are getting this wrong?

I support the 80k pivot, and the blue dot page seems ok (but yes, I'd maybe prefer smg more opinionated).

While these concerns make sense in theory I'm not sure whether it's a problem in practice

Nice!

I think that condition is equivalent to saying that A_cog explodes iff either

• phi_cog + lambda > 1 and phi_exp + lambda > 1, or
• phi_cog > 1

Where the second possibility is the unrealistic one where it could explode with just human input

Agree that i wouldn't particularly expect the efficiency curves to be the same. 

But if the phi>0 for both types of efficiency, then I think this argument will still go through.

To put it in math, there would be two types of AI software technology, one for experimental efficiency and one for cognitive labour efficiency: A_exp and A_cog. The equations are then:

dA_exp = A_exp^phi_exp F(A_exp K_res, A_cog K_inf)

dA_cog = A_cog^phi_cog F(A_exp K_res, A_cog K_inf)

And then I think you'll find that, even with sigma < 1, it explodes when phi_exp>0 and phi_cog>0.