Rerunning the Time of Perils

This post is an experiment in two ways that I explain in comments here and here. But you don't need to read those comments before reading this post. 

1. Introduction

In What We Owe the Future I argued that even very severe non-extinction catastrophes are unlikely to permanently derail civilisation. Even if a catastrophe killed 99.9% of people (leaving 8 million survivors), I'd put the chance we eventually re-attain today's technological level at well over 95%.

On the usual longtermist picture, this means such catastrophes are extraordinarily bad for present people but don't rival full existential catastrophes in long-run importance. Either we go extinct (or lock into a permanently terrible trajectory), or we navigate the "time of perils" once and reach existential security.

In this article I'll describe a mechanism by which non-extinction global catastrophes would have existential-level importance. I'll define:

• A catastrophic setback as a catastrophe that causes civilisation to revert to the technological level it had at least 100 years prior.
• Sisyphus risk as the extra existential risk society incurs from the possibility of catastrophic setbacks. (Though, unlike with Sisyphus’s plight, we will not suffer such setbacks indefinitely.)

The mechanism is straightforward: if a catastrophic setback occurs after we've already survived the transition to superintelligence, civilisation has to re-run the time of perils—redevelop advanced AI and face the alignment problem all over again. The magnitude of this additional risk could be meaningful: if AI takeover risk is 10% and the chance of a post-superintelligence catastrophic setback is 10%, Sisyphus risk adds about an extra percentage point of existential risk.

The structure of this article is as follows. In section 2 I present a simple model of Sisyphus risk, and in section 3 I describe potential avenues for post-superintelligence catastrophic setbacks. Section 4 asks whether a post-setback society would retain alignment knowledge, and section 5 addresses objections. Section 6 extends the simple model to varying rerun risk and multiple cycles. Section 7 presents what I see as the most important upshots:

• The magnitude of existential risk from engineered pandemics is meaningfully higher than you might otherwise think.
• The magnitude of existential risk from nuclear war and other non-AI non-bio sources is much higher than you might otherwise think.
• Unipolar post-superintelligence scenarios seem more desirable than they otherwise would.

• The value of saving philanthropic resources to deploy post-superintelligence is greater than it otherwise would be.

   

2. A Simple Model of Sisyphus Risk

2.1 One catastrophic setback

Start with a deliberately stripped-down picture:

• The first run is the period from now until we reach robust existential security (say, stable aligned superintelligence plus reasonably good global governance).
• During this run there is some total probability of existential catastrophe, p.
• Conditional on avoiding existential catastrophe, there is a probability q of a post-AGI catastrophic setback that knocks us back technologically by a hundred years or more.
• After such a setback, civilisation eventually rebounds and goes through a second run with existential catastrophe probability p₂.

In this stripped-down picture, we'll ignore multiple setbacks and suppose p₂ = p. Then existential catastrophe can happen in the first run (probability p), or in the second run—which requires avoiding catastrophe in the first run (1−p), suffering a setback (q), then catastrophe in the rerun (p).  So:

P(existential catastrophe) = p + (1−p) × q × p

For smallish probabilities, (1−p) ≈ 1, so:

P(existential catastrophe) ≈ p(1+q)

Whatever your first-run risk p, a probability q of catastrophic setback multiplies it by about (1+q). The extra existential risk from Sisyphus risk is approximately qp.

2.2 Ord's numbers

In The Precipice, Toby Ord gives rough 100-year existential risk estimates:

• Artificial intelligence: ~10% (1 in 10)
• Engineered pandemics: ~3.3% (1 in 30)
• Nuclear war: ~0.1% (1 in 1,000)
• Extreme climate change: ~0.1% (1 in 1,000)
• Natural risks (asteroids, supervolcanoes, etc.): ~0.01% total

Overall, he puts this century's existential risk at about 1 in 6 (~17%).

Treat that 17% as first-run risk p. And suppose that conditional on surviving the AGI transition, there's a 10% chance (q) of some post-AGI catastrophic setback.

Sisyphus risk ≈ 0.83 × 0.1 × 0.17 ≈ 1.4%. Total risk rises to roughly 18.4%—about a 10% relative increase over the "1 in 6" figure.

For specific causes, Sisyphus channels can dominate direct extinction probabilities. Ord puts direct nuclear-war extinction risk at ~0.1%. But suppose there's a 5% chance of a post-AGI nuclear war causing a catastrophic setback, and the rerun's existential risk is ~10%. Then nuclear war's indirect contribution via Sisyphus risk is approximately 0.05 × 0.10 = 0.5%. Total nuclear-related existential risk becomes ~0.6%—six times the direct figure.

3. What Forms Could Post-AGI Catastrophic Setbacks Take?

Many familiar global catastrophic risks create post-AGI setback risk.

3.1 Engineered pandemics

Engineered pandemics are widely regarded as among the most plausible routes to civilisational collapse, especially once advanced AI can help design pathogens.

The chance of human extinction from engineered pandemics this century might be around a few percent. But the chance of a catastrophic pandemic killing a very large fraction of the population without killing everyone is substantially higher. Isolated populations—uncontacted tribes, remote islanders, Antarctic researchers, submarine crews, people with rare genetic resistance—are extremely hard to reach with any pathogen. So even for worst-case engineered pandemics, the modal outcome is vast death and societal collapse, but not literal extinction.

Such catastrophes could well occur after superintelligence: AI could make the ability to create such pandemics widespread, while it might take considerable time to adequately protect society against them.

3.2 Nuclear war and nuclear winter

A full-scale nuclear war—especially in a future with larger arsenals or more nuclear states—could kill billions directly and through nuclear-winter-induced famines. Again, complete extinction is unlikely: some regions, especially in the Southern Hemisphere (New Zealand, parts of South America, some islands), would likely suffer less severe climatic impacts and avoid direct strikes. But the chance of a nuclear war that kills ≳90% of people and collapses civilisation is much higher—especially if an AI-driven arms race leads to massively expanded global stockpiles before war breaks out.

3.3 AI-driven catastrophe

Some AI-involved catastrophes could collapse civilisation without causing extinction. For example, there could be a failed AI takeover, where a powerful coalition of AIs attempt to seize control, humans (and/or aligned AIs) thwart it, but in the process unleash massive destruction—nuclear exchanges, catastrophic cyber-attacks on critical infrastructure, or near-complete destruction of the electrical grid.

3.4 Butlerian backlash and fertility decline

One more speculative possibility involves a Butlerian backlash against AI combined with technological regression from fertility decline. Imagine:

• Humanity navigates the first AGI transition without extinction, but with significant scares and close calls.
• As a result, a broad anti-tech, anti-AI movement gains momentum—driven by religious or ideological currents and fear of loss of control—and results in a complete ban on advanced AI.
• Simultaneously, steep global fertility decline continues.
• Over centuries, technological suppression and demographic decline gradually unwind complex civilisation: key industries shut down, knowledge institutions wither, and the capacity to maintain sophisticated systems is lost.

This is a slow drift into technological regression rather than a single violent shock. But from the perspective of the far future, the effect is similar: by 2400, the world population might be a few hundred million living in low-tech polities, no longer able to sustain advanced semiconductor manufacturing. Eventually the population rebounds—and must face the development of AGI all over again.

4. Would a Post-Setback Society Retain Alignment Knowledge?

For Sisyphus risk, it matters greatly what we take into the rerun (H/T Tom Davidson for this objection). Two extremes:

• Optimistic: we carry forward a reasonably complete picture of how to align AGI (perhaps even copies of aligned systems). The second run is still dangerous but much easier.
• Pessimistic: almost all that knowledge is lost—both weights and code, and the tacit expertise—so the rerun faces similar alignment difficulties on a poorer, more depleted planet.

I think the default is closer to the pessimistic end unless we take targeted action.

4.1 Digital fragility

Most alignment work today exists as digital bits: arXiv papers, lab notes, GitHub repos, model checkpoints. Digital storage is surprisingly fragile without continuous power and maintenance.

SSDs store bits as charges in floating-gate cells; when unpowered, charge leaks, and consumer SSDs may start losing data after a few years. Hard drives retain magnetic data longer, but their mechanical parts degrade; after decades of disuse they often need clean-room work to spin up safely. Data centres depend on air-conditioning, fire suppression, and regular maintenance.

In a global collapse where grids are down for years, almost all unmaintained digital archives eventually succumb to bit-rot, corrosion, fire, or physical decay.

Some attempts at very long-term archiving exist—the GitHub Arctic Code Vault stored repository snapshots on archival film intended to last 500–1,000 years, and experimental technologies like quartz-glass "5D" storage and DNA-based media show promise. But these require fairly sophisticated equipment to read. A collapsed civilisation is unlikely to have polarisation microscopes and ML-based decoders at hand.

Unless we make a special effort to record alignment ideas on robust analogue media (discussed later), it's very unlikely a post-collapse society will resurrect our exact algorithms or trained models, or even broad alignment techniques. 

4.2 Hardware and software compatibility

Even if some digital archives survive, there's another problem: hardware and software stacks. Running a 21st-century AGI or reading its weights requires suitable hardware still working, compatible drivers and operating systems, and a chain of compilers, libraries, and containerisation tools.

Modern microelectronics don't age gracefully. Capacitors dry out, solder joints crack, chips suffer long-term degradation. Many high-end systems are locked behind licensing that requires phoning home to servers. If grids are down for decades and no one maintains server rooms, survivors will likely find, by the time they can run a data centre again, a collection of dead, unbootable hardware.

The chance a future civilisation can simply turn the old aligned AGI back on without reinventing much of the semiconductor and computing stack looks very small.

4.3 Tacit knowledge

Alignment isn't just code; it's a body of tacit knowledge: which training tricks worked in practice, how to interpret confusing safety-relevant behaviour, what failed and why. This is held by a small number of researchers and engineers. A collapse killing 90–99% of people would almost certainly kill most alignment researchers and scatter the rest into survival mode.

Luisa Rodriguez's work on collapse suggests that many kinds of practical expertise (small-scale farming, improvising power) survive reasonably well—people are inventive, and some skills exist in large numbers. But alignment research is exactly the opposite: tiny communities at the frontier of abstract theory and empirical ML.

Absent deliberate efforts to create "alignment textbooks for the dark age," the tacit knowledge probably won't make it through.

5. Won't AGI make post-AGI catastrophes essentially irrelevant?

A natural thought: once we have aligned superintelligence, surely it will either prevent pandemics, wars, and other disasters altogether, or kill or disempower us directly (making other risks moot). On this view, a post-AGI world is nearly binary—utopia or extinction—leaving little room for Sisyphean scenarios.

But I think this is too optimistic about the speed and completeness of the transition to globally deployed, robustly aligned "guardian" systems. There are many plausible worlds where AI capabilities are at or beyond human level in many domains, multiple actors control superintelligence, global coordination remains shaky, and deployment is messy and entangled with geopolitical rivalries.

In those worlds, AI may reduce some risks (better pandemic surveillance) while simultaneously increasing others (cheaper bioweapons, more destabilising autonomous weapons, faster escalation cycles). 

An aligned AGI trusted by everyone to override national sovereignty at will, and itself indefinitely stable, is one possible endpoint. It's not guaranteed we reach it quickly; the first few decades post-AGI could still contain plenty of room for very large mistakes.

6. Implications and Strategic Upshots

6.1 The importance of non-AI risks, especially non-AI non-bio

Ord gives ~3.3% existential risk from engineered pandemics and ~0.1% each from nuclear war and extreme climate change. But, as I understand them, those estimates focus mainly on direct extinction-like outcomes. Adding Sisyphean channels increases the long-run importance of some risks:

• Sisyphean risk could easily add a percentage point of existential risk to biorisk (if the chance of a catastrophic setback from a pandemic is 10% and second-run existential risk is also 10%). This increases total existential risk from pandemics by a meaningful amount.
• Nuclear war's significance rises even more sharply: from 0.1% direct existential risk to perhaps ~0.5–1% once we factor in its role as a catastrophic-setback trigger.

6.2 When to donate

Sisyphus risk has implications for how longtermist philanthropists should allocate resources over time. Standard longtermist thinking often treats the AGI transition as the critical period—survive it, and the problem is largely solved. If that's right, the case for spending now rather than later is very strong.

But Sisyphus risk complicates this picture. If substantial catastrophic-setback risk persists after AGI—from pandemics, wars, or other sources—then the post-AGI world still has important risk-reduction work to do. Resources that can be stored safely and deployed later would have somewhere valuable to go.

Sisyphus risk makes it more reasonable to devote a larger fraction of longtermist resources to patient strategies—endowments, value-preserving institutions, or other structures designed to retain influence in the post-AGI world and help guard against collapse.

6.3 A modest argument for more unipolar futures

Sisyphus risk also bears on unipolar versus multipolar futures. In a strongly multipolar world with several powerful states and AGI systems, there are more pathways to post-AGI catastrophic setbacks: AI-augmented great-power wars, unstable deterrence, arms races over bioweapons or dangerous tech.

In a more unipolar world—where a single broadly trusted coalition controls the leading AI system and a large share of coercive power—there may be fewer actors able to start truly global wars, more scope for coordinated biosecurity and nuclear risk reduction, and a clearer path to retiring high-risk technologies once no longer needed.

A malevolent or incompetent unipolar power is terrifying and could lock in a terrible trajectory. But Sisyphus risk provides an extra argument on the unipolar side: fewer centres of independent catastrophic capability means fewer opportunities for post-AGI catastrophic setbacks—hence lower q.

This doesn't settle governance questions by itself—unipolarity comes with serious lock-in and abuse-of-power concerns. But Sisyphus risk is one more consideration tilting the trade-off slightly away from highly multipolar futures.

6.4 The value of knowledge preservation and civilisational kernels

Given how fragile alignment knowledge and infrastructure are, Sisyphus risk increases the appeal of:

• Resilient knowledge repositories: printing and archiving key alignment insights and general scientific knowledge on robust media in multiple locations. For example, paper, microfilm, and etched metal have very long lifetimes and are human-readable with simple tools. Microfilm and acid-free paper can last centuries under decent conditions.
• Possibly even preserving copies of aligned systems in robust forms—though technically this would be tricky.
• Civilisational refuges: well-resourced, physically secure sites designed to ride out extreme global catastrophes while maintaining advanced capability and knowledge.

These interventions don't remove Sisyphus risk, but they can shorten and soften the rerun.

7. Conclusion

We often think of the "time of perils" as something we either navigate once or fail once. In fact, a post-AGI world still has many routes to non-extinction global catastrophes that could set civilisation back by a century or more. In those worlds, our descendants must rerun the dangerous trajectory—re-industrialise, rebuild powerful AI, face alignment problems again—typically without our full technological base or alignment knowledge.

I call this extra exposure to existential danger Sisyphus risk. For reasonable parameter choices, it’s not a tiny correction; it can add one or several percentage points to lifetime existential risk and significantly increase the importance of non-AI risks like biorisk, nuclear war, or other causes of post-AGI catastrophe.

Strategically, this gives us additional reason to care about non-AI risks; a modest push towards patient philanthropy; extra motivation for knowledge-preservation and civilisational resilience projects; and a small but real argument in favour of more unified global governance over a dangerously multipolar AGI landscape.

Appendix: Extensions

A.1 Multiple cycles

So far we've considered just one potential setback. But what if setbacks can happen repeatedly—each time sending civilisation back to face the dangerous period again?

Let's model this more explicitly. Suppose each "run" through the time of perils has three possible outcomes:

• Existential catastrophe, with probability p.
• Catastrophic setback, with probability q conditional on not going extinct.
• Safe exit to existential security, with the remaining conditional probability 1−q.

So the unconditional probabilities per run are:

• Extinction: p.
• Setback: (1−p)q.
• Safe exit: (1−p)(1−q).

Let E be the eventual probability of extinction, starting from a fresh run. On any given run:

• With probability p, we go extinct immediately.
• With probability (1−p)(1−q), we safely exit to existential security.
• With probability (1−p)q, we suffer a setback and are effectively back where we started, facing the same eventual extinction probability E again.

So we can write the simple recursion:

E = p + (1−p)q × E

Rearranging:

E = p / (1 − (1−p)q)

In other words, the possibility of multiple setbacks multiplies the first-run risk p by the factor 1 / (1 − (1−p)q).

For small p, this is very close to 1/(1−q). If q = 0.5 and p is modest, then E ≈ 2p: multiple runs roughly double eventual extinction risk. If q = 0.33, the multiplier is about 1.5; and so on. More setbacks means more runs, and more runs mean more opportunities for extinction.

We can describe the expected number of runs in the same way. Let N be the expected number of times civilisation passes through the time of perils. On the first attempt we definitely get one run; with probability (1−p)q we suffer a setback and expect another N runs:

N = 1 + (1−p)q × N, which gives N = 1 / (1 − (1−p)q)

For small p, this again is approximately 1/(1−q). If, conditional on not going extinct, there is a 50% chance of a catastrophic setback, we expect about two runs; if it's 33%, about 1.5 runs; and so on.

This multiple-run model helps us see when Sisyphus risk actually matters for our decisions:

If q is very small, the multiplier 1/(1 − (1−p)q) is extremely close to 1. Even if we conceptually allow infinitely many potential reruns, the expected number of reruns is so small that eventual extinction probability barely changes: E ≈ p.

If q is very large—close to 1—then civilisation is very likely to collapse and retry many times. In the limit as q → 1, eventual extinction probability E approaches 1 even for modest p: we almost surely keep rerunning the time of perils until one run finally kills us. Sisyphus dynamics are "decisive" here, but in a grim way: unless we can dramatically change p or q, eventual failure becomes close to inevitable.

The interesting cases lie in between, when p is neither negligible nor overwhelming—we're neither "almost certainly safe" nor "almost certainly doomed" on a single run—and q is large enough that multiple attempts are a real possibility, but not so large that extinction is virtually guaranteed.

In that intermediate regime, it's natural to ask: when is it more important to reduce catastrophic setback risk q, rather than first-run existential risk p?

From the formula E = p / (1 − (1−p)q), a bit of algebra shows:

• For a small absolute reduction in p, the resulting reduction in E is proportional to (1−q).
• For a small absolute reduction in q, the resulting reduction in E is proportional to p(1−p).

So, for equal small changes Δp and Δq, reducing q has a bigger impact on eventual extinction probability only if p(1−p) > 1−q—i.e. only if q is already extremely close to 1. For example:

• If p = 0.5, you'd need q > 0.75.
• If p = 0.2 or 0.8, you'd need q > 0.84.
• If p = 0.1 or 0.9, you'd need q > 0.91.

For more modest values of q (say, 10–50%), a unit reduction in p does more to reduce eventual extinction probability than a unit reduction in q. Intuitively, this is because p matters on every run, while q only matters insofar as it creates extra runs. But, unintuitively, as your estimate of first-run existential risk increases from, say, 0.1 to 0.5, you start to care more about reducing q compared to reducing p (by the same small absolute amount).

We can push this comparison a bit further. The ratio between the marginal impact of reducing q and the marginal impact of reducing p (for the same small absolute change in each) is:

(∂E/∂q) / (∂E/∂p) = p(1−p) / (1−q)

Given this, we can ask when reducing q has at least one-tenth as much impact as reducing p. The condition is p(1−p)/(1−q) ≥ 1/10, which implies q ≥ 1 − 10p(1−p).

For p = 5% this gives q ≳ 0.53; for p = 10% it's q ≳ 0.10; and for p above about 11%, the right-hand side becomes negative, which means that any positive q makes reductions in q at least one-tenth as valuable on the margin as equal-sized reductions in p. 

For a more general overview, here’s a table of how many times greater the value of a one‑percentage‑point reduction in 𝑝 is compared to a one‑percentage‑point reduction in 𝑞, depending on different assumptions about p and q.

Even if work on catastrophic setbacks is rarely as directly impactful (per percentage point) as work on first-run existential risk, it can still matter a lot: as soon as first-run risk is non-trivial and setbacks are more than very rare, shaving q down becomes far from a rounding error in the overall picture of existential risk.

A.2 Higher or lower risk in the rerun

So far I've assumed the rerun's existential risk (p₂) equals first-run risk (p₁). But this need not be true. If not, then the Sisyphean contribution becomes (1−p₁) × q × p₂, structurally the same but sensitive to p₂. 

One can make arguments that p₂ would be greater than p₁ and vice-versa:

• Rerun risk could be higher if post-collapse governance is worse, the environment and resource base is extremely depleted (making re-industrialisation harder), or survivors inherit dangerous technologies without the surrounding safety culture.
• Rerun risk could be lower if the catastrophe serves as a strong warning shot, survivors build much more cautious institutions, or knowledge about what went wrong and how to avoid catastrophe survives.

If later runs are riskier than the first, Sisyphus considerations become more important. If later runs are safer than the first, then Sisyphus effects are muted.  So, a crucial question is whether we think that society is doing unusually well or poorly at handling AI takeover risk, and so whether to think that regression to the mean is a reason for thinking that rerun risk is greater or lesser than first-run risk. 

My personal view is that society is on track to do unusually well (though bear in mind that this is a low bar), which makes the importance of Sisyphus risk even greater, as we should expect rerun risk to be higher than first-run risk. To illustrate: if we face AI existential risk of 5%, but the rerun risk would be 20%, and the chance of post-AGI catastrophe is 10%, then Sisyphus risk is 2%: 40% as large as AI risk itself. 

A.3 Trajectory change

So far I've focused on how catastrophic setbacks change the probability of existential catastrophe. But they also matter enormously for the quality of the future conditional on no existential catastrophe.

In other work I've suggested decomposing an action's long-term impact into its existential impact (the part that comes from changing the probability of existential catastrophe) and its trajectory impact (the part that comes from changing the value of the world conditional on no existential catastrophe).

Catastrophic setbacks affect both. On the existential side, they add extra rounds of existential risk. On the trajectory side, they plausibly change expected value substantially, because the civilisation that eventually emerges from collapse could be very different from what would have emerged otherwise.

I suspect that for many plausible parameter values, the trajectory impact of avoiding catastrophic setbacks is at least as large as—and perhaps larger than—their existential impact. But it’s unclear to me in what direction this consideration points. 

Perhaps current civilisation is unusually good (in particular because it’s unusually liberal and democratic), and if so then by regression to the mean the society we would get post-catastrophe would be worse (H/T Fin Moorhouse for this worry). And consider that, conditional on a post-AGI catastrophic setback occurring, it’s more likely that the post-AGI society is liberal rather than authoritarian, so that post-AGI society is unusually good. If so, then the trajectory impact of preventing a post-AGI  catastrophic setback is positive, too. This would increase the importance of catastrophic setbacks — perhaps considerably.