earlier this year I was a part of the latest cohort at AI safety camp (AISC). the project I had joined — Investigating assumptions of the doom debate — had a goal to build a probabilistic assumption tree for AI doom arguments.
several members made versions of the tree, but the one that we collectively worked on, started from a root node “Dangerous AI occurs.” one branch — “model capacity”broke down into a dozen subnodes, from inference and reasoning capability, to model optimization. the other branch —“external environment” — was less developed, with only two nodes: “conducive human policy,” and “sustained capacity for development.” now, both of these could reasonably be argued as already certain, making assigning probabilities to them all but meaningless. nonetheless, this was the branch I took on that sent me down a divergent path from the rest of the team: why was the environment being treated as an afterthought?
it reflected something at the core of the canonical AI doom argument: it is fundamentally grounded around the agent — what it can do, what it wants, whether it could become too powerful to control. but the environment, in which harm actually occurs, barely appears, treated as a passive backdrop, instead of as an active component of the risk.
my earliest version of the tree started from first principles: what are the necessary conditions for AI to cause existential harm?
I landed on three:
these three conditions would eventually survive dozens of iterations complementing a complete paradigm shift in how I thought about P(doom).
I dug into the canonical AI safety concepts: misalignment, instrumental convergence, orthogonality, goal specification, capability thresholds. mapping this to a tree quickly became a convoluted mess. every decomposition led to questions about the definition of the terms we were using—what counts as a goal? does indifference count as misalignment? which capabilities does a system need to be considered dangerous?
our trees were sprawling with over a dozen nodes full of these conceptual claims rather than verifiable ones. it hit me: the mapping wasn’t at an impasse because the questions were hard, but because we hadn’t even agreed on what we were talking about.
taking a step back, I assessed what kind of claim each node was making, and categorized them:
the canonical doom argument relies heavily on definitional claims treated as empirical ones. but probability estimates on definitional claims are just intuitions cosplaying in mathematical structure. we cannot build a meaningful probability tree until we’ve resolved what terms mean. the definitions are a prerequisite.
the root node of the canonical tree says ‘dangerous AI occurs,’ but dangerous how, to what? unpacking it revealed that the terms are hierarchically nested: AI optimizes, optimization affects something, that something is human welfare. ‘dangerous’ becomes ‘harmful’ once you specify what it’s harmful to.
“human welfare” encompasses three conditions that we want to preserve or improve: human life, the environment that sustains human life, and what human life has materially built (infrastructure, civilization). these are the targets of optimization, and the things through which harm, at doom scale, is actualized.
“harmful” is then defined relative to humans’ ability to stop or reverse adverse effects on human welfare.
instead of being a property of the AI, it’s a relational property between the optimization and the corrective capacity of the environment it’s operating in.
so under these definitions, harmful AI already exists — recommendation algorithms erode the social contract. autonomous trading bots create financial fragility. finite resource extraction is scaling to civilization levels for AI infrastructure.
the doom debate is often framed as some future risk from AGI or superintelligence, but the risk from AI is already here, which is why we need to talk about trajectory.
I decided to ask a different question.
the canonical approach asks: what properties must AI have to be dangerous? this got us stuck in debates about capability thresholds, goal specification, alignment definitions.
what if instead we start from: what makes an environment vulnerable to catastrophic harm, and how does AI push toward those conditions?
AI dropped from the top of the tree to the bottom. it went from being an assumed dangerous agent to being one mechanism among many that could push environmental conditions toward doom.
danger is not a property of the AI, but rather a property of the configuration between optimization and environment.
the tree became pathway agnostic—it didn’t care whether harm came from a misaligned superintelligence, a misused system, or the compound effects of thousands of deployed bots with complex interactions.
the canonical argument treats the external world as a passive recipient of the agent’s actions. the environment is where harm happens, but it’s altogether missing from the formalization. ignoring it means ignoring most of the actual risk surface, which is in itself dynamic and antifragile.
E — environment, internally coupled such that changes propagate to K
K — dangerous conditions, a subset of E
K₀ — current state of K at time t
K+ — K worsening
t — point in time at which K and R₀ are sampled
R₀ — rate of change of K at t (current trajectory)
Rc — rate at which corrective capacity can negate K+
O — optimization, preference ordering of actions to change E
M — K is mutable (accessible, changeable)
C — O sufficient to mutate K (directly or through E)
DAI = AI where K+ | O > R₀ — optimization accelerates worsening beyond current trajectory
Lock in = K+ > Rc — worsening outpaces corrective capacity
Doom = DAI ∧ Lock in
deliberate design decisions:
the formalization initially included θ, an irrecoverability threshold. doom occurred when K crossed θ, but any threshold is debatable.
to solve this I reframed P(doom) as a trajectory you can’t get off.
at any snapshot in time, dangerous conditions have a state and a rate of change. AI is dangerous if it accelerates that rate. lock-in is when corrective capacity can’t keep up. the question becomes purely about whether or not the worsening is faster than the correction (and if that gap is widening). we’re looking at the probability of a condition instead of an event.
P(Doom) = P(M) × P(C) × P(Lock in)
Doom — D = DAI ∧ Lock in
├── AI increases P(Doom) — DAI = K+ | O > R₀
│ ├── Dangerous conditions are changeable — M
│ └── Optimization sufficient to change them — C
└── Lock in — K+ > Rc
this formalization was implemented as an interactive tool with two modes:
the tool can be found here at: https://kvncaldwll.github.io/AISC-P-Doom-Tool/
my first stab at mapping to the tree was the three conditions: sufficiently capable, cannot be stopped, cannot recover from the harm. dozens of iterations later, those three conditions survived, and became more precisely defined. “sufficiently capable” became M ∧ C — dangerous conditions are changeable and optimization is sufficient to change them. “cannot be stopped” and “cannot recover” merged into lock-in — the rate of worsening outpaces corrective capacity.
everything, from the definitions audit, the environment-first inversion, the dangerous configuration framing, the trajectory reframe — was the process of making those three intuitions rigorous.
developed during AISC, cohort 11, Project 19 - Investigating assumptions of the doom debate. the environment-first framework, formalization, notation, and tool are independent contributions. the project context and team conversations informed the development.