Cluelessness: Summary of the argument, why it matters, and counterarguments

I’d like to elicit direct, productive critiques of the argument for cluelessness from my sequence on “unawareness”, which I’ll call the unawareness argument.

To that end, this post will:

• break down the unawareness argument at a high level;
• explain why the EA community should care about this argument; and
• summarize the angles for critiquing the argument that I expect to be most productive, and how the sequence responds to existing critiques.

Argument breakdown

Here’s a new framing of the unawareness argument (compared to how I present it in the sequence). I expect this framing to help readers disentangle different types of disagreements they might have, corresponding to three different premises of the argument.

Roughly: What would justify preferring action A over B on impartial altruistic grounds? We’d need to “expect” that according to our epistemically idealized self, A has better expected total consequences across the cosmos (normative premise). But if our understanding of these actions’ consequences is too coarse, then we can’t say how our idealized self would compare their expected values (conceptual premise). And our understanding of any given action’s cosmos-wide consequences is in fact that coarse (empirical premise). So there’s no impartial altruistic justification for preferring any action over another.

More precisely:

• Let’s say that we c-prefer^[1] A over B if the reason we prefer A is an impartial altruistic comparison of the actions’ possible consequences.
• P1. Normative premise: To justify c-preferring A over B, it’s not enough to say (e.g.) that A seems heuristically good. Rather, we need to argue that A has higher “expected value” broadly speaking, meaning: In some sense we “expect” that, if we were idealized agents who could aggregate all of A’s and B’s possible consequences into literal EVs, then we’d say A has higher EV. (We ourselves don’t need literal EVs to justify c-preferences, hence the scare quotes.^[2]) Otherwise, it’s unacceptably arbitrary to c-prefer A.
• P2. Conceptual premise: If our understanding of A’s and B’s possible consequences is sufficiently coarse-grained, then we don’t have an argument for “expecting” our idealized self’s EV for A to be higher, lower, or equal to B’s.^[3] So A’s and B’s “EVs” are incomparable. In particular:
  • a. Against precise EVs: We shouldn’t represent actions’ degrees of c-preferability with literal precise expected values.
  • b. Against “best guesses”: Even if we don’t use literal precise EVs, we shouldn’t always force ourselves to compare A’s and B’s “EVs”.
• P3. Empirical^[4] premise: Due to unawareness (at least), our understanding of any pair of actions’ possible consequences is indeed very coarse-grained — enough that the conclusion of (P2) follows (i.e., these actions’ “EVs” are incomparable). In particular, the actions’ “EVs” are too severely imprecise to compare them, regardless of whether we (a) formally model these “EVs” or (b) appeal to informal/heuristic arguments.
• Conclusion: We aren’t justified in c-preferring any action over any other.

(See Appendix for how each section of the sequence maps onto this form of the argument.)

Why cluelessness matters

Here’s a natural reaction to this argument: “If we’re clueless, nothing we do matters anyway. So there’s no point looking into the argument, and we should just act as if we aren’t clueless.”

But I don’t think it’s that simple. I’ll explain why, then say what kinds of “looking into the argument” I expect to have the highest value of information.

First, the unawareness argument doesn’t imply that “nothing we do matters” all things considered. It only implies that impartial altruism, or any very far-reaching value system, isn’t action-guiding. Other values and moral norms still matter to us, for example, rules like avoiding dishonesty or virtues like compassion. These can be action-guiding even if we’re clueless about total consequences.

Second, if you think the argument goes wrong somewhere, it makes a difference where it goes wrong:^[5]

• Do you reject the normative premise, because you think it counts as an impartial altruistic justification if we say “This action has good ‘expected’ consequences after bracketing the consequences we’re clueless about”? Then arguably you should prioritize neartermist causes.
• Do you accept the normative premise but reject the conceptual one, implying that we should form “best guesses” about the balance of all cosmos-wide consequences? Then you should look for interventions that are best after accounting for as many “galaxy-brained” considerations as possible, rather than simply ignore those considerations. (It’s been argued that mainstream x-risk reduction meets that bar — e.g., Shulman; Adelstein; Carlsmith — but I think this should be spelled out a lot more carefully.)
• Do you agree with the normative and conceptual premises, but think some cautious or “meta” interventions are justified without arbitrary calls about the considerations we’re unaware of? Like, say, saving resources until we’re in a clearer epistemic situation? Then you should do those interventions, rather than various other popular interventions whose justification does rely on arbitrary calls.

Third, relatedly, unawareness probably has some implications for impartial altruists, even if we don’t think it makes us clueless. There’s older work on crucial considerations, but I don’t think such work has rigorously fleshed out the implications much, relative to the scale of the problem. As an example of such an implication: Suppose we want to make forecasts about post-ASI civilization, but we’re worried that forecasting methods that worked well in better understood systems won’t generalize to this case. We could study various methods in domains with (e.g.) different frequencies of past crucial consideration discovery, and see which methods are relatively robust as the frequency of crucial considerations increases.^[6]

Finally, if nothing else, it seems epistemically virtuous to be clear about the reasons for our decisions. Sure, perhaps there’s no behavioral difference between “I’m working on AI risk because I’ve really weighed up all the possible consequences, and it doesn’t seem arbitrary to say this work is impartially good ‘in expectation’”, and “I have no clue if my idealized self would favor working on AI risk, but I’m doing it because no one has offered something better”. But I think if we’re honest with ourselves that our reasoning is the latter, we’ll have more open minds if and when “something better” comes along.

Critiques: What I expect to be productive, and what’s been said so far

So, if the unawareness argument is worth engaging with, what should we focus on when scrutinizing it? Going forward, I expect newer or sharper critiques of the normative premise, and part (b) of the conceptual premise, to be most productive. This is because:

• I feel relatively confused about these premises, and they seem underexplored, at least in precise terms. I’m not aware of many writings explicitly aiming to answer: “What is the standard that non-idealized impartial altruists (should) use to judge which actions are rational? If it’s ‘approximating EV’, or ‘going with our best guess, even if not with literal precise EVs’, what exactly do these things mean, and what justifies them?” (See also Violet Hour’s discussion of this open problem.)
  • I think a relatively promising framework to start with here is reasons-based choice.
• By contrast:
  • Part (a) of the conceptual premise is more crisp and therefore less confusing. There has already been a lot of academic work on the arguments for and against precise Bayesianism (for example).
  • Disagreements about the empirical premise partly hinge on different intuitions about, e.g., how we should extrapolate from our history of discovering certain backfire risks. These intuitions can be hard to make legible. (That said, we can still make progress by considering arguments like those in posts #3 and #4 of the unawareness sequence.)

As far as I’m aware, almost all existing critiques of the unawareness argument are addressed by the sequence, or by other references in this resource guide — see the following table. (The exception, included in the table as well, is the critique that incomparability violates decision-theoretic deference principles.)

(Edit June 27, 2026:) Several objections in the table are classified as objections to P1, because they’re of the form: “We should always have a c-preference (rather than say the options are incomparable w.r.t. c-preferability) because of [some reason other than a comparison of the ‘expected’ consequences].”

Acknowledgments

Thanks to Toby Tremlett, Clare Harris, and Konrad Kozaczek for comments.

Appendix: Sequence summary annotated with the corresponding premises

Here’s a copy of the unawareness sequence summary from post #1, where each section is tagged with the premises of the argument supported by that section.

1. The challenge of unawareness for impartial altruist action guidance: Introduction:

1. Unawareness consists of two problems: The possible outcomes we can conceive of are too coarse to precisely evaluate, and there are some outcomes we don’t conceive of in the first place. (1.1)
   • (P3): makes precise a basic sense in which our information about total consequences is coarse-grained.
2. Thus, unlike standard uncertainty, under unawareness it’s unclear how to make tradeoffs between the possible outcomes we consider when making decisions. (1.1.1)
   • (P2.a): blocks the most straightforward motivation for precise EVs.
3. We can’t dissolve this problem by avoiding explicit models of the future, or by only asking what works empirically. (1.1.2)
   • (P1): argues that we need some all-things-considered model of the consequences, not just (e.g.) brute appeals to heuristics.
4. A vignette illustrates how unawareness might undermine even intuitively robust interventions, like trying to reduce AI x-risk. (1.2)
   • (P3): concretely illustrates how our information about some longtermist intervention seems very coarse-grained.

2. Why intuitive comparisons of large-scale impact are unjustified:

1. The generalization of “expected value” to the case of unawareness should be imprecise, i.e., not a single number, but an interval. This is because assigning precise values to outcomes we’re not precisely aware of would be arbitrary. This imprecision doesn’t represent uncertainty about some “true EV” we’d endorse with more thought. Rather, it reflects irreducible indeterminacy: there is no single value pinned down by our evidence and epistemic principles. (2.1)
   • (P2.a).
2. Suppose we have (A) a deep understanding of the mechanisms determining a strategy’s consequences on some scale, and (B) evidence of consistent success in similar contexts. Then, we can trust that our intuitions factor in unawareness precisely enough to justify comparing strategies, relative to that scale. (2.2)
   • (P2.b), (P3.b): argues that (P2.b), (P3.b) don’t “prove too much”.
3. Unlike in everyday decision situations, we have neither (A) nor (B) when making choices based on the impartial good.
   1. Our understanding of our effects on high-stakes outcomes seems too shallow for us to have precisely calibrated intuitions. This is due to the novel and empirically inaccessible dynamics of, e.g., the development of superintelligence, civilization after space colonization, and possible interactions with other universes. (2.3.1)
      • (P2.b), (P3.b): argues that our tacit understanding of total consequences is coarse-grained, and hence “best guess” comparisons are inappropriate.
   2. The mechanisms we’re unaware of might be qualitatively distinct from those we’re aware of. They’re not merely the minor variations we know superforecasters can handle. (2.3.1.1)
      • (P2.b), (P3.b): rebuts a counterargument to the previous point.
   3. Instead of consistent success, we have a history of consistently fragile models of how to promote the impartial good. Based on EAs’ track record of discovering sign-flipping considerations and new scales of impact, we’re likely unaware of more such discoveries. (2.3.2)
      • (P2.b), (P3.b): argues that we’re likely unaware of many considerations of which we have only very coarse-grained understanding, and hence “best guess” comparisons are inappropriate.
4. If we don’t know how to weigh up evidence about our overall impact that points in different directions, then an intuitive precise guess is not a tiebreaker. This intuition is just one more piece of evidence to weigh up. (2.4)
   • (P1); (P2.a), (P2.b): rebuts “you have to choose something, so nothing is incomparable”; rebuts arguments for precise EVs and for “best guess” comparisons.

3. Why impartial altruists should suspend judgment under unawareness:

1. To get the imprecise “EV” of a strategy under unawareness, we take the EV with respect to all plausible ways of precisely evaluating coarse outcomes. Given two strategies, if neither strategy is net-better than the other under all these ways of making precise evaluations, then we’re not justified in comparing these strategies. (3.1; 3.1.1)
   • (P2.a), (P2.b): defends imprecise “EV”, and rebuts an argument for aggregating the set of “EVs” to make comparisons.
2. Our evaluations of pairs of strategies should be so severely imprecise that they’re incomparable, absent arguments to the contrary. This is for two reasons:
   1. Given the possibilities we’re aware of, there are very few constraints on how to precisely model the possibilities we’re unaware of. This lack of constraints is worsened by systematic biases in how we discover new hypotheses. For example, we may be disproportionately likely to consider hypotheses that we happen to find interesting. (3.2.1)
      • (P3.a): argues that our coarse-grained understanding of the catch-all implies severe imprecision.
   2. Suppose we try breaking down the space of possible outcomes into manageable categories. Since we can only break things down so far, the categories we can model remain too coarse-grained to pin down whether a strategy’s expected upsides outweigh its downsides. (3.2.2)
      • (P3.a): argues that our coarse-grained understanding of hypotheses we’re aware of also implies severe imprecision.
3. We have unawareness at the level of both (P1) how good different world-states are (like “misaligned AIs take over”) relative to each other, and (ii) how effective concrete interventions are at steering toward vs. away from a given world-state. (3.3.1)
   • (Framing for the argument in the next point.)
4. When we model the impact of the AI safety intervention from the vignette in (1d), the structural problems from (3b) and (3c) undermine the case for that intervention. That is, given reasonable ranges of parameter estimates, the intervention is positive under some estimates and negative under others, and it’s arbitrary how we weigh up their plausibility. (3.3.2)
   • (P3.a): supports the arguments about severe imprecision above with a specific formal example.

4. Why existing approaches to cause prioritization are not robust to unawareness:

1. We can’t assume the considerations we’re unaware of “cancel out”, because when we discover a new consideration, this assumption no longer holds. (4.1.1)
   • (P3.a): rebuts counterargument.
2. We can’t trust that the hypotheses we’re aware of are a representative sample (see 3.b.i), so we can’t naïvely extrapolate from them. Although we don’t know the net direction of our biases, this doesn’t justify the very strong assumption that we’re precisely unbiased in expectation. (4.1.2)
   • (P3.a): rebuts counterargument.
3. Similarly, we can’t trust that a strategy’s past success under smaller-scale unawareness is representative of how well it would promote the impartial good. The mechanisms that made a strategy work historically could actively mislead us when predicting its success on a deeply unfamiliar scale. (4.1.3)
   • (P3.a): rebuts counterargument.
4. The argument that heuristics are robust assumes we can neglect complex effects (i.e., effects beyond the “first order”), either in expectation or absolutely. But under unawareness, we have no reason to think these effects cancel out, and should expect them to matter a lot collectively. (4.1.4)
   • (P3.a): rebuts counterargument.
5. Even if we focus on near-term lock-in, we can’t control our impact on these lock-in events precisely enough, nor can we tease apart their relative value when we only picture them coarsely. The “punt to the future” approach doesn’t help for similar reasons. (4.1.5; 4.1.6)
   • (P3.a): rebuts counterargument.
6. Suppose that when we choose between strategies, we only consider the effects we can weigh up under unawareness, because (we think) the other effects aren’t decision-relevant. Then, it seems arbitrary how we group together “effects we can weigh up”. (4.2)
   • (P1): rebuts an argument that A can be c-preferable to B even if A’s total consequences aren’t better.