One counterexample is enough

This competition entry has been selected for publication by the Forum team.

Option 1 — challenging P1 (the normative premise)

P1 is a universal claim about comparing actions: a justified preference for A over B always requires an assessment that A's downstream consequences are better. I take it in its broad form — not only literal expected values, but informal assessments too. Even an informal assessment must be made with some model of the world.

I grant P3 entirely. Our models are compressions. The universe changes while we model it, and it changes partly because we are within it. Our understanding of cosmos-wide consequences is therefore too coarse to compare actions, both in formal models and in informal reasoning.

My objection is to P1 itself. It is a universal claim, so one exception is sufficient.

Suppose an agent is a learning prediction model. I take myself to be such an agent. Then there are moments when that agent discovers that its actions have an effect it had not previously represented. Call that effect X.

The discovery is not the choice. What follows is. The agent can add X to its model (A), or it can continue calculating as before while knowing that X exists (B). Both are actions available to me, now.

I prefer A. Yet I do not need an expected value to justify this preference. Indeed, I cannot have one.

To justify A by comparing consequences, I must run that comparison inside some model. Which model?

If I use the model that already counts X, then I have already chosen A. The comparison presupposes the very revision it is meant to justify.

If I use the model that does not count X, then X appears nowhere in the comparison. From inside that model, adding X registers as no change at all.

Going one level up does not solve the problem. Any model capable of comparing A and B must itself already determine whether X belongs among the relevant considerations. The same problem therefore returns at every level.

The issue is not that the calculation is difficult. It is that there is no neutral calculation to perform. What a model counts is fixed before it can compute anything. Therefore what to count cannot itself be decided by computation within that model.

My preference also does not depend on whether I am ultimately correct about what I learned. Nor does it depend on predicting that adding X will improve future decisions. Once I judge X to be a relevant part of reality,^[1] knowingly maintaining a model that excludes it is defective. No forecast is required.

Moreover, B does not merely omit one effect from one calculation. It preserves a defect in the instrument that produces future assessments. For an impartial altruist, this is not merely an epistemic failure. It is a failure to remain responsive to consequences one already recognises as relevant.

P1 requires assessments; assessments require models. But model revision determines what the model is capable of assessing. Therefore P1 cannot remain neutral between A and B. It requires the choice of A before it can ask for the comparison that is supposed to justify A.

Therefore P1 cannot be universally true. 

1. ^{^}

   One might object that judging X relevant is itself an informal assessment of downstream consequences, so P1 was satisfied all along. It is not. Judging that X bears on outcomes is not judging that A's outcomes are better than B's. Relevance is a property of one thing; P1 demands a comparison between two. I can know that X matters without knowing which action it favours — indeed, P3 guarantees that I usually cannot know the latter.