Ah, sorry, I thought you were making the first-order wager argument (Q3 here), but IIUC you're making a metanormative wager argument as Toby suggested. I discuss why I'm unconvinced of that here. (And as another commenter pointed out, this is supplemented by "Why cluelessness matters" in the OP.)

taking your probability-weighted expectation of that range

I think you're misunderstanding the framework. The whole problem is that we can't assign a (non-arbitrary) "probability-weighted expectation". That's the motivation for representing with a range rather than a single expectation.

(ETA: By default I plan not to reply further.)

I address this objection here (Q3), if I understand what you're saying correctly. (I'd recommend first reading sec. 2.1 of the post for crucial background on the epistemology, though, as I noted in another comment.)

(In general, I think you should not expect this post to be a self-contained explanation of the argument by any means. It's a high-level summary.)

I'm not saying we have information that updates us in a particular direction about the bias. I'm saying we have information suggesting various different directions, and it's ambiguous what the update should be overall — which is fundamentally different from "no update". I strongly recommend reading sections 2.1 and 2.4 of this post, as well as 3.2.1 of this post, to understand the epistemology that's at play here.

(ETA: The final paragraphs of sec 4.1.1, linked right after the part you quoted, also discuss this point.)

(Edited for tone)

Sorry, I don't understand. The snippet I quoted — about acausal stuff and simulations — is what's at issue in this discussion.

Regardless, I'm still interested in where you object to my response to Extrapolation. Could you please say more on that?

they seem to argue more for "we're clueless about how much we should do ECL"?

I think they suggest that there's just a lot of subtlety in working out the implications of acausal decision theories in practice. Which is reason to expect more crucial considerations in this domain generally / reason to doubt your "by definition" argument.

but why should I expect their attempts to do so to backfire

Why should you expect them to be positive in expectation either? (The broader point of the unawareness sequence is that there's an ambiguous pile of positive and negative effects to weigh up.)

On the simulators, it just seems like its hard to think of possible simulator-motivations where us reaching good outcomes in the simulation would be bad for the base reality, and easy to think of ones where it would be neutral or good.

But then we're back to the Extrapolation argument, which you claimed you weren't committed to. I'm saying, even if the balance of effects we can think of looks good, we're looking at a super tiny sliver of the set of effects our fully aware selves would be weighing up — and it's a biased sample of such effects, so extrapolating from that sample is dubious.

Hmm, these arguments seem too anchored on what we happen to currently be aware of.

• I think we're very confused about the theory of acausal control / what "similarity" is, and should expect lots of crucial considerations to come up there. (Some relevant writings on this here and here.) As one example: If you try to do something cooperative because this gives evidence (or logically causes) that other agents try to do good things for your values, their attempts to help your values might backfire.
• I don't see what the basis is for inferring "it's probably good for their reality too if we reach a good outcome in the simulation". Why would we think we can understand their motivations? They'd be alien intelligences.

Most of our impact comes from acausal effects, and effects on the base reality if we are in a simulation: I’m confused here like everyone else, but I currently don’t buy this as a major factor because we only know our reality, and therefore the same things that are good here should naively also have good acausal effects in expectation. 

If I understand correctly, this is the "Extrapolation" response to unawareness I discuss here. What do you think of my response?

Linkpost: Bracketing violates the sure-thing principle

I'd been meaning to turn this into a proper post, but didn't get around to it. So I've just linked to this Google doc, which shows that the "bracketing" decision rule violates a version of the sure-thing principle. I personally don't think this is actually much of a problem (see the conclusion of this post, for one), but it might be of interest.

Oh, I don't think the worry hinges on particular infohazards that aren't public in EA. I'm thinking of a pretty general problem like: "The value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More 'wisdom' or knowledge for actors who are misaligned with you can be quite bad." And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.