Oh, I don't think the worry hinges on particular infohazards that aren't public in EA. I'm thinking of a pretty general problem like: "The value of the future from the perspective of your altruistic values, epistemology, and decision theory upon reflection is probably a non-monotonic function of how much you increase wisdom etc. broadly. More 'wisdom' or knowledge for actors who are misaligned with you can be quite bad." And this is at least somewhat borne out by examples like AI movement building, biorisk, and technological progress making factory farming worse.

Cool! For other readers, I think the most relevant sections of the sequence to your question here are 4.1.3 "Meta-extrapolation" and 4.1.6 "Capacity-building". They don't go into much concrete detail on backfire risks of "promoting wisdom, cooperativeness, knowledge, etc". But yeah, mostly stuff like infohazards and dual use, plus the unknown unknown downsides we should expect from pessimistic induction. The idea is that:

• The historical mechanisms by which promoting wisdom/cooperativeness/knowledge made things better occurred in the context of fairly non-weird human socioeconomics. AI takeoff and space colonization are much weirder contexts.
• Even if the downsides seem unlikely in absolute terms, the intended upsides from promoting wisdom etc. are so indirect that I think we should also consider them similarly unlikely.

(More controversial, probably: On #6, IDK, the abstraction of general "intelligence" seems too coarse to me. LLMs' (and humans'!) capability profile seems to depend on a lot more domain-specific fiddly things than the intelligence explosion argument suggests. But I'd be interested in evidence otherwise. ETA: Put differently, I basically co-sign this post.)

The ordering of #7-#9 at least seems debatable to me, because many aspects of (e.g.) natural sciences and medical breakthroughs seem bottlenecked on things other than remote-only labor. Like physical understanding and dexterity (for the scientific labor), and feedback from the physical world (for medical breakthroughs).

Thanks! I do something like this in post #2, sec. 2.3. This first post is just meant to set up the problem at a high level. Interested to hear your thoughts on those examples.

(See also here - these aren't "really weird", but they do seem like pretty plausible backfire mechanisms to me.)

I should have played the game more. I thought BOTECs for alignment work were dumb — how would estimating differential x-risk reduction basis points mean anything? I figured smart funders would just get the obvious argument around reducing elicitation overhang. They didn’t, and ego stopped me from making up the numbers anyway. Probably should have.

I think it would be a shame if you took away this lesson. A perfectly good reason to not make up numbers, that has nothing to do with ego, is: "Made-up numbers are arbitrary! There's no reason to think these numbers are calibrated with enough precision to be action-guiding, or at least, as action-guiding as people often take them to be."

(My gripes discussed in the linked post above, FWIW: Re: (1), the typical EA operationalization of "well-calibrated", and judgments about how to defer to people based on their calibration on some reference class of past questions, are based on very questionable epistemological assumptions. See also this great post.)

When talking about forecasting, people often ask questions like “How can we leverage forecasting into better decisions?” This is the wrong way to go about solving problems. You solve problems by starting with the problem, and then you see which tools are useful for solving it.

I definitely have my own gripes about EA/rationalist attitudes towards forecasting (see here), but maybe your objection is a level confusion:

1. I think when people talk about "leveraging forecasting into better decisions", they're saying: "'Better' decisions just are decisions guided by the normatively correct beliefs. Namely, they're decisions that make reasonable-seeming tradeoffs between possible outcomes given the normatively correct beliefs about the plausibility of those outcomes. So our decisions will be more aligned with this standard of 'better' if our beliefs are formed by deferring to well-calibrated forecasts."
   1. E.g. they're saying, "When navigating AI risk, we'll make decisions that we endorse more if those decisions are guided by the credences of folks who've been unusually successful at forecasting AI developments."
   2. (At least, that's the steelman. Maybe I'm being too charitable!)
2. Whereas you seem to be asking something like: "We already know which beliefs are reasonable. Do these beliefs tell us that 'plug forecasts into some decision-making procedure' seems likely to lead to good outcomes (i.e., that this is a 'useful tool')?"

I meant my future decisions would be the same in reality if I could not gather additional evidence

Perhaps, but that's consistent with incomparability. Given the independent motivations we've discussed (/given in my post) for calling the two options incomparable, I'd say you should call them incomparable.

I think I address your questions in the second paragraph in "Why we're especially unaware of large-scale consequences" (this post) and "Meta-extrapolation" (post #4). See also my discussion with Richard here.

I don't know what you mean by "practically the same", can you say more?

Regardless, the problem is that "gathering evidence" vs "doing something else" is itself a decision, whose consequences you'll be clueless about. I discuss this more here.