A 10% chance of transformative AI this decade justifies current EA efforts to make AI go well.

Not necessarily. It depends on 

a) your credence distribution of TAI after this decade, 

b) your estimate of annual risk per year of other catastrophes, and 

c) your estimate of the comparative longterm cost of other catastrophes.

I don't think it's unreasonable to think, for example, that

• there's a very long tail to when TAI might arrive, given that its prospects of arriving in 2-3 decades are substantially related to to its prospects of arriving this decade) it arriving this decade (e.g. if we scale current models substantially and they still show no signs of becoming TAI, that undermines the case for future scaling getting us there under same paradigm); or
• the more pessimistic annual risk estimates I talked about in the previous essay of 1-2% per year are correct, and that future civilisations will have a sufficiently increased difficulty for a collapse to cost to have near 50% the expected cost of extinction

And either of these beliefs (and others) would suggest we're relatively overspending on on AI.

It's also important to understand that Hendrycks and Yudkowsky were simply describing/predicting the geopolitical equilibrium that follows from their strategies, not independently advocating for the airstrikes or sabotage.

This is grossly disingenuous. Yudkowsky frames his call for airstrikes as what we 'need' to do, and describes them in the context of the hypothetical 'if I had infinite freedom to write laws'. Hendrycks is slightly less direct in actively calling for it, claiming that it's the default, but the document clearly states the intent of supporting it 'we outline measures to maintain the conditions for MAIM'. 

These aren't the words of people dispassionately observing a phenomenon - they are both clearly trying to bring about the scenarios they describe when the lines they've personally drawn are crossed.

But the expected value of existential risk reduction is—if not infinite, which I think it clearly is in expectation—extremely massive.

I commented something similar on your blog, but as soon as you allow that one decision is infinite in expectation you have to allow that all outcomes are, since whatever possibility of infinite value you have given that action must still be present without it.

If you think the Bostrom number of 10^52 happy people has a .01% chance of being right, then you’ll get 10^48 expected future people if we don’t go extinct, meaning reducing odds of existential risks by 1/10^20 creates 10^28 extra lives.

Reasoning like this seems kind of scope insensitive to me. In the real world, it's common to see expected payoffs declining as offered rewards get larger, and I don't see any reason to think this pattern shouldn't typically generalise to most such prospects, even when the offer is astronomically large.

The odds are not trivial that if we get very advanced AI, we’ll basically eliminate any possibility of human extinction for billions of years.

I think the stronger case is just security in numbers. Get a civilisation around multiple star systems and capable of proliferating, and the odds of its complete destruction rapidly get indistinguishable from 0.

I agree with Yarrow's anti-'truth-seeking' sentiment here. That phrase seems to primarily serve as an epistemic deflection device indicating 'someone whose views I don't want to take seriously and don't want to justify not taking seriously'.

I agree we shouldn't defer to the CEO of PETA, but CEOs aren't - often by their own admission - subject matter experts so much as people who can move stuff forwards. In my book the set of actual experts is certainly murky, but includes academics, researchers, sometimes forecasters, sometimes technical workers - sometimes CEOs but only in particular cases - anyone who's spent several years researching the subject in question. 

Sometimes, as you say, they don't exist, but in such cases we don't need to worry about deferring to them. When they do, it seems foolish to not to upweight their views relative to our own unless we've done the same, or unless we have very concrete reasons to think they're inept or systemically biased (and perhaps even then).

I agree that the OP is too confident/strongly worded, but IMO this

which is more than enough to justify EA efforts here. 

could be dangerously wrong. As long as AI safety consumes resources that might have counterfactually gone to e.g. nuclear disarmament, stronger international relations, it might well be harmful in expectation.

This is doubly true for warlike AI 'safety' strategies like Aschenbrenner's call to intentionally arms race China, Hendrycks, Schmidt and Wang's call to 'sabotage' countries that cross some ill-defined threshold, and Yudkowsky calling for airstrikes on data centres. I think such 'AI safety' efforts are very likely increasing existential risk.

Well handled, Peter! I'm curious how much of that conversation was organic, how much scripted or at least telegraphed in advance?

I'm warming to CoGi

That makes some sense, but leaves me with questions like

• Which projects were home runs, and how did you tell that a) they were successful at achieving their goals and b) that their goals were valuable?
• Which projects were failures that you feel were justifiable given your knowledge state at the time?
• What do these past projects demonstrate about the team's competence to work on future projects?
• What and how was the budget allocated to these projects, and do you expect future projects projects to have structurally similar budgets?
• Are there any other analogies you could draw between past and possible future projects that would enable us update on the latter's probability of success?

MIRI is hardly unique even in the EA/rat space in having special projects - Rethink Priorities, for e.g., seem to be very fluid in what they work on; Founders Pledge and Longview are necessarily driven to some degree by the interests of their major donors; Clean Air Task force have run many different political campaigns, each seemingly unlike the previous ones in many ways; ALLFED are almost unique in their space, so have huge variance in the projects they work on; and there are many more with comparable flexibility. 

And many EA organisations in the space that don't explicitly have such a strategy have nonetheless pivoted after learning of a key opportunity in their field, or realising an existing strategy was failing. 

In order to receive funds - at least from effectiveness-minded funders - all these orgs have to put a certain amount of effort into answering questions like those above.

And ok, you say you're not claiming to be entitled to dollars, but it still seems reasonable to ask why a rational funder should donate to MIRI over e.g. any of the above organisations - and to hope that MIRI has some concrete answers.

IMO it would help to see a concrete list of MIRI's outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically. 

If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for those, anything you can say about the more dangerous ones (e.g. number of work hours that went into them, what class of project they were, whether they're still live, any downstream effect you can point to, and so on).

Might be confusing with SoGive.