Hide table of contents

MIRI is running its first fundraiser in six years, targeting $6M. The first $1.6M raised will be matched 1:1 via an SFF grant. Fundraiser ends at midnight on Dec 31, 2025. Support our efforts to improve the conversation about superintelligence and help the world chart a viable path forward.


MIRI is a nonprofit with a goal of helping humanity make smart and sober decisions on the topic of smarter-than-human AI.

Our main focus from 2000 to ~2022 was on technical research to try to make it possible to build such AIs without catastrophic outcomes. More recently, we’ve pivoted to raising an alarm about how the race to superintelligent AI has put humanity on course for disaster.

In 2025, those efforts focused around Nate Soares and Eliezer Yudkowsky’s book (now a New York Times bestseller) If Anyone Builds It, Everyone Dies, with many public appearances by the authors; many conversations with policymakers; the release of an expansive online supplement to the book; and various technical governance publications, including a recent report with a draft of an international agreement of the kind that could actually address the danger of superintelligence.

Millions have now viewed interviews and appearances with Eliezer and/or Nate, and the possibility of rogue superintelligence and core ideas like “grown, not crafted” are increasingly a part of the public discourse. But there is still a great deal to be done if the world is to respond to this issue effectively.

In 2026, we plan to expand our efforts, hire more people, and try a range of experiments to alert people to the danger of superintelligence and help them make a difference.

To support these efforts, we’ve set a fundraising target of $6M ($4.4M from donors plus 1:1 matching on the first $1.6M raised, thanks to a $1.6M matching grant), with a stretch target of $10M ($8.4M from donors plus $1.6M matching).

Donate here

or read on to learn more.[1]

The Big Picture

As stated in If Anyone Builds It, Everyone Dies:

If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

We do not mean that as hyperbole. We are not exaggerating for effect. We think that is the most direct extrapolation from the knowledge, evidence, and institutional conduct around artificial intelligence today. In this book, we lay out our case, in the hope of rallying enough key decision-makers and regular people to take AI seriously. The default outcome is lethal, but the situation is not hopeless; machine superintelligence doesn't exist yet, and its creation can yet be prevented.

The leading AI labs are explicitly rushing to create superintelligence. It looks to us like the world needs to stop this race, and that this will require international coordination. MIRI houses two teams working towards that end:

  1. A communications team working to alert the world to the situation.
  2. A governance team working to help policymakers identify and implement a response.

Activities

Communications

If Anyone Builds It, Everyone Dies has been the main recent focus of the communications team. We spent substantial time and effort preparing for publication, executing the launch, and engaging with the public via interviews and media appearances.
 

May be an image of television, newsroom and text that says 'ARTIFICIA FUTURE UNDERSTANDING UNDERSTANDING A NEWSLIVE 美 ARTIFICIAL FUTURE? UNDERSTANDING AI NEW WARNS OF THREATS To HUMANITY FROM ARTIFICIAL FROMARTIFICIALSUPERINTELIGENCE SUPERINTELLIGENCE'


The book made a pretty significant splash:

The end goal is not media coverage, but a world in which people understand the basic situation and are responding in a reasonable, adequate way. It seems early to confidently assess the book's impact, but we see promising signs.

The possibility of rogue superintelligence is now routinely mentioned in mainstream coverage of the AI industry. We’re finding in our own conversations with strangers and friends that people are generally much more aware of the issue, and taking it more seriously. Our sense is that as people hear about the problem through their own trusted channels, they are more receptive to concerns.

Our conversations with policymakers feel meaningfully more productive today than they did a year ago, and we have been told by various U.S. Members of Congress that the book had a valuable impact on their thinking. It remains to be seen how much this translates into action. And there is still a long way to go before world leaders start coordinating an international response to this suicide race.

Today, the MIRI comms team comprises roughly seven full-time employees (if we include Nate and Eliezer). In 2026, we’re planning to grow the team. For example:

  • We need someone whose job is to track AI developments and how the global conversation is responding to those developments, and help coordinate a response.
  • We need someone to assess and measure the effectiveness of various types of communications and arguments, and notice what’s working and what’s not.
  • We need someone to track and maintain relationships with various colleagues and allies (such as neighboring organizations, safety teams at the labs, journalist contacts, and so on) and make sure the right resources are being deployed at the right times.

We will be making a hiring announcement soon, with more detail about the comms team’s specific models and plans. We are presently unsure (in part due to funding constraints/budgetary questions!) whether we will be hiring one or two new comms team members, or many more.

Going into 2026, we expect to focus less on producing new content, and more on using our existing library of content to support third parties who are raising the alarm about superintelligence for their own audiences. We also expect to spend more time responding to news developments and taking advantage of opportunities to reach new audiences.

Governance

Our governance strategy primarily involves:

  1. Figuring out solutions, from high-level plans to granular details, for how to effectively halt the development of superintelligence.
  2. Engaging with policymakers, think tanks, and others who are interested in developing and implementing a response to the growing dangers.

There's a ton of work still to be done. To date, the MIRI Technical Governance Team (TGT) has mainly focused on high-level questions such as "Would it even be possible to monitor AI compute relevant to frontier AI development?" and "What would an international halt to the superintelligence race look like?" We're only just beginning to transition into more concrete specifics, such as writing up A Tentative Draft of a Treaty, with Annotations, which we published on the book website to coincide with the book release, followed by a draft international agreement.

We plan to push this a lot further, and work towards answering questions like:

  • What, exactly, are the steps that could be taken today, assuming different levels of political will?
  • If there is will for chip monitoring and verification, what are the immediate possible legislative next-steps? What are the tradeoffs between the options?
  • Technologically, what are the immediate possible next steps for, e.g., enabling tamper-proof chip usage verification? What are the exact legislative steps that would require this verification?

We need to extend that earlier work into concrete, tractable, shovel-ready packages that can be handed directly to concerned politicians and leaders (whose ranks grow by the day).

To accelerate this work, MIRI is looking to support and hire individuals with relevant policy experience, writers capable of making dense technical concepts accessible and engaging, and self-motivated and competent researchers.[2]

We’re also keen to add additional effective spokespeople and ambassadors to the MIRI team, and to free up more hours for those spokespeople who are already proving effective. Thus far, the bulk of our engagement with policymakers and national security professionals has been done either by our CEO (Malo Bourgon), our President (Nate Soares), or the TGT researchers themselves. That work is paying dividends, but there’s room for a larger team to do much, much more.

In our conversations to date, we’ve already heard that folks in government and at think tanks are finding TGT’s write-ups insightful and useful, with some calling it top-of-its-class work. TGT’s recent outputs and activities include:

  • In addition to collaborating with Nate, Eliezer, and others to produce the treaty draft, the TGT has further developed this document into a draft international agreement, along with a collection of supplementary posts that expand on various points.
  • The team published a research agenda earlier this year. Much of their work (to date and going forward) falls under this agenda, which is further explored in a number of papers digging into various specifics. TGT has also participated in relevant conferences and workshops, and has been supervising and mentoring junior researchers through external programs.
  • TGT regularly provides input on RFCs and RFIs from various governmental bodies, and engages with individuals in governments and elsewhere through meetings, briefings, and papers.
  • Current efforts are mostly focused on the U.S. federal government, but not exclusively. For example, in 2024 and 2025, TGT participated in the EU AI Act Code of Practice Working Groups, working to make EU regulations more likely to be relevant to misalignment risks from advanced AI. Just four days ago, Malo was invited to provide testimony to a committee of the Canadian House of Commons; and TGT researcher Aaron Scher was invited to speak to the Scientific Advisory Board of the Secretary-General of the UN on AI verification as part of an expert panel.
     

 

The above isn’t an exhaustive description of what everyone at MIRI is doing; e.g., we continue to support a small amount of in-house technical alignment research.

As noted above, we expect to make hiring announcements in the coming weeks and months, outlining the roles we’re hoping to add to the team. But if your interest has already been piqued by the general descriptions above, you’re welcome to reach out to contact@intelligence.org. For more updates, you can subscribe to our newsletter or periodically check our careers pages (MIRI-wide, TGT-specific).

Fundraising

Our goal at MIRI is to have at least two years’ worth of reserves on hand. This enables us to plan more confidently: hire new staff, spin up teams and projects with long time horizons, and balance the need to fundraise with other organizational priorities. Thanks to generous support we received in 2020 and 2021, we didn’t need to run any fundraisers in the last six years.

We expect to hit December 31st having spent approximately $7.1M this year (similar to recent years[3]), and with $10M in reserves if we raise no additional funds.[4]

Going into 2026, our budget projections have a median of $8M[5], assuming some growth and large projects, with large error bars from uncertainty about the amount of growth and projects. On the upper end of our projections, our expenses would hit upwards of $10M/yr.

Thus, our expected end-of-year reserves puts us $6M shy of our two-year reserve target of $16M.

This year, we received a $1.6M matching grant from the Survival and Flourishing Fund, which means that the first $1.6M we receive in donations before December 31st will be matched 1:1. We will only receive the grant funds if it can be matched by donations.

Therefore, our fundraising target is $6M ($4.4M from donors plus 1:1 matching on the first $1.6M raised). This will put us in a good place going into 2026 and 2027, with a modest amount of room to grow.

It’s an ambitious goal and will require a major increase in donor support, but this work strikes us as incredibly high-priority, and the next few years may be an especially important window of opportunity. A great deal has changed in the world over the past few years. We don’t know how many of our past funders will also support our comms and governance efforts, or how many new donors may step in to help. This fundraiser is therefore especially important for informing our future plans.

We also have a stretch target of $10M ($8.4M from donors plus the first $1.6M matched). This would allow us to move much more quickly on pursuing new hires and new projects, embarking on a wide variety of experiments while still maintaining two years of runway.

For more information or assistance on ways to donate, view our Donate page or contact development@intelligence.org.


The default outcome of the development of superintelligence is lethal, but the situation is not hopeless; superintelligence doesn't exist yet, and humanity has the ability to hit the brakes.

With your support, MIRI can continue fighting the good fight.

Donate today
  1. ^

    This post's lead author is Alex Vermeer.

  2. ^

    In addition to growing our team, we plan to do more mentoring of new talent who might go on to contribute to TGT's research agenda, or who might contribute to the field of technical governance more broadly.

  3. ^

    Our yearly expenses in 2019–2024 ranged from $5.4M to $7.7M, with the high point in 2020 (when our team was at its largest), and the low point in 2022 (after scaling back).

  4. ^

    It’s worth noting that despite the success of the book, book sales will not be a source of net income for us. As the authors noted prior to the book’s release, “unless the book dramatically exceeds our expectations, we won’t ever see a dime”. From MIRI’s perspective, the core function of the book is to try to raise an alarm and spur the world to action, not to make money; even with the book’s success to date, the costs to produce and promote the book have far exceeded any income.

  5. ^

    Our projected expenses are roughly evenly split between Operations, Outreach, and Research, where our communications efforts fall under Outreach and our governance efforts largely fall under Research (with some falling under Outreach). Our median projection breaks down as follows: $2.6M for Operations ($1.3M people costs, $1.2M cost of doing business), $3.2M Outreach ($2M people costs, $1.2M programs), and $2.3M Research ($2.1M people costs, $0.2M programs). This projection includes roughly $0.6–1M in new people costs (full-time-equivalents, i.e., assuming the people are not all hired on January 1st).

    Note that the above is an oversimplified summary; it's useful for high-level takeaways, but for the sake of brevity, I've left out a lot of caveats, details, and explanations.

  6. Show all footnotes

56

1
1

Reactions

1
1

More posts like this

Comments18
Sorted by Click to highlight new comments since:

IMO it would help to see a concrete list of MIRI's outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically. 

If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for those, anything you can say about the more dangerous ones (e.g. number of work hours that went into them, what class of project they were, whether they're still live, any downstream effect you can point to, and so on).

(Speaking in my capacity as someone who currently works for MIRI)

I think the degree to which we withheld work from the public for fear of accelerating progress toward ASI might be a little overrepresented in the above.  We adopted a stance of closed-by-default research years ago for that reason, but that's not why e.g. we don't publish concrete and exhaustive lists of outputs and budget.

We do publish some lists of some outputs, and we do publish some degree of budgetary breakdowns, in some years.

But mainly, we think of ourselves as asking for money from only one of the two kinds of donors.  MIRI feels that it's pretty important to maintain strategic and tactical flexibility, to be able to do a bunch of different weird things that we think each have a small chance of working out without exhaustive justification of each one, and to avoid the trap of focusing only on clearly legible short chains of this—>that (as opposed to trying both legible and less-legible things).

(A colleague of mine once joked that "wages are for people who can demonstrate the value of their labor within a single hour; I can't do that, which is why I'm on a salary."  A similar principle applies here.)

In the past, funding MIRI led to outputs like our alignment research publications.  In the more recent past, funding MIRI has led to outputs like the work of our technical governance team, and the book (and its associated launch campaign and various public impacts).

That's enough for some donors—"If I fund these people, my money will go into various experiments that are all aimed at ameliorating existential risk from ASI, with a lean toward the sorts of things that no one else is trying, which means high variance and lots of stuff that doesn't pan out and the occasional home run."

Other donors are looking to more clearly purchase a specific known product, and those donors should rightly send fewer of their dollars to MIRI, because MIRI has never been and does not intend to ever be quite so clear and concrete and locked-in.

(One might ask "okay, well, why post on the EA forum, which is overwhelmingly populated by the other kind of donor, who wants to track the measurable effectiveness of their dollars?" and the answer is "mostly for the small number who are interested in MIRI-like efforts anyway, and also for historical reasons since the EA and rationality and AI safety communities share so much history."  Definitely we do not feel entitled to anyone's dollars, and the hesitations of any donor who doesn't want to send their money toward MIRI-like efforts are valid.)

That makes some sense, but leaves me with questions like

  • Which projects were home runs, and how did you tell that a) they were successful at achieving their goals and b) that their goals were valuable?
  • Which projects were failures that you feel were justifiable given your knowledge state at the time?
  • What do these past projects demonstrate about the team's competence to work on future projects?
  • What and how was the budget allocated to these projects, and do you expect future projects projects to have structurally similar budgets?
  • Are there any other analogies you could draw between past and possible future projects that would enable us update on the latter's probability of success?

MIRI is hardly unique even in the EA/rat space in having special projects - Rethink Priorities, for e.g., seem to be very fluid in what they work on; Founders Pledge and Longview are necessarily driven to some degree by the interests of their major donors; Clean Air Task force have run many different political campaigns, each seemingly unlike the previous ones in many ways; ALLFED are almost unique in their space, so have huge variance in the projects they work on; and there are many more with comparable flexibility. 

And many EA organisations in the space that don't explicitly have such a strategy have nonetheless pivoted after learning of a key opportunity in their field, or realising an existing strategy was failing. 

In order to receive funds - at least from effectiveness-minded funders - all these orgs have to put a certain amount of effort into answering questions like those above.

And ok, you say you're not claiming to be entitled to dollars, but it still seems reasonable to ask why a rational funder should donate to MIRI over e.g. any of the above organisations - and to hope that MIRI has some concrete answers.

Noting that this is more "opinion of an employee" than "the position of MIRI overall"—I've held a variety of positions within the org and can't speak for e.g. Nate or Eliezer or Malo:

  • The Agent Foundations team feels, to me, like it was a slam dunk at the time; the team produced a ton of good research and many of their ideas have become foundational to discussions of agency in the broader AI sphere
  • The book feels like a slam dunk
  • The research push of 2020/2021 (that didn't pan out) feels to me like it was absolutely the right bet, but resulted in (essentially) nothing; it was an ambitious, many-person project for a speculative idea that had a shot at being amazing.

I think it's hard to generalize lessons, because various projects are championed by various people and groups within the org ("MIRI" is nearly a ship of Theseus).  But some very basic lessons include:

  • Things pretty much only have a shot at all when there are people with a clear and ambitious vision/when there's an owner
  • When we say to ourselves "this has an X% chance of working out" we seem to be actually pretty calibrated
  • As one would expect, smaller projects and clearer projects work out more frequently than larger or vaguer ones

(Sorry, that feels sort of useless, but.)

From my limited perspective/to the best of my ability to see and descrbe, budget is essentially allocated in a "Is this worth doing?  If so, how do we find the resources to make it work?" sense.  MIRI's funding situation has always been pretty odd; we don't usually have a pie that must be divided up carefully so much as a core administrative apparatus that needs to be continually funded + a preexisting pool of resources that can be more or less freely allocated + a sense that there are allies out there who are willing to fund specific projects if we fall short and want to make a compelling pitch.  

Unfortunately, I can't really draw analogies that help an outsider evaluate future projects.  We're intending to try stuff that's different from anything we've tried before, which means it's hard to draw on the past (except insofar as the book and surrounding publicity were also something we'd never tried before, so you can at least a little bit assess our ability to pivot and succeed at stuff outside our wheelhouse by looking at the book).

Some choice quotes from Clara Collier's incisive review of If Anyone Builds It, Everyone Dies in Asterisk Magazine:

It’s true that the book is more up-to-date and accessible than the authors’ vast corpus of prior writings, not to mention marginally less condescending. Unfortunately, it is also significantly less coherent. The book is full of examples that don’t quite make sense and premises that aren’t fully explained. But its biggest weakness was described many years ago by a young blogger named Eliezer Yudkowsky: both authors are persistently unable to update their priors.

About the unexplained shift of focus from symbolic AI, which Yudkowsky was still claiming as of around 2015 or 2016 — quite late in the game, all things considered — was more likely than deep learning to lead to AGI, to deep learning:

We’ve learned a lot since 2008. The models Yudkowsky describes in those old posts on LessWrong and Overcoming Bias were hand-coded, each one running on its own bespoke internal architecture. Like mainstream AI researchers at the time, he didn’t think deep learning had much potential, and for years he was highly skeptical of neural networks. (To his credit, he’s admitted that that was a mistake.) But If Anyone Builds It, Everyone Dies very much is about deep learning-based neural networks. The authors discuss these systems extensively — and come to the exact same conclusions they always have. The fundamental architecture, training methods and requirements for progress for modern AI systems are all completely different from the technology Yudkowsky imagined in 2008, yet nothing about the core MIRI story has changed.

Building on this:

In fact, there are plenty of reasons why the fact that AIs are grown and not crafted might cut against the MIRI argument. For one: The most advanced, generally capable AI systems around today are trained on human-generated text, encoding human values and modes of thought.

I still have no idea when, why, or how exactly Eliezer Yudkowsky changed his mind about symbolic AI vs. deep learning. This seems quite fundamental to his, Nate Soares', and MIRI's case, yet as far as I know, it's never been discussed. I've looked, and I've asked around. I'm not reassured Yudkowsky has a good understanding of deep learning, and, per Clara Collier's review, it really doesn't seem like the core MIRI case has been updated since the pre-deep learning era in the late 2000s. If deep learning doesn't change things, Yudkowsky/MIRI should explain why not. If it does change things, then Yudkowsky/MIRI's views and arguments should be updated to reflect that.

Also, it's worth reflecting on how unrealistic Yudkowsky's belief in symbolic AI now appears given that we've had over a decade of deep learning-based and deep reinforcement learning-based AI systems that are astronomically more capable than any symbolic AI systems ever were, and yet these systems are still far below human-level. Deep learning is vastly more capable than symbolic AI and even deep learning is still vastly less capable than the average human (or, on some dimensions, the average cat). So, it really seems unrealistic to think symbolic AI could have led to AGI, especially on the short timescales Yudkowsky was imagining in the 2000s and early-to-mid 2010s. 

It makes the whole thing look a little odd. If deep neural networks had never been invented, eventually, at some point, surely it would have become evident that symbolic AI was never going to lead to anything interesting or powerful. Maybe in this counterfactual timeline, by the 2040s, with no meaningful progress in AI, people who had believed Yudkowsky's arguments would start to have doubts. It's odd that deep neural networks were invented and then Yudkowsky abandoned this forlorn theory about symbolic AI, yet changed very little, if anything, when creating a new version of the theory about deep learning. It's also quite odd that he switched from the old version of the theory to the new version with no public explanation, as far as I've been able to find. 

This seems consistent with a general pattern of reluctance to admit mistakes.

If deep learning doesn't change things, Yudkowsky/MIRI should explain why not.

Speaking in my capacity as someone who currently works for MIRI, but who emphatically does not understand all things that Eliezer Yudkowsky understands, and can't authoritatively represent him (or Nate, or the other advanced researchers at MIRI who are above my intellectual pay grade):

My own understanding is that Eliezer has, all along, for as long as I've known him and been following his work, been fairly agnostic as to questions of how AGI and ASI will be achieved, and what the underlying architectures of the systems will be.

I've often seen Eliezer say "I think X will not work" or "I think Y is less doomed than X," but in my experience it's always been with a sort of casual shrug and an attitude of "but of course these are very hard calls" and also with "and it doesn't really matter to the ultimate outcome except insofar as some particular architecture might make reliable alignment possible at all."

Eliezer's take (again, as I understand it) is something like "if you have a system that is intelligent enough and powerful enough to do the actual interesting work that humans want to do, such as end all wars and invent longevity technology and get us to the stars (and achieve these goals in the real world, which involves also being competent at things like persuasion and communication), then that system is going to be very, very, very hard to make safe.  It's going to be easier by many orders of magnitude to create systems that are capable of that level of sophisticated agency that don't care about human flourishing, than it will be to hit the narrow target of a sufficiently sophisticated system that also does in fact happen to care."

That's true regardless of whether you're working with deep learning or symbolic AI.  In fact, deep learning makes it worse—Eliezer was pointing at "even if you build this thing out of nuts and bolts that you thoroughly understand, alignment is a hard problem," and instead we have ended up in a timeline where the systems are grown rather than crafted, giving us even less reason to be confident or hopeful.

(This is a trend: people often misunderstand MIRI's attempts to underscore how hard the problem is as being concrete predictions about what will happen, c.f. the era in which people were like, well, obviously any competent lab trying to build ASI will keep their systems airgapped and secure and have a very small number of trusted and monitored employees acting as intermediaries.  MIRI's response was to demonstrate how even in such a paradigm, a sufficiently sophisticated system would have little trouble escaping the box.  Now, all of the frontier labs routinely feed their systems the entire internet and let those systems interact with any human on Earth and in many cases let those systems write and deploy their own code with no oversight, and some people say "haha, look, MIRI was wrong."  Those people are confused.)

Symbolic AI vs. deep learning was never a crux, for Eliezer or the MIRI view.  It was a non-crucial sidebar in which Eliezer had some intuitions and guesses, some of which he was more confident about and others less confident, and some of those guesses turned out wrong, and none of that ever mattered to the larger picture.  The crucial considerations are the power/sophistication/intelligence of the system, and the degree to which its true goals can be specified/pinned-down, and being wrong about whether deep learning or symbolic AI specifically were capable of reaching the required level of sophistication is mostly irrelevant.

One could argue "well, Eliezer proved himself incapable of predicting the future with those guesses!" but this would be, in my view, disingenuous.  Eliezer has long said, and continues to say, "look, guesses about how the board will look in the middle of the chess game are fraught, I'm willing to share my intuitions but they are far more likely to be wrong than right; it's hard to know what moves Stockfish will make or how the game will play out; what matters is that it's still easy to know with high confidence that Stockfish will win."

That claim was compelling to me in 2015, and it remains compelling to me in 2025, and the things that have happened in the world in the ten years in between have, on the whole, made the case for concern stronger rather than weaker.

To draw up one comment from your response below:

The author of the review's review does not demonstrate to me that they understand Collier's point.

...Collier's review does not even convincingly demonstrate that they read the book, since they get some extremely basic facts about it loudly, loudly wrong, in a manner that's fairly crucial for their criticisms.  I think that you should hold the reviewer and the review's reviewer to the same standard, rather than letting the person you agree with more off the hook.

Fair warning: I wrote this response less for Yarrow specifically and more for the benefit of the EA forum userbase writ large, so I'm not promising that I will engage much beyond this reply.  I might!  But I also might not.  I think I said the most important thing I had to say, in the above.

EDIT: oh, for more on how this:

In fact, there are plenty of reasons why the fact that AIs are grown and not crafted might cut against the MIRI argument. For one: The most advanced, generally capable AI systems around today are trained on human-generated text, encoding human values and modes of thought.

...is badly, badly wrong, see the supplemental materials for the book, particularly chapters 2, 3, and 4, which exhaustively addressed this point long before Collier ever made it, because we knew people would make it.  (It's also addressed in the book, but I guess Collier missed that in their haste to say a bunch of things it seems they already believed and were going to say regardless.)

I think you might be engaging in a bit of Motte-and-Baileying here. Throughout this comment, you're stating MIRI's position as things like "it will be hard to make ASI safe", and that AI will "win", and that it will be hard for an AI to be perfectly aligned with "human flourishing"  Those statements seem pretty reasonable. 

But the actual stance of MIRI, which you just released a book about, is that there is an extremely high chance that building powerful AI will result in everybody on planet earth being killed. That's a much narrower and more specific claim. You can imagine a lot of scenarios where AI is unsafe, but not in a way that kills everyone. You can imagine cases where AI "wins", but decides to cut a deal with us. You can imagine cases where an AI doesn't care about human flourishing because it doesn't care about anything, it ends up acting like a tool that we can direct as we please. 

I'm aware that you have counterarguments for all of these cases (that I will probably disagree with). But these counterarguments will have to be rooted in the actual nuts and bolts details of how actual, physical AI works. And if you are trying to reason about future machines, you want to be able to get a good prediction about their actual characteristics. 

I think in this context, it's totally reasonable for people to look at your (in my opinion poor) track record of prediction and adjust their credence in your effectiveness as an institution. 

I disagree re: motte and bailey; the above is not at all in conflict with the position of the book (which, to be clear, I endorse and agree with and is also my position).

re: "you can imagine," I strongly encourage people to be careful about leaning too hard on their own ability to imagine things; it's often fraught and a huge chunk of the work MIRI does is poking at those imaginings to see where they collapse.  

I'll note that core MIRI predictions about e.g. how machines will be misaligned at current levels of sophistication are being borne out—things we have been saying for years about e.g. emergent drives and deception and hacking and brittle proxies.  I'm pretty sure that's not "rooted in the actual nuts and bolts details" in the way you're wanting, but it still feels ... relevant.

Thanks @Duncan Sabien for this excellent explanation. Don't undersell yourself, I rate your communication here at least as good (if not better) than that of other senior MIRI people in recent years.

About the unexplained shift of focus from symbolic AI, which Yudkowsky was still claiming as of around 2015 or 2016

This is made up, as far as I can tell (at least re: symbolic AI as described in the wikipedia article you link).  See Logical or Connectionist AI? (2008):

As it so happens, I do believe that the type of systems usually termed GOFAI will not yield general intelligence, even if you run them on a computer the size of the moon.

Wikipedia, on GOFAI (reformatted, bolding mine):

In the philosophy of artificial intelligence, GOFAI (good old-fashioned artificial intelligence) is classical symbolic AI, as opposed to other approaches, such as neural networks, situated robotics, narrow symbolic AI or neuro-symbolic AI.

Even earlier is Levels of Organization in General Intelligence.  It is difficult to excerpt a quote but it is not favorable to the traditional "symbolic AI" paradigm.

This seems quite fundamental to his, Soares', and MIRI's case, yet as far as I know, it's never been discussed. I've looked, and I've asked around.

I really struggle to see how you could possibly have come to this conclusion, given the above.

And see here re: Collier's review.

Curated and popular this week
Relevant opportunities