IMO it would help to see a concrete list of MIRI's outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically.
If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for those, anything you can say about the more dangerous ones (e.g. number of work hours that went into them, what class of project they were, whether they're still live, any downstream effect you can point to, and so on).
(Speaking in my capacity as someone who currently works for MIRI)
I think the degree to which we withheld work from the public for fear of accelerating progress toward ASI might be a little overrepresented in the above. We adopted a stance of closed-by-default research years ago for that reason, but that's not why e.g. we don't publish concrete and exhaustive lists of outputs and budget.
We do publish some lists of some outputs, and we do publish some degree of budgetary breakdowns, in some years.
But mainly, we think of ourselves as asking for money from only one of the two kinds of donors. MIRI feels that it's pretty important to maintain strategic and tactical flexibility, to be able to do a bunch of different weird things that we think each have a small chance of working out without exhaustive justification of each one, and to avoid the trap of focusing only on clearly legible short chains of this—>that (as opposed to trying both legible and less-legible things).
(A colleague of mine once joked that "wages are for people who can demonstrate the value of their labor within a single hour; I can't do that, which is why I'm on a salary." A similar principle applies here.)
In the past, funding MIRI led to outputs like our alignment research publications. In the more recent past, funding MIRI has led to outputs like the work of our technical governance team, and the book (and its associated launch campaign and various public impacts).
That's enough for some donors—"If I fund these people, my money will go into various experiments that are all aimed at ameliorating existential risk from ASI, with a lean toward the sorts of things that no one else is trying, which means high variance and lots of stuff that doesn't pan out and the occasional home run."
Other donors are looking to more clearly purchase a specific known product, and those donors should rightly send fewer of their dollars to MIRI, because MIRI has never been and does not intend to ever be quite so clear and concrete and locked-in.
(One might ask "okay, well, why post on the EA forum, which is overwhelmingly populated by the other kind of donor, who wants to track the measurable effectiveness of their dollars?" and the answer is "mostly for the small number who are interested in MIRI-like efforts anyway, and also for historical reasons since the EA and rationality and AI safety communities share so much history." Definitely we do not feel entitled to anyone's dollars, and the hesitations of any donor who doesn't want to send their money toward MIRI-like efforts are valid.)
That makes some sense, but leaves me with questions like
Which projects were home runs, and how did you tell that a) they were successful at achieving their goals and b) that their goals were valuable?
Which projects were failures that you feel were justifiable given your knowledge state at the time?
What do these past projects demonstrate about the team's competence to work on future projects?
What and how was the budget allocated to these projects, and do you expect future projects projects to have structurally similar budgets?
Are there any other analogies you could draw between past and possible future projects that would enable us update on the latter's probability of success?
MIRI is hardly unique even in the EA/rat space in having special projects - Rethink Priorities, for e.g., seem to be very fluid in what they work on; Founders Pledge and Longview are necessarily driven to some degree by the interests of their major donors; Clean Air Task force have run many different political campaigns, each seemingly unlike the previous ones in many ways; ALLFED are almost unique in their space, so have huge variance in the projects they work on; and there are many more with comparable flexibility.
And many EA organisations in the space that don't explicitly have such a strategy have nonetheless pivoted after learning of a key opportunity in their field, or realising an existing strategy was failing.
In order to receive funds - at least from effectiveness-minded funders - all these orgs have to put a certain amount of effort into answering questions like those above.
And ok, you say you're not claiming to be entitled to dollars, but it still seems reasonable to ask why a rational funder should donate to MIRI over e.g. any of the above organisations - and to hope that MIRI has some concrete answers.
Noting that this is more "opinion of an employee" than "the position of MIRI overall"—I've held a variety of positions within the org and can't speak for e.g. Nate or Eliezer or Malo:
The Agent Foundations team feels, to me, like it was a slam dunk at the time; the team produced a ton of good research and many of their ideas have become foundational to discussions of agency in the broader AI sphere
The book feels like a slam dunk
The research push of 2020/2021 (that didn't pan out) feels to me like it was absolutely the right bet, but resulted in (essentially) nothing; it was an ambitious, many-person project for a speculative idea that had a shot at being amazing.
I think it's hard to generalize lessons, because various projects are championed by various people and groups within the org ("MIRI" is nearly a ship of Theseus). But some very basic lessons include:
Things pretty much only have a shot at all when there are people with a clear and ambitious vision/when there's an owner
When we say to ourselves "this has an X% chance of working out" we seem to be actually pretty calibrated
As one would expect, smaller projects and clearer projects work out more frequently than larger or vaguer ones
(Sorry, that feels sort of useless, but.)
From my limited perspective/to the best of my ability to see and descrbe, budget is essentially allocated in a "Is this worth doing? If so, how do we find the resources to make it work?" sense. MIRI's funding situation has always been pretty odd; we don't usually have a pie that must be divided up carefully so much as a core administrative apparatus that needs to be continually funded + a preexisting pool of resources that can be more or less freely allocated + a sense that there are allies out there who are willing to fund specific projects if we fall short and want to make a compelling pitch.
Unfortunately, I can't really draw analogies that help an outsider evaluate future projects. We're intending to try stuff that's different from anything we've tried before, which means it's hard to draw on the past (except insofar as the book and surrounding publicity were also something we'd never tried before, so you can at least a little bit assess our ability to pivot and succeed at stuff outside our wheelhouse by looking at the book).
[Edited on 2026-01-18 at 20:29 UTC to add: I misremembered some important details about my exchanges on Facebook with Eliezer Yudkowsky and another person at MIRI, Rob Bensinger, around 2016-2018. Take my recollections below as unreliable memory. I went through the trouble of digging up those old Facebook comments and detailed what I found here.]
Some choice quotes from Clara Collier's incisive review of If Anyone Builds It, Everyone Dies in Asterisk Magazine:
It’s true that the book is more up-to-date and accessible than the authors’ vast corpus of prior writings, not to mention marginally less condescending. Unfortunately, it is also significantly less coherent. The book is full of examples that don’t quite make sense and premises that aren’t fully explained. But its biggest weakness was described many years ago by a young blogger named Eliezer Yudkowsky: both authors are persistently unable to update their priors.
About the unexplained shift of focus from symbolic AI, which Yudkowsky was still claiming as of around 2015 or 2016 — quite late in the game, all things considered — was more likely than deep learning to lead to AGI, to deep learning:
We’ve learned a lot since 2008. The models Yudkowsky describes in those old posts on LessWrong and Overcoming Bias were hand-coded, each one running on its own bespoke internal architecture. Like mainstream AI researchers at the time, he didn’t think deep learning had much potential, and for years he was highly skeptical of neural networks. (To his credit, he’s admitted that that was a mistake.) But If Anyone Builds It, Everyone Dies very much is about deep learning-based neural networks. The authors discuss these systems extensively — and come to the exact same conclusions they always have. The fundamental architecture, training methods and requirements for progress for modern AI systems are all completely different from the technology Yudkowsky imagined in 2008, yet nothing about the core MIRI story has changed.
Building on this:
In fact, there are plenty of reasons why the fact that AIs are grown and not crafted might cut against the MIRI argument. For one: The most advanced, generally capable AI systems around today are trained on human-generated text, encoding human values and modes of thought.
I still have no idea when, why, or how exactly Eliezer Yudkowsky changed his mind about symbolic AI vs. deep learning. This seems quite fundamental to his, Nate Soares', and MIRI's case, yet as far as I know, it's never been discussed. I've looked, and I've asked around. I'm not reassured Yudkowsky has a good understanding of deep learning, and, per Clara Collier's review, it really doesn't seem like the core MIRI case has been updated since the pre-deep learning era in the late 2000s. If deep learning doesn't change things, Yudkowsky/MIRI should explain why not. If it does change things, then Yudkowsky/MIRI's views and arguments should be updated to reflect that.
Also, it's worth reflecting on how unrealistic Yudkowsky's belief in symbolic AI now appears given that we've had over a decade of deep learning-based and deep reinforcement learning-based AI systems that are astronomically more capable than any symbolic AI systems ever were, and yet these systems are still far below human-level. Deep learning is vastly more capable than symbolic AI and even deep learning is still vastly less capable than the average human (or, on some dimensions, the average cat). So, it really seems unrealistic to think symbolic AI could have led to AGI, especially on the short timescales Yudkowsky was imagining in the 2000s and early-to-mid 2010s.
It makes the whole thing look a little odd. If deep neural networks had never been invented, eventually, at some point, surely it would have become evident that symbolic AI was never going to lead to anything interesting or powerful. Maybe in this counterfactual timeline, by the 2040s, with no meaningful progress in AI, people who had believed Yudkowsky's arguments would start to have doubts. It's odd that deep neural networks were invented and then Yudkowsky abandoned this forlorn theory about symbolic AI, yet changed very little, if anything, when creating a new version of the theory about deep learning. It's also quite odd that he switched from the old version of the theory to the new version with no public explanation, as far as I've been able to find.
This seems consistent with a general pattern of reluctance to admit mistakes.
If deep learning doesn't change things, Yudkowsky/MIRI should explain why not.
Speaking in my capacity as someone who currently works for MIRI, but who emphatically does not understand all things that Eliezer Yudkowsky understands, and can't authoritatively represent him (or Nate, or the other advanced researchers at MIRI who are above my intellectual pay grade):
My own understanding is that Eliezer has, all along, for as long as I've known him and been following his work, been fairly agnostic as to questions of how AGI and ASI will be achieved, and what the underlying architectures of the systems will be.
I've often seen Eliezer say "I think X will not work" or "I think Y is less doomed than X," but in my experience it's always been with a sort of casual shrug and an attitude of "but of course these are very hard calls" and also with "and it doesn't really matter to the ultimate outcome except insofar as some particular architecture might make reliable alignment possible at all."
Eliezer's take (again, as I understand it) is something like "if you have a system that is intelligent enough and powerful enough to do the actual interesting work that humans want to do, such as end all wars and invent longevity technology and get us to the stars (and achieve these goals in the real world, which involves also being competent at things like persuasion and communication), then that system is going to be very, very, very hard to make safe. It's going to be easier by many orders of magnitude to create systems that are capable of that level of sophisticated agency that don't care about human flourishing, than it will be to hit the narrow target of a sufficiently sophisticated system that also does in fact happen to care."
That's true regardless of whether you're working with deep learning or symbolic AI. In fact, deep learning makes it worse—Eliezer was pointing at "even if you build this thing out of nuts and bolts that you thoroughly understand, alignment is a hard problem," and instead we have ended up in a timeline where the systems are grown rather than crafted, giving us even less reason to be confident or hopeful.
(This is a trend: people often misunderstand MIRI's attempts to underscore how hard the problem is as being concrete predictions about what will happen, c.f. the era in which people were like, well, obviously any competent lab trying to build ASI will keep their systems airgapped and secure and have a very small number of trusted and monitored employees acting as intermediaries. MIRI's response was to demonstrate how even in such a paradigm, a sufficiently sophisticated system would have little trouble escaping the box. Now, all of the frontier labs routinely feed their systems the entire internet and let those systems interact with any human on Earth and in many cases let those systems write and deploy their own code with no oversight, and some people say "haha, look, MIRI was wrong." Those people are confused.)
Symbolic AI vs. deep learning was never a crux, for Eliezer or the MIRI view. It was a non-crucial sidebar in which Eliezer had some intuitions and guesses, some of which he was more confident about and others less confident, and some of those guesses turned out wrong, and none of that ever mattered to the larger picture. The crucial considerations are the power/sophistication/intelligence of the system, and the degree to which its true goals can be specified/pinned-down, and being wrong about whether deep learning or symbolic AI specifically were capable of reaching the required level of sophistication is mostly irrelevant.
One could argue "well, Eliezer proved himself incapable of predicting the future with those guesses!" but this would be, in my view, disingenuous. Eliezer has long said, and continues to say, "look, guesses about how the board will look in the middle of the chess game are fraught, I'm willing to share my intuitions but they are far more likely to be wrong than right; it's hard to know what moves Stockfish will make or how the game will play out; what matters is that it's still easy to know with high confidence that Stockfish will win."
That claim was compelling to me in 2015, and it remains compelling to me in 2025, and the things that have happened in the world in the ten years in between have, on the whole, made the case for concern stronger rather than weaker.
To draw up one comment from your response below:
The author of the review's review does not demonstrate to me that they understand Collier's point.
...Collier's review does not even convincingly demonstrate that they read the book, since they get some extremely basic facts about it loudly, loudly wrong, in a manner that's fairly crucial for their criticisms. I think that you should hold the reviewer and the review's reviewer to the same standard, rather than letting the person you agree with more off the hook.
Fair warning: I wrote this response less for Yarrow specifically and more for the benefit of the EA forum userbase writ large, so I'm not promising that I will engage much beyond this reply. I might! But I also might not. I think I said the most important thing I had to say, in the above.
EDIT: oh, for more on how this:
In fact, there are plenty of reasons why the fact that AIs are grown and not crafted might cut against the MIRI argument. For one: The most advanced, generally capable AI systems around today are trained on human-generated text, encoding human values and modes of thought.
...is badly, badly wrong, see the supplemental materials for the book, particularly chapters 2, 3, and 4, which exhaustively addressed this point long before Collier ever made it, because we knew people would make it. (It's also addressed in the book, but I guess Collier missed that in their haste to say a bunch of things it seems they already believed and were going to say regardless.)
I think you might be engaging in a bit of Motte-and-Baileying here. Throughout this comment, you're stating MIRI's position as things like "it will be hard to make ASI safe", and that AI will "win", and that it will be hard for an AI to be perfectly aligned with "human flourishing" Those statements seem pretty reasonable.
But the actual stance of MIRI, which you just released a book about, is that there is an extremely high chance that building powerful AI will result in everybody on planet earth being killed. That's a much narrower and more specific claim. You can imagine a lot of scenarios where AI is unsafe, but not in a way that kills everyone. You can imagine cases where AI "wins", but decides to cut a deal with us. You can imagine cases where an AI doesn't care about human flourishing because it doesn't care about anything, it ends up acting like a tool that we can direct as we please.
I'm aware that you have counterarguments for all of these cases (that I will probably disagree with). But these counterarguments will have to be rooted in the actual nuts and bolts details of how actual, physical AI works. And if you are trying to reason about future machines, you want to be able to get a good prediction about their actual characteristics.
I think in this context, it's totally reasonable for people to look at your (in my opinion poor) track record of prediction and adjust their credence in your effectiveness as an institution.
I disagree re: motte and bailey; the above is not at all in conflict with the position of the book (which, to be clear, I endorse and agree with and is also my position).
re: "you can imagine," I strongly encourage people to be careful about leaning too hard on their own ability to imagine things; it's often fraught and a huge chunk of the work MIRI does is poking at those imaginings to see where they collapse.
I'll note that core MIRI predictions about e.g. how machines will be misaligned at current levels of sophistication are being borne out—things we have been saying for years about e.g. emergent drives and deception and hacking and brittle proxies. I'm pretty sure that's not "rooted in the actual nuts and bolts details" in the way you're wanting, but it still feels ... relevant.
Thanks @Duncan Sabien for this excellent explanation. Don't undersell yourself, I rate your communication here at least as good (if not better) than that of other senior MIRI people in recent years.
Thank you for attempting to explain Yudkowsky’s views as you understand them.
I don’t think anybody would be convinced by an a priori argument that any and all imaginable forms of AGI or ASI are highly existentially dangerous as a consequence of being prone to the sort of alignment failures Yudkowsky imagined in the 2000s, in the pre-deep learning era, regardless of the technological paradigm underlying them, simply by virtue of being AGI or ASI. I think you have to make some specific assumptions about how the underlying technology works. It’s not clear what assumptions Yudkowsky is making about that.
A lot of Yudkowsky’s (and Yudkowsky and Soares’) arguments about why deep learning is dangerous seem to depend on very loose, hazy analogies (like this). Yudkowsky doesn’t have any expertise, education, training, or research/engineering experience with deep learning. Some deep learning experts say he doesn’t know what he’s talking about, on even a basic level. He responded to this, but was unable to defend the technical point he was making — his response was more about vaguely casting aspersions and a preoccupation with social status over and above technical matters, as is often the case. So, I’m not sure he actually understands the underlying technology well enough to make a convincing, substantive, deep, detailed version of the argument he wants to make.
Whether deep learning is dissimilar from human cognition, and specifically dissimilar in a way that makes it 99.5% likely to cause human extinction, is not some side issue, but the topic on which the whole debate depends. If deep learning is highly similar to human cognition, or dissimilar but in a way that doesn’t make it existentially dangerous, then AGI and ASI based on deep learning would not have a 99.5% chance of causing human extinction. That’s not a minor detail. That’s the whole ballgame.
Also, as an aside, it’s bewildering to me how poorly Yudkowsky and others at MIRI take criticism of their ideas. In response to any sort of criticism or disagreement, Yudkowsky and other folks’ default response seems to be to fly into a rage and to try to attack or humiliate the person making the criticism/expressing the disagreement. Yudkowsky has explicitly said he believes he’s by far the smartest person on Earth, at least when it comes to the topic of AGI safety/alignment. He seems indignant at having to talk to people who are so much less intelligent than he is. Unfortunately, his attitude seems to have become the MIRI culture. (Soares has also apparently contributed to this dynamic.)
If you’re doing technical research, maybe you can get away with that — but even then I don’t think you can, because an inability to hear criticism/disagreement makes you much worse at research. But now that MIRI has pivoted to communications and advocacy, this will be an even more serious problem. If Yudkowsky and others at MIRI are incapable of engaging in civil intellectual debate, or are simply unwilling to, how on Earth are they going to be effective at advocacy and communications?
"Some experts downvote Yudkowsky's standing to opine" is not a reasonable standard; some experts think vaccines cause autism. You can usually find someone with credentials in a field who will say almost anything.
The responsible thing to do (EDIT: if you're deferring at all, as opposed to evaluating the situation for yourself) is to go look at the balance of what experts in a field are saying, and in this case, they're fairly split, with plenty of respected big names (including many who disagree with Eliezer on many questions) saying he knows enough of what he's talking about to be worth listening to. I get that Yarrow is not convinced, but I trust Hinton, who has reservations of his own but not of the form "Eliezer should be dismissed out of hand for lack of some particular technical expertise."
Also: when the experts in a field are split, and the question is one of existential danger, it seems that the splitness itself is not reassuring. Experts in nuclear physics do not drastically diverge in their predictions about what will happen inside a bomb or reactor, because we understand nuclear physics. When experts in the field of artificial intelligence have wildly different predictions and the disagreement cannot be conclusively resolved, this is a sign of looseness in everyone's understanding, and when you ask normal people on the street "hey, if one expert says an invention will kill everyone, and another says it won't, and you ask the one who says it won't where their confidence comes from, and they say 'because I'm pretty sure we'll muddle our way through, with unproven techniques that haven't been invented yet, the risk of killing everyone is probably under 5%,' how do you feel?"
they tend to feel alarmed.
And that characterization is not uncharitable—the optimists in this debate do not have an actual concrete plan. You can just go check. It all ultimately boils down to handwaving and platitudes and "I'm sure we'll stay ahead of capabilities [for no explicable reason]."
And we're intentionally aiming at something that exceeds us along the very axis that led us to dominate the planet, so ... ?
Another way of saying this: it's very, very weird that the burden of proof on this brand-new and extremely powerful technology is "make an airtight case that it's dangerous" instead of "make an airtight case that it's a good idea." Even a 50/50 shared burden would be better than the status quo.
I'll note that
In response to any sort of criticism or disagreement, Yudkowsky and other folks’ default response seems to be to fly into a rage and to try to attack or humiliate the person making the criticism/expressing the disagreement.
The responsible thing to do is to go look at the balance of what experts in a field are saying, and in this case, they're fairly split
This is not a crux for me. I think if you were paying attention, it was not hard to be convinced that AI extinction risk was a big deal in 2005–2015, when the expert consensus was something like "who cares, ASI is a long way off." Most people in my college EA group were concerned about AI risk well before ML experts were concerned about it. If today's ML experts were still dismissive of AI risk, that wouldn't make me more optimistic.
Oh, I agree that if one feels equipped to go actually look at the arguments, one doesn't need any argument-from-consensus. This is just, like, "if you are going to defer, defer reasonably." Thanks for your comment; I feel similarly/endorse.
This seems like a motte-and-bailey. The question at hand is not about experts’ opinions on the general topic of existential risk from AGI, but specifically their assessment of Yudkowsky’s competence at understanding deep learning. You can believe that deep learning-based AGI is a serious existential risk within the next 20 years and also believe that Yudkowsky is not competent to understand the topic at a technical level.
As far as I know, Geoffrey Hinton has only commented on Yudkowsky’s high-level comments about existential risk from AGI — which is a concern Hinton shares — and not said anything about Yudkowsky’s technical competence on deep learning.
If you know any examples of prominent experts in deep learning vouching for Yudkowsky’s technical competence in deep learning, specifically, I invite you to give citations.
Yudkowsky has said he believes he’s by far the smartest person in the world at least when it comes to AI alignment/safety — as in, the second smartest doesn’t come close — and maybe the smartest person in the world in general. AI alignment/safety has been his life’s work since before he decided — seemingly sometime in the mid-to-late 2010s — deep learning was likely to lead to AGI. MIRI pays him about $600,000 a year to do research. By now, he’s had plenty of opportunity to learn about deep learning. Given this, shouldn’t he show a good grasp on concepts in deep learning? Shouldn’t he be competent at making technical arguments about deep learning? Shouldn’t he be able to clearly, coherently explain his reasoning?
It seems like Yudkowsky must at least be wrong about his own intelligence because if he really were as intelligent as he thinks, he wouldn’t struggle with basic concepts in deep learning or have such a hard time defending the technical points he wants to make about deep learning. He would just be able to make a clear, coherent case, demonstrating an understanding of the definitions of widely-used terms and concepts. Since he can’t do that, he must be overestimating his own abilities by quite a lot.
In domains other than AI, such as Japanese monetary policy, he has expressed views with a similar level of confidence and self-assurance as what he says about deep learning that turned out to be wrong, but, notably, never acknowledged the mistake. This speaks to Clara Collier’s point about not updating his views based on his new evidence. It’s not clear that any amount of evidence would (at least publicly) change his mind about any topic where he would lose face if he admitted being wrong. (He’s been wrong many times in the past. Has this ever happened before?) And if he doesn’t understand deep learning in the first place, then the public shouldn’t care whether he changes his mind or not.
You would most likely get fired for agreeing with me about this, so I can’t reasonably expect you to agree, but I might as well say the things that people on the payroll of a Yudkowsky-founded organization can’t say. For me, the cost isn’t losing a job, it’s just a bit of negative karma on a forum.
Sorry to be so blunt, but you’re asking for $6M to $10M to be redirected from possibly the world’s poorest people or animals in factory farms — or even other organizations working on AI safety — to your organization, led by Yudkowsky, so that you can try to influence policy on a national U.S. and international scale. Yudkowsky has indicated if his preferred policy were enacted at an international scale, it might increase the risk of wars. This calls for a high level of scrutiny. No one should accept weak, flimsy, hand-wavy arguments about this. No one should tiptoe around Yudkowsky’s track record of false or extremely dubious claims, or avoid questioning his technical competence in deep learning, which is in serious doubt, out of fear or politeness. If you, MIRI, or Yudkowsky don’t want this level of scrutiny, don’t ask for donations from the EA community and don’t try to influence policy.
About the unexplained shift of focus from symbolic AI, which Yudkowsky was still claiming as of around 2015 or 2016
This is made up, as far as I can tell (at least re: symbolic AI as described in the wikipedia article you link). See Logical or Connectionist AI? (2008):
As it so happens, I do believe that the type of systems usually termed GOFAI will not yield general intelligence, even if you run them on a computer the size of the moon.
[Edited on 2026-01-18 at 20:30 UTC to add: I misremembered some important details about my exchanges on Facebook with Eliezer Yudkowsky and another person at MIRI, Rob Bensinger, around 2016-2018. Take my recollections below as unreliable memory. I went through the trouble of digging up those old Facebook comments and detailed what I found here.]
This is made up, as far as I can tell
Nope, it was Yudkowsky in a Facebook group about AI x-risk around 2015 or 2016. He specifically said he didn't think deep learning was the royal road to AGI.
I'm not sure what sort of system Yudkowsky had in mind, specifically, when he said he thought symbolic AI approaches were more likely to get there. He didn't give more details. Maybe he saw a difference between symbolic AI and "the type of systems usually termed GOFAI". Some people seem to draw a distinction between the two, whereas other people say they're just the same thing. Maybe he changed his mind about GOFAI between 2008 and whenever that discussion took place. I don't know. I'm only going off of what he said, and he didn't explain this.
In any case, at some point after 2008 and before 2023 he changed his mind about deep learning, and he never explained why, as far as I can tell. This seems like an important topic to discuss, and not a minor detail or an afterthought. What gives?
Again, it's not even clear that Yudkowsky even understands deep learning particularly well, so to apply his pre-deep learning theory to deep learning specifically, we need a substantive explanation from him. It's the kind of thing that if you were writing a book about this topic (or many long-form posts over the span of years), you would probably want to address.
The review of Collier's review that you linked does not, in my view, adequately address the point that I raised from Collier's review. The author of the review's review does not demonstrate to me that they understand Collier's point. They might understand it or they might not, but it's not clear from the review whether they do or don't, so there isn't the basis for a convincing reply, there.
By the way, the last time we interacted on the EA Forum, you refused to retract a false accusation against me after I disproved it. I gave you the opportunity to apologize and try to have a good faith discussion from that point, but you didn't apologize and you didn't retract the accusation. Given this, I don't particularly have much patience for engaging with you further. Take care.
Nope, it was Yudkowsky in a Facebook group about AI x-risk around 2015 or 2016. He specifically said he didn't think deep learning was the royal road to AGI.
Nope, it was Yudkowsky in a Facebook group about AI x-risk around 2015 or 2016. He specifically said he didn't think deep learning was the royal road to AGI.
Would you be able to locate the post in question? If Yudkowsky did indeed say that, I would agree that it would constitute a relevant negative update about his overall prediction track record.
MIRI is running its first fundraiser in six years, targeting $6M. The first $1.6M raised will be matched 1:1 via an SFF grant. Fundraiser ends at midnight on Dec 31, 2025. Support our efforts to improve the conversation about superintelligence and help the world chart a viable path forward.
MIRI is a nonprofit with a goal of helping humanity make smart and sober decisions on the topic of smarter-than-human AI.
Our main focus from 2000 to ~2022 was on technical research to try to make it possible to build such AIs without catastrophic outcomes. More recently, we’ve pivoted to raising an alarm about how the race to superintelligent AI has put humanity on course for disaster.
In 2025, those efforts focused around Nate Soares and Eliezer Yudkowsky’s book (now a New York Times bestseller) If Anyone Builds It, Everyone Dies, with many public appearances by the authors; many conversations with policymakers; the release of an expansive online supplement to the book; and various technical governance publications, including a recent report with a draft of an international agreement of the kind that could actually address the danger of superintelligence.
Millions have now viewed interviews and appearances with Eliezer and/or Nate, and the possibility of rogue superintelligence and core ideas like “grown, not crafted” are increasingly a part of the public discourse. But there is still a great deal to be done if the world is to respond to this issue effectively.
In 2026, we plan to expand our efforts, hire more people, and try a range of experiments to alert people to the danger of superintelligence and help them make a difference.
To support these efforts, we’ve set a fundraising target of $6M ($4.4M from donors plus 1:1 matching on the first $1.6M raised, thanks to a $1.6M matching grant), with a stretch target of $10M ($8.4M from donors plus $1.6M matching).
If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.
We do not mean that as hyperbole. We are not exaggerating for effect. We think that is the most direct extrapolation from the knowledge, evidence, and institutional conduct around artificial intelligence today. In this book, we lay out our case, in the hope of rallying enough key decision-makers and regular people to take AI seriously. The default outcome is lethal, but the situation is not hopeless; machine superintelligence doesn't exist yet, and its creation can yet be prevented.
The leading AI labs are explicitly rushing to create superintelligence. It looks to us like the world needs to stop this race, and that this will require international coordination. MIRI houses two teams working towards that end:
A communications team working to alert the world to the situation.
A governance team working to help policymakers identify and implement a response.
Activities
Communications
If Anyone Builds It, Everyone Dies has been the main recent focus of the communications team. We spent substantial time and effort preparing for publication, executing the launch, and engaging with the public via interviews and media appearances.
Content creators have also made videos explaining parts of the book, or that mention the book in their broader coverage of the topic.
The book received strong blurbs from the national security establishment (e.g., from a retired three-star general and a number of former directors of the White House National Security Council) and from top scientists and academics (e.g., George Church, Yoshua Bengio, and Ben Bernanke). The book was boosted by endorsements from an extremely wide mix of public figures, from Whoopi Goldberg to Matt Yglesias to Steve Bannon. (See also earlier discussion and takeaways about the initial blurbs we received.)
The end goal is not media coverage, but a world in which people understand the basic situation and are responding in a reasonable, adequate way. It seems early to confidently assess the book's impact, but we see promising signs.
The possibility of rogue superintelligence is now routinely mentioned in mainstreamcoverage of the AI industry. We’re finding in our own conversations with strangers and friends that people are generally much more aware of the issue, and taking it more seriously. Our sense is that as people hear about the problem through their own trusted channels, they are more receptive to concerns.
Our conversations with policymakers feel meaningfully more productive today than they did a year ago, and we have been told by various U.S. Members of Congress that the book had a valuable impact on their thinking. It remains to be seen how much this translates into action. And there is still a long way to go before world leaders start coordinating an international response to this suicide race.
Today, the MIRI comms team comprises roughly seven full-time employees (if we include Nate and Eliezer). In 2026, we’re planning to grow the team. For example:
We need someone whose job is to track AI developments and how the global conversation is responding to those developments, and help coordinate a response.
We need someone to assess and measure the effectiveness of various types of communications and arguments, and notice what’s working and what’s not.
We need someone to track and maintain relationships with various colleagues and allies (such as neighboring organizations, safety teams at the labs, journalist contacts, and so on) and make sure the right resources are being deployed at the right times.
We will be making a hiring announcement soon, with more detail about the comms team’s specific models and plans. We are presently unsure (in part due to funding constraints/budgetary questions!) whether we will be hiring one or two new comms team members, or many more.
Going into 2026, we expect to focus less on producing new content, and more on using our existing library of content to support third parties who are raising the alarm about superintelligence for their own audiences. We also expect to spend more time responding to news developments and taking advantage of opportunities to reach new audiences.
Governance
Our governance strategy primarily involves:
Figuring out solutions, from high-level plans to granular details, for how to effectively halt the development of superintelligence.
Engaging with policymakers, think tanks, and others who are interested in developing and implementing a response to the growing dangers.
There's a ton of work still to be done. To date, the MIRI Technical Governance Team (TGT) has mainly focused on high-level questions such as "Would it even be possible to monitor AI compute relevant to frontier AI development?" and "What would an international halt to the superintelligence race look like?" We're only just beginning to transition into more concrete specifics, such as writing up A Tentative Draft of a Treaty, with Annotations, which we published on the book website to coincide with the book release, followed by a draft international agreement.
We plan to push this a lot further, and work towards answering questions like:
What, exactly, are the steps that could be taken today, assuming different levels of political will?
If there is will for chip monitoring and verification, what are the immediate possible legislative next-steps? What are the tradeoffs between the options?
Technologically, what are the immediate possible next steps for, e.g., enabling tamper-proof chip usage verification? What are the exact legislative steps that would require this verification?
We need to extend that earlier work into concrete, tractable, shovel-ready packages that can be handed directly to concerned politicians and leaders (whose ranks grow by the day).
To accelerate this work, MIRI is looking to support and hire individuals with relevant policy experience, writers capable of making dense technical concepts accessible and engaging, and self-motivated and competent researchers.[2]
We’re also keen to add additional effective spokespeople and ambassadors to the MIRI team, and to free up more hours for those spokespeople who are already proving effective. Thus far, the bulk of our engagement with policymakers and national security professionals has been done either by our CEO (Malo Bourgon), our President (Nate Soares), or the TGT researchers themselves. That work is paying dividends, but there’s room for a larger team to do much, much more.
In our conversations to date, we’ve already heard that folks in government and at think tanks are finding TGT’s write-ups insightful and useful, with some calling it top-of-its-class work. TGT’s recent outputs and activities include:
In addition to collaborating with Nate, Eliezer, and others to produce the treaty draft, the TGT has further developed this document into a draft international agreement, along with a collection of supplementary posts that expand on various points.
The team published a research agenda earlier this year. Much of their work (to date and going forward) falls under this agenda, which is further explored in a number of papers digging into various specifics. TGT has also participated in relevant conferences and workshops, and has been supervising and mentoring junior researchers through external programs.
TGT regularly provides input on RFCs and RFIs from various governmental bodies, and engages with individuals in governments and elsewhere through meetings, briefings, and papers.
Current efforts are mostly focused on the U.S. federal government, but not exclusively. For example, in 2024 and 2025, TGT participated in the EU AI Act Code of Practice Working Groups, working to make EU regulations more likely to be relevant to misalignment risks from advanced AI. Just four days ago, Malo was invited to provide testimony to a committee of the Canadian House of Commons; and TGT researcher Aaron Scher was invited to speak to the Scientific Advisory Board of the Secretary-General of the UN on AI verification as part of an expert panel.
The above isn’t an exhaustive description of what everyone at MIRI is doing; e.g., we continue to support a small amount of in-house technical alignment research.
As noted above, we expect to make hiring announcements in the coming weeks and months, outlining the roles we’re hoping to add to the team. But if your interest has already been piqued by the general descriptions above, you’re welcome to reach out to [email protected]. For more updates, you can subscribe to our newsletter or periodically check our careers pages (MIRI-wide, TGT-specific).
Fundraising
Our goal at MIRI is to have at least two years’ worth of reserves on hand. This enables us to plan more confidently: hire new staff, spin up teams and projects with long time horizons, and balance the need to fundraise with other organizational priorities. Thanks to generous support we received in 2020 and 2021, we didn’t need to run any fundraisers in the last six years.
We expect to hit December 31st having spent approximately $7.1M this year (similar to recent years[3]), and with $10M in reserves if we raise no additional funds.[4]
Going into 2026, our budget projections have a median of $8M[5], assuming some growth and large projects, with large error bars from uncertainty about the amount of growth and projects. On the upper end of our projections, our expenses would hit upwards of $10M/yr.
Thus, our expected end-of-year reserves puts us $6M shy of our two-year reserve target of $16M.
This year, we received a $1.6M matching grant from the Survival and Flourishing Fund, which means that the first $1.6M we receive in donations before December 31st will be matched 1:1. We will only receive the grant funds if it can be matched by donations.
Therefore, our fundraising target is $6M ($4.4M from donors plus 1:1 matching on the first $1.6M raised). This will put us in a good place going into 2026 and 2027, with a modest amount of room to grow.
It’s an ambitious goal and will require a major increase in donor support, but this work strikes us as incredibly high-priority, and the next few years may be an especially important window of opportunity. A great deal has changed in the world over the past few years. We don’t know how many of our past funders will also support our comms and governance efforts, or how many new donors may step in to help. This fundraiser is therefore especially important for informing our future plans.
We also have a stretch target of $10M ($8.4M from donors plus the first $1.6M matched). This would allow us to move much more quickly on pursuing new hires and new projects, embarking on a wide variety of experiments while still maintaining two years of runway.
For more information or assistance on ways to donate, view our Donate page or contact [email protected].
The default outcome of the development of superintelligence is lethal, but the situation is not hopeless; superintelligence doesn't exist yet, and humanity has the ability to hit the brakes.
With your support, MIRI can continue fighting the good fight.
In addition to growing our team, we plan to do more mentoring of new talent who might go on to contribute to TGT's research agenda, or who might contribute to the field of technical governance more broadly.
Our yearly expenses in 2019–2024 ranged from $5.4M to $7.7M, with the high point in 2020 (when our team was at its largest), and the low point in 2022 (after scaling back).
It’s worth noting that despite the success of the book, book sales will not be a source of net income for us. As the authors noted prior to the book’s release, “unless the book dramatically exceeds our expectations, we won’t ever see a dime”. From MIRI’s perspective, the core function of the book is to try to raise an alarm and spur the world to action, not to make money; even with the book’s success to date, the costs to produce and promote the book have far exceeded any income.
Our projected expenses are roughly evenly split between Operations, Outreach, and Research, where our communications efforts fall under Outreach and our governance efforts largely fall under Research (with some falling under Outreach). Our median projection breaks down as follows: $2.6M for Operations ($1.3M people costs, $1.2M cost of doing business), $3.2M Outreach ($2M people costs, $1.2M programs), and $2.3M Research ($2.1M people costs, $0.2M programs). This projection includes roughly $0.6–1M in new people costs (full-time-equivalents, i.e., assuming the people are not all hired on January 1st).
Note that the above is an oversimplified summary; it's useful for high-level takeaways, but for the sake of brevity, I've left out a lot of caveats, details, and explanations.
I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences.
Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts:
* Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do
* NEST is an opinionated ideas-fi...
Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job.
My channel is using that same cinematic, high-production value beauty to talk about everything in the EA universe that isn't AI.
...
IMO it would help to see a concrete list of MIRI's outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically.
If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for those, anything you can say about the more dangerous ones (e.g. number of work hours that went into them, what class of project they were, whether they're still live, any downstream effect you can point to, and so on).
(Speaking in my capacity as someone who currently works for MIRI)
I think the degree to which we withheld work from the public for fear of accelerating progress toward ASI might be a little overrepresented in the above. We adopted a stance of closed-by-default research years ago for that reason, but that's not why e.g. we don't publish concrete and exhaustive lists of outputs and budget.
We do publish some lists of some outputs, and we do publish some degree of budgetary breakdowns, in some years.
But mainly, we think of ourselves as asking for money from only one of the two kinds of donors. MIRI feels that it's pretty important to maintain strategic and tactical flexibility, to be able to do a bunch of different weird things that we think each have a small chance of working out without exhaustive justification of each one, and to avoid the trap of focusing only on clearly legible short chains of this—>that (as opposed to trying both legible and less-legible things).
(A colleague of mine once joked that "wages are for people who can demonstrate the value of their labor within a single hour; I can't do that, which is why I'm on a salary." A similar principle applies here.)
In the past, funding MIRI led to outputs like our alignment research publications. In the more recent past, funding MIRI has led to outputs like the work of our technical governance team, and the book (and its associated launch campaign and various public impacts).
That's enough for some donors—"If I fund these people, my money will go into various experiments that are all aimed at ameliorating existential risk from ASI, with a lean toward the sorts of things that no one else is trying, which means high variance and lots of stuff that doesn't pan out and the occasional home run."
Other donors are looking to more clearly purchase a specific known product, and those donors should rightly send fewer of their dollars to MIRI, because MIRI has never been and does not intend to ever be quite so clear and concrete and locked-in.
(One might ask "okay, well, why post on the EA forum, which is overwhelmingly populated by the other kind of donor, who wants to track the measurable effectiveness of their dollars?" and the answer is "mostly for the small number who are interested in MIRI-like efforts anyway, and also for historical reasons since the EA and rationality and AI safety communities share so much history." Definitely we do not feel entitled to anyone's dollars, and the hesitations of any donor who doesn't want to send their money toward MIRI-like efforts are valid.)
That makes some sense, but leaves me with questions like
MIRI is hardly unique even in the EA/rat space in having special projects - Rethink Priorities, for e.g., seem to be very fluid in what they work on; Founders Pledge and Longview are necessarily driven to some degree by the interests of their major donors; Clean Air Task force have run many different political campaigns, each seemingly unlike the previous ones in many ways; ALLFED are almost unique in their space, so have huge variance in the projects they work on; and there are many more with comparable flexibility.
And many EA organisations in the space that don't explicitly have such a strategy have nonetheless pivoted after learning of a key opportunity in their field, or realising an existing strategy was failing.
In order to receive funds - at least from effectiveness-minded funders - all these orgs have to put a certain amount of effort into answering questions like those above.
And ok, you say you're not claiming to be entitled to dollars, but it still seems reasonable to ask why a rational funder should donate to MIRI over e.g. any of the above organisations - and to hope that MIRI has some concrete answers.
Noting that this is more "opinion of an employee" than "the position of MIRI overall"—I've held a variety of positions within the org and can't speak for e.g. Nate or Eliezer or Malo:
I think it's hard to generalize lessons, because various projects are championed by various people and groups within the org ("MIRI" is nearly a ship of Theseus). But some very basic lessons include:
(Sorry, that feels sort of useless, but.)
From my limited perspective/to the best of my ability to see and descrbe, budget is essentially allocated in a "Is this worth doing? If so, how do we find the resources to make it work?" sense. MIRI's funding situation has always been pretty odd; we don't usually have a pie that must be divided up carefully so much as a core administrative apparatus that needs to be continually funded + a preexisting pool of resources that can be more or less freely allocated + a sense that there are allies out there who are willing to fund specific projects if we fall short and want to make a compelling pitch.
Unfortunately, I can't really draw analogies that help an outsider evaluate future projects. We're intending to try stuff that's different from anything we've tried before, which means it's hard to draw on the past (except insofar as the book and surrounding publicity were also something we'd never tried before, so you can at least a little bit assess our ability to pivot and succeed at stuff outside our wheelhouse by looking at the book).