Thanks for this question Stephen.  Let me start with my overall perspective on the probability of use:  Over the 'deep time' periods about which I have learned much from the EA Community -- periods of hundreds or thousands of years – I believe a CNE is a near certainty unless we dramatically reduce this risk.  For a while, states acquired nuclear weapons roughly at the rate of one every five years; this rate slowed after the Nuclear Non-Proliferation Treaty was negotiated, but has never stopped.  Over the next century, we will either manage the risks of nuclear technology more effectively, or the number of actors with nuclear weapons will increase until someone makes a mistake.  After that, our ability to manage these risks may be destroyed completely.

That said, I also want to share some more insight into the 0.5% figure that I shared at the EA Global meeting.  I thought it was important for me to offer the EA community my own assessment about the probability of a catastrophic nuclear event (CNE). 

I want to be clear, however, that any probability assessment in this space is, of necessity, a judgment call, no matter who provides it.  The nuclear “system” is a massive, complex, system of many sub-systems, and it’s impossible to quantify the risks of use the way one might, for example, conduct a probabilistic risk assessment of the likelihood of failure of a specific engineered technology (where specific data about system performance over time can be employed to calculate probabilities).  With the nuclear weapons system, we are talking about thousands of individual weapon and delivery systems, tens of thousands of humans in the decision-making and launch execution loop, hundreds of thousands of technical (and digital) components, and constantly evolving political dynamics and uncertainties which are not quantifiable beyond a qualitative judgment. 

Ultimately, I’ve given you my judgment about the probability of nuclear use, on average, in any given year based on:

• my experience working within the nuclear system and my understanding of the system design and its vast complexity;
• my knowledge of (and in one case direct experience with) failures in the nuclear weapons system over the last 7 decades that led to loss of life or weapons being lost, damaged, or almost used -from numerous operational accidents, incidents, near misses, component failures, faulty warnings, etc (see examples here: www.motherjones.com/politics/2014/11/timeline-nuclear-weapons-accidents-mishaps-near-misses/ and here: www.newyorker.com/news/news-desk/world-war-three-by-mistake);
• my understanding and experience of the culture of control of the weapons and its weak points, (for example after the cold war ended, the culture of nuclear control in the US military declined significantly – see a report on this here:   https://www.hsdl.org/?abstract&did=486958)
• my understanding and experience of global politics and the dynamics of great power conflict.

The key point I wanted to illustrate with my probability assessment is that even if we think the risk of use is small in any given year, the cumulative risk is unacceptably large over time.  I did not say, but perhaps should have highlighted, that nuclear risk is not static, and some years my judgment about the risk would be much higher than 0.5%.  For example, we are watching Russian forces mass along the border with Ukraine as I write, and tensions are very high right now between the US/Europe and Russia.  In this moment of heightened risk, and until we back away from posturing our forces for conflict, I would say the risk of use is much higher, perhaps by as much as a factor of 10.  If that persists over time, and between more nuclear adversaries, the probability of use over time becomes much higher.  Sadly, I suspect that’s the world we are heading for – an increasing use probability – if we don’t succeed in dramatically reducing risks through a variety of measures that are policy, technical and political in nature.

Thanks for this very thoughtful question, Tessa.

1. What are the most important problems for this project, and what is stopping you (/NTI) from working on them right now?

The Biosecurity Innovation and Risk Reduction Initiative (BIRRI) was born from the recognition that advances in biotechnology are rapidly outpacing the ability of governments to provide effective oversight. Through BIRRI, we are working to identify efforts that would be most impactful in safeguarding science and reducing the risk of catastrophic events that could result from deliberate abuse or accidental misuse of bioscience and biotechnology.  

The research ecosystem that enables biotechnological developments is diverse—involving publishers, academic scientists, private industry, research funders, bioethicists, managers, security experts, statisticians, and more. Although each group has a shared interest in the overall success of research, their near-term priorities may not always be aligned. BIRRI seeks to bring together these diverse stakeholders to identify and advance practical tools that, when paired with appropriate incentive structures, have a real chance at reducing risks. 

Projects under the umbrella of the initiative were collaboratively developed and are designed to cut across the research lifecycle, providing a layered defense against deliberate or accidental misuse of biotechnologies.

We believe all five of the initiatives you listed are critical components of this layered defense, and we’re working on each of these to different degrees. 

For example, NTI is partnering with Stanford University on the Visibility Initiative for Responsible Science in an effort to understand, improve, and standardize risk reduction practices among communities involved in research, with a particular focus on funders and publishers. 

We’re also partnering with the World Economic Forum and global leaders from industry, academic, philanthropy, and international organizations to develop an international Common Mechanism for DNA synthesis screening to protect this critical service from malicious actors who may seek to exploit it to cause harm.

To create systemic and sustainable change in the global biosecurity architecture, NTI and partners have also prioritized efforts to establish the International Biosecurity and Biosafety Initiative for Science (IBBIS), a new organization that will work collaboratively to strengthen global biosecurity norms and develop innovative tools to uphold them. The goal of IBBIS will be to safeguard science and reduce the risk of catastrophic events that could result from deliberate abuse or accidental misuse of bioscience and biotechnology.

IBBIS is slated to launch in 2022 and will begin with a narrow focus on improving DNA synthesis screening practices internationally with the flexibility to expand its remit over time to encompass, but also expand beyond, the other goals outlined in each initiative. 

We see our current approach as developing urgently needed, risk-reducing pilot efforts while simultaneously building out the larger institutional structure required to fully implement and sustain these efforts.  

2. What is the limiting factor on the project’s growth and progress?

We’re pleased with the progress of our efforts, however, ultimately, limited team resources – both in terms of human and financial resources - is a limiting factor through much of our work. There are many gaps in the international biosecurity architecture and many important solutions to develop. To make the best use of our limited staff time, we try to prioritize projects that are important, timely, and have the potential to advance biosecurity in the long term. 

Additionally, engaging stakeholders and building communities is crucial to our work. Strong relationships take time to build, but are essential to gaining trust, avoiding “one-off” solutions, and ensuring that our efforts create sustainable and durable positive change.

3. What problems in this project are the largest order of magnitude? What changes could you make that would result in a 100x or 1000x increase in this project’s positive impact?

Referring to a previous point about developing a common mechanism for screening DNA synthesis orders, creating a tool and set of standards to simplify screening is an important first step to increasing the depth and breadth of screening coverage.

Coupling the development of tools like the common mechanism with the launch and sustainment of a global organization like IBBIS, with the capacity to work with diverse partners to identify, socialize, and implement stronger biosecurity norms though both practical tools and accompanying incentive structures, will increase the positive impact of these efforts on risk reduction. This will require a careful design and approach and NTI is working with an international steering group to shape this effort. 

Thanks for asking.  I see a high degree of alignment and overlap between the EA community’s framing and cause areas, and NTI’s work.  At NTI, our vision is a world safe from preventable global catastrophe, and our mission is to transform global security by driving systemic solutions to nuclear and biological threats imperiling humanity.  It’s been exciting for me to get to know the EA community better and I look forward to exploring ways we can leverage each others’ strengths, capacities and skills to expand our collective impact.  At NTI, we have a team of some of the world’s best experts in nuclear and biosecurity risk reduction.  We are happy to provide our advice and expertise to members of the EA community who are interested in bio and nuclear risk reduction.  

Members of the EA community can help us build our capacity for action, through helping to build global awareness about the risks to humanity of catastrophic risks, by choosing careers in nuclear and bio risk reduction, by providing financial resources to institutions doing impactful work in this issue space, and through helping to build the political space for action.  After watching the nuclear and bio fields decline for several decades, it’s great to see the building momentum of effective altruism!  Many in our traditional NGO community were beginning to feel like a dying tribe trying to protect a small patch of the galaxy that humanity calls home.  I think the EA community represents the new vanguard, and a new generation of ideas and energy for problem solving associated with catastrophic threats to humanity.  We are excited to work together!

Thanks for this question. I want to begin by saying how enormously appreciative we are for MacArthur’s support over the years. 

The MacArthur Foundation announced at the beginning of 2021 that they were ending their funding for the nuclear field after more than 40 years of support for civil society groups doing important nuclear risk reduction work.  By MacArthur’s own estimate, their nuclear funding represented around 45% of philanthropic funding for civil society groups working on nuclear issues (principally in the US).  The nuclear field was already small, fragile and declining even before the MacArthur withdrawal, so their departure is expected to have a significant impact on the sustainability and capacity of many civil society groups in this space.  It’s causing an already neglected field to shrink further, absorb painful staffing cuts and in some cases will force closures. That’s what I meant by “a big blow.”

Regarding their reasons, I can’t speak for MacArthur, but I can make the following observations: MacArthur’s decision coincides with a change of leadership at the foundation at multiple levels.  A new President was recently appointed at MacArthur and he set out to do a review of the foundation’s grantmaking and realign its priorities (as one might expect of a new leader).  As public awareness and understanding of, and interest in, nuclear issues has waned since the end of the Cold War, so too had the knowledge and expertise of MacArthur leadership and Board on nuclear issues.  Around the same time as the new President taking the helm, quite coincidentally, the foundation’s lead nuclear program officer departed the foundation for a leadership position with a different foundation.  So, by mid-2020, there were no champions left at MacArthur to help guide and advocate for the nuclear portfolio, and the foundation decided to invest its resources in other areas deemed more relevant by the new President and the Board (Climate and Social Justice issues). 

It’s important to note here that MacArthur hired a consultant to review their nuclear portfolio as part of their strategic review and the consultant’s report concluded that MacArthur funding had enabled important achievements within the nuclear space.  But the report also found that MacArthur’s most recent investment in a specific nuclear project, a “Big Bet”, was not likely to payoff within the remaining five-year target they had set.  Instead of updating their nuclear “Big Bet” with a different strategy or a bet more likely to succeed, MacArthur decided to scrap nuclear funding entirely.  They never did offer an explanation for that decision.

One of my key takeaways: the nuclear community has some really fundamental work to do to make the case to funders and to global publics about our collective global stake in nuclear risk reduction.  We’ve not done a good job of making nuclear issues relevant for lay audiences, or of connecting nuclear risks to our day-to-day lives, or the sustainability of humanity’s future. In that sense, the MacArthur departure is a wake-up call that we must pay attention to.  We are now working on this!

Yes and yes! NTI has worked with Andrew in recent years, and his book does a good job of explaining just how dangerous cyber threats to nuclear weapons systems can be – whether in the U.S., UK, Russia, China, or the other states with nuclear weapons. There’s no question that nuclear weapons and related systems are increasingly vulnerable to sophisticated cyberattacks, and nuclear-armed countries have a responsibility to cooperate and accelerate efforts to prevents an attack that could have catastrophic consequences. But they need help. Developing innovative, actionable ways to reduce cyber nuclear risks—including to nuclear planning systems, early-warning systems, communications systems, and nuclear weapon delivery systems—is a priority for our team, and we’ve gathered some of the foremost experts in the world to study the issue.

To learn more, I recommend Nuclear Weapons in the New Cyber Age, a report from our Cyber-Nuclear Weapons Study Group that explores four plausible scenarios for a cyberattack and offers  recommendations for how to reduce risks as part of the U.S. government’s modernization of nuclear weapons now underway. You can also read about our work to bring together U.S. and Russian experts to find technical and policy solutions.

We need to do both – in parallel and with urgency – until we mitigate both risks. At NTI, we use a mix of strategies to maximize our effectiveness in each area. We engage with stakeholders and governments, we develop and field test innovative ideas, we convene and consult with the world’s best experts – and then we adjust our strategies as we go. Through it all, we use our evidence-based findings and solutions to keep the pressure on global leaders, whether in government or the private sector, to prioritize risk reduction in both areas. 

For nuclear risk reduction, I believe AI/ML and the growing availability of big data are potentially game changing.  In a world where the vast majority of collective human actions and behaviors leave a digital footprint, we now have a set of new tools for detecting illicit nuclear behavior (like illicit trade in nuclear dual use components, or the production of weapons grade uranium or plutonium). Like any new technology, we have to work to capture the benefits while minimizing the risks they bring, so we at NTI are looking at both sides.

Last year NTI published a report on a pilot project we undertook with a partner, C4ADS, that demonstrated how we were able to use some large trade datasets and machine learning to detect illicit nuclear trade. After we published our work, a number of the entities we found were added to government sanctions lists. We believe this work can be significantly expanded to build and demonstrate an entirely new approach to a verification system– one that enables a safer nuclear strategy for the future.

In April, the National Academies published an Interim Report, (led by Jill Hruby) on their ongoing exploration of Nuclear Proliferation and Arms Control Monitoring, Detection, and Verification.  The Interim Report finds that technological advances provide unprecedented opportunities for staying ahead of complex and expanding monitoring, detection, and verification challenges; and that monitoring, detection, and verification technology should be a higher national security priority with a long-term vision and regular evaluation of progress.  Technological change is creating transformational possibilities for nuclear threat reduction and we should work diligently to develop national and global innovation ecosystems that can deliver on this promise.

On the flip side, AI/ML and automated decision-making also pose new challenges and risks for the nuclear system.  In August, NTI published a report titled Assessing and Managing the Benefits and Risks of Artificial Intelligence in Nuclear-Weapon Systems.  This report, by Jill Hruby, then Sam Nunn Distinguished Fellow (and now Administrator of the National Nuclear Security Administration) and former NTI intern and MIT doctoral student M. Nina Miller explores the possible applications of AI to nuclear-weapons systems and assesses the benefits, risks, and strategic stability implications.  The report recommends: 1) research on low technical-risk approaches and fail-safe protocols for AI use in high-consequence applications; 2) that states with nuclear weapons make declaratory statements about the role of human operators in nuclear-weapon systems and/or the prohibition or limits of AI use in their nuclear-weapon systems; and 3) increased international dialog on the implications of AI use in nuclear-weapon systems, including how AI could affect strategic and crisis stability, and explore areas where international cooperation or development of international norms, standards, limitations, or bans could be beneficial.

On the biosecurity side, we’re really excited about our work on a prototype to globally expand DNA synthesis screening practices and to prevent exploitation by malicious actors and misuse. 

NTI | bio, with the support of talented technical consultants and in partnership with the World Economic Forum, has developed several critical elements of the Common Mechanism prototype, which improves on current industry best practices. The Common Mechanism prototype includes: (1) novel databases of ‘benign’ and ‘biorisk’ sequences informed by publicly available literature and industry expertise and inputs and (2) and a screening algorithm to compare incoming orders with the contents of the aforementioned databases. A key feature of the algorithm is that it uses statistical models known as ‘profile hidden Markov models,’ which have been shown through early tests to be resilient against subversion attempts. 

As a next step, the project team will construct a “decision support system” which will present the results of the screening algorithm, while providing important context about the kind of organism the sequence comes from, the function it might have, and a recommendation regarding whether to proceed with the order. These tools will allow DNA synthesis providers to perform an effective and efficient assessment as to whether the DNA synthesis order in question is benign or constitutes a risk. 

Future steps include acquiring additional data sets to test the Common Mechanism prototype, improving the quality of the reference databases, and rigorously testing the Common Mechanism with close partners and a wider circle of other peer reviewers.

They are inextricably linked. The 2021 Global Health Security Index – a project led by NTI in partnership with the Johns Hopkins Center for Health Security –  found that 94% of countries have no national level oversight measures for dual use research, no agency responsible for the oversight, and no evidence of national assessments of dual use research. Additionally, there is no international entity that has dedicated, as its top priority, efforts to strengthen biosecurity and bioscience governance and to reduce emerging biological risks associated with technology advances. While the development of risk-reducing technologies is critical, to ensure these tools are applied and that actors are held accountable for reducing risk, we must also make progress on the policy front. 

NTI’s efforts to design and launch the International Biosecurity and Biosafety Initiative for Science (IBBIS) recognize the need to focus on both aspects of biosecurity, which is clear in the mission:  IBBIS works collaboratively to strengthen global biosecurity norms and develop innovative tools to uphold them. We undertake this work to safeguard science and reduce the risk of catastrophic events that could result from deliberate abuse or accidental misuse of bioscience and biotechnology.     

Technology in the nuclear weapon space is quickly evolving. I can imagine numerous scenarios where emerging technologies could lead to nuclear escalation and devastating conflict. For example, there are plausible scenarios involving a conflict that evolves from a cyber attack or cyber interference in nuclear command and control systems. During a moment of high tension, such an attack could trigger a nuclear response. This could also happen due to a cyber attack from a nonstate actor. Confusion about intent and attribution could lead to a breakdown in nuclear deterrence. Nuclear-armed states cannot completely prevent such scenarios (short of nuclear disarmament), but they can reduce their likelihood by agreeing to rules of the road that would prohibit cyber attacks against nuclear weapons and command and control and warning systems. Moreover, nuclear-armed states could conduct their own “failsafe reviews” to consider additional steps they could take unilaterally or potentially cooperatively to reduce the risk of unauthorized or miscalculated use of nuclear weapons.

For more on the risks and need for a fail-safe review process, see the recent Washington Post op-ed by NTI Co-Chairs Ernie Moniz and Sam Nunn.

Thanks for asking, Stephen. Please see my answer above to Hauke Hillebrandt's question on MacArthur.

We have to pursue multiple strategies for reducing nuclear risks – no one strategy alone is sufficient. Because governments possess nuclear weapons and have the resources necessary for implementing risk reduction measures at the scale that’s needed, it’s imperative to continue to leverage policy change by governments as a core focus of nuclear risk reduction efforts. This includes generating creative ideas for policy solutions that governments could adopt. But a strategy that focuses only on persuading government leaders to adopt policy changes has proven to be insufficient on its own.  The adoption of nuclear risk reduction measures has recently been outpaced by growing risks, and policy makers’ attention to nuclear issues has been declining as the awareness and attention of global publics has also declined.

So we must also aim to drive a strategy of culture change by working to raise awareness of nuclear risks among broader segments of the public.  We have work to do to communicate more effectively about the risks of nuclear weapons to global publics.  The more awareness we can create, the more likely it is that governments will feel they have the political support (the Overton window) for reducing the threat of nuclear weapons.  

 Individuals can educate themselves on the dangers posed by nuclear weapons. There are many places to learn about nuclear risks and nuclear policy (including on NTI’s website!).  A few specific reading suggestions include:

•  2034: A Novel of the Next World War by Admiral James Stavridis;
• My Journey at the Nuclear Brink by former U.S. Secretary of Defense William Perry, and
• Command and Control by Eric Schlosser. 

These could be discussed in book groups or informal dinners with friends and neighbors. Individuals can then make their views known by communicating to political leaders their concerns about the danger of nuclear weapons and the need for governments to do more to reduce them. Even small actions, such as hosting a book group or dinner party and discussing these issues can go a long way to raise awareness.

My bad, the first paragraph should have my thesis statement right? That and the remaining paragraphs are the supporting arguments. Good luck. The HTML embedding is not working for links. Their is probably a breakdown somewhere but this site needs a better UI.

My point was abstract threats are not motivating. People are terrified of war. But we have been beaten down and have no political agency. For instance, your comment sucks Optimist. You got the moderator mad. Yeah I hope so. People want this problem solved myself included, but the academics are worried about funding and impact factors.

Threads and the day after were the reason the stockpile was reduced. Diplomats had no impact on it until the public was motivated.

Unfortunately I think there are multiple pathways to nuclear use or an exchange involving several pairings or groupings of states with nuclear weapons including: US-Russia and scenarios that could also involve the UK and France along with the US; US-China; India- Pakistan; China – India; DPRK – US; and potentially Iran and other countries should Iran decide to build a nuclear weapon, not to mention the potential for terrorists to get hold of nuclear weapons or materials.  

So I believe our priorities in the nuclear space must be first to build awareness of the true risks and recognize that the risk is increasing that nuclear weapons will be used again; second to demand that governments pursue policies and concrete actions that will reduce the risks of nuclear use; and third build political will to ultimately end nuclear weapons as a threat by eliminating them and implementing safeguards for a world in which nuclear technology exists and will continue to be used for civilian purposes, but where possession of nuclear weapons is verifiably banned. 

The reality is that as long as other states possess nuclear weapons, the United States will continue to maintain nuclear forces for deterrence.  I believe their purpose should be only to deter the use of nuclear weapons by others, and that the US should maintain the minimum number of forces we believe we need to serve that purpose.  At the same time, the US must vigorously pursue further nuclear reductions and limits with Russia, and eventually, multilateral arms control to reduce and ultimately eliminate the nuclear forces of all states that have them including China.  We must also strengthen the nuclear nonproliferation regime to prevent the emergence of new nuclear weapons states. It is essential to return to an agreement with Iran to make sure it does not develop nuclear weapons, and we need to reinvigorate diplomacy with North Korea to stop, reverse and ultimately eliminate its nuclear weapons program while providing security and economic incentives in return.    

In my view, the most immediate catastrophic/existential threats facing humanity are biological risks, nuclear risks,  climate change, and AI. 

To address these global catastrophic risks to humanity, the world must first and foremost recognize the enormity of these threats and the catastrophic consequences of not addressing them with the urgency or creativity required to solve them.  

Second, these threats have in common the reality that their mitigation will require unprecedented global cooperation and an approach that takes advantage of the expertise, perspectives and wisdom of the “whole of society.” No one state can mitigate these threats on its own or isolate itself from their consequences. 

Third, they all require radically new ways of thinking and working together to devise solutions. As humans we tend to think linearly, but we are entering an unprecedented period of disruption and discontinuity, where incremental, linear solutions are unlikely to produce the kind of change we need at the pace we need it.  We need to untether from the past in thinking about the future and remain open to novel solutions.

Fourth,  enhanced political will is required to do what is necessary to avert the catastrophic course the world is on in all four of these dimensions. The EA community can play a really significant role in building the political will for change that is so urgently needed.