Cross posted on lesswrong https://www.lesswrong.com/posts/qkvk22oc3YeEJpEfC/agi-safety-and-losing-electricity-industry-resilience-cost

Below is a paper about to be submitted. The focus is on interventions that could improve the long-term outcome given catastrophes that disrupt electricity/industry, such as solar storm, high-altitude electromagnetic pulse (HEMP), narrow AI computer virus, and extreme pandemic. Work on these interventions is even more neglected than interventions for feeding everyone if the sun is blocked. Cost-effectiveness is compared to a slightly modified AGI safety cost-effectiveness model posted earlier on the EA forum. Two different cost-effectiveness estimates for losing industry interventions were developed: one by Denkenberger and a poll at EA Global San Francisco 2018, and the other by Anders Sandberg at Future of Humanity Institute. There is great uncertainty in both AGI safety and interventions for losing industry. However, the models have ~99% confidence that funding interventions for losing industry now is more cost effective than additional funding for AGI safety beyond the expected ~$3 billion. This does not take into account model or theory uncertainty, so the confidence would likely decrease. However, in order to make AGI safety more cost effective, this required changing four variables in the Sandberg model to the 5th percentile on the pessimistic end simultaneously. For the other model, it required changing seven variables. Therefore, it is quite robust that a significant amount of money should be invested in losing industry interventions now. There is closer to 50%-88% confidence that spending the ~$40 million on interventions for losing industry is more cost effective than AGI safety. Overall, AGI safety is more important and more total money should be spent on it. The modeling concludes that additional funding would be justified on both causes even for the present generation.

Long Term Cost-Effectiveness of Interventions for Loss of Electricity/Industry Compared to Artificial General Intelligence Safety

David Denkenberger 1,2, Anders Sandberg 3, Ross Tieman *1, and Joshua M. Pearce 4,5

1. Alliance to Feed the Earth in Disasters (ALLFED), Fairbanks, AK 99775, USA
2. University of Alaska Fairbanks, Fairbanks, AK 99775, USA
3. Future of Humanity Institute, University of Oxford, Oxford, UK
4. Department of Material Science and Engineering and Department of Electrical and Computer Engineering, Michigan Technological University, Houghton, MI 49931, USA
5. Department of Electronics and Nanoengineering, School of Electrical Engineering, Aalto University, FI-00076 Espoo, Finland
* corresponding author

Abstract

Extreme solar storms, high-altitude electromagnetic pulses, and coordinated cyber attacks could disrupt regional/global electricity. Since electricity basically drives industry, industrial civilization could collapse without it. This could cause anthropological civilization (cities) to collapse, from which humanity might not recover, having long-term consequences. Previous work analyzed technical solutions to save nearly everyone despite industrial loss globally, including transition to animals powering farming and transportation. The present work estimates cost-effectiveness for the long-term future with a Monte Carlo (probabilistic) model. Model 1, partly based on a poll of Effective Altruism conference participants, finds a confidence that industrial loss preparation is more cost effective than artificial general intelligence safety of ~88% and ~99+% for the 30 millionth dollar spent on industrial loss interventions and the margin now, respectively. Model 2 populated by one of the authors produces ~50% and ~99% confidence, respectively. These confidences are likely to be reduced by model and theory uncertainty, but the conclusion of industrial loss interventions being more cost effective was robust to changing the most important 4-7 variables simultaneously to their pessimistic ends. Both cause areas save expected lives cheaply in the present generation and funding to preparation for industrial loss is particularly urgent.

Disclaimer/Acknowledgements: Funding was received from the Centre for Effective Altruism. Anders Sandberg received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 669751). The Oxford Prioritisation Project developed the artificial general intelligence safety cost effectiveness submodel. Owen Cotton-Barratt, Daniel Dewey, Sindy Li, Ozzie Gooen, Tim Fist, Aron Mill, Kyle Alvarado, Ratheka Stormbjorne, and Finan Adamson contributed helpful discussions. This is not the official position of the Centre for Effective Altruism, the Future of Humanity Institute, nor the Alliance to Feed the Earth in Disasters (ALLFED).

1. Introduction

The integrated nature of the electric grid, which is based on centralized generation makes the entire system vulnerable to disruption.(1) There are a number of anthropogenic and natural catastrophes that could result in regional-scale electrical grid failure, which would be expected to halt the majority of industries and machines in that area. A high-altitude electromagnetic pulse (HEMP) caused by a nuclear weapon could disable electricity over part of a continent (Bernstein, Bienstock, Hay, Uzunoglu, & Zussman, 2012; Foster et al., 2004; Kelly-Detwiler, 2014; Oak Ridge National Laboratory, 2010). This could destroy the majority of electrical grid infrastructure, and as fossil fuel extraction and industry is reliant on electricity (Foster, Jr et al., 2008), industry would be disabled. Similarly, solar storms have destroyed electrical transformers connected to long transmission lines in the past (Space Studies Board, 2008). The Carrington event in 1859 damaged telegraph lines, which was the only electrical infrastructure in existence at the time. It also caused Aurora Borealis that was visible in Cuba and Jamaica (Klein, 2012). This could potentially disable electrical systems at high latitudes, which could represent 10% of electricity/industry globally. Though solar storms may last less than the 12 hours that would be required to expose the entire earth with direct line of sight, the earth's magnetic field lines redirect the storm to affect the opposite side of the earth (Space Studies Board, 2008).

Lastly, both physical (M. Amin, 2002, 2005; Kinney, Crucitti, Albert, & Latora, 2005; Motter & Lai, 2002; Salmeron, Wood, & Baldick, 2004) and cyber attacks (Aitel, 2013; Hébert, 2013; Nai Fovino, Guidi, Masera, & Stefanini, 2011; Onyeji, Bazilian, & Bronk, 2014; Sridhar, Hahn, & Govindarasu, 2012; Umbach, 2013; Watts, 2003) could also compromise electric grids. Physical attacks include traditional acts of terrorism such as bombing or sabotage (Watts, 2003) in addition to EMP attacks. Significant actors could scale up physical attacks, for example by using drones. A scenario could include terrorist groups hindering individual power plants (Tzezana, 2016), while a large adversary could undertake a similar operation physically to all plants and electrical grids in a region.

Unfortunately, the traditional power grid infrastructure is simply incapable of withstanding intentional physical attacks (National Research Council, 2012). Damage to the electric grid resulting in physical attack could be long lasting, as most traditional power plants operate with large transformers that are difficult to move and source. Custom rebuilt transformers require time for replacement ranging from months and even up to years (National Research Council, 2012). For example, a relatively mild 2013 sniper attack on California’s Pacific Gas and Electric (PG&E) substation, which injured no one directly, was able to disable 17 transformers supplying power to Silicon Valley. Repairs and improvements cost PG&E roughly $100 million and lasted about a month (Avalos, 2014; Pagliery, 2015). A coordinated attack with relatively simple technology (e.g. guns) could cause a regional electricity disruption.

However, a high-tech attack could be even further widespread. The Pentagon reports spending roughly $100 million to repair cyber-related damages to the electric grid in 2009 (Gorman, 2009). There is also evidence that a computer virus caused an electrical outage in the Ukraine (Goodin, 2016). Unlike simplistic physical attacks, cyber attackers are capable of penetrating critical electric infrastructure from remote regions of the world, needing only communication pathways (e.g. the Internet or infected memory sticks) to install malware into the control systems of the electric power grid. For example, Stuxnet was a computer worm that destroyed Iranian centrifuges (Kushner, 2013) to disable their nuclear industry. Many efforts are underway to harden the grid from such attack (Gent & Costantini, 2003; Hébert, 2013). The U.S. Department of Homeland Security responded to ~200 cyber incidents in 2012 and 41% involved the electrical grid (Prehoda, Schelly, & Pearce, 2017). Nations routinely have made attempts to map current critical infrastructure for future navigation and control of the U.S. electrical system (Gorman, 2009).

The electric grid in general is growing increasingly dependent upon the Internet and other network connections for data communication and monitoring systems (Bessani, Sousa, Correia, Neves, & Verissimo, 2008; Schainker, Douglas, & Kropp, 2006; Sridhar et al., 2012; Ulieru, 2007; Wu, Moslehi, & Bose, 2005). Although this conveniently allows electrical suppliers management of systems, it increases the susceptibility of the grid to cyber-attack, through denial of webpage services to consumers, disruption to supervisory control and data acquisition (SCADA) operating systems, or sustained widespread power outages (Aitel, 2013; Krotofil, Cardenas, Larsen, & Gollmann, 2014; Sridhar et al., 2012; Ten, Manimaran, & Liu, 2010). Thus global or regional loss of the Internet could have similar implications.

A less obvious potential cause is a pandemic that disrupts global trade. Countries may ban trade for fear of the disease entering their country, but many countries are dependent on imports for the functioning of their industry. If the region over which electricity is disrupted had significant agricultural production, the catastrophe could be accompanied by a ~10% food production shortfall as well. It is uncertain whether countries outside the affected region would help the affected countries, do nothing, or conquer the affected countries.

Larger versions of these catastrophes could disrupt electricity/industry globally. For instance, it is possible that multiple HEMPs could be detonated around the world, due to a world nuclear war (Pry, 2017) or due to terrorists gaining control of nuclear weapons. There is evidence that, in the last 2000 years, two solar storms occurred that were much stronger than the Carrington event (Mekhaldi et al., 2015). Therefore, it is possible that an extreme solar storm could disable electricity and therefore industry globally. It is conceivable that a coordinated cyber or physical attack (or a combination) on many electric grids could also disrupt industry globally. Many of the techniques to harden the electric grid could help with this vulnerability as well as moving to more distributed generation and microgrids (Che & Shahidehpour, 2014; Colson, Nehrir, & Gunderson, 2011; Lasseter, 2007; Lasseter & Piagi, 2004; Prehoda et al., 2017; Shahidehpour & Khodayar, 2013). An extreme pandemic could cause enough people to not show up to work such that industrial functioning could not be maintained. Though this could be mitigated by directing military personnel to fill vacant positions, if the pandemic were severe enough, it could be rational to retreat from high human contact industrial civilization in order to limit disease mortality.

The global loss of electricity could even be self-inflicted as a way of stopping rogue artificial general intelligence (AGI) (Turchin & Denkenberger, 2018a). As the current high agricultural productivity depends on industry (e.g. for fertilizers) it has been assumed that there would be mass starvation in these scenarios (Robinson, 2007).

Repairing these systems and re-establishing electrical infrastructure would be a goal of the long term and work should ideally start on it immediately after a catastrophe. However, human needs would need to be met immediately (and continually) and since there is only a few months of stored food, it would likely run out before industry is restored with the current state of preparedness. In some of the less challenging scenarios, it may be possible to continue running some machines on the fossil fuels that had previously been brought to the surface or from the use microgrids or shielded electrical systems. In addition, it may be feasible to run some machines on gasified wood (Dartnell, 2014). However, in the worst-case scenario, all unshielded electronics would be destroyed.

Here we focus on catastrophes that only disrupt electricity/industry, rather than catastrophes that could disable industry and obscure the sun (Cole, Denkenberger, Griswold, Abdelkhaliq, & Pearce, 2016) or catastrophes that only obscure the sun (or affect crops directly in other ways) ( Denkenberger & Pearce, 2015b). This paper analyzes the cost effectiveness of interventions from a long term perspective. First, this study will review interventions to both avoid a loss of electricity, but also to feed everyone with this loss. Then the benefits of artificial general intelligence (AGI) safety on the long term future will be reviewed and quantified. Next, two loss of industry interventions submodels are developed. The cost for an intervention based on alternative food communication is estimated.

2. Background

2.1 Review of Potential Solutions

An obvious intervention for HEMP is preventing a nuclear exchange, which would be the best outcome. However, it is not neglected, as it has been worked on for many decades (Barrett, Baum, & Hostetler, 2013; D. C. Denkenberger & Pearce, 2018; Helfand, 2013; McIntyre, 2016a; Turchin & Denkenberger, 2018b) and is currently funded at billions of dollars per year quality adjusted (McIntyre, 2016b). Other obvious interventions for HEMP that would also work for solar storms, and coordinated physical or cyber threats would be hardening the electrical grid against these threats. However, hardening just the U.S. electrical grid against solar storm and HEMP would cost roughly $20 billion (Pry, 2014). Therefore globally, just from these two threats it would be around $100 billion. Furthermore, adding hardening to cyber threats would be even more expensive. Again, preventing the collapse of electricity/industry would be the preferable option, but given the high cost, it may not happen. Even if it occurs eventually, it would still be preferable to have a backup plan in the near term and in the case that hardening is unsuccessful at stopping loss of industry.

A significant problem in loss of industry catastrophes is that of food supply (Cole et al., 2016). One intervention is storing years worth of food, but it is too expensive to have competitive cost effectiveness (and it would take many years so it would not protect humanity right away, and it would exacerbate current malnutrition) (Baum, Denkenberger, & Pearce, 2016). Furthermore, if electricity/industry is disabled for many years, food storage would be impractical. Stockpiling of industrial goods could be another intervention, but again it would be much more expensive than the interventions considered here.

Interventions for food production given the loss of industry include burning wood from landfills to provide fertilizer and high use of nitrogen fixing crops including legumes (peas, beans, peanuts, etc.) (Cole et al., 2016). Also, nonindustrial pest control could be used. Despite pre-industrial agricultural productivity (~1.3 dry tons per hectare per year) (Cole et al., 2016), this could feed everyone globally. However, not everyone would be nearby the food sources, and losing industry would severely hamper transportation capability. Solutions for this problem include backup plans for producing more food locally, including expanding planted area (while minimizing impact to biodiversity e.g. by expanding into the boreal forest/tundra enhanced by the nutrients from tree decomposition/combustion) and favoring high calorie per hectare foods such as potatoes, yams, sweet potatoes, lentils, and groundnuts (Oke, Redhead, & Hussain, 1990). Though clearing large areas of forest with hand saws would not be practical, it is possible to girdle the trees (remove a strip of bark around the circumference), let the trees dry out, and burn them. This has the advantage of releasing fertilizer to the soils. Another option involves producing “alternative foods,” which were proposed for sun-blocking catastrophes (D. Denkenberger & Pearce, 2014). Some of these alternative foods would require industry, but producing non-industrial lower cost ones such as extracting calories from leaves (D. Denkenberger, Pearce, Taylor, & Black, 2019) could be feasible. For transporting the food and other goods, ships could be modified to be wind powered and animals could pull vehicles (Abdelkhaliq, Denkenberger, Griswold, Cole, & Pearce, 2016). A global network of shortwave radio transmitters and receivers would facilitate disseminating the message that there is a plan and people need not panic, and also allow for continuing coordination globally (see below).

Current awareness of interventions given loss of electricity/industry (hereafter “interventions”) is very low, likely in the thousands of people. Also, many of the interventions are theoretical only and need to be tested experimentally. There may be a significant amount of shortwave radio systems that are shielded from HEMP and have shielded backup power systems, but likely some addition to this capacity would be beneficial. This paper analyzes the cost effectiveness of interventions from a long term perspective. It is unlikely that the loss of industry would directly cause human extinction. However, by definition, there would be a loss of industrial civilization for the global catastrophes. Furthermore, there could be a loss of anthropological civilization (basically cities or cooperation outside the clan). One definition of the collapse of civilization involves short-term focus, loss of long distance trade, widespread conflict, and collapse of government (Coates, 2009). Reasons that civilization might not recover include: i) easily accessible fossil fuels and minerals are exhausted (Motesharrei, Rivas, & Kalnay, 2014) (though there would be minerals in landfills), ii) the future climate might not be as stable as it has been for the last 10,000 years (Gregory et al., 2007), or iii) technological and economic data and information might be lost permanently because of the trauma and genetic selection of the catastrophe (Bostrom, 2013). If the loss of civilization were prolonged, a natural catastrophe, such as a super volcanic eruption or an asteroid/comet impact, could cause the extinction of humanity. Another way to far future impact is the trauma associated with the catastrophe making future catastrophes more likely, e.g. global totalitarianism (Bostrom & Cirkovic, 2008). A further route is worse values caused by the catastrophe could be locked in by artificial general intelligence (AGI) (Bostrom, 2014), though with the loss of industrial civilization, the advent of AGI would be significantly delayed, so the bad values could have decayed out by then.

2.2 Artificial General Intelligence

AGI itself represents a major, independent risk. The artificial intelligence available now is narrow AI, i.e. it can generally only do a specific task, such as playing Jeopardy! (Schaul, Togelius, & Schmidhuber, 2011). However, there are concerns that as AI systems become more advanced, AGI will eventually be achieved (Bostrom, 2014). Since AGI could perform all human tasks as well as or better than humans, this would include reprogramming the AGI. This would enable recursive self-improvement, so there could be an intelligence explosion (Good, 1966). Since the goals of the intelligence may not be aligned with human interests (Bostrom, 2014) and could be pursued with great power, this implies a potentially serious risk (Good, 1966). AGI safety is a top priority in the existential risk community that seeks to improve humanity’s long term future (Turchin & Denkenberger, 2018b). Though there is uncertainty in when and how AGI may be developed, there are concrete actions that can be taken now to increase the probability of a good outcome (Amodei et al., 2016).

We seek to compare the cost effectiveness of losing industry interventions with AGI safety to discover whether these interventions should also be a top priority. Comparisons to other risks, such as asteroids (Matheny, 2007), climate change (Halstead, 2018) and pandemics (Millett & Snyder-Beattie, 2017), are possible, though these are generally regarded by the existential risk community as lower priority and therefore less informative.

3. Methods

Given the large uncertainties in input parameters, we model cost-effectiveness using a Monte Carlo simulation, producing a probability distribution of cost-effectiveness. Probabilistic uncertainty analysis is used widely in insurance, decision-support and cost-effectiveness modelling (Garrick, 2008). In these models, uncertain parameters are represented by samples drawn from defined distributions that are combined into output samples that form a resultant distribution.

The models consist of a loss of industry submodel estimating the risk and mitigation costs of industrial loss, and an AGI risk submodel estimating risk and mitigation costs of AGI scenarios. These two submodels then allow us to estimate the ratio and confidence of cost-effectivenesses.

Monte Carlo estimation was selected because the probability distributions for various parameters do not come in a form that provides analytically tractable combinations. It also allows exploring parameter sensitivity.

The open source software called Guesstimate(2) was originally used to implement the models, and they are available online. However, to enable more powerful analysis and plotting, the models were also implemented on the software Analytica 5.2.9. Combining the uncertainties in all the inputs was performed utilizing a Median Latin Hypercube analysis (similar to Monte Carlo, but better performing (Keramat & Kielbasa, 1997)) with the maximum uncertainty sample of 32,000 (run time on a personal computer was seconds). The results from the two software agreed within uncertainties due to finite number of samples, giving greater confidence in the results.

Figures 1 to 4 illustrate the interrelationships of the nodes for Model 1; Model 2 is identical with the following exception. The input variable Mitigation of far future impact of industrial loss from ALLFED so far for 10% industrial loss node was removed from Model 1 due to the poll question not requiring this input.

Figure 1. Model overview

Figure 2. 100% Industry loss catastrophes submodel (10% industry loss is nearly identical)

Figure 3. AGI safety cost effectiveness submodel

Figure 4. Overall cost effectiveness ratios

3.1 Loss of Industry Interventions Submodel

Table 1 shows the key input parameters for Model 1 (largely Denkenberger and conference poll of effective altruists)(D. Denkenberger, Cotton-Barrat, Dewey, & Li, 2019a) and Model 2 (D. Denkenberger, Cotton-Barratt, Dewey, & Li, 2019) (Sandberg inputs)(3). Though the authors here are associated with research on loss of industry, two out of four also published in AGI safety. Also, opinions outside of the loss of industry field have been solicited for one of the models. Therefore, we believe the results are representative. All distributions are lognormal unless otherwise indicated. The absolute value of the long term future is very difficult to quantify, so losses are expressed as a percent.

Table 1. Losing industry interventions input variables

The potential causes of the disabling of 1/10 of global industry include Carrington-type solar storm, single HEMP, coordinated physical or cyber attack, conventional world war, loss of the Internet, and pandemic disrupting trade. We are not aware of quantitative estimates of the probability of a coordinated cyber attack, loss of the Internet, a pandemic that significantly disrupts trade, or a conventional world war that destroys significant industry and does not escalate to the use of nuclear weapons. Quantitative model estimates of the probability of full-scale nuclear war between the U.S. and Russia such as (Barrett et al., 2013) may give some indication of the probability of HEMP. HEMP could accompany nuclear weapons destroying cities, and this would be a combination losing industry/losing the sun scenario, which would benefit from the preparation considered here. Asymmetric warfare, where one country is significantly less powerful than another, could use HEMP because it only requires one or two nuclear weapons to disable an entire country. There are significantly more nuclear pairs that could result in HEMP than could result in full-scale nuclear war (the latter is basically the dyads between US, Russia, and China). And yet one quantitative model estimate of the probability of full-scale nuclear war only between U.S. and Russia was 1.7% per year mean (Barrett et al., 2013). In 2012, there was a near miss of a solar storm similar size to the Carrington event (Baker et al., 2013). One probability estimate of a Carrington-sized event is ~0.033% per year (Roodman, 2015). However, an estimate of the probability per year of a superflare 20 times as powerful as the Carrington event is 0.1%/year (Lingam & Loeb, 2017), which disagrees by orders of magnitude for the same intensity. Another study proposes that a Carrington-sized event recurrence interval is less than one century (Hayakawa et al., 2019). Given the large uncertainty of solar storms and significant probability of single EMP, pandemic and regional cyber attack, Model 1 uses a mean of 3% per year. Model 2 uses a mean of 0.4% per year.

Intuitively, one would expect that the probability of near-total loss of industry would be significantly lower than 10% loss of industry. Complete loss of industry may correspond to the superflares that may have occurred in the first millennium A.D. (~0.1% per year). We are not aware of quantitative estimates of the probability of multiple EMP, industry-halting pandemic or global cyber attack. Model 1 mean is 0.3% per year for near-total loss of industry. Model 2 mean is 0.09% per year.

At the Effective Altruism Global 2018 San Francisco conference, with significant representation of people with knowledge of existential risk, a presentation was given and the audience was asked about the 100% loss of industry catastrophes. The questions involved the reduction in far future potential due to the catastrophes with current preparation and if ~$30 million were spent to get prepared. The data from the poll were used directly instead of constructing continuous distributions.

To determine the marginal impact of additional funding, the contribution due to work so far should be quantified. The Alliance to Feed the Earth in Disasters (ALLFED)(ALLFED, 2019) (and ALLFED researchers before the organization was officially formed) have published several papers on interventions for losing industry. They have a website with these papers and summaries. They have also run workshops to investigate planning for these interventions. However, we expect the contribution of ALLFED to reducing the long term impact of loss of industry to be significantly lower than in the case of obscuring of the sun because the loss of the Internet may be immediate if there are multiple simultaneous EMPs. However, the loss of electricity may not be simultaneous globally due to cyber attack. Furthermore, there may be several days warning for an extreme solar storm. The other reason why current work may be less valuable in a global loss of industry scenario is that fewer people know about the loss of industry work of ALLFED than the food without the sun work. Model 1 estimates a reduction in long-term future potential loss from a global loss of industry due to ALLFED so far as a mean of 0.1%. Model 2 uses 0.004% due to emphasizing lack of communication scenarios.

In the case of a 10% loss of industry, with the exception of the scenario of loss of Internet everywhere, the Internet in most places would be functioning. Even if the Internet is not functioning, mass media would generally be functioning. Therefore, possible mechanisms for impact due to work so far include the people already aware of the interventions getting the message to decision makers/media in a catastrophe, decision makers finding the three papers (Abdelkhaliq et al., 2016; Cole et al., 2016; David C Denkenberger et al., 2017) on these interventions, or the people in the media who know about these interventions spreading the message. However, even though people outside of the affected countries could get the information, it may not be feasible to get the information to the people who need it most. Model 2 estimates a reduction in long-term future potential loss from a global loss of industry due to ALLFED so far as a mean of 0.004%, again due to the likely lack of communications in the affected region. Model 1 does not use a value in its calculation.

The mean estimate of the conference participants was 16% reduction in the long-term future of humanity due to loss of global industry with current preparedness. Model 2 estimate mean was 7%.

The 10% industry loss catastrophes could result in instability and full scale nuclear war or other routes to far future impact. Though the poll was not taken for this level of catastrophe, a survey of GCR researchers estimated a mean of 13% reduction in long-term potential of humanity due to a 10% food shortfall (Denkenberger, Sandberg, & Pearce, unpublished results). Some 10% loss of industry catastrophes could cause a ~10% global food shortfall. However, if the affected area were largely developed countries, since they would likely need to become near vegan to survive, human edible food demand could fall 10% because of the reduction of feeding animals. Still, given the possible overlap of these catastrophes, this analysis uses the survey estimate for Model 1. Model 2 estimate mean is 0.4% reduction in long-term potential due to 10% loss of industry.

The means of the percent further reduction in far future loss due to global loss of industry due to spending ~$30 million were 40% for the poll and 3% for Model 2. Note that in Model 1, the poll did not ask for the further reduction in far future loss from spending money, but instead a new far future loss after the money was spent. Therefore, the 40% mean further reduction is a calculated value and does not appear in Table 1. For the 10% industrial shortfalls, our estimate of the mean reduction is 12% for Model 1 because the contribution of additional spending on the aid from outside the affected region would be smaller. On the other hand, it was 5% for Model 2 because he thought the likelihood of success would be greater than for the global loss of industry given the outside aid.

Moral hazard would occur if awareness of interventions makes catastrophes more likely or more intense. Global use of EMP or coordinated cyber attack could be perpetrated by a terrorist organization trying to destroy civilization. However, if the organization knew of backup plans that could maintain civilization, the terrorist might actually be deterred from attempting such an attack. This would result in negative moral hazard (additional benefit of preparation). However, it is possible that knowledge of a backup plan could result in people expending less effort to harden systems to EMP, solar storm or cyber attack, creating moral hazard. Therefore, Model 1 uses a mean moral hazard of zero, and Model 2 uses a point value of zero.

For the 10% loss of industry scenarios, the same moral hazard values are used as for the global loss of industry.

3.2 Costs of Interventions

The costs of the proposed interventions are made up of a backup communication system, developing instructions and testing them for distributed food production, and making response plans at different levels of governments.

Currently the long distance shortwave radio frequencies are used by government and military stations, ships at sea, and by amateur (ham) radio operators. Because of security considerations, data on the number of government/military stations is difficult to compile. The use by ships has declined because of the availability of low cost satellite phones but there are an estimated three million ham operators worldwide (Silver, 2004). Not all of those are licensed to use the shortwave bands, however. In the U.S., about half of the approximately 800,000 American ham operators do hold the necessary license. Assuming such a pattern worldwide that would mean potentially about 1.5 million ham radio shortwave stations globally.

However, this analysis conservatively ignores the possibility that there would be existing ham radios that are disconnected with unplugged backup power systems. Therefore, the cost of the backup communication system of 5 million USD is based on the cost of 10 larger two-way shortwave communication systems (with backup power) that can transmit across oceans (see Appendix A). Then there would be 4000 smaller one-way shortwave receivers (with backup power) that, when connected to a laptop computer and printer, would have the ability to print out information. This could be called REcovering Civilization Using Radio (RECUR). This would cover 80% of the world’s population within one day nonmotorized transportation distance (~40 km) according to Geographical Information Systems (GIS) analysis (Fist et al., unpublished results). It is critical to very quickly get the message out that there is a plan and not to panic. Subsequent communication would be instructions for meeting basic needs immediately like food, shelter, and water. This initial planning would be considered open-loop control because it would not have immediate feedback (Liptak, 2018).

In the ensuing months, as reality always deviates from plans, feedback would be required. This could be accomplished by coordinating additional undamaged shortwave and electrical generation equipment to allow two-way communication for many cities. Also, depending on distance, some messages could be communicated through non-electronic means such as horses, smoke signals, and sun reflecting heliographs of the kind that were used in the Western USA before telegraphs (Rolak, 1975; Sterling, 2008).

Instructions would include how to get safe water or treat it (e.g. by filling containers including cleaned bathtubs with water in water towers and treating with bleach for a limited amount of time, solar water pasteurization (Burch et al., 1998; ) or boiling). Additional instructions would be on how to keep warm if it is cold outside (Abdelkhaliq et al., 2016). Other instructions would be how to retrofit a light duty vehicle to be pulled by a large animal. Because cattle and horses can eat food that is not edible to humans and because the wheel is so efficient, this would be a much more effective way of moving people than people walking. Additional instructions would be how to create wood-burning stoves and hand and animal farming tools, e.g. from repurposed or landfill materials. A similar project is Open Source Ecology, where blueprints have been developed of essential equipment for civilization that can be made from scratch (Open Source Ecology, 2019). All of this should be tested on realistically untrained people and the instructions should be modified accordingly.

Planning involves determining where different people would need to be relocated in order to have their basic needs met. The critical short-term factors are shelter and water, while food is slightly longer term. The economically optimal plan could be achieved with GIS analysis. However, in order for this to be politically feasible, there would need to be negotiations and precommitments. This may have similar cost to the government planning for food without the sun of $1 million to $30 million (Denkenberger & Pearce, 2016).

Overall, Model 1 estimates the communications, instructions/testing, and planning for global industry loss would cost roughly 30 million USD (see Table 1). For the regional loss of industry, it is difficult to predict where it might occur, so generally communications and planning should be done for the entire world, and thus the instructions/experiments would be similar. Therefore, there is a high correlation of preparation for the two catastrophes, so this is assumed to be the cost of the preparation to both scales of catastrophe. Model 2 has somewhat higher costs ($50 million mean).

The time horizon of effectiveness of the interventions would depend on the intervention. Modern shortwave radio communications equipment has few moving parts (chiefly cooling fans and motors to rotate directional antennas) and serviceability measured in decades.(5)

Furthermore, these systems need to be disconnected from the grid to be protected from HEMP. This would reduce wear and tear, but regular testing would be prudent. Some of the budget could be used for this and for repair of the units. As for the instructions, since the hand and animal tools are not changing, directions should stay relevant. Planning within governments is susceptible to turnover, but some money could be used to transfer the knowledge to new employees. Model 1 estimates a 25 year mean for the time horizon. Model 2 has a slightly shorter time horizon mean of 20 years driven by a conservative estimate of the communications equipment lifetime.

3.3 Artificial Intelligence Submodel

The submodel for AGI safety cost-effectiveness was based on work of the Oxford Prioritisation Project, Owen Cotton-Barratt and Daniel Dewey (both while at the Future of Humanity Institute at the University of Oxford) (D. Denkenberger, Cotton-Barrat, Dewey, & Li, 2019b; Li, 2017). We modified it (Denkenberger et al., unpublished results), with major changes including increasing the cost of an AGI safety researcher, making better behaved distributions, removing one method of calculation and changing the analysis from average to marginal for number of researchers. These changes increased the cost effectiveness of AGI safety by roughly a factor of two and increased the uncertainty considerably (because the method of calculation retained had much greater uncertainty than the one removed). The cost-effectiveness was found at the margin assuming $3 billion expenditure.

4. Results and Discussion

4.1 Results

In order to convert average cost effectiveness to marginal for interventions, we use logarithmic returns (Cotton-Barratt, 2014), which results in the relative marginal cost effectiveness being one divided by the cumulative money spent. An estimate is needed of the cumulative money spent so far for interventions. Under $100,000 equivalent (mostly volunteer time) has been spent so far directly on this effort, nearly all by ALLFED. A very large amount of money has been spent on trying to prevent nuclear war, hardening military installations to HEMP, and on cyber security. However, note that even though US military infrastructure is supposedly hardened to EMP, it may not be able to withstand a “super” EMP weapon that some countries may possess (P. Pry, 2017) or sophisticated cyber attacks. More relevant, money has been spent on farming organically and less industrially for traditional sustainability reasons. Also, Open Source Ecology has developed instructions for critical equipment. These could be tens of millions of dollars that would have needed to be spent for catastrophe preparation. So this would be relevant for the marginal $30 million case. However, there are still very high value interventions that should be done first, such as collecting instructions for producing hand/animal farm tools without industry and giving them to at least some governments and owners of disconnected shortwave radios and backup power sources. Though the interventions would not work as well as with ~$30 million of research/communications backup, simply having some critical people know about them and implement them in their own communities/countries without trade could still significantly increase the chance of retaining anthropological civilization. The cost of these first interventions would be very low, so they would have very high cost effectiveness.

Table 2 shows the ranges of the far future potential increase per $ due to loss of industry preparation average over ~$30 million Model 1, average over ~$50 million for Model 2, and AGI safety research at the $3 billion margin. The distributions are shown in Figure 5. Because the variance of Model 1 is very high, the mean cost-effectiveness is high, driven by the small probability of very high cost-effectiveness.

Table 2. Cost-effectiveness comparison

Figure 5. Far future potential increase per $ due to loss of industry preparation average over ~$30 million Model 1, due to loss of industry preparation average over ~$50 million Model 2, and AGI safety research at the $3 billion margin. Further to the right is more cost-effective.

With logarithmic returns, cost-effectivenesses of the marginal dollar now (100,000th dollar) and of the last dollar are about 50 times greater than, and 6 times less than, the average cost effectiveness of spending $30 million, respectively. For Model 2, the corresponding numbers are about 70 times greater than and 6 times less than the average cost effectiveness of spending $50 million. Ratios of mean of the distributions of cost effectivenesses are reported in Table 3.6 Comparing to AGI safety at the margin, Model 1 yields the 30 millionth dollar on losing industry being 20 times more cost effective, the average $30 million on interventions being 100 times more cost effective, and the marginal dollar now on interventions being 5000 times more cost effective (Table 3). Model 2 yields the last dollar on interventions being 0.05 times as cost effective, the average ~$50 million on interventions being 0.2 times as cost effective, and the marginal dollar now on interventions being 20 times as cost effective. Given orders of magnitude uncertainty and sensitivity of these ratios to the relative uncertainty of the interventions, likely more robust are the probabilities that one is more cost effective than the other. Comparing to AGI safety at the margin, Model 1 finds ~88% probability that the 30 millionth dollar on interventions is more cost effective, ~95% probability that the average $30 million on interventions is more cost effective, and ~99+% probability that the marginal dollar now on interventions is more cost effective (see Table 3). Model 2 finds ~50% probability that the 50 millionth dollar on interventions is more cost effective than AGI safety, ~76% probability that the average $50 million on interventions is more cost effective, and ~99% probability that the marginal dollar now on interventions is more cost effective. Note that the greater than 50% probability for the average cost effectiveness despite the ratio of the means of cost-effectiveness being less than one is due to the relatively smaller variance of Model 2 cost-effectiveness estimate (see Figure 5).

Table 3. Key cost effectiveness outputs of losing industry interventions

Overall, the mean cost-effectiveness of Model 1 is about 2.5 orders of magnitude higher than Model 2. However, due to the smaller variance in Model 2 distributions, there was similar confidence that losing industry interventions at the margin now are more cost-effective than AGI safety. Another large difference is that Model 1 found that 10% loss of industry scenarios are similar cost effectiveness for the far future as global loss. This was because the greater probability of these catastrophes counteracted the smaller far future impact. However, Model 2 rated the cost-effectiveness of the 10% industry loss as ~1.5 orders of magnitude lower than for global loss. Given the agreement of high confidence that further work is justified at this point, some of this further work could be used to resolve the significant uncertainties to determine if more money is justified: value of information (Barrett, 2017).

Being prepared for loss of industry might protect against unknown risks, meaning the cost-effectiveness would increase.

According to Model 1, every year acceleration in preparation for losing industry would increase the long term value of humanity by 0.00009% to 0.4% (mean of 0.07%). The corresponding Model 2 numbers are 0.00006% to 0.0004% (mean of 0.00017%). Either way, there is great urgency to get prepared.

It is not necessary for interventions to be more cost effective than AGI safety in order to fund losing industry interventions on a large scale. Funding in the existential risk community goes to other causes, e.g. an engineered pandemic. One estimate of cost effectiveness of biosecurity was much lower than for AGI safety and losing industry interventions, but the authors were being very conservative (Millett & Snyder-Beattie, 2017). Another area of existential risk that has received investment is asteroid impact, which again has much lower cost-effectiveness than for losing industry interventions (Matheny, 2007).

The importance, tractability, neglectedness (ITN) framework (Effective Altruism Concepts, 2019) is useful for prioritizing cause areas. The importance is the expected impact on the long-term future of the risk. Tractability measures the ease of making progress. Neglectedness quantifies how much effort is being directed towards reducing the risk. Unfortunately this framework cannot be applied to interventions straightforwardly. This is because addressing a risk could have many potential interventions. Nevertheless, some semi-quantitative insights can be gleaned. The importance of AGI is larger than industry loss catastrophes, but industry loss interventions are far more neglected.

Though these interventions for the loss of industry are not compared directly to food without the sun interventions, they are both compared to the same AGI safety submodel. Overall, Model 2 indicates that spending $50 million on interventions for the loss of industry is competitive with AGI safety. However, Model 1 here and both models for the food without sun indicate that significantly larger than the proposed amount to be spent (~$100 million) would be justified from the long-term future perspective.

The AGI safety submodel was used to estimate the cost effectiveness of saving expected lives in the present generation, finding $16-$12,000 per expected life saved ((Denkenberger et al., unpublished results). This is generally more cost effective than GiveWell estimates for global health interventions: $900-$7,000 (GiveWell, 2017). Food without the sun is significantly better ($0.20-$400 per expected life) for only 10% global food production shortfalls ( Denkenberger & Pearce, 2016) and generally better only considering one country ($1-$20,000 per expected life) and only nuclear winter ( Denkenberger & Pearce, 2016). Model 2 for interventions for losing industry has similar long term future cost-effectiveness to AGI safety, indicating that the lifesaving cost-effectiveness of interventions for losing industry would likely be competitive with AGI safety and global health, but this requires future work. Model 1 for interventions for losing industry has similar long term future cost-effectiveness to food without the sun, indicating that loss of industry preparations may save lives in the present generation less expensively than AGI safety and global health. Since AGI safety appears to be underfunded from the present generation perspective, it would be extremely underfunded when taking into account future generations. If this were corrected, then in order for interventions for losing industry to stay similar cost-effectiveness to AGI safety, more funding for losing industry interventions would be justified.

4.2 Timing of Funding

If one agrees that interventions for losing industry should be a significant part of the existential risk reduction portfolio, there remains the question of how to allocate funding to the different causes over time. For AGI safety, there are arguments both for funding later and funding now (Ord, 2014). For interventions for losing industry, since most of the catastrophes could happen right away, there is significantly greater urgency to fund interventions for losing industry now. Furthermore, it is relatively more effective to scale up the funding quickly because, through requests for proposals, the effort could co-opt relevant existing expertise (e.g. in shortwave radio). Since we have not monetized the value of the far future, we cannot use conventional cost-effectiveness metrics such as the benefit to cost ratio, net present value, payback time, and return on investment. However, in the case of saving expected lives in the present generation for the global case and 10% food shortfalls, the return on investment was from 100% to 5,000,000% per year (Denkenberger & Pearce, 2016) based on monetized life savings. This suggests that the $40 million or so for interventions for losing industry should be mostly spent in the next few years to optimally reduce existential risk (a smaller amount would maintain preparedness into the future).

4.3 Uncertainty and parameter sensitivity

Parameter sensitivities of Model 1 and Model 2 were investigated using the Analytica importance analysis function. This uses the absolute rank-order correlation between each input and the output as a measure of the strength of monotonic relations between each uncertain input and a selected output, both linear and otherwise (Chrisman et al., 2007; Morgan & Henrion, 1990). Analysis was focused on the alternative foods submodel i.e. Global loss of industry and 10% industry loss catastrophes. Parameter sensitivity within AGI safety was not investigated as this submodel was adapted from previous work by the Oxford Prioritisation Project, which discussed uncertainties within the AGI safety cost effectiveness submodel (Denkenberger et al., 2019b; Li, 2017)).

The key outputs nodes in Table 3 were unable to be investigated directly using the importance analysis function due to the node outputs being point values, a result of calculating the ratio of means (the Analytica importance analysis function requires the variable be a chance variable to perform absolute rank-order correlation). Therefore the previous node in the models Far future potential increase per $ due to loss of industry preparation was used to investigate the importance of input variables of the alternate foods submodel.

Importance analysis of node: Far future potential increase per $ due to loss of industry preparation showed Model 1 had greatest sensitivity to input variables Reduction in far future potential due to 10% industrial loss with current preparation closely followed by Reduction in far future potential due to global loss of industry with current preparation (Figure 6). Model 2 showed greatest sensitivity to input variable Cost of interventions ($ million) (global loss of industry) (Figure 7).

Figure 6. Importance analysis results for Far future potential increase per $ due to loss of industry preparation for Model 1.

Figure 7. Importance analysis results for Far future potential increase per $ due to loss of industry preparation for Model 2.

Successive rounds of parametric analysis were performed to determine combinations of input parameters sufficiently unfavorable to alternative foods, until cost effectiveness ratios (Table 3) switched to favoring AGI safety. Unfavorable input values were limited to 5th or 95th percentile values of original input distributions. Model 1 required 7 unfavorable input parameters to switch to AGI safety being more cost effective than losing industry interventions at the margin now while Model 2 required 4 input variables (see Table 4).

Table 4: Combination of input variables resulting in AGI safety being more cost effective than losing industry interventions at the margin now.

5. Conclusions and Future Work

There are a number of existential risks that have the potential to reduce the long-term potential of humanity. These include AGI and electricity/industry disrupting catastrophes including extreme solar storm, EMP, and coordinated cyber attack. Here we present the first long term future cost-effectiveness analyses for interventions for losing industry. There is great uncertainty in both AGI safety and interventions for losing industry. However, the models have 99%-99+% confidence that funding interventions for losing industry now is more cost effective than additional funding for AGI safety beyond the expected $3 billion. In order to make AGI safety more cost effective than losing industry interventions according to the mean of their distributions, this required changing four variables in Model 2 to the 5th percentile on the pessimistic end simultaneously. For Model 1, it required changing seven variables. Therefore, it is quite robust that a significant amount of money should be invested in losing industry interventions now. There is closer to 50%-88% confidence that spending the ~$40 million on interventions for losing industry is more cost effective than AGI safety. These interventions address catastrophes that have significant likelihood of occurring in the next decade, so funding is particularly urgent. Both AGI safety and interventions for losing industry save expected lives in the present generation more cheaply than global poverty interventions, so funding should increase for both. The cost-effectiveness at the margin of interventions for the loss of industry is similar to that for food without the sun (for industry versus sun, Model 1 is ~1 order of magnitude more cost effective, but Model 2 is ~1 order of magnitude less cost effective). Because the electricity/industry catastrophes could happen immediately and because existing expertise relevant to food without industry could be co-opted by charitable giving, it is likely optimal to spend most of this money in the next few years.

Since there may be scenarios of people eating primarily one food, micronutrient sufficiency should be checked, though it would be less of an issue than for food without the sun (D. Denkenberger & Pearce, 2018; Griswold et al., 2016). Higher priority future research includes ascertaining the number and distribution of unplugged shortwave radio systems with unplugged power systems that could be utilized in a catastrophe. Additional research includes the feasibility of the continuation of improved crop varieties despite loss of industry. Further research is estimating the rapidity of scale up of hand and animal powered farm tools. Estimating the efficacy of pest control without industry would be valuable. Better quantifying the capability of using fertilizer based on ash would be aided by GIS analysis. Additional work is surveying whether there have been experiments of the agricultural productivity produced by people inexperienced in farming by hand.

Another piece of future work would be to analyze the cost-effectiveness of AGI safety and preparation for the loss of industry in terms of species saved. Rogue AGI could cause the extinction of nearly all life on earth. If there were mass starvation due to the loss of electricity/industry, humans would likely eat many species to extinction. Therefore, being able to meet human needs would save species. These cost effectivenesses could be compared to the cost effectiveness of conventional methods of saving species. Finally, additional future work involves better quantifying the cost of preparedness to the loss of industry. Furthermore, research for the actual preparedness should be done, including estimating the amount of unplugged communications hardware and backup power, testing the backup communications system, experiments demonstrating the capability to quickly construct hand/animal farm tools and developing quick training to use them. Also investigating alternative food sources that do not require industry would be beneficial, such as seaweed (Mill et al., unpublished results).

Footnotes

(1) This vulnerability can be addressed with distributed generation and microgrids (S. M. Amin, 2010; Lovins & Lovins, 1982; Prehoda et al., 2017; Zerriffi, Dowlatabadi, & Strachan, 2002), but these technologies are still far from ubiquitous.

(2) One can change numbers in viewing mode to see how outputs change, but alterations will not save. If one wants to save a new version, one can make a copy of the model. Click View, visible to show arrows of relationships between cells. Drag mouse over cells to see comments. Click on the cell to show the equation.

(3) Lognormal results in the median being the geometric mean of the bounds (multiply the 5th and 95th percentiles and raise to the 0.5 power). Note that with large variances, the mean is generally much higher than the median.

(4) The global loss poll gave people ranges, including <0.1%, 0.1% to 1%, 1% to 10%, and 10% to 100%. All responses in the range were recorded as approximately the geometric mean of the range. Half of people were therefore recorded as 30% loss of the far future. If the people had been able to provide exact values, likely one of them would have recorded greater than 40%, which was the upper bound for the 10% loss of industry, making these results consistent. However, even with the constraints of the data, the mean and median are higher for the global loss of industry than the 10% loss of industry.

(5) On any given day Ebay lists numerous used shortwave radio transmitter/receivers still in fully operational condition, some of them manufactured in the 1960s.

(6) Ratios of means require manual changes in Guesstimate, which we note in all caps in the model.

Appendix A: Radio component costs - available on https://osf.io/rgq2z/

References

Abdelkhaliq, M., Denkenberger, D., Griswold, M., Cole, D., & Pearce, J. (2016). Providing Non-food Needs if Industry is Disabled.

Aitel, D. (2013). Cybersecurity Essentials for Electric Operators. The Electricity Journal, 26(1), 52–58. https://doi.org/10.1016/j.tej.2012.11.014

ALLFED. (2019, April 10). Home. Retrieved April 10, 2019, from ALLFED website: http://allfed.info/

Amin, M. (2002). Security challenges for the electricity infrastructure. Computer, 35(4), supl8–supl10. https://doi.org/10.1109/MC.2002.1012423

Amin, M. (2005). Energy Infrastructure Defense Systems. Proceedings of the IEEE, 93(5), 861–875. https://doi.org/10.1109/JPROC.2005.847257

Amin, S. M. (2010). Electricity infrastructure security: Toward reliable, resilient and secure cyber-physical power and energy systems. IEEE PES General Meeting, 1–5. IEEE.

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety. ArXiv:1606.06565 [Cs]. Retrieved from http://arxiv.org/abs/1606.06565

Avalos, G. (2014, August 27). PG&E substation in San Jose that suffered a sniper attack has a new security breach. Retrieved August 8, 2019, from The Mercury News website: https://www.mercurynews.com/2014/08/27/pge-substation-in-san-jose-that-suffered-a-sniper-attack-has-a-new-security-breach/

Baker, D. N., Li, X., Pulkkinen, A., Ngwira, C. M., Mays, M. L., Galvin, A. B., & Simunac, K. D. C. (2013). A major solar eruptive event in July 2012: Defining extreme space weather scenarios. Space Weather, 11(10), 585–591. https://doi.org/10.1002/swe.20097

Barrett, A. M. (2017). Value of GCR Information: Cost Effectiveness-Based Approach for Global Catastrophic Risk (GCR) Reduction. Forthcoming in Decision Analysis.

Barrett, A. M., Baum, S. D., & Hostetler, K. R. (2013). Analyzing and reducing the risks of inadvertent nuclear war between the United States and Russia. Sci. Global Secur., 21(2), 106–133.

Baum, S. D., Denkenberger, D. C., & Pearce, J. M. (2016). Alternative Foods as a Solution to Global Food Supply Catastrophes. Solutions.

Bernstein, A., Bienstock, D., Hay, D., Uzunoglu, M., & Zussman, G. (2012). Sensitivity analysis of the power grid vulnerability to large-scale cascading failures. ACM SIGMETRICS Performance Evaluation Review, 40(3), 33. https://doi.org/10.1145/2425248.2425256

Bessani, A. N., Sousa, P., Correia, M., Neves, N. F., & Verissimo, P. (2008). The CRUTIAL way of critical infrastructure protection. IEEE Security & Privacy, (6), 44–51.

Bostrom, N. (2013). Existential Risk Prevention as Global Priority. Global Policy, 4(1), 15–31. https://doi.org/10.1111/1758-5899.12002

Bostrom, N. (2014). Superintelligence: paths, dangers, strategies (First edition). Oxford: Oxford University Press.

Bostrom, N., & Cirkovic, M. M. (Eds.). (2008). Global Catastrophic Risks. New York: Oxford University Press.

Burch, J.D. and Thomas, K.E., 1998. Water disinfection for developing countries and potential for solar thermal pasteurization. Solar Energy, 64(1-3), pp.87-97.

Che, L., & Shahidehpour, M. (2014). DC microgrids: Economic operation and enhancement of resilience by hierarchical control. IEEE Transactions on Smart Grid, 5(5), 2517–2526.

Chrisman, L., Henrion, M., Morgan, R., Arnold, B., Brunton, F., Esztergar, A., & Harlan, J. (2007). Analytica user guide. Los Gatos, CA: Lumina Decision Systems.

Coates, J. F. (2009). Risks and threats to civilization, humankind, and the earth. Futures, 41(10), 694–705. https://doi.org/10.1016/j.futures.2009.07.010

Cole, D. D., Denkenberger, D., Griswold, M., Abdelkhaliq, M., & Pearce, J. (2016). Feeding Everyone if Industry is Disabled. Proceedings of the 6th International Disaster and Risk Conference. Presented at the 6th International Disaster and Risk Conference, Davos, Switzerland.

Colson, C., Nehrir, M., & Gunderson, R. (2011). Distributed multi-agent microgrids: a decentralized approach to resilient power system self-healing. 83–88. IEEE.

Cotton-Barratt, O. (2014, October). The law of logarithmic returns. Retrieved April 10, 2019, from The Future of Humanity Institute website: http://www.fhi.ox.ac.uk/law-of-logarithmic-returns/

Dartnell, L. (2014). The Knowledge: How to Rebuild Our World from Scratch. Random House.

Denkenberger, D. C., & Pearce, J. M. (2018, June 14). A National Pragmatic Safety Limit for Nuclear Weapon Quantities. Safety 2018, 4(2), 25; https://doi.org/10.3390/safety4020025

Denkenberger, D. and Pearce, J., 2018. Design optimization of polymer heat exchanger for automated household-scale solar water pasteurizer. Designs, 2(2), 11; https://doi.org/10.3390/designs2020011

Denkenberger, D., Cotton-Barrat, O., Dewey, D., & Li, S. (2019a, August 10). Foods without industry and AI X risk cost effectiveness general far future impact Denkenberger. Retrieved August 10, 2019, from Guesstimate website: https://www.getguesstimate.com/models/11599

Denkenberger, D., Cotton-Barrat, O., Dewey, D., & Li, S. (2019b, August 12). Machine Intelligence Research Institute - Oxford Prioritisation Project. Retrieved August 12, 2019, from Guesstimate website: https://www.getguesstimate.com/models/8789

Denkenberger, D., Cotton-Barratt, O., Dewey, D., & Li, S. (2019, April 10). Food without the sun and AI X risk cost effectiveness general far future impact publication. Retrieved April 10, 2019, from Guesstimate website: https://www.getguesstimate.com/models/13082

Denkenberger, D., & Pearce, J. (2018). Micronutrient availability in alternative foods during agricultural catastrophes. Agriculture, 8(11), 169.

Denkenberger, D., & Pearce, J. M. (2014). Feeding Everyone No Matter What: Managing Food Security After Global Catastrophe. Academic Press.

Denkenberger, D., Pearce, J., Taylor, A. R., & Black, R. (2019). Food without sun: Price and life-saving potential. Foresight, 21(1), 118–129.

Denkenberger, D., Sandberg, A., & Pearce, J. M. (unpublished results). Long Term Cost-Effectiveness of Alternative Foods for Global Catastrophes.

Denkenberger, D. C, Cole, D. D., Abdelkhaliq, M., Griswold, M., Hundley, A. B., & Pearce, J. M. (2017). Feeding everyone if the sun is obscured and industry is disabled. International Journal of Disaster Risk Reduction, 21, 284–290.

Denkenberger, D.C. and Pearce, J.M., 2016. Cost-effectiveness of interventions for alternate food to address agricultural catastrophes globally. International Journal of Disaster Risk Science, 7(3), pp.205-215.

Denkenberger, D. C, & Pearce, J. M. (2015b). Feeding everyone: Solving the food crisis in event of global catastrophes that kill crops or obscure the sun. Futures, 72, 57–68.

Denkenberger, D. C., & Pearce, J. M. (2016). Cost-Effectiveness of Interventions for Alternate Food to Address Agricultural Catastrophes Globally. International Journal of Disaster Risk Science, 7(3), 205–215. https://doi.org/10.1007/s13753-016-0097-2

Effective Altruism Concepts. (2019, April 10). Importance, tractability, neglectedness framework. Retrieved April 10, 2019, from Effective Altruism Concepts website: https://concepts.effectivealtruism.com/concepts/importance-neglectedness-tractability/

Foster, J. S., Gjelde, E., Graham, W. R., Hermann, R. J., Kluepfel, H. (Hank) M., Lawson, R. L., … Woodard, J. B. (2004, July 22). Report of the Commission to Assess the Threat to the United States from Electromagnetic Pulse (EMP) Attack. Retrieved June 30, 2016, from Committee on Armed Services House of Representatives website: http://commdocs.house.gov/committees/security/has204000.000/has204000_0.HTM

Foster, Jr, J. S., Gjelde, E., Graham, W. R., Hermann, R. J., Kluepfel, H. (Hank) M., Lawson, R. L., … Woodard, J. B. (2008). Report of the commission to assess the threat to the united states from electromagnetic pulse (emp) attack: Critical national infrastructures. Retrieved from DTIC Document website: http://www.empcommission.org/docs/A2473-EMP_Commission-7MB.pdf

Garrick, B. J. (2008). Quantifying and controlling catastrophic risks. Academic Press.

Gent, M. R., & Costantini, L. P. (2003). Reflections on security [power systems]. IEEE Power and Energy Magazine, 1(1), 46–52.

GiveWell. (2017, November). Cost-Effectiveness. Retrieved April 10, 2019, from GiveWell website: https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness

Good, I. J. (1966). Speculations concerning the first ultraintelligent machine. In Advances in computers (Vol. 6, pp. 31–88). Elsevier.

Goodin, D. (2016, January 4). First known hacker-caused power outage signals troubling escalation. Retrieved from http://arstechnica.com/security/2016/01/first-known-hacker-caused-power-outage-signals-troubling-escalation/

Gorman, S. (2009, April 9). Electricity Grid in U.S. Penetrated By Spies. Wall Street Journal. Retrieved from https://www.wsj.com/articles/SB123914805204099085

Gregory, J., Stouffer, R. J., Molina, M., Chidthaisong, A., Solomon, S., Raga, G., … Stone, D. A. (2007). Climate Change 2007: The Physical Science Basis. Retrieved from http://copa.acguanacaste.ac.cr:8080/handle/11606/461

Griswold, M., Denkenberger, D., Abdelkhaliq, M., Cole, D., Pearce, J., & Taylor, A. R. (2016). Vitamins in Agricultural Catastrophes. Proceedings of the 6th International Disaster and Risk Conference. Presented at the 6th International Disaster and Risk Conference, Davos, Switzerland.

Halstead, J. (2018, May). Climate Change Cause Area Report. Founders Pledge.

Hayakawa, H., Ebihara, Y., Willis, D. M., Toriumi, S., Iju, T., Hattori, K., … Ribeiro, J. R. (2019). Temporal and Spatial Evolutions of a Large Sunspot Group and Great Auroral Storms around the Carrington Event in 1859. Space Weather.

Hébert, C. (2013). The Most Critical of Economic Needs (Risks): A Quick Look at Cybersecurity and the Electric Grid. The Electricity Journal, 26(5), 15–19. https://doi.org/10.1016/j.tej.2013.05.009

Helfand, I. (2013). Nuclear famine: Two billion people at risk. International Physicians for the Prevention of Nuclear War, 20.

Kelly-Detwiler, P. (2014, July 31). Failure to Protect U.S. Against Electromagnetic Pulse Threat Could Make 9/11 Look Trivial Someday. Retrieved August 7, 2019, from https://www.forbes.com/sites/peterdetwiler/2014/07/31/protecting-the-u-s-against-the-electromagnetic-pulse-threat-a-continued-failure-of-leadership-could-make-911-look-trivial-someday/#2ed092db7a14

Keramat, M., & Kielbasa, R. (1997). Latin hypercube sampling Monte Carlo estimation of average quality index for integrated circuits. In Analog Design Issues in Digital VLSI Circuits and Systems (pp. 131–142). Springer.

Kinney, R., Crucitti, P., Albert, R., & Latora, V. (2005). Modeling cascading failures in the North American power grid. The European Physical Journal B, 46(1), 101–107. https://doi.org/10.1140/epjb/e2005-00237-9

Klein, C. (2012, March 14). A Perfect Solar Superstorm: The 1859 Carrington Event. Retrieved August 7, 2019, from HISTORY website: https://www.history.com/news/a-perfect-solar-superstorm-the-1859-carrington-event

Krotofil, M., Cardenas, A., Larsen, J., & Gollmann, D. (2014). Vulnerabilities of cyber-physical systems to stale data—Determining the optimal time to launch attacks. International Journal of Critical Infrastructure Protection, 7(4), 213–232.

Kushner, D. (2013). The real story of stuxnet. IEEE Spectrum, 50(3), 48–53. https://doi.org/10.1109/MSPEC.2013.6471059

Lasseter, R. H. (2007). Microgrids and distributed generation. Journal of Energy Engineering, 133(3), 144–149.

Lasseter, R. H., & Piagi, P. (2004). Microgrid: A conceptual solution. 6, 4285–4291. Citeseer.

Li, S. (2017, May 12). A model of the Machine Intelligence Research Institute - Oxford Prioritisation Project - EA Forum. Retrieved August 12, 2019, from https://forum.effectivealtruism.org/posts/NbFZ9yewJHoicpkBr/a-model-of-the-machine-intelligence-research-institute

Lingam, M., & Loeb, A. (2017). Risks for life on habitable planets from superflares of their host stars. The Astrophysical Journal, 848(1), 41.

Liptak, B. G. (2018). Instrument Engineers’ Handbook, Volume Two: Process Control and Optimization. CRC Press.

Lovins, A. B., & Lovins, L. H. (1982). Brittle power. Brick House Publishing Company.

Matheny, J. G. (2007). Reducing the risk of human extinction. Risk Analysis: An International Journal, 27(5), 1335–1344.

McIntyre, P. (2016a, April 12). How you can lower the risk of a catastrophic nuclear war. Retrieved August 13, 2019, from 80,000 Hours website: https://80000hours.org/problem-profiles/nuclear-security/

McIntyre, P. (2016b, April 12). How you can lower the risk of a catastrophic nuclear war. Retrieved August 9, 2019, from 80,000 Hours website: https://80000hours.org/problem-profiles/nuclear-security/

Mekhaldi, F., Muscheler, R., Adolphi, F., Aldahan, A., Beer, J., McConnell, J. R., … Synal, H.-A. (2015). Multiradionuclide evidence for the solar origin of the cosmic-ray events of ᴀᴅ 774/5 and 993/4. Nature Communications, 6.

Millett, P., & Snyder-Beattie, A. (2017). Existential Risk and Cost-Effective Biosecurity. Health Security, 15(4), 373–383. https://doi.org/10.1089/hs.2017.0028

Morgan, M. G., & Henrion, M. (1990). Uncertainty: a Guide to dealing with uncertainty in quantitative risk and policy analysis Cambridge University Press. New York, New York, USA.

Motesharrei, S., Rivas, J., & Kalnay, E. (2014). Human and nature dynamics (HANDY): Modeling inequality and use of resources in the collapse or sustainability of societies. Ecological Economics, 101, 90–102. https://doi.org/10.1016/j.ecolecon.2014.02.014

Motter, A. E., & Lai, Y.-C. (2002). Cascade-based attacks on complex networks. Physical Review E, 66(6), 065102. https://doi.org/10.1103/PhysRevE.66.065102

Nai Fovino, I., Guidi, L., Masera, M., & Stefanini, A. (2011). Cyber security assessment of a power plant. Electric Power Systems Research, 81(2), 518–526. https://doi.org/10.1016/j.epsr.2010.10.012

National Research Council. (2012). Terrorism and the electric power delivery system. National Academies Press.

Oak Ridge National Laboratory. (2010). Electromagnetic Pulse: Effects on the U.S. Power Grid. 6.

Oke, O., Redhead, J., & Hussain, M. (1990). Roots, tubers, plantains and bananas in human nutrition. FAO Food and Nutrition Series, 24, 182.

Onyeji, I., Bazilian, M., & Bronk, C. (2014). Cyber Security and Critical Energy Infrastructure. The Electricity Journal, 27(2), 52–60. https://doi.org/10.1016/j.tej.2014.01.011

Open Source Ecology. (2019, August 10). Open Source Ecology. Retrieved August 10, 2019, from https://www.opensourceecology.org/

Ord, T. (2014, July 3). The timing of labour aimed at reducing existential risk. Retrieved April 10, 2019, from The Future of Humanity Institute website: https://www.fhi.ox.ac.uk/the-timing-of-labour-aimed-at-reducing-existential-risk/

Pagliery, J. (2015, October 16). Sniper attack on California power grid may have been “an insider,” DHS says. Retrieved August 8, 2019, from CNNMoney website: https://money.cnn.com/2015/10/16/technology/sniper-power-grid/index.html

Prehoda, E. W., Schelly, C., & Pearce, J. M. (2017). US strategic solar photovoltaic-powered microgrid deployment for enhanced national security. Renewable and Sustainable Energy Reviews, 78, 167–175.

Pry, P. (2017). NUCLEAR EMP ATTACK SCENARIOS AND COMBINED-ARMS CYBER WARFARE. 65.

Pry, P. V. (2014, May 8). - ELECTROMAGNETIC PULSE (EMP): THREAT TO CRITICAL INFRASTRUCTURE. Retrieved August 14, 2019, from https://www.govinfo.gov/content/pkg/CHRG-113hhrg89763/html/CHRG-113hhrg89763.htm

Robinson, R. A. (2007). Crop histories. Sharebooks Pub.

Rolak, B. J. (1975). General Miles’ Mirrors: The Heliograph in the Geromino Campaign of 1886. The Journal of Arizona History, 16(2), 145–160.

Roodman, D. (2015). The risk of geomagnetic storms to the grid. 56.

Salmeron, J., Wood, K., & Baldick, R. (2004). Analysis of Electric Grid Security Under Terrorist Threat. IEEE Transactions on Power Systems, 19(2), 905–912. https://doi.org/10.1109/TPWRS.2004.825888

Schainker, R., Douglas, J., & Kropp, T. (2006). Electric utility responses to grid security issues. IEEE Power and Energy Magazine, 4(2), 30–37.

Schaul, T., Togelius, J., & Schmidhuber, J. (2011). Measuring intelligence through games. ArXiv Preprint ArXiv:1109.1314.

Shahidehpour, M., & Khodayar, M. (2013). Cutting campus energy costs with hierarchical control: The economical and reliable operation of a microgrid. IEEE Electrification Magazine, 1(1), 40–56.

Silver, W. H. (2004). Ham Radio for Dummies. Wiley Publishing, Inc.

Space Studies Board (Ed.). (2008). Severe Space Weather Events--Understanding Societal and Economic Impacts: A Workshop Report. National Academies Press.

Sridhar, S., Hahn, A., & Govindarasu, M. (2012). Cyber–Physical System Security for the Electric Power Grid. Proceedings of the IEEE, 100(1), 210–224. https://doi.org/10.1109/JPROC.2011.2165269

Sterling, C. H. (2008). Military Communications: From Ancient Times to the 21st Century. ABC-CLIO.

Ten, C.-W., Manimaran, G., & Liu, C.-C. (2010). Cybersecurity for critical infrastructures: Attack and defense modeling. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 40(4), 853–865.

Turchin, A., & Denkenberger, D. (2018a). Classification of global catastrophic risks connected with artificial intelligence. AI & SOCIETY. https://doi.org/10.1007/s00146-018-0845-5

Turchin, A., & Denkenberger, D. (2018b). Global catastrophic and existential risks communication scale. Futures, 102, 27–38. https://doi.org/10.1016/j.futures.2018.01.003

Tzezana, R. (2016). Scenarios for crime and terrorist attacks using the internet of things. European Journal of Futures Research, 4, 18. https://doi.org/10.1007/s40309-016-0107-z

Ulieru, M. (2007). Design for resilience of networked critical infrastructures. 540–545. IEEE.

Umbach, F. (2013, June 29). World Review | Energy infrastructure targeted as cyber attacks increase globally. Retrieved August 8, 2019, from https://web.archive.org/web/20130629041842/https://worldreview.info/content/energy-infrastructure-targeted-cyber-attacks-increase-globally

Watts, D. (2003). Security & Vulnerability in Electric Power Systems. Th North American Power Symposium, 8.

Wu, F. F., Moslehi, K., & Bose, A. (2005). Power system control centers: Past, present, and future. Proceedings of the IEEE, 93(11), 1890–1908.

Zerriffi, H., Dowlatabadi, H., & Strachan, N. (2002). Electricity and conflict: Advantages of a distributed system. The Electricity Journal, 15(1), 55–65.