Max_Daniel's Quick takes

[Some of my high-level views on AI risk.]

[I wrote this for an application a couple of weeks ago, but thought I might as well dump it here in case someone was interested in my views. / It might sometimes be useful to be able to link to this.]

[In this post I generally state what I think ​before ​updating on other people’s views – i.e., what’s ​sometimes known as​ ‘impressions’ as opposed to ‘beliefs.’]

Summary

• Transformative AI (TAI) – the prospect of AI having impacts at least as consequential as the Industrial Revolution – would plausibly (~40%) be our best lever for influencing the long-term future if it happened this century, which I consider to be unlikely (~20%) but worth betting on.
• The value of TAI depends not just on the technological options available to individual actors, but also on the incentives governing the strategic interdependence between actors. Policy could affect both the amount and quality of technical safety research and the ‘rules of the game’ under which interactions between actors will play out.

Why I'm interested in TAI as a lever to improve the long-run future

I expect my perspective to be typical of someone who has become interested in TAI through their engagement with the effective altruism (EA) community. In particular,

• My overarching interest is to make the lives of as many moral patients as possible to go as well as possible, no matter where or when they live; and
• I think that in the world we find ourselves in – it could have been otherwise –, this goal entails strong longtermism,​ i.e. the claim that “the primary determinant of the value of our actions today is how those actions affect the very long-term future.”

Less standard but not highly unusual (within EA) high-level views I hold more tentatively:

• The indirect long-run impacts of our actions are extremely hard to predict and don’t ‘cancel out’ in expectation. In other words, I think that what ​Greaves (2016)​ calls ​complex cluelessness​ is a pervasive problem. In particular, evidence that an action will have desirable effects in the short term generally is ​not​ a decisive reason to believe that this action would be net positive overall, and neither will we be able to establish the latter through any other means.
• Increasing the relative influence of longtermist actors is one of the very few strategies we have good reasons to consider net positive. Shaping TAI is a particularly high-leverage instance of this strategy, where the main mechanism is reaping an ‘epistemic rent’ from having anticipated TAI earlier than other actors. I take this line of support to be significantly more robust than any ​particular story on how TAI might pose a global catastrophic risk including even broad operationalizations of the ‘value alignment problem.’

My empirical views on TAI

I think the strongest reasons to expect TAI this century are relatively outside-view-based (I talk about this century just because I expect that later developments are harder to predictably influence, not because I think a century is particularly meaningful time horizon or because I think TAI would be less important later):

• We’ve been able to automate an increasing number of tasks (with increasing performance and falling cost), and I’m not aware of a convincing argument for why we should be ​highly confident​ that this trend will stop short of ​full automation –​ i.e., AI systems being able to do all tasks more economically efficiently than humans –, despite moderate scientific and economic incentives to find and publish one.
• Independent types of weak evidence such as ​trend extrapolation​ and ​expert​ ​surveys​ suggest we might achieve full automation this century.
• Incorporating full automation into macroeconomic growth models predicts – at least under some a​ssumptions – a sustained higher rate of economic growth (e.g. ​Hanson 2001​, Nordhaus 2015​, ​Aghion et al. 2017​), which arguably was the main driver of the welfare-relevant effects of the Industrial Revolution.
• Accelerating growth this century is consistent with extrapolating historic growth rates, e.g. Hanson (2000[1998])​.

I think there are several reasons to be skeptical, but that the above succeeds in establishing a somewhat robust case for TAI this century not being wildly implausible.

My impression is that I’m less confident than the typical longtermist EA in various claims around TAI, such as:

• Uninterrupted technological progress would eventually result in TAI;
• TAI will happen this century;
• we can currently anticipate any specific way of positively shaping the impacts of TAI;
• if the above three points were true then shaping TAI would be the most cost-effective way of improving the long-term future.

My guess is this is due to different priors, and due to frequently having found extant specific arguments for TAI-related claims (including by staff at FHI and Open Phil) less convincing than I would have predicted. I still think that work on TAI is among the few best shots for current longtermists.

What's the right narrative about global poverty and progress? Link dump of a recent debate.

The two opposing views are:

(a) "New optimism:" [1] This is broadly the view that, over the last couple of hundred years, the world has been getting significantly better, and that's great. [2] In particular, extreme poverty has declined dramatically, and most other welfare-relevant indicators have improved a lot. Often, these effects are largely attributed to economic growth.

• Proponents in this debate were originally Bill Gates, Steven Pinker, and Max Roser. But my loose impression is that the view is shared much more widely.
• In particular, it seems to be the orthodox view in EA; cf. e.g. Muehlhauser listing one of Pinker's books in his My worldview in 5 books post, saying that "Almost everything has gotten dramatically better for humans over the past few centuries, likely substantially due to the spread and application of reason, science, and humanism."

(b) Hickel's critique: Anthropologist Jason Hickel has criticized new optimism on two grounds:

• 1. Hickel has questioned the validity of some of the core data used by new optimists, claiming e.g. that "real data on poverty has only been collected since 1981. Anything before that is extremely sketchy, and to go back as far as 1820 is meaningless."
• 2. Hickel prefers to look at different indicators than the new optimists. For example, he has argued for different operationalizations of extreme poverty or inequality.

Link dump (not necessarily comprehensive)

If you only read two things, I'd recommend (1) Hasell's and Roser's article explaining where the data on historic poverty comes from and (2) the take by economic historian Branko Milanovic.

By Hickel (i.e. against "new optimism"):

• https://www.jasonhickel.org/blog/2018/12/13/what-max-roser-gets-wrong-about-inequality
• https://www.theguardian.com/commentisfree/2019/jan/29/bill-gates-davos-global-poverty-infographic-neoliberal
• https://www.jasonhickel.org/blog/2019/2/3/pinker-and-global-poverty
• https://www.jasonhickel.org/blog/2019/2/6/response-to-max-roser
• https://www.jasonhickel.org/blog/2019/2/19/response-to-vox-global-poverty
• https://newint.org/features/2019/07/01/long-read-progress-and-its-discontents

By "new optimists":

• Joe Hasell and Max Roser: https://ourworldindata.org/extreme-history-methods
• Steven Pinker: https://whyevolutionistrue.wordpress.com/2019/01/31/is-the-world-really-getting-poorer-a-response-to-that-claim-by-steve-pinker/
• [ETA 2021-05-23:] Max Roser on Twitter on why he has "no respect at all for [Hickel] anymore".

Commentary by others:

• Branko Milanovic (a leading economic historian): https://www.globalpolicyjournal.com/blog/11/02/2019/global-poverty-over-long-term-legitimate-issues
• Dylan Matthews: https://www.vox.com/future-perfect/2019/2/12/18215534/bill-gates-global-poverty-chart
• LW user ErickBall: https://www.lesswrong.com/posts/eTMgL7Cx8TsA9nedn/is-the-world-getting-better-a-brief-summary-of-recent-debate
• [ETA 2021-05-23:] Noah Smith, Against Hickelism. Poverty is falling, and it isn't because of free-market capitalism

My view

• I'm largely unpersuaded by Hickel's charge that historic poverty data is invalid. Sure, it's way less good than contemporary data. But based on Hasell's and Roser's article, my impression is that the data is better than I would have thought, and its orthodox analysis and interpretation more sophisticated than I would have thought. I would be surprised if access to better data would qualitatively change the "new optimist" conclusion.
• I think there is room for debate over which indicators to use, and that Hickel makes some interesting points here. I find it regrettable that the debate around this seems so adversarial.
• Still, my sense is that there is an important, true, and widely underappreciated (particularly by people on the left, including my past self) core of the "new optimist" story. I'd expect looking at other indicators could qualify that story, or make it less simplistic, point to important exceptions etc. - but I'd probably consider a choice of indicators that painted an overall pessimistic picture as quite misleading and missing something important.
• On the other hand, I would quite strongly want to resist the conclusion that everything in this debate is totally settled, and that the new optimists are clearly right about everything, in the same way in which orthodox climate science is right about climate change being anthropogenic, or orthodox medicine is right about homeopathy not being better than placebo. But I think the key uncertainties are not in historic poverty data, but in our understanding of wellbeing and its relationship to environmental factors. Some examples of why I think it's more complicated
• The Easterlin paradox
• The unintuitive relationship between (i) subjective well-being in the sense of the momentary affective valence of our experience on one hand and (ii) reported life satisfaction. See e.g. Kahneman's work on the "experiencing self" vs. "remembering self".
• On many views, the total value of the world is very sensitive to population ethics, which is notoriously counterintuitive. In particular, on many plausible views, the development of the total welfare of the world's human population is dominated by its increasing population size.
• Another key uncertainty is the implications of some of the discussed historic trends for the value of the world going forward, about which I think we're largely clueless. For example, what are the effects of changing inequality on the long-term future?

[1] It's not clear to me if "new optimism" is actually new. I'm using Hickel's label just because it's short and it's being used in this debate anyway, not to endorse Hickel's views or make any other claim.

[2] There is an obvious problem with new optimism, which is that it's anthropocentric. In fact, on many plausible views, the total axiological value of the world at any time in the recent past may be dominated by the aggregate wellbeing of nonhuman animals; even more counterintuitively, it may well be dominated by things like the change in the total population size of invertebrates. But this debate is about human wellbeing, so I'll ignore this problem.

[See this research proposal for context. I'd appreciate pointers to other material.]

[WIP, not comprehensive] Collection of existing material on 'impact being heavy-tailed'

Conceptual foundations

• Newman (2005) provides a good introduction to power laws, and reviews several mechanisms generating them, including: combinations of exponentials; inverses of quantities; random walks; the Yule process [also known as preferential attachment]; phase transitions and critical phenomena; self-organized criticality.
• Terence Tao, in Benford’s law, Zipf’s law, and the Pareto distribution, offers a partial explanation for why heavy-tailed distributions are so common empirically.
• Clauset et al. (2009[2007]) explain why it is very difficult to empirically distinguish power laws from other heavy-tailed distributions (e.g. log-normal). In particular, seeing a roughly straight line in a log-log plot is not sufficient to identify a power law, despite such inferences being popular in the literature. Referring to power-law claims by others, they find that “the distributions for birds, books, cities, religions, wars, citations, papers, proteins, and terrorism are plausible power laws, but they are also plausible log-normals and stretched exponentials.” (p. 26) [NB on citations Golosovsky & Solomon, 2012, claim to settle the question in favor of a power law - and in fact an extreme tail even heavier than that -, and they are clearly aware of the issues pointed out by Clauset et al. On the other hand, Brzezinski, 2014, using a larger data set, seems to confirm the results of Clauset et al., finding that a pure power law is the single best fit only for physics & astronomy papers, while in all other disciplines we either can't empirically distinguish between several heavy-tailed distributions or a power law can be statistically rejected.]
• Lyon (2014) argues that, unlike commonly believed, the Central Limit Theorem cannot explain why normal distributions are so common. (The critique also applies to the analog explanation for the log-normal distribution, an example of a heavy-tailed distribution.) Instead, an appeal to the principle of maximum entropy is suggested.

Impact in general / cause-agnostic

• Kokotajlo & Oprea (2020) argue that there also is a heavy left tail of harmful interventions.
• Tobias Baumann, Is most expected suffering due to worst-case scenarios?
• Open Phil's approach of hits-based giving, in which they "expect the very few best projects to account for much (or most) of our impact".
• Owen Cotton-Barratt's talk Prospecting for gold, section Heavy-tailed distributions
• Brian Tomasik, Why Charities Usually Don't Differ Astronomically in Expected Cost-Effectiveness
• Discussion between Ben Pace and Richard Ngo on "how valuable it is for people to go into AI policy and strategy work" - but one of Ben's premises is that impact generally has been heavy-tailed since the Industrial Revolution.
• Recorded GWWC donations are extremely heavy-tailed. From here:

Less than 1% of our donors account for 50% of our recorded donations. This amounts to dozens of people, while the next 40% of donations (from both pledge donors and non-pledge donors) is distributed among hundreds. This suggests that most of our impact comes from a small-to-medium-size group of large donors (rather than from a very small group of very large donors, or from a large group of small donors).[6]

EA community building

• CEA models of community building, especially the discussion of How much might these factors vary? in the ‘three-factor model’.
• CEA’s current thinking [no longer endorsed by CEA], particularly Talent is high variance
• Discussion between Ben Pace and Jan Kulveit on "whether you should expect to be able to identify individuals who will most shape the long term future of humanity".
• Discussion on this Shortform between me, Buck, and others, on whether "longtermism is bottlenecked by 'great people'".

Global health

• Toby Ord's classic paper The Moral Imperative toward Cost-Effectiveness in Global Health, in particular his observation that "According to the DCP2 data, if we funded all of these [health] interventions equally, 80 percent of the benefits would be produced by the top 20 percent of the interventions." (p. 3)
• Jeff Kaufman, The Unintuitive Power Laws of Giving (mostly based on the same DCP2 data)
• Agenty Duck, Eva Vivalt Did Not Show QALYs/$ of Interventions Follow a Gaussian Curve
• Jeff Kaufman, Effectiveness: Gaussian? (pushing back against the same claim by Robin Hanson as the Agenty Duck blog post above)

Misc

• Will MacAskill, in an interview by Lynette Bye, on the distribution of impact across work time for a fixed individual: "Maybe, let's say the first three hours are like two thirds the value of the whole eight-hour day. And then, especially if I'm working six days a week, I'm not convinced the difference between eight and ten hours is actually adding anything in the long term."

[Longtermist EA vs. human progress/progress studies.

I'm posting a quick summary of my current understanding, which I needed to write anyway for an email conversation. I'm not that familiar with the human progress/progress studies communities and would be grateful if people pointed out where my impression of them seems off, as well as for takes on whether I seem correct about what the key points of agreement and disagreement are.]

[ETA: See this reply from Tony from the 'progress studies' community .]
 

Here's a quick summary of my understanding of the 'longtermist EA' and 'progress studies' perspectives, in a somewhat cartoonish way to gesture at points of agreement and disagreement. 

EA and progress studies mostly agree about the past. In particular, they agree that the Industrial Revolution was a really big deal for human well-being, and that this is often overlooked/undervalued. E.g. here's a blog post by someone somewhat influential in EA:

https://lukemuehlhauser.com/industrial-revolution/
 

Looking to the future, the progress studies community is most worried about the Great Stagnation. They are nervous that science seems to be slowing down, that ideas are getting harder to find, and that economic growth may soon be over. Industrial-Revolution-level progress was by far the best thing that ever happened to humanity, but we're at risk of losing it. That seems really bad. We need a new science of progress to understand how to keep it going. Probably this will eventually require a number of technological and institutional innovations since our current academic and economic systems are what's led us into the current slowdown.

If we were making a list of the most globally consequential developments from the past, EAs would in addition to the Industrial Revolution point to the Manhattan Project and the hydrogen bomb: the point in time when humanity first developed the means to destroy itself. (They might also think of factory farming as an example for how progress might be great for some but horrible for others, at least on some moral views.) So while they agree that the world has been getting a lot better thanks to progress, they're also concerned that progress exposes us to new nuclear-bomb-style risks. Regarding the future, they're most worried about existential risk -- the prospect of permanently forfeiting our potential of a future that's much better than the status quo. Permanent stagnation would be an existential risk, but EAs tend to be even more worried about catastrophes from emerging technologies such as misaligned artificial intelligence or engineered pandemics. They might also be worried about a potential war between the US and China, or about extreme climate change. So in a sense they aren't as worried about progress stopping than they are about progress being mismanaged and having catastrophic unintended consequences. They therefore aim for 'differential progress' -- accelerating those kinds of technological or societal change that would safeguard us against these catastrophic risks, and slowing down whatever would expose us to greater risk. So concretely they are into things like "AI safety" or "biosecurity" -- e.g. making machine learning systems more transparent so we could tell if they were trying to deceive their users, or implementing better norms around the publication of dual-use bio research.

The single best book on this EA perspective is probably The Precipice by my FHI colleague Toby Ord.

Overall, EA and the progress studies perspective agree on a lot -- they're probably closer than either would be to any other popular 'worldview'. But overall EAs probably tend to think that human progress proponents are too indiscriminately optimistic about further progress, and too generically focused on keeping progress going. (Both because it might be risky and because EAs probably tend to be more "optimistic" that progress will accelerate anyway, most notably due to advances in AI.) Conversely, human progress proponents tend to think that EA is insufficiently focused on ensuring a future of significant economic growth and the risks imagined by EAs either aren't real or that we can't do much to prevent them except encouraging innovation in general.

[On https://www.technologyreview.com/s/615181/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/ ]

• [ETA: After having talked to more people, it now seems to me that disagreeing on this point more often explains different reactions than I thought it would. I'm also now less confident that my impression that there wasn't bad faith from the start is correct, though I think I still somewhat disagree with many EAs on this. In particular, I've also seen plenty of non-EA people who don't plausibly have a "protect my family" reaction say the piece felt like a failed attempt to justify a negative bottom line that was determined in advance.] (Most of the following doesn't apply in cases where someone is acting in bad faith and is determined to screw you over. And in fact I've seen the opposing failure mode of people assuming good faith for too long. But I don't think this is a case of bad faith.)
• I've seen some EAs react pretty negatively or angrily to that piece. (Tbc, I've also seen different reactions.) Some have described the article as a "hit piece".
• I don't think it qualifies as a hit piece. More like a piece that's independent/pseudo-neutral/ambiguous and tried to stick to dry facts/observations but in some places provides a distorted picture by failing to be charitable / arguably missing the point / being one-sided and selective in the observation it reports.
• I still think that reporting like this is net good, and that the world would be better if there was more of it at the margin, even if it has flaws similarly severe to that one. (Tbc, I think there would have been a plausibly realistic/achievable version of that article that would have been better, and that there is fair criticism one can direct at it.)
• To put it bluntly, I don't believe that having even maximally well-intentioned and intelligent people at key institutions is sufficient for achieving a good outcome for the world. I find it extremely hard to have faith in a setup that doesn't involve a legible system/structure with things like division of labor, checks and balances, procedural guarantees, healthy competition, and independent scrutiny of key actors. I don't know if the ideal system for providing such outside scrutiny will look even remotely like today's press, but currently it's one of the few things in this vein that we have for nonprofits, and Karen Hao's article is an (albeit flawed) example of it.
• Whether this specific article was net good or not seems pretty debatable. I definitely see reasons to think it'll have bad consequences, e.g. it might crowd out better reporting, might provide bad incentives by punishing orgs for trying to do good things, ... I'm less wedded to a prediction of this specific article's impact than to the broader frame for interpreting and reacting to it.
• I find something about the very negative reactions I've seen worrying. I of course cannot know what they were motivated by, but some seemed like I would expect someone to react who's personally hurt because they judge a situation as being misunderstood, feels like they need to defend themself, or like they need to rally to protect their family. I can relate to misunderstandings being a painful experience, and have sympathy for it. But I also think that if you're OpenAI, or "the EA community", or anyone aiming to change the world, then misunderstandings are part of the game, and that any misunderstanding involves at least two sides. The reactions I'd like to see would try to understand what has happened and engage constructively with how to productively manage the many communication and other challenges involved in trying to do something that's good for everyone without being able to fully explain your plans to most people. (An operationalization: If you think this article was bad, I think that ideally the hypothesis "it would be good it we had better reporting" would enter your mind as readily as the hypothesis "it would be good if OpenAI's comms team and leadership had done a better job".)

[Is longtermism bottlenecked by "great people"?]

Someone very influential in EA recently claimed in conversation with me that there are many tasks X such that (i) we currently don't have anyone in the EA community who can do X, (ii) the bottleneck for this isn't credentials or experience or knowledge but person-internal talent, and (iii) it would be very valuable (specifically from a longtermist point of view) if we could do X. And that therefore what we most need in EA are more "great people".

I find this extremely dubious. (In fact, it seems so crazy to me that it seems more likely than not that I significantly misunderstood the person who I think made these claims.) The first claim is of course vacuously true if, for X, we choose some ~impossible task such as "experience a utility-monster amount of pleasure" or "come up with a blueprint for how to build safe AGI that is convincing to benign actors able to execute it". But of course more great people don't help with solving impossible tasks.

Given the size and talent distribution of the EA community my guess is that for most apparent X, the issue either is that (a) X is ~impossible, or (b) there are people in EA who could do X, but the relevant actors cannot identify them, or (c) acquiring the ability to do X is costly (e.g. perhaps you need time to acquire domain-specific expertise), even for maximally talented "great people", and the relevant actors either are unable to help pay that cost (e.g. by training people themselves, or giving them the resources to allow them to get training elsewhere) or make a mistake by not doing so.

My best guess for the genesis of the "we need more great people" perspective: Suppose I talk a lot to people at an organization that thinks there's a decent chance we'll develop transformative AI soon but it will go badly, and that as a consequence tries to grow as fast as possible to pursue various ambitious activities which they think reduces that risk. If these activities are scalable projects with short feedback loops on some intermediate metrics (e.g. running some super-large-scale machine learning experiments), then I expect I would hear a lot of claims like "we really need someone who can do X". I think it's just a general property of a certain kind of fast-growing organization that's doing practical things in the world that everything constantly seems like it's on fire. But I would also expect that, if I poked a bit at these claims, it would usually turn out that X is something like "contribute to this software project at the pace and quality level of our best engineers, w/o requiring any management time" or "convince some investors to give us much more money, but w/o anyone spending any time transferring relevant knowledge". If you see that things break because X isn't done, even though something like X seems doable in principle (perhaps you see others do it), it's tempting to think that what you need is more "great people" who can do X. After all, people generally are the sort of stuff that does things, and maybe you've actually seen some people do X. But it still doesn't follow that in your situation "great people" are the bottleneck ...

Curious if anyone has examples of tasks X for which the original claims seem in fact true. That's probably the easiest way to convince me that I'm wrong.

[EA's focus on marginal individual action over structure is a poor fit for dealing with info hazards.]

I tend to think that EAs sometimes are too focused on optimizing the marginal utility of individual actions as opposed to improving larger-scale structures. For example, I think it'd be good if there was much content and cultural awareness on how to build good organizations as there is on how to improve individual cognition. - Think about how often you've heard of "self improvement" or "rationality" as opposed to things like "organizational development".

(Yes, this is similar to the good old 'systemic change' objection aimed at "what EAs tend to do in practice" rather than "what is implied by EAs' normative views".)

It occurred to me that one instance where this might bite in particular are info hazards.

I often see individual researchers agonizing about whether they can publish something they have written, which of several framings to use, and even which ideas are safe to mention in public. I do think that this can sometimes be really important, and that there are areas with a predictably high concentration of such cases, e.g. bio.

However, in many cases I feel like these concerns are far-fetched and poorly targeted.

• They are far-fetched when they overestimate the effects a marginal publication by a non-prominent person can have on the world. E.g. the US government isn't going to start an AGI project because you posted a thought on AI timelines on LessWrong.
• They are poorly targeted when they focus on the immediate effects of marginal individual action. E.g., how much does my paper contribute to 'AI capabilities'? What connotations will readers read into different terms I could use for the same concept?

On the other hand, in such cases often there are important info hazards in the areas researchers are working about. For example, I think it's at least plausible that there is true information on, say, the prospects and paths to transformative AI, that would be bad to bring to the attention of, say, senior US or Chinese government officials.

It's not the presence of these hazards but the connection with typical individual researcher actions that I find dubious. To address these concerns, rather than forward chaining from individual action one considers to take for other reasons, I suspect it'd be more fruitful to backward-chain from the location of large adverse effects (e.g. the US government starting an AGI project, if you think that's bad). I suspect this would lead to a focus on structure for the analysis, and a focus on policy for solutions. Concretely, questions like:

• What are the structural mechanisms for how information gets escalated to higher levels of seniority within, e.g., the US government or Alphabet?
• Given current incentives, how many publications of potentially hazardous information do we expect, and through which channels?
• What are mechanisms that can massively amplify the visibility of information? E.g. when will media consider something newsworthy, when and how do new academic subfields form?

[PAI vs. GPAI]

So there is now (well, since June 2020) both a Partnership on AI and a Global Partnership on AI.

Unfortunately GPAI's and PAI's FAQ pages conspicuously omit "how are you differnet from (G)PAI?".

Can anyone help?

At first glance it seems that:

• PAI brings together a very large number of below-state actors of different types: e.g., nonprofits, academics, for-profit AI labs, ...
• GPAI members are countries
• PAI's work is based on 4 high-level goals that each are described in about two sentences [?]
• GPAI's work is based on the OECD Recommendation on Artificial Intelligence
• I don't know how/where the substantive work undertaken by PAI gets done - e.g., by PAI staff or by ad-hoc joint projects between some members or ... ?
• GPAI has two "centers of excellence" in Montreal and Paris. I would guess [?] that a lot of the substantive work for/by it gets done there.

I also note that it's slightly ironic that GPAI differs from PAI by having added the adjective "global". It's based on an OECD recommendation, but the OECD is very much not a "global" organization - it's a club of rich market democracies. (Though GPAI membership differs a lot from the OECD - fewer than half of OECD members have joined GPAI, and some notable non-OECD member such as India have.)

(Max Daniel) Effective Altruism:

“Looking to the future, the progress studies community is most worried about the Great Stagnation. They are nervous that science seems to be slowing down, that ideas are getting harder to find, and that economic growth may soon be over. Industrial-Revolution-level progress was by far the best thing that ever happened to humanity, but we're at risk of losing it. That seems really bad. We need a new science of progress to understand how to keep it going. Probably this will eventually require a number of technological and institutional innovations since our current academic and economic systems are what's led us into the current slowdown.”

(Tony Morley) Human Progress – Progress Studies

The Industrial Revolution was a pivotal and critical point for the launch of human progress and prosperity; however, it came with costs and problems which needed to be solved in turn. The Industrial Revolution was the kick-off point where civilization mastered the ability to utilize the chemical energy banked in geological formation to invest in building a prosperous civilization. I do not think we are at risk of losing the progress of civilization build on the Industrial Revolution, but rather that the progress which the revolution kicked off, needs to continue to mature. The human progress proponent seeks as a priority to continue to advance the dramatic story of improving global living standards, while making rational and fact-based choices with investment and risk. When it comes to “what is progress?” as defined elegantly by Pinker in the following text, we have not yet reached the top of the s-curve for the majority of the global population. There is still room for progress in progress. 

Pinker: What is Progress? 

“What is progress? You might think that the question is so subjective and culturally relative as to be forever unanswerable. In fact, it’s one of the easier questions to answer. Most people agree that life is better than death. Health is better than sickness. Sustenance is better than hunger. Abundance is better than poverty. Peace is better than war. Safety is better than danger. Freedom is better than tyranny. Equal rights are better than bigotry and discrimination. Literacy is better than illiteracy. Knowledge is better than ignorance. Intelligence is better than dull-wittedness. Happiness is better than misery. Opportunities to enjoy family, friends, culture, and nature are better than drudgery and monotony. All these things can be measured. If they have increased over time, that is progress.” - Enlightenment Now, The Case for Reason, Science, Humanism, and Progress – Steven Pinker c2018

(Max Daniel) Effective Altruism:

“So while they agree that the world has been getting a lot better thanks to progress, they're also concerned that progress exposes us to new nuclear-bomb-style risks.”

(Tony Morley) Human Progress – Progress Studies

Agreed, many of the technologies and systems which have driven human progress forward, have also created novel risks and consequences that in turn must be mitigated or eliminated. The utilization of coal is a perfect example of this. Coal represented the metaphorical equivalent of an energy seed investment which humans have used since the Industrial Revolution through to today, to get civilization up and running. However, the ubiquitous use of coal for heating, smelting, and later electricity generation, has introduced or exacerbated issues including climate change, environmental surface disturbance and water and air pollution. Which are issues of human progress that in turn require further solutions. 

Similarly, our mastery of nuclear theory, and subsequently nuclear energy; provided both great manifest opportunities to push civilization forward, while at the same time it opened a pandora’s box of risk with regards to the use of nuclear energy for mass destruction and or loss of life. Our command of nuclear theory is widely disseminated, and as such there is little chance of fully reducing the risk to zero. There are, however, effective means of control for reducing the likelihood and consequence, of nuclear risk, and I would argue that utilizing those controls and mitigations are an example of human progress in action.

With regards to the article by Patrick Collison and Tyler Cowen, “We Need a New Science of Progress”, I do think there is value here. While the primary thesis for the history of progress and its principal drivers is fairly well established, it is poorly communicated, and not universally available or understood. What is needed is to take the human progress thesis as outlined in books like Progress, and Open by Johan Norberg, Factfulness by Hans Rosling, The Rational Optimist by Matt Ridley and Enlightenment Now by Steven Pinker, The Birth of Plenty: How the Prosperity of the Modern Work was Created – William J. Bernstein, amongst others, and use the insights contained therein to motivate our civilization to build a better future based on sound, rational and effective modes of operation. We can debate at length whether indefinite economic growth and ongoing human progress is possible, desirable, or ethical, once we have lifted nearly all of the worlds’ people out of extreme poverty – and not before then. “First comes a full stomach, then comes ethics.” – Bertolt Brecht, The Threepenny Opera c1928

(Max Daniel) Effective Altruism:

“(They might also think of factory farming as an example for how progress might be great for some but horrible for others, at least on some moral views.)”

(Tony Morley) Human Progress – Progress Studies

Agreed, factory farming has been both a blessing and a curse, particularly with respect to our use of animals for food. Access to more food, better food, and more meat and protein, is a driving force for progress at the developing country level, but causes enormous issues in the highly developed West. There’s certainly room for progress in agriculture and livestock management and wellbeing. Synthetic meat anyone? See, “Why Meat is the Best Worst Thing in the World” by Kurzgesagt c2018

(Max Daniel) Effective Altruism:

So while they agree that the world has been getting a lot better thanks to progress, they're also concerned that progress exposes us to new nuclear-bomb-style risks.

(Tony Morley) Human Progress – Progress Studies

I fully agree. As we attempt to solve problems and improve our personal and collective standards of living as a species, we generate other, largely unforeseen problems and consequences that require yet further solutions, mitigations or substitutions in turn. This has been an active trend in humanity for more than a hundred thousand years - a classic example of “Hatchet, Ratchet, Pivot” theory, advanced by Ruth DeFries

“Our species long lived on the edge of starvation. Now we produce enough food for all 7 billion of us to eat nearly 3,000 calories every day. This is such an astonishing thing in the history of life as to verge on the miraculous. The Big Ratchet is the story of how it happened, of the ratchets, the technologies and innovations, big and small, that propelled our species from hunters and gatherers on the savannahs of Africa to shoppers in the aisles of the supermarket. The Big Ratchet itself came in the twentieth century, when a range of technologies, from fossil fuels to scientific plant breeding to nitrogen fertilizers, combined to nearly quadruple our population in a century, and to grow our food supply even faster. To some, these technologies are a sign of our greatness to others, of our hubris. MacArthur fellow and Columbia University professor Ruth DeFries argues that the debate is the wrong one to have. Limits do exist, but every limit that has confronted us, we have surpassed. That cycle of crisis and growth is the story of our history indeed, it is the essence of The Big Ratchet. Understanding it will reveal not just how we reached this point in our history, but how we might survive it.” - The Big Ratchet, How Humanity Thrives in the Face of Natural Crisis – Ruth DeFries

(Max Daniel) Effective Altruism:

“Regarding the future, they're most worried about existential risk -- the prospect of permanently forfeiting our potential of a future that's much better than the status quo. Permanent stagnation would be an existential risk, but EAs tend to be even more worried about catastrophes from emerging technologies such as misaligned artificial intelligence or engineered pandemics. They might also be worried about a potential war between the US and China, or about extreme climate change. So in a sense they aren't as worried about progress stopping than they are about progress being mismanaged and having catastrophic unintended consequences.”

(Tony Morley) Human Progress – Progress Studies

I agree, however, civilization cannot mitigate existential risk without the courage, confidence, drive to do so. From my perspective, the human progress movement seeks to make the following broad case, 

1. All things considered; the world/living standards used to be much worse. 
2. All things considered; the world/living standards have improved dramatically for nearly everyone. 
3. All things considered; the world/living standards still need to improve dramatically / there are still problems to be solved. 
4. We should keep making the world a better place, improve living standards etc. (preferably actively)

Now, there is at times, much debate amongst the Effective Altruism community and the Human Progress community about 3) and 4), however I think we universally agree on the majority of 1) – 4). 

(Max Daniel) Effective Altruism:

“ Permanent stagnation would be an existential risk, but EAs tend to be even more worried about catastrophes from emerging technologies such as misaligned artificial intelligence or engineered pandemics. They might also be worried about a potential war between the US and China, or about extreme climate change.”

(Tony Morley) Human Progress – Progress Studies

I agree, civilization is not doing enough to consider (low likelihood, high consequence risks) such as “misaligned artificial intelligence”, “engineered pandemics” and other ‘LLHC’ risks, e.g., catastrophic climate change, nuclear war, asteroid impact, massive volcanic eruption, multi-state conventional warfare, genetically advanced selective human development (see Homo Deus: A History of Tomorrow by Yuval Noah Harari) etc. These are risks which need ongoing risk assessment and appropriate mitigation. On a side note, I have formal tertiary qualifications in risk assessment and mitigation, as it has been the field of my normal non-human progress vocation for the last decade.  

(Max Daniel) Effective Altruism:

“They therefore aim for 'differential progress' -- accelerating those kinds of technological or societal change that would safeguard us against these catastrophic risks, and slowing down whatever would expose us to greater risk. So concretely they are into things like "AI safety" or "biosecurity" -- e.g. making machine learning systems more transparent so we could tell if they were trying to deceive their users, or implementing better norms around the publication of dual-use bio research.”

(Tony Morley) Human Progress – Progress Studies

I’m afraid commentary on this section falls outside my scope of expertise. 

(Max Daniel) Effective Altruism:

“Overall, EA and the progress studies perspective agree on a lot -- they're probably closer than either would be to any other popular 'worldview'. 

(Tony Morley) Human Progress – Progress Studies

#Agreed 

(Max Daniel) Effective Altruism:

"But overall EAs probably tend to think that human progress proponents are too indiscriminately optimistic about further progress, and too generically focused on keeping progress going."

(Tony Morley) Human Progress – Progress Studies:

The human progress and progress studies movement is not a blindly optimistic look at a mission completed, or progress concluded – but rather that the world has become a much better place, while remaining in need of enormous progress. “Bad and Better”

“The solution is not to balance out all the negative news with more positive news. That would just risk creating a self-deceiving, comforting, misleading bias in the other direction. It would be as helpful as balancing too much sugar with too much salt. It would make things more exciting, but maybe even less healthy. A solution that works for me is to persuade myself to keep two thoughts in my head at the same time. It seems that when we hear someone say things are getting better, we think they are also saying “don’t worry, relax” or even “look away.” But when I say things are getting better, I am not saying those things at all. I am certainly not advocating looking away from the terrible problems in the world. I am saying that things can be both bad and better. Think of the world as a premature baby in an incubator. The baby’s health status is extremely bad and her breathing, heart rate, and other important signs are tracked constantly so that changes for better or worse can quickly be seen. After a week, she is getting a lot better. On all the main measures, she is improving, but she still has to stay in the incubator because her health is still critical. Does it make sense to say that the infant’s situation is improving? Yes. Absolutely. Does it make sense to say it is bad? Yes, absolutely. Does saying “things are improving” imply that everything is fine, and we should all relax and not worry? No, not at all. Is it helpful to have to choose between bad and improving? Definitely not. It’s both. It’s both bad and better. Better, and bad, at the same time. That is how we must think about the current state of the world.” - Factfulness, Ten Reasons We're Wrong About The World - And Why Things Are Better Than You Think – Hans Rosling, Ola Rosling, Anna Rosling Rönnlund

[Some of my tentative and uncertain views on AI governance, and different ways of having impact in that area. Excerpts, not in order, from things I wrote in a recent email discussion, so not a coherent text.]

1. In scenarios where OpenAI, DeepMind etc. become key actors because they develop TAI capabilities, our theory of impact will rely on a combination of affecting (a) 'structure' and (b) 'content'. By (a) I roughly mean how the relevant decision-making mechanisms look like irrespective of the specific goals and resources of the actors the mechanism consists of; e.g., whether some key AI lab is a nonprofit or a publicly traded company; who would decide by what rules/voting scheme how Windfall profits would be redistributed; etc. By (b) I mean something like how much the CEO of a key firm, or their advisors, care about the long-term future. -- I can see why relying mostly on (b) is attractive, e.g. it's arguably more tractable; however, some EA thinking (mostly from the Bay Area / the rationalist community to be honest) strikes me as focusing on (b) for reasons that seem ahistoric or otherwise dubious to me. So I don't feel convinced that what I perceive to be a very stark focus on (b) is warranted. I think that figuring out if there are viable strategies that rely more on (a) is better done from within institutions that have no ties with key TAI actors, and also might be best done my people that don't quite match the profile of the typical new EA that got excited about Superintelligence or HPMOR. Overall, I think that making more academic research in broadly "policy relevant" fields happen would be a decent strategy if one ultimately wanted to increase the amount of thinking on type-(a) theories of impact.

2. What's the theory of impact if TAI happens in more than 20 years? More than 50 years? I think it's not obvious whether it's worth spending any current resources on influencing such scenarios (I think they are more likely but we have much less leverage). However, if we wanted to do this, then I think it's worth bearing in mind that academia is one of few institutions (in a broad sense) that has a strong track record of enabling cumulative intellectual progress over long time scales. I roughly think that, in a modal scenario, no-one in 50 years is going to remember anything that was discussed on the EA Forum or LessWrong, or within the OpenAI policy team, today (except people currently involved); but if AI/TAI was still (or again) a hot topic then, I think it's likely that academic scholars will read academic papers by Dafoe, his students, the students of his students etc. Similarly, based on track records I think that the norms and structure of academia are much better equipped than EA to enable intellectual progress that is more incremental and distributed (as opposed to progress that happens by way of 'at least one crisp insight per step'; e.g. the Astronomical Waste argument would count as one crisp insight); so if we needed such progress, it might make sense to seed broadly useful academic research now. 

[...]

My view is closer to "~all that matters will be in the specifics, and most of the intuitions and methods for dealing with the specifics are either sort of hard-wired or more generic/have different origins than having thought about race models specifically". A crux here might be that I expect most of the tasks involved in dealing with the policy issues that would come up if we got TAI within the next 10-20 years to be sufficiently similar to garden-variety tasks involved in familiar policy areas that as a first pass: (i) if theoretical academic research was useful, we'd see more stories of the kind "CEO X / politician Y's success was due to idea Z developed through theoretical academic research", and (ii) prior policy/applied strategy experience is the background most useful for TAI policy, with usefulness increasing with the overlap in content and relevant actors; e.g.: working with the OpenAI policy team on pre-TAI issues > working within Facebook on a strategy for how to prevent the government to split up the firm in case a left-wing Democrat wins > business strategy for a tobacco company in the US > business strategy for a company outside of the US that faces little government regulation > academic game theory modeling. That's probably too pessimistic about the academic path, and of course it'll depend a lot on the specifics (you could start in academia to then get into Facebook etc.), but you get the idea.

[...]

Overall, the only somewhat open question for me is whether ideally we'd have (A) ~only people working quite directly with key actors or (B) a mix of people working with key actors and more independent ones e.g. in academia. It seems quite clear to me that the optimal allocation will contain a significant share of people working with key actors [...]

If there is a disagreement, I'd guess it's located in the following two points: 

(1a) How big are countervailing downsides from working directly with, or at institutions having close ties with, key actors? Here I'm mostly concerned about incentives distorting the content of research and strategic advice. I think the question is broadly similar to: If you're concerned about the impacts of the British rule on India in the 1800s, is it best to work within the colonial administration? If you want to figure out how to govern externalities from burning fossil fuels, is it best to work in the fossil fuel industry? I think the cliche left-wing answer to these questions is too confident in "no" and is overlooking important upsides, but I'm concerned that some standard EA answers in the AI case are too confident in "yes" and are overlooking risks. Note that I'm most concerned about kind of "benign" or "epistemic" failure modes: I think it's reasonably easy to tell people with broadly good intentions apart from sadists or even personal-wealth maximizers (at least in principle -- if this will get implemented is another question); I think it's much harder to spot cases like key people incorrectly believing that it's best if they keep as much control for themselves/their company as possible because after all they are the ones with both good intentions and an epistemic advantage (note that all of this really applies to a colonial administration with little modification, though here in cases such as the "Congo Free State" even the track record of "telling personal-wealth maximizers apart from people with humanitarian intentions" maybe isn't great -- also NB I'm not saying that this argument would necessarily be unsound; i.e. I think that in some situations these people would be correct).

(1b) To what extent to we need (a) novel insights as opposed to (b) an application of known insights or common-sense principles? E.g., I've heard claims that the sale of telecommunication licenses by governments is an example where post-1950 research-level economics work in auction theory has had considerable real-world impact, and AFAICT this kind of auction theory strikes me as reasonably abstract and in little need of having worked with either governments or telecommunication firms. Supposing this is true (I haven't really looked into this), how many opportunities of this kind are there in AI governance? I think the case for (A) is much stronger if we need little to no (a), as I think the upsides from trust networks etc. are mostly (though not exclusively) useful for (b). FWIW, my private view actually is that we probably need very little of (a), but I also feel like I have a poor grasp of this, and I think it will ultimately come down to what high-level heuristics to use in such a situation.

[Epistemic status: speculation based on priors about international organizations. I know next to nothing about the WHO specifically.]

[On the WHO declaring COVID-19 a pandemic only (?) on March 12th. Prompted by this Facebook discussion on epistemic modesty on COVID-19.]

- [ETA: this point is likely wrong, cf. Khorton's comment below. However, I believe the conclusion that the timing of WHO declarations by itself doesn't provide a significant argument against epistemic modesty still stands, as I explain in a follow-up comment below.] The WHO declaring a pandemic has a bunch of major legal and institutional consequences. E.g. my guess is that among other things it affects the amounts of resources the WHO and other actors can utilize, the kind of work the WHO and others are allowed to do, and the kind of recommendations the WHO can make.

- The optimal time for the WHO to declare a pandemic is primarily determined by these legal and institutional consequences. Whether COVID-19 is or will in fact be a pandemic in the everyday or epidemiological sense is an important input into the decision, but not a decisive one.

- Without familiarity with the WHO and the legal and institutional system it is a part of, it is very difficult to accurately assess the consequences of the WHO declaring a pandemic. Therefore, it is very hard to evaluate the timing of the WHO's declaration without such familiarity. And being even maximally well-informed about COVID-19 itself isn't even remotely sufficient for an accurate evaluation.

- The bottom line is that the WHO officially declaring that COVID-19 is a pandemic is a totally different thing from any individual persuasively arguing that COVID-19 is or will be a pandemic. In a language that would accurately reflect differences in meaning, me saying that COVID-19 is a pandemic and the WHO declaring COVID-19 is a pandemic would be done using different words. It is simply not the primary purpose of this WHO speech act to be an early, accurate, reliable, or whatever indicator of whether "COVID-19 is a pandemic", to predict its impact, or any other similar thing. It isn't primarily epistemic in any sense.

- If just based on information about COVID-19 itself someone confidently thinks that the WHO ought to have declared a pandemic earlier, they are making a mistake akin to the mistake reflected by answering "yes" to the question "could you pass me the salt?" without doing anything.

So did the WHO make a mistake by not declaring COVID-19 to be a pandemic earlier, and if so how consequential was it? Well, I think the timing was probably suboptimal just because my prior is that most complex institutions aren't optimized for getting the timing of such things exactly right. But I have no idea how consequential a potential mistake was. In fact, I'm about 50-50 on whether the optimal time would have been slightly earlier or slightly later. (Though substantially earlier seems significantly more likely optimal than substantially later.)

[Context: This is a research proposal I wrote two years ago for an application. I'm posing it here because I might want to link to it. I plan to spend a few weeks looking into a subquestion: how heavy-tailed is EA talent, and what does this imply for EA community building?]

Research proposal: Assess claims that "impact is heavy-tailed"

Why is this valuable?

EAs frequently have to decide how much resources to invest to estimate the utility of their available options; e.g.:

• How much time to invest to identify the best giving opportunity?
• How much research to do before committing to a small set of cause areas?
• When deciding whether to hire someone now or a more talented person in the future, when is it worth to wait?

One major input to such questions is how heavy-tailed the distribution of altruistic impact is: The better the best options are relative to a random option, the more valuable it is to identify the best options.

Claims like “impact is heavy-tailed” are widely accepted in the EA community—with major strategic consequences (e.g. [1], “Talent is high variance”)—but have sometimes been questioned [2, 3, 4, 5].

These claims are often made in an imprecise way, which makes it hard to estimate the extent of their practical implications (should you spend a month or a year doing research before deciding?), and hard to check if one actually disagrees about them. E.g., is the claim that we can now see that Einstein did much more for progress in physics than 90% of the world population at his time, or that in 1900 our subjective expected value for the progress Einstein would make would have been much higher than the value for a random physics graduate student, or something in between?

Suggested approach

1. Collect several claims of this type that have been made.
2. Review statistical measures of heavy-tailedness.
3. Limit the project’s scope appropriately. E.g., focus just on the claim that “talent is heavy-tailed” and its implications for community building.
4. Refine claims into precise candidate versions, i.e. something like “looking backwards, the empirical distribution of the number of published papers by researcher looks like it was sampled from a distribution that doesn’t have finite variance” rather than “researcher talent is heavy-tailed”.
5. Assess the veracity of those claims, based on published arguments about them and general properties of heavy-tailed distributions (e.g. [6]). Perhaps gather additional data.
6. Write up the results in an accessible way that highlights the true, precise claims and their practical implications.

Concerns

• There probably are good reasons for why “impact is heavy-tailed” is widely accepted. I’m therefore unlikely to produce actionable results.
• The proposed level of analysis may be too general.

[A failure mode of culturally high mental health awareness.]

In my experience, there is a high level of mental health awareness in the EA community. That is, people openly talk about mental health challenges such as depression, and many will know about how to help people facing such challenges (e.g. by helping them to get professional treatment). At least more so than in other communities I've known.

I think this is mostly great, and on net much preferable over low mental health awareness.

However, I recently realized one potential failure mode: There is a risk of falsely overestimating another individual's mental health awareness. For example, suppose I talk to an EA who appears to struggle with depression; I might then think "surely they know that depression is treatable, and most likely they're already doing CBT", concluding there isn't much I can do to help. I might even think "it would be silly for me to mention CBT because it's common knowledge that depression can often be treated that way, and stating facts that are common knowledge is at best superfluous and at worst insulting (because I'd imply the other person might lack some kind of basic knowledge)".

Crucially, this would be a mistake even if I was correct that the person I'm talking to was, by virtue of exposure to the EA community, more likely than usual to have heard of CBT. This is because of a large asymmetry in value: It can be extremely valuable for both one's personal well-being and one's expected impact on the world to e.g. start treatment for depression; the cost of saying something obvious or even slightly annoying pales by comparison.

This suggests a few lessons:

• If you may be able to help someone cope with mental health challenges, don't forfeit that opportunity just because you assume they've already got all the help they could. (Of course, there may be other valid reasons: e.g., it could be too costly, or inappropriate in a particular context.) Even if you've recently talked to many similar people of whom this was true.
• If you're still worried about stating the obvious, it may help to be explicit about why. E.g., "I know you may be aware of this, but I'd like to mention something because I think it could have outsized importance if not".

[A rebuttal of one argument for hard AI takeoff due to recursive self-improvement which I'm not sure anyone was ever making.

Wrote this as a comment in a Google doc, but thought I might want to link to it sometimes.]

I'm worried that naive appeals to self-improvement are sometimes due to a fallacious inference from current AIs / computer programs to advanced AIs. I.e. the implicit argument is something like:

1. It's much easier to modify code than to do brain surgery.
2. Therefore, advanced AI (self-)improvement will be much easier than human (self-)improvement.

But my worry is that the actual reason why "modifying code" seems more feasible to us is the large complexity difference between current computer programs and human cognition. Indeed, at a physicalist level, it's not at all clear that moving around the required matter on a hard disk or SDD or in RAM or whatever is easier than moving around the required matter in a brain. The difference instead is that we have developed intermediate levels of abstraction (from assembly to high-level programming languages) that massively facilitate the editing process -- they bridge the gap between the hardware level and our native cognition in just the right way. But especially to someone with functionalist or otherwise substrate-neutral inclinations it may seem likely that they key feature that enabled us to construct the intermediate-level abstractions was precisely the small complexity of the "target cognition" compared to our native cognition.

**Would some moral/virtue intuitions prefer small but non-zero x-risk?**

[Me trying to lean into a kind of philosophical reasoning I don't find plausible. Not important, except perhaps as cautionary tale for what kind of things could happen if you wanted to base the case for reducing x-risk on purely non-consequentialist reasons.]

(Inspired by a conversation with Toby Newberry about something else.)

The basic observation: we sometimes think that a person achieving some success is particularly praiseworthy, remarkable, virtuous, or similar if they could have failed. (Or if they needed to expend a lot of effort, suffered through hardship, etc.)

Could this mean that we removed one source of value if we reduced x-risk to zero? Achieving our full potential would then no longer constitute a contingent achievement - it would be predetermined, with failure no longer on the table.

We can make the thought more precise in a toy model: Suppose that at some time t_0 x-risk is permanently reduced to zero. The worry is that acts happening after t_0 (or perhaps acts of agents born after t_0), even if they produce value, are less valuable in one respect: In their role of being a part of us eventually achieving our full potentially, they can no longer fail. More broadly, humanity's great generation-spanning project (whatever that is) can no longer fail. Those humans living after t_0 therefore have a less valuable part in that project. They merely help push along a wagon that was set firmly in its tracks by their ancestors. Their actions may have various valuable consequences, but they no longer increase the probability of humanity's grand project succeeding.

(Similarly, if we're inclined to regard humanity as a whole, or generations born after t_0, as moral agents in their own right we might worry that zero x-risk detracts from the value of their actions.)

Some intuition pumps:

• Suppose that benevolent parents arrange things (perhaps with the help of fantastic future technology) in such a way that their child will with certainty have a highly successful career. (They need not remove that child's agency or autonomy: perhaps the child can choose if she'll become a world-class composer, scientist, political leader, etc. - but no matter what she does, she'll excel and will be remembered for their great achievements.) We can even suppose that it will seem to the child as if it could have failed, and that she did experience small-scale setbacks: perhaps she went rock climbing, slipped, and broke her leg; but unbeknownst to her she was watched by invisible drones that would have caught her if she would have otherwise died. And so on for every single moment of her life.
• Suppose that the matriarch of a family dynasty arranges things in such a way that the family business's success will henceforth be guaranteed. Stock price, cash flow, and so on may still go up and down, but the firm can no longer go bankrupt and will broadly be the most profitable company in its sector, forever. Her heirs might on some occasions think that their management has turned things around, and prevented the company's downfall. But really the shots were called by a great grandmother generations ago.

We may think that there is something objectionable, even dystopian about these situations. At the very least, we may think that the apparent successes of the child prodigy or the family business leaders count for less because, in one respect, they could not have failed.

If we give a lot of weight to such worries we may not want to eliminate x-risk. Instead, perhaps, we'd conclude that it's best to carefully manage x-risk: at any point, it should not be so high that we run an irresponsible risk of squandering all future potential - but it also should not be so low that our children are robbed of more value than we protect.

--

Some reasons why I think this is either implausible or irrelevant:

• I think the intuitions underlying the basic observation aren't about axiology, probably not even about normative ethics in any sense. I think they are best explained away by evolutionary debunking arguments, or taken to be heuristics about instrumental value, or (if one wants to take them as seriously as plausibly defensible) about some primitive notion of 'virtue' or 'praiseworthiness' that's different from axiology.
• This at least partly relies on intuitions about the import of "could have done otherwise" claims, which may be confused or deceptive anyway. For example, we might think that Frankfurt-style examples show that such claims aren't actually relevant for moral responsibility, despite appearing to be at first glance. And we might think this is a reason to also not take them seriously in this context.
• Even if there's something to this, on plausible views the loss of value from removing x-risk would be overcompensated by the gains.
• Reducing x-risk to exactly zero, as opposed to something tiny, is implausible anyway. (Though we might worry that all of this applies if x-risk falls below some positive threshold, or that there is some gradual loss of value as x-risk falls.)
• The worry straightforwardly only applies in scenarios where we can permanently reduce x-risk to zero, no matter what future agents do. This seems extremely implausible. Even if we could reduce x-risk to zero, most likely it would require the continuing effort of future agents to keep it at zero.