A while back (as I've just been reminded by a discussion on another thread), David Thorstad wrote a bunch of posts critiquing the idea that small reductions in extinction risk have very high value, because the expected number of people who will exist in the future is very high: https://reflectivealtruism.com/category/my-papers/mistakes-in-moral-mathematics/. The arguments are quite complicated, but the basic points are that the expected number of people in the future is much lower than longtermists estimate because:
-Longtermists tend to neglect the fact that even if your intervention blocks one extinction risk, there are others it might fail to block; surviving for billions (or more) of years likely requires driving extinction risk very low for a long period of time, and if we are not likely to survive that long, even conditional on longtermist interventions against one extinction risk succeeding, the value of preventing extinction (conditional on more happy people being valuable) is much lower.
-Longtermists tend to assume that in the future population will be roughly as large as the available resources can support. But ever since the industrial revolution, as countries get richer, their fertility rate falls and falls until it is below replacement. So we can't just assume future population sizes will be near the limits of what the available resources will support.
Thorstad goes on to argue that this weakens the case for longtermism generally, not just the value of extinction risk reductions, since the case for longtermism is that future expected population is many times the current population, or at least could be given plausible levels of longtermist extinction risk reduction effort. He also notes that if he can find multiple common mistakes in longtermist estimates of expected future population, we should expect that those estimates might be off in other ways. (At this point I would note that they could also be missing factors that bias their estimates of future population size down rather than up: Thorstad probably hasn't been looking for those with the same level of effort.)
I am interested in whether there has been any kind of "official" or quasi-official response to Thorstad on these points from leading EA orgs or at least leading individual long-termists. (I know there has been discussion in forum comments, but I mean something more than that.) After all, 80k has now effectively gone all in on AI risk as the cause partly on the basis of longtermist arguments (though they've always been a primarily longtermist org I think) and Open Phil also spends a lot of money on projects that arguably are only amongst the most effective uses of the money if longtermism is true. (It's possible I guess that AI safety work could be high expected value per $ just for saving current lives.) Thorstad used to work for the Global Priorities Institute and I think it is great that they were prepared to employ someone to repeteadly harshly critique the famous theory they are most associated with. But there's not much point in EA soliciting serious criticism if people then mostly ignore it.
Reactions
7 Answers sorted by
I know David well, and David, if you're reading this, apologies if it comes across as a bit uncharitable. But as far as I've ever been able to see, every important argument he makes in any of his papers against longtermism or the astronomical value of x-risk reduction was refuted pretty unambiguously before it was written. An unfortunate feature of an objection that comes after its own rebuttal is that sometimes people familiar with the arguments will skim it and say "weird, nothing new here" and move on, and people encountering it for the first time will think no response has been made.
For example,[1] I think the standard response to his arguments in "The Scope of Longtermism" would just be the Greaves and MacAskill "Case for Strong Longtermism".[2] The Case, in a nutshell, is that by giving to the Planetary Society or B612 Foundation to improve our asteroid/comet monitoring, we do more than 2x as much good in the long term, even on relatively low estimates of the value of the future, than giving to the top GiveWell charity does in the short term. So if you think GiveWell tells us the most cost-effective way to improve the short term, you have to think that, whenever your decision problem is "where to give a dollar", the overall best action does more good in the long term than in the short term.
You can certainly disagree with this argument on various grounds--e.g. you can think that non-GiveWell charities do much more good in the short term, or that the value of preventing extinction by asteroid is negative, or for that matter that the Planetary Society or B612 Foundation will just steal the money--but not with the arguments David offers in "The Scope of Longtermism".
His argument [again, in a nutshell] is that there are three common "scope-limiting phenomena", i.e. phenomena that make it the case that the overall best action does more good in the long term than in the short term in relatively few decision situations. These are
- rapid diminution (the positive impact of the action per unit time quickly falls to 0),
- washing out (the long-term impact of the action has positive and negative features which are hard to predict and cancel out in expectation), and
- option unawareness (there's an action that would empirically have large long-term impact but we don't know what it is).
He grants that when Congress was deciding what to do with the money that originally went into an asteroid monitoring program called the Spaceguard Survey, longtermism seems to have held. So he's explicitly not relying on an argument that there isn't much value to trying to prevent x-risk from asteroids. Nevertheless, he never addresses the natural follow-up regarding contributing to improved asteroid monitoring today.
Re (1), he cites Kelly (2019) and Sevilla (2021) as reasons to be skeptical of claims from the "persistence" literature about various distant cultural, technological, or military developments having had long-term effects on the arc of history. Granting this doesn't affect the Case that whenever your decision problem is "where to give a dollar", the overall best action does more good in the long term than in the short term.[3]
Re (2), he says that we often have only weak evidence about a given action's impact on the long-term future. He defends this by pointing out (a) that attempts to forecast actions' impacts on a >20 year timescale have a mixed track record, (b) that professional forecasters are often skeptical of the ability to make such forecasts, and (c) that the overall impact of an action on the value of world is typically composed of its impacts on various other variables (e.g. the number of people and how well-off they are), and since it's hard to forecast any of these components, it's typically even harder to forecast the action's impact on value itself. None of this applies to the Case. We can grant that most actions have hard-to-predict long-term consequences, and that forecasters would recognize this, without denying that in most decision-situations (including all those where the question is where to give a dollar), there is one action that has long-term benefits more than 2x as great as the short-term benefits of giving to the top GiveWell charity: namely the action of giving to the Planetary Society or B612 Foundation. There is not a mixed track record of forecasting the >20 year impact of asteroid/comet monitoring, and no evidence that professional forecasters are skeptical of making such forecasts, and he implicitly grants that the complexity of forecasting its long-term impact on value isn't an issue in this case when it comes to the Space Guard Survey.
Re (3), again, the claim the Case makes is that we have identified one such action.
- ^
I also emailed him about an objection to his "Existential Risk Pessimism and the Time of Perils" in November and followed up in February, but he's responded only to say that he's been too busy to consider it.
- ^
Which he cites! Note that Greaves and MacAskill defend a stronger view than the one I'm presenting here, in particular that all near-best actions do much more good in the long term than in the short term. But what David argues against is the weaker view I lay out here.
- ^
Incidentally, he cites the fact that "Hiroshima and Nagasaki returned to their pre-war population levels by the mid-1950s" as an especially striking illustration of lack of persistence. But as I mentioned to him at the time, it's compatible with the possibility that those regions have some population path, and we "jumped back in time" on it, such that from now on the cities always have about as many people at t as they would have had at t+10. If so, bombing them could still have most of its effects in the future.
I'm sure @David Thorstad can defend himself here, but to quote his most recent post:
The claim, then, cannot be that the scope of longtermism is empty. The claim is rather that cases such as the Space Guard Survey are more special than they may appear. When we turn our eyes to many other proposed existential risk mitigation efforts, the scope-limiting factors begin to get a more solid take.
Ie: he has acknowledged already that his objections don't really apply to asteroid monitoring, a field where predictions are backed to scientific precision. But he says t...
First, to clarify, Greaves and MacAskill don’t use the Spaceguard Survey as their example. They use giving to the Planetary Society or B612 Foundation as their example, which do similar work.
Could you spell out what you mean by “the actual scope of longtermism”? In everyday language this might sound like it means “the range of things it’s justifiable to work on for the sake of improving the long term”, or something like that, but that’s not what either Thorstad or Greaves and MacAskill mean by it. They mean [roughly; see G&M for the exact definition] the set of decision situations in which the overall best act does most of its good in the long term.
Long before either of these papers, people in EA (and of course elsewhere) had been making fuzzy arguments for and against propositions like “the best thing to do is to lower x-risk from AI because this will realize a vast and flourishing future”. The project G&M, DT, and other philosophers in this space were engaged in at the time was to go back and carefully, baby step by baby step, formalize the arguments that go into the various building blocks of these "the best thing to do is..." conclusions, so that it’s easier to identify which elements of the overall conclusion follow from which assumptions, how someone might agree with some elements but disagree with others, and so on. The “[scope of] longtermism” framing was deliberately defined broadly enough that it doesn’t make claims about what the best actions are: it includes the possibility that giving to the top GiveWell charity is the best act because of its long-term benefits (e.g. saving the life of a future AI safety researcher).
The Case offers a proof that if you accept the premises (i) giving to the top GiveWell charity is the way to do the most good in the short term and (ii) giving to PS/B612F does >2x more good [~all in the long term] than the GiveWell charity does in the short term, then you accept (iii) that the scope of longtermism includes every decision situation in which you’re giving money away. It also argues for premise (ii), semi-formally but not with anything like a proof.
Again, the whole point of doing these sorts of formalizations is that it helps to sharpen the debate: it shows that a response claiming that the scope of longtermism is actually narrow has to challenge one of those premises. All I’m pointing out is that Thorstad's “Scope of Longtermism” doesn’t do that. You’ve done that here, which is great: maybe (ii) is false because giving to PS/B612F doesn’t actually do much good at all.
By "scope of longtermism" I took Thorstad's reference to "class of decision situations" in terms of permutations to be evaluated (maximising welfare, maximising human proliferation, minimising suffering etc) rather than categories of basic actions (spending, voting, selecting clothing).[1] I'm not actually sure it makes a difference to my interpretation of the thrust of his argument (diminution, washing out and unawareness means solutions whose far future impact swamps short term benefits are vanishingly rare and generally unknowable) either way.
Sure, Thorstad absolutely starts off by conceding that under certain assumptions about the long term future,[2] a low probability but robustly positive action like preparing to stop asteroids from hitting earth which indirectly enables benefits to accrue over the very long term absolutely can be a valid priority.[3] But it doesn't follow that one should prioritise the long term future in every decision making situation in which money is given away. The funding needs of asteroid monitoring sufficient to alert us to impending catastrophe are plausibly already met[4], and his core argument is we're otherwise almost always clueless about what the [near] best solution for the long term future is. It's not a particularly good heuristic to focus spending on outcomes you are most likely to be clueless about, and a standard approach to accumulation of uncertainty is to discount for it, which of course privileges the short term.
I mean, I agree that Thorstad makes no dent in arguments to the effect that if there is an action which leads to positive utility sustained over a very long period of time for a very large number of people it will result in very high utility relative to actions which don't have that impact: I'm not sure that argument is even falsifiable within a total utilitarian framework.[5] But I don't think his intention is to argue with [near] tautologies, so much as to insist that the set of decisions which credibly result in robustly positive long term impact is small enough to usually be irrelevant.
- ^
all of which can be reframed in terms of making money to spend available to spend on priorities" in classic "hardcore EA" style anyway...
- ^
Some of the implicit assumptions behind the salience of asteroid x-risk aren't robust: if AI doomers are right then that massive positive future we're trying to protect looks a lot smaller. On the other hand compared with almost any other x-risk scenario, asteroids are straightforward: we don't have to factor in the possibility of asteroids becoming sneaky in response to us monitoring them, or attach much weight to the idea that informing people about asteroids will motivate them to try harder to make it hit the earth.
- ^
you correctly point out his choice of asteroid monitoring service is different from Greaves and MacAskill's. I assume he does so partly to steelman the original, as the counterfactual impact of a government agency incubating the first large-scale asteroid monitoring programme is more robust than that of the marginal donation to NGOs providing additional analysis. And he doesn't make this point, but I doubt the arguments that decided its funding actually depended on the very long term anyway....
- ^
this is possibly another reason for his choice of asteroid monitoring service...
- ^
Likewise, pretty much anyone familiar with total utilitarianism can conceive a credible scenario in which the highest total utility outcome would be to murder a particular individual (baby Hitler etc), and I don't think it would be credible to insist such a situation could never occur or never be known. This would not, however, fatally weaken arguments against the principle of "murderism" that focused on doubting there were many decision situations where murder should be considered as a priority
Thanks for saying a bit more about how you’re interpreting “scope of longtermism”. To be as concrete as possible, what I'm assuming is that we both read Thorstad as saying “a philanthropist giving money away so as to maximize the good from a classical utilitarian perspective” is typically outside the scope of decision-situations that are longtermist, but let me know if you read him differently on that. (I think it’s helpful to focus on this case because it’s simple, and the one G&M most clearly argue is longtermist on the basis of those two premises.)
It’s a tautology that the G&M conclusion that the above decision-situation is longtermist follows from the premises, and no, I wouldn't expect a paper disputing the conclusion to argue against this tautology. I would expect it to argue, directly or indirectly, against the premises. And you’ve done just that: you’ve offered two perfectly reasonable arguments for why the G&M premise (ii) might be false, i.e. giving to PS/B612F might not actually do 2x as much good in the long term as the GiveWell charity in the short term. (1) In footnote 2, you point out that the chance of near-term x-risk from AI may be very high. (2) You say that the funding needs of asteroid monitoring sufficient to alert us to impending catastrophe are plausibly already met. You also suggest in footnote 3 that maybe NGOs will do a worse job of it than the government.
I won’t argue against any of these possibilities, since the topic of this particular comment thread is not how strong the case for longtermism is all things considered, but whether Thorstad’s “Scope of LTism” successfully responds to G&M’s argument. I really don't think there's much more to say. If there’s a place in “Scope of LTism” where Thorstad offers an argument against (i) or (ii), as you’ve done, I’m still not seeing it.
Carl Shulman's response here responds to objection 1. You can also see the tag for the time of perils hypothesis for a bit more discussion.
On 2, the structure of the objection is similar to Shulman's response on 1: we're not vanishingly unlikely to have very large (or even near maximal) population sizes. For instance, a variety of people (including longtermists) are interested in ultimately creating vast numbers of digital minds or other sources of value and there aren't clearly opposing groups which directly have preferences against this happening. I don't see a very strong analogy between current low fertility and long run cosmic resource utilization, and at a more basic level, current low fertility isn't stable: even if the status quo continues for a long time (without e.g. the creation of powerful AI resulting in much faster progress in technology), selection will likely lead to the fertility rate increasing at some point in the future unless this is actively suppressed.
Thanks for the good points, Ryan.
I can see the annual probability of the absolute value of the welfare of Earth-originating beings dropping to 0 becoming increasingly low, and their population increasingly large. However, I do not think this means decreasing the nearterm risk of human extinction is more cost-effective than donating to GiveWell's top charities, or organisations working on invertebrate welfare.
Longtermists often estimate the expected value of the future from EV = p*V = "probability of reaching existential safety"*"expected value of the futur...
We asked David about longtermists' responses to his work in the podcast episode we did with him. Here's the (rough, automatically generated) transcript, but you can listen to the relevant section here; it starts at ~33:50.
David: I think to contextualize that, let me use the word I'm going to use in my book, namely a strategy of shedding zeros. So, longtermists say, look, the axiological case for longtermism is 10 orders of magnitude or 15 orders of magnitude better than the case for competing short-termist interventions.
So, therefore, unless you are radically non-consequentialist, longtermism is going to win at the level. And I want to chip away a lot of zeros in those value estimates, and then maybe do some other deontic things too. And so if the longtermist is just in one swoop gonna hand me five or ten or twenty zeros, I think there's two things to say. The first is they might run out of zeros just there.
5, 10, 20 orders of magnitude is a lot. But the second is this isn't the only time I'm gonna ask them for some orders of magnitude back. And this thing that they do, which is correct, is they point at every single argument I make and they say "I can afford to pay that cost and that cost and that cost." But the question is whether they can afford to pay them all together, and I think, at least as the line of the argument in my book, that if we're really tossing orders of magnitude around that freely, we're probably going to run out of orders of magnitude quite quickly.
Leah: Got it. Okay. And, I just want to follow up on the last thing you said. So has that been the response of the people who are writing on these issues? Like, do they read your work and say, yeah, I concede that?
David: I get, well, sometimes it's concessive, sometimes it's not, but almost always, somebody raised their hand, they say, David, couldn't I believe that, and still be a longtermist? So I had to rewrite some of the demographics section in my paper. They said, look, aren't you uncertain about demographics?
Maybe there's a 10 to the 8th probability I'm right about demographics, so maybe I lose eight orders of magnitude, and the response there is, okay, maybe you do. And then they'll say about the time of perils, maybe there's a 10 to the 9th chance I'm right about the time of perils, maybe I lose nine orders of magnitude, and okay, you do.
Obviously, we have a disagreement about how many orders of magnitude are lost each time, but I think it's a response I see in isolation every time I give a paper, and I'd like people to see it as a response that works in isolation, but can't just keep being repeated.
Rethink Priorities did a more comprehensive survey on models of future x-risk and population sizes.
I'm going to actually disagree with your initial premise - the basic points are that the expected number of people in the future is much lower than longtermists estimate - because, at least in the Reflective Altruism blog series, I don't see that as being the main objection David has to (Strong) Longtermism. Instead, I think he instead argues That the interventions Longtermists support require additional hypotheses (the time of perils) which are probably false and that the empirical evidence longtermists give for their existential pessimism are often non-robust on further inspection.[1] Of course my understanding is not complete, David himself might frame it differently, etc etc.
One interesting result from his earlier Existential risk pessimism and the time of perils paper is that on a simple model, though he expands the results to more complex ones, people with low x-risk should be longtermists about value, and those with high x-risk estimates should be focused on the short term, which is basically the opposite of what we see happening in real life. The best way out for the longtermist, he argues, is to believe in 'the time of perils hypothesis'. I think the main appeals to this being the case are either a) interstellar colonisation giving us existential security so we're moral value isn't tethered to one planet,[2] or of course from b) aligned superintelligence allowing us unprecedented control over the universe and the ability to defuse any sources of existential risk. But of course, many working on Existential AI Risk are actually very pessimistic about the prospects for alignment and so, if they are longtermist,[3] why aren't they retiring from technical AI Safety and donating to AMF? More disturbingly, are longtermists just using the 'time of perils' belief to backwards-justify their prior beliefs that interventions in things like AI are the utilitarian-optimal interventions to be supporting? I haven't seen a good longtermist case answering these questions, which is not to say that one doesn't exist.
Furthermore, in terms of responses from EA itself, what's interesting is that when you look at the top uses of the Longtermism tag on the Forum, all of the top 8 were made ~3 years ago, and only 3 of the top 20 within the last 3 years. Longtermism isn't a used a lot even amongst EA any more - the likely result of negative responses from the broader intelligensia during the 2022 soft launch, and then the incredibly toxic result of the FTX collapse shortly after the release of WWOTF. So while I find @trammell's comment below illuminating in some aspects about why there might be fewer responses than expected, I think sociologically it is wrong about the overarching reasons - I think longtermism doesn't have much momentum in academic philosophical circles right now. I'm not plugged into the GPI-Sphere though, so I could be wrong about this.
So my answer to your initial question is "no" if you mean 'something big published post-Thorstad that responds directly or implicitly to him from a longtermist perspective'. Furthermore, were they to do so, or to point at one already done (like The Case for Strong Longtermism) I'd probably just reject many of the premises that give the case legs in the first place, such as that it's reasonable to do risk-neutral-expected-value reasoning about the very long run future in the first place as a guide to moral action. Nevertheless, other objections to Longtermism I am sympathetic to are those from Eric Schwitzgebel (here, here) among others. I don't think this is David's perspective though, I think he believes that the empirical warrant for the claims aren't there but that he would support longtermist policies if he believed they could be supported this way.
I'm also somewhat disturbed by the implication that some proportion of the EA Brain-Trust, and/or those running major EA/AI Safety/Biorisk organisations, are actually still committed longtermists or justify their work in longtermist terms. If so they should make sure this is known publicly and not hide it. If you think your work on AI Policy is justified on strong longtermist grounds, then I'd love to see your the model used for that, and parameters used for the length of the time of perils, the marginal difference to x-risk the policy would make, and the evidence backing up those estimates. Like if 80k have shifted to be AI Safety focused because of longtermist philosophical commitments, then lets see those commitments! The inability of many longtermist organisations to do that is a sign of what Thorstad calls the regression to the inscrutable,[4] which is I think one of his stronger critiques.
- ^
Disagreement about future population estimates would be a special case of the latter here
- ^
In The Epistemic Challenge to Longtermism, Tarsney notes that:
More concretely, the case for longtermism seems to depend to a significant
extent on the possibility of interstellar settlement - ^
Note these considerations don't apply to you if you're not an impartial longtermist, but then again, if many people working in this area don't count themselves as longtermists, it certainly seems like a poor sign for longtermism
- ^
Term coined in this blog post about WWOTF
- ^
A general good rule for life
- ^
(I am not a time-invariant-risk-neutral-totally-impartial-utilitarian, for instance)
Fair enough, I think the lack of a direct response has been due to an interaction between the two things. At first, people familiar with the existing arguments didn't see much to respond to in David's arguments, and figured most people would see through them. Later, when David's arguments had gotten around more and it became clear that a response would be worthwhile (and for that matter when new arguments had been made which were genuinely novel), the small handful of people who had been exploring the case for longtermism had mostly moved on to other proje...
I think the standard response by longtermists is encapsulated in Tarsney's "The Epistemic Challenge to Longtermism": https://link.springer.com/article/10.1007/s11229-023-04153-y Tarsney concludes: "if we simply aim to
maximize expected value, and don’t mind premising our choices on minuscule
probabilities of astronomical payoffs, the case for longtermism looks robust.
But on some prima facie plausible empirical worldviews, the expectational
superiority of longtermist interventions depends heavily on these ‘Pascalian’
probabilities. So the case for longtermism may depend either on plausible but
non-obvious empirical claims or on a tolerance for Pascalian fanaticism." I don't have time to compare the two papers at the moment, but - in my memory - the main difference to Thorstad's conclusion is that Tarsney explicitly considers uncertainty about different models and model parameters regarding future population growth and our possibility to affect the probability of extinction.
First, to clarify, Greaves and MacAskill don’t use the Spaceguard Survey as their example. They use giving to the Planetary Society or B612 Foundation as their example, which do similar work.
Could you spell out what you mean by “the actual scope of longtermism”? In everyday language this might sound like it means “the range of things it’s justifiable to work on for the sake of improving the long term”, or something like that, but that’s not what either Thorstad or Greaves and MacAskill mean by it. They mean [roughly; see G&M for the exact definition] the set of decision situations in which the overall best act does most of its good in the long term.
Long before either of these papers, people in EA (and of course elsewhere) had been making fuzzy arguments for and against propositions like “the best thing to do is to lower x-risk from AI because this will realize a vast and flourishing future”. The project G&M, DT, and other philosophers in this space were engaged in at the time was to go back and carefully, baby step by baby step, formalize the arguments that go into the various building blocks of these "the best thing to do is..." conclusions, so that it’s easier to identify which elements of the overall conclusion follow from which assumptions, how someone might agree with some elements but disagree with others, and so on. The “[scope of] longtermism” framing was deliberately defined broadly enough that it doesn’t make claims about what the best actions are: it includes the possibility that giving to the top GiveWell charity is the best act because of its long-term benefits (e.g. saving the life of a future AI safety researcher).
The Case offers a proof that if you accept the premises (i) giving to the top GiveWell charity is the way to do the most good in the short term and (ii) giving to PS/B612F does >2x more good [~all in the long term] than the GiveWell charity does in the short term, then you accept (iii) that the scope of longtermism includes every decision situation in which you’re giving money away. It also argues for premise (ii), semi-formally but not with anything like a proof.
Again, the whole point of doing these sorts of formalizations is that it helps to sharpen the debate: it shows that a response claiming that the scope of longtermism is actually narrow has to challenge one of those premises. All I’m pointing out is that Thorstad's “Scope of Longtermism” doesn’t do that. You’ve done that here, which is great: maybe (ii) is false because giving to PS/B612F doesn’t actually do much good at all.