If we try to maximize expected utility with an unbounded utility function, we will sometimes be reckless: we will accept gambles with an arbitrarily small chance of success if the payout is large enough. And it is not just expected utility maximizers who encounter this problem. Beckstead and Thomas have shown that any decision framework will be either reckless, timid (unwilling to take obviously good gambles), or non-transitive. This issue becomes important when considering the case for strong longtermism, which says that protecting the far future is overwhelmingly important because of the small probability that it contains an astronomically huge amount of value. It is also at the heart of the Pascal mugger problem, where someone threatens to use alleged supernatural powers to do some astronomically huge amount of harm/good, unless/if we give them our wallet.
But there is a loophole in these arguments. This loophole allows us to avoid the most problematic implications of recklessness in practice. The loophole: we can choose to adopt prior probabilities which make it unlikely that our actions would have such large effects. The larger the potential effect of our action in some scenario, the less probability we assign that scenario in our prior, in proportion. We can give zero probability to scenarios that would allow us to influence infinite utility, removing all of the associated issues that infinite value introduces. I usually see this referred to as the "dogmatic" response to recklessness. We simply adopt extreme confidence that the Pascal mugger is lying to us, or mistaken. If they claim to be able to do infinite harm, then we say that they are lying, or maybe mistaken, with certainty.
It seems to me that given results like Beckstead and Thomas, dogmatism is the most promising way to justify the obvious fact that it is not irrational to refuse to hand over your wallet to a Pascal mugger. (If anyone disagrees that this is an obvious fact, please get in touch, and be prepared to hand over lots of cash).
It is true that penalising large utilities in our priors will not eliminate recklessness entirely. For example, a 1/x decay in our credences could still lead us to take some long-shot gambles. Also, although our prior might severely penalise the possibility of our actions having large effects, we may encounter evidence which causes us to update our initial scepticism. But I think a dogmatic prior would eliminate the most problematic forms of reckless behaviour. To convince a dogmatic decision maker to be reckless, you would now need to make an argument of the form:
Here is a good reason to believe that X has this particular small but non-tiny chance of leading to Y, and Y would be extremely good/bad, so you should do this reckless thing.
But a dogmatic decision maker would not be susceptible to arguments of the form:
We can't rule out that X would lead to Y, so you surely can't assign it that small a probability, and Y would be extremely good/bad, so you should do this reckless thing.
In other words: we avoid the Pascal mugger problem, as well as, I believe, the strongest form of longtermism. But we continue to allow other reckless conclusions that actually seem perfectly fine, like the expected utility based justification for why you should vote in an election (there is a ~1/N chance of your vote changing the result in a close election, and ~N impact on utility if it does, where N is the number of voters).
Given how neat this solution to the problem seems to be, I am confused about why I don't see it defended more often. I believe it is defended by Holden Karnofsky here. But apart from that, I typically see this solution raised, labelled as 'dogmatic' (with obvious negative connotations), and then the discussion moves on. I'm not a philosopher though. I would be interested to read anything anyone can point me to that discusses this in more depth.
In the next section, I'll try to explain what I think is supposed to be wrong with the dogmatic approach. Then, in the following section, I'll explain why I don't actually find this problem to be that bad. I suggest that there could be a close analogy between dogmatism and Occam's razor: both are assumptions that maybe can't be justified in a purely epistemological way, but which nevertheless should be adopted by practical decision makers.
What I think is supposed to be wrong with dogmatism
The obvious objection to dogmatism is that it seems to be a form of motivated reasoning. It tells us to adopt extreme confidence that certain claims are false, for apparently no good reason except that we dislike their consequences. Isn't this the wrong way for a truth-seeker to behave?
Here is another way of looking at it: the dogmatic approach looks like it has things backwards. Epistemology should come first, and decision theory second. That is, first, we should look at the world and form beliefs. Then, given those beliefs, we should try to make good decisions. Proposing dogmatic priors as a solution to recklessness seems strange, because it is an argument which goes in the other direction. It starts by considering problems in decision theory, and then lifts those into conclusions about epistemology, which feels wrong.
Holden Karnofsky's blogpost side-steps this criticism by attempting to provide a first principles defence of dogmatic priors. In his account, the prior is not chosen because it resolves the Pascal mugger problem. That is merely a nice consequence. Instead he argues that a dogmatic prior should emerge naturally from our 'life experience':
Say that you’ve come to believe – based on life experience – in a “prior distribution” for the value of your actions, with a mean of zero and a standard deviation of 1.
(and he advocates taking a normal, or log-normal, distribution with these parameters).
He then argues that you can refuse the Pascal mugger on this basis.
But I can see how this argument might not be convincing. Something seems wrong with using your experience of ordinary every-day past decisions to make such a confident judgement on the possibility of this mugger being a sorceror from another dimension. If you go around making these kinds of conclusions based on your past life experiences, that seems like it would lead you into an unjustified level of scepticism of 'black swan' events. It is also not clear to me how Karnofsky's argument would handle the possibility of infinite value.
Personally, I think I'd like to make the dogmatic assumption a part of my true prior. That is, the probability distribution I adopt before I have looked at any evidence at all. It is one of my starting assumptions, and not derived from my life experience. But if I do that, then it does look like I might be open to the 'motivated reasoning' criticism described above.
Why I think dogmatism isn't so bad: the analogy with Occam's razor
In the last section I described how dogmatism feels wrong because its logic seems to go in the wrong direction: from decisions to epistemology, instead of from epistemology to decisions. But in this section I explain why I'm actually not so concerned about this, because I think all of our knowledge about the world ultimately depends on a similarly suspicious argument.
I'll start by explicitly making the argument for dogmatic priors as best I can, and then I'll discuss the problem of induction and Occam's razor, and why the best defence of Occam's razor probably takes a similar form.
An (outline of a possible) argument for dogmatism:
- We should make decisions to maximize expected utility (see e.g. von Neumann theorem, Savage's axioms)
- There should be no bound on our utility function, at least when we're doing ethics (if an action affecting N individuals has utility X, the same action affecting 2N individuals should have utility 2X).
- If there are actions available to us with infinite expected utility, then decision making under an expected utility framework becomes practically impossible (I'm sure you could write many papers on whether this claim is true or not, but here I'm just going to take it for granted).
- We should therefore adopt dogmatic priors which penalise large utilities. This is the only way to rule out the possibility of infinite expected utility actions. (Note: the penalising of large finite utilities comes for free, even though we only required that infinite expected value be removed from the theory, because we need to rule out St Petersberg paradox type scenarios).
- If dogmatic priors are a good model for the world, we make good decisions, and if they're not, we were always doomed anyway.
This argument might seem unsatisfying, but I think there could be a close analogy between this argument and the basis for Occam's razor, on which all our knowledge about the world ultimately rests.
The problem of induction concerns the apparent impossibility of learning anything at all about the world, from experience. In Machine Learning, it manifests itself as the No Free Lunch theorem. Suppose we see a coin tossed 99 times and it lands heads every time. What can we say about the probability that the next toss will be heads instead of tails? If we adopt the maximum entropy prior over the 2100 possible coin toss results, where each of the 2100 results is equally likely, and then apply Bayes' theorem to our observations, we can say nothing. The probability of heads or tails on the 100th toss is still 50/50.
In order to make inferences about the unknown, from the known, we need to start by assuming that not all of the 2100 possible results are a priori equally likely. We start off believing, based on no evidence, that some possibilities are more likely than others. For example, we might typically assume that the coin has some constant unknown bias, p. But this might be too strong an assumption. What if this coin obeyed a rule where each toss was 99.9% likely to be the same as the immediately preceding toss? That seems a priori possible, and consistent with the observations, but would be ruled out by the constant unknown bias model. In general, the best way of describing the approach we actually take in these situations, is that we pick a prior over the 2100 coin tosses which is consistent with Occam's razor (and you might try to formalize this using Solomonoff induction). We assume that the world follows rules, and that simpler rules are a priori more likely to be true than more complex rules. Under this assumption, we can be justified in stating that the 100th coin toss is very likely to be heads.
Occam's razor seems to be necessary in order to learn anything at all about the world from experience, but it remains an assumption. It is something we must take for granted. It is extremely tempting to try to justify Occam's razor based on our past experience of the world, but that would be circular. We would be using induction to justify induction.
It is troubling to discover that all of our knowledge potentially rests on an unjustified assumption. It would certainly be convenient for us if the Occam's razor principle were valid, but is there any other reason for believing in it? Or are we engaging in motivated reasoning here? How could we go about trying to defend ourselves against this charge? I think the best defence would actually look very similar to the defence of dogmatic priors given above:
- We want to have external reasons for our decisions, so we need learning to be possible.
- Learning is only possible if the Occam's razor principle is true (I'm sure you could write many papers on whether this claim is true or not, or perhaps how to re-phrase it slightly so that it becomes true, but I'm just going to take it for granted).
- We should therefore adopt priors consistent with Occam's razor.
- If Occam's-razor -consistent priors are a good model for the world, we make good decisions, and if they're not, we were always doomed anyway.
Hopefully the analogy with dogmatism is clear.
Conclusion
I'd be very interested to read a more in-depth discussion of whether the so-called dogmatic approach is a reasonable response to the problem of recklessness or not. To me, it seems like the best candidate for resolving some of the thorny issues associated with Pascal mugger type problems.
On the face of it, dogmatism looks like it might involve irrationally extreme confidence that the world happens to be arranged in a certain way. But I think there could actually be a close analogy between adopting dogmatic priors, and adopting priors consistent with Occam's razor. Everyone happily does that already without fretting too much.
One problem with the dogmatic solution against infinities and (sub)distributions with infinite EV is that it means no finite amount of information could allow you to assign a nonzero probability to the possibilities.
Suppose you wake up in what appears to be Biblical Heaven (or Hell), with many of the details basically right. Years pass, and then decades, centuries and milennia. You'd still have to assign 0 probability to it lasting forever. Nothing could convince you to do otherwise.
Or, you watch us colonize space. All the evidence against a neverending (in time or space) universe was badly mistaken (e.g. the universe was never actually expanding, so heat death seems unlikely), and we make it past the edge of the currently observable universe (relative to here) and go further than any specific finite model scientists propose and have any specific reason to believe in should allow. In that kind of situation, you'd still have to be absolutely certain it (the generation of value, or the counterfactual gain in value) will all end eventually and assign 0 probability to heavy tails.
FWIW, unless the mugger is offering you infinite EV or gives you very persuasive evidence of their powers, there might be higher EV things you could pursue.
Personally if I had these experiences I would think it's much more likely that I have gone insane and that these experiences are hallucinations.
I would give substantial weight to that possibility in the case of Heaven, but I wouldn't absolutely rule out infinities 100%. I don't think the space colonization scenario should really make you doubt your sanity much, though, and especially if the evidence we had so far never supported an end to the universe (rather than just undermining the evidence later). We can also posit a future human or agent instead and consider their POV. The most unusual experience in the space colonization scenario is the space colonization itself, not the revisions to our understanding of physics which wouldn't really affect every day life, anyway.
The space colonization scenario doesn't offer the possibility of infinite utility though? Just very large but finite utility?
Regardless of whether or not you think the universe is infinite, my understanding is that the part of the universe that is reachable without faster-than-light travel is still finite.
I tried to set up the space colonization scenario so that the reachable universe (from now) grows without bound over time, but could have been clearer.
Even if it is bounded, if our descendants go on forever (or have an end date that's finite but has infinite expected value), that would still generate infinite expected utility.
I think that's a very persuasive way to make the case against assigning 0 probability to infinities. I think I've got maybe three things I'd say in response which could address the problem you've raised:
I'd be interested in hearing you elaborate more on your final sentence (or if you've got any links you could point me to which elaborate on it).
(Also, I don't think heat death depends on an expanding universe. It's an endgame for existence that would require a change to the 2nd law of thermodynamics in order to escape, in my understanding.)
I think the original Pascal's wager addresses your first point (assuming you think a higher probability of infinity is infinitely better than a lower but still nonzero probability of infinity). Also, if our descendents will create infinite (or undefined) value in expectation, then extinction risk reduction plausibly affects the probability here. There's also acausal influence over infinitely many agents in an infinite universe, but that might be more tractable to treat as finite EVs, after some kind of averaging.
On your 2nd and 3rd points, I think we should aim for justified beliefs, and 0 to infinity and infinite EV subdistributions doesn't seem justifiable to me. Is it reasonable for someone to have an arbitrarily strong prior that the average number of heads per coin flip (for a specific coin, flipped a million times) is 0.9 with no particular reason favouring heads over tails (or only motivated reasons), and only barely adjusting their beliefs as the average comes out to about 0.5? I guess, from the outside, with "more reasonable" beliefs, this person is predictably losing by their own lights. The same seems true of those assigning 0 credence to infinities, although it's much harder (maybe impossible?) to get good enough feedback to show they're losing, and one-shots seem pretty different from events that are basically repeated many times.
On my last sentence from my first comment, see my point 5 here:
https://forum.effectivealtruism.org/posts/qcqTJEfhsCDAxXzNf/what-reason-is-there-not-to-accept-pascal-s-wager?commentId=Ydbz56hhEwxg9aPh8
Also, see this post and discussion:
https://forum.effectivealtruism.org/posts/sEnkD8sHP6pZztFc2/fanatical-eas-should-support-very-weird-projects
And https://reducing-suffering.org/lab-universes-creating-infinite-suffering/
I guess I am calling into question the use of subjective probabilities to quantify beliefs.
I think subjective probabilities make sense in the context of decisions, to describe your decision-making behaviour (see e.g. Savage's derivation of probabilities from certain properties of decision-making he thinks we should abide by). But if you take the decisions out of picture, and try to talk about 'beliefs' in abstract, and try to get me to assign a real number between 0 and 1 to them, I think I am entitled to ask "why would I want to do something like that?" Especially if it's going to lead me into strange conclusions like "you should give your wallet to a Pascal mugger".
I think rejecting the whole business of assigning subjective probabilities to things is a very good way to reply to the Pascal's mugger problem in general. Its big weakness is that there are several strong arguments you can make that tell you you should quanity subjective uncertainty with mathematical probabilities, of which I think Savage's is the strongest. But the actual fundamental interpretation of subjective probabilities in Savage's argument is that they describe how you will act in the context of decisions, not that they quantify some more abstract notion like a degree of belief (he deliberately avoids talking in those terms). For example, P(A)>P(B) fundamentally means that you will choose to receive a prize in the event that A occurs, rather than in the event that B occurs, if forced to choose between the two.
If that's what subjective probabilities fundamentally mean, then it doesn't seem necessarily absurd to assign zero probability to something that is concievable. It at least doesn't violate any of Savage's axioms. It seems to violate our intuition of how quantification of credences should behave, but I think I can reply to that by resorting to the "why would I want to do something like that?" argument. Quantifying credences is not actually what I'm doing. I'm trying to make decisions.
Thanks for the links! I'm still not sure I quite understand point 5. Is the idea that instead of giving my wallet to the mugger, I should donate it to somewhere else that I think has a more plausible sounding way of achieving infinite utility? I suppose that might be true, but doesn't really seem to solve the Pascal mugger problem to me, just reframe it a bit.
If you aren't (ever) using subjective probabilities to guide decisions, then what would you use instead and why? If you're sometimes using subjective probabilities, how do you decide when to and when not to, and why?
Unbounded utilities do violate Savage's axioms, though, I think because of St. Petersburg-like lotteries. Savage's axioms, because of completeness (you have to consider all functions from states to outcomes, so all lotteries), force your utility function and probability function to act in certain ways even over lotteries you would assign 0 probability to ever countering. But you can drop the completeness axiom and assume away St. Petersburg-like lotteries, too. See also Toulet's An Axiomatic Model of Unbounded Utility Functions (which I only just found and haven't read).
I am comfortable using subjective probabilities to guide decisions, in the sense that I am happy with trying to assign to every possible event a real number between 0 and 1, which will describe how I will act when faced with gambles (I will maximize expected utility, if those numbers are interpreted as probabilities).
But the meaning of these numbers is that they describe my decision-making behaviour, not that they quantify a degree of belief. I am rejecting the use of subjective probabilities in that context, if it is removed from the context of decisions. I am rejecting the whole concept of a 'degree of belief', or of event A being 'more likely' than event B. Or at least, I am saying there is no meaning in those statements that goes beyond the meaning: 'I will choose to receive a prize if event A happens, rather than if event B happens, if forced to choose'.
And if that's all that probabilities mean, then it doesn't seem necessarily wrong to assign probability zero to something that is concievable. I am simply describing how I will make decisions. In the Pascal mugger context: I would choose the chance of a prize in the event that I flip 1000 heads in a row on a fair coin, over the chance of a prize in the event that the mugger is correct.
That's still a potentially counter-intuitive conclusion to end up at, but it's a bullet I'm comfortable biting. And I feel much happier doing this than I do if you define subjective probabilities in terms of degrees of belief. I believe this language obscures what subjective probabilities fundamentally are, and this, previously, made me needlessly worried that by assigning extreme probabilities, I was making some kind of grave epistemological error. In fact I'm just describing my decisions.
Savage's axioms don't rule out unbounded expected utility, I don't think. This is from Savage's book, 'Foundations of Statistics', Chapter 5, 'Utility', 'The extension of utility to more general acts':
"If the utility of consequences is unbounded, say from above, then, even in the presence of P1-7, acts (though not gambles) of infinite utility can easily be constructed. My personal feeling is that, theological questions aside, there are no acts of infinite or minus infinite utility, and that one might reasonable so postulate, which would amount to assuming utility to be bounded."
The distinction between 'acts' and 'gambles' is I think just that gambles are acts with a finite number of possible consequences (which obviously stops you constructing infinite expected value), but the postulates themselves don't rule out infinite utility acts.
I'm obviously disagreeing with the Savage's final remark in this post. I'm saying that you could also shift the 'no acts of infinite or minus infinite utility' constraint away from the utility function, and onto the probabilities themselves.
I think this doesn't really answer my question or is circular. I don't think that you decide how to act based on probabilities that come from how you decide to act, but that seems to be what you're saying if I interpet your response as an answer to my question. It might also justify any course of action, possibly even if you fix the utility function (I think the subjective probabilities would need to depend on things in very weird ways, though). I think you still want to be able to justify specific acts, and I want to know how you'll do this.
Maybe we can make this more explicit with an example. How do you decide which causes to prioritize? Or, pick an intervention, and how would you decide whether it is net positive or net negative? And do so without assigning probabilities as degrees of belief. How else are you going to come up with those probabilities? Or are you giving up probabilities as part of your procedure?
On Savage’s axioms, if your state space is infinite and your utility function is unbounded, then completeness requires the axioms to hold over acts that would have infinite expected utility, even if none is ever accessible to you in practice, and I think that would violate other axioms (the sure thing principle; if not Savage’s version, one that would be similarly irrational to violate; see https://onlinelibrary.wiley.com/doi/full/10.1111/phpr.12704 ). If your state space is finite and no outcome has infinite actual utility, then that seems to work, but I'm not sure you'd want to commit to a finite state space.
I still don't think the position I'm trying to defend is circular. I'll have a go at explaining why.
I'll start with aswering your question: in practice the way I would come up with probabilities to assess a charitable intervention is the same as the way you probably would. I'd look at the available evidence and update my priors in a way that at least tries to approximate the principle expressed in Bayes' theorem. Savage's axioms imply that my decision-describing-numbers between 0 and 1 have to obey the usual laws of probability theory, and that includes Bayes' theorem. If there is any difference between our positions, it will only be in how we should pick our priors. You pick those before you look at any evidence at all. How should you do that?
Savage's axioms don't tell you how to pick your priors. But actually I don't know of any other principle that does either. If you're trying to quantify 'degrees of belief' in an abstract sense, I think you're sort of doomed (this is the problem of induction). My question for you is, how do you do that?
But we do have to make decisions. I want my decisions to be constrained by certain rational sounding axioms (like the sure thing principle), but I don't think I want to place many more constraints on myself than that. Even those fairly weak constraints turn out to imply that there are some numbers, which you can call subjective probabilities, that I need to start out with as priors over states of the world, and which I will then update in the usual Bayesian way. But there is very little constraint in how I pick those numbers. They have to obey the laws of probability theory, but that's quite a weak constraint. It doesn't by itself imply that I have to assign non-zero probability to things which are concievable (e.g. if you pick a real number at random from the uniform distribution between 0 and 1, every possible outcome has probability 0).
So this is the way I'm thinking about the whole problem of forming beliefs and making decisions. I'm asking the question:
" I want to make decisions in a way that is consistent with certain rational seeming properties, what does that mean I must do, and what, if anything, is left unconstrained?"
I think I must make decisions in a Bayesian-expected-utility-maximising sort of way, but I don't think that I have to assign a non-zero probability to every concievable event. In fact, if I make one of my desired properties be that I'm not susceptible to infinity threatening Pascal muggers, then I shouldn't assign non-zero probability to situations that would allow me to influence infinite utility.
I don't think there is anything circular here.
Ok, this makes more sense to me.
FWIW, I think most of us go with our guts to assign probabilities most of the time, rather than formally picking priors, likelihoods and updating based on evidence. I tend to use ranges of probabilities and do sensitivity analysis instead of committing to precise probabilities, because precise probabilities also seem epistemically unjustified to me. I use reference classes sometimes.