literally, you pick one precise credence over many others for no reason
At some foundational level, a credence has no deeper reason than 'some neurons fired that way'. But you don't need to restrict yourself to concerns about the whole future lightcone to run into this problem - at the foundational level this is true of every statement.
There are various ways one might respond to this challenge, but if we don't view it as insurmountable elsewhere, I don't see why we should do so with credences (which are of course usually non-foundational statements). And if we do, it undermines e.g. any argument about unawareness.
Why does "our best effort" need to be precise?
It only needs to be as precise as is necessary for decision-making. I will probably never need to forecast rain to 8 decimal places. But if you're saying forecasting rain as 'less than .5' is ok, but that forecasting 0.1234567% chance of rain if the extra precision was actually decision-relevant, would be un-ok/impossible/qualitatively different, then I disagree.
(from link) It doesn’t follow from “we don’t know the net direction of the consequences we’re unaware of” that we should regard the positives and negatives as precisely symmetric. One reason symmetry is implausible: If we become aware of a new possible consequence, this should update our beliefs about the others we’re unaware of, breaking the symmetry.
If you 'become aware' of something, you've gained information and should update your priors accordingly. That doesn't move me away from being happy to treat genuine unknowns as EV-0. Your counterpoint seems to be that in some cases that feel sort-of- equal (and about which, in the cases you describe we actually have a lot of information), we might be inclined to give equal credence. But it seems to me correct to say 'if you have meaningful knowledge of two possible outcomes, and the weight you assign to them is decision-relevant, giving them equal credence is a mistake', which fixes this purported problem without radically undermining our epistemology.
My claim is that we don't have a positive argument in favor of doing what the precise EV approach recommends
The precise EV approach is well evidenced in short-term decision-making, so the positive argument is that there isn't any principled difference between short and long-term decision-making
I don't know how to challenge the premises because the key premise seems to be an assertion that I don't find convincing.
In 'Should you go with your best guess?', which appears to be the primary argument against the idea of 'deterministic' Bayesian credences, DiGiovanni repeatedly signposts that he's going to give an argument against them... but I can't see anything that constitutes one.
In the section 'Background on degrees of belief and what makes them rational', he talks about how we don't get to find out which beliefs outperform others, but doesn't say why this means we shouldn't/can't pick credences according to our best effort. It also doesn't say why, if we can measure short term value, we shouldn't use that as a justification for our decisionmaking process and assume EV from events that we don't think we can assess is 0.
In the section 'Motivating example', he gives an example vignette, at the end of which we're given that 'this feels so arbitrary'. But it doesn't seem like 'feeling arbitrary' is a reason not to do something - especially when we're not given an alternative (or at least, no other decision process that seems less or equally arbitrary).
So my response is just to say 'using credences still seems fine, if occasionally emotionally uncomfortable (maybe using distributions is sometimes empirically better, and if so I support it)' - in which case I don't see a problem in need of solving.
Makes sense that range of threats should be wider (arbitrarily wide I guess, but a function of what scenarios you consider and differentiate between). I don't see why error estimates should be thin though - there are certainly people guessing close to 100% for some risks and various mechanisms that we might not even have considered that we would consider high risk if we knew more about them, which lead to us underestimating the risks of innocuous actions by a huge amount (c.f. the recent upsurge in concern about mirror bacteria)
I think 1) implies that you should give up some substantial optimisation for the sake of greater versatility (which seems approx titotal's view with reference to overcommitting)
2) feels correct and important to me, also since I've been arguing in the post op linked and elsewhere that treating extinction as special is a heuristic that was useful for initial cause prioritisation but isn't a valid reason for focusing on it two decades later.
I think there are many strong reasons to be sceptical of a lot of EA orthodoxy - many people including me have written about them at length (also keep an eye out for a post by titotal in the next few days that comes up with a magnitude estimate for such errors).
These reasons might turn out to be wrong, but I strongly disagree that e.g. classic longtermist/x-risk/AI doomer arguments are anywhere near the standard of economics - as evidenced e.g. by economists vigorouslydisagreeing with them.
I do my best to reply to everyone who reaches out about careers at CG, even if sometimes this is just a quick note celebrating their pursuit of impactful work and sharing career resources with them. At the time of writing, this has worked out to roughly 1,200 messages and 600 calls/in-person meetings.
is really inspiring. It sounds exhausting, but as someone who's often struggled to get feedback from decisionmakers in the EA space, I'm sure those 1800 people very much appreciated it - and I'd be amazed if it didn't lead to some substantially positive counterfactual changes in their plans :)
(and maybe I'll reach out to you for some feedback soon!)
A relatively simple way of making the repugnant conclusion more intuitive to me is to recognise that individual selves are largely an illusion, i.e. that empty individualism (or, its better marketed/more spiritual sounding but functional equivalent open individualism) is correct.
Suppose you've set up your parameters such that pinpricks actually involve negative utility - because (trigger warning of slightly graphical image for the second of these links) in manycases it obviously isn't actually negative, which muddies our intuitions. Then for empty individualism a tiny amount of torture either is actually closely analogous to a large number of pinpricks (that just happen to be locally clustered). The OI equivalent is that a small amount of torture is analogous to a pinprick on the cosmic entity
Even in the case of humans, we can imagine how pinpricks could add up. A single superficial pinprick isn't that bad - and the difference between a 1mm and 2mm insertion would be very slight. But if you insert hundreds or thousands of pins 10+mm deep, gradually you move towards an experience that seems as bad as any other torture.
To put it simply, sufficiently many pinpricks, of sufficient depth to be unpleasant, are torture. And they can be torture at a degree of virtually any level of pain a human is capable of experiencing - so I don't see a need to introduce dramatic discontinuities in our moral evaluation to explain why other tortures are somehow still morally worse.
That was fascinating - I really like the idea of reframing EA ideas as a way of saving on future research costs.
I'm a bit unclear what we're currently to make of the the '3–5x' estimate - you say it's illustrative, but also 'plausible'. Assuming that is your current best guess, could you say how you reached it?
I don't encounter many people who still identify as longtermist, but as someone who does, I recently wrote thesearguments for why longtermists should be less extinction-focused.
The tl;dr is that I think that other than extinction there are predictable patterns, with perhaps the most prominent related to entropy, and that those patterns provide more nuanced ways to estimate the cost of lesser catastrophes - and that while assessing the costs of lesser catastrophes precisely is infeasible, that's not a basis for thinking they would be negligible compared to extinction.
At some foundational level, a credence has no deeper reason than 'some neurons fired that way'. But you don't need to restrict yourself to concerns about the whole future lightcone to run into this problem - at the foundational level this is true of every statement.
There are various ways one might respond to this challenge, but if we don't view it as insurmountable elsewhere, I don't see why we should do so with credences (which are of course usually non-foundational statements). And if we do, it undermines e.g. any argument about unawareness.
It only needs to be as precise as is necessary for decision-making. I will probably never need to forecast rain to 8 decimal places. But if you're saying forecasting rain as 'less than .5' is ok, but that forecasting 0.1234567% chance of rain if the extra precision was actually decision-relevant, would be un-ok/impossible/qualitatively different, then I disagree.
If you 'become aware' of something, you've gained information and should update your priors accordingly. That doesn't move me away from being happy to treat genuine unknowns as EV-0. Your counterpoint seems to be that in some cases that feel sort-of- equal (and about which, in the cases you describe we actually have a lot of information), we might be inclined to give equal credence. But it seems to me correct to say 'if you have meaningful knowledge of two possible outcomes, and the weight you assign to them is decision-relevant, giving them equal credence is a mistake', which fixes this purported problem without radically undermining our epistemology.
The precise EV approach is well evidenced in short-term decision-making, so the positive argument is that there isn't any principled difference between short and long-term decision-making
I don't know how to challenge the premises because the key premise seems to be an assertion that I don't find convincing.
In 'Should you go with your best guess?', which appears to be the primary argument against the idea of 'deterministic' Bayesian credences, DiGiovanni repeatedly signposts that he's going to give an argument against them... but I can't see anything that constitutes one.
In the section 'Background on degrees of belief and what makes them rational', he talks about how we don't get to find out which beliefs outperform others, but doesn't say why this means we shouldn't/can't pick credences according to our best effort. It also doesn't say why, if we can measure short term value, we shouldn't use that as a justification for our decisionmaking process and assume EV from events that we don't think we can assess is 0.
In the section 'Motivating example', he gives an example vignette, at the end of which we're given that 'this feels so arbitrary'. But it doesn't seem like 'feeling arbitrary' is a reason not to do something - especially when we're not given an alternative (or at least, no other decision process that seems less or equally arbitrary).
So my response is just to say 'using credences still seems fine, if occasionally emotionally uncomfortable (maybe using distributions is sometimes empirically better, and if so I support it)' - in which case I don't see a problem in need of solving.
Makes sense that range of threats should be wider (arbitrarily wide I guess, but a function of what scenarios you consider and differentiate between). I don't see why error estimates should be thin though - there are certainly people guessing close to 100% for some risks and various mechanisms that we might not even have considered that we would consider high risk if we knew more about them, which lead to us underestimating the risks of innocuous actions by a huge amount (c.f. the recent upsurge in concern about mirror bacteria)
I think 1) implies that you should give up some substantial optimisation for the sake of greater versatility (which seems approx titotal's view with reference to overcommitting)
2) feels correct and important to me, also since I've been arguing in the post op linked and elsewhere that treating extinction as special is a heuristic that was useful for initial cause prioritisation but isn't a valid reason for focusing on it two decades later.
I think there are many strong reasons to be sceptical of a lot of EA orthodoxy - many people including me have written about them at length (also keep an eye out for a post by titotal in the next few days that comes up with a magnitude estimate for such errors).
These reasons might turn out to be wrong, but I strongly disagree that e.g. classic longtermist/x-risk/AI doomer arguments are anywhere near the standard of economics - as evidenced e.g. by economists vigorously disagreeing with them.
I just want to say this
is really inspiring. It sounds exhausting, but as someone who's often struggled to get feedback from decisionmakers in the EA space, I'm sure those 1800 people very much appreciated it - and I'd be amazed if it didn't lead to some substantially positive counterfactual changes in their plans :)
(and maybe I'll reach out to you for some feedback soon!)
A relatively simple way of making the repugnant conclusion more intuitive to me is to recognise that individual selves are largely an illusion, i.e. that empty individualism (or, its better marketed/more spiritual sounding but functional equivalent open individualism) is correct.
Suppose you've set up your parameters such that pinpricks actually involve negative utility - because (trigger warning of slightly graphical image for the second of these links) in many cases it obviously isn't actually negative, which muddies our intuitions. Then for empty individualism a tiny amount of torture either is actually closely analogous to a large number of pinpricks (that just happen to be locally clustered). The OI equivalent is that a small amount of torture is analogous to a pinprick on the cosmic entity
Even in the case of humans, we can imagine how pinpricks could add up. A single superficial pinprick isn't that bad - and the difference between a 1mm and 2mm insertion would be very slight. But if you insert hundreds or thousands of pins 10+mm deep, gradually you move towards an experience that seems as bad as any other torture.
To put it simply, sufficiently many pinpricks, of sufficient depth to be unpleasant, are torture. And they can be torture at a degree of virtually any level of pain a human is capable of experiencing - so I don't see a need to introduce dramatic discontinuities in our moral evaluation to explain why other tortures are somehow still morally worse.
That was fascinating - I really like the idea of reframing EA ideas as a way of saving on future research costs.
I'm a bit unclear what we're currently to make of the the '3–5x' estimate - you say it's illustrative, but also 'plausible'. Assuming that is your current best guess, could you say how you reached it?
Have you had any takers (who you can publicly name) other than Greg?
I don't encounter many people who still identify as longtermist, but as someone who does, I recently wrote these arguments for why longtermists should be less extinction-focused.
The tl;dr is that I think that other than extinction there are predictable patterns, with perhaps the most prominent related to entropy, and that those patterns provide more nuanced ways to estimate the cost of lesser catastrophes - and that while assessing the costs of lesser catastrophes precisely is infeasible, that's not a basis for thinking they would be negligible compared to extinction.