Bostrom is generally vague and quasi-inconsistent in his probability-related wordings. The original paper is here. The statement in the abstract is 'agents having any of a wide range of final goals will pursue similar intermediary goals', so you could read 'will' as 'I think this is basically guaranteed', or 'this is what the hypothesis in which I have an unspecified credence predicts'; or you could read 'will' as a sort of IPCC-ish statement meaning anything between '>50%' and '100%'.
Later on in the same paper he modifies the language to 'according to what we may term the “instrumental convergence” thesis, there are some instrumental goals likely to be pursued by almost any intelligent agent', which sounds more like the latter interpretation of 'will' (it could end up meaning a <50% min-threshold if you similarly interpret 'almost any' as something like '90%' and multiply that by a likeliness of (50+ϵ)%).
Finally in the conclusion his language shifts to 'we cannot blithely assume' and 'It might be possible to set up a situation', which reads to me like a statement of, say, >1%, or even a lower minimum.
I've been chatting to Vasco via email about this. I guess I'll import some of my critiques here.
I feel like Brian's right that consciousness has some fuzzy edges (maybe it's a combination of many hard-to-recognise-in-isolation subphenomena), but his actual takes on this and most other philosophical subjects seem to me to reduce to an equivocation between nihilism and extreme egoism - they always seem to boil down to 'what "I" (the author) "care" about' as some kind of acceptance test. In the case of consciousness, this seems like you could understand it in a couple of ways, neither of which seem very informative to the reader:
Brian defines consciousness as what Brian/perhaps the reader cares about, and we are to assume that Brian's 'caring' is a real phenomenon, not a meaningless string. Then his definition is circular or at least unfinished, since the reader has no access to what it means for Brian to care (and possibly Brian doesn't either), or how one could predict what future and other Brian- and reader-likes will care about. Nonetheless we are to assume that morality will perenially boil down to internecine struggles among different spheres of caring among different individuals
Brian defines consciousness as what Brian/perhaps the reader cares about, and Brian recognises the circularity and implies that there's really nothing resolving it - his strings are ultimately meaningless, produced, for all anyone including he can tell, by an automaton. Then either the reader is an automaton too, and their actions are equally meaningless, or they are not, and have no cause to read the arbitrarily generated output of an automaton
Though to do that I need to digress into an extended caveat that any purported contradiction comes with the caveat that on interpretation two, Brian's words and all words are referentless. At the very least, if we abandon the notion of consciousness, I think we most likely lose our entire vocabulary of non-physically-descriptive nouns, including but not limited to
Contradiction/validity
Reference classes
Prediction
Moral norms
Metaethics
Rational norms
(is metarationality an analogous thing? If not, why not, and does that suggest something about metaethics and normative ethics)
Decision theory
Axiology (‘values’)
Causality
Free will
Meaning
Need
Epistemology
Internal (non revealed) preferences
Beliefs
Desires
Goal-seeking
Personal identity/the self
Analyticity/maths
Syntheticity/science (these last two already unified courtesy of Quine)
Valence (though this maybe has two distinct phenomena)
Ontology
Revealed preferences
Theology
Moral uncertainty
Political science
Truth/falsity
(not to say I would miss all of these) (also heads up so it doesn't catch Vasco off guard, I added a couple of key ones since our email discussion)
And I think Vasco is arguing for interpretation two. So if so I think there's no discussion to be had - we are both just flapping our meat. It might turn out somehow that one of us counterfactually changes their behaviour if in range of the others' meat-flapping, but we can't ascribe that to causality - since that's a non-physically-descriptive noun - and we certainly can't ascribe it to anything like semantic properties of the words.
So I have to proceed by assuming that the first interpretation was correct, or that I somehow misunderstand the nature of the absence of consciousness, which is a strange position to be in. Perhaps Vasco can help me understand a different way of 'thinking about' it, but I am sceptical.
On this first-and-a-halfth interpretation Brian has gone from defining consciousness as 'thing Brian cares about' to 'we need to know what other things have raw feelings'. If the first essay were correct, 'need' wouldn't mean much, and if it did, the answer would be 'whatever things Brian cares about'. So perhaps Brian is saying there he anticipates caring about future things and cares about that anticipation. But learning information about the world couldn't give him information about other consciousnesses, because that information would be entirely contained within the stochastic process of him caring about things or not.
In the latter essay, Brian writes:
Reductionism: The world ultimately reduces to physics—quarks, leptons, etc., or probably something else more fundamental. "Consciousness" is not one of those physical primitives. Thus, consciousness must be an emergent phenomenon—a process that involves many parts working together in various ways. But if consciousness is emergent, then it must have fuzzy boundaries just like any concept does. There's no "true essence" underlying an emergent phenomenon. It just is what its components are, and we decide the boundaries of labeling which emergent phenomena belong to which categories that we make up. Consciousness must be like this.
Which Vasco approves of. I think these claims are far too confident:
Depending on what we mean by a physical primitive, consciousness could be a property of one or more physical primitives. Perhaps panpsychists are correct and it's a property of all of them (I find this view weird, but easier to parse than the entirely dissolutionary view of consciousness)
'if consciousness is emergent, then it must have fuzzy boundaries just like any concept does.' - doesn't seem to follow. It might be a crisp subset that emerges from a specific minimum set of objects given by some as-yet unknown parameter; it might be inherent in a well-defined mathematical property, as the QRI guys think; it might be fuzzy-in-principle but so tightly defined in practice that for almost all practical purposes we can treat it as discrete at the macroscropic level (as e.g. the position of particles)
Ultimately I think that consciousness, despite being a philosophical black box and a neuroscientifically grey one, is too powerful a high level theory to abandon without a much more solid replacement. If you were to put me on a rack for long enough, I would recant the positions I'm arguing for here. But my prior explanation of why I would recant refers to a relatively simple, localisable statement that some -currently but not inevitably hard to specify - measurable quantity had gone negative, and that 'caused' my action.
I won't put words into Vasco's mouth, but I wonder whether he thinks there's a way of describing that process without using 'reference classes' that's simpler than '{full mathematical derivation that} that's the all the interacting fields and forces in that local area'.
I also want to ask Vasco how he can be confident that, even if it's true that he has no consciousness (and it does seem to be a contingent and disablable property in humans), that lack of a light would generalise to those of us who profess to have one.
I wanted to keep it abstract to reduce the risk of priming people, but something like 'doesn't give reason to suppose any particular of the supposedly convergent behaviours are strongly linked in nontrivial ways or likely to dominate in any particular AI system we should expect to encounter in the near future'.
I'm working on an essay arguing that its conclusions basically aren't well-supported in this respect. See also the series Vasco posted in one of the other comments to this poll.
It's great to see some EAs writing about positive(ish) futures :)
I think it's important to factor in the (opportunity) cost of reliable error correction. Here's my BOTEC effort (sorry about the lack of Latex, posted in a hurry):
Estimating the actual cost of reliable error correction on the timescales we're talking about looks pretty involved, so we can parameterise it for now, such that 'cost' is the reduction in probes sent out you would incur for some level of error correction. We can define this as a function of 'degradation', which would be expected rate of change of alignment per probe over time.
Maybe degradation needs to be a function of something else we need to consider, but naively at the moment we could consider it an independent variable such that we're looking for the value of degradation that optimises total benefit. We can normalise expected value as as 1 per fully aligned probe per unit time. Then we have:
ev_production_at_t = ∫ from 0..t probe_launch_rate * alignment(i, t) * probe_coverage_volume(i, t) di (where i is launch times of probes)
probe_launch_rate = 1/cost
alignment(i, t) = f(degradation, t-i) (specifically some exponent or integral)
cost = g(degradation)
degradation = TODO (something like a constant of change, maybe proportional, except that you might want to express alignment being able to go negative under particular cirucmstances)
probe_coverage_volume(i, t) = h((t-i)^3)
total_benefit_at_T = ∫ from 0..T of ev_production_at_t with respect to t
where f(x), g(x), h(x) are just 'some as-yet undefined function of x'
So you can vary the amount you're willing to pay to avoid degradation; with (close enough to) 0 degradation on the timescales we're looking at probably coming at some high cost. So we probably don't want 0 degradation even in an uncontested universe.
But also, at least in the limit, probe_coverage_volume is also going to be bounded by the total number of probes any national, corporate or alien competitors send out, which will be more the slower you send your own out. So even in some idealised world where everyone in our own civilisation is acting altruistically, we'd have some motivation to send out imperfect probes.
In practice I suspect our own world won't be so neat, and we'll have various factions having to choose how much degradation they're willing to accept to avoid being outcompeted.
I just tried it out and had some similar issues that have been mentioned. It flagged my main thesis as 'false', and then cites most of the arguments I'm explicitly arguing against as proof.
One thing that I think would make it feel much better to use is if the UI allowed its commentsto be the start of a conversation. Then I could correct what I think are its errors, and see if once we fix those it can still poke useful holes.
(I have a slight preference for it being more tonally constructive. Atm it reads like I pasted the post into an LLM chat with the prompt 'some idiot wrote this: please explain why it's wrong.' Though I wouldn't want it to lose fidelity/become more sycophantic for the sake of improving tone)
As others said it was handy for picking up confusing turns of phrase - and did end up giving me some useful-looking extra reading to do.
literally, you pick one precise credence over many others for no reason
At some foundational level, a credence has no deeper reason than 'some neurons fired that way'. But you don't need to restrict yourself to concerns about the whole future lightcone to run into this problem - at the foundational level this is true of every statement.
There are various ways one might respond to this challenge, but if we don't view it as insurmountable elsewhere, I don't see why we should do so with credences (which are of course usually non-foundational statements). And if we do, it undermines e.g. any argument about unawareness.
Why does "our best effort" need to be precise?
It only needs to be as precise as is necessary for decision-making. I will probably never need to forecast rain to 8 decimal places. But if you're saying forecasting rain as 'less than .5' is ok, but that forecasting 0.1234567% chance of rain if the extra precision was actually decision-relevant, would be un-ok/impossible/qualitatively different, then I disagree.
(from link) It doesn’t follow from “we don’t know the net direction of the consequences we’re unaware of” that we should regard the positives and negatives as precisely symmetric. One reason symmetry is implausible: If we become aware of a new possible consequence, this should update our beliefs about the others we’re unaware of, breaking the symmetry.
If you 'become aware' of something, you've gained information and should update your priors accordingly. That doesn't move me away from being happy to treat genuine unknowns as EV-0. Your counterpoint seems to be that in some cases that feel sort-of- equal (and about which, in the cases you describe we actually have a lot of information), we might be inclined to give equal credence. But it seems to me correct to say 'if you have meaningful knowledge of two possible outcomes, and the weight you assign to them is decision-relevant, giving them equal credence is a mistake', which fixes this purported problem without radically undermining our epistemology.
My claim is that we don't have a positive argument in favor of doing what the precise EV approach recommends
The precise EV approach is well evidenced in short-term decision-making, so the positive argument is that there isn't any principled difference between short and long-term decision-making
I don't know how to challenge the premises because the key premise seems to be an assertion that I don't find convincing.
In 'Should you go with your best guess?', which appears to be the primary argument against the idea of 'deterministic' Bayesian credences, DiGiovanni repeatedly signposts that he's going to give an argument against them... but I can't see anything that constitutes one.
In the section 'Background on degrees of belief and what makes them rational', he talks about how we don't get to find out which beliefs outperform others, but doesn't say why this means we shouldn't/can't pick credences according to our best effort. It also doesn't say why, if we can measure short term value, we shouldn't use that as a justification for our decisionmaking process and assume EV from events that we don't think we can assess is 0.
In the section 'Motivating example', he gives an example vignette, at the end of which we're given that 'this feels so arbitrary'. But it doesn't seem like 'feeling arbitrary' is a reason not to do something - especially when we're not given an alternative (or at least, no other decision process that seems less or equally arbitrary).
So my response is just to say 'using credences still seems fine, if occasionally emotionally uncomfortable (maybe using distributions is sometimes empirically better, and if so I support it)' - in which case I don't see a problem in need of solving.
Makes sense that range of threats should be wider (arbitrarily wide I guess, but a function of what scenarios you consider and differentiate between). I don't see why error estimates should be thin though - there are certainly people guessing close to 100% for some risks and various mechanisms that we might not even have considered that we would consider high risk if we knew more about them, which lead to us underestimating the risks of innocuous actions by a huge amount (c.f. the recent upsurge in concern about mirror bacteria)
I think 1) implies that you should give up some substantial optimisation for the sake of greater versatility (which seems approx titotal's view with reference to overcommitting)
2) feels correct and important to me, also since I've been arguing in the post op linked and elsewhere that treating extinction as special is a heuristic that was useful for initial cause prioritisation but isn't a valid reason for focusing on it two decades later.
Bostrom is generally vague and quasi-inconsistent in his probability-related wordings. The original paper is here. The statement in the abstract is 'agents having any of a wide range of final goals will pursue similar intermediary goals', so you could read 'will' as 'I think this is basically guaranteed', or 'this is what the hypothesis in which I have an unspecified credence predicts'; or you could read 'will' as a sort of IPCC-ish statement meaning anything between '>50%' and '100%'.
Later on in the same paper he modifies the language to 'according to what we may term the “instrumental convergence” thesis, there are some instrumental goals likely to be pursued by almost any intelligent agent', which sounds more like the latter interpretation of 'will' (it could end up meaning a <50% min-threshold if you similarly interpret 'almost any' as something like '90%' and multiply that by a likeliness of (50+ϵ)%).
Finally in the conclusion his language shifts to 'we cannot blithely assume' and 'It might be possible to set up a situation', which reads to me like a statement of, say, >1%, or even a lower minimum.
Wow, ty :)
I've been chatting to Vasco via email about this. I guess I'll import some of my critiques here.
I feel like Brian's right that consciousness has some fuzzy edges (maybe it's a combination of many hard-to-recognise-in-isolation subphenomena), but his actual takes on this and most other philosophical subjects seem to me to reduce to an equivocation between nihilism and extreme egoism - they always seem to boil down to 'what "I" (the author) "care" about' as some kind of acceptance test. In the case of consciousness, this seems like you could understand it in a couple of ways, neither of which seem very informative to the reader:
Brian subsequently wrote the essay My Confusions about the Hard Problem of Consciousness, which I think (though Vasco disagrees) basically contradicts this one.
Though to do that I need to digress into an extended caveat that any purported contradiction comes with the caveat that on interpretation two, Brian's words and all words are referentless. At the very least, if we abandon the notion of consciousness, I think we most likely lose our entire vocabulary of non-physically-descriptive nouns, including but not limited to
(not to say I would miss all of these) (also heads up so it doesn't catch Vasco off guard, I added a couple of key ones since our email discussion)
And I think Vasco is arguing for interpretation two. So if so I think there's no discussion to be had - we are both just flapping our meat. It might turn out somehow that one of us counterfactually changes their behaviour if in range of the others' meat-flapping, but we can't ascribe that to causality - since that's a non-physically-descriptive noun - and we certainly can't ascribe it to anything like semantic properties of the words.
So I have to proceed by assuming that the first interpretation was correct, or that I somehow misunderstand the nature of the absence of consciousness, which is a strange position to be in. Perhaps Vasco can help me understand a different way of 'thinking about' it, but I am sceptical.
On this first-and-a-halfth interpretation Brian has gone from defining consciousness as 'thing Brian cares about' to 'we need to know what other things have raw feelings'. If the first essay were correct, 'need' wouldn't mean much, and if it did, the answer would be 'whatever things Brian cares about'. So perhaps Brian is saying there he anticipates caring about future things and cares about that anticipation. But learning information about the world couldn't give him information about other consciousnesses, because that information would be entirely contained within the stochastic process of him caring about things or not.
In the latter essay, Brian writes:
Which Vasco approves of. I think these claims are far too confident:
Ultimately I think that consciousness, despite being a philosophical black box and a neuroscientifically grey one, is too powerful a high level theory to abandon without a much more solid replacement. If you were to put me on a rack for long enough, I would recant the positions I'm arguing for here. But my prior explanation of why I would recant refers to a relatively simple, localisable statement that some -currently but not inevitably hard to specify - measurable quantity had gone negative, and that 'caused' my action.
I won't put words into Vasco's mouth, but I wonder whether he thinks there's a way of describing that process without using 'reference classes' that's simpler than '{full mathematical derivation that} that's the all the interacting fields and forces in that local area'.
I also want to ask Vasco how he can be confident that, even if it's true that he has no consciousness (and it does seem to be a contingent and disablable property in humans), that lack of a light would generalise to those of us who profess to have one.
I wanted to keep it abstract to reduce the risk of priming people, but something like 'doesn't give reason to suppose any particular of the supposedly convergent behaviours are strongly linked in nontrivial ways or likely to dominate in any particular AI system we should expect to encounter in the near future'.
I'm working on an essay arguing that its conclusions basically aren't well-supported in this respect. See also the series Vasco posted in one of the other comments to this poll.
It's great to see some EAs writing about positive(ish) futures :)
I think it's important to factor in the (opportunity) cost of reliable error correction. Here's my BOTEC effort (sorry about the lack of Latex, posted in a hurry):
Estimating the actual cost of reliable error correction on the timescales we're talking about looks pretty involved, so we can parameterise it for now, such that 'cost' is the reduction in probes sent out you would incur for some level of error correction. We can define this as a function of 'degradation', which would be expected rate of change of alignment per probe over time.
Maybe degradation needs to be a function of something else we need to consider, but naively at the moment we could consider it an independent variable such that we're looking for the value of degradation that optimises total benefit. We can normalise expected value as as 1 per fully aligned probe per unit time. Then we have:
ev_production_at_t = ∫ from 0..t probe_launch_rate * alignment(i, t) * probe_coverage_volume(i, t) di (where i is launch times of probes)
probe_launch_rate = 1/cost
alignment(i, t) = f(degradation, t-i)
(specifically some exponent or integral)
cost = g(degradation)
degradation = TODO
(something like a constant of change, maybe proportional, except that you might want to express alignment being able to go negative under particular cirucmstances)
probe_coverage_volume(i, t) = h((t-i)^3)
total_benefit_at_T = ∫ from 0..T of ev_production_at_t with respect to t
where f(x), g(x), h(x) are just 'some as-yet undefined function of x'
So you can vary the amount you're willing to pay to avoid degradation; with (close enough to) 0 degradation on the timescales we're looking at probably coming at some high cost. So we probably don't want 0 degradation even in an uncontested universe.
But also, at least in the limit, probe_coverage_volume is also going to be bounded by the total number of probes any national, corporate or alien competitors send out, which will be more the slower you send your own out. So even in some idealised world where everyone in our own civilisation is acting altruistically, we'd have some motivation to send out imperfect probes.
In practice I suspect our own world won't be so neat, and we'll have various factions having to choose how much degradation they're willing to accept to avoid being outcompeted.
I just tried it out and had some similar issues that have been mentioned. It flagged my main thesis as 'false', and then cites most of the arguments I'm explicitly arguing against as proof.
One thing that I think would make it feel much better to use is if the UI allowed its commentsto be the start of a conversation. Then I could correct what I think are its errors, and see if once we fix those it can still poke useful holes.
(I have a slight preference for it being more tonally constructive. Atm it reads like I pasted the post into an LLM chat with the prompt 'some idiot wrote this: please explain why it's wrong.' Though I wouldn't want it to lose fidelity/become more sycophantic for the sake of improving tone)
As others said it was handy for picking up confusing turns of phrase - and did end up giving me some useful-looking extra reading to do.
At some foundational level, a credence has no deeper reason than 'some neurons fired that way'. But you don't need to restrict yourself to concerns about the whole future lightcone to run into this problem - at the foundational level this is true of every statement.
There are various ways one might respond to this challenge, but if we don't view it as insurmountable elsewhere, I don't see why we should do so with credences (which are of course usually non-foundational statements). And if we do, it undermines e.g. any argument about unawareness.
It only needs to be as precise as is necessary for decision-making. I will probably never need to forecast rain to 8 decimal places. But if you're saying forecasting rain as 'less than .5' is ok, but that forecasting 0.1234567% chance of rain if the extra precision was actually decision-relevant, would be un-ok/impossible/qualitatively different, then I disagree.
If you 'become aware' of something, you've gained information and should update your priors accordingly. That doesn't move me away from being happy to treat genuine unknowns as EV-0. Your counterpoint seems to be that in some cases that feel sort-of- equal (and about which, in the cases you describe we actually have a lot of information), we might be inclined to give equal credence. But it seems to me correct to say 'if you have meaningful knowledge of two possible outcomes, and the weight you assign to them is decision-relevant, giving them equal credence is a mistake', which fixes this purported problem without radically undermining our epistemology.
The precise EV approach is well evidenced in short-term decision-making, so the positive argument is that there isn't any principled difference between short and long-term decision-making
I don't know how to challenge the premises because the key premise seems to be an assertion that I don't find convincing.
In 'Should you go with your best guess?', which appears to be the primary argument against the idea of 'deterministic' Bayesian credences, DiGiovanni repeatedly signposts that he's going to give an argument against them... but I can't see anything that constitutes one.
In the section 'Background on degrees of belief and what makes them rational', he talks about how we don't get to find out which beliefs outperform others, but doesn't say why this means we shouldn't/can't pick credences according to our best effort. It also doesn't say why, if we can measure short term value, we shouldn't use that as a justification for our decisionmaking process and assume EV from events that we don't think we can assess is 0.
In the section 'Motivating example', he gives an example vignette, at the end of which we're given that 'this feels so arbitrary'. But it doesn't seem like 'feeling arbitrary' is a reason not to do something - especially when we're not given an alternative (or at least, no other decision process that seems less or equally arbitrary).
So my response is just to say 'using credences still seems fine, if occasionally emotionally uncomfortable (maybe using distributions is sometimes empirically better, and if so I support it)' - in which case I don't see a problem in need of solving.
Makes sense that range of threats should be wider (arbitrarily wide I guess, but a function of what scenarios you consider and differentiate between). I don't see why error estimates should be thin though - there are certainly people guessing close to 100% for some risks and various mechanisms that we might not even have considered that we would consider high risk if we knew more about them, which lead to us underestimating the risks of innocuous actions by a huge amount (c.f. the recent upsurge in concern about mirror bacteria)
I think 1) implies that you should give up some substantial optimisation for the sake of greater versatility (which seems approx titotal's view with reference to overcommitting)
2) feels correct and important to me, also since I've been arguing in the post op linked and elsewhere that treating extinction as special is a heuristic that was useful for initial cause prioritisation but isn't a valid reason for focusing on it two decades later.