I think your notion of counting distinct paths has a less satisfactory answer than 'standard' util though.
Here's a scenario I think shows the weirdness. Imagine a single mind. We copy this mind into two instantiations, which have slightly different experiences for a tenth of a second. We then merge these minds. We repeat this process a thousand times. There are then 2^1000 distinct experience paths here. But counting the future experiences of the 2^1000 merged mind at that full multiple seems quite silly.
If two toll roads merge into one, should the merged road cost twice as much per mile? No, because its now one road. Having two entries does not make it twice as valuable. Why are minds different?
Also, the notion of merging minds only makes sense (to me) under some form of dualism. Ie, you need some non-physical mind stuff to do a complete merge. I don't think imagining me & you merging minds can result in one entity that carries both 100% of my weight and 100% of your weight. At best we blur together and get a fraction of each. More likely, both of us are functionally dead post-merge and there is a new 3rd conscious being.
Due to 2, I think I might go as high as 0.1% risk over the next 20 years (with incredibly wide error bars). That still seems high enough to seriously consider, although I agree other X-risk concerns are likely even higher.
This post feels like it is saying that lone actors are currently incapable of creating an X-risk bioweapon. I agree. I think almost everyone in the X-risk scene agrees.
The concern is instead about whether this capability will exist soon. To show that being impossible (or very unlikely), IMO you need to show that there is at least one critical step that is not getting democratized. IE, bio-expertise is getting cheaper because of increasingly powerful open source models. Robots with sufficient dexterity to do bio experiments are getting more capable and cheaper. Where is the bottleneck that will remain unsolved for 2+ decades, and why are we confident it stays unsolved?
To give an analogy, 2 years ago non-experts couldn't solve major open math problems. Today, with access to frontier models, they can. How can we be confident that the bottlenecks against independently creating novel & powerful plagues will not also fall?
Reading that section does not change my opinion on this at all, fwiw.
Maybe this will help convey my intuition. Imagine
A100*: As A100, but every shock is accompanied by a man saying a different random word.
Now, the experience of each shock in A100* is different. So A100* is not 100X as bad as A1. But I think most people would agree that A100 and A100* are not meaningfully different.
I think your notion of 'subjectively identical experiences' is not a natural way to count. And using a boolean criterion for this (identical experiences are given no extra weight, minorly different experiences are given full weight) feels awkward.
I guess I'm not sure how you're going to figure out that the subjective experience of multiple copies is identical. Even if you did, I would personally view experiencing identical joy twice (literally indistinguishable, but happened 2 times) as twice as valuable.
I think you're saying that each subjectively indistinguishable experience class only counts as 1, regardless of the number of instantiations of that experience. I can understand that, but I definitely disagree (I count the number of instantiations).
Im very unsure how we'd settle this disagreement. Maybe the following thought experiment would persuade you:
A1: You are put in a chair, and given a harmless (but painful) electric shock. Your memory of this experience is then wiped, and you are returned to your state before the shock.
A100: As above, but we repeat the procedure 100 times. Each experience is subjectively identical.
IMO A100 is worse than A1. My naive expectation is that you are neutral between the two.
My intuition is that if N copies experience something, that experience gets multiplied by N. I suspect this is closer to most people's intuition.
To give an example, 10k chickens being debeaked is about 10k as bad as one chicken being debeaked. Individual chickens are pretty close to copies from a conscious perspective (not completely so, but much closer to copies than two randomly chosen humans).
I think the cringe reason is why I'll say I'm EA, and then describe it as "quote unquote effective altruism" (if they ask). Which solves the problem decently well, in the same way folks describe themselves as 'rats' instead of rationalists.
Merging & copying gets weird, no questions.
I think your notion of counting distinct paths has a less satisfactory answer than 'standard' util though.
Here's a scenario I think shows the weirdness. Imagine a single mind. We copy this mind into two instantiations, which have slightly different experiences for a tenth of a second. We then merge these minds. We repeat this process a thousand times. There are then 2^1000 distinct experience paths here. But counting the future experiences of the 2^1000 merged mind at that full multiple seems quite silly.
If two toll roads merge into one, should the merged road cost twice as much per mile? No, because its now one road. Having two entries does not make it twice as valuable. Why are minds different?
Also, the notion of merging minds only makes sense (to me) under some form of dualism. Ie, you need some non-physical mind stuff to do a complete merge. I don't think imagining me & you merging minds can result in one entity that carries both 100% of my weight and 100% of your weight. At best we blur together and get a fraction of each. More likely, both of us are functionally dead post-merge and there is a new 3rd conscious being.
Oh thanks for adding concrete figures, that is very helpful (I should have done the same)!
This post feels like it is saying that lone actors are currently incapable of creating an X-risk bioweapon. I agree. I think almost everyone in the X-risk scene agrees.
The concern is instead about whether this capability will exist soon. To show that being impossible (or very unlikely), IMO you need to show that there is at least one critical step that is not getting democratized. IE, bio-expertise is getting cheaper because of increasingly powerful open source models. Robots with sufficient dexterity to do bio experiments are getting more capable and cheaper. Where is the bottleneck that will remain unsolved for 2+ decades, and why are we confident it stays unsolved?
To give an analogy, 2 years ago non-experts couldn't solve major open math problems. Today, with access to frontier models, they can. How can we be confident that the bottlenecks against independently creating novel & powerful plagues will not also fall?
Reading that section does not change my opinion on this at all, fwiw.
Maybe this will help convey my intuition. Imagine A100*: As A100, but every shock is accompanied by a man saying a different random word.
Now, the experience of each shock in A100* is different. So A100* is not 100X as bad as A1. But I think most people would agree that A100 and A100* are not meaningfully different.
I think your notion of 'subjectively identical experiences' is not a natural way to count. And using a boolean criterion for this (identical experiences are given no extra weight, minorly different experiences are given full weight) feels awkward.
Well, biting that bullet at least keeps you consistent.
Fwiw, I think my take is more common. I asked a few folks, everyone asked agreed A1 is less bad than A100.
Not saying that is strong evidence that you're wrong, but it does call for more justification of your non-standard choice.
I guess I'm not sure how you're going to figure out that the subjective experience of multiple copies is identical. Even if you did, I would personally view experiencing identical joy twice (literally indistinguishable, but happened 2 times) as twice as valuable.
I think you're saying that each subjectively indistinguishable experience class only counts as 1, regardless of the number of instantiations of that experience. I can understand that, but I definitely disagree (I count the number of instantiations).
Im very unsure how we'd settle this disagreement. Maybe the following thought experiment would persuade you:
A1: You are put in a chair, and given a harmless (but painful) electric shock. Your memory of this experience is then wiped, and you are returned to your state before the shock.
A100: As above, but we repeat the procedure 100 times. Each experience is subjectively identical.
IMO A100 is worse than A1. My naive expectation is that you are neutral between the two.
Do you mean it is bad for one's own mental health (strong agree) or that it should be discouraged (unsure if I agree)?
My intuition is that if N copies experience something, that experience gets multiplied by N. I suspect this is closer to most people's intuition.
To give an example, 10k chickens being debeaked is about 10k as bad as one chicken being debeaked. Individual chickens are pretty close to copies from a conscious perspective (not completely so, but much closer to copies than two randomly chosen humans).
I think that this post is wrong for the same reason that eigenism is wrong. Bentham's bulldog gives a better rebuttal than I would: https://benthams.substack.com/p/dan-hendrycks-moral-theory-is-very
I think the cringe reason is why I'll say I'm EA, and then describe it as "quote unquote effective altruism" (if they ask). Which solves the problem decently well, in the same way folks describe themselves as 'rats' instead of rationalists.