Thank you for the extensive response, I will try to address each point.
1)
If I understand this correctly, this point is related to the notion that a physical process can be interpreted as executing an arbitrary algorithm, so there is no way to say what's the real algorithm that's being executed, therefore which algorithm is being executed depends entirely on the interpretation of a conscious observer about said function. This would make algorithms not real beyond said interpretation. To quote Michael Edward Johnson:
"Imagine you have a bag of popcorn. Now shake it. There will exist a certain ad-hoc interpretation of bag-of-popcorn-as-computational-system where you just simulated someone getting tortured, and other interpretations that don’t imply that. Did you torture anyone? If you’re a computationalist, no clear answer exists—you both did, and did not, torture someone. This sounds like a ridiculous edge-case that would never come up in real life, but in reality it comes up all the time, since there is no principled way to objectively derive what computation(s) any physical system is performing."
This is a potent argument against an algorithm being enough to determine qualia, and maybe even (on its own) agency. However, if algorithms are an arbitrary interpretation of consciousness, how could then a non-sentient AGI or ASI, if possible, maintain such robust "illusory" agency so as to reliably steer reality into very specific directions and even outsmart conscious intelligence? What's the source of such causal potency in a non-conscious algorithm that makes it observably different from the pop-corn tortured mind? The pop-corn tortured mind is an arbitrary interpretation among many possible, but if a Paperclip Maximizer starts consuming entire star systems, its goals could be independetly deduced by any kind of smart enough alien mind.
I think agency can be best understood as a primarily physical phenomenon, a process of information modeling the world and that model being able to cause transformations in matter and energy, and agency increasing means the capacity for that model to steer the outer world into more impactful and specific states increases. Therefore an algorithm is necessary but not sufficient for agency, an algorithm's interpretation would be real so long as it described the structure of such physical phenomenon. However abstract algorithms on their own also describe blueprints for agents, even if on their own they aren't the physically instatiated agent.
To take Searle's book as example, the book itself doesn't become more of an agent when the instructions are written on it, physically it's the same as any other book. However the instructions are the blueprint of general intelligence that any human capable of understanding their language could execute manually with pen and paper, thus the physical transformation of the world is performed by the human but is structured like the instructions in the book. The algorithm is as objective as any written communication can be, and anyone willing to execute it will reliably get the same effects in the world, thus any human willing to follow it is acting as a channel.
But what if said algorithm appeared when shaking pop-corn? Just like if staying written in the book without a facilitator, the pattern would be disconnected from the physical effects in the world it implies. There is an alternative to the algorithm objectively being in the pop-corn (or the book) and it being merely an interpretation, and that is that the algorithm as a pattern always latently exists, and is merely discovered (and communicated), just like any other abstract pattern.
In order to discover it in a random physical process, one needs to in some sense already know it, just like pareidolia allows us to see faces where "there are none". It being written in some language instead requires to merely understand said language. And in either case it only becomes a physically instatiated agent when guiding a physical transformation in its image, rather than merely being represented.
2)
That all referents to agency and its causal effects we have direct epistemological access to are structures within consciousness itself doesn't imply that they are the same as those structures, because insofar as agency is implemented through consciousness one would expect consciousness to be able to track it. However this implies that there is a world outside bounded conscious experiences that such model and affect, which leads directly to the third point.
3)
That there are no states of the world outside of moments of experience is a very strong claim so, first and foremost, I want to make sure I'm interpreting it correctly: Does this imply that if a tree falls in a forest, and no sentient being witnesses it (suppose there are no animals), then it didn't actually happen? Or it happened if later a sentient being discovers it, but otherwise not?
I don't think this is what you mean, however I'm not sure of the alternative. The tree, or rocks. etc are as far as I understand idealistic physicalism still made of qualia, but not bound into a unified experience that could experience the tree falling. If there are no states of the world outside experiences, then what happens in that case?
(On a side note, this instrumentalization would also be a case of an abstract algorithm physically instatiating itself through consciousness, in a way perhaps much more invasive than self-awarely following clearly external instructions, but still ultimately possible to detatch from).
But even when separated from the evolved, suboptimal phenomenology, I don't think agency merely substracts from valence. As I understand the state of permanent unconditional almost maxed out well-being doesn't destroy one's agency but increase it, and even within the context of pursuing only experiences, the search for novelty seemingly outwheighs merely wanting to "reach the maximum valence and stay there".
It could be the case that anything consciousness-as-consciousness could want is better realized within its own world simulation, whereas blindminds are optimal for pursuing values pertaining rearranging the rest of reality. If that was the case we could expect both types of minds to ideally be divorced from each other for the most part, but that doesn't inherently imply a conflict.
5)
Of this I will just ask if you have read the Part 2 and if not suggest you do, there I elaborate on a framework that can generalize to include both conscious and non-conscious agents.
6)
This is an interesting explanation for blindsight and might as well be true. While, if true, it would make blindminds more distant to us in the immediate (not being even partially one), I don't think that's ultimately as central as the notion that our own agency could be run by a blindmind (if mind uploading is possible but uploads can't be sentient). Or that if the entire informational content of our qualia was replaced we wouldn't be ourselves anymore, except in the Open Individualist sense.
Also, if the brain runs on multiple separate pockets of experience, wouldn't the intelligence that runs on all of them still be phenomenally unbound? Like a little "China Brain" inside the actual brain.
7) (Pragmatic and political reasons).
Would the argument here remain the same when talking about sentient minds with alien values?
Like you said, sentient but alien minds could also be hostile and disempower us. However, you have been mapping out universal (to consciousness) scheling points, ways to coordinate, rather than just jumping to the notion that alien values means irreconciliable hostility.
I think this is because of Valence Realism making values (among sentient beings) not being fully arbitrary and irreconciliable, but non-sentient minds of course would have no valence.
And yet, I don't think the issue changes that much when considering blindminds, insofar as they have values (in the sense an AGI as a physical system can be expected to steer the universe in certain ways, as opposed to an algorithm interpreted into a popcorn shaking) then game theory and methods of coordination apply to them too.
Here I once again ask if you have read and if not suggest you read the Part 2, for what a coalition of agents in general, sentient and not, could look like.
While I agree it's different to give consideration/rights than to conclude something is a moral patient (by default respecting entities of unknown sentience as precaution), my point is also that the exclusion of blindminds from intrinsic concern easily biases towards considering their potential freedom more catasthrophic than than the potential enslavement of sentients.
Discussions of AI rights often emphasize a call for precaution in the form of "because we dónde know how to measure sentience, we should be very careful about granting personhood to AI that acts like a person".
And, charitably, sometimes that could just mean not taking AI's self-assesment at face-value. But often it's less ambiguous: That legal personhood of actually human-level AI shouldn't be given on the risk that it's not sentient. And this I call an inversion of the precautionary principle.
So I argue, even if one doesn't see intrinsic value in blindminds, there is no reason to see them as net negative monsters, and so effective defense of the precautionary principle regarding sentient AI needs to be willing to at least say "and if non-sentient AI gets rights, that wouldn't be a catasthrophe".
But, apart from that, I also argue why non-sentient intelligences could be moral patients intrinsically anyway.
Your interpretation isn't exactly wrong, I'm proposing an onthological shift on the understanding of what's more central to the self, the thing to care about (i.e. is the moral patient fundamentally a qualia that has or can have an agent, or an agent that has or can have qualia?).
The intuition is that if qualia, on its own, is generic and completely interchangeable among moral patients, it might not be what makes them such, even if it's an important value. A blindmind upload has ultimately far more in common with the sentient person they are based on than said person has with a phenomenal experience devoid of all the content that makes up their agency.
Thus it would be the agent the thing that primarily values the qualia (and everything else), rather than the reverse. This decenters qualia even if it is exceptionally valuable, being valuable not a priori (and thus, the agent being valued instrumentally in order to ensure its existence) but because it was chosen (and the thing that would have intrinsic value would be that which can make such choices).
A blindmind that doesn't want qualia would be valuable then in this capacity to value things about the world in general, of which qualia is just a particular type (even if very valuable for sentient agents).
The appropiate type to compare rather than a Paperclip Maximizer (who, in Part 2 I argue, represents a type of agent whose values are inherently an aggression against the possibility of universal cooperation) would be aliens with strange and hard to comprehend values but no more intrinsically tied to the destruction of everything else than human values. If the moral patiency in them is only their qualia, then the best thing we could do for them is to just give them positive feelings, routing around whatever they valued in particular as means to that (and thus ultimately not really being about changing the outer world).
Respecting their agency would mean at least trying to understand what they are trying to do, from their perspective, not necessarily to give them everything they want (that's subject to many considerations and their particular values), but to respect their goals in the sense that, when a human wants to make some great art, we take that helping them means helping them with that, rather than puting them in an experience machine where they think they did it.
Thank you for the extensive response, I will try to address each point.
1)
If I understand this correctly, this point is related to the notion that a physical process can be interpreted as executing an arbitrary algorithm, so there is no way to say what's the real algorithm that's being executed, therefore which algorithm is being executed depends entirely on the interpretation of a conscious observer about said function. This would make algorithms not real beyond said interpretation. To quote Michael Edward Johnson:
"Imagine you have a bag of popcorn. Now shake it. There will exist a certain ad-hoc interpretation of bag-of-popcorn-as-computational-system where you just simulated someone getting tortured, and other interpretations that don’t imply that. Did you torture anyone? If you’re a computationalist, no clear answer exists—you both did, and did not, torture someone. This sounds like a ridiculous edge-case that would never come up in real life, but in reality it comes up all the time, since there is no principled way to objectively derive what computation(s) any physical system is performing."
This is a potent argument against an algorithm being enough to determine qualia, and maybe even (on its own) agency. However, if algorithms are an arbitrary interpretation of consciousness, how could then a non-sentient AGI or ASI, if possible, maintain such robust "illusory" agency so as to reliably steer reality into very specific directions and even outsmart conscious intelligence? What's the source of such causal potency in a non-conscious algorithm that makes it observably different from the pop-corn tortured mind? The pop-corn tortured mind is an arbitrary interpretation among many possible, but if a Paperclip Maximizer starts consuming entire star systems, its goals could be independetly deduced by any kind of smart enough alien mind.
I think agency can be best understood as a primarily physical phenomenon, a process of information modeling the world and that model being able to cause transformations in matter and energy, and agency increasing means the capacity for that model to steer the outer world into more impactful and specific states increases. Therefore an algorithm is necessary but not sufficient for agency, an algorithm's interpretation would be real so long as it described the structure of such physical phenomenon. However abstract algorithms on their own also describe blueprints for agents, even if on their own they aren't the physically instatiated agent.
To take Searle's book as example, the book itself doesn't become more of an agent when the instructions are written on it, physically it's the same as any other book. However the instructions are the blueprint of general intelligence that any human capable of understanding their language could execute manually with pen and paper, thus the physical transformation of the world is performed by the human but is structured like the instructions in the book. The algorithm is as objective as any written communication can be, and anyone willing to execute it will reliably get the same effects in the world, thus any human willing to follow it is acting as a channel.
But what if said algorithm appeared when shaking pop-corn? Just like if staying written in the book without a facilitator, the pattern would be disconnected from the physical effects in the world it implies. There is an alternative to the algorithm objectively being in the pop-corn (or the book) and it being merely an interpretation, and that is that the algorithm as a pattern always latently exists, and is merely discovered (and communicated), just like any other abstract pattern.
In order to discover it in a random physical process, one needs to in some sense already know it, just like pareidolia allows us to see faces where "there are none". It being written in some language instead requires to merely understand said language. And in either case it only becomes a physically instatiated agent when guiding a physical transformation in its image, rather than merely being represented.
2)
That all referents to agency and its causal effects we have direct epistemological access to are structures within consciousness itself doesn't imply that they are the same as those structures, because insofar as agency is implemented through consciousness one would expect consciousness to be able to track it. However this implies that there is a world outside bounded conscious experiences that such model and affect, which leads directly to the third point.
3)
That there are no states of the world outside of moments of experience is a very strong claim so, first and foremost, I want to make sure I'm interpreting it correctly: Does this imply that if a tree falls in a forest, and no sentient being witnesses it (suppose there are no animals), then it didn't actually happen? Or it happened if later a sentient being discovers it, but otherwise not?
I don't think this is what you mean, however I'm not sure of the alternative. The tree, or rocks. etc are as far as I understand idealistic physicalism still made of qualia, but not bound into a unified experience that could experience the tree falling. If there are no states of the world outside experiences, then what happens in that case?
4)
Even if agency can only manifest within consciousness by substracting from valence in some way, I don't think it follows that it's inherently disvaluable, and in particular not in the case of blindminds. If it's the case that qualia in animals is enslaved/instrumentalized by the brain to use its intelligence, then a blindmind would be able to not even cause suffering in order to pursue its values.
(On a side note, this instrumentalization would also be a case of an abstract algorithm physically instatiating itself through consciousness, in a way perhaps much more invasive than self-awarely following clearly external instructions, but still ultimately possible to detatch from).
But even when separated from the evolved, suboptimal phenomenology, I don't think agency merely substracts from valence. As I understand the state of permanent unconditional almost maxed out well-being doesn't destroy one's agency but increase it, and even within the context of pursuing only experiences, the search for novelty seemingly outwheighs merely wanting to "reach the maximum valence and stay there".
It could be the case that anything consciousness-as-consciousness could want is better realized within its own world simulation, whereas blindminds are optimal for pursuing values pertaining rearranging the rest of reality. If that was the case we could expect both types of minds to ideally be divorced from each other for the most part, but that doesn't inherently imply a conflict.
5)
Of this I will just ask if you have read the Part 2 and if not suggest you do, there I elaborate on a framework that can generalize to include both conscious and non-conscious agents.
6)
This is an interesting explanation for blindsight and might as well be true. While, if true, it would make blindminds more distant to us in the immediate (not being even partially one), I don't think that's ultimately as central as the notion that our own agency could be run by a blindmind (if mind uploading is possible but uploads can't be sentient). Or that if the entire informational content of our qualia was replaced we wouldn't be ourselves anymore, except in the Open Individualist sense.
Also, if the brain runs on multiple separate pockets of experience, wouldn't the intelligence that runs on all of them still be phenomenally unbound? Like a little "China Brain" inside the actual brain.
7) (Pragmatic and political reasons).
Would the argument here remain the same when talking about sentient minds with alien values?
Like you said, sentient but alien minds could also be hostile and disempower us. However, you have been mapping out universal (to consciousness) scheling points, ways to coordinate, rather than just jumping to the notion that alien values means irreconciliable hostility.
I think this is because of Valence Realism making values (among sentient beings) not being fully arbitrary and irreconciliable, but non-sentient minds of course would have no valence.
And yet, I don't think the issue changes that much when considering blindminds, insofar as they have values (in the sense an AGI as a physical system can be expected to steer the universe in certain ways, as opposed to an algorithm interpreted into a popcorn shaking) then game theory and methods of coordination apply to them too.
Here I once again ask if you have read and if not suggest you read the Part 2, for what a coalition of agents in general, sentient and not, could look like.
I consider Claude's summary accurate.
While I agree it's different to give consideration/rights than to conclude something is a moral patient (by default respecting entities of unknown sentience as precaution), my point is also that the exclusion of blindminds from intrinsic concern easily biases towards considering their potential freedom more catasthrophic than than the potential enslavement of sentients.
Discussions of AI rights often emphasize a call for precaution in the form of "because we dónde know how to measure sentience, we should be very careful about granting personhood to AI that acts like a person".
And, charitably, sometimes that could just mean not taking AI's self-assesment at face-value. But often it's less ambiguous: That legal personhood of actually human-level AI shouldn't be given on the risk that it's not sentient. And this I call an inversion of the precautionary principle.
So I argue, even if one doesn't see intrinsic value in blindminds, there is no reason to see them as net negative monsters, and so effective defense of the precautionary principle regarding sentient AI needs to be willing to at least say "and if non-sentient AI gets rights, that wouldn't be a catasthrophe".
But, apart from that, I also argue why non-sentient intelligences could be moral patients intrinsically anyway.
Your interpretation isn't exactly wrong, I'm proposing an onthological shift on the understanding of what's more central to the self, the thing to care about (i.e. is the moral patient fundamentally a qualia that has or can have an agent, or an agent that has or can have qualia?).
The intuition is that if qualia, on its own, is generic and completely interchangeable among moral patients, it might not be what makes them such, even if it's an important value. A blindmind upload has ultimately far more in common with the sentient person they are based on than said person has with a phenomenal experience devoid of all the content that makes up their agency.
Thus it would be the agent the thing that primarily values the qualia (and everything else), rather than the reverse. This decenters qualia even if it is exceptionally valuable, being valuable not a priori (and thus, the agent being valued instrumentally in order to ensure its existence) but because it was chosen (and the thing that would have intrinsic value would be that which can make such choices).
A blindmind that doesn't want qualia would be valuable then in this capacity to value things about the world in general, of which qualia is just a particular type (even if very valuable for sentient agents).
The appropiate type to compare rather than a Paperclip Maximizer (who, in Part 2 I argue, represents a type of agent whose values are inherently an aggression against the possibility of universal cooperation) would be aliens with strange and hard to comprehend values but no more intrinsically tied to the destruction of everything else than human values. If the moral patiency in them is only their qualia, then the best thing we could do for them is to just give them positive feelings, routing around whatever they valued in particular as means to that (and thus ultimately not really being about changing the outer world).
Respecting their agency would mean at least trying to understand what they are trying to do, from their perspective, not necessarily to give them everything they want (that's subject to many considerations and their particular values), but to respect their goals in the sense that, when a human wants to make some great art, we take that helping them means helping them with that, rather than puting them in an experience machine where they think they did it.