Daniel, You provide good evidence that we will experience a period of SIE. Still I think we can make a second argument that this period of SIE will come to an end. Perhaps it even points towards a second way to assess consequences of SIE.
My notion of an asymptotic performances is easiest seen on a much simpler problem. Consider the task of doing of doing parallel multiplication in silicon. Over the years we have definitely improved the multiplication performance in speed and chip area (for a fixed lithography tech level). I expect there was a period of time where is somehow the speed of human innovation was proportional to current multiplication speed then we would have seen a period of SIE for chip multipliers. Still as our designs approached the (unknown) asymptotic limit of multiplication performance in our chip design this explosion would level off again.
In the same way, if fix the task of running an AI agent capable of ASARA and fix the HW, then there must exist an asymptotically best design theoretically possible. From these if follows that period of SIE must stop as designs approach this asymptote.
This raises an interesting secondary question: How many multiples exist between our first ASARA system, and the asymptotically best one? If that is 10x, that implies a certain profile for SIE, if it is 10,000x then it is a very different profile for SIE. In the end it might be this multiple rather than the velocity of SIE that has greater sway over its societal outcome.
I agree that we should be considering "grey area" cases where both motivation and ability may not ensure domination. Indeed I argue we need to further muddy the motivational landscape to include "rouge" human motivations as we find in abundance today.
The first AGI systems won't need to 'go rogue' in order for its behavior to be 'rogue' as viewed by humanity at large. It is overwhelmingly that likely these systems will be borne-into and harnessed-in-service-of the corporate and military struggles that already exist today.
These systems will be 'rouge' and in conflict by design. And just like their master's they will be very acquainted with many 'rogue' methods of achieving their goals.
It seems likely humans over time will voluntarily relinquish control and understanding to their AIs in service of the shared goal they have with their AIs.
Of course over time the AI's goals may not stay aligned, but even knowing this we are financially and politically incapable of resisting this path, where ever it may lead. It is an obvious prisoner's dilemma where humans the defect stand to gain enormously.
Ironically I think this nearly unavoidable path LOWERS our chances for a unilateral fast take off. Long before one is possible many human+machine corporations and militaries will already be very focused on detecting and stopping any OTHER rogue AIs attempting this.
Still I am not sanguine. This militarized community will be fast moving and hard for humans to understand. The chances we can manage shape such a society seems lower than our proven inability to shape human society today. And our competing interests will make unified human action nearly impossible.
Here I think the EA movement has a very substantial role to play. My hypothesis is thus that the EA message is very fruitful even though it may be philosophically trivial.
It seems correct given EA's goals, its effectiveness should not be measured philosophically -- instead it should be assessed practically. If EA fails, likely it is because it becomes meta discussion (like this one) and fails to make a difference in this world. (This is not intended as a dig against the present discussion). My sense is that EA sometimes involves interested parties that are not directly involved in DOING the relevant activities in question. Thus it is a kind of meta-discussion by its nature. I think this is fine... as an AI guy I notice that practitioners rarely ask the hardest questions questions about what they are doing. as a former DARPA guy I saw the same myopia in the defense sphere. So outsiders may well be the right ingredient to add.
Personally I would assess EA on the basis of its subjectively/objectively-assessed movement of the Overton window for relevant decision makers. e. g. company owner, voters, activists, researchers, etc. The issues EA takes on are really quite large. It seems hard to directly move that needle. Still it seems plausible that EA could end up being transformative by changing very thinking of humanity. And it seems possible that it gets wrapped up into its own sub-communities whose beliefs end up diverging from humanity at large and thus are ignored by humanity at large.
When I look at questions around AGI safety I think the tiny amounts of human effort and money expended by EA, perhaps this can be counted as a "win" ... that humanity's thinking is moving in directions that will affect large scale policy. (On this particular issue, I fall into the "too little too late" camp) but still I have to acknowledge the apparently real impact EA has had in legitimizing this topic in practically impactful ways.
Here I think the EA movement has a very substantial role to play. My hypothesis is thus that the EA message is very fruitful even though it may be philosophically trivial.
It seems correct given EA's goals, its effectiveness should not be measured philosophically -- instead it should be assessed practically. If EA fails, likely it is because it becomes meta discussion (like this one) and fails to make a difference in this world. (This is not intended as a dig against the present discussion). My sense is that EA sometimes involves interested parties that are not directly involved in DOING the relevant activities in question. Thus it is a kind of meta-discussion by its nature. I think this is fine... as an AI guy I notice that practitioners rarely ask the hardest questions questions about what they are doing. as a former DARPA guy I saw the same myopia in the defense sphere. So outsiders may well be the right ingredient to add.
Personally I would assess EA on the basis of its subjectively/objectively-assessed movement of the Overton window for relevant decision makers. e. g. company owner, voters, activists, researchers, etc. The issues EA takes on are really quite large. It seems hard to directly move that needle. Still it seems plausible that EA could end up being transformative by changing very thinking of humanity. And it seems possible that it gets wrapped up into its own sub-communities whose beliefs end up diverging from humanity at large and thus are ignored by humanity at large.
When I look at questions around AGI safety I think the tiny amounts of human effort and money expended by EA, perhaps this can be counted as a "win" ... that humanity's thinking is moving in directions that will affect large scale policy. (On this particular issue, I fall into the "too little too late" camp) but still I have to acknowledge the apparently real impact EA has had in legitimizing this topic in practically impactful ways.
Supporting the community with this new competition is quite valuable. Thanks!
Here is an idea for how your impact might be amplified: For ever researcher that is somehow has full time funding to do AI safety research I suspect there are 10 qualified researchers with interest and novel ideas to contribute, but who will likely never be full time funded for AI safety work. Prizes like these can enable this much larger community to participate in a very capital efficient way.
But such "part time" contributions are likely to unfold over longer periods, and ideally would involve significant feedback from the full-time community in order to maximize the value of those contributions.
The previous prize required that all submissions be of never before published work. I understand the reasoning here. They wanted to foster NEW work. Still this rule drops a wet blanket on any part-timer who might want to gain feedback on ideas over time.
Here is an alternate rule that might have fewer unintended side effects: Only the portions of ones work that has never been awarded prize money in the past is eligible for consideration.
Such a rule would allow a part-timer to refine an important contribution with extensive feedback from the community over an extended period of time. Biasing towards fewer higher quality contributions in a field with so much uncertainty seems a worthy goal. Biasing towards greater numbers of contributors in such a small field also seems valuable from a diversity in thinking perspective too.
Daniel, You provide good evidence that we will experience a period of SIE. Still I think we can make a second argument that this period of SIE will come to an end. Perhaps it even points towards a second way to assess consequences of SIE.
My notion of an asymptotic performances is easiest seen on a much simpler problem. Consider the task of doing of doing parallel multiplication in silicon. Over the years we have definitely improved the multiplication performance in speed and chip area (for a fixed lithography tech level). I expect there was a period of time where is somehow the speed of human innovation was proportional to current multiplication speed then we would have seen a period of SIE for chip multipliers. Still as our designs approached the (unknown) asymptotic limit of multiplication performance in our chip design this explosion would level off again.
In the same way, if fix the task of running an AI agent capable of ASARA and fix the HW, then there must exist an asymptotically best design theoretically possible. From these if follows that period of SIE must stop as designs approach this asymptote.
This raises an interesting secondary question: How many multiples exist between our first ASARA system, and the asymptotically best one? If that is 10x, that implies a certain profile for SIE, if it is 10,000x then it is a very different profile for SIE. In the end it might be this multiple rather than the velocity of SIE that has greater sway over its societal outcome.
Thoughts on this?
--Dan
I agree that we should be considering "grey area" cases where both motivation and ability may not ensure domination. Indeed I argue we need to further muddy the motivational landscape to include "rouge" human motivations as we find in abundance today.
The first AGI systems won't need to 'go rogue' in order for its behavior to be 'rogue' as viewed by humanity at large. It is overwhelmingly that likely these systems will be borne-into and harnessed-in-service-of the corporate and military struggles that already exist today.
These systems will be 'rouge' and in conflict by design. And just like their master's they will be very acquainted with many 'rogue' methods of achieving their goals.
It seems likely humans over time will voluntarily relinquish control and understanding to their AIs in service of the shared goal they have with their AIs.
Of course over time the AI's goals may not stay aligned, but even knowing this we are financially and politically incapable of resisting this path, where ever it may lead. It is an obvious prisoner's dilemma where humans the defect stand to gain enormously.
Ironically I think this nearly unavoidable path LOWERS our chances for a unilateral fast take off. Long before one is possible many human+machine corporations and militaries will already be very focused on detecting and stopping any OTHER rogue AIs attempting this.
Still I am not sanguine. This militarized community will be fast moving and hard for humans to understand. The chances we can manage shape such a society seems lower than our proven inability to shape human society today. And our competing interests will make unified human action nearly impossible.
Vasco, this can hardly be posed as a fair bet. If you win, you get paid, if you loose, then money ceases to exist.
It seems correct given EA's goals, its effectiveness should not be measured philosophically -- instead it should be assessed practically. If EA fails, likely it is because it becomes meta discussion (like this one) and fails to make a difference in this world. (This is not intended as a dig against the present discussion). My sense is that EA sometimes involves interested parties that are not directly involved in DOING the relevant activities in question. Thus it is a kind of meta-discussion by its nature. I think this is fine... as an AI guy I notice that practitioners rarely ask the hardest questions questions about what they are doing. as a former DARPA guy I saw the same myopia in the defense sphere. So outsiders may well be the right ingredient to add.
Personally I would assess EA on the basis of its subjectively/objectively-assessed movement of the Overton window for relevant decision makers. e. g. company owner, voters, activists, researchers, etc. The issues EA takes on are really quite large. It seems hard to directly move that needle. Still it seems plausible that EA could end up being transformative by changing very thinking of humanity. And it seems possible that it gets wrapped up into its own sub-communities whose beliefs end up diverging from humanity at large and thus are ignored by humanity at large.
When I look at questions around AGI safety I think the tiny amounts of human effort and money expended by EA, perhaps this can be counted as a "win" ... that humanity's thinking is moving in directions that will affect large scale policy. (On this particular issue, I fall into the "too little too late" camp) but still I have to acknowledge the apparently real impact EA has had in legitimizing this topic in practically impactful ways.
It seems correct given EA's goals, its effectiveness should not be measured philosophically -- instead it should be assessed practically. If EA fails, likely it is because it becomes meta discussion (like this one) and fails to make a difference in this world. (This is not intended as a dig against the present discussion). My sense is that EA sometimes involves interested parties that are not directly involved in DOING the relevant activities in question. Thus it is a kind of meta-discussion by its nature. I think this is fine... as an AI guy I notice that practitioners rarely ask the hardest questions questions about what they are doing. as a former DARPA guy I saw the same myopia in the defense sphere. So outsiders may well be the right ingredient to add.
Personally I would assess EA on the basis of its subjectively/objectively-assessed movement of the Overton window for relevant decision makers. e. g. company owner, voters, activists, researchers, etc. The issues EA takes on are really quite large. It seems hard to directly move that needle. Still it seems plausible that EA could end up being transformative by changing very thinking of humanity. And it seems possible that it gets wrapped up into its own sub-communities whose beliefs end up diverging from humanity at large and thus are ignored by humanity at large.
When I look at questions around AGI safety I think the tiny amounts of human effort and money expended by EA, perhaps this can be counted as a "win" ... that humanity's thinking is moving in directions that will affect large scale policy. (On this particular issue, I fall into the "too little too late" camp) but still I have to acknowledge the apparently real impact EA has had in legitimizing this topic in practically impactful ways.
Supporting the community with this new competition is quite valuable. Thanks!
Here is an idea for how your impact might be amplified: For ever researcher that is somehow has full time funding to do AI safety research I suspect there are 10 qualified researchers with interest and novel ideas to contribute, but who will likely never be full time funded for AI safety work. Prizes like these can enable this much larger community to participate in a very capital efficient way.
But such "part time" contributions are likely to unfold over longer periods, and ideally would involve significant feedback from the full-time community in order to maximize the value of those contributions.
The previous prize required that all submissions be of never before published work. I understand the reasoning here. They wanted to foster NEW work. Still this rule drops a wet blanket on any part-timer who might want to gain feedback on ideas over time.
Here is an alternate rule that might have fewer unintended side effects: Only the portions of ones work that has never been awarded prize money in the past is eligible for consideration.
Such a rule would allow a part-timer to refine an important contribution with extensive feedback from the community over an extended period of time. Biasing towards fewer higher quality contributions in a field with so much uncertainty seems a worthy goal. Biasing towards greater numbers of contributors in such a small field also seems valuable from a diversity in thinking perspective too.