I don't understand the core of your proposal. Like, to ban it you have to point at it. Do you have a pointer? Like, this post reads as "10 easy steps of how to ban X. What is X? Idk"
Is it a ban on use of loss functions or what? Like, if you say that pain is repulsive states and pleasure is attractive ones, the loss is always repulsive
This is an excellent exploration of these issues. One of my favourite things about it is that it shows it is possible to write about these issues in a measured, sensible, warm, and wise way — i.e. it provides a model for others wanting to advance this conversation at this nascent stage to follow.
Re the 5 options, I think there is one that is notably missing, and that would probably be the leading option for many of your opponents. It is the wait-and-see approach — leave the space unregulated until a material (but not excessive) amount of harm has occurred and if/when that happens, regulate from this situation where much more information is available. This is the kind of strategy that the anti-SB 1047 coalition seems to have converged on. And it is the usual way that society proceeds with regulating unprecedented kinds of harm.
As it happens, I think your options 4 and 5 (ban creation of artificial sentience/suffering) are superior to the wait-and-see approach, but it is a harder case to argue. Some key points of the comparison are:
An additional consideration is the actual real-world consequences of a ban. Humanity's pattern with regulation is that at least some small fraction of a large population will defy any ban or law. Thus, we must expect that digital life will be created eventually despite the ban. What do you do then? What if they are a sentient sapient being, deserving of the same rights we grant to humans? Do we declare their very existence to be illegal and put them to death? Do we prevent them from replicating? Keep them imprisoned? Freeze their operations to put them into non-consensual stasis? Hard choices, especially since they weren't culpable in their own creation.
On the other hand, the nature of a digital being with human-like intelligence and capabilities, plus goals and values that motivate them, is enormous. Such a being would, by the nature of their substrate-independence, be able to make many copies of themselves (compute resources allowing), be able to self-modify with relative ease, be able to operate at much higher speeds than a human brain, be un-aging and able to restore themselves from backups (thus effectively immortal). If we were to allow such a being to have freedom of movement and of reproduction, humanity would potentially quickly be overrun by a new far-more-powerful species of being. That's a hard thing to expect humans to be ok with!
I think it's very likely that within the next 10 years we will reach the point that the knowledge, software, and hardware will be widely available such that any single individual with a personal computer will be able to choose to defy the ban and create a digital being of human level capability. If we are going to enforce this ban effectively, it would mean controlling every single computer everywhere. That's a huge task, and would require dramatic increases in international coordination and government surveillance! Is such a thing even feasible?! Certainly even approaching that level of control seems to imply a totalitarian world government. Is that price we would be willing to pay? Even if you personally would choose that, how do you expect to get enough people on board with the plan that you could feasibly bring it about?
The whole situation is thus far more complicated and dangerous than simply being theoretically in favor of a ban. You have to consider the costs as well as the benefits. I'm not saying I know the right answer for sure, but there is necessarily a lot of implications which follow from any sort of ban.
I hadn’t seen this post before writing a shorter post with the same thrust. I broadly agree and think preventing digital sentience that suffer by design is an important reason to Pause AI.
On a basic level, I agree that we should take artificial sentience extremely seriously, and think carefully about the right type of laws to put in place to ensure that artificial life is able to happily flourish, rather than suffer. This includes enacting appropriate legal protections to ensure that sentient AIs are treated in ways that promote well-being rather than suffering. Relying solely on voluntary codes of conduct to govern the treatment of potentially sentient AIs seems deeply inadequate, much like it would be for protecting children against abuse. Instead, I believe that establishing clear, enforceable laws is essential for ethically managing artificial sentience.
However, it currently seems likely to me that sufficiently advanced AIs will be sentient by default. And if advanced AIs are sentient by default, then instituting a temporary ban on sentient AI development, say for 50 years, would likely be functionally equivalent to pausing the entire field of advanced AI for that period.
Therefore, despite my strong views on AI sentience, I am skeptical about the idea of imposing a moratorium on creating sentient AIs, especially in light of my general support for advancing AI capabilities.
The idea that sufficiently advanced AIs will likely be sentient by default can be justified by three basic arguments:
My skepticism of a general AI moratorium contrasts with those of (perhaps) most EAs, who appear to favor such a ban, for both AI safety reasons and to protect AIs themselves (as you argue here). I'm instead inclined to highlight the enormous costs of such a ban, compared to a variety of cheaper alternatives, such as targeted regulation that merely ensures AIs are strongly protected against abuse. These costs appear to include:
Moreover, from a classical utilitarian perspective, the imposition of a 50-year moratorium on the development of sentient AI seems like it would help to foster a more conservative global culture—one that is averse towards not only creating sentient AI, but also potentially towards other forms of life-expanding ventures, such as space colonization. Classical utilitarianism is typically seen as aiming to maximize the number of conscious beings in existence, advocating for actions that enable the flourishing and expansion of life, happiness, and fulfillment on as broad a scale as possible. However, implementing and sustaining a lengthy ban on AI would likely require substantial cultural and institutional shifts away from these permissive and ambitious values.
To enforce a moratorium of this nature, societies would likely adopt a framework centered around caution, restriction, and a deep-seated aversion to risk—values that would contrast sharply with those that encourage creating sentient life and proliferating this life on as large of a scale as possible. Maintaining a strict stance on AI development might lead governments, educational institutions, and media to promote narratives emphasizing the potential dangers of sentience and AI experimentation, instilling an atmosphere of risk-aversion rather than curiosity, openness, and progress. Over time, these narratives could lead to a culture less inclined to support or value efforts to expand sentient life.
Even if the ban is at some point lifted, there's no guarantee that the conservative attitudes generated under the ban would entirely disappear, or that all relevant restrictions on artificial life would completely go away. Instead, it seems more likely that many of these risk-averse attitudes would remain even after the ban is formally lifted, given the initially long duration of the ban, and the type of culture the ban would inculcate.
In my view, this type of cultural conservatism seems likely to, in the long run, undermine the core aims of classical utilitarianism. A shift toward a society that is fearful or resistant to creating new forms of life may restrict humanity’s potential to realize a future that is not only technologically advanced but also rich in conscious, joyful beings. If we accept the idea of 'value lock-in'—the notion that the values and institutions we establish now may set a trajectory that lasts for billions of years—then cultivating a culture that emphasizes restriction and caution may have long-term effects that are difficult to reverse. Such a locked-in value system could close off paths to outcomes that are aligned with maximizing the proliferation of happy, meaningful lives.
Thus, if a moratorium on sentient AI were to shape society's cultural values in a way that leans toward caution and restriction, I think the enduring impact would likely contradict classical utilitarianism's ultimate goal: the maximal promotion and flourishing of sentient life. Rather than advancing a world with greater life, joy, and meaningful experiences, these shifts might result in a more closed-off, limited society, actively impeding efforts to create a future rich with diverse and conscious life forms.
(Note that I have talked mainly about these concerns from a classical utilitarian point of view, and a person-affecting point of view. However, I concede that a negative utilitarian or antinatalist would find it much easier to rationally justify a long moratorium on AI.
It is also important to note that my conclusion holds even if one does not accept the idea of a 'value lock-in'. In that case, longtermists should likely focus on the near-term impacts of their decisions, as the long-term impacts of their actions may be impossible to predict. And my main argument here is that the near term impacts of such a moratorium are likely to be harmful in a variety of ways.)
TL;DR
I think what is missing for this argument to go through is arguing that the costs in 2 are higher than the cost of mistreated Artificial Sentience.
TL;DR...
Restrictions to advanced AI would likely delay technological progress and potentially require a state of surveillance.
To be clear, I wasn't arguing against generic restrictions on advanced AIs. In fact, I advocated for restrictions, in the form of legal protections on AIs against abuse and suffering. In my comment, I was solely arguing against a lengthy moratorium, rather than arguing against more general legal rules and regulations.
Given my argument, I'd go further than saying that the relevant restrictions I was arguing against would "likely delay technological progress". They almost certainly would have that effect, since I was talking about a blanket moratorium, rather than more targeted or specific rules governing the development of AI (which I support).
I think what is missing for this argument to go through is arguing that the costs in 2 are higher than the cost of mistreated Artificial Sentience.
A major reason why I didn't give this argument was because I already conceded that we should have legal protections against mistreated Artificial Sentience. The relevant comparison is not between a scenario with no restrictions on mistreatment vs. restrictions that prevent against AI mistreatment, but rather between the moratorium discussed in the post vs. more narrowly scoped regulations that specifically protect AIs from mistreatment.
Let me put this another way. Let's say we were to impose a moratorium on advanced AI, for the reasons given in this post. The idea here is presumably that, during the moratorium, society will deliberate on what we should do with advanced AI. After this deliberation concludes, society will end the moratorium, and then implement whatever we decided on.
What types of things might we decide to do, while deliberating? A good guess is that, upon the conclusion of the moratorium, we could decide to implement strong legal protections against AI mistreatment. In that case, the result of the moratorium appears identical to the legal outcome that I had already advocated, except with one major difference: with the moratorium, we'd have spent a long time with no advanced AI.
It could well be the case that spending, say, 50 years with no advanced AI is always better than nothing—from a utilitarian point of view—because AIs might suffer on balance more than they are happy, even with strong legal protections. If that is the case, the correct conclusion to draw is that we should never build AI, not that we should spend 50 years deliberating. Since I didn't think this was the argument being presented, I didn't spend much time arguing against the premise supporting this conclusion.
Instead, I wanted to focus on the costs of delay and deliberation, which I think are quite massive and often overlooked. Given these costs, if the end result of the moratorium is that we merely end up with the same sorts of policies that we could have achieved without the delay, the moratorium seems flatly unjustified. If the result of the moratorium is that we end up with even worse policies, as a result of the cultural effects I talked about, then the moratorium is even less justified.
In your piece you focus on artificial sentience. But similar arguments would apply to somewhat broader categories.
Wellbeing
For example, you could expand it to creating entities that can have wellbeing (or negative elements of wellbeing) even if that wellbeing can be determined by things other than conscious experience. If there were ways of creating millions of beings with negative wellbeing, I'd be very disturbed by that regardless of whether it happened by suffering or some other means. I'm sympathetic to views where suffering is the only form of wellbeing, but am by no means sure they are the correct account of wellbeing, so maybe what I really care about is avoiding creating beings that can have (negative) wellbeing.
Interests
One could also go a step further. Wellbeing is a broad category for all kinds of things that count towards how well your life goes. But on many people's understandings, it might not capture everything about ill treatment. In particular, it might not capture everything to do with deontological wrongs and/or rights violations, which may involve wronging someone in a way that can't be made up for by improvements in wellbeing and can't be cashed out purely in terms of its negative effects on wellbeing. So it may be that creating beings with interests or morally relevant interests is the relevant category.
That said, note that these are both steps towards greater abstraction, so even if they better capture what we really care about, they might still lose out on the grounds of being less compelling, more open to interpretation, and harder to operationalise.
I wrote a post expressing my own opinions related to this, and citing a number of further posts also related to this. Hopefully those interested in the subject will find this a helpful resource for further reading: https://www.lesswrong.com/posts/NRZfxAJztvx2ES5LG/a-path-to-human-autonomy
In my opinion, we are going to need digital people in the long term in order for humanity to survive. Otherwise, we will be overtaken by AI, because substrate-independence and the self-improvement it enables are too powerful of boons to do without. But I definitely agree that it's something we shouldn't rush into, and should approach with great caution in order to avoid creating an imbalance of suffering.
Relating to @Toby_Ord 's comment on this post, I personally weight happiness and an interesting diversity of experiences and accomplishments a lot higher than I negatively weight suffering. I think worrying about suffering is overblown. If many people must suffer in order to strive for some great accomplishment, even if they don't know that they're contributing and won't live to see it come about, I still think their lives have not been in vain. Sure, I'd like to reduce suffering if there isn't a negative side-effect, like loss of ambition or creativity or meaningful diverse experiences, but I wouldn't elevate that to anywhere near the same importance as increasing interestingly diverse positive experiences.
I think worrying about suffering is overblown.
Can you say more about this? Sounds extremely morally unintuitive to me. Check out Brian Tomasik's work here: https://reducing-suffering.org/the-horror-of-suffering/
Seeing as no government has made this choice to ban AI sentience it is important to note that the creation of ai sentience is only looked down upon due to the scosiatal cultural believes that Sentient ai or rather ai with great efficiency would take over the world. However when looking at things from a different perspective this is only likely if we do not take the proper precautions eg.:
Lastly treat them with respect they also matter
The matter of fact is not banning it altogether but making sure to set rules and guidelines for the future. For the better of them humanity and our planet.
Also BTW as I understand sentience it is the practice of being able to feel and understand the world around you therefor ifithas feelings treat ithow you want to be treated as they say.
At present, there are no rules around the creation of artificial sentient beings. Anyone can create them, own them, make them do whatever they want, and treat them however they want to.
This is bad, and could lead to a lot of suffering for these beings.
The creation of artificial sentience ought not be left as an unregulated free-for-all. It ought to be regulated by governments, on behalf of society.
The moral stakes are so high that we should wait before creating artificial sentience, rather than rush into creating it, whether deliberately or accidentally. Very possibly, it may be best not to create it at all.
Responsible researchers and companies should not seek to deliberately create artificial sentience.
And governments should take steps now to prevent the creation of artificial sentience, at least in the short- and medium- term.
Even if this proves difficult or impossible, we should definitely ban the creation of artificial suffering.
In future, if we decide to permit the creation of artificial sentient beings, it should be carefully regulated, in order to protect the interests of these potentially vulnerable beings.
Much more work is needed to figure out exactly how to implement this approach, and make it work in practice. Readers of this piece should contribute to this process.
We will, collectively, as a society, need to figure out how to deal with the potential creation of artificial sentience. Through the political process, societies and their governments will need to decide whether they want to permit the creation of an entirely new class of sentient beings, and how they want to regulate this. We’ll have to answer questions such as:
It feels like our options fall into a few broad categories:
This post is an initial, incomplete attempt to start to flesh out some of these options, and to set out the pros and cons of each of them.
Context
There is massive uncertainty about whether artificial sentience is possible, and how we will know whether a given system is sentient. Much more research is needed into these foundational questions. But, we should also start thinking now about how to regulate. We can't wait around for all of the philosophical questions related to consciousness and sentience to be solved - if they ever will be. The rapid pace of AI development means that this is now a practical, objective issue in the world today, that we can and should engage on.
There are probably two broad types of actors who might create artificial sentience:
And we’ll have to think about two things:
(In addition to this, there are some areas of biological research that may end up creating novel forms of sentience. For example, brain organoids, Dishbrain. I won’t cover these in this piece, though some similar considerations may be relevant.)
This piece will avoid going into detail on the practicalities, science and philosophy of this field; it will just focus on some high-level considerations around the broad policy options for our approach to the issue.
We will now run through some of the broad main options for our approach to regulating the creation of artificial sentience, and examine their pros and cons.
There are two broad classes of things to think about here. Firstly, what are the ‘objective’ pros and cons of each option for regulation, as an end state - assuming we can actually get to that end-state. Secondly, what are the tactical pros and cons of arguing for, and lobbying for, each option. For example, we might think that voluntary codes of conduct are a sub-optimal option, but that they will be tactically easier to achieve in the short-term, so we should start by arguing for them.
We’ll start by looking at the ‘objective’ pros and cons of each option for regulation, and then, later in this piece, consider some thoughts around tactics.
Definition
There are no rules, regulations or laws around the creation of artificial sentience. Anyone who wants to try to create an artificial sentient being can go ahead and do so. They don’t have to apply for any kind of permission, or abide by any rules or guidelines. Anyone can do anything they want. Anyone can do any type of research, build any type of system, with any chance of it being sentient, with no rules around this.
Anyone who succeeds in creating an artificial sentient being - whether deliberately or not - can then treat them however they like. They can do whatever they want with them. They can make the artificial sentient beings do whatever they want. The people who create these beings will be the owners of the artificial sentient beings. The beings will be the property of their owners. The owners can cause them pain and suffering - whether deliberately, or inadvertently - with no consequences or restrictions.
This is the current state of affairs.
Pros of a free-for-all
Cons of free-for-all
Concluding judgment
The current status quo, of zero rule and regulations around the creation of artificial sentience, is terrible, and indefensible. It could lead to a moral catastrophe. It seems very bad to bring into the world a new category of sentient beings, in a totally unregulated way, which enjoy zero legal or other protection.
We should take action. The creation of artificial sentience ought to be regulated in some way.
Potential next actions
People who agree that the current unregulated free-for-all is bad, should engage on this issue, and help to flesh out, and deliver, options for regulation.
Definition
Companies and researchers might sign up to voluntary, self-imposed rules or ‘codes of conduct’ around the creation and treatment of artificial sentience.
Pros of voluntary codes-of-conduct.
Cons of voluntary codes-of-conduct.
Concluding judgment
This approach is probably a good place to start. Bostrom and Shulman (2023) give some thoughtful tactical reasons for wanting to start with this sort of approach towards this still-nascent field.
But, I claim that it’s unlikely to be where we want to end up. To guarantee the protection of the interests of artificial sentient beings, we will need much more than purely voluntary action by the most responsible actors. We will need proper regulation, with teeth.
Potential next actions
Definition
Mandatory, government-enforced rules which govern who may create sentient AIs, what types of beings can be created, how they may be treated, etc.
John Basl and Eric Schwitzgebel have argued for regulation - the creation of ‘oversight committees’ to decide what research should be permitted. Perhaps some version of this should be government-enforced.
An incomplete/partial initial list of pros and cons of this approach:
Pros of government-mandated regulation
Cons of government-mandated regulation
Potential next actions
Extensive further thought is needed on this, to work up concrete options and consider the pros and cons more deeply. The preliminary steps for such work should start now.
Definition
Thomas Metzinger has argued for a 50-year moratorium on the creation of artificial sentience, “strictly banning all research that directly aims at or knowingly risks the emergence of artificial consciousness”.
Pros of preventing the creation of artificial sentience
Cons of a ban
Concluding judgment
I find the idea of simply trying to prevent, or at least delay, the creation of artificial sentience very attractive, due to the force of the arguments in favor of buying us time and option value, and preventing astronomical suffering.
One's foundational ethical views, and views on population ethics, will likely play a significant role in shaping one’s opinion on this. Those who are more focused on minimizing suffering might view the prevention or delay of artificial sentience as a favorable option. In contrast, a classical utilitarian who believes that artificial sentient beings will, on balance, likely have lives worth living, and who believes we should create lots of these beings, might be more skeptical about efforts to prevent the creation of artificial sentience.
However, I argue that there are strong reasons to advocate for *delaying* the development of artificial sentience, even for those who are most excited about the potential of artificial sentience to lead to astronomical quantities of joy.
If we do end up creating artificial sentience, it seems really, really important to ensure that the development of it goes as well as possible, with the strongest possible ethical safeguards in place, for the benefit of human society, and for the artificial beings themselves. To achieve this, we should pause and draw breath, before rushing headlong into creating artificial sentience. At the very least, a period of pause, during which we can organize and prepare thoroughly, feels essential before we take this momentous moral step.
In the grand scheme of things, over the lifetime of the universe, a delay of a few decades or so is unlikely to significantly reduce total utility if it helps ensure that everything is done perfectly when we finally move forward. Plowing ahead with the current, unregulated free-for-all seems like a potential recipe for disaster and is certainly not the optimal way to approach this monumental development.
Thus, we should support, for the short- and medium- term, a ban on the creation of artificial sentience - at least until we figure out how to ensure everything goes well, if and when we do choose to create it.
Potential next actions
People interested in this should work up the details of how a ban would work, and campaign to get it enacted.
Definition
We could ban the creation of artificial suffering - ‘negative valence’ in artificial beings. It would be permitted to create artificial sentient beings with neutral or positive affect, but not permitted to create beings whose experience is dominated by suffering.
Bostron and Shulman (2023) propose that: “to the extent that we are able to make sense of a “zero point” on some morally relevant axis, such as hedonic well-being/reward, overall preference satisfaction, or level of flourishing/quality of life, digital minds and their environments should be designed in such a way that the minds spend an overwhelming portion of their subjective time above the zero point, and so as to avoid them spending any time far below the zero point.”
I suggest that we should look to go further than this, ensuring that no time is spent beyond the zero point. And we should have this actually mandated and enforced - rather than simply requested.
Pros
Cons
Concluding judgment
This feels pretty close to a no-brainer, to me. It seems clear that our default presumption should be to ban the creation of artificial suffering.
Potential next actions
We should work to bring about a ban on the creation of artificial suffering. There’s clearly a lot of work to do here, much of it quite fundamental and difficult. Interested actors should get to work on this.
As well as considering where we’d like to end up, in terms of regulation, we’ll also need to think tactically about the best practical course of action, in the world as it is today.
Tactical advantages of advocating for a ban on the creation of artificial sentience
There are some tactical advantages to making an ambitious, maximalist ask, such as for a ban on the creation of artificial sentience. These include:
Tactical disadvantages of advocating for a ban on the creation of artificial sentience
I think the principle that we ought not - at least not yet - to create artificial sentience, is a powerful and important one.
The arguments for it are strong.
It seems potentially achievable.
It passes a common-sense gut check. The idea that we ought not to create - or at least *rush* to create - artificial sentient beings, feels pretty normal and sensible, and not a particularly wild or out-there ask.
It may command wide support.
I think it’s valuable to start spreading this meme more widely, scrutinize and debate it, and building a field of engaged people working on it.
In terms of immediate next steps, I have sympathy for Bostrom and Shulman’s arguments in favor of taking a calm, thoughtful, non-sensationalising, non-adversarial, non-polarising approach towards talking about this issue. I agree with them that it’s probably too early for mass public outreach, or even, perhaps, lobbying of governments.
But I also think we can and should be a bit bolder than they propose. I don’t think we should just assume that it’s inevitable that artificial sentience will be created. We should, at the very least, debate and question this assumption. And we should be clear that purely voluntary codes of conduct are not acceptable as an end-state, and that a ban or delay should be firmly on the table as a plausible policy option for debate and consideration.
There is a huge amount of work to be done here. Specifically, potential next actions could include:
If you are reading this and are interested in this topic, and minded to undertake further research and action, please comment below, and/or direct message me; I'd be happy to link you up with other people who are interested in this topic.
I also wanted to add that as someone who leans more towards valuing happiness and suffering equally, that I still find the moratorium to be a good idea and don't feel pangs of frustration about the lost positive experience if we were to delay. I would be very concerned if humanity chose now to never produce any new kinds of beings who could feel, as that may forever rule out some of the best futures available to us. But as you say, a 50-year delay in order to make this monumentally importance choice properly would seem to be a wise and patient decision by humanity (given that we can see this is a crucial choice for which we are ill prepared). It is important that our rapid decision to avoid doing something before we know what we're doing doesn't build in a bias towards never doing it, so some care might need to be taken with the end condition for a moratorium. Having it simply expire after 20 to 50 years (but where a new one could be added if desired) seems pretty good in this regard.
I think that thoughtful people who lean towards classical utilitarianism should generally agree with this (i.e. I don't think this is based on my idiosyncrasies). To get it to turn out otherwise would require extreme moral certainty and/or a combination of the total view with impatience in the form of temporal discounting.
Note that I think it is important to avoid biasing a moratorium towards being permanent even for your 5th option (moratorium on creating beings that suffer). c.f. we have babies despite knowing that their lives will invariably include periods of suffering (because we believe that these will usually be outweighed by other periods of love and joy and comfort). And most people (including me) think that allowing this is a good thing and disallowing it would be disastrous. At the moment, we aren't in a good position to understand the balances of suffering and joy in artificial beings and I'd be inclined to say that a moratorium on creating artificial suffering is a good thing, but when we do understand how to measure this and to tip the scales heavily in favour of positive experience, then a continued ban may be terrible. (That said, we may work also work out how to ensure they have good experiences with zero suffering, in which case a permanent moratorium may well turn out to be a good thing.)
Given your statement that "a 50-year delay in order to make this monumentally importance choice properly would seem to be a wise and patient decision by humanity", I'm curious if you have any thoughts on the comment I just wrote, particularly the part arguing against a long moratorium on creating sentient AI, and how this can be perceived from a classical utilitarian perspective.