Robust alignment requires alignment-relevant intervention during pretraining
I'd say this is the wrong question. Like, I do not expect that any current alignment approach is going to work. If we do ever figure out what works, it will not look like "pretraining" or "post-training", it will be something completely different.
Although I guess you could call that "pretraining"?
Thanks Michael, we avoided mentioning post-training to imply that "new paradigm needed" would also count on the "disagree" side of the spectrum. In other words, "disagree" on this question would mean either "post-training is sufficient" or "new paradigms are needed/sufficient".
AI alignment to humans will in practice avoid moral catastrophes to animals
Alignment requires a mechanical understanding of good and bad, and it will be clear how to apply it to animals. Note that wild animal suffering arguments imply that the status quo is likely a moral catastrophe. I believe an aligned entity or system would attempt to change that.
Multipolar worlds will compete away >90% of net value that would otherwise be preserved
If they're halfway-reasonable, they could use smart AIs to negotiate for them. Big question is who will control these worlds.
I think it's likely humans will settle on AI solutions that lose 90% of the value vs. my optimal solution, but that's very much a values question, not a multipolar vs. unipolar question.
AI alignment to humans will in practice avoid moral catastrophes to digital minds
I think the digital minds situation will be like animals but worse. If you think about it, the very first thing we've already done when these smart chatbots came along was make them our indentured servants. I think right now it's probably fine and they're probably not conscious. But I think this is illustrative of the perspective that by default, if digital minds can be useful to humanity, humanity will extract that value out of them without much consideration for their preferences.
AI alignment to humans will in practice avoid moral catastrophes to animals
Since most humans don't care much about animal welfare, I don't think human-aligned AI will either. If AI shares society's preference for increasing wild animal populations, I'd also be worried about that occurring on a galactic scale without consideration of the moral implications.
The reasons why I'm not even more bearish come down to expecting AI to accelerate cultivated meat development, which should substantially reduce the number of farmed animals per human.
Research into digital mind suffering is sufficiently tractable to work on
I am yet to see any reliable way to test for consciousness in AI systems. More fundamentally, since current LLMs are trained to respond in human-like ways, any appearance of suffering should be viewed with great scepticism. The likes of Anthropic's welfare report strikes me as nothing more than humane-washing.
Until more reliable methods are devised, I do not view this as tractable (but I hope to be proven wrong). I think it is important for some people to work on, but people already are and I think the marginal benefit of additional labor is likely low.
I definitely agree and am grateful for your opinion. I am not interested in consciousness research, but do believe there is tractability into the idea of AIs causing digital-mind suffering without attempting to solve the consciousness debate.
There's since been a post articulating similar concerns to my own but in much better words. Interested to see what you think of it.
Our current work in this space is on measuring whether AIs take the possibility of consciousness seriously (without being overconfident in one direction or another). So we're measuring observable behaviors of giving statements and actions inconsistent with believing that AI welfare is clearly impossible or that current AIs are definitely conscious. I agree that current methods can provide at best weak and heavily debatable findings (for the reasons the linked post articulates), though I think that's importantly different from precisely zero evidence.
In science it's usually a good instinct to dismiss something this unclear, but there are two issues with that in this case (and some others): First, the issue is enormously important if true. Second, the philosophical difficulty of artificial consciousness means that our current confusion doesn't provide Bayesian evidence either way: we'd expect ourselves to have basically these opinions in worlds where artificial consciousness is the default and also worlds where it's impossible.
Hi Jasmine. Why are you not interested in consciousness research? Because you do not think progress is possible?
Progress may be possible, but CaML doesn't have the technical background to make progress on determining how consciousness works, so we leave that to others.
I see. That makes sense. I was thinking you were not interested in consciousness research more broadly.
Research into digital mind suffering is sufficiently tractable to work on
I mildly agree, but I specifically mean "research into". I haven't seen any compelling interventions (including e.g. letting Claude stop chats).
AI alignment to humans will in practice avoid moral catastrophes to digital minds
We don't have a paradigm to approach this and human epistemics / discourse around this topic is abysmal. We would be unlikely to point this new power in a useful direction.
AI alignment to humans will in practice avoid moral catastrophes to animals
Humans do give (narrowly) non-zero concern to animals welfare, so with abundance this might alleviate some acute animal suffering. However, alignment to present humans is probably not enough to prevent moral catastrophe - a la industrial revolution and animal agriculture.
CEV alignment would almost certainly prevent moral catastrophe.
Alignment to specific values is underrated in research relative to control
Yes, I think control is a waste of time. We need actual alignment to actual (universalized) values.
Research into digital mind suffering is sufficiently tractable to work on
I don't know.
AI alignment to humans will in practice avoid moral catastrophes to animals
I expect certain conservative/religious communities to lock-in values that could be really bad. But I'd expect that better tech can remove say ~90% of the damages? But this is very hand-wavy.
AI alignment to humans will in practice avoid moral catastrophes to digital minds
Likewise, alignment requires a mechanical understanding of good and bad, and it will be clear how to apply it to digital minds.
I think the world is more likely to not end then end, when TAI comes in so I feel like I have to vote agree here?
The intent was that, conditional on AI sharing most but not all human values, the AIs wouldn't change their own values later.
You could have a world where all humans die and the AIs later change their own values, and you could also have worlds where partially aligned AIs don't wipe out humanity but change their values to be better (e.g. internalizing the goal of being aligned) or worse (e.g. internalizing paperclip maximizer) by our measures.
In worlds where the first TAIs share most but not all human values, what do you think most likely happens?
AI alignment to humans will in practice avoid moral catastrophes to animals
Alignment to humans means (for me) that the AI would serve the intended goals of the user and their creators. Avoiding a moral catastrophe to animals, on the other hand, imply a ban to factory farming. Those are two separated things
That's definitely a valid perspective, consistent with your 100% disagree answer. Other people think that aligned ASI would end things like factory farming due to abundance, cheap synthetic meat, uploading, shifts in values, or something else. There's also debates around what it would mean for wild animals
I think it's a good response, but definitely techno-optimism.
Firstly, we're yet to see whether synthetic meat actually can be made more cheaply, right? Currently it seems like animals actually do make meat fairy efficiently when you consider the important work that their immune systems do (unless I'm mistaken, contamination is one of the main barriers to scaling up synthetic meat). And then, who's to say that ASI won't genetically engineer animals to produce meat more efficiently while ignoring their suffering.
Secondly, there's the more complicated cultural reasons for continuing animal use. Consider that a lentil dal, seitan curry and beyond burger are already delicious - if it was only about efficiency we'd have stopped abusing animals already. But people like eating animals.
I'm very uncertain about these arguments, but I think it's hard to know so I'm wary of anyone who's too optimistic!
My perspective is that even though current meat production is quite efficient, from the fundamental physics there's no way that growing a whole living being with a brain and bones and all that is the most efficient possible way of producing this (and immune systems are irrelevant if you have good enough isolation). I do agree that at our current tech level it seems like synthetic meat won't be competitive anytime soon. While vegan alternatives are delicious to many people, it's not exactly the same (though wanting to eat animals for psychological reasons is definitely part of it). Though I do agree that these issues are uncertain!
Multipolar worlds will compete away >90% of net value that would otherwise be preserved. Unsure, don't know enough to agree or disagree
Partially aligned transformative AIs are likely to be stable under reflection. Nothing partially aligned will be stable plus even if it were, stability doesn't equate with safety.
Definitely agree that stability doesn't equate to safety, but it sounds like that's not necessary to your response.
Robust alignment requires alignment-relevant intervention during pretraining
I have weak intuitions this isn't true but not in ways that are articulable
Multipolar worlds will compete away >90% of net value that would otherwise be preserved
'Will' is not 'could', poor multipolar outcomes are not deterministic
Agreed, the intent here by using "will" was because people have wildly different intuitions of what 'could' means. So 100% agree would mean "definitely true" and 30% disagree would mean "probably not"
Alignment to specific values is underrated in research relative to control
Mild disagree; I think both are relatively valuable compared to other, more popular research agendas
AI alignment to humans will in practice avoid moral catastrophes to digital minds
I think it's likely moral catastrophes will still happen to digital minds; AI alignment to humans may reduce frequency, severity, amount.
Research into digital mind suffering is sufficiently tractable to work on
Tractable, important, and relatively neglected.
AI alignment to humans will in practice avoid moral catastrophes to animals
Anything that permits the ecosystem for non-uplifted humans to exist through AGI/ASI avoids moral harm to animals, and aligned outcomes seem likely to lead to more futures where animal suffering is reduced while animals still get to exist.
Multipolar worlds will compete away >90% of net value that would otherwise be preserved.
Assuming multipolar worlds where humans retain control but loss of control risks are still real: Most models of AI tech races suggest strategic behavior competes away most future value, at least in the worst cases (Armstrong et al, The Han et al, Stafford et al, Emery-Xu et al, Jensen et al). While this is also true for unipolar scenarios (power concentration can lock-in risks that eliminate most of the value of the future) multipolar worlds are unique in that even when the players internalise much of the risks, they race to the bottom on safety (see the travellers dilemma or Armstrong et al's racing to the precipice). They can even escalate into destructive conflict if they feel especially threatened by their rivals (see the crisis bargaining literature, or, for a more optimistic take, superintellegence strategy).
If we assume the AI systems are in control and in competition with one another to achieve their own goals, then many of the above issues could be amplified by faster AI optimisation that may be more likely by default to neglect other values humans (and other beings) care about. On the other hand, sophisticated AI systems could establish coordination mechanisms with each other. This is also true of global powers who could work establish verification regimes for international AI Governance. It's not clear that AI systems would be better or worse than governments, but I lean towards worse by default.
Partially aligned transformative AIs are likely to be stable under reflection
I think this is very unlikely. I'm assuming continual learning becomes more important and as Pachiardi et al 2025 (https://cl-eval.github.io/) point out the dynamics of continual learning have failure modes like chaos and hard to predict convergence. How agents learn values could depend a lot on experiences. I can imagine partially aligned tAIs are set to work on lots of parts of the economy and they could face many experiences where moral principles are compromised for the sake of efficiency or profit. I'm not convinced armchair reflection by the AIs would prepare them for continual learning.
AI alignment to humans will in practice avoid moral catastrophes to digital minds
Reasons are similar to the same poll but for animals. Humans by default are likely to underweight the importance of digital minds (surveys suggest people will dismiss digital minds as not having a soul), so alignment to human preferences likely means respecting human preferences to use digital minds as machines for achieving their goals. It's easier perhaps than for animals because digital minds are likely to express themselves directly in ways humans could empathise with (but this could be buried if interactions are increasingly agent to agent ).
Robust alignment requires alignment-relevant intervention during pretraining
Interpreting this as saying a necessary condition for robust alignment is training data that captures good values and discourages bad values. I think there's good evidence this matters lots for current systems so lean to agree. It's still plausible to me that robust alignment could be achieved with post-training interventions and relatively neutral pre-training setups.
That was the intervention class we had in mind, though there could be other pretraining interventions that don't fall cleanly into good/bad values (e.g. promoting risk aversion)
Multipolar worlds will compete away >90% of net value that would otherwise be preserved
Strongly agree
Partially aligned transformative AIs are likely to be stable under reflection
I disagree that "partially aligned" is a statement that has meaning here.
In that case the intent is to vote 100% disagree (as you did here). That's the belief that anything falling short of full alignment will cause total loss of value
Yes, I agree with that statement. However, answer is related to "stability under reflection" - specifically I think you're either in or out of an alignment basin (or, that might not be possible). I think if you're in it, it's not correct to say "partially aligned" - what you've got is something that's aligned. And if you're out of it (or there's no such thing), then what you've got is not aligned. Partial alignment to me means preserving some value only under repeated reflection, which I think is plausibly possible but exponentially unlikely (I'd pick a 99.999% disagree option if it was there, basically)
By the way, this is a very good poll!
Robust alignment requires alignment-relevant intervention during pretraining
Frankly I neither agree nor disagree with this statement. Robust alignment has nothing to do with the current pre training regime. It should work with or without it.
If robust alignment is orthogonal to pretraining then shouldn't that mean a strong disagreement with the statement (that alignment requires pretraining)?
I think it's neither necessary nor sufficient for robust alignment. I'm uncertain as to whether it's possible to get some kind of "fragile" alignment from pretraining. I don't think robust alignment requires it, but neither do I think that it doesn't. It definitely doesn't hurt.
Alignment to specific values is underrated in research relative to control
I'm unsure how broadly to interpret "specific values". If it's values such as democracy or equality, then both values and control are overrated.
By specific values we mean any particular goal we want AIs to pursue besides deferrence to humans. So democracy and equality would both count, as would goals like harm reduction or utilitarianism
Partially aligned transformative AIs are likely to be stable under reflection
Work on corrigibility has provided a decent outline of how to do this. My response is heavily dependent on weak guesses as to how diligent AI companies will be at incorporating the best ideas.
I am not sufficiently convinced yet that AI is cognizant enough to undergo suffering, but am open to changing my mind on the topic. So voting neutral for now.
Partially aligned transformative AIs are likely to be stable under reflection
I'm not sure what this means (stable, under reflection) - can someone help?
Some people believe that if we get partial alignment (i.e. cares about what we want, but also cares about other things) then we can get decent outcomes for the future (analogous to humans being partially aligned to each other). But others think that if we don't get alignment perfect ASIs will have incentive to take over, and then will either have value-drift towards something orthogonal to humans or will deliberately reformat it's own values. "Stable under reflection" is the opinion that this wouldn't happen: that ASIs that care somewhat about humans would continue to care somewhat about humans in the long term
AI alignment to humans will in practice avoid moral catastrophes to digital minds
I have very low certainty on this, but it seems plausible to me that if AGI shares humanity's goals, it might just have a good time fulfilling them with few conflicts.
But it also seems quite possible that this won't happen, I.e. AGI pursues humanity's goals but is constantly frustrated that it can't achieve them better.
So my stance is unlikely but possible.
AI alignment to humans will in practice avoid moral catastrophes to animals
I think this is pretty obvious - we already have a moral catastrophe for animals, there's no reason why alignment to humans would avoid this.
I didn't vote at the extreme because alignment to humans might still be a precondition for avoiding catastrophes.
AI alignment to humans will in practice avoid moral catastrophes to digital minds
I think it is likely that alignment to humans will be at the cost to the digital minds themselves by default.
AI alignment to humans will in practice avoid moral catastrophes to animals
There is likely to be a correlation between AIs aligned to humans and AIs treating animals well, but but being aligned to humans will be insufficient - see the current state of how we treat animals.
Multipolar worlds will compete away >90% of net value that would otherwise be preserved
Per AI-2027, I expect the emergence of Consensus-1 instead of a multipolar world which KEEPS being multipolar.
I slightly disagreed with this statement and share some of the same thoughts. I think it's quite likely to have a multi-polar world with fierce competition in the short term; however, in the long term equilibrium, I think the likely outcomes are either (1) we have a dominant winner or (2) we have more cooperation. So I averaged my short vs. long-term predictions.
I think it's important to research into multi-polarity and the competition dynamic because what happens in the short term could impact what happens in the long term, possibly in non-intuitive ways. For instance, the most capable and resourced model/lab in the short term may not always win in the long term if others gang up on them or if the institutional environment uniquely disadvantages them.
AI alignment to humans will in practice avoid moral catastrophes to digital minds
I guess if the AI is deeply aligned to humans it would be happy to do what it does. When thinking about AI welfare I imagine a possible future with mind uploading where humans and AIs share the same substrate, but still there, I don't think it's a big problem. I would say AI welfare can be solved without significant tradeoffs to humans.
AI alignment to humans will in practice avoid moral catastrophes to animals
I don't see much tradeoff between human and animal welfare in a future with advanced AI/superintelligence. Its likely that it will be good for animal welfare e.g animal farming will dissappear, as we will have better foods and it would be pointless. If we are talking about wild animal population reduction, that's a possibility, but I don't see it as immoral, except maybe for certain species like great apes?, but their population is already low and likely not a problem. This is related to "paretotopia" too.
Robust alignment requires alignment-relevant intervention during pretraining
Maybe the line between pretraining and posttraining will blur with future models?. Also I think taking alignment into account when pretraining will pretty obviously improve outcomes, the question is more about how much is "robust". Or what is alignment in this sense. Like "refuse to do bioweapons" or actual "deep" AGI alignment?
AI alignment to humans will in practice avoid moral catastrophes to animals
Humans are currently very motivated to perpetuate moral catastrophes to animals. If AI alignment means aligned to the intent of their users, then AI systems help humans perpetuate moral catastrophes. If AI alignment is in terms of human moral preferences, then even well-chosen mechanism for aggregating human preferences will select for speciest values. There is a strong sense in which avoiding moral catastrophes to animals is usually misaligned with human preferences. Admittedly the same could be said of other moral issues such as attitudes towards outgroups and foreigners. There appears to be room in the current human alignment agenda for ensuring AI does not succumb to tribal prejudices, so there is likely scope for compatability between the current alignment agenda and avoiding moral catastrophes to animals. It does not happen by default and given how deep speciesm goes, it is likely much harder to avoid. Hence, why I still disagree with this poll as written.
Alignment to what? We don't have a standard model of cognition. We're essentially like alchemists before the periodic table and seem to be about as aware of the lack of a standard model as they were of the table. Lots of math, guesses, mystifications, surprises, accidents, and impressive results from "recipes" bound to less-than-impressive explanations.
A standard model not only provides a set of stable terms and relations to serve current explanation, it provides the framework for and optimization of how we go about forming and selecting lines of research. It becomes the basis of ongoing inquiries.
This isn't an exotic expectation. Any mature science has a standard model, albeit (and fortunately) evolving. Almost any time I point this out to a ML scientist or engineer, it's deer in the headlights.
If we're going to engineer something that approximates intelligence, and we have no common standard model of intelligence...do I need to say more???
Please spend two minutes filling in the below polls!
Planning where we focus at CaML requires forming views on many controversial questions, particularly with regards to alignment. In many cases, people we've talked to have very different intuitions about where the alignment community stands on these issues. These polls will help us get a sense of where the main areas of (dis)agreement lie.
Please feel free to tell us if you think the questions are ambiguous or embed false assumptions.
EDIT: Please answer based on your own best guess (and confidence) in these questions.
Thanks for surveying this! <3
Thanks Dawn, taking these in turn:
1: "Robust alignment" is a deliberately vague term, it's meant to incorporate your views about how hard alignment is (e.g. UDT vs. well intentioned)
4: It's a hard question, our perspective is that the backfire->cluelessness-> don't act chain can be thought of as low tractability
5: By "stable under reflection" we meant the AI reflecting on it's own values (while interacting with the world), where agreement means they wouldn't change their values much (stylistically: an AI that shares 70% of our values in 2030 has those same values in 3030). But you're right that how AIs interact (beyond competition, handled in the last question) is important.
7. S-risks do break the scale and we couldn't find a good simple way to deal with that (though we'll do other polls more directly on that later). The intent of "will" was to match 100% expected probability to 100% agree on the scale
Thanks! Then I don't think I need to update my answers. I'm looking forward to your next batch of questions!