I don't think there are any such guarantees? I think the point is not that ASI will definitely be rational, it is that ASI being rational and individually aligned is not a guarantee for good outcomes?

I guess it depends? I wouldn't think "AI welfare" as a whole would fall under cooperative AI, I think there's probably a lot of work there which is not about cooperative AI (and I don't think that is what you are asking but just to be really clear).

But cooperation between biological and digital minds (more or less cooperation between humans and AI?) seems clearly to fall under cooperative AI, with the caveat that the single-human-to-single-AI is a bit of a special case which is less central to cooperative AI.

It's a while since I wrote this now, and today I would probably put more emphasis on that cooperative AI is typically focused on mixed-motive settings where the entities involved are not fully aligned in their objectives but neither fully opposed/adversarial. So if you have mixed-motive settings involving both biological and digital minds, and especially if there are many of them, figuring out how to get good outcomes (rather than cooperation failures) in these settings seems like it would be a question for cooperative AI research.

Thanks - yes I agree, and study of collusion is often included into the scope of cooperative AI (e.g. methods for detecting and preventing collusion between AI models is among the priority areas of our current grant call at Cooperative AI Foundation).

Thank you Shaun!

I found myself wondering where we would fit AI Law / AI Policy into that model.

I would think policy work might be spread out over the landscape? As an example, if we think of policy work aiming to establishing the use of certain evaluations of systems, such evaluations could target different kinds of risk/qualities that would map to different parts of the diagram?

Interesting perspective!

I personally believe that many, if not most, of the world's most pressing problems are political problems, at least in part.

I agree! But if this is true, doesn't it seem very problematic if a movement that means to do the most good does not have tools for assessing political problems? I think you may be right that we are not great at that at the moment, but it seems... unambitious to just accept that?

I also think that many people in EA do work with political questions, and my guess would be that some do it very well - but that most of those do it in a full-time capacity that is something different from "citizen politics". Could it be than rather than EA being poorly suited to assessing political issues, EA does not (yet) have great tools for assessing part-time activism, which would be a much more narrow claim?

Thanks you for this comment - this is indeed very relevant context, much of which I was not previously aware of.

Thanks for commenting!

I think there are two different things to figure out: 1) should we engage with the situation at all? and 2) if we engage, what should we do/advocate for?
I might be wrong about this, but my perception so far is that many EAs based on some ITN reasoning answer the first question with a no, and then the second question becomes irrelevant. My main point here is that I think it is likely that the answer to the first question could be yes?
For this specific case I personally believe that a ceasefire would be more constructive than the alternative, but even if you disagree with that this would not automatically mean that the best thing is not to engage at all. Or do you think it does?

Strongly agree. Of course it's different what works for different people but I think it's a little odd that both EAG and EAGx seem to always be over the weekend, and I would be curious to see how the composition of attendees would shift if an event was held on work days.

Thanks, I'm glad you found it useful!

- Having spent a couple of months working on this topic, do you still think AI science capabilities are especially important to explore, cf AI in other contexts? I ask because I've been thinking and reading a lot about this recently, and I keep changing my mind about the answer.

Answering just for myself and not for the team: I don't have a confident answer to this. I have updated in the direction that capabilities for autonomous science work are more similar to general problem-solving capabilities than I thought previously. I think that means that these capabilities could be more likely to emerge from a powerful general model than from a narrow "science model".
Still, I think there is something specific about how the scientific process develops new knowledge and then builds on that, and how new findings can update the world-view in a way that might discredit a lot of the previous training data (or change how it's interpreted).

Interesting!

What is your assessment of current risk awareness among the researchers you work with (outside of survey responses), and their interest in such perspectives?