Thanks for this.

In each the examples you give, i'm thinking that the pause would be significantly more beneficial (plausibly by 10x) if we pause when AI is already capable enough that it can significantly help us solve the issue. In general, they seem like the kinds of issues where AI could massively accelerate progress.

So if i'm choosing between international pause now vs international pause in 2 years, I choose the latter. (I assume we're talking about international pauses here rather than just the U.S. but lmk if you also support a unilateral pause now!)

I do find Holly's point that it might be damaging to quibble about exactly when we pause if that reduces the chance of a pause happening at all. And today we are very far from a pause actually happening, and one may well be needed in two years' time, so I def support efforts to get us closer to a pause!

I'm hesitant about saying "pause now" because I actually think a different policy might be much more effective. But I think a world where we were about to do an international pause would be better than the actual world.

(I want to think more about this topic and all of this is v tentative.)

Thanks for the kind words, Rafael!

I can see how the island analogy is confusing:

• It suggests that the task of society is to reach some very particular kind of state(s), and otherwise it (presumably) flounders.
• One way that's inaccurate is that you can ask how well things are going on the ship, before it reaches the island, or after it misses the island.
• It's also unclear that the landscape of possible futures looks like a relatively discrete region of "success" and its complement, a region of failure.
• Finally, the value of the expedition, whether or not it reaches the island, might depend on the course which the ship took. In jargon, the value of society at a time might be stateful with respect to its history, or the value over time might not be time-separable. In words, it's about the journey…

So if that's what you had in mind, I agree.

And then as I read you, the worry is something like: "Better Futures argues that great futures are likely to be difficult to reach. So even if great futures are somehow many times better than mediocre futures, shouldn't we plan to make mediocre futures slightly better (or less bad), rather than throw a Hail Mary at the best futures?"

I wonder if there are two versions of this worry. The first might be: "the strategy which maximises the chance of passing some (or any) 'great' threshold of value is meaningfully worse, in expected value terms, than other strategies". In particular it could be that the max(p(great future)) strategy involves doing common-sensically bad stuff with a slim chance of paying off, and which otherwise does harm. For example, strategies which involve massively centralising power and then hoping that whoever holds all the power makes the right decisions.

I think some version of this worry is decently plausible. But I don't think anyone thinks that you absolutely should take whatever course maximises the chance of a great future. Rather, I think the rough idea in Better Futures is that, as a heuristic, it seems generally worthwhile to choose actions in light of whether they make great futures more likely. This is similar to Bostrom's "maxipok" principle, which Will and Guive Assadi argue against here. But both principles are derived from asking "what heuristic seems like it does a good job at approximating trying to take the max-EV option in a more granular way"; rather than suggesting a direct alternative. So the question is whether it's a good approximation of max-EV, rather than a good alternative to it.

The second version of the worry is something like: "on the Better Futures worldview, the max-EV strategy may well involve a Hail Mary with a slim chance of a great future, and otherwise a very likely outcome which is mediocre or bad. And this seems fanatical, or otherwise wrong."

Here again I would be pretty sympathetic, and it might be useful to distinguish which actions are EV maximising, from which actions are right. If you are a consequentialist (with caveats and asterisks) you think the right action is the EV-maximising one. But we don't try to argue that you should think right action is always EV-maximising and vice-versa (and IIRC there is a footnote trying to make this clear-ish). As with other cases of fanaticism, you could think that the plan which results in the most expected value is not the right plan!

Practically speaking, my hope is that aiming for truly great futures, and just trying to improve incrementally on 'default' futures, recommend quite similar and compatible courses. For example, it seems like power concentration looks pretty bad for both, in practice.

Thanks James!

What advice would have for someone who wanted to work there as a researcher?

Some things I appreciate in my colleagues: having some discernment for which questions or ideas are most important, rather than just conceptually interesting but not urgent; being able to contribute to group conversations by driving at cruxes, being willing to ask naive questions and avoiding the impulse to sound clever for the sake of it, and being able to spot and entertain "big if true" hypotheses; and being able to clearly communicate ideas where you often don't have an especially deep literature to draw on.

Do you think it's important to have a strong understanding of how LLMs work?

I think it's important to understand how AI works in the fundamentals; including some of the theory. I don't think it's important to have deep technical knowledge of LLMs, unless you can think why those details could end up being relevant for macrostrategy.

On your second question, many of those points seem good to me. I'll single out "Locking-in one’s values" since I've been thinking about it recently. It seems to me that some people roughly think that great futures are futures which resemble our own (or which carry on our values) in many particular ways. In particular, maybe great futures are futures which are recognisably human in their values. Inhuman futures, like futures where AI successors call the shots, might just seem empty of what we today care about; even if they involve a lot of moral reflection and nothing morally offensive from a human perspective. We could call this a "humanity forever" view.

On the other hand, some people roughly think that great futures are necessarily futures which are radically different from humanity today, including in the values which guide it, and perhaps the kind of actors living there. See Dan Faggella on the "Worthy Successor" idea (and here), which I see as one version of this view.

Both these views care about preventing obvious catastrophes from AGI, but it seems to me like they might end up disagreeing quite profoundly on what should come next. It's possible that there is opportunity for trade and compromise between the two views, but in any case this strikes me as a potentially important difference in approach to post-AGI futures.

To me, the ideas presented in the series (strong self-modification, modification of descendants, selection of beliefs by evolutionary pressures) indicate that we should expect future humans to be very different from us, and that, as a result, we should expect the future to be neutral in expectation.

Firstly, you're right that the series doesn't discuss negative futures, but I should say that's not because Will or I think they are worth ignoring, or very unlikely in absolute terms. We didn't discuss them more just so we could make a more focused argument about how to think about making good futures even better.

I think your point (quoted) touches on the difference I mentioned above between "humanity forever" views and views which are more open to change in values. I think it's coherent to take a view such as:

• You want to value whatever is ultimately valuable. You're unsure what that is, but you trust the processes which guide the future to converge on it;
• You want to value whatever you would value under some idealised process of reflection, and you think the processes which guide the future will emulate idealised reflection on your own values closely enough;
• You value roughly what you currently value. But you're scope-insensitive: in order to think we've reached a great future, you just need your neck of the woods to be how you want it, and the rest of the future to avoid things you think are morally repugnant. You expect almost the entire future not to be guided by what you value; but you're confident you can get the things you want to be satisfied the future is great, and you're confident the rest of the future will avoid the morally repugnant (perhaps through trade)
• Similar to above, but what you personally value is cheap by the lights of other value systems which guide the future, and vice-versa. So you are confident you can secure a great future by your lights through trade.

Better Futures argues that these views may be less tenable than they first appear, but I think they're not totally doomed.

Additionally I would point out a potential "missing mood" in the framing we adopt of cardinally quantifying the value of the future in terms of a fraction of the value of the best feasible future. This suggests futures which are only, say, 10E-5 the value of the best feasible future are barren, hollow, 'neutral'. But this would be a mistake: potentially our own world, even with all the harm and pain removed, is achieving a tiny fraction of what a great future could achieve. So we might imagine (as Better Futures points out) a "common-sense eutopia" which is radically better than the world today, but still only a fraction as good as things could get. That could be true, but it doesn't undermine the value of such a future, which would also truly be (by stipulation) wildly better than the world today! All the joy and freedom and discovery and so on, in this near-zero world, would be entirely real and could dwarf all the good we have achieved and enjoyed so far.

Currently, there are a wide range of ideas about how a post-AGI future will go and what features it will contain. To me, this strongly indicates that we should expect the post-AGI future could go in a very broad range of ways and that we should prepare for the many different ways it could go.

Maybe I'm misreading but I don't think it follows from uncertainty about how things go that many different things will actually happen. For example, if you're uncertain who wins a political election, you don't infer that everyone wins and shares power.

At the same time, I get the sense that Forethought has a very specific vision about how a post-AGI future will go (there will be an intelligence explosion, tools for epistemics will be beneficial, we might begin acquiring resources in other solar systems, small sets of actors could use AGI in malicious ways.) I'm wondering how do you decide what ideas you think are likely, and do you guys have any measures in place to ensure you're receiving criticism of your ideas so you don't create an epistemic bubble?

I'm in a few minds about this, so I'll just list some reactions:

• You say the Forethought vision is "very specific", and then you list some claims (e.g. "small sets of actors could use AGI in malicious ways") which seem… surprisingly anodyne? Or in particular it doesn't strike me as egregious or unusual to put a decent amount of credence in those claims being true. I think that's all you need to take them seriously and work on them. Indeed I don't myself feel extremely confident in any one of them.
• I think there is a way to do criticism in a performative way, where you invite people you know to disagree, for reasons you are already familiar with. I don't think that is totally useless, because performing these dialogues in public can be useful for other people to decide what they think.
• On the other hand, I think the best kind of outside criticism for the sake of throwing out bad ideas often isn't very flashy, and can look like outside experts telling you "this isn't really how [my domain of expertise works], so [ABC] seems confused but [XYZ] seems plausible".
• From my perspective there is quite a lot of internal disagreement, including between broad worldviews, although that's relative.
• Speaking personally, I worry a bit that there are components of the implicit shared Forethought worldview which are tricky to pin down from the outside, and thus more likely to influence research decisions in an unscrutinised way. I do think this is a generic problem, and think the most useful place from which to notice and communicate these implicit beliefs is straddling being enough of an insider to have context, and enough of an outsider to see alternatives.
• On the other hand I think you do at some point just need to pick some assumptions and some worldview and work within it to make any progress at all. In my experience simply pointing out that those assumptions could be wrong is often less valuable than proposing more fleshed-out alternative assumptions and worldviews, which themselves can be criticised and so on…

I'm wondering, if you think it's valuable for additional people to work in the field, why do you think this?

We are at Forethought, running a research programme on space right now, which I guess reflects a view that it does seem worth investigating more. I don't think the central case for space runs through the hope for binding international treaties because I agree that we shouldn't expect them to hold. I think there are a few other reasons to want to investigate space. One is that the space economy could be somewhat relevant for the course of AGI development, for example if orbital data centres are a big deal, or because of the role of sensing satellites in peace and security.

Another is that most of the physical stuff is in space. At some point it seems likely to me, if the human project continues at all, that most of the important stuff will also eventually be in space. AGI + automated manufacturing + rapid R&D progress suggests that expanding into space could happen in the time span of decades rather than centuries or millennia; and that seems generically worth planning for. And it seems like there are some policy levers which don't root through international treaties.

To be clear I don't currently think that space governance should be the next big cause in EA or anything like that.

It seems like longtermism is an unhelpful idea since it requires people to believe that our actions could persist for millions of years.

This feels like a slightly odd sentence construction, because you seem to be saying that longtermism is unhelpful because it requires people to believe one of its central claims. I agree it's contentious and I'm certainly not confident that the effects of our actions could persist for millions of years but it seems plausible enough that the anticipated long-term effects of our actions should meaningfully weigh into what we prioritise, at least where you can tell a story about how your decisions could have some systematic long-run effects.

It also seems like the idea has been somewhat harmful to EA as a movement since people can always point out that some of the founders of the movement are focused on helping people millions of years from now, which sounds pretty crazy.

I do think that is plausible. Although, to state the obvious, there is a difference between which ideas have good or bad PR effects when you say them out loud, and which ideas are actually true or important. So questions about communicating longtermist ideas are, naturally, different from the question of whether longtermist ideas are worth taking seriously as ideas.

And then, I also want to say: the full-on version of longtermism — that the very long-run effects of our actions are overwhelmingly important for what we prioritise — just doesn't feel especially necessary for working on most or even all of the topics that Forethought is focused on. There is a far more common-sense and mundane reason to focus on them, which is that they could matter enormously within our own lifetimes! Another way of putting that is that when trying to prioritise between possible focuses within Forethought, my personal view is that longtermism is rarely a crux. Maybe my colleagues disagree with that; obviously I'm not speaking on their behalf.

In "How To Make The Future Better," MacAskill argues that we should make AIs encourage humans to be good people and use them as a source of moral reflection. This seems like it could be deeply problematic in case moral sense theory is true, but AIs lack a moral sense. Do you agree with this?

I'm not sure I'm entirely following your points but I don't see a strong reason why AIs or non-human entities could not in principle engage in genuine moral reasoning in the same way that humans do. Maybe instead the AIs will do something which superficially resembles real moral reasoning, but which is closer to just telling humans what they want to hear.

I do think that is not a crazy thing to worry about because it is much easier to train some skill where an uncontroversial and abundant source of ground truth data exists. Moral reasoning is not one of those domains because people often don't agree on what good moral reasoning looks like. So I think there is much work to be done on that front although I'm not sure that answers your question.

Thanks again for your questions!