Really enjoyed this, great work! I get the impression medium and long timelines are significantly under-prioritized by the community, perhaps due to social/systemic biases.

I’m thrilled to see this as currently a leading forum debate week topic Toby Ord has suggested, I think this could be really high value for the community to collectively reckon with, and of course Toby’s work on the surprisingly bad scaling of chain of thought post-training (1 & 2) seems highly relevant here.

Hey again, sorry to spam you as I just commented on another piece of yours but am really vibing with your content!

I’m really hoping we can get something like this, I’ve been calling this “existential compromise.”

I worry that it may be difficult to get humanity to agree that we should use even a small fraction of future the resources optimally (see my research on this here), as I agree it seems like it will be a very weird^[1] thing that is optimal.

I think a compromise like this, with things split between optimal use and more (trans-) human friendly world makes a lot of sense and could perhaps be achieved if we can get society onto a good viatopian path, I’ve described what I hope that could look like here.

I would also say that I think Will Macaskill’s “grand bargain” and suggestions that we should try to aim for compromise in order to achieve “the good de dicto” feels to me like he actually is arguing that we need to aim for what is in fact best with significant fraction of future resources.

Nick Bostrom has also argued (page 14) that it should be possible for humans to get almost all of what they want and yet most resources could be optimized for “super-beneficiaries”.

Essentially I think we need to get whatever process^[2] is used to decide what we do with future resources to be extremely careful and thorough. I think we should target that process specifically, but it might also be good to broadly try to differentially accelerate society‘s wisdom (perhaps with AI).

Additionally, it could be really good to delay or slow things down so that we have more time to realize the existential stakes giving us the time that it takes to collectively mature and wisen. 

1. ^{^}

    Recently Will Macaskill’s research debated whether or not the best possible use of resources is something very extreme, calling this thesis “extremity of the best” (EOTB). I think this is very likely to be the case.

2. ^{^}

   I think this includes things like AGI/superintelligence governance, global constitutional conventions, and obviously any kind of long reflection  or coherent extrapolated volition that we collectively decide to undergo. 

Hi Peter, good to meet you! If you are interested in the long reflection you might  be interested in my research which I will link here which is on the broader class of interventions that the long reflection belongs, I really appreciate any feedback or comments on it.

Additionally, if this is something you’re interested in, you might be interested in this as a future forum debate topic. I raised it as a potential candidate here, I’m really hoping it gets enough initial upvotes to be a finalist candidate as I really think it’s an important crux for whether or not we achieve a highly valuable future!

Thanks for putting this out! 

I agree with quite a lot of this, however I think one of the most important points for whether or not we get something close to a near-best future is whether or not we have some kind of relatively comprehensive process for determining what humanity does with our future; what I’ve been calling “deep reflection,” which could be something like a long reflection or coherent extrapolated volition.

I think if such a process is used to determine how to use at least some percentage of future resources, then at least that percentage could be close to optimal value; if for some reason there is not a comprehensive process that determines any significant percentage of future resources, then I am quite pessimistic that we would get very far above 0% possible value, as getting high value seems quite difficult because it seems like there are many factors or crucial considerations (e.g. here and here) which could decrease future value, and like getting even a few of these factors wrong may mean we get close to zero value. 

I am also quite sympathetic to extremity of the best. My intuition is that it is very likely that optimal use of resources is far, far better than most other likely uses. Would be happy to expand more on this if there is interest.

That said, I think it’s important to recognize the importance of such a comprehensive process as an essential unifying strategic element; conditional on avoiding any existential catastrophes, it seems to me the vast majority of future value^[1] will be determined by two factors:

1. Whether or not we get a comprehensive reflection process at all
2. What percentage of future resources are used optimally according to the outcome of such a process

I think all of the other factors you mentioned are quite important, but I believe that they gain the vast majority of their value through how they influence the above two factors. Again, this is simply because (i) I believe we need such a process to determine how to get much above zero value with any degree of confidence, and (ii) it seems quite possible there will not be widespread agreement that we should use resources optimally according to the outcome of this process, but hopefully some grand bargain/existential compromise can be reached, or more optimistically perhaps we could try to get on a path where more convergence happens.

Would be curious to hear if you disagree with the previous paragraph, it feels like a pretty big crux for me.

1. ^{^}

   Rather than "the vast majority of future value," perhaps it would be slightly more accurate to say "the vast majority of positive impact we will have in expectation"; perhaps we are already predestined to achieve very high value or very low value, or more importantly, perhaps there are things we could do that seem like they will have a large impact (positive or negative); but unless our actions are leading to comprehensive strategic clarity (comprehensive reflection), and the likelihood of acting on this clarity (deep reflection), if seems hard to be confident ex ante our actions are highly likely to have highly positive impact in the final analysis.

And I might add – not just a deep understanding of how the world is, but of how the world could be:

• Large knowledge models for grounded, efficient information retrieval
• Decomposable tasks for superintelligent systems without superintelligent agents
• The potential for coordinated small models to outcompete large models on narrow tasks, making superintelligence potentially nearer but also safer
• Structured transparency enabling verifiable commitments and de-escalation of races
• Massively positive sum possibilities making coordination much more desirable

That is to say, I think Eric is a futurist in the best sense; he is someone who sees how the future could be and strives to paint a highly legible and compelling vision of this that at times can make it feel like it might just be inevitable, but at the very least, and perhaps more importantly, shows that it’s both possible and desirable.

Thanks for posting this Owen, couldn’t agree more! 

I often find myself referencing Eric’s work in specific contexts, in fact I just recommended it last night to someone working on AI control via task decomposition. I have been meaning to do a link-post on Why AI Systems Don’t Want Anything as soon as I get some free time, as it’s the biggest update I have had on AI existential risk since ChatGPT was released.

Eric has the keen ability to develop a unique, nuanced, first principles perspective. I agree his work is dense and I think this is one of its greatest virtues; when I recommend his blog I always have to comment in amazement that you can read the whole thing in an afternoon and come away with an entirely novel viewpoint on the world of AI.

This is a great overview of the virtues of his work, and the things his work leaves out. I especially like how you talk about deep study and the five steps you describe in order to internalize and reinvent. I think this also hints that what I see as Eric’s greatest strength; he looks at things very deeply in order to understand from first principles. I hope studying his work deeply in this way might help inspire others to develop similar first principles insight.

Max Tegmark explains it best I think. Very clear and compelling and you don’t need any technical background to understand what he’s saying.

I believe his third or maybe it was second appearance on Lex Fridman’s podcast where I first heard his strongest arguments, although those are quite long with extraneous content, here is a version that is just the arguments. His solutions are somewhat specific, but overall his explanation is very good I think:

Quick link-post highlighting Toner quoting Postrel’s dynamist rules + her commentary. I really like the dynamist rules as a part of the vision of the AGI future we should aim for:

“Postrel does describe five characteristics of ‘dynamist rules’:

As an overview, dynamist rules:

1. Allow individuals (including groups of individuals) to act on their own knowledge.
2. Apply to simple, generic units and allow them to combine in many different ways.
3. Permit credible, understandable, enduring, and enforceable commitments.
4. Protect criticism, competition, and feedback.
5. Establish a framework within which people can create nested, competing frameworks of more specific rules.

I see some overlap with existing ideas in AI policy:

• Transparency, everyone’s favorite consensus recommendation, fits well into a dynamist worldview. It helps with Postrel’s #1 (giving individuals access to better information that they can act on as they choose), #3 (facilitating commitments), and #4 (facilitating criticism and feedback). Ditto whistleblower protections.
• Supporting the development of a third-party audit ecosystem also fits—it helps create and enforce credible commitments, per #3, and could be considered a kind of nestable framework, per #5.
• The value of open models in driving decentralized use, testing, and research is obvious through a dynamist lens, and jibes with #1 and #4. (I do think there should be some precautionary friction before releasing frontier models openly, but that’s a narrow exception to the broader value of open source AI resources.)

Another good bet is differential technological development, aka defensive accelerationism—proactively building technologies that help manage challenges posed by other technologies—though I can’t easily map it onto Postrel’s five characteristics. I’d be glad to hear readers’ ideas for other productive directions to push in.”

Hey Will, very excited to see you posting more on viatopia, couldn't agree more that some conception of viatopia might be an ideal north star for navigating the intelligence explosion.

As crazy as this seems, I just last night wrote a draft of a piece on what I have been calling primary and secondary cruxes/crucial considerations, (in previous work I also used a perhaps even more closely related concept of “robust viatopia proxy targets”) which seems closely related to your "societal version of Rawls' primary goods," though I had not been previously aware of this work by Rawls. I continue to be quite literally shocked at the convergence of our research, in this case profoundly (if you happen to be as incredulous as I am, I do by chance have my work on this time-stamped through a few separate modalities I’d be happy to share.)

I believe figuring out primary goods and primary cruxes should be a key priority of macrostrategy research, we don't need to figure out everything, we just need to get the right processes and intermediate conditions in order to move us progressively in the right direction. 

I think what is ultimately most important is that we reach a state of what I have been calling “deep reflection”; a state in which we have both comprehensively reflected to determine how to achieve a high value future, and simultaneously are in such a state in which society is likely to act on that knowledge. This is not quite the same as viatopia, as it’s more of an end state that would occur right before we actualize our potential, hence I think it can act as another useful handle as the kind of thing we should hope viatopia is ultimately moving us toward.

I’m really looking forward to seeing more essays in your series!