A note on AI, increasing/decreasing technological determinism, and wise institutions:

One really important facet of highly general advanced AI that I've been thinking about a lot lately is that it seems like, in general, AI will dramatically accelerate humanity's ability to choose its path. While human labor has a lot of inertia and resistance to being directed in certain directions, AGI could be directed toward whichever technology we decide is most desirable.

If things are sufficiently decentralized, or perhaps a better word would be anarchic, we might expect a race to the bottom. Competitive dynamics will incentivize a Molochian dynamic where we target the technologies that are most profitable, even when this produces technologies that are harmful or dangerous.

But if humanity is sufficiently coordinated or in some other way able to have wise collective governance/leadership/decision-making, then we could choose to develop the safest and most beneficial technologies first, building out the tech tree in whatever way is optimal for existential security and a flourishing future.

So I think path-dependence will be much more dependent on the way institutions are set up and how they make decisions around AI, such that we could either build a new dynamic that moves us toward a wisdom attractor, or on the other hand, if commercial or competitive dynamics are unleashed to their maximum, perhaps technological determinism could actually accelerate instead of decrease.

Anyone else ever feel a strong discordance between emotional response and cognitive worldview when it comes to EA issues?

Like emotionally I’m like “save the animals! All animals deserve love and protection and we should make sure they can all thrive and be happy with autonomy and evolve toward more intelligent species so we can live together in a diverse human animal utopia, yay big tent EA…”

But logically I’m like “AI and/or other exponential technologies are right around the corner and make animal issues completely immaterial. Anything that detracts from progress on that is a distraction and should be completely and deliberately ignored. Optimally we will build an AI or other system that determines maximum utility per unit of matter, possibly including agency as a factor and quite possibly not, so that we can tile the universe with sentient simulations of whatever the answer is.”

OR, a similar discordance between what was just described and the view that we should also co-optimize for agency, diversity of values and experience, fun, decentralization, etc., EVEN IF that means possibly locking in a state of ~99.9999+percent of possible utility unrealized.

Very frustrating, I usually try to push myself toward my rational conclusion of what is best with a wide girth for uncertainty and epistemic humility, but it feels depressing, painful, and self-de-humanizing to do so.

Quick link-post highlighting Toner quoting Postrel’s dynamist rules + her commentary. I really like the dynamist rules as a part of the vision of the AGI future we should aim for:

“Postrel does describe five characteristics of ‘dynamist rules’:

As an overview, dynamist rules:

1. Allow individuals (including groups of individuals) to act on their own knowledge.
2. Apply to simple, generic units and allow them to combine in many different ways.
3. Permit credible, understandable, enduring, and enforceable commitments.
4. Protect criticism, competition, and feedback.
5. Establish a framework within which people can create nested, competing frameworks of more specific rules.

I see some overlap with existing ideas in AI policy:

• Transparency, everyone’s favorite consensus recommendation, fits well into a dynamist worldview. It helps with Postrel’s #1 (giving individuals access to better information that they can act on as they choose), #3 (facilitating commitments), and #4 (facilitating criticism and feedback). Ditto whistleblower protections.
• Supporting the development of a third-party audit ecosystem also fits—it helps create and enforce credible commitments, per #3, and could be considered a kind of nestable framework, per #5.
• The value of open models in driving decentralized use, testing, and research is obvious through a dynamist lens, and jibes with #1 and #4. (I do think there should be some precautionary friction before releasing frontier models openly, but that’s a narrow exception to the broader value of open source AI resources.)

Another good bet is differential technological development, aka defensive accelerationism—proactively building technologies that help manage challenges posed by other technologies—though I can’t easily map it onto Postrel’s five characteristics. I’d be glad to hear readers’ ideas for other productive directions to push in.”

Opportunity Cost Ethics

“Every man is guilty of all the good he did not do.”

~Voltaire

Opportunity Cost Ethics is a term I invented to capture the ethical view that failing to do good ought to carry the same moral weight as doing harm.

You could say that in Opportunity Cost Ethics, sins of omission are equivalent to sins of commission.

In this view, if you walk by a child drowning in a pond, and there is zero cost to you to saving the child, it would be equally morally inexcusable for you not to save the child from drowning as it would be to take a non-drowning child and drown them in the pond.

Moreover, if you have good reasons to suspect there are massive opportunities to do good, such as a reasonable probability you could save millions of lives, Opportunity Cost Ethics states that the moral thing to do is to seek out such opportunities and find the best possible opportunities, after accounting for search costs.

Opportunity Cost Ethics is a consequentialist ethical theory, as it states the only morally relevant fact is the ultimate effect of our actions or non-actions.

The strongest version of Opportunity Cost Ethics, which I hold, states that:

"The best moral action at any given time requires considering as many actions as possible, until marginal search costs exceed the expected value of finding better options; at which point an optimal stopping point has been reached. Considerations of proximity also need to be taken into account."

We could call the above "Strong Opportunity Cost Ethics."

Possibly the most important implication of Opportunity Cost Ethics is that we should spend a massive amount of time, energy, and resources trying to determine what the best possible actions we can take are.

It is very difficult to know how much time we should spend trying to figure how good various actions are, and how certain we should be before we act. Opportunity Cost Ethics states that it is morally right to continue searching for the best action you can take until you predict the marginal cost of searching for a better action outweighs the amount of additional good that the better action would achieve.

Due to its obvious importance I assume something like this already exists, if so please point me to it, thank!

edit: Forum user tc pointed out that the “doing / allowing” distinction in ethics, whether causing harm is equivalent to preventing harm, which inspired trolley dilemmas, is highly relevant.

This is the kind of thing I was looking for! I am, however, uncertain if it fully captures what I am saying, which is more about the potential to do a massive amount of good, and the ethical responsibility to seek out such opportunities if we suspect they are available and the search costs do not cancel out the good.

At some point I hope to extend this short-form into a more well-formed robust theory.

The Proximity Principle

Excerpt from my upcoming book, "Ways To Save The World"

As was noted earlier, this does not mean we should necessarily pursue absolute perfect selflessness, if such a thing is even possible. We might conceive that this would include such activities as not taking any medicine so that those who are more sick can have it, not eating food and giving all of your food away to those who are starving, never sleeping but instead continually working for those who are less fortunate than yourself. As is obvious, all of these would lead imminently to death and so would ultimately not really be that good in the grand scheme of things.

Instead, the most effective way to be selfless is to be intelligently selfish in such a way that enables you to maximize your own capacity to serve others. This I call the “proximity principle,” which says that the most efficient way to do good for others is to prioritize that which is closest in proximity, both because by helping yourself and those closest to you, you are most likely to maximize your power to be effective in the future, and also because, all else being equal, it is simply easiest and most efficient to help those who are closest to you in time and space, rather than those who are on the opposite side of the world or billions of years in the future.

For example, it might be best to educate yourself, become financially independent, have a family and friends that meets your emotional needs, take care of yourself psychologically, and give these much higher prioritization than helping others, at least insofar as is practicable, since if you are falling apart in one or several of these ways, it might make it much more difficult for you to be effective in helping others. Once you have your own house in order, you can be exponentially more effective in helping the world at large and creating long-term impact.

For reference, I originally created this short-form so I could reference in an estimate of lives saved by longtermists. Here is the original context in which I mention it:

The Proximity Principle

The take-away from this post is not that you should agonize over the trillions of trillions of trillions of men, women, and children you are thoughtlessly murdering each time you splurge on a Starbucks pumpkin spice latte or watch cat videos on tik-tok — or in anyway whatsoever commit the ethical sin of making non-optimal use of your time.

The point of this post is not to create a longtermist “dead children currency” analogue. Instead it is meant to be motivating background information, giving us all the more good reason to be thoughtful about our self-care and productivity.

I call the principle of strategically caring for yourself and those closest to you “The Proximity Principle,” something I discovered after several failed attempts to be perfectly purely altruistic. It roughly states that:

1. It is easiest to affect those closest to you (in time, space, and relatedness)
2. Taking care of yourself and those closest to you is high leverage for multiplying your own effectiveness in the future

To account for proximity, perhaps in addition to conversion rates for time and money into lives saved, we also need conversion rates for time and money into increases in personal productivity, personal health & wellbeing, mental health, self-development, personal relationships, and EA community culture.

These things may be hard to quantify, but probably less hard than we think, and seem like fruitful research directions for social-science oriented EAs. I think these areas are highly valuable relative to time and money, even if only valued instrumentally.

In general, for those who feel compelled to over-work to a point that feels unhealthy, have had a tendency to burn out in the past, or think this may be a problem for them, I would suggest erring on the side of over-compensating.

This means finding self-care activities that make you feel happy, energized, refreshed, and a sense of existential hope — and, furthermore, doing these activities regularly, more than the minimum you feel you need to in order to work optimally.

I like to think if this as keeping my tank nearly full, rather than perpetually halfway full or nearly empty. From a systems theory perspective, you are creating a continuous inflow and keeping your energy stocks high, rather than waiting til they are fully depleted and panic mode alerts you to refill.

For me, daily meditation, daily exercise, healthy diet, and good sleep habits are most essential. But each person is different, so find what works for you.

Remember, if you want to change the future, you need to be at your best. You are your most valuable asset. Invest in yourself.

Highly Pessimistic to Pessimistic-Moderate Estimates of Lives Saved by X-Risk Work

This short-form supplements a post estimating how many lives x-risk work saves on average.

Following are four alternative pessimistic scenarios, two of which are highly pessimistic, and two of which fall between pessimistic and moderate.

Except where stated, each has the same assumptions as the original pessimistic estimate, and is adjusted from the baseline estimates of 10^16 lives possible and one life saved per hour of work or $100 donated.

1. It is 100% impossible to prevent existential risk, or it is 100% impossible to accurately predict what will reduce X-risk. In this case, we get an estimate that in expectation, on average, x-risk work may extremely pessimistically have zero positive impact and have the negative impact of wasting resources. I think it is somewhat unreasonable to conclude with absolutely certainty an existential catastrophe is inevitable or unpredictable, but others may disagree.
2. Humanity lasts as long as the typical mammalian species, ~1 million years. This would lead to three orders of magnitude reduction in expected value from the pessimistic estimate, giving an estimate that over the next 10,000 years, in expectation, on average, x-risk work will very pessimistically save one life for every 1,000 hours of work or every $100,000 donated. *Because humanity goes extinct in a relatively short amount of time in this scenario, x-risk work has not technically sustainably prevented existential risk, but this estimate has the benefit of using other species to give an outside view.
3. Digital minds are possible, but interstellar travel is impossible. This estimate is highly speculative. My understanding is that Bostrom estimated 15 additional orders of magnitude if digital minds are possible, given that we are able to inhabit other star systems. I have no idea if anything like this holds up if we only inhabit earth. But if it does, assuming a 1/10 chance digital minds are possible, the possibility of digital minds gives a 14 orders of magnitude increase from the original pessimistic estimate so that, over the next 10,000 years, in expectation, on average, x-risk work will moderately pessimistically save approximately one trillion lives per minute of work or per dollar donated.
4. Interstellar travel is possible, but digital minds are impossible. Nick Bostrom estimates that if emulations are not possible and so humans must remain in biological form, there could be 10^37 biological human lives at 100 years per life, or 21 orders of magnitude greater than the original pessimistic estimate. Assuming a 1/10 chance interstellar travel is possible, this adds 20 orders of magnitude so that, over the next 10,000 years, in expectation, on average, x-risk work will moderately pessimistically save approximately a billion billion (10^18) lives per minute of work or per dollar donated.

How Happy Could Future People Be?

This short-form is a minimally edited outtake from a piece estimating how many lives x-risk work saves on average. It conservatively estimates how much better future lives might be with pessimistic, moderate, and optimistic assumptions.

A large amount of text is in footnotes because it was highly peripheral to the original post.

Pessimistic

(human remain on earth, digital minds are impossible)

We could pessimistically assume that future lives can only be approximately as good as present lives for some unknown reasons concerning the limits of well-being, or well-being in the long-term future.

Moderate

(digital minds are possible)

It seems likely future lives will have significantly greater well-being or happiness than present lives, especially if brain emulations are possible.

If humans become digital people, we can edit our source code such that we are much happier. Additionally, by designing our digital environment we will be better able to achieve our desires, experience pleasure and avoid pain, and have the good things in life that support happiness.

Because this is speculative, let’s conservatively assume future people could be 10 times happier than present people, leading to one order of magnitude increase in expected value.^[1] I will come back to why I think this is conservative and a higher estimate in the optimistic section.

Optimistic

(digital minds are possible)

In the moderate estimate we assumed future lives would have ten times greater well-being or happiness than the average present human life. Yet there isn’t any reason in principle to assume future people couldn’t be many orders of magnitude happier.^[2]

In the moments before a seizure, Dostoyevsky reported feeling:

“A happiness unthinkable in the normal state and unimaginable for anyone who hasn’t experienced it… I am then in perfect harmony with myself and the entire universe”

And he wrote this experience into one of his characters, who stated,

“I would give my whole life for this one instant”

Taken literally, if we assume this experience lasted 10 seconds, that would make it over 100 million times better than his average experience.

I can say that I have personally experienced altered states of consciousness which were many orders of magnitude better than my average experience, and far better than even other very good experiences.^[3]

Most people have occasional peak experiences that are extremely good when compared with average experience. We may postulate maximum happiness is a Bostromian utopia of unimaginable ecstasy, (maximized hedonic well-being) or a Karnofskian future of maximum choice and freedom (“‘meta’ option”,) (maximized preference satisfaction), or perhaps gradients of bliss so that we can freely navigate choices while also being in unimaginable ecstasy.

In any case, humans are not currently optimized for happiness. We could optimistically estimate that if future minds were optimized for happiness there could be something like a minimum of one million times more happiness per unit of time than we currently experience, leading to 6 orders of magnitude increase in expected value.^[4]

1. ^{^}

   Because I am using a utilitarian expected value framework, I will assume that an individual being 10 times happier is equivalent to 10 times as many well-being adjusted lives;

   In other words,

   Average happiness * number of lives = well-being adjusted lives

2. ^{^}

   I will not give special attention to the impact on suffering in this analysis.

   Although I lean toward prioritarianism (utilitarianism which prioritizes reducing suffering over increasing positive wellbeing,) my naive intuition it that it is insufficiently likely there will be a large enough amount of suffering in the far-future for there to be much impact on expected value (which was the focus of the piece this short-form was originally embedded in.) Admittedly, I have not spent much time studying s-risks and would appreciate other perspectives.

3. ^{^}

   Additionally, I once briefly dated a woman and remember distinctly thinking to myself multiple times “I would trade 1 minute of kissing this woman for every other kiss, maybe even every other sexual experience I have ever had,” and this still feels accurate. I have repeatedly tried (unsuccessfully) to capture this experience in a poem.

   Assuming this intuition was accurate, if I had previously had an average of 12 minutes of kissing/sexual experience per day over about 2 aggregated years of dating by that point in my life, that 1 minute was about 10,000 times better than my average kissing/sexual experience - and these, in turn, were worth perhaps 100 average minutes in a day (I like kissing and sex quite a lot), making a minute of kissing this woman one million times better than my average experience, or about 3 years of my life. While this seems absurd, I doubt it is off by more than ~two orders of magnitude. 

^{^}

Concretely, this means that if the happiest minute of your life was worth 100 days of normal experience, this would be one order of magnitude greater happiness than that.

Such a state may seem difficult to maintain long-term. But considering we can already induce and sustain profoundly intense states of wellbeing over several hours with certain chemical substances, it seems plausible this could be the case.

This section is highly speculative, as it deals with aspects of phenomenology that we are not yet able to empirically test. It is possible that optimal experience requires ups and downs, complexity, moving from a worse past to a better future, or can’t be sustained indefinitely.

That said, this could also be underestimating the real possible happiness by many orders of magnitude. Complete scientific understanding of phenomenology and powerful consciousness engineering techniques could unlock levels of happiness we currently have no way to conceive.

Show all footnotes