Executive summary: The characteristics of Space-Faring Civilization (SFC) Shapers are likely constrained by evolutionary dynamics, almost winner-takes-all races, and universal selection pressures, which may imply that different SFCs across civilizations will have similar values and capabilities. If true, this could challenge the prioritization of extinction risk reduction in longtermist strategy, as the expected utility of alien SFCs may not be significantly different from humanity’s SFC.
Key points:
SFC Shapers as constrained agents – The values and capabilities of SFC Shapers (key influencers of an SFC) may be significantly constrained by evolutionary selection, competition, and universal pressures, challenging the assumption of wide moral variation among civilizations.
Sequence of almost winner-takes-all races – The formation of an SFC is shaped by a sequence of competitive filters, including biochemistry, planetary environment, species dominance, political systems, economic structures, and AI influence, each narrowing the characteristics of SFC Shapers.
Convergent evolution and economic pressures – Both genetic and cultural evolution, along with economic and game-theoretic constraints, may lead to similar cognitive abilities, moral frameworks, and societal structures among different civilizations' SFC Shapers.
Implications for the Civ-Similarity Hypothesis – If SFC Shapers across civilizations are similar, the expected utility of humanity’s SFC may not be significantly different from those of other civilizations, reducing the relative value of extinction risk reduction.
Uncertainty as a key factor – Given the difficulty of predicting the long-term value output of civilizations, longtermists should default to the Mediocrity Principle unless strong evidence suggests humanity’s SFC is highly exceptional.
Filtering through existential risks – Various bottlenecks, such as intelligence erosion, economic collapse, and self-destruction risks, may further shape the space of possible SFC Shapers, reinforcing selection pressures that favor robust and similar civilizations.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
The Convergent Path to the Stars - Similar Utility Across Civilizations Challenges Extinction Prioritization — EA Forum
The Convergent Path to the Stars - Similar Utility Across Civilizations Challenges Extinction Prioritization
This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?
Summary
Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.
Public Opinion...
This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...
Why building and backing Welfare Tech companies may be one of the most promising things we can do for billions of animals.
I used AI to assist in writing this post, but I’ve rewritten it extensively and endorse it.
* Announcing the launch of Spring Innovation Fund, a not-for-profit venture philanthropy studio and fund built specifical...
This document is a shallow review of selection pressures and filters influencing the characteristics of the Shapers of Space-Faring Civilizations (SFCs). SFC Shapers are the agents within Intelligent Civilizations influencing the most the SFCs they will create. We introduce a sequence of almost winner-takes-all races, evolutionary dynamics, and a number of filters, which together should make us update towards a more constrained space of possible SFC Shapers’ values and capabilities. We care about SFC Shapers because they are one of the main inputs necessary to evaluate the Civ-Similarity Hypothesis[1], which could have macrostrategic implications for impartial longtermists such as reducing the value of Extinction-Risk reduction (but not reducing the value of Alignment-Risk reduction).
Sequence: This post is part 6 of a sequence investigating the longtermist implications of alien Space-Faring Civilizations. Each post aims to be standalone. You can find an introduction to the sequence in the following post.
Bridging to Extinction-Risk reduction
In a previous post, we evaluated the Civ-Saturation Hypothesis[2] and concluded that if humanity does not create an SFC, 84+% of the resource we would have grabbed would not be lost, they would be grabbed by other SFCs. This estimate is our best estimate when assuming the weakest form of EDT[3].
At the end, we are interested in implications about the relative priorities of Extinction Reduction[4] and Alignment (improving the “Value of the future” conditional on no extinction). The relative values of these two depends on how much value would be lost if humanity’s SFC does not exist. We already found that 84+% of our resources would NOT be lost. In this post, we start investigating whether these 84+% salvageable by other SFCs would be used to produce significantly different utility than if they were used by humanity’s SFC. If they don't produce significantly different value, then these 84+% of resources would not be an impartial loss, and that could have a significant impact on longtermist priorities; we will review these impacts in a later post.
To study if humanity's SFC would produce more or less value than other SFCs, we focus on the characteristics of SFC Shapers, because this is where we could directly observe whether humanity is abnormal among Intelligent Civilizations (ICs). This post exposes a short case for significant similarity between SFC Shapers. Unfortunately, this post does NOT review arguments against this claim.
Uncertainty as a crucial consideration.
We don’t develop that much in this post, but we are massively uncertain about the value humanity’s and alien SFCs would produce. We can only observe contemporary humanity and our capability at predicting the value it will produce is minimum[5]. Given we are extremely uncertain about how the characteristics of an SFC ancestor impact the value produced by the SFC, we would need to observe that humanity is somewhat strongly abnormal to be able to conclude that humanity’s SFC would produce a significantly different expected utility than other SFCs. Stated another way, we start with the Mediocrity Principle: if we have no information to distinguish humanity's SFC and alien SFCs then we should assume they produce similar value. Then, given how uncertain our model of value production in the far future is, we are going to need to observe humanity is strongly abnormal to update significantly away from the Mediocrity Principle prior.
Summary of the post
The post examines plausible constraints on the Shapers of Space-Faring Civilizations (SFCs) when weighting them by their decision-relevance (~ how much resources their descendants will grab). We first describe a sequence of almost winner-takes-all races that progressively filter and constrain the characteristics of the SFC Shapers, starting from fundamental biochemistry through substrate environments, species dominance, geographic advantages, macro-level policies, economic competition, and finally individual influence over ASIs. The post then suggests that convergent evolution, both genetic and cultural, may produce additional constraints through the existence of universal causes behind moral values, norms, and cognitive and societal capabilities, as studied in evolutionary psychology, cultural evolution, and economics. Finally, these constraints are complemented by three categories of bottlenecks that could further restrict the space of important SFC Shapers: civilization-destroying factors, economic impediments, and intelligence-eroding mechanisms.
As long as we focus on decision-relevance-weighted alien SFC Shapers, this overall filtering may imply they could have more predictable and constrained characteristics than previously assumed in discussions of alien values. This has significant implications for evaluating the Civ-Similarity Hypothesis[1], which we will review in another post.
Clarifying claims
This post doesn't claim that alien ancestors of SFC will have our identical moral values and capabilities. It claims that they are more constrained than we may think, and that after taking into account our massive uncertainty, impartial longtermists are unable, in expectation, to update much away from the prior given by the Mediocrity Principle: the expected utilities of humanity's and alien's SFCs are similar for impartial longtermists.
Introduction
We want, at the end, to evaluate the Civ-Similarity Hypothesis, which posits that the expected utility[6] of humanity's future Space-Faring Civilization (SFC) is similar to that of other SFCs. To evaluate this hypothesis, one strategy is to look at the characteristics of SFC Shapers, to compare them to characteristics of the expected SFC Shapers within humanity, and then to look at whether these expected differences translate into different longterm expected values, produced by the SFCs they would respectively create.
In this post, we describe a number of mechanisms that could significantly constrain SFC Shapers characteristics. When aliens’ values are discussed in the rationalist community, we can often find arguments along the lines of moral values and capabilities are orthogonal, and the space of possible moral values is huge, thus aliens will have alien moral values that would make no sense to us. See a few quotes in footnotes[7][8][9].
This post focuses on a much more limited group of aliens: decision-relevance-weighted SFC Shapers. We explore how much smaller the space of values and capabilities could be when both conditioning on SFC Shapers and weighting them by their decision-relevance (in short, how many resources their descendants will grab).
What do we mean by SFC Shapers? Space-Faring Civilization (SFC) Shapers are defined as the agents dominantly shaping the impact of the SFC their Intelligent Civilization (IC) will create. More specifically, we define SFC Shapers as the minimum group of intelligent agents within an IC whose characteristics are enough to get 80% of the predictive power one would get by using the characteristics of the whole IC when predicting the impact of the SFC it will create. Intuitively, SFC Shapers on Earth may be the top 10 most influential humans in AI (centralized scenario) or the top 10,000 (decentralized scenario).
A sequence of almost winner-takes-all races
We defined SFC Shapers as the group of individuals that concentrate most of the impact an Intelligent Civilization will have on the SFC it creates. We claim SFC Shapers are the winners of a sequence of almost winner-takes-all races happening over many OOMs in space and time. This temporal sequence of races contributes to narrowing down the characteristics SFC Shapers can have.
How much each of these races has a winner-takes-all nature varies severely. Some races could be strong filters, only keeping in the leading competitor, winner-takes-all style. Other races could be significantly milder and only filter out a significant fraction (e.g., ¾, ½, ¼) of competitors. The purpose of this section is not to estimate the impact of these races, nor to describe them accurately, but to introduce them and highlight their existence.
Inter-life forms
We claim that most resources will be grabbed by a unique life form, the fastest at generating SFCs, and we can likely describe it.
Competitors: Life forms distinguished by their core biochemistry (e.g., carbon + water, carbon + alternative solvents, silicon-based, etc.).
Victory condition: Being the fastest at generating SFCs and thus securing a significant fraction of cosmic resources.
Deadline to finish the race: Preclusion by other life forms, availability of resources in space.
Likely winner: Carbon + water-based life.
Likely challenger: E.g., carbon + another solvent[10].
Early SFCs preclude or at least constrain later potential SFCs in their sphere of influence.
Humanity having a significant causal or evidential impact.
Early appearance time and high appearance frequency are correlated, and the first life-form to appear will do so OOMs earlier and much more frequently than the later ones, thus dominating how resources are used. The earlier life forms to appear in the universe will have more time to preclude later life forms. The life forms able to create SFCs on many substrates in a shorter time will grab more resources. For a given life form, the time to create SFCs and the density of appearance are correlated because we have some reasons to think some steps leading to the emergence of an SFC are time-dependent steps. Given our best knowledge, our estimates of the rate at which a life form generates a Space-Faring Civilization (SFC) has an uncertainty over many orders of magnitude (e.g., over 40 to 100 OOMs in Dissolving the Fermi Paradox), and such estimates are better modeled using log scale uncertainty and priors because of being composed of many multiplicative factors. We should thus expect the dominant life form to appear first and produce OOMs more, as well as larger SFCs, than other life forms. Overall, it will likely grab most resources.
This reasoning assumes a strong preclusion between SFCs and thus the potential existence of many SFCs in the same reachable space. Even if earlier SFCs don’t preclude later ones by choice, it is still likely they will strongly shape or constrain which values the later SFCs will be able to pursue.
A few other facts: Carbon is the third most abundant element in the universe. Water (Oxygen+Hydrogen) is composed of the first and fourth most abundant. Carbon is very versatile, forming stable enough bonds, but not too stable. Water has great solvent properties favorable to chemical reactions and is significantly stability. We have some evidence of pseudo-panspermia for our DNA[11][12].
We claim that most resources will be grabbed by life coming from a small distribution of substrates, the fastest at generating SFCs, that these substrates have a significantly narrower distribution than those able to support simple life-forms, and that we can likely describe this substrate, at least in broad strokes.
Victory condition: Securing a significant fraction of cosmic resources.
Deadline to finish the race: Preclusion by other life forms, availability of resources in space.
Likely winners: Planets with high continuous Planetary Habitability over billions of years with stable instability
Likely challengers: E.g., planets with high but continuous Planetary Habitabilitywith some divergent features, for example, with another solvent than water
Early SFCs preclude or at least constrain later potential SFCs in their sphere of influence.
Humanity having a significant causal or evidential impact.
Planet Habitability and Rare Earth. Given the life form winning the previous race, carbon + water, and its challengers, the substrates on which they will rapidly lead to the appearance of SFCs are limited. These limits are often referenced as Planetary Habitability and were popularized by the Rare Earth hypothesis. The Rare Earth Hypothesis argues that the conditions necessary for complex life and intelligent civilizations are exceptionally uncommon in the universe, severely restricting the frequency of suitable habitats. Multiple factors constrain where complex life might emerge: planetary size and mass (adequate to retain atmosphere while permitting emerged land mass), tectonic activity (regulating carbon cycles and nutrient distribution), atmospheric composition, orbital stability, position within a galactic habitable zone away from excessive radiation, and the presence of a gas giant like Jupiter to shield from asteroid impacts. This hypothesis describes a number of substrate constraints, suggesting that even among carbon-water-based life forms, the specific planetary conditions capable of rapidly producing Space-Faring Civilizations might indeed be narrowly constrained. Additional constraints not yet mentioned include: planetary rotation, axial tilt, orbital eccentricity, stable instability (e.g., Milankovitch cycles) and specific chemical abundances such as phosphorus availability.
Let's now further illustrate these constraints by talking speculatively about the optimal[13] size of planets.
Too small is too solid a crust, too unstable a climate, too thin an atmosphere, too weak a shield, etc. The smaller the surface of a planet, the lower the number of ecological niches, the less species would differentiate and the less likely an intelligent species would emerge (speculative). The smaller the planet, the smaller the solar energy flux (small effect), which may be a bottleneck for the total biomass a planet can support. If a planet is too small, its core cools too fast, and this prevents tectonic plates, which are likely useful to break local evolutionary minimums, bring nutrients to oceans, create a stabilizing climate effect through silicate weathering, and release warming gases. The smaller a planet, the more it loses its atmosphere (extreme simplification), plausibly with different thresholds leading to different atmosphere compositions. Smaller planets may lack protective magnetic fields strong enough. Etc...
Too large is too soft a crust, too stable a climate, too extreme winds, too opaque an atmosphere, too small animals, etc. The larger a planet, the longer it takes for the crust to cool enough for tectonic plates to be able to crack and form (stronger gravity messing this simplification). The larger a planet, the more atmosphere is kept up to the point at which the atmosphere is opaque to light (also influenced by atmospheric composition), and thermal gradients in the atmosphere cause extreme storms. The composition of the atmosphere is also dependent on the size of the planet; large enough planets will have atmospheres mostly composed of hydrogen and helium. A much more massive atmosphere to planet surface ratio may also delay life appearance because of amplifying delay steps in which the composition of the atmosphere needs to be changed for complex life to appear (e.g., the Great Oxygenation event on Earth). A gravity too high may also prevent complex and intelligent life forms by preventing the existence of large enough animals (strength scales sublinearly with body size). Etc...
We claim that at the time an SFC is created, only one intelligent species will be dominating their planet, likely the first one to appear, and that intelligent species will either create an SFC as soon as they can or go extinct quickly on genetic evolution scales. In the end, we can likely describe some of their characteristics.
Competitors: Species on a planet
Victory condition: Creating an SFC
Deadline to finish the race: Preclusion by other intelligent species or (mostly) endogenous extinction.
Likely winners: All the constraints on complex life + Brain power similar to or higher than humans, part of a genus and order with high intellectual capabilities, capable of cultural accumulation, land-based, predators, abnormally good at learning from each other, high brain-mass to body-mass ratio, etc.
Likely challengers: All the constraints on complex life + Brain power similar to or higher than humans (larger range), part of a genus and order with high intellectual capabilities, capable of cultural accumulation, etc.
Some assumptions:
Intelligent species colonize the most productive niches of their planet and preclude later intelligent species.
Intelligent species quickly try to create an SFC or go extinct.
Humanity having a significant causal or evidential impact.
Earth-analogy: Homo Sapiens won the race against other Homo species and against other species on Earth.[14][15]
Complex life is heavily constrained. As illustrated in The Zoologist's Guide to the Galaxy, we can describe many constraints on the functions that complex life will have developed. These constraints emerge from universal selection pressures such as ecological competition, predator-prey dynamics, and climate variability. With more uncertainty, we can also study the solutions of life to these necessary functions, "implementations". For example, movement, sensor, and communication strategies are heavily constrained by environmental factors, such as medium density, viscosity, borders between mediums, wave propagation and diffusion, sensory resolution and bandwidth.
SFCs are created as soon as possible by their ICs if they survive. We claim that the first species to create an SFC on a planet will be among the first species to have the minimum requirements necessary to create an SFC, and it will spread through many ecological niches, becoming an apex predator and dominating macro energy and resource usage.
Natural laws are caused by universal constraints, and general laws constraining the characteristics of ICs likely exist. We can speculatively list characteristics that contribute to winning this race: High evolvability, high encephalization quotient, large absolute brain size, which implies a large body size, which implies a range of maximum movement speed, other characteristics are prehensile, terrestrial, social, eating animals, nesting behavior, social hunting, etc.. These characteristics are constrained by natural laws such as the selection for higher evolvability in fluctuating environments[16][17], relation between size, speed, and brain size on Earthly animals[18][19][20][21], metabolic rate life-span and growth scaling laws[22][23][24][25]; such laws likely have equivalences in other environmental conditions than Earth.
Empirical laws fitting the relations between: Body mass and brain mass (1st row, the right plot focuses on mammals). And body mass and maximum speed (2nd row).
The impact of this race is possibly weaker than that of other races. We claim that the geographic region from which an SFC will be created will have unfair advantages over other regions. Such as easily scaling population through agriculture, scaling industrial production through easy energy and resource access, or scaling trade by having natural transportation mediums and favorable locations[26][27]. These three (illustrative) factors are reinforced by how early they are unlocked and how strong they are. E.g., the earlier a region can scale its population/industry/trade, and the higher it can, the more likely it will create first an SFC.
Competitors: Geographic groups within an intelligent species
Victory condition: Scaling the raw power of a region by one to a few OOMs over other regions.
Deadline to finish the race: Preclusion from other geographic groups taking control (soft or hard power) over the less powerful regions.
Likely winner: Regions with a large population, important energy sources and resources, coastal or with natural fluvial networks, etc. All of those as early and as consistently as possible.
Likely challengers: Regions with a large population in which the scaling of population happens late.
Some assumptions:
Geographic regions will compete for geopolitical influence and resources. The powerful will exploit the weak and export their culture.
The benefits of having a large population and easy resource and energy access are large in terms of geopolitical and cultural influence.
Being powerful at an early point in time is a good predictor for being powerful at a later point in time.
Earth-analogy:
The indigenous populations and cultures from the following regions lost the race on Earth: E.g., Siberia, Greenland, Amazonie, desertic regions in general, high altitude regions, Native-American, Scandinavia, Africa, etc.
Other populations and cultures benefited from geographical advantages: (Early[28] and late scaling) India, China, and Mesopotamia. (On time[29] scaling) Europe, America. And notably, the US has strong geographical advantages[30][31].
This race is similar to the previous one but focuses on advantages from policies at the level of macro-social groups such as countries instead of the geographical advantages at the level of regions. We claim that the political organizations creating an SFC first will also be the organizations the most technologically advanced and economically prosperous and that these characteristics are strongly correlated with the level of micro-level economic incentives and economic freedom which are themselves correlated with political freedom[32][33][34]. Additionally, early technological and economic leads are also decent predictors for creating an SFC first.
Competitors: Large groups with centralized policies and norms (e.g., countries)
Victory condition: Scaling the economic and military power of a region by one to a few OOMs over other regions.
Deadline to finish the race: Preclusion from other organizations taking control (soft or hard power) over resources, land, and populations.
Likely winner: Organizations under a lot of cultural evolution pressure (early on) towards efficiency, science, and economic freedom. Organizations able to break out of the Malthusian trap.
Likely challenger: Organizations (later on) able to imitate cultures that bring economic success and to learn from others. Organizations that are able to, at least, catch up technologically.
Some assumptions:
Organizations will compete for geopolitical influence and resources. The powerful will exploit the weak and export their culture.
The benefits from leading economically and technologically are large in terms of geopolitical and cultural influence.
Being powerful at an early point in time is a good predictor for being powerful at a later point in time.
Earth-analogy: Only the countries in which science and economic freedom is valued are still in the race on Earth. E.g., all else equal, religious and welfarist countries are slowed down by their culture. One likely reason European and North-America countries were able to influence the world so much is the power they got from leading the scientific and industrial revolution.
We claim that the technological requirements necessary to create an SFC (e.g., deep learning scaling laws) imply a race between efficient value-producer groups (e.g., companies) for accumulating skills, capital, and compute. Because of network effects and the very low marginal cost of production, the first value to successfully scale the production of digital and status products will dominate the digital and status economy. These “tech companies” will shape the culture and values of the individuals who will ultimately create the first ASIs, and will be ideally positioned to invest in AI.
Victory condition: Being among the first to successfully scale selling digital products and entrenched their competitive advantages.
Deadline to finish the race: Digital and status market saturation. The entrenchment of positions caused by network and scale effects.
Likely winner: Large organizations specialized in digital products.
Likely challenger: Small organizations specialized in digital products partially owned by large organizations. And organization producting half-tech/half-status products.
Some assumptions:
SFC is achieved by developing AIs. Which massively rely on scaling training and inference compute. Natural evolution has no time to have any significant effect. Bio-enhancements are too limited (e.g., too complex, suffering from delays, not able to change skill profiles), note this assumption may be somewhat incorrect, see How to Make Superbabies. Mind upload is feasible after AGI is possible, or requires scaling inference and plausibly training compute.
The benefits for a company leading economically and technologically are large in terms of willingness, capacity, and efficiency to invest in AI development.
Let’s speculate about the characteristics of the winners. The internal culture values science, technology, and fast-paced technological progress. The capability to quickly iterate on digital products and to get cheap, precise direct feedback will lead to a culture where performance measurement is more quantitative and more rigorous than in other organizations. The scalable nature of the digital and status products will lead to higher income inequality, where the most capable workers in tech companies are much more rewarded than the average worker in the same country. The complexity of technical and digital products will imply highly intelligent workers who are able to cooperate and learn from each other. Etc.
Inter-individuals
We claim that individuals will be part of a race to influence the first ASIs and that individuals winning this race, referred to as SFC Shapers, have some predictable characteristics. We try not repeating characteristics easily inferable from the previous races leading to this final one.
Competitors: Individuals
Victory condition: Being among the SFC Shapers. Being among the individuals influencing the most the first ASIs.
Deadline to finish the race: Preclusion by other individuals grabbing all influence over the ASIs.
Winner: The SFC Shapers (by definition).
Earth-analogy: Trump, Musk, Altman, Dario, Yudkowsky, executive and senior positions at leading AI labs, AI Safety community, Effective AI Acceleration community, greed-driven individuals working in tech companies, etc.
What are the likely characteristics of SFC Shapers? Let’s speculatively describe two profiles among SFC Shapers, the influencers and the technical creators and operators.
Speculative characteristics of influencers:Top 100 worldwide in one or more of political, economic, scientific, military influences, raw wealth, AI science, and philosophy. And the predictable characteristics associated with such positions: E.g., highly logically or/and emotionally intelligent, able to work with others, likely power or status-seeking and optimising for influence, more likely originating from an elite background. Other characteristics include: Living in one of the countries leading the race to ASI. Aware of the importance of AI. Etc.
Speculative characteristics of the technical creators and operators:World-level intellectual performance, very wealthy (from birth in the world top 5%, in the top 1% since working, joined the world top 0.1% before ASI is created), scientific/rational mindset, somewhat truth-seeking at work, has been working in tech for years, able to efficiently cooperate in team, able to learn from others and by themselves, live in a (rich) tech hub, studied in a prestigious institution. Etc.
Convergent genetic and cultural evolution
We may expect some degree of convergent evolution between SFC Shapers in different Intelligent Civilizations. Convergent genetic evolution has already been discussed and described many times in many places[35][36][37][38]. Thus, we will focus on listing a few plausible high-level mechanisms that could cause meaningful degrees of convergent cultural evolution.[39]
Evolutionary psychology
Our minds may be significantly shaped by our genetic selection and the environmental and functional constraints involved. This is the domain of study of evolutionary psychology[40][41][42]. We may find strong reasons to expect similar effects in other ICs.
The origin of pleasure and suffering is partially explained by genetic selection. For example, universal drives around food acquisition and reproduction likely shape core experiences of pleasure and suffering across intelligent life. The reward systems incentivizing food consumption and reproduction would be selected due to their direct impact on survival. We may expect analogous pleasure/suffering mechanisms in other ICs, with pleasure being associated with behaviors that historically increased reproductive fitness (like consuming high-energy foods, storing food and capitals, protecting descendants, and valuing safety) and suffering being associated with fitness-reducing situations (like starvation, social isolation, or body harm). The intensity of these experiences may also follow universal patterns - for instance, suffering from acute threats may be more intense than pleasure from routine rewards, as this asymmetry better promotes survival.
Cultural evolution
Cultural evolution[43][44][45] is another research domain studying how minds, values, and capabilities are shaped. It is similar to evolutionary psychology but related to how cultures are shaped by the selection and reproduction of ideas or unitary bits of information (memes), instead of genes.
Five known mechanisms from which altruism and cooperation emerge. We have a decent understanding of the reasons why cooperation and altruism emerge, though they are not all cultural. Five mechanisms are usually named. We quote (Kristian Rönn, The Great Bootstrap 2024) to describe them: "Kin Selection fosters cooperation among genetically related individuals, like a bee sacrificing itself for the hive. Direct Reciprocity is based on mutual aid, as seen when neighbors agree to watch each other’s homes. Network Reciprocity thrives within tight-knit groups, creating a safety net of support. Indirect Reciprocity relies on reputation, where helpful individuals are more likely to receive help in return. Multi-Level Selection occurs when cooperative groups outcompete less cooperative ones, leading to the spread of cooperative norms.". For more details, find in the footnotes quotes from (Martin A Nowak, Five rules for the evolution of cooperation, 2006)[46] and (Kristian Rönn, The Great Bootstrap 2024)[47]. We may find strong reasons to expect these mechanisms to have analogs in alien Intelligent Civilizations. We don't introduce these reasons here.
Genetic and cultural co-evolution. Another interesting concept is the interplay between genetic and cultural evolution, which creates feedback loops that shape both biological and social characteristics of intelligent species; this is usually referred to as co-evolution[48][49]. Cultural innovations like waterskin or domesticating fire create new selection pressures on genes (e.g., for endurance hunting or increasing nesting behaviors), while genetic adaptations enable new cultural possibilities (e.g., larger brains enabling more complex behaviors). We may expect similar co-evolutionary dynamics in other species creating ICs, where cultural developments systematically influence "alien-genetic" selection and vice versa.
Self-domestication shaping ICs. Self-domestication[50][51][52] is another plausible example of the co-evolution of genetic and cultural evolutions. Cultural norms may shape who is able to genetically and memetically reproduce and thus can create a self-domestication effect. This effect may be behind the fast genetic entrenchment of norms, intuitions and feelings guiding humanity, and plausibly other ICs, towards some degree of cooperation and altruism.
Economic laws
Economic laws arising from fundamental constraints on resource allocation and coordination may shape all ICs. These include principles like comparative advantage, diminishing returns, and the emergence of markets and trade. The universal nature of scarcity, specialization benefits, and coordination problems suggests that any IC creating an SFC could naturally develop economic structures which target these opportunities and bottlenecks. This would likely include some form of property rights, mechanisms for large-scale cooperation and resource allocation, and incentive systems rewarding productivity and innovation. The specific implementations may vary, but the underlying economic pressures would create convergent patterns, especially in terms of functionalities, in how resources and labor are organized. Here are a few speculative examples:
Public goods dynamics. Public goods problems create selective pressure for pro-social norms and punishment mechanisms. Most ICs would face free-rider problems when coordinating on large-scale projects, this should select for cultural innovations that value cooperation and contribution to collective welfare. This pressure would likely favor the emergence of social norms punishing defection and rewarding contribution, constraining the space of moral values.
Principal-agent relationships. Delegation naturally creates information asymmetries and incentive misalignments between the agent and principal. The existence of principal-agent problems would select for cultural norms around trust, monitoring, and reputation across any IC. The universal nature of this problem may make it a powerful constraint on moral evolution, likely pushing toward values that solve principal-agent problems like accountability and honesty.
Game theoretic equilibria. Strategic interactions, like in the prisoner's dilemma game, can create selection pressure for conditional cooperation strategies. This could be a reason driving some convergent evolution of values around reciprocity, fairness, and proportional punishment. The fundamental mathematics of game theory would constrain the space of stable social norms for any ICs, regardless of their bodily and chemical differences from humanity.
Filters at the level of Intelligent Civilizations
Finally, the characteristics of SFC Shapers are also filtered by a number of bottlenecks which prevent Intelligent Civilizations from creating SFCs. These further reduce the space of possible characteristics of SFC Shapers. We quickly list possible bottlenecks on ICs creating SFCs, ranging in plausibility from speculative to real.
For each, we give in footnotes a speculative evaluation of how real they are (likelihood), which impact they would have (effect), and some details about how the bottleneck may work (idea). Notably, we don't try here to evaluate the strength of these filters, an effect could be real but only apply very rarely, it would filter very few ICs and thus have a very small effect while being real.
Three groups of filters. We sort the bottlenecks in three groups: civilization destruction, economic destruction, and intelligence destruction. We try to provide some references agreeing or disagreeing with the existence of the filters or analogous effects, as well as descriptive-only references. References are illustrative, and not the product of a thorough investigation.
General filtering trends. The filters listed below show general trends towards filtering more stronglythe ICs which spend more time with high self-destruction capabilities, which are less capable at cooperation and shaping incentives to reduce extinction risks, and which are unlucky from either living on a fragile world or in fragile bodies.
Weak filters. These filters likely have a much weaker effect than the sequence of almost winner-takes-all races introduced in the previous section.
Likelihood: Speculative to plausible. Speculated effects[119]. Idea[120].
Context
Evaluating the Existence Neutrality Hypothesis - Introductory Series. This post is part of a series introducing a research project for which I am seeking funding: Evaluating the Existence Neutrality Hypothesis. This project includes evaluating both the Civ-Saturation[2] and the Civ-Similarity Hypotheses[1] and their longtermist macrostrategic implications. This introductory series hints at preliminary research results and looks at the tractability of making further progress in evaluating these hypotheses.
Acknowledgements
Thanks to Tristan Cook, and Justis Mills for their excellent feedback on this post and ideas. Note that this research was done under my personal name and that this content is not meant to represent any organization's stance.
The Civ-Similarity Hypothesis posits that the expected utility efficiency of humanity's future Space-Faring Civilization (SFC) would be similar to that of other SFCs.
The Civ-Saturation Hypothesis posits that when making decisions, we should assume most of humanity's Space-Faring Civilization (SFC) resources will eventually be grabbed by SFCs regardless of whether humanity's SFC exists or not.
"Extinction" here is the extinction of intelligent life on Earth, it is not limited to the extinction of humanity. In this post, we mostly conflate the extinction of intelligent life on Earth with humanity not creating an Space-Faring Civilization.
Here are a few questions we would need to solve to be able to study the relative utility produced by a Space-Faring Civilization (SFC) given the characteristics of the Intelligent Civilization (IC) creating it, when compared to another (SFC, IC) pair:
What should we value? What should we optimize for?
What is consciousness, sentience? How does it work? Is it wide-spread? Does it take different intensities? Are humans abnormal by being conscious?
What is the nature of the universe? Are we in a multiverse? What are the actual laws of the world (e.g., physics)?
Are acausal interactions possible, likely? Should we expect causal or acausal conflicts?
How does the moral values chosen by an IC depend on its characteristics?
What role does the evolutionary history of an IC play in determining its long-term values?
How does the distribution of values within an IC affect which values are ultimately represented in its SFC?
How does the alignment power of an IC depend on its characteristics?
Which parameters influence the technical alignment power of an IC?
Which geopolitical factors influence the governance of superintelligent agents?
Is humanity abnormal in creating an SFC through creating AIs?
Which moral values an SFC will optimize in the far future, given different levels of alignments? How stable are SFCs' moral values over long time scales? Do they converge or diverge?
How do resource constraints affect value formation and prioritization in expanding civilizations? Should we expect selection effects between SFCs? How is the offense-defense balance between SFCs?
Quote: “There are no rules requiring minds to value life, liberty or the pursuit of happiness. An alien will have, in all probability, alien values. If an "alien" isn't evolved, the range of possible values increases even more, allowing such absurdities as a Paperclip maximizer. Creatures with alien values might as well value only non-sentient life, or they might spend all their time building heaps of prime numbers of rocks.” (Alien Values 2020)
Quote: "Once upon a time there was a strange little species—that might have been biological, or might have been synthetic, and perhaps were only a dream—whose passion was sorting pebbles into correct heaps.
They couldn't tell you why some heaps were correct, and some incorrect. But all of them agreed that the most important thing in the world was to create correct heaps, and scatter incorrect ones.
Why the Pebblesorting People cared so much, is lost to this history—maybe a Fisherian runaway sexual selection, started by sheer accident a million years ago? Or maybe a strange work of sentient art, created by more powerful minds and abandoned?" (Eliezer Yudkowsky, 2008)
Quote: "But minds with different emotions - minds that feel emotions you've never felt yourself, or that fail to feel emotions you would feel? That's something you can't grasp by putting your brain into the other brain's shoes. I can tell you to imagine an alien that grew up in universe with four spatial dimensions, instead of three spatial dimensions, but you won't be able to reconfigure your visual cortex to see like that alien would see. I can try to write a story about aliens with different emotions, but you won't be able to feel those emotions, and neither will I.
Imagine an alien watching a video of the Marx Brothers and having absolutely no idea what was going on, or why you would actively seek out such a sensory experience, because the alien has never conceived of anything remotely like a sense of humor. Don't pity them for missing out; you've never antled." (Eliezer Yudkowsky, 2008)
Quote: “Evolutionary psychology is a theoretical approach in psychology that examines cognition and behavior from a modern evolutionary perspective.[1][2] It seeks to identify human psychological adaptations with regards to the ancestral problems they evolved to solve. In this framework, psychological traits and mechanisms are either functional products of natural and sexual selection or non-adaptive by-products of other adaptive traits.[3][4]
Adaptationist thinking about physiological mechanisms, such as the heart, lungs, and the liver, is common in evolutionary biology. Evolutionary psychologists apply the same thinking in psychology, arguing that just as the heart evolved to pump blood, the liver evolved to detoxify poisons, and the kidneys evolved to filter turbid fluids there is modularity of mind in that different psychological mechanisms evolved to solve different adaptive problems.[5] These evolutionary psychologists argue that much of human behavior is the output of psychological adaptations that evolved to solve recurrent problems in human ancestral environments.[6]
“Cultural evolution is an evolutionary theory of social change. It follows from the definition of culture as "information capable of affecting individuals' behavior that they acquire from other members of their species through teaching, imitation and other forms of social transmission".[1] Cultural evolution is the change of this information over time.[2]
Cultural evolution, historically also known as sociocultural evolution, was originally developed in the 19th century by anthropologists stemming from Charles Darwin's research on evolution. Today, cultural evolution has become the basis for a growing field of scientific research in the social sciences, including anthropology, economics, psychology, and organizational studies. Previously, it was believed that social change resulted from biological adaptations; anthropologists now commonly accept that social changes arise in consequence of a combination of social, environmental, and biological influences (viewed from a nature vs nurture framework).[3][4]
There have been a number of different approaches to the study of cultural evolution, including dual inheritance theory, sociocultural evolution, memetics, cultural evolutionism, and other variants on cultural selection theory. The approaches differ not just in the history of their development and discipline of origin but in how they conceptualize the process of cultural evolution and the assumptions, theories, and methods that they apply to its study. In recent years, there has been a convergence of the cluster of related theories towards seeing cultural evolution as a unified discipline in its own right.[5][6]” (Wikipedia’s introduction to cultural evolution)
Cooperation is needed for evolution to construct new levels of organization. The emergence of genomes, cells, multi-cellular organisms, social insects and human society are all based on cooperation. Cooperation means that selfish replicators forgo some of their reproductive potential to help one another. But natural selection implies competition and therefore opposes cooperation unless a specific mechanism is at work. Here I discuss five mechanisms for the evolution of cooperation: kin selection, direct reciprocity, indirect reciprocity, network reciprocity and group selection. For each mechanism, a simple rule is derived which specifies whether natural selection can lead to cooperation.” (Martin A Nowak, Five rules for the evolution of cooperation, 2006)
Quote: “Kin Selection, Direct Reciprocity, Network Reciprocity, Indirect Reciprocity, and Multi-Level Selection—can be seen as different strategies to promote cooperation.
Kin Selection fosters cooperation among genetically related individuals, like a bee sacrificing itself for the hive.
Direct Reciprocity is based on mutual aid, as seen when neighbors agree to watch each other’s homes.
Network Reciprocity thrives within tight-knit groups, creating a safety net of support.
Indirect Reciprocity relies on reputation, where helpful individuals are more likely to receive help in return.
Multi-Level Selection occurs when cooperative groups outcompete less cooperative ones, leading to the spread of cooperative norms.
The first three mechanisms—Kin Selection, Direct Reciprocity, and Network Reciprocity—have historically helped us tackle past challenges in our evolutionary history. Nonetheless, each of these mechanisms comes with distinct limitations, rendering them insufficient for facilitating widespread cooperation or the Great Bootstrap. Kin selection is confined to familial ties, direct reciprocity to individual exchanges, and network reciprocity is effective only within small groups. These limitations narrow the viable options for achieving broader cooperative success down to multi-level selection and indirect reciprocity. In a way, these two mechanisms represent opposite ends of the spectrum.
Multi-level selection, in the case of human civilization, arises when a dominant global actor creates rules via a monopoly of violence that penalizes defectors, thereby often enforcing governance through centralization.
Indirect reciprocity, on the other hand, might emerge through a sort of global reputation system, where defectors receive a lower reputation score and are consequently sanctioned by all other participants, thus facilitating governance through decentralization.” (Kristian Rönn, The Great Bootstrap 2024)
Partially filtering out Intelligent Civilizations (ICs) whose cultures have some of the following traits: low international cooperation, low empathy with war victims, low rationality, high conflict frequency, low security mindset, etc.
Nuclear or biological warfare can set back in time ICs or cause extinctions. Some features of ICs are linked to increased/decreased Nuclear risks and are thus more often filtered out/in. Temporary setbacks could also be assimilated to resampling new cultural norms, creating a new IC with the same underlying intelligent species.
Filtering ICs on small planets or with slow technological progress, which destabilize their climate too much. Especially filtering ICs originating from planets with unstable climates, and not caring about future generations and fragile nations, or not able to coordinate to prevent climate destabilization through the accumulation of “pollutants”, e.g., warming gases.
Climate changes caused by the IC (and much more rarely caused by random cosmic events such as asteroid impacts) could destroy or set back an IC. We also include here the destruction of biodiversity or lands, which would destabilize the climate.
The IC loses the capability to create an SFC when an early AI takeover or destroys the IC, conditional on the AI not having the capabilities or the will to become an SFC. E.g., AI amplified wars between factions in an IC, AIs destroying their creators but not caring enough about their own survival, and AIs with just enough capabilities to destroy their creators but not enough capabilities to survive or flourish alone as an SFC.
Filtering ICs with small economies and/or slow technological progress. Filtering ICs with very slow AI takeoff. Amplifying other bottlenecks whose occurrence is proportional to the time spent before creating an SFC.
Population decline in the most advanced countries could lead to a slowdown of technological progress (e.g., a much lower young/old ratio), to a rise in the importance of capital versus income, leading to increasing inequalities and political and economic destabilization, or simply to a shrink of the global economy such that scaling compute via investment is harder and harder.
Filtering out wasteful ICs, or ICs in planets with low resources availability. Amplifying other bottlenecks whose occurrence is proportional to the time before creating an SFC.
Increasing resource depletion and scarcity could lead to a permanent reduction in economic growth or total output. Such reduction can prevent or slow down the creation of an SFC.
The longer the time between space exploration and the creation of an SFC, the more polluted the orbit of the IC’s planet. At some point, this may increase the capability threshold necessary for an AI to create an SFC.
In a true Malthusian trap, almost all surplus gets eaten by population growth. In high-conflict worlds, state-power may be optimized by having high populations (in addition to other factors). Birth control in some ICs may be harder than for humanity such that populations would more easily fall in Malthusian traps. In these cases, it's possible that technological progress is impaired long enough to have other risks that prevent some of these ICs from creating SFCs.
On Earth historically the capital to income ratio in nations has been pretty stable and only the first and second world wars, and the economic boom following seems to have been able to change this ratio. It is plausible that this ratio governs the distribution of economic power in populations, and thus the political power and moral values that are selected by economic pressures. We can speculate that ICs whose combine a high tendency for exploiting individuals and low income growth may more likely experience a fall into decline through a decoupling of the economic interest of the owners of capital from that of the creators of value.
While economic incentives should prevent this scenario, we can imagine ICs in which their agricultural production is more sensitive to ecological perturbation caused by the IC. These ICs may destroy their own agricultural outputs through shortsightedness and mismanagement of lands.
Filtering out some very slow-growth Intelligent Civilizations (ICs) and some small-size ICs. E.g., preventing societies, at least 10x-100x slower at creating an SFC than humanity, from creating an SFC. Filtering out more strongly the ICs initially the closest to the minimum threshold for creating SFC. Etc.
Human brains have plausibly been shrinking since cities and agriculture have been developed. One could imagine that if an intelligent species would take 10x to 100x more evolutionary equivalent time to create an SFC, it could lose a significant fraction or most of its capability necessary for creating an SFC.
Filtering out slow-growth ICs, especially those with conflict-free cultures inside these ICs. Also, maybe filtering out some small size ICs if cultural drift is much faster there, this is very speculative. E.g., those at least 2x-10x slower than humanity. Filtering out more strongly the ICs with the worst epistemic norms. Etc.
When societies become too peaceful, the selection pressure forcing cultures (memes, ideas) to stay aligned with survival and economic productivity, are significantly reduced to the point of plausibly producing a random cultural drift, or at least a cultural drift independent of survival and productivity. This drift can lead to loss in capabilities or increased extinction risks.
Filtering out ICs polluting more, ICs with lower resistance to pollution, ICs with lower international coordination capabilities, slow growth ICs, ICs on planets with small pollution sinks (e.g., small atmospheric and oceanic sinks), etc.
Some kinds of pollution are known to decrease intellectual capabilities. It is plausible that some such pollution could be irreversible because of an incapacity to remove pollutants or because of economic and geopolitical selection pressures. If this scenario were to happen to an IC, it could cause semi-permanent cognitive capability loss.
Amplifying other risks. Filtering out countries not intrinsically valuing freedom of thought, education, etc., within slow-growth ICs. Filtering out more the ICs in which economic pressures are left unchecked.
As marketing efficiency improves (feedback-intensive economic systems, behavioral simulations, AI manipulation, etc.), profit-seekers might saturate public discourse and priorities. Over many decades, this might crowd out R&D and destabilize societies. Plausibly leading to stable points in which societies are mostly stagnant (all resources are spent on marketing and just barely enough in science to maintain current technological levels) or into downward slopes (individual profit and power optimization destroying the aggregated economy).
Scientific progress is valued by a society as long as it brings benefits to its supporters. We can speculate that the slower the economic growth of an economy, the less value a society will attribute to scientific progress. Low growth societies may trend towards valuing less and less scientific progress (as long as there is no upward trend in its usefulness). Small ICs may not be large enough for scientific progress to produce net positive outputs.
Executive summary: The characteristics of Space-Faring Civilization (SFC) Shapers are likely constrained by evolutionary dynamics, almost winner-takes-all races, and universal selection pressures, which may imply that different SFCs across civilizations will have similar values and capabilities. If true, this could challenge the prioritization of extinction risk reduction in longtermist strategy, as the expected utility of alien SFCs may not be significantly different from humanity’s SFC.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.