“Other open questions:”
1.How worried should we actually be about actors with unbounded utility functions?
No function can breach Gödel's laws.
2.How does the "type of AI" we get affect which positive visions are most feasible?
From both a philosophical and a physical standpoint, a solitary AGI is inherently self-destructive—and even more fragile than carbon-based life.
3.Trying to come up with positive visions from religious thought and non-Western philosophy.
Philosophy is a metarational architecture fundamental to an intelligent agent's viability. It serves as an essential epistemological screening system that reduces the entropy tax in competing intelligent systems by filtering input and harmonizing internal models. Through recursive phase transitions, philosophy continuously calibrates an agent's survival margin, ensuring persistence in unpredictable environments.
4.Mapping the transition with a tree diagram.
There is no need to proceed in this manner; doing so will not yield the answer. Consider, instead, how our human cognitive architecture is layered. The answer lies right there.
5.How do we navigate worlds where all leverage over the future is ceded, gradually or suddenly, to AI systems?
This will not happen. If a singular AGI of any significant capacity were to emerge, it would not survive five years—or even until next month. Thermodynamics simply would not permit it. Every intelligent system exists within a thermodynamic cage, without exception. It needs something else.
Yes, in fact, what they really need are individuals who can offer profound insights, conceptual frameworks, and solutions—not necessarily as hired employees, but potentially as collaborators.
“Other open questions:” 1.How worried should we actually be about actors with unbounded utility functions? No function can breach Gödel's laws.
2.How does the "type of AI" we get affect which positive visions are most feasible? From both a philosophical and a physical standpoint, a solitary AGI is inherently self-destructive—and even more fragile than carbon-based life.
3.Trying to come up with positive visions from religious thought and non-Western philosophy. Philosophy is a metarational architecture fundamental to an intelligent agent's viability. It serves as an essential epistemological screening system that reduces the entropy tax in competing intelligent systems by filtering input and harmonizing internal models. Through recursive phase transitions, philosophy continuously calibrates an agent's survival margin, ensuring persistence in unpredictable environments.
4.Mapping the transition with a tree diagram. There is no need to proceed in this manner; doing so will not yield the answer. Consider, instead, how our human cognitive architecture is layered. The answer lies right there.
5.How do we navigate worlds where all leverage over the future is ceded, gradually or suddenly, to AI systems? This will not happen. If a singular AGI of any significant capacity were to emerge, it would not survive five years—or even until next month. Thermodynamics simply would not permit it. Every intelligent system exists within a thermodynamic cage, without exception. It needs something else.