We’re the co-founders of Better Futures Guide (BFG), a charity evaluator specifically focused on livelihoods. We were founded last year through Ambitious Impact’s Charity Entrepreneurship Incubation Program. We welcome questions, criticism and opportunities for collaboration, in the comments or at
[email protected].
...
Random thought: does the idea of explosive takeoff of intelligence assume the alignment is solvable?
If the alignment problem isn’t solvable, then an AGI, in creating ASI, would face the same dilemma as humans: The ASI wouldn’t necessarily have the same goals, would disempower the AGI, instrumental convergence, all the usual stuff.
I suppose one counter argument is that the AGI rationally shoudn’t create ASI, for these reasons, but, similar to humans, might do so anyway due to competitive/racing dynamics. Whichever AGI doesn’t creates ASI will be left behind, etc.
not if the ai increases intelligence via speed up or other methods which don't change the goals.