Good post. It's worth mentioning that this already happened. This is what the Singularity Institute, now called the Machine Intelligence Research Institute (MIRI), wrote in 2000. MIRI predicted that transhuman AI — what we'd now call artificial general intelligence — would be invented no later than 2020, and probably much earlier:
Our best guess for the timescale is that our final-stage AI will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels.
In 1999, Eliezer Yudkowsky, MIRI's founder, wrote that he believed dangerous nanotechnology would wipe out humanity unless we (or, specifically he) invented friendly AI first. (Now he says don't just slow down, stop. Back then he said we had to go fast.) Nowadays, we'd call friendly AI something like aligned AGI or safe AGI. Yudkowsky's name for his friendly AI project was "Ellison".
He wrote:
My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015. The most optimistic estimate for project Elisson would be 2006; the earliest nanowar, 2003.
So, what happens when people get the timeline wrong by at least 6 years? You've seen one example of what has happened! Look at MIRI and at Eliezer Yudkowsky, and at the LessWrong community.
Another example I personally found interesting to look at is the eschatology of Jehovah's Witnesses. In the 1960s, the Jehovah's Witnesses leadership embraced an end of the world prophecy with a specific year attached: 1975.
This prophecy was widely taught and believed among Jehovah's Witnesses. People took this deadly seriously and did things like sell their house and devote themselves full-time to evangelism.
When 1976 rolled around and the world was not over, the church leadership backpedaled and did revisionist history on what it had actually said. Recruitment suffered greatly for at least a few years, although the church did not collapse and obviously still exists. One leader even said the prophecy's falsification was a good thing, since it separated the true faithful wheat from the wavering chaff.
Would it be so surprising if the most devoted Singularity believers kept kicking the can down the road over and over forever until they eventually died of old age 50 or 60 years from now? Not to me, at least. That seems perfectly human.
On the other hand, many people whose belief is not as strong will probably rethink things when the AI bubble pops.
I think this question is pretty hard for me to answer because it's hard for me to imagine what the future would look like if we have AIs that are actually much more capable than today but the world mostly stays normal. I think it's much more plausible to me that I'm wrong because AIs don't become much capable, than because they do become much more capable and nothing external much changes.
What about AI becoming unsafe at a normal you-can-be-sued-for-this level strongly enough that you don't really get much forward progress because nobody can release or use their models, and this drives some combo of an AI financial collapse and a shift towards specialized models/harnesses/other anti-bitter lesson stuff? Does this scenario put too much trust in the AI companies?
Right, so to be a bit more concrete, maybe you're imagining that after HuggingFace 3.0 the push for generalized AIs stall out and we see some impressive specialized innovations like (say) self-driving cars and specialized biology models but no more general advances? Seems plausible.
Such a world I think look a lot like some combination of my #1 and #3, except with probably less available funding for AI safety than those worlds (though likely still higher than 2025 and 2026).
Thanks for mapping those scenarios out and yes, I'm most curious about 4. I suppose I’m interested in whether there’s a conceivable future in which the field accepts that it overestimated the danger - without immediately retreating to its predictions just being premature.