Good post. It's worth mentioning that this already happened. This is what the Singularity Institute, now called the Machine Intelligence Research Institute (MIRI), wrote in 2000. MIRI predicted that transhuman AI — what we'd now call artificial general intelligence — would be invented no later than 2020, and probably much earlier:

Our best guess for the timescale is that our final-stage AI will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010. As always with basic research, this is only a guess, and heavily contingent on funding levels.

In 1999, Eliezer Yudkowsky, MIRI's founder, wrote that he believed dangerous nanotechnology would wipe out humanity unless we (or, specifically he) invented friendly AI first. (Now he says don't just slow down, stop. Back then he said we had to go fast.) Nowadays, we'd call friendly AI something like aligned AGI or safe AGI. Yudkowsky's name for his friendly AI project was "Ellison".

He wrote:

My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015. The most optimistic estimate for project Elisson would be 2006; the earliest nanowar, 2003.

So, what happens when people get the timeline wrong by at least 6 years? You've seen one example of what has happened! Look at MIRI and at Eliezer Yudkowsky, and at the LessWrong community.

Another example I personally found interesting to look at is the eschatology of Jehovah's Witnesses. In the 1960s, the Jehovah's Witnesses leadership embraced an end of the world prophecy with a specific year attached: 1975.

This prophecy was widely taught and believed among Jehovah's Witnesses. People took this deadly seriously and did things like sell their house and devote themselves full-time to evangelism.

When 1976 rolled around and the world was not over, the church leadership backpedaled and did revisionist history on what it had actually said. Recruitment suffered greatly for at least a few years, although the church did not collapse and obviously still exists. One leader even said the prophecy's falsification was a good thing, since it separated the true faithful wheat from the wavering chaff.

Would it be so surprising if the most devoted Singularity believers kept kicking the can down the road over and over forever until they eventually died of old age 50 or 60 years from now? Not to me, at least. That seems perfectly human.

On the other hand, many people whose belief is not as strong will probably rethink things when the AI bubble pops.

I think this question is pretty hard for me to answer because it's hard for me to imagine what the future would look like if we have AIs that are actually much more capable than today but the world mostly stays normal. I think it's much more plausible to me that I'm wrong because AIs don't become much capable, than because they do become much more capable and nothing external much changes.

1. If I'm imagining that world, the most likely hypothesis that comes to mind is that we had some significant global governance to either slow down further AI development or to manage the risks imposed by the existing very powerful models (likely both).
   1. In such a world, whether I should work on AI safety or not depends a lot on the details of whether the global governance is in a good place and/or benefits from my help.
2. Another possibility (that seems less likely to me but not impossible) is that we were wrong about the difficulty of aligning AI and we have some form of "alignment by default" world where the AIs are actually just much more cognitively capable than us but they're aligned to our intentions.
   1. In that world, what I should work on is in some sense not up to me (and certainly not up to my 2026-self!). I should instead be asking the new and better AIs what I should be doing, to the extent my efforts matter at all.
3. A third possibility is that capabilities stalled at a level slightly higher than today's, but not generally dangerous (for example if AI conquered a few fields the way it conquered math and cybersecurity, but otherwise stopped). And we've seen a few years with minimal further progress. In particular, this is a world where we have convincing evidence that AI R&D is not automatable, robotics has stalled out, human persuasion and long-run autonomy seems to have stalled, etc.
   1. In such a world I think probably some people should still work on AI safety but mostly it's manageable. Maybe people should go back to theory and conceptual stuff, if it looks clear that prosaic AI won't get to true AGI/ASI. Probably I'd personally work on something else. I'm not sure.
4. A fourth possibility is that AI capabilities are at an insanely high level and still growing, and alignment and governance still are far from solved, but we aren't yet doomed. In that world (which I think is the one you're most worried about/think is most central?) I basically think it's reasonable to think AI x-risk is just around the corner and we should still be working really hard on it.
   1. If we had, say, 3-5 more years of capabilities with AIs that are superhumanly good at science, long-term planning, real-world robotics (or side channels like persuasion and nanotech), but no major catastrophes, than I'm probably wrong about this for some key reason. #1 or #2 are the most obvious that comes to mind but maybe there's a secret third thing here that I didn't account for.

Loved the reflection! 

I would say caring for s-risks and x-risks will be still important and worth the funding and effort (the magnitude deserves it, even with a slight likelihood). 

However we should also care for more immediate and factual risks we usually leave to AI Ethics (e.g. capitalism surveillance, algorithmic bias, suicidal prevention), that could become a core part of AI Safety as well in the scenario you present.

What is (almost) certain by late 2026 is that AI is going to play a massive transformative role into society. Billions of human beings are already interacting with LLMs, AI agents are being deployed at a rapid scale, robotics is a few years ahead... 

AI Safety may need to adapt depending on how the risks scenarios unfold, but our industry is going to play an enormous role in doing the most good in the upcoming future, no matter when and if s-risks and x-risks happen. 

At least this is my view, and what motivates me to contribute to the field. I will be looking forward to hearing any thoughts you have about this! =)

Yes, that would probably be sufficient for me to change my mind. I think I a central example of someone who is deep into AI Safety.

I assign 5% probability to the scenario you are outlining - nothing dramatic happening until 2031.

Conditional on that scenario actually happening, I am 80% certain that I would dramatically reduce my involvement in AI Safety.

Let’s imagine it’s November 2031. We’re both still here, AI is much more capable than it is today, but governments are governing, civilisation is still (mostly) civilised, and there has been no catastrophic loss of control. It just kind of didn’t happen.

Depends on what "much more capable" means. But on the sharp left turn worldview, or a worldview similar to AI 2027, this is what the world is expected to look like shortly before AI kills everyone. When misaligned AI is smart enough to be transformative but not yet powerful enough to reliably seize control, its best strategy is to pretend to be aligned until it has enough power to wipe out humanity with minimal opposition.

The point at which I'd be less worried is where AI is clearly powerful enough to kill everyone if it wants to, and it's had plenty of time to do that, but it hasn't.

This is a great question! 

I only decided to work on AI safety in the last year or so, largely because the current rate of progress convinced me there was a high enough chance that the next 1-5 years are pivotal. Thinking about what I'll be working on in 2031 assumes (1) we're alive and (2) humans have retained enough control over the world for my actions to matter. The answer entirely depends on why (1) and (2) are true. 

I can think of a few scenarios: 

1. We're alive and in control in 2031 because we rose to the challenge and created some sort of (likely international) regulatory regime and/or solved the alignment problem. What I do depends on how much maintenance the solution requires, i.e. if we appear to be in a stable equilibrium. My best guess is that we won't be and there will still be lots of work to be done to maintain the fragile peace, so to speak.
2. The tech fizzles; models plateau around the current frontier or slightly past it. AI would still be a huge deal in this world (the frontier is currently internal models that can solve major unsolved math problems), but catastrophe seems further off. This is the world where I'm most likely to pivot to working on something else. I'd keep an eye on things and possibly go back to working in AI safety if a new paradigm takes off and it looks like we're in crunch time again.
3. Funding for scaling temporarily dries up. The Anthropic IPO is a bust, Wall St gets very nervous about the AI buildout, and some hyperscalers go under. The remaining frontier labs have a harder time convincing investors that $1B+ training runs are worth it and the rate of AI progress slows dramatically. I'm probably still working on AI safety in this world because nothing has happened to show that the underlying technology doesn't pose an existential risk. This future still has a button that maybe ends the world that anyone can pay $100B to press. The fact that no one is currently willing to pay $100B (or whatever the actual number is) to press the button doesn't change the fact that the world ending button is still waiting around for someone to press it. 

These predictions are all admittedly qualitative. I did have a quantitative prediction that played a big role in convincing me to pivot to AI safety - in mid-2024, I said that I would pivot if the METR long-horizon tasks trend line held through 2026. I updated a lot when that benchmark was saturated in the middle of this year. At some point I'll need to think about a comparable metric I can use to signal that it's time to pivot off AI safety. I don't feel a lot of urgency to do that though, given the way things are going. 

The best I have right now is "2027 and 2028 feel significantly less crazy than 2026." That's still vibes based, but feels tangible enough. I wouldn't automatically stop working on AI safety if this happened, but it would prompt me to think a lot harder about if I should be.