Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
Subtitle: Will we agree about AI risks in time?
Science is a social institution like the judicial system. Just like courts make you throw out perfectly good Bayesian evidence like whether the defendant seems like a shifty guy to you or rumors you heard about his whereabouts, the scientific process has epistemically conservative norms about what kinds of evidence and arguments are admissible. This means that the process of science gets to the truth much slower than a perfectly rational Bayesian should, and indeed much slower than many particular farseeing individuals actually did. Svante Arrhenius made the first estimate of global warming from human carbon dioxide emissions back in 1896, probably about eighty years before something like a “scientific consensus” emerged and ninety four years before the first IPCC Assessment Report.
But they say if you want to go fast go alone, if you want to go far go together. At its best, the point of the scientific process is to add bricks to a solid foundation of facts that we can establish “beyond a reasonable doubt,” that we can all agree on regardless of our priors because the weight of the evidence we’ve built up is enough to overwhelm those differences. This is a good piece of social technology to have. We’re never going to agree on subtle questions of priors, and it’s great to have a process that lets us at least agree about where the giant likelihood ratios lie. And like our adversarial justice system, scientific institutions are to a significant extent designed around the fact that we all have biases and conflicts of interest — personal virtue isn’t as verifiable or scalable as specific norms like preregistration, reproducibility, and peer review.
This is why, when AI Snake Oil (now known as AI As a Normal Technology) wrote AI existential risk probabilities are too unreliable to inform policy two years ago, I was much more sympathetic to it than most people who are as concerned as I am about existential risks from AI. While I strongly disagree with the authors on the object level, my guess is that loosening the norms around how much policymaking should be based on science is much more likely to lead to vaccine bans and other destructive crackpottery than thoughtfully crafted AI safety regulation.
The arguments about superintelligence and the Singularity are speculative. “Speculative” should not be a pejorative term meaning “obviously wrong and dismissible” (Arrhenius’ arguments about CO2 emissions were speculative), but it does make sense that society draws a normative distinction between speculative and scientifically established claims.
Unfortunately, these conservative norms have a serious chance of getting us killed in the case of AI. The pace of change in the field is vastly outstripping our ability to develop a well-grounded scientific picture of it. Just last year, METR generated a lot of buzz with an RCT showing that early 2025 AI tools actually seemed to make open source developers a little slower at their work tasks. By the time my colleagues started looking at late 2025 AI tools, the experiment design broke because participants started dropping out of the study or changing what they worked on because they were so afraid of being randomized into the no-AI arm for their normal tasks.
Still, the project of scientific research on models’ capabilities and risks doesn’t seem doomed. We could have lived in a world where superintelligence truly came as a bolt from the blue, like a lot of the speculative writing from ten or fifteen years ago contemplated. The world we find ourselves in is extremely far from that. We used to debate questions like whether people would make AI agents at all, or whether they would be connected to the internet, or whether they would be allowed to run on their own without a human in the loop. We know the answers to all those questions now. We have seen thousands of public examples of AI agents lying, cheating, and working around safeguards to try to achieve their task. With every new model there are new researchers who take catastrophic risks seriously. And precisely because we are so far from a bulletproof consensus on anything and the field moves at such a breakneck pace, there is so much grist for research and testing and sensemaking. The scientific fruit is on the floor.
We almost certainly won’t be able to develop an evidence base about AI risks anywhere near as robust as what we have for greenhouse gases causing climate change or tobacco causing lung cancer. But science is a process made out of humans, and a “scientific consensus” is ultimately a set of facts the relevant scientists managed to agree about. This develops through some messy combination of building up the foundation of established facts through shared experience and solid experiments, making logical arguments on top of established facts, arguing with one another to establish the right frames to think about things, observing whose predictions turn out to be more right, and much more. I think we do have a shot at building a real-life scientific consensus robust enough to motivate serious technical standards before it’s too late. AI is going to keep improving at a breakneck pace, and in some ways that makes this an easier problem than tobacco or climate change. Scientific consensus, like everything else, could be dragged along at the speed of AI.
