Let’s say you want to know how likely it is that an innovative new product will succeed, or that China will invade Taiwan in the next decade, or that a global pandemic will sweep the world — basically any question for which you can’t just use “predictive analytics,” because you don’t have a giant dataset you can plug into some statistical models like (say) Amazon can when predicting when your package will arrive.
Is it possible to produce reliable, accurate forecasts for such questions?
Somewhat amazingly, the answer appears to be “yes, if you do it right.”
Prediction markets are one promising method for doing this, but they’re mostly illegal in the US, and various implementation problems hinder their accuracy for now. Fortunately, there is also the “superforecasting” method, which is completely legal and very effective.
How does it work? The basic idea is very simple. The steps are:
-
First, bother to measure forecasting accuracy at all. Some industries care a lot about their forecasting accuracy and therefore measure it, for example hedge funds. But most forecasting-heavy industries do not even bother to measure their forecasting accuracy, for example the US intelligence community or philanthropy.
-
Second, identify the people who are consistently more accurate than everyone else — say, those in the top 0.1% for accuracy, for multiple years in a row. These are your “superforecasters.”
-
Finally, pose your forecasting questions to the superforecasters, and use an aggregate of their predictions.
Technically, the usual method is a bit more complicated than that, but these three simple steps are the core of the superforecasting method.
So, how well does this work?
A few years ago, the US intelligence community tested this method in a massive, rigorous forecasting tournament that included multiple randomized controlled trials and produced over a million forecasts on >500 geopolitical forecasting questions such as “Will there be a violent incident in the South China Sea in 2013 that kills at least one person?” This study found that:
-
This method produced forecasts that were very well-calibrated, in the sense that forecasts made with 20% confidence came true 20% of the time, forecasts made with 80% confidence came true 80% of the time, and so on. The method is not a crystal ball; it can’t tell you for sure whether China will invade Taiwan in the next decade, but if it tells you there’s a 10% chance, then you can be pretty confident the odds really are pretty close to 10%, and decide what policy is appropriate given that level of risk.
-
This method produced forecasts that were far more accurate than those of a typical forecaster or other approaches that were tried, and ~30% more accurate than intelligence community analysts who (unlike the superforecasters) had access to expensively-collected classified information and years of training in the geopolitical issues they were making forecasts about.
Those are pretty amazing results! And from an unusually careful and rigorous study, no less!
So you might think the US intelligence community has eagerly adopted the superforecasting method, especially since the study was funded by the intelligence community, specifically for the purpose of discovering ways to improve the accuracy of US intelligence estimates used by policymakers to make tough decisions. Unfortunately, in my experience, very few people in the US intelligence and national security communities have even heard of these results, or even the term “superforecasting.”
A large organization such as the CIA or the Department of Defense has enough people, and makes enough forecasts, that it could implement all steps of the superforecasting method itself, if it wanted to. Smaller organizations, fortunately, can just contract already-verified superforecasters to make well-calibrated forecasts about the questions of greatest importance to their decision-making. In particular:
-
The superforecasters who out-predicted intelligence community analysts in the forecasting tournament described above are available to be contracted through Good Judgment Inc.
-
Another company, Hypermind, offers aggregated forecasts from “champion forecasters,” i.e. the most accurate forecasters across thousands of forecasting questions for corporate clients going back (in some cases) almost two decades.
-
Several other projects, for example Metaculus, are also beginning to identify forecasters with unusually high accuracy across hundreds of questions.
These companies each have their own strengths and weaknesses, and Open Philanthropy has commissioned forecasts from all three in the past couple years. If you work for a small organization that regularly makes important decisions based on what you expect to happen in the future, including what you expect to happen if you make one decision vs. another, I suggest you try them out. (All three offer “conditional” questions, e.g. “What’s the probability of outcome X if I make decision A, and what’s the probability of that same outcome if I instead make decision B?”)
If you work for an organization that is very large and/or works with highly sensitive information, for example the CIA, you should consider implementing the entire superforecasting process internally. (Though contracting one or more of the above organizations might be a good way to test the model cheaply before going all-in.)
My worry is this: if enough people are trying to make forecasts, just by random chance you will get some people that attain whatever arbitrary amount of accuracy you desire (e.g. getting several forecasts in a row right for several years). How do we tell if a "superforecaster" is really that or they just got lucky until now? If the latter, their past success is not an indicator of future success.