Higher-order forecasting could be a useful concept for prediction markets and forecasting systems more broadly.
The core idea is straightforward:
Nth-order forecasts are forecasts about (N-1)th order forecasts.
Examples
Here are some examples:
0-Order Forecasting (i.e., the ground truth)
- Biden won the 2020 U.S. presidential election
- The US GDP in 2023 was $27 trillion
1st-Order Forecasting (i.e., regular forecasting)
- What is the chance that Trump will win the 2024 U.S. presidential election?
- What will be the GDP of the US in 2024?
2nd-Order Forecasting
- How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?
- How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?
- If the question “What is the chance that a Republican will win the 2028 Presidential Election?” was posted to Manifold, with a subsidy of 100k Mana, what would the prediction be, after 1 month?”
3rd-Order Forecasting
- How much will the forecasts, [How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?] and [How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?], be correlated, from now until 2024?
- How valuable were all the forecasts for the question, [‘How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?’]
As forecasting systems mature, higher-order forecasts could play a role analogous to financial derivatives in markets. Derivatives allow for more efficient pricing, risk transfer, and information aggregation by letting market participants express views on the relationships between assets. Similarly, higher-order forecasts could allow forecasters to express views on the relationships between predictions, leading to a more efficient and informative overall forecasting ecosystem.
Benefits
Some potential benefits of higher-order forecasting include:
- Identify Overconfidence
- Improve the accuracy of forecasts by having participants directly predict and get rewarded for estimating overconfidence or poor calibration in other forecasts.
- "How overconfident is [forecast/forecaster] X"
- Prioritize Questions
- Prioritize the most important and decision-relevant questions by forecasting the value of information from different predictions.
- "How valuable is the information from forecasting question X?"
- Surface Relationships
- Surface key drivers and correlations between events by letting forecasters predict how different questions relate to each other.
- "How correlated will the forecasts for questions X and Y be over [time period]?"
- Faster Information Aggregation
- Enable faster aggregation of information by allowing forecasts on future forecast values, which may update more frequently than the underlying events.
- "What will the forecast for question X be on [future date], conditional on [other forecasts or events]?"
- Leverage Existing Infrastructure
- Leverage the existing infrastructure and resolution processes of prediction platforms, which are already designed to handle large numbers of forecasting questions.
We've already seen some early examples of higher-order forecasts on platforms like Manifold Markets. For example, with the recent questions:
- Will there be a Manifold bot that makes profitable bets on random 1-month markets by December 2024? (Ṁ3,000 subsidy!)
- Manifold Top Traders Leaderboard Ranking Prediction (2024)
- If Manifold begins allowing real-money withdrawals, will its accuracy improve?
- Is Manifold's P(Doom) by 2050 currently between 10% and 90%? [Resolves to Poll]
- Will Manifold be more accurate than real-money markets in forecasting the 2024 election?
Challenges
Of course, there are also challenges and risks to consider with higher-order forecasts:
- The accuracy of higher-order forecasts depends on the accuracy of the lower-order forecasts they build on. If the underlying forecasts are poorly calibrated or noisy, that will limit the value of higher-order forecasts.
- Higher-order forecasts inherently add complexity to forecasting systems, which could create challenges for participation, interpretation, and managing systemic risks.
- A substantial base of lower-order forecasting questions is needed before higher-order forecasts can be productively created on top of them.
Alternative Names
I considered a few options for names, asked around a bit, and settled on "higher-order" for this term. Here are some other options I considered:
- Derivatives: In the financial market, "markets about markets" are called derivatives. However, "derivative" is often understood as a term specific to markets, which could make it more confusing for forecasting.
- Meta-forecasts: I used this term before. It's a catchy term, but it doesn't differentiate between layers easily. There's no straightforward way to refer to "meta-layer 1."
- Higher-Layer: Similar to "higher-order," but less precise.
If there's contention on this later, it could be useful to have some formal discussion, to make sure that we share consistent terminology. Right now, I doubt many people care about it.
Conclusion
Over time, I expect higher-order forecasts to go from a niche idea to a key component of mature forecasting systems. Just as financial markets would be far less efficient without derivatives, forecasting platforms could see substantial accuracy and liquidity gains from higher-order forecasts.
An application I was expecting you to mention was longer term forecasts. E.g. if there was a market about, say, something in 2050, for example, the incentives for forecasters are perhaps less good, because the time until resolution is so long. But a "chained" forecast capturing something like "what will next year's forecast say" (and next year's forecast is about the following year's forecast, and so until you hit 2050, when it resolves to the ground truth).
This assumes that forecasters are less effective when it comes to markets which don't resolve for a long time.
In principle, normal markets should work this way. That is, if there's a market that won't settle for a year, but you think next week the price is going to go up a bunch, you will want to buy it now, and then sell it when the price goes up. If lots of people do this, the price goes up now, instead of next week (and in fact, if everyone saw that coming, it went up last week instead, and so on). If the market is reasonably liquid and/or there are market makers, you're not committing yourself until settlement, you can just sell out of your position when the p... (read more)