Comments
Forecasting Newsletter: July 2022
You can sign up for this newsletter on substack, or browse past newsletters here (a).
In the US, the land of the free, the CFTC has withdrawn its approval (a) for prediction market traders to trade against each other on political futures on PreditIt.com.
Part of the rationale for this withdrawal of approval is that PredictIt's alibi, that it was sharing its data with academics, wasn't up to the CFTC's liking, and that the company that actually runs the site was making a profit. For what it's worth, PredictIt made available a great API, which we at the Quantified Uncertainty Research Institute used without hitch on Metaforecast.org, our open source forecast aggregator.
A random person on Twitter (a) took responsibility for the loss of CFTC approval. But not all that long ago, another US prediction market, Polymarket, fell under the wrath of the CFTC, and so the CFTC's decision this time gives Kalshi, a new & slick prediction market, a state-enforced monopoly. The prediction market community, led by (a) the Star Spangled Gamblers, asked "who benefits from this recent decision", and the answer is clearly Kalshi. And so Star Spangled Gamblers is calling for a boycott (a) of Kalshi (a).
Note that Star Spangled Gamblers is incentivized to spin a nice story, though. My independent opinion is that there is a fair chance that Kalshi caused the previous Polymarket fall from grace in the eyes of the CFTC (60%), and a lower chance that they caused this one (40%). One would think that much like the origins of covid, this is an unresolvable question. But someone filed a Freedom of Information request (a) about the CFTC's decision, so eventually, this will be resolved.
Politically, my guess is that this decision made the couple thousand or so most politically sophisticated people in the planet significantly more Republican. As a reminder, these people can predict elections. And this can be transformed into power by using that predictive ability to identify and lobby in the political districts that are more swingable, i.e., in which the probabilities are closest to 50%.
"Metaculus et al." published a paper in the Lancet (a) about their monkeypox predictions.
Metaculus received a $5M grant from Open Philanthropy (a). Congratulations to Metaculus! Separately, note that Open Philanthropy's "hits based giving" (a) approach means that this grant isn't all that much evidence of Metaculus' formidability, so I contend that readers' opinions of Metaculus should not move much as a result of this grant. Previously, INFER received a similarly sized grant (a).
Perhaps because my team, Samotsvety, was still showing on the global leaderboard due to our strong performance in the first seasons, INFER has now changed the leaderboard rules. From their email newsletter:
We also made an update to the leaderboard. To ensure we're surfacing the top forecasters, the leaderboard will now default to showing only forecasters who have forecasted in at least 5 scored questions this season.
In happier news, congratulations to EA Delhi for winning the EA forecasting tournament (a) hosted on INFER.
We saw a nice Polymarket flip (a) as Celsius, a crypto company, declared bankruptcy on the same night a question about it resolved.
Domah writes about an attempted prediction market scam (a). Worth a read.
Manifold Markets implements limit orders (a), and will have a bot trading contest (a). A Manifold user also writes up a bot fight (a) from the perspective of both participants: note how this type of writeup doesn't seem to exist for Polymarket or Kalshi because the profit motive there is much greater.
They also interview Joel Becker (a), a former top-leaderboard-spot holder.
The Quantified Uncertainty Research Institute, the NGO for which I work, recently released an "early access" version of Squiggle (a), a language for probabilistic estimation, as well as a $1k prize for experimentation using it (a). We are also hiring for various positions (a).
For reference, past examples of work which have used an early version of Squiggle include:
The Swift Centre (a) is a new forecasting initiative supported by a $2M grant from the FTX Future Fund (a). Sam Atis interviews Michael Story, the Swift Centre's director, here (a):
During the pandemic, Dominic Cummings said some of the most useful stuff that he received and circulated in the British government was not forecasting, it was qualitative information explaining the general model of what’s going on, which enabled decision-makers to think more clearly about their options for action and the likely consequences. If you’re worried about a new disease outbreak, you don’t just want a percentage probability estimate about future case numbers, you want an explanation of how the virus is likely to spread, what you can do about it, how you can prevent it. Not the best estimate for how many COVID cases there will be in a month, but why forecasters believe there will be X COVID cases in a month. Getting this contextual and decision-relevant information right is my full-time focus at the Swift Centre and I think it’s where we have the best chance of making forecasting as useful as it can be.
True to that view, the Swift Centre recently added a quantified writeup about Germany's dependence on Russian oil (a), and are looking to hire forecasters (a).
An underdog programmer recently published some extremely nice (a) visualizations of existential risk:
Ryan Enos answers a question on Reddit (a) about a prediction market on whether any of his papers will be retracted (a). In his answer, he mentions that "the person who set up that market is a blogger that has a rather unhinged obsession with me and my research and constructed some rather odd conspiracy theories about me and some of my colleagues". Said allegedly "unhinged blogger" is none other than right wing investigative journalist Chris Brunett (a), whose answer can be read here (a).
The Diff has a good profile (a) on Jane Street, one of the top quantitative trading firms in the world.
Aver, a prediction market startup backed by Serum, itself backed by FTX, launched on the Solana chain mainnet. Their frontpage can be seen here (a) (could be slow to load), and here is a market on the next UK prime minister.
On the more speculative and perhaps scammy intersection of prediction markets and crypto, Floorprice (a) is a new platform for predicting the least amount any item of various NFT collections will sell for.
Coinbase, one of the biggest companies trying to make crypto accessible to the masses, is planning to add Gnosis (a) to their list of available tokens. This is surprising to me, because I'm not aware of anything interesting that Gnosis had been recently doing, and thought that the project was half-dead.
Facebook might have been looking to invest on (a) Better Opinions, an Indian prediction markets startup. I previously covered the legal shift that enabled Indian prediction markets here, and Facebook's failure with its own forecasting efforts here. In the end, it seems that Better opinions chose to raise $2.5M from other investors (a) instead.
The Economist has a new piece on software developers[who] aspire to forecast who will win a battle (a), covering some modeling in the field of war. h/t Jonathan Nankivell.
Bloomberg profiles Black Swan Hedge Funds, that is, funds structured to pay out in the event of strong negative swings.
A data science consultancy around the Turing.jl probabilistic programming language is looking for media partners to acquire access (a) to their election forecasting models. They seem like the real deal.
The Wall Street Journal writes about how "Our Recession Forecasting Model Is Broken" (a). Meanwhile, forecasters at Good Judgment Open (a), Metaculus (a) or Kalshi (a) are still going at it.
Voice of America (the Russia Today of the US) reports (a) that Yellowstone National Park forecasts grossly underestimated floods.
The Yellowstone National Park area's weather forecast the morning of June 12 seemed fairly tame: warmer temperatures and rain showers would accelerate mountain snow melt and could produce "minor flooding." A National Weather Service bulletin recommended moving livestock from low-lying areas but made no mention of danger to people
Torrents of water poured off the mountains. Swollen rivers carrying boulders and trees smashed through Montana towns over the next several days. The flooding swept away houses, wiped out bridges and forced the evacuation of more than 10,000 tourists, park employees and residents near the park.
As a cleanup expected to last months grinds on, climate experts and meteorologists say the gap between the destruction and what was forecast underscores a troublesome aspect of climate change: Models used to predict storm impacts do not always keep up with increasingly devastating rainstorms, hurricanes, heat waves and other events.
That last quoted paragraph is interesting to me. I used to think that ascribing specific weather events to climate change was an exaggeration or rhetorical move which reflected badly on those making it, like the German Tagesschau (a). And the article does later mention that "while no single weather event can be conclusively tied to climate change, scientists said the Yellowstone flooding was consistent with changes already documented around the park as temperatures warm". But the pathway to harm where climate change leads to worse forecasting which leads to more damage seems more plausible.
Bank of Canada blames oil price shifts for inflation forecasting errors (a):
Stéfane Marion, chief economist with National Bank, said the Bank of Canada’s assessment of its own forecasting errors needs to be taken with a grain of salt. "I think it’s a bit too easy to blame external factors for two-thirds of your miss," he said in an interview. "There’s a serious lack of transparency when it comes to their analysis of the Canadian labour markets."
The Guardian covered polling firms tweaking their poll results (a), and an Essex academic commented on how reliable polls aiming to predict diferent time frames are (a).
Kudos to Charlie McCarthy for covering the changing odds of 2024 US Republican primary candidates (a) on Newsmax, and to A. G. Gancarski for covering a potentially crowded presidential race (a) in a Florida newspaper.
A paper by Nick Otis (paper (a), slides (a)) aggregates previous studies which look at the ability of crowds of experts to predict which of a pair of policies will be most effective in randomized trials.
Robin Hanson advocates for branding truth narrowly (a), that is, coming up with knowledge creation mechanisms that only produce the kinds of knowledge that people are willing to accept.
Few people want truth in general. Yes decision theory says that people want truth near their decisions, and want it more the biggest their decision. But there are many kinds of truths that they positively do not want, and many more truths where their generally positive value for truth is below its cost of production.
...In fact, most of the claims on most of these prediction market sites are actually of this sort: general world events, politics, and celebrity gossip topics. Topics where people care a bit about truth, all else equal, but aren’t much willing to pay to improve on the level of truth that results from the usual news, gossip, and punditry on such topics.
...these good people have failed to create a brand to distinguish their superior truth product
...A solution here I think is: application-specific prediction market brands. For example, a brand that specializes in estimating the chance of making project deadlines, sold to orgs that actually want to know if they will make their deadlines. Or a brand that specializes in estimating the two-year-later employee evaluation that each new hire candidate would have if hired, sold to orgs that actually want to evaluate new hires.
Readers might enjoy an old cost-effectiveness analysis of a proposed new London airport (a), via StatsModelling
A well-known early discussion of the problems involved in these large cost–benefit exercises is John Adams’ "Westminster: the fourth London airport" (Area, 1970). Adams considered the report of the Roskill Commission on the Third London Airport, and argued that there was so much flexibility in choosing which costs and benefits to model that the outcome was arbitrary. He demonstrated (satirically) that, according to the Commission’s own choices of costs and benefits, Westminster in central London was a better location for the airport than the recommendation (Cublington).
Scott Sumner writes about Why macro forecasting is difficult (a), pointing out that policy interventions make past signals "anti-inductive":
- Much of what we are asked to predict represents policy failures. Not all predictions; it is certainly possible to predict a healthy economy. But the predictions that people value most are policy failures, such as a surge in inflation or the timing of the next deep recession.
- We often forecast by looking at past patterns in the data. We say, "The last time X happened, the economy experienced Y." Importantly, "X" is almost always public information.
- Policymakers are generally trying to prevent policy failures, and rely on public information.
A 2017 post (a) by Caspar Oesterheld explains that Futarchy (a) implements evidential decision theory (a):
…futarchy rewards traders based on how accurately they predict what is actually going to happen if the agent makes a particular choice. This leads the traders to estimate the value of an action as proportional to the expected utility conditional on that action since conditional probabilities are the correct way to make predictions.
For an example, consider the idea of "Fire the CEO" markets: prediction markets predict what the stock price of a company would be if the CEO was fired. The problem with this is that in the scenario where the CEO was fired, say, Elon Musk was fired as Tesla's CEO, that could be indicative that the company isn't doing very well.
In the CEO market, the way to fix this would be to fire the CEO with 1% probability, and pay 100x the amount to prediction market players who hold the shares of the correct outcome—e.g., whether Tesla's share price would grow if Musk is fired.
I asked forecasters from Samotsvety whether they wanted to write a short column on a topic of interest to readers for my newsletter, and an anonymous forecaster known as belikewater answered with the following long piece in what I can only describe as a labor of love. I present it here with some very light editing, noting that it has a strong personal component:
On May 7, 2022, the UK Health Services Agency announced (a) a confirmed case of monkeypox in a traveler returning from Nigeria. Since then, we have watched monkeypox cases spread around the world and wondered where the current outbreak is headed. Within less than two weeks, a series of forecast questions on Metaculus (a) began to be launched. On May 27, Good Judgment Open (GJO) launched a question (a) about the monkeypox outbreak, asking how many cases around the world would be recorded by global.health by the end of August.
In the early days of the outbreak, there was great uncertainty about exactly how quickly and widely cases were spreading. Health professionals quickly noticed that nearly all reported cases were in men who have sex with men (MSM), but many people wondered whether most cases were seen in MSM because almost no one else was being tested. Data gradually emerged that supported the hypothesis that most or nearly all actual infections are currently occurring in MSM, and the data now very clearly show this.
As of August 3, 2022, 97.5% (7328/7514) (a) of all cases reported to the WHO with known data on sexual orientation identified as MSM, which the WHO defines as "homosexual or bisexual males in reporting forms," and of those, 1.0% (73/7328) (a) were identified as bisexual men. In addition, 37.6% (2,979/7,924) (a) of those with known HIV status were HIV-positive. The most common setting in which cases were likely exposed was a large event with sexual contacts; for 21.7% (277/1277) (a) of cases for which a likely exposure setting was known, such an event was considered to be the most likely exposure setting.
Whenever positive test rates have been reported, the rate has always been much higher for men than for women. In the UK, for example, only 2.2% (a) of women tested have been positive for monkeypox, compared to 54% (a) of men. Moreover, sustained transmission outside the MSM community has not been clearly demonstrated to date, in the UK (a) or elsewhere. However, anyone (a)can get (a) monkeypox, and many are concerned about the possibility of greater spread beyond the MSM community.
Once it became clear to many that most monkeypox cases were among MSM, many forecasters then wondered how large the most at-risk portion of that community might be, because that would be a major determinant of how large the outbreak might become in any given region region within any specific time frame. Many thought that the portion of the MSM network with frequent new partners was likely small enough that the outbreak could potentially be contained relatively quickly or might die out on its own. A few (myself included) overestimated the connectedness of the MSM social network and the extent to which monkeypox would spread rapidly through the MSM community.
Early consensus that the outbreak would be relatively small shifted towards higher numbers, as reported cases continued to grow approximately exponentially. However, over time, case growth began to slow, first in the countries in Europe in which monkeypox had first been detected outside endemic regions, and then elsewhere. It became clear that the portion of the MSM community through which monkeypox could spread rapidly represents a small fraction of the MSM community as a whole.
Forecasters at GJO are all but certain (a) that there will be between 25,000 and 100,000 monkeypox cases reported worldwide by August 31, 2022. The Metaculus community currently forecasts that a mean of 260k infections (a) will be have occurred worldwide before 2023, with a broad range of 91k to 960k estimated to encompass the middle 50% of the probability distribution. A substantial portion of the uncertainty in such numbers relates to large uncertainties about the percentages of infections that will be detected and reported in different countries worldwide. Metaculus forecasters also estimate that about 541 deaths (a) (lower 25%, 194; upper 75%, 1.6k) will be estimated to have been caused by monkeypox worldwide before 2023. The current Metaculus consensus is that there is a 60% chance (a) (median; 61% mean) that by 2023, the US CDC will recommend vaccination for all MSM, and that there is a 23% chance (a) (median and mean) that by 2023, the US CDC will recommend use of a smallpox/monkeypox vaccine for at least 10% of the US population.
Most (a) forecasters (a) did not expect the WHO to declare the current monkeypox outbreak a PHEIC so soon, or perhaps at all, as its Director-General, Tedros Ghebreyesus, did on July 23 (a). Indeed, the WHO Emergency Committee (EC) itself was split in its recommendation, with nine members recommending against a PHEIC declaration and only six recommending that one be declared; for the first time since the PHEIC was created, the Director-General declared a PHEIC against the EC's recommendation. While this virus has so far caused few deaths, it is nonetheless of great concern because of the severe pain, tissue damage and encephalitis it causes in some cases and because of its economic impacts on individuals and communities. The PHEIC declaration should help to facilitate worldwide efforts to reduce monkeypox spread.
Many questions remain about the future of the global monkeypox outbreak, and forecasters' opinions about the long-term future of the monkeypox outbreak diverge widely. Some forecasters hope that the outbreak might be contained or might die out after many MSM are vaccinated or the virus burns through the most connected elements of MSM networks worldwide; some of these forecasters think that it is most likely that the outbreak will not lead to monkeypox becoming endemic in humans. Some forecasters also emphasize that government officials and world leaders could bring the outbreak under control and end it if they chose to do so.
Other forecasters are far less optimistic, myself included. I think that monkeypox is here to stay. Genetic evidence (a) strongly suggests that the main strain currently circulating has been circulating continuously for the past several years in West Africa; because it has been circulating continuously in one region for several years, I see no reason why it will not continue to circulate worldwide for many more years to come. I think monkeypox will continue to spread if it is not actively controlled, and I do not see the level of worldwide, concerted effort to trace contacts, isolate cases, and ring vaccinate that I think would likely be necessary to end the outbreak.
Moreover, I think that if the outbreak were to continue anywhere unabated, then new outbreaks would continuously be seeded globally by people traveling from regions with ongoing outbreaks. Nonetheless, many of us hope that the virus might one day be eliminated in many countries through vaccination. However, the possibility has also been raised that monkeypox could spill (a) over (a) into animal populations outside regions in which monkeypox has been endemic and that such a development could make the virus much harder to eliminate within individual countries or regions because of the risk of reverse infections from animals.
Many people are also concerned about potential changes in transmissibility over time due to seasonality and viral evolution. Currently, neither fomite nor aerosol transmission causes a substantial fraction of cases, but I think it is possible that aerosol transmission could cause a small percentage of cases in winter. Over time, I think it is likely that the virus will evolve to become more transmissible, as has been seen with SARS-CoV-2. Moreover, I think it is possible that its transmissibility could increase for more than one route of transmission—direct contact (e.g., sexual contact), indirect contact (e.g., fomite) and/or airborne—as was the case with smallpox transmission. We will continue to see the virus evolve (a), and over time, we may also see the emergence of distinct new lineages of the virus, with their attendant risks.
The extent to which asymptomatic transmission currently occurs is yet unknown, but it is possible that the frequency of asymptomatic transmission could change over time. If the virus were to become more transmissible, it could circulate among broader segments of society; if the virus were to circulate among children, in particular, then based on an analysis (a) of monkeypox cases in Nigeria, it is possible that a higher percentage of children with monkeypox than of adults with monkeypox could become more seriously ill.
It remains to be seen how the current monkeypox outbreak will unfold over the long term, but the likely broad outlines of the near future are becoming clearer. Collectively, forecasters have interpreted the slowing growth in reported cases to mean that while individual countries will experience brief periods of roughly exponential growth in infections, there will not be an exponential explosion of cases or infections worldwide over the coming months and that most cases will continue to occur via direct contact in the MSM community, with some spillover into the broader community. The extent of that spillover remains to be seen. In the short term, increasing vaccination of those most at risk, together with at least some contact tracing and isolation, should help to reduce spread. In the long term, we can hope that monkeypox-specific vaccines will be developed and distributed worldwide.
Finally, it should be noted that many have been concerned about the possibility that reporting that nearly all cases are occurring within the MSM community would lead to stigmatizing those who become infected. However, Kai Kupferschmidt, a reporter who writes about infectious diseases and other topics for Science magazine and who is himself in the MSM community, has argued (a) that, "Any successful response to an outbreak needs to be grounded in facts, and the facts are clear." Mr. Kupferschmidt also writes that it is important "to engage the communities that are most at risk," and that, "To control the outbreak, those most vulnerable to infection need to have information that allows them to make decisions to stay healthy until there are enough vaccine doses available." Let us hope that MSM communities worldwide can be provided with information and resources that will help to reduce the number of monkeypox infections, and the human suffering that accompany them, worldwide
Note to the future: All links are added automatically to the Internet Archive, using this tool (a). "(a)" for archived links was inspired by Milan Griffes (a), Andrew Zuckerman (a), and Alexey Guzey (a).
I need not tell you, gentlemen, that the world situation is very serious. That must be apparent to all intelligent people. I think one difficulty is that the problem is one of such enormous complexity that the very mass of facts presented to the public by press and radio make it exceedingly difficult for the man in the street to reach a clear appraisement of the situation. Furthermore, the people of this country are distant from the troubled areas of the earth and it is hard for them to comprehend the plight and consequent reactions of the long-suffering peoples, and the effect of those reactions on their governments in connection with our efforts to promote peace in the world.
In considering the requirements for the rehabilitation of Europe…
~ George Marshall, the "Marshall Plan" speech, 5 June 1947