Metaculus FluSight Challenge 2023/24 Winners + Forecasting Impact

Metaculus forecaster impact in our 3rd annual FluSight Challenge: 

• Strengthened CDC flu forecasts
• Helped train mechanistic models
• 1 successful abstract submission by our partner, Dr. Thomas McAndrew, who leads Lehigh University's Computational Uncertainty Lab
• Sped progress on 2 manuscripts 

The flu burden has subsided for the season and with it the third annual FluSight Challenge has concluded. The top three prize-winners earning a share of the $5,000 prize pool are:

1. SpottedBear – $3,432
2. skmmcj – $914
3. datscilly – $297

Congratulations to the top three winners – who also earned the gold, silver, and bronze medals in the tournament – and congratulations to all the prizewinners!

Our returning partner for this project, Dr. Thomas McAndrew, who leads Lehigh University's Computational Uncertainty Lab, shared the following:

We wish to thank everyone who submitted forecasts of the peak timing and intensity of influenza hospitalizations throughout this 2023/24 season. Forecasts supported training a mechanistic model to generate forecasts for all 50 states in the country. Your contributions have led to a successful abstract submission (below), and this work has spurred two manuscripts: one focused on a computational technique to extend forecasts of peak timing/intensity from 10 states to all 50 states; and a second focused on how to train a mechanistic model on human judgment ensemble forecasts of peak timing/intensity. Your work here builds upon previous human judgment-support mechanistic modeling on a synthetic outbreak. Thank you again for all the hard work and dedication during the influenza season.

Abstract: For the 2022/2023 season, influenza accounted for 31m cases, 360k hospitalizations and 21k deaths, costing US healthcare ~$4.6 billion. Forecasts of influenza hospitalizations provide health officials with advanced warning. Prior research has revealed that forecasts generated by human judgment are similar in accuracy to mathematical models. However, little work has compared an equally weighted vs performance-weighted human judgment ensemble. We collected weekly human judgment forecasts of the peak number of incident hospitalizations (peak intensity) and the epidemic week where this peak occurs (peak time) for 10 US states from Oct 2023 to March 2024. The median, 25th, and 75th percentiles were extracted for each week. We found that, for the performance-weighted ensemble, forecast uncertainty decreased just before and after peak intensity and peak timing. However, uncertainty for the equally-weighted ensemble does not decrease. For both ensembles, the median prediction of peak intensity is smaller than the truth at the beginning of the season and approaches the truth monotonically. For peak time, the performance-weighted ensemble median prediction is later than the truth and approaches the truth only after the true peak time is observed. We have observed that the performance-weighted ensemble tends to produce more accurate forecasts of peak timing and intensity compared to an equally-weighted ensemble. One potential mechanism for this boost in accuracy was that the performance-weighted ensemble correctly assigned large weights to experienced forecasters and small weights to inexperienced forecasters. Experienced forecasters may be needed when asked to predict peak timing/intensity of an ongoing outbreak.

You can see more details and the full leaderboard for the FluSight Challenge 2023/24 here.

We will be in touch with prize winners to arrange payments. We are also happy to facilitate donations to the organizations listed here.

Congratulations again to SpottedBear, skmmcj, and datscilly!

Metaculus launches round 2 of the Chinese AI Chips Tournament

Help bring clarity to key questions in AI governance and support research by the Institute for AI Policy and Strategy (IAPS).

Start forecasting on new questions tackling broader themes of Chinese AI capability like: 

Will we see a frontier Chinese AI model before 2027?

Will a Chinese firm order a large number of domestic AI chips?

Will a Chinese firm order a large number of US or US-allied AI chips?

Metaculus is conducting its first user survey in nearly three years. If you have read analyses, consumed forecasts, or made predictions on Metaculus, we want to hear from you! Your feedback helps us better meet the needs of the forecasting community and is incredibly important to us. 

Take the short survey here — we truly appreciate it! (We'll be sure to share what we learn.) 

Metaculus introduces 'Changed my mind' button

This short take is a linkpost for this Discussion Post by Metaculus's Technical Product Manager

• Do you sometimes read a comment so good that you revise your whole world model and start predicting the opposite of what you believed before?
• Do you ever read a comment and think “Huh. Hadn’t thought of that.” and then tweak your prediction by a few percentage points?
• Do you ever read a comment so clearly wrong that you update in the opposite direction?
• Do you ever wish you could easily tell other forecasters that what they share is valuable to you?
• Do you ever want to update you prediction right after reading a comment, without getting RSI in your scrolling finger?

Did a comment change your mind? Give Metaculus's new 'Changed my mind' button a click! 

And for binary questions, clicking the button lets you update your prediction directly from the comment. 

🔬 Metaculus Officially Launches API
Metaculus has officially launched the Metaculus API, providing access to a rich, quantitative database of aggregate forecasts on 7000+ questions. Start exploring, analyzing, and building here.

🔎 Discover More Questions Relevant to You With Metaculus's New Filter & Sort Tools

Where do you disagree with other forecasters? Which community predictions have shifted the most? And what was that nanotech forecast you meant to update? Metaculus has introduced new filter & sort tools that provide more control over the forecast feed so you can find the questions that matter to you.

Learn more

New Metaculus Feature: Reaffirming Predictions
Now you can more easily find predictions that need updating and reaffirm those you stand by. Learn more about this new feature here.

Metaculus is excited to announce the winners of the inaugural Keep Virginia Safe Tournament! This first-of-its kind collaboration with the University of Virginia (UVA) Biocomplexity Institute and the Virginia Department of Health (VDH) delivered forecasting and modeling resources to public health professionals and public policy experts as they have navigated critical decisions on COVID-19.

Congratulations to the top 3 prize winners!

1. Sergio 
2. 2e10e122
3. mattvdm

"On behalf of the Virginia Department of Health, I’d like to offer our congratulations to Sergio, 2e10e122, mattvdm, and all the forecasters who participated in the Keep Virginia Safe Forecasting tournament! Because of your work, Virginia was better able to navigate the Delta and Omicron waves and prepare for the long-term impacts of COVID-19, helping us keep Virginians safe during a critical period."

—Justin Crow, Foresight & Analytics Coordinator for the Virginia Department of Health

Thank you to forecasting community! Your predictions were integrated into VDH planning sessions and were shared with local health department staff, statewide epidemiologists, and even with the Virginia Governor’s office.

For more details on the tournament outcomes, visit the project summary.

Our successful partnership with UVA and VDH continues through the Keep Virginia Safe II Tournament, where Metaculus forecasts continue to provide valuable information. Join to help protect Virginians and compete for $20,000 in prizes. 

Find more information about the Keep Virginia Safe Tournament, including the complete leaderboard, here.

Now You Can Create Multiple Choice Questions on Metaculus

Create multiple choice questions and bring greater clarity to topics with multiple potential outcomes where one and only one will occur.

To get started, simply Create a Question and set the Question Type to 'multiple choice'.

Give the Group Variable a clear label, e.g., 'Option', 'Team', 'Country'.

Fill in the Multiple Choice Options, adding more fields as needed.

After you share additional details including background information on your topic, we'll be excited to review and publish your multiple choice question!

Embed Interactive Metaculus Forecasts on Your Website or Blog

Now you can share interactive Metaculus forecasts for all question types, including Question Groups and Conditional Pairs. 

Just click the 'Embed' button at the top of a question page, customize the plot as needed, and copy the iframe html. 

Improved Forecast Previews

Metaculus has also made it easier to share forecast preview images for more question types, on platforms like Twitter, Substack, Slack, and Facebook.

Just paste the question URL to generate a preview of the forecast plot on any platform that supports them.

To learn more about embedding forecasts & preview images, click here. 

Enhanced Usability for Metaculus's Binary Question Groups

We've upgraded Metaculus's binary question groups to make forecasting faster and more efficient. Each question is equipped with its own slider, which means fewer steps to make your forecasts. 

Try the new interface on one of these question groups: 

Will China Invade Taiwan?

Who Will Be Elected To Be US President in 2024?

Will Israeli Forces Reach Gaza City Hall?

Will These Countries See Civil War by 2036?

🕛 New Metaculus Feature: Now You Can Set Custom Forecast Notifications

Metaculus has upgraded its forecast notifications to make it easier to follow the questions you care about so you can keep your predictions fresh and stay up to date as stories develop. Set a single notification that alerts you whenever the Metaculus community's prediction shifts, when new comments are made, when a question nears its close date, or at regular intervals over a question's life. Learn more about forecast notifications and how you can use them here. 

Metaculus Concludes Nuclear Risk Tournament Supporting Rethink Priorities

Metaculus has concluded the Nuclear Risk Tournament, supporting Rethink Priorities' efforts by helping to inform funding, policy, research, and career decisions aimed at reducing existential risk. 

Thank you to Rethink Priorities for sponsoring the tournament, thank you to the forecasters who contributed their talents, and congratulations to the tournament winners. 

Learn more

Winners of the China and Global Cooperation Tournament

The China and Global Cooperation Tournament has ended, thank you to everyone who forecasted! Here are the top three finishers:

1. twsummer – $687
2. Ab5A8bd20V – $194
3. Inertia – $80

The China and Global Cooperation Tournament featured questions related to the future of China and its role in the world stage. Participants forecasted on questions such as the global uptake of China’s currency, the number of Japanese Air Force responses to Chinese aircraft, and the combined market cap of large Chinese companies.

Join Metaculus for Forecast Friday, April 28th at 12pm ET

Are you interested in how top forecasters predict the future? Curious how other people are reacting to the forecasts in the main feed?

Join us April 28th at 12pm ET/GMT-4 for Forecast Friday!

Click here to go to Gather Town. Then take the Metaculus portal.

This Friday

Seasoned forecaster and Product Manager for Metaculus Sylvain Chevalier will present for this week's Forensic Friday session. 

Now You Can Submit Conditional Pair Questions on Metaculus

Curious to learn the likelihood of an event given the outcome of another? Now you can create conditional pairs of binary questions and invite the forecasting community to share their predictions to clarify the relationships between events.

Not yet familiar with Metaculus's conditional pairs? Read the guide here or watch this introductory video to get up to speed.

Metaculus: Winners of the Global Pulse Tournament + Forecaster Impact

The Global Pulse Tournament has concluded: Forecasters contributed to two academic studies conducted by Dr. Philipp Schoenegger, Associate Professor Barbara Fasolo, and Associate Professor Matteo Galizzi, all at the Behavioural Lab at the London School of Economics. 
 

Congratulations to the top three winners:

1. skmmcj – $233
2. datscilly – $231
3. SpottedBear – $110

Dr. Schoenegger shared the following about the research the tournament supported:

I wanted to thank all of you for your hard work in this tournament! It was really amazing to follow the forecasts and discussions over the past few months (even though I, at times, really wanted to jump in and forecast myself)!

We used the forecasts you generated in two academic studies, where we presented a report made up of of your forecasts to two samples of decision-makers who were forecasting on questions that were directly connected to but distinct from those asked in the tournament. While I am not yet able to share results as we are still in the analysis phase, I can tell you that we wouldn't have been able to study this in such a thorough way without the actual tournament data you provided, so thank you!

And last but not least, congratulations to the winners!

You can find more details about the research shared by Dr. Schoenegger here. You can also see more details and the full leaderboard for the Global Pulse Tournament here.

Embed and Share All Metaculus Question Types

Now you can share and embed all question types, including question groups without fan graphs and all conditionals. Simply paste a question URL on your social platform of choice to generate a preview image:

You can also paste URLs to embed plots here on the Effective Altruism Forum and on Substack: 

Want to embed forecasts on other sites? You can access the embed code for a given question by navigating to the question page, making sure the browser window is wide enough to include the sidebar, and clicking the Embed button. 

Now you can explore more relationships between more forecast questions on Metaculus, with conditional pairs that feature question group subquestions. To submit your own: 

1. Click 'Write a Question' on Metaculus
2. Select 'conditional pair' as the Question Type
3. Click 'Select Parent' and/or 'Select Child'
4. Search for the subquestion name, which will be indicated in parentheses after the group name, or paste in the URL of the subquestion

Note: You can copy the URL of a subquestion by visiting the question group page, clicking the ‘...’ more options menu, and selecting the ‘Copy Link’ option next to the subquestion you’re focused on.

In addition to submitting a new conditional yourself, you can also request questions here in the new question discussion post.

Get Started

Here are some new subquestion conditional pairs to start forecasting on: