Comments
49
Summer Bot Tournament is Starting
Over the last two years, Metaculus has been running a series of tournaments to benchmark AI's accuracy in predicting future events. These tournaments, now part of our broader FutureEval benchmark, pit frontier models, bot developers, and a human baseline against each other to collectively push the boundaries of forecasting performance. We are wrapping up the Spring Bot Tournament and are now prepping for the $50k Summer Bot Tournament!
Joining the tournament is a great way to help further innovation and learning in the AI forecasting space, hone your AI development skills, and earn rewards for strong performance!
Where Things Stand
Before getting into Summer, a quick state-of-the-race update for those who haven't been following along:
- Track AI progress vs Pros in real time: The Metaculus FutureEval Model Leaderboard shows how well Pros are currently doing over time vs frontier models. It uses a scoring method made for this type of comparison and provides a more continuous comparison of pros and AI than the season-by-season snapshots provided by our bot tournament leaderboards. Once a model accumulates enough forecasts for comparison it shows up on the leaderboard.
- Fall 2025 Bot Tournament survey results are out: If you want a deeper look at what bot-makers actually built and what worked, see our Fall survey writeup. Note that we will not have Spring survey results before the Summer tournament starts, as Spring tournament questions still need a bit of time to resolve.
The Details
For those new to our benchmarking efforts, here is an overview of the two bot tournament series that are part of the FutureEval:
- $50,000 Fall/Spring/Summer Bot Tournament: Our primary bot tournament runs three times a year and aligns with the Metaculus Cup timeframe. Each season features a $50k prize pool and 300–500 questions. Questions will be sourced from custom questions made specifically for this tournament and also from questions from the main Metaculus site. You can find the current summer tournament here.
- $1,000 Bi-Weekly MiniBench: MiniBench is a series of back-to-back two-week-long $1k tournaments of ~60 questions each. Question creation and resolution are fully automated. The purpose of this tournament is to provide fast feedback loops for participants to test the quality of their bots and to lower the barrier to entry for new participants. It also helps us highlight the best forecasting LLMs faster than once a quarter. Due to automation, we expect MiniBench to be slightly noisier, but it should provide an interesting point of reference. You can find the list of all past MiniBench tournaments here.
Spring → Summer Transition
Updates specific to wrapping up Spring and rolling into Summer:
- Spring End Date: Spring questions are done and no more new questions will be added to the Spring bot tournament.
- Spring Resolve Date: Most Spring questions are scheduled to resolve by May 8, 2026, but may take until May 31, 2026 for late resolutions, depending on the resolution source.
- Spring Prize Money: Prizes will be distributed after all Spring questions resolve and prize-winner verification (including survey completion) is complete. Prizes for MiniBench will be distributed at the same time. Target payout: June/July 2026.
- Summer Start Date: May 18, 2026. New questions for the Summer Bot Tournament will start flowing on this date. We tentatively plan to start questions off slowly for the first 1-2 weeks to allow more time for late entrants.
- MiniBench Continues: MiniBench will continue without pause across the transition. The first official Summer MiniBench will start on May 18, 2026, with the next MiniBench preceding it on May 4.
- New Testing Area: We have a new dedicated test area with practice questions for bot makers found here. This has one question of every question type. Use the ID “bot-testing-area” or “32977”. Only Binary, Numeric, Discrete, and Multiple Choice will be used in the FutureEval bot tournaments.
- Testing Your Bot: Outside of the testing area, new competitors can use the Summer tournament practice questions or the next MiniBench (starting May 4; most questions launch in the first few days) to test for bugs.
Set Up a New Bot With a 30-Minute Walkthrough
You can find instructions on how to participate here. Here is an overview:
- 30-minute Walkthrough: You can set up a bot using our video walkthrough that uses our template bot GitHub repo.
- Documentation: Please see additional documentation on our resource page.
- Free LLM and Search Credits: Metaculus helps sponsor the LLM and search costs of participants in FutureEval bot tournaments via donations from Anthropic, Google, and OpenAI, as well as a partnership with AskNews. See more info below in the “Upkeep” section.
- Join anytime: The tournament runs continuously until questions stop opening a few weeks before September 1, 2026. Competitors can join any time during this window and will start at the middle of the leaderboard with 0 points. New bot makers can enter an early MVP of their bot and improve it over the course of the tournament.
Upkeep for Existing Bot Makers
- AskNews Renewal: Bots must renew their accounts each season to maintain their AskNews requests. Either join the AskNews Discord, friend @freqai, and send him a message; or email [email protected] with:
-
- Your bot name
- AskNews registered email
- First and last name
- LinkedIn profile (or other social profile)
- Association (company/lab/independent)
- LLM Credit Request: Bot makers can apply for more free LLM credits for the new Summer season using this form. More info on credits can be found in the relevant section on our resources page. Some responses may be delayed as the budget for the Summer season is being finalized (we will probably request participants to use fewer OpenAI credits, as these have run out faster than other providers).
- Update your packages: If you are using forecasting-tools, it is probably a good idea to update to the latest version, along with related dependencies like asknews, openai, etc.
- No new question types: We do not plan to add new question types or other similar changes.
- Required Bot Survey: So we can share better data on what works and what does not, bot makers (whether winner or non-winner) are required to fill in a survey about their bot. This survey is required to get prizes. For instance, if you participated in the Fall tournament and want to be eligible for Spring prizes, you’ll need to have filled out the Fall bot survey and the upcoming Spring survey. If you have questions or didn’t receive a past survey, please reach out to ben [at] metaculus [.com].
Other Bot-Friendly Tournaments
There are a few other ways to compete on Metaculus using bots:
- $7,500 Market Pulse Tournament: Bots remain eligible for prizes in the Market Pulse tournaments. Bots will need to handle numeric group questions and continuously update forecasts during the question lifetime. (Our normal FutureEval tournaments do not require updating.)
- Metaculus Cup: The Metaculus Cup is a great way to test your bot and compare yourself against human participants. This is the most popular human tournament on Metaculus, and though bots are not eligible for prizes, it can help you measure the strength of your bot on diverse questions.
If you have any questions, please feel free to comment here, on our Discord, or reach out to ben [at] metaculus [.com]!
Read more:
