Hide table of contents

Forecasting tools and prediction markets allow you to state your credence in something precisely and view others' credences. This can be an important way of gathering information from a group or sharing information precisely between individuals. 

This post will introduce a few tools for generating and sharing predictions. Many of these tools are excellent; they've been grouped together because of their similarity and number. 

One way I think forecasting tools can be useful for collaborative truth-seeking is that spending some time using them can improve both your calibration and your ability to precisely define questions. I think this can then greatly improve your ability to communicate your implicit beliefs explicitly in conversation or text. I'd be interested to hear other people's opinions on this view! 

Forecasting tools and putting precise numbers on predictions in general is important because it lets a community approach truth over time better by letting people have a more transparent track record to learn who is in fact good at predicting.

This post will discuss the following tools:

  • Metaculus 
  • Manifold Markets
  • Pastcasting
  • Confido
  • PredictionBook
  • Elicit

When can forecasting/prediction tools be useful?

  • Practice predicting so that you're better able to communicate your beliefs verbally (especially PredictionBook, Pastcasting)
  • Get community predictions relevant to deciding where to donate, e.g.:
    •  AI milestones
    • Cultured meat
    • Factory farming
    • Global poverty
    • Politics 
  • Make a private question to ask people in your organisation to predict project outcomes or engagement (Confido)
  • Add a prediction question on your post to see if people's views change after reading (Elicit)
  • Find out how people feel about a controversial topic (with people who feel more strongly able to influence the result more; e.g. here)
  • Learn to forecast (especially Pastcasting)
  • Perform curious self-referential research on prediction markets[1] (Manifold Markets)

Metaculus

Metaculus is a reputation-based site for soliciting and aggregating predictions.

The official Metaculus tutorials are suitable for beginners as a way to introduce forecasting, probability distributions, and a basic understanding of how to behave under uncertainty. 

Tutorial #1 is over-explaining for those with a basic understanding of the forecasting, but still a good introduction to using Metaculus specifically. 

For those familiar with forecasting, tutorials #2 and #3 are more useful as a (small) amount of calibration practice and building familiarity with the Metaculus interface. 

Examples

Note that you can embed metaculus questions in forum posts by posting the direct link into the editor:

Manifold Markets 

Manifold Markets is a play money prediction market with user-created questions. The play money can be redeemed as a charitable donation. Because it works as a market, it can be used in lots of interesting ways-- you can ask people to predict which opportunities your organisation will fund (as one factor to influence your decision, maybe), you can play chess, and you can even implement futarchy.

A text tutorial is available on the wiki here

Examples

Note that you can embed Manifold Markets questions into forum posts by pasting the direct link into the editor:

Other Tools

Pastcasting is a forecasting platform using resolved questions from other sites (e.g. Metaculus) to shorten the feedback loop and practice forecasting, particularly over long time horizons. The official FAQ is a great introduction. In partcular, I'd like to highlight Pastcasting's feature Vantage Search:

Vantage Search is a custom search engine that only shows results from before the vantage point to prevent information leaks from the future.

Confido is tool that provides an instance of a closed forecasting platform. Forecasts are entered similarly to on metaculus, but questions are intended for a single group or organisation. This is useful for questions requiring insider knowledge like "How will the EA conference we are planning go" or "How many employees will we have by the end of the year". Trying the demo version is a good way to understand the tool.

PredictionBook is simple tool for prediction by entering a % credence only. You write can write a question, put the date when you should know the true answer by, and enter a credence. On the date you entered, you'll be notified to give the true answer. 

Predictions can be public or private; I find it's particularly useful for producing personal questions "in 2023, will I go 60 days without allergy symptoms?" as a way of improving your calibration (including an automatic calibration graph[2]). 

Elicit Forecast is a tool that lets you easily set up  questions people can put credences on, and shows an aggregate of them, but is currently unsupported[3]. At present, only binary questions (with a % chance input) work as far as I can tell; date or number estimate questions are currently non-functioning. There is no guarantee any functionality will continue. 

Binary elicit forecasts currently work as embeds:

1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
toribrk (15%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
Saul Munn (56%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
Will the US retail price of lemons be above $2.10 on 01-12-23?
99%

Personal Experience

I don't honestly use prediction or forecasting tools much in my day-to-day life[4]. If this were to change I think I'd be most likely to use PredictionBook to improve my calibration on everyday questions that I need to be able to predict, or manifold markets simply for fun. 

I think if you're regularly making big decisions like grant making or running projects in AI, biosecurity, etc., forecasting tools like Metaculus or Manifold could be useful, particularly the comments (where other people explain their reasoning). Similarly, I think Confido would be useful for a sufficiently-sized organisation. 

Elicit is useful because of its ability to be embedded & interacted with in forum posts. If somebody picks up supporting the tool & fixes the current issues, I imagine I'd add it to a lot more forum posts. 

Try it Yourself!

You can access the below market to buy shares yourself here.

I'd also suggest making an account on PredictionBook and formalising 3 predictions that you're currently making implicitly! Here are some suggestions to get the ball rolling:

  • The completion date of a current project
  • How much money you'll donate to charity before 31/12/2023
  • How many people will read your next forum post
  • How successful you will be with your new year's resolution
  • Your total spending this month
  • What time you'll get home today

Tomorrow at 6pm GMT we'll also be running a short event  in the EA GatherTown to discuss forecasting tools & do a short exercise about them!

Also, join us at 6pm GMT today in the GatherTown to discuss the use of Squiggle!

 

  1. ^
  2. ^

    i.e. a graph of "% you assigned to an outcome" against "% chance that that actually happened" (which would be the line y=x if you were perfectly calibrated)

  3. ^

    Private communication, Ought

  4. ^

    Though we did get feedback from people who do on various drafts of this post, don't worry!

  5. Show all footnotes
Comments


No comments on this post yet.
Be the first to respond.
Curated and popular this week
 ·  · 4m read
 · 
TLDR When we look across all jobs globally, many of us in the EA community occupy positions that would rank in the 99.9th percentile or higher by our own preferences within jobs that we could plausibly get.[1] Whether you work at an EA-aligned organization, hold a high-impact role elsewhere, or have a well-compensated position which allows you to make significant high effectiveness donations, your job situation is likely extraordinarily fortunate and high impact by global standards. This career conversations week, it's worth reflecting on this and considering how we can make the most of these opportunities. Intro I think job choice is one of the great advantages of development. Before the industrial revolution, nearly everyone had to be a hunter-gatherer or a farmer, and they typically didn’t get a choice between those. Now there is typically some choice in low income countries, and typically a lot of choice in high income countries. This already suggests that having a job in your preferred field puts you in a high percentile of job choice. But for many in the EA community, the situation is even more fortunate. The Mathematics of Job Preference If you work at an EA-aligned organization and that is your top preference, you occupy an extraordinarily rare position. There are perhaps a few thousand such positions globally, out of the world's several billion jobs. Simple division suggests this puts you in roughly the 99.9999th percentile of job preference. Even if you don't work directly for an EA organization but have secured: * A job allowing significant donations * A position with direct positive impact aligned with your values * Work that combines your skills, interests, and preferred location You likely still occupy a position in the 99.9th percentile or higher of global job preference matching. Even without the impact perspective, if you are working in your preferred field and preferred country, that may put you in the 99.9th percentile of job preference
 ·  · 5m read
 · 
Summary Following our co-founder Joey's recent transition announcement we're actively searching for exceptional leadership to join our C-level team and guide AIM into its next phase. * Find the full job description here * To apply, please visit the following link * Recommend someone you think could be a great fit here * Location: London strongly preferred. Remote candidates willing to work from London at least 3 months a year and otherwise overlapping at least 6 hours with 9 am to 5 pm BST will be considered. We are happy to sponsor UK work visas. * Employment Type: Full-time (35 hours) * Application Deadline: rolling until August 10, 2025 * Start Date: as soon as possible (with some flexibility for the right candidate) * Compensation: £45,000–£90,000 (for details on our compensation policy see full job description) Leadership Transition On March 15th, Joey announced he's stepping away from his role as CEO of AIM, with his planned last day as December 1st. This follows our other co-founder Karolina's completed transition in 2024. Like Karolina, Joey will transition to a board member role while we bring in new leadership to guide AIM's next phase of growth. The Opportunity AIM is at a unique inflection point. We're seeking an exceptional leader to join Samantha and Devon on our C-level team and help shape the next era of one of the most impactful organizations in the EA ecosystem. With foundations established (including a strong leadership team and funding runway), we're ready to scale our influence dramatically and see many exciting pathways to do so. While the current leadership team has a default 2026 strategic plan, we are open to a new CEO proposing radical departures. This might include: * Proposing alternative ways to integrate or spin off existing or new programs * Deciding to spend more resources trialling more experimental programs, or double down on Charity Entrepreneurship * Expanding geographically or deepening impact in existing region
 ·  · 6m read
 · 
I am writing this to reflect on my experience interning with the Fish Welfare Initiative, and to provide my thoughts on why more students looking to build EA experience should do something similar.  Back in October, I cold-emailed the Fish Welfare Initiative (FWI) with my resume and a short cover letter expressing interest in an unpaid in-person internship in the summer of 2025. I figured I had a better chance of getting an internship by building my own door than competing with hundreds of others to squeeze through an existing door, and the opportunity to travel to India carried strong appeal. Haven, the Executive Director of FWI, set up a call with me that mostly consisted of him listing all the challenges of living in rural India — 110° F temperatures, electricity outages, lack of entertainment… When I didn’t seem deterred, he offered me an internship.  I stayed with FWI for one month. By rotating through the different teams, I completed a wide range of tasks:  * Made ~20 visits to fish farms * Wrote a recommendation on next steps for FWI’s stunning project * Conducted data analysis in Python on the efficacy of the Alliance for Responsible Aquaculture’s corrective actions * Received training in water quality testing methods * Created charts in Tableau for a webinar presentation * Brainstormed and implemented office improvements  I wasn’t able to drive myself around in India, so I rode on the back of a coworker’s motorbike to commute. FWI provided me with my own bedroom in a company-owned flat. Sometimes Haven and I would cook together at the residence, talking for hours over a chopping board and our metal plates about war, family, or effective altruism. Other times I would eat at restaurants or street food booths with my Indian coworkers. Excluding flights, I spent less than $100 USD in total. I covered all costs, including international transportation, through the Summer in South Asia Fellowship, which provides funding for University of Michigan under