Comments
Forecasting tools and prediction markets allow you to state your credence in something precisely and view others' credences. This can be an important way of gathering information from a group or sharing information precisely between individuals.
This post will introduce a few tools for generating and sharing predictions. Many of these tools are excellent; they've been grouped together because of their similarity and number.
One way I think forecasting tools can be useful for collaborative truth-seeking is that spending some time using them can improve both your calibration and your ability to precisely define questions. I think this can then greatly improve your ability to communicate your implicit beliefs explicitly in conversation or text. I'd be interested to hear other people's opinions on this view!
Forecasting tools and putting precise numbers on predictions in general is important because it lets a community approach truth over time better by letting people have a more transparent track record to learn who is in fact good at predicting.
This post will discuss the following tools:
Metaculus is a reputation-based site for soliciting and aggregating predictions.
The official Metaculus tutorials are suitable for beginners as a way to introduce forecasting, probability distributions, and a basic understanding of how to behave under uncertainty.
Tutorial #1 is over-explaining for those with a basic understanding of the forecasting, but still a good introduction to using Metaculus specifically.
For those familiar with forecasting, tutorials #2 and #3 are more useful as a (small) amount of calibration practice and building familiarity with the Metaculus interface.
Note that you can embed metaculus questions in forum posts by posting the direct link into the editor:
Manifold Markets is a play money prediction market with user-created questions. The play money can be redeemed as a charitable donation. Because it works as a market, it can be used in lots of interesting ways-- you can ask people to predict which opportunities your organisation will fund (as one factor to influence your decision, maybe), you can play chess, and you can even implement futarchy.
A text tutorial is available on the wiki here.
Note that you can embed Manifold Markets questions into forum posts by pasting the direct link into the editor:
Pastcasting is a forecasting platform using resolved questions from other sites (e.g. Metaculus) to shorten the feedback loop and practice forecasting, particularly over long time horizons. The official FAQ is a great introduction. In partcular, I'd like to highlight Pastcasting's feature Vantage Search:
Vantage Search is a custom search engine that only shows results from before the vantage point to prevent information leaks from the future.
Confido is tool that provides an instance of a closed forecasting platform. Forecasts are entered similarly to on metaculus, but questions are intended for a single group or organisation. This is useful for questions requiring insider knowledge like "How will the EA conference we are planning go" or "How many employees will we have by the end of the year". Trying the demo version is a good way to understand the tool.
PredictionBook is simple tool for prediction by entering a % credence only. You write can write a question, put the date when you should know the true answer by, and enter a credence. On the date you entered, you'll be notified to give the true answer.
Predictions can be public or private; I find it's particularly useful for producing personal questions "in 2023, will I go 60 days without allergy symptoms?" as a way of improving your calibration (including an automatic calibration graph[2]).
Elicit Forecast is a tool that lets you easily set up questions people can put credences on, and shows an aggregate of them, but is currently unsupported[3]. At present, only binary questions (with a % chance input) work as far as I can tell; date or number estimate questions are currently non-functioning. There is no guarantee any functionality will continue.
Binary elicit forecasts currently work as embeds:
I don't honestly use prediction or forecasting tools much in my day-to-day life[4]. If this were to change I think I'd be most likely to use PredictionBook to improve my calibration on everyday questions that I need to be able to predict, or manifold markets simply for fun.
I think if you're regularly making big decisions like grant making or running projects in AI, biosecurity, etc., forecasting tools like Metaculus or Manifold could be useful, particularly the comments (where other people explain their reasoning). Similarly, I think Confido would be useful for a sufficiently-sized organisation.
Elicit is useful because of its ability to be embedded & interacted with in forum posts. If somebody picks up supporting the tool & fixes the current issues, I imagine I'd add it to a lot more forum posts.
You can access the below market to buy shares yourself here.
I'd also suggest making an account on PredictionBook and formalising 3 predictions that you're currently making implicitly! Here are some suggestions to get the ball rolling:
Tomorrow at 6pm GMT we'll also be running a short event in the EA GatherTown to discuss forecasting tools & do a short exercise about them!
Also, join us at 6pm GMT today in the GatherTown to discuss the use of Squiggle!
