EA Forum Bot Site
EA Forum

Hide table of contents
Tools for collaborative truth seeking
by brook, ChanaMessinger
4 min read 0

19

ForecastingTools for collaborative truth-seeking
Frontpage

Forecasting tools and prediction markets allow you to state your credence in something precisely and view others' credences. This can be an important way of gathering information from a group or sharing information precisely between individuals. 

This post will introduce a few tools for generating and sharing predictions. Many of these tools are excellent; they've been grouped together because of their similarity and number. 

One way I think forecasting tools can be useful for collaborative truth-seeking is that spending some time using them can improve both your calibration and your ability to precisely define questions. I think this can then greatly improve your ability to communicate your implicit beliefs explicitly in conversation or text. I'd be interested to hear other people's opinions on this view! 

Forecasting tools and putting precise numbers on predictions in general is important because it lets a community approach truth over time better by letting people have a more transparent track record to learn who is in fact good at predicting.

This post will discuss the following tools:

  • Metaculus 
  • Manifold Markets
  • Pastcasting
  • Confido
  • PredictionBook
  • Elicit

When can forecasting/prediction tools be useful?

  • Practice predicting so that you're better able to communicate your beliefs verbally (especially PredictionBook, Pastcasting)
  • Get community predictions relevant to deciding where to donate, e.g.:
    •  AI milestones
    • Cultured meat
    • Factory farming
    • Global poverty
    • Politics 
  • Make a private question to ask people in your organisation to predict project outcomes or engagement (Confido)
  • Add a prediction question on your post to see if people's views change after reading (Elicit)
  • Find out how people feel about a controversial topic (with people who feel more strongly able to influence the result more; e.g. here)
  • Learn to forecast (especially Pastcasting)
  • Perform curious self-referential research on prediction markets[1] (Manifold Markets)

Metaculus

Metaculus is a reputation-based site for soliciting and aggregating predictions.

The official Metaculus tutorials are suitable for beginners as a way to introduce forecasting, probability distributions, and a basic understanding of how to behave under uncertainty. 

Tutorial #1 is over-explaining for those with a basic understanding of the forecasting, but still a good introduction to using Metaculus specifically. 

For those familiar with forecasting, tutorials #2 and #3 are more useful as a (small) amount of calibration practice and building familiarity with the Metaculus interface. 

Examples

Note that you can embed metaculus questions in forum posts by posting the direct link into the editor:

Manifold Markets 

Manifold Markets is a play money prediction market with user-created questions. The play money can be redeemed as a charitable donation. Because it works as a market, it can be used in lots of interesting ways-- you can ask people to predict which opportunities your organisation will fund (as one factor to influence your decision, maybe), you can play chess, and you can even implement futarchy.

A text tutorial is available on the wiki here. 

Examples

Note that you can embed Manifold Markets questions into forum posts by pasting the direct link into the editor:

Other Tools

Pastcasting is a forecasting platform using resolved questions from other sites (e.g. Metaculus) to shorten the feedback loop and practice forecasting, particularly over long time horizons. The official FAQ is a great introduction. In partcular, I'd like to highlight Pastcasting's feature Vantage Search:

Vantage Search is a custom search engine that only shows results from before the vantage point to prevent information leaks from the future.

Confido is tool that provides an instance of a closed forecasting platform. Forecasts are entered similarly to on metaculus, but questions are intended for a single group or organisation. This is useful for questions requiring insider knowledge like "How will the EA conference we are planning go" or "How many employees will we have by the end of the year". Trying the demo version is a good way to understand the tool.

PredictionBook is simple tool for prediction by entering a % credence only. You write can write a question, put the date when you should know the true answer by, and enter a credence. On the date you entered, you'll be notified to give the true answer. 

Predictions can be public or private; I find it's particularly useful for producing personal questions "in 2023, will I go 60 days without allergy symptoms?" as a way of improving your calibration (including an automatic calibration graph[2]). 

Elicit Forecast is a tool that lets you easily set up  questions people can put credences on, and shows an aggregate of them, but is currently unsupported[3]. At present, only binary questions (with a % chance input) work as far as I can tell; date or number estimate questions are currently non-functioning. There is no guarantee any functionality will continue. 

Binary elicit forecasts currently work as embeds:

1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
toribrk (15%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
Saul Munn (56%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
Will Howard🔹 (66%)
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
Will the US retail price of lemons be above $2.10 on 01-12-23?
99%

Personal Experience

I don't honestly use prediction or forecasting tools much in my day-to-day life[4]. If this were to change I think I'd be most likely to use PredictionBook to improve my calibration on everyday questions that I need to be able to predict, or manifold markets simply for fun. 

I think if you're regularly making big decisions like grant making or running projects in AI, biosecurity, etc., forecasting tools like Metaculus or Manifold could be useful, particularly the comments (where other people explain their reasoning). Similarly, I think Confido would be useful for a sufficiently-sized organisation. 

Elicit is useful because of its ability to be embedded & interacted with in forum posts. If somebody picks up supporting the tool & fixes the current issues, I imagine I'd add it to a lot more forum posts. 

Try it Yourself!

You can access the below market to buy shares yourself here.

I'd also suggest making an account on PredictionBook and formalising 3 predictions that you're currently making implicitly! Here are some suggestions to get the ball rolling:

  • The completion date of a current project
  • How much money you'll donate to charity before 31/12/2023
  • How many people will read your next forum post
  • How successful you will be with your new year's resolution
  • Your total spending this month
  • What time you'll get home today

Tomorrow at 6pm GMT we'll also be running a short event  in the EA GatherTown to discuss forecasting tools & do a short exercise about them!

Also, join us at 6pm GMT today in the GatherTown to discuss the use of Squiggle!

 

  1. ^

    "This market resolves to NO if there's at least two people holding 157 YES shares, otherwise it resolves YES.", Manifold Markets

  2. ^

    i.e. a graph of "% you assigned to an outcome" against "% chance that that actually happened" (which would be the line y=x if you were perfectly calibrated)

  3. ^

    Private communication, Ought

  1. ^

    Though we did get feedback from people who do on various drafts of this post, don't worry!

Show all footnotes

19

0
0

Reactions

0
0
Previous:
Excalidraw: Why and How to Use it
14 comments29 karma
Next:
Squiggle: Why and how to use it
4 comments44 karma
Mentioned in
Comments


No comments on this post yet.
Be the first to respond.
More from brook
35
Small EA Groups and Invisible Impact
brook
· · 2m read
3
3
82
Announcing Introductions for Collaborative Truth Seeking Tools
2 authors
· · 3m read
7
7
44
Squiggle: Why and how to use it
2 authors
· · 3m read
4
4
View more
Curated and popular this week
Paul Present
 ·  · 28m read
 · 
Note: I am not a malaria expert. This is my best-faith attempt at answering a question that was bothering me, but this field is a large and complex field, and I’ve almost certainly misunderstood something somewhere along the way. Summary While the world made incredible progress in reducing malaria cases from 2000 to 2015, the past 10 years have seen malaria cases stop declining and start rising. I investigated potential reasons behind this increase through reading the existing literature and looking at publicly available data, and I identified three key factors explaining the rise: 1. Population Growth: Africa's population has increased by approximately 75% since 2000. This alone explains most of the increase in absolute case numbers, while cases per capita have remained relatively flat since 2015. 2. Stagnant Funding: After rapid growth starting in 2000, funding for malaria prevention plateaued around 2010. 3. Insecticide Resistance: Mosquitoes have become increasingly resistant to the insecticides used in bednets over the past 20 years. This has made older models of bednets less effective, although they still have some effect. Newer models of bednets developed in response to insecticide resistance are more effective but still not widely deployed.  I very crudely estimate that without any of these factors, there would be 55% fewer malaria cases in the world than what we see today. I think all three of these factors are roughly equally important in explaining the difference.  Alternative explanations like removal of PFAS, climate change, or invasive mosquito species don't appear to be major contributors.  Overall this investigation made me more convinced that bednets are an effective global health intervention.  Introduction In 2015, malaria rates were down, and EAs were celebrating. Giving What We Can posted this incredible gif showing the decrease in malaria cases across Africa since 2000: Giving What We Can said that > The reduction in malaria has be
Rory Fenton
 ·  · 6m read
 · 
Cross-posted from my blog. Contrary to my carefully crafted brand as a weak nerd, I go to a local CrossFit gym a few times a week. Every year, the gym raises funds for a scholarship for teens from lower-income families to attend their summer camp program. I don’t know how many Crossfit-interested low-income teens there are in my small town, but I’ll guess there are perhaps 2 of them who would benefit from the scholarship. After all, CrossFit is pretty niche, and the town is small. Helping youngsters get swole in the Pacific Northwest is not exactly as cost-effective as preventing malaria in Malawi. But I notice I feel drawn to supporting the scholarship anyway. Every time it pops in my head I think, “My money could fully solve this problem”. The camp only costs a few hundred dollars per kid and if there are just 2 kids who need support, I could give $500 and there would no longer be teenagers in my town who want to go to a CrossFit summer camp but can’t. Thanks to me, the hero, this problem would be entirely solved. 100%. That is not how most nonprofit work feels to me. You are only ever making small dents in important problems I want to work on big problems. Global poverty. Malaria. Everyone not suddenly dying. But if I’m honest, what I really want is to solve those problems. Me, personally, solve them. This is a continued source of frustration and sadness because I absolutely cannot solve those problems. Consider what else my $500 CrossFit scholarship might do: * I want to save lives, and USAID suddenly stops giving $7 billion a year to PEPFAR. So I give $500 to the Rapid Response Fund. My donation solves 0.000001% of the problem and I feel like I have failed. * I want to solve climate change, and getting to net zero will require stopping or removing emissions of 1,500 billion tons of carbon dioxide. I give $500 to a policy nonprofit that reduces emissions, in expectation, by 50 tons. My donation solves 0.000000003% of the problem and I feel like I have f
LewisBollard
 ·  · 8m read
 · 
> How the dismal science can help us end the dismal treatment of farm animals By Martin Gould ---------------------------------------- Note: This post was crossposted from the Open Philanthropy Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post. ---------------------------------------- This year we’ll be sharing a few notes from my colleagues on their areas of expertise. The first is from Martin. I’ll be back next month. - Lewis In 2024, Denmark announced plans to introduce the world’s first carbon tax on cow, sheep, and pig farming. Climate advocates celebrated, but animal advocates should be much more cautious. When Denmark’s Aarhus municipality tested a similar tax in 2022, beef purchases dropped by 40% while demand for chicken and pork increased. Beef is the most emissions-intensive meat, so carbon taxes hit it hardest — and Denmark’s policies don’t even cover chicken or fish. When the price of beef rises, consumers mostly shift to other meats like chicken. And replacing beef with chicken means more animals suffer in worse conditions — about 190 chickens are needed to match the meat from one cow, and chickens are raised in much worse conditions. It may be possible to design carbon taxes which avoid this outcome; a recent paper argues that a broad carbon tax would reduce all meat production (although it omits impacts on egg or dairy production). But with cows ten times more emissions-intensive than chicken per kilogram of meat, other governments may follow Denmark’s lead — focusing taxes on the highest emitters while ignoring the welfare implications. Beef is easily the most emissions-intensive meat, but also requires the fewest animals for a given amount. The graph shows climate emissions per tonne of meat on the right-hand side, and the number of animals needed to produce a kilogram of meat on the left. The fish “lives lost” number varies significantly by
Relevant opportunities
15
Evals projects I'd like to see, and a call to apply to OP's evals RFP
cb
· · 4m read
2
2
20
Hiring pre-docs
Eva
· · 1m read
0
0
15
Apply to the Cambridge ERA:AI Fellowship 2025
Harrison 🔸
· · 3m read
0
0
View more