EA Forum Bot Site
EA Forum

Hide table of contents
Tools for collaborative truth seeking
by brook, ChanaMessinger
4 min read 0

19

ForecastingTools for collaborative truth-seeking
Frontpage

Forecasting tools and prediction markets allow you to state your credence in something precisely and view others' credences. This can be an important way of gathering information from a group or sharing information precisely between individuals. 

This post will introduce a few tools for generating and sharing predictions. Many of these tools are excellent; they've been grouped together because of their similarity and number. 

One way I think forecasting tools can be useful for collaborative truth-seeking is that spending some time using them can improve both your calibration and your ability to precisely define questions. I think this can then greatly improve your ability to communicate your implicit beliefs explicitly in conversation or text. I'd be interested to hear other people's opinions on this view! 

Forecasting tools and putting precise numbers on predictions in general is important because it lets a community approach truth over time better by letting people have a more transparent track record to learn who is in fact good at predicting.

This post will discuss the following tools:

  • Metaculus 
  • Manifold Markets
  • Pastcasting
  • Confido
  • PredictionBook
  • Elicit

When can forecasting/prediction tools be useful?

  • Practice predicting so that you're better able to communicate your beliefs verbally (especially PredictionBook, Pastcasting)
  • Get community predictions relevant to deciding where to donate, e.g.:
    •  AI milestones
    • Cultured meat
    • Factory farming
    • Global poverty
    • Politics 
  • Make a private question to ask people in your organisation to predict project outcomes or engagement (Confido)
  • Add a prediction question on your post to see if people's views change after reading (Elicit)
  • Find out how people feel about a controversial topic (with people who feel more strongly able to influence the result more; e.g. here)
  • Learn to forecast (especially Pastcasting)
  • Perform curious self-referential research on prediction markets[1] (Manifold Markets)

Metaculus

Metaculus is a reputation-based site for soliciting and aggregating predictions.

The official Metaculus tutorials are suitable for beginners as a way to introduce forecasting, probability distributions, and a basic understanding of how to behave under uncertainty. 

Tutorial #1 is over-explaining for those with a basic understanding of the forecasting, but still a good introduction to using Metaculus specifically. 

For those familiar with forecasting, tutorials #2 and #3 are more useful as a (small) amount of calibration practice and building familiarity with the Metaculus interface. 

Examples

Note that you can embed metaculus questions in forum posts by posting the direct link into the editor:

Manifold Markets 

Manifold Markets is a play money prediction market with user-created questions. The play money can be redeemed as a charitable donation. Because it works as a market, it can be used in lots of interesting ways-- you can ask people to predict which opportunities your organisation will fund (as one factor to influence your decision, maybe), you can play chess, and you can even implement futarchy.

A text tutorial is available on the wiki here. 

Examples

Note that you can embed Manifold Markets questions into forum posts by pasting the direct link into the editor:

Other Tools

Pastcasting is a forecasting platform using resolved questions from other sites (e.g. Metaculus) to shorten the feedback loop and practice forecasting, particularly over long time horizons. The official FAQ is a great introduction. In partcular, I'd like to highlight Pastcasting's feature Vantage Search:

Vantage Search is a custom search engine that only shows results from before the vantage point to prevent information leaks from the future.

Confido is tool that provides an instance of a closed forecasting platform. Forecasts are entered similarly to on metaculus, but questions are intended for a single group or organisation. This is useful for questions requiring insider knowledge like "How will the EA conference we are planning go" or "How many employees will we have by the end of the year". Trying the demo version is a good way to understand the tool.

PredictionBook is simple tool for prediction by entering a % credence only. You write can write a question, put the date when you should know the true answer by, and enter a credence. On the date you entered, you'll be notified to give the true answer. 

Predictions can be public or private; I find it's particularly useful for producing personal questions "in 2023, will I go 60 days without allergy symptoms?" as a way of improving your calibration (including an automatic calibration graph[2]). 

Elicit Forecast is a tool that lets you easily set up  questions people can put credences on, and shows an aggregate of them, but is currently unsupported[3]. At present, only binary questions (with a % chance input) work as far as I can tell; date or number estimate questions are currently non-functioning. There is no guarantee any functionality will continue. 

Binary elicit forecasts currently work as embeds:

1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
toribrk (15%)
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
Saul Munn (56%)
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
Will Howard🔹 (66%)
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
Will the US retail price of lemons be above $2.10 on 01-12-23?
99%

Personal Experience

I don't honestly use prediction or forecasting tools much in my day-to-day life[4]. If this were to change I think I'd be most likely to use PredictionBook to improve my calibration on everyday questions that I need to be able to predict, or manifold markets simply for fun. 

I think if you're regularly making big decisions like grant making or running projects in AI, biosecurity, etc., forecasting tools like Metaculus or Manifold could be useful, particularly the comments (where other people explain their reasoning). Similarly, I think Confido would be useful for a sufficiently-sized organisation. 

Elicit is useful because of its ability to be embedded & interacted with in forum posts. If somebody picks up supporting the tool & fixes the current issues, I imagine I'd add it to a lot more forum posts. 

Try it Yourself!

You can access the below market to buy shares yourself here.

I'd also suggest making an account on PredictionBook and formalising 3 predictions that you're currently making implicitly! Here are some suggestions to get the ball rolling:

  • The completion date of a current project
  • How much money you'll donate to charity before 31/12/2023
  • How many people will read your next forum post
  • How successful you will be with your new year's resolution
  • Your total spending this month
  • What time you'll get home today

Tomorrow at 6pm GMT we'll also be running a short event  in the EA GatherTown to discuss forecasting tools & do a short exercise about them!

Also, join us at 6pm GMT today in the GatherTown to discuss the use of Squiggle!

 

  1. ^

    "This market resolves to NO if there's at least two people holding 157 YES shares, otherwise it resolves YES.", Manifold Markets

  2. ^

    i.e. a graph of "% you assigned to an outcome" against "% chance that that actually happened" (which would be the line y=x if you were perfectly calibrated)

  3. ^

    Private communication, Ought

  1. ^

    Though we did get feedback from people who do on various drafts of this post, don't worry!

Show all footnotes

19

0
0

Reactions

0
0
Previous:
Excalidraw: Why and How to Use it
14 comments29 karma
Next:
Squiggle: Why and how to use it
4 comments44 karma
Mentioned in
Comments


No comments on this post yet.
Be the first to respond.
More from brook
82
Announcing Introductions for Collaborative Truth Seeking Tools
2 authors
· · 3m read
7
7
44
Squiggle: Why and how to use it
2 authors
· · 3m read
4
4
35
Small EA Groups and Invisible Impact
brook
· · 2m read
3
3
View more
Curated and popular this week
 ·  · 5m read
 · 
[Cross-posted from my Substack here] If you spend time with people trying to change the world, you’ll come to an interesting conundrum: Various advocacy groups reference previous successful social movements as to why their chosen strategy is the most important one. Yet, these groups often follow wildly different strategies from each other to achieve social change. So, which one of them is right? The answer is all of them and none of them. This is because many people use research and historical movements to justify their pre-existing beliefs about how social change happens. Simply, you can find a case study to fit most plausible theories of how social change happens. For example, the groups might say: * Repeated nonviolent disruption is the key to social change, citing the Freedom Riders from the civil rights Movement or Act Up! from the gay rights movement. * Technological progress is what drives improvements in the human condition if you consider the development of the contraceptive pill funded by Katharine McCormick. * Organising and base-building is how change happens, as inspired by Ella Baker, the NAACP or Cesar Chavez from the United Workers Movement. * Insider advocacy is the real secret of social movements – look no further than how influential the Leadership Conference on Civil Rights was in passing the Civil Rights Acts of 1960 & 1964. * Democratic participation is the backbone of social change – just look at how Ireland lifted a ban on abortion via a Citizen’s Assembly. * And so on… To paint this picture, we can see this in action below: Source: Just Stop Oil which focuses on…civil resistance and disruption Source: The Civic Power Fund which focuses on… local organising What do we take away from all this? In my mind, a few key things: 1. Many different approaches have worked in changing the world so we should be humble and not assume we are doing The Most Important Thing 2. The case studies we focus on are likely confirmation bias, where
 ·  · 1m read
 · 
Although some of the jokes are inevitably tasteless, and Zorrilla is used to set up punchlines, I enjoyed it and it will surely increase concerns and donations for shrimp. I'm not sure what impression the audience will have of EA in general.  Last week The Daily Show interviewed Rutger Bregman about his new book Moral Ambition (which includes a profile of Zorrilla and SWP). 
 ·  · 2m read
 · 
Americans, we need your help to stop a dangerous AI bill from passing the Senate. What’s going on? * The House Energy & Commerce Committee included a provision in its reconciliation bill that would ban AI regulation by state and local governments for the next 10 years. * Several states have led the way in AI regulation while Congress has dragged its heels. * Stopping state governments from regulating AI might be okay, if we could trust Congress to meaningfully regulate it instead. But we can’t. This provision would destroy state leadership on AI and pass the responsibility to a Congress that has shown little interest in seriously preventing AI danger. * If this provision passes the Senate, we could see a DECADE of inaction on AI. * This provision also violates the Byrd Rule, a Senate rule which is meant to prevent non-budget items from being included in the reconciliation bill.   What can I do? Here are 3 things you can do TODAY, in order of priority: 1. (5 minutes) Call and email both of your Senators. Tell them you oppose AI preemption, and ask them to raise a point of order that preempting state AI regulation violates the Byrd Rule. * Find your Senators here. * Here’s an example of a call:  “Hello, my name is {YOUR NAME} and I’m a resident of {YOUR STATE}. The newest budget reconciliation bill includes a 10-year ban pre-empting state AI legislation without establishing any federal guardrails. This is extremely concerning to me – leading experts warn us that AI could cause mass harm within the next few years, but this provision would prevent states from protecting their citizens from AI crises for the next decade. It also violates the Byrd Rule, since preempting state AI regulation doesn’t impact federal taxes or spending. I’d like the Senator to speak out against this provision and raise a point of order that this provision should not be included under the Byrd Rule.” See here for sample call + email temp
Relevant opportunities
14
Project Proposal Looking for Feedback: Making Policy Impacts Transparent — A Reasoning Model for Trade, Jobs, and Prices
Echo Huang
· · 9m read
4
4
60
FLF Fellowship on AI for Human Reasoning: $25-50k, 12 weeks
Oliver Sourbut
·
2
2
19
Evals projects I'd like to see, and a call to apply to OP's evals RFP
cb
· · 4m read
2
2
View more