Good question. I think AI researchers views inform/can inform me. A few examples from the recent NLP Community Metasurvey. I would quote bits from this summary.

Few scaling maximalists: 17% agreed that Given resources (i.e., compute and data) that could come to exist this century, scaled-up implementations of established existing techniques will be sufficient to practically solve any important real-world problem or application in NLP.

This was surprsing and updated me somewhat against shorter timelines (and higher risk) as, for example, it clashes with the "+12 OOMS Enough" premise of the Kokotajlo's argument for short timelines of the Carlsmith report (see also this and his review of Carlsmith report).

NLP is on a path to AGI: 58% agreed that Understanding the potential development of artificial general intelligence (AGI) and the benefits/risks associated with it should be a significant priority for NLP researchers. Related: 57% agreed that Recent developments in large-scale ML modeling (such as in language modeling and reinforcement learning) are significant steps toward the development of AGI.

If these numbers were significantly lower or higher, it would also probably update my views.

AGI could be catastrophic: 36% agreed that It is plausible that decisions made by AI or machine learning systems could cause a catastrophe this century that is at least as bad as an all-out nuclear war. 46% of women and 53% of URM respondents agreed. The comments suggested that people took a pretty wide range of interpretations to this, including things like OOD robustness failures leading to weapons launches.

This number is puzzling and hard to interpret. It seems appropriate in light of AI Impacts' What do ML researchers think about AI in 2022? where "48% of respondents gave at least 10% chance of an extremely bad outcome".

I don't fully understand what this implies about the ML community's views on AI alignment. But I can see myself updating positively if these concerns would lead to more safety culture, alignment research, etc.

Note that we'd probably also look at the object level reasons for why they think that. E.g., new scaling laws findings could definitely shift our/my forecast by 10%.

Fair question. I say little weight but if it was far enough from my view I would update a little. My view also may not be representative of other forecasters, as is evident from Misha's comment.

For example is there any level of consensus against ~AGI by 2070 (or some other date) that would be strong enough to move your forecast by 10 percentage points?

From the original Grace et al. survey (and I think the more recent ones as well? but haven't read as closely) the ML researchers clearly had very incoherent views depending on the question being asked and elicitation techniques, which I think provides some evidence they haven't thought about it that deeply and we shouldn't take it too seriously (some incoherence is expected, but I think they gave wildly different answers for HLMI (human-level machine intelligence) and full automation of labor).

So I think I'd split up the thresholds by somewhat coherent vs. still very incoherent.

My current forecast for ~AGI by 2100 barring pre-AGI catastrophe is 80%. To move it to 70% based just a survey of ML experts, I think I'd have to see something like one of:

1. ML experts still appear to be very incoherent, but are giving a ~10% chance of ~AGI by 2100 on average across framings.
2. ML experts appear to be somewhat coherent, and are giving a ~25% chance of ~AGI by 2100.

(but I haven't thought about this a lot, these numbers could change substantially on reflection or discussion/debate)

Yup, I linked the text "APS-AI" in the post to the relevant section of the report. Sorry if it wasn't that noticeable!

Thanks for the suggestion and glad you found the forecasts helpful :)

I personally have a distaste for academic credentialist culture so am probably not the best person to turn this into a more prestigious looking report. I agree it might be valuable despite my distaste, so if anyone reading this is interested in doing so feel free to DM me and I can probably help with funding and review if you have a good writing track record.

In terms of forecasting accuracy on Metaculus, Eli's individual performance is comparable^[1] to the community aggregate on his own, despite him having optimised for volume (he's 10th on the heavily volume weighted leaderboard). I expect that were he to have pushed less hard for volume, he'd have significantly outperformed the community aggregate even as an individual.^[2]

Assuming the other Samotsvety forecasters are comparably good, I'd expect the aggregated forecasts from the group to very comfortably outperform the community aggregate, even if they weren't paying unusual attention to the questions (which they are).

1. ^{^}

   Comparing 'score at resolution time', Eli looks slightly worse than the community. Comparing 'score across all times', Eli looks better than the community. Score across all times is a better measure of skill when comparing individuals, but does disadvantage the community prediction, because at earlier times questions have fewer predictors.

2. ^{^}

   As some independent evidence of this, I comfortably outperform the community aggregate, having tried less hard than Eli to optimise for volume. Eli has beaten me in more than one competition, and think he's a better forecaster.

In addition to the points above, there have been a few jokes on questions like that about the scoring rule not being proper (if the world ends, you don't get the negative points for being wrong!). Not sure how much of a factor that is, though, and I could imagine it being minimal.

My take is that we should give little weight to Metaculus. From footnote 2 of the post:

Why do I give little weight to Metaculus’s views on AI? Primarily because of the incentives to make very shallow forecasts on a ton of questions (e.g. probably <20% of Metaculus AI forecasters have done the equivalent work of reading the Carlsmith report), and secondarily that forecasts aren’t aggregated from a select group of high performers but instead from anyone who wants to make an account and predict on that question.

(Edited to add: I see the post you linked also includes the "Metaculus prediction" which theoretically performs significantly better than the community prediction by weighting stronger predictors more heavily. But if you look at its actual track record, it doesn't do much better than the community.  For binary questions at resolve time, it has a log score of 0.438 vs. 0.426 for community. At all times, it gets 0.280 vs. 0.261.  For continuous questions at resolve time, it has a log score of 2.19 vs. 2.12. At all times, it gets 1.57 vs. 1.55.)

 That said:

Or is forecasting risks just really super unreliable and not a thing to put much weight on?

I wouldn't want people to overestimate the precision of the estimates in this post! Take them as a few data points among many. I also think it's very healthy for the community if many people are forming inside views about AI risk, though I understand it's difficult and had a hard time with it myself for a while.

More generally, though probably especially on AI. I'm not exactly sure how to handle some of the evidence for why I'm pretty confident on this, but I can gesture at a few data points that seem fine to share:

1. Nuño, Misha and I each did much better on INFER than at least a few superforecasters (and we were 3 of the top 4 on the leaderboard; no superforecasters did better than us).
2. I've seen a few superforecaster aggregate predictions that seemed pretty obviously bad in advance, which I pre-registered and in both cases I was right.
   1. I said on Twitter I thought supers were overconfident on Russia invasion of Ukraine; supers were at 84% no invasion, l was at 55% (and Metaculus was similar, to be fair). Unfortunately, Russia went on to invade Ukraine a few weeks later.
   2. I noticed that superforecasters may be overconfident on the rise of the Delta COVID variant, and it seems they likely were: they predicted a 14% chance 7-day median would rise above 140k cases and a 2% (!) chance it would rise above 210k; it ended up peaking at about 150k so this is weak evidence, but it still seems like 2% was crazy low.

I'm hesitant to make it seem like I'm bashing on superforecasters, I think they're amazing overall relative to the general public. But I think Samotsvety is even more amazing :D I also think some superforecasters are much better forecasters than others.

Good idea, I made a very quick version. Anyone should feel free to update it.

Includes the highest and lowest

To add to Eli's comment, I think on such complex topics, it's just common for even personal estimates to fluctuate quite a bit. For example, here is an excerpt from footnote 181 of Carlsmith report:

[...] And my central estimate varies between ~1-10% depending on my mood, what considerations are salient to me at the time, and so forth. This instability is yet another reason not to put too much weight on these numbers.

My impression is wide ranges are pretty common on questions as difficult/complex as these. I think large differences can often come from very-hard-to-resolve deep disagreements in intuitions, as we've seen with the MIRI conversations.

If the range was too small on this type of question I might be worried about herding/anchoring. In a few cases there are 1-2 outlier forecasts on each end and the rest are relatively close together.

I believe your linked text for existential catastrophe (in the second table) is incorrect- I get a page not found error.

Thanks, fixed.

Substantively, I realize this is probably not something you originally asked (nor am I asking for it since presumably this’d take a bunch of time), but I’d be super curious to see what kind of uncertainty estimates folks put in this, and how combining using those uncertainties might look. If you have some intuition on what those intervals look like, that’d be interesting.

The reason I’m curious about this is probably fairly transparent, but given the pretty extensive broader community uncertainty on the topic, aggregating using those might yield a different point estimate, but more importantly might help people understand the problem better by seeing the large degree uncertainty involved. For example, it’d be interesting/useful to see how much probability people put outside a 10-90% range.

Good question! Perhaps we can include better uncertainty information in a future post at some point. For now, regarding my personal uncertainty, I'll quote what I wrote about a week ago:

[Regarding chance of misaligned takeover by 2100] I’m going to stick to 35% for now, but it’s a very tough question and I could see ending up at anywhere between 10-90% on further reflection and discussion.

[Regarding chance of TAI by 2100] Mashing all these intuitions together gives me a best guess of 80%, though I think I could end up at anywhere between 60% and 90% on further reflection and discussion.