One thing I'd flag is that models are extremely good at telling who is prompting them, and this leads to them being sycophantic, in very subtle ways. I'm not quite sure how they do it, but I've seen this in multiple instances.

For starters, you'd want to lobby for external safety testing on internal models.  You'd want to make sure external safety researchers had access to the models. You'd want certain reporting, etc.

I think he would include a lot of people who work at Anthropic, for example, on pre-training, some of whom went through MATS or something.

Yea, this is fair. I am much more sympathetic to non-PM forecasting than I am PM/judgemental forecasting. The ideas in this post were really developed in 2023/2024 when I saw EAs spending a ton of time on Manifold/Metaculus, investing at high valuations, generally revering prediction markets for decision making, etc. whereas what I was seeing was completely different.

This post really belabours the first and second bullet point, perhaps because that is where a lot of money has gone to, but there can be a lot of value in the third.

I really believe in following the money. I think if we spend $100M on forecasting and $90M of it went to prediction market-style forecasting, I think it's fair to basically lump it all together. It'd be one thing if PMs were a small experiment within broad forecasting, but its been the main thing.

Hi Eva,

I think the Social Science Prediction Platform (alongside a friend of mine who is doing something similar for clinical trials) are among the more interesting uses of forecasting/PMs but I'm skeptical they will be uptaken to the degree/impact you might hope for.

do forecasts inform 1% of their funding or what?

I'm skeptical of things of the form "small percentage chance * big number". I think humans are really bad at estimating small percentages.

Would be happy to talk privately about any situations you are thinking of.

As promised, my reply (a couple days late).

1. I think it's a bit of a cop out to suggest that the money was spent poorly and therefore it's not fair to judge forecasting on the merits. Not sure if this is what you are saying though.
2. I'm not sure I agree. Blockchains were funded. Lots of academic research is funded by governments, etc.
3. By this logic, why aren't we paying people in the top, say 100 on Polymarket on non-sports questions and saying "hey guys, we want to pay you to make some forecasts for us".
4. I'm not sure how I feel about these. I will say, I am not nearly as big a fan of AI 2027 as others seem to be, and I think it is going to severely discredit AI risks because we have been crying wolf when frankly, most of the scary stuff they say doesn't happen. I am very happy to make some bets with Daniel or Eli on some of these (and give them extra time).
5. Useless is a stretch and I didn't intend to claim it originally. Sorry if I implied it. I merely think they are overrated, hence the title.

I agree that there is a lot of stuff being conflated in "forecasting". I suppose, I want to single out Prediction Markets and Judgemental Forecasting.

I think far more than $10M/year is going into forecasting. Many grants for forecasting are awarded outside the forecasting fund, such as the Navigating Transformative AI Fund. It depends on what you count, but I think it is closer to $25M/year.

I really question if people are really getting much, if anything, from all these forecasts that they didn't already have before.

Hi Josh, thanks for the response.

I hate to do this, especially at the start, but I want to point out for you and others who have jobs related to forecasting that it's difficult to convince someone of something when their job relies on them not believing it. I think you should assume that you will think forecasting is more useful than it is.

As for your points, I'll respond to some of them.

1. If you want to DM me, I can sign an NDA, and I may update my opinion depending on what these non-public uses of forecasting are.
2. I don't think this is all that relevant. I'm not sure what forecasting research has really elicited on AI timelines. I agree that talk about timelines creates a lot of "buzz" around AI but depending on your views, this is good or bad.
3. I agree that the impact of measurement-oriented research is difficult to measure, but importantly, not impossible. OWID for example should count how much their work is being cited and looked up. Conversely, I think it would be good to estimate, for FRI, how much $$ the change of the decision was worth and by what amount/percentage did FRI make that change more likely. I don't think you really gave a good reason that FRI should be funded over anything else that simply has very diffuse benefits.
4. When do you think it's reasonable, if ever, for the EA community to "give up" on funding more forecasting work?
5. If I'm being cynical, almost every field can say "AI will transform the field" though I'm not sure how much this is worth debating.

Yes, my fault. I should have published sooner.