Also what are the main or best open source projects in the space? Or if someone wanted to actually use LMs for forecasting, what is better than just asking o3 to produce a forecast?

I haven’t been following this area closely, but why aren’t they making a lot of money on polymarket?

I was going for the second, adding some quotes to make it clearer.

“Plausibly best people who have ever lived” is a much lower bar than “best people who have ever lived”.

I’m a huge fan of having high standards. Posts that are like “we reproduced this published output and think they made these concrete errors” are often great. But I notice much more “these people did a bad job or spent too much money” takes often from people who afaict haven’t done a bunch of stuff themselves so aren’t very calibrated, and don’t seem very scope sensitive. If people saw their projects being critiqued and were then motivated to go and do more things more quickly I’d think that was great (or were encouraged to do more things more quickly from “fear” of critiques) I think we’d be in a better equilibrium.

For example people often point out that LW and the forum are somewhat expensive per user as evidence they are being mismanaged and imo this is a bad take which is rarely made by people who have built or maintained popular software projects/forums or used the internet enough to know that discussion of the kind in these venues is really quite rare and special.

To be clear, I think the “but have they actually done stuff” critique should also be levelled at grantmakers. I’m sympathetic to grantmakers who are like “the world is burning and I just need to do a leveraged thing right now” but my guess is that if more grantmakers had run projects in the reference class of things they want to fund (or founded any complicated or unusual and ambitious projects) we’d be in a better position. I think this general take is very common in YC/VC spaces, which perform a similar function to grantmaking for their ecosystem.

Many examples of criticism in replies, are high quality posts that I think improve standards. I may spend an hour going through the criticism tag and sorting them into posts I think are useful/anti-useful to check.

Disproportionately deterring bad projects is a crux. I think if people are running a “minimise criticism policy” they aren’t going to end up doing very useful things (or they’ll do everything in secret or far from EA). I currently don’t think nearly enough people are trying to start projects and many project explorations seem good to me on net, so the discrimination power needs to be pretty strong for the benefits to pencil.

I think there are positives about criticism which I didn’t focus on, but yeah if I were to write a more comprehensive post I think the points you raised are good to include.

I’m not convinced that criticism is very counterfactually educational for 3rd parties. Particularly when imo lots of criticism is bad. Feels like it could go either way. If more criticism came from people who had run substantial projects or operated in the same field or whatever I think I’d trust their takes more. Many of the examples raised in this thread are good imo and have this property.

I agree that this is one of the upsides of criticism on the forum. I don’t think it outweighs the costs in many cases.