yeah, as an academic, I find writing blog posts to be much more pleasant and enjoyable than writing scientific papers. Part of the reason for that is that scientific papers are much more technical and rigorous, and you are expected to back up all your claims in order to pass peer review. 

The flipside, of course, is that I trust blog posts a lot less than scientific papers, my own posts included. I still think more academics should have a go at blogging though, it can help bridge the gap between the technical work and the public. 

I'm sure @David Thorstad can defend himself here, but to quote his most recent post:

The claim, then, cannot be that the scope of longtermism is empty. The claim is rather that cases such as the Space Guard Survey are more special than they may appear. When we turn our eyes to many other proposed existential risk mitigation efforts, the scope-limiting factors begin to get a more solid take.

Ie: he has acknowledged already that his objections don't really apply to asteroid monitoring, a field where predictions are backed to scientific precision. But he says this is a special case, and that his objections apply much more strongly to other longtermist causes like AI risk. 

My understanding is that peer review is somewhat less common in computer science fields because research is often published in conference proceedings without extensive peer review. Of course, you could say that the conference itself is doing the vetting here, and computer science often has the advantage of easy replication by running the supplied code. This applies to some of the papers people are providing... but certainly not all of them. 

Peer review is far from perfect, but if something isn't peer reviewed I won't fully trust it unless it's gone through an equivalent amount of vetting by other means. I mean, I won't fully trust a paper that has gone through external peer review, so I certainly won't immediately trust something that has gone through nothing. 

I'm working on an article about this, but I consider the lack of sufficient vetting to be one of the biggest epistemological problems in EA. 

Why? What advantage does this have over a lower stakes bet of $50:$250? 

Do you have an alternate suggestion for how flaws and mistakes made by projects in the EA sphere can be discovered?

As a scientist, one of the reasons people trust our work is the expectation that the work we publish has been vetted and checked by other experts in the field (and even with peer review, sloppy work gets published all the time). Isn't one of the goals of the EA forum to crowdsource at least some of this valuable scrutiny? 

Scaled voting power is part of why moderation on the Forum is sustainable

On a typical day the forum has like 10 posts and like 30 comments. A subreddit like r/excel is roughly 3 times as active as this forum, and is moderated entirely by volunteers. I do not think it would be very difficult to moderate the forum without a karma system if people chose to do so. 

Therefore, high karma correlates pretty well with people having good Forum content taste. 

I would say the people with the most karma are the people who comment a lot with a content that is in line with the tone of the forum. This seems like it serves to reinforce the status quo of forum norms. Whether this is a good or bad thing will depend on your opinion of said norms: for example, I would prefer this place be a bit  more tolerant of humour and levity.  

I am having trouble understanding why AI safety people are even trying to convince the general public that timelines are short. 

If you manage to convince an investor that timelines are very short without simultaneously convincing them to care a lot about x-risk, I feel like their immediate response will be to rush to invest briefcases full of cash into the AI race, thus helping make timelines shorter and more dangerous. 

Also, if you make a bold prediction about short timelines and turn out to be wrong, won't people stop taking you seriously the next time around? 

As I said, I don't think your statement was wrong, but I want to give people a more accurate perception as to how AI is currently affecting scientific progress: it's very useful, but only in niches which align nicely with the strengths of neural networks. I do not think similar AI would produce similarly impressive results in what my team is doing, because we already have more ideas than we have the time and resources to execute on. 

I can't really assess how much speedup we could get from a superintelligence, because superintelligences don't exist yet and may never exist. I do think that 3xing research output with AI in science is an easier proposition than building digital super-einstein, so I expect to see the former before the latter. 

I found this article well written, although of course I don't agree that AGI by 2030 is likely. I am roughly in agreement with this post by an AI expert responding to the other (less good) short- timeline article going around. 

I thought instead of critiquing the parts that I'm not an expert in, I might take a look at the part of this post that intersects with my field, when you mention material science discovery, and pour just a little bit of cold water on it. 

A recent study found that an AI tool made top materials science researchers 80% faster at finding novel materials, and I expect many more results like this once scientists have adapted AI to solve specific problems, for instance by training on genetic or cosmological data.

So, an important thing to note is that this was not an LLM (neither was alphafold), but a specially designed deep learning model for generating candidate material structures. I covered a bit about them in my last article, this is a nice bit of evidence for their usefulness. The possibility space for new materials is ginormous and humans are not that good at generating new ones: the paper showed that this tool boosted productivity by making that process significantly easier. I don't like how the paper described this as "idea generation": it evokes the idea that the AI is making it's own newtonian flashes of scientific insight, but actually it's just mass generating candidate materials that an experienced professional can sift through. 

I think your quoted statement is technically true, but it's worth mentioning that the 80% faster figure was just for the people previously in the top decile of performance (ie the best researchers), for people who were not performing well there was not evidence of a real difference. In practice the effect of the tool on progress was less than this: it was plausibly attributed to increasing the number of new patents at a firm by roughly 40%, and increasing the number of actual prototypes by 20%. You can also see that the productivity is not continuing to increase: they got their boost from the improved generation pipeline, and now the bottleneck is somewhere else. 

To be clear, this is still great, and a clear deep learning success story, but it's not really in line with colonizing the mars in 2035 or whatever the ASI people are saying now. 

In general, I'm not a fan of the paper, and it really could have benefited from some input from an actual material scientist. 

I think if you surveyed any expert on LLMs and asked them "which was a greater jump in capabilities, Gpt2 to GPT3 or GPT3 to GPT4?" the vast majority would say the former, and I would agree with them. This graph doesn't capture that, which makes me cautious about overelying on it.