I'm a computational physicist, I generally donate to global health. I am skeptical of AI x-risk and of big R Rationalism, and I intend on explaining why in great detail.
I definitely agree that Bayesianism is overrated and overused. I do wonder if some of these pitfalls could be avoided if people started using actual Bayesian statistics, rather than this fake pop-bayes thing where people make single numerical estimates (like saying "10% chance of apocalypse") and nudge the number up and down a bit in response to news.
Maybe I'm being a grumbly math nerd here, but outside of toy examples, actual bayesianism involves updating distributions, not single numbers. And you need to actually run calculations using bayes formula, not guesstimate it.
Another point in favour of your position is that there are a lot of people like me, who think that extinction risk is massively overblown, but are seriously concerned about smaller-scale harms from AI. I would say this is a majority view among the people I know in real life, and is the dominant view on left-wing social media networks such as bluesky.
I will not join a movement that is solely focussed on human extinction risk, because I think the arguments for it are wrong, and I believe such a movement would backfire anyway by helping AI companies with their marketing hype. I would join a movement that cares about the negative effects of AI that are plainly visible right now.
I can add four more to the list of possible dangers:
Effective altruists tend to be more utilitarian, more "maximizing" with regards to ethics, and more willing to bite philosophical bullets, compared to the general population. EA's involvement in AI safety could end up spreading these beliefs to future AI. It's well established how this sort of utilitarian reasoning can go wrong (utility monsters, etc), so this could increase the chance of disaster.
Effective altruists also have an implicit epistemology that differs from typical subject matter experts. If there are flaws in parts of the epistemology of EA or other groups involved in AI safety (which I believe to be true), then these flaws could be absorbed by future AI systems, and give rise to negative outcomes due to poor decisonmaking, etc.
People that are informed by AI safety arguments could decide that extreme actions are necessary to avoid AI doom, resulting in violence or rash actions that cause extreme harm.
The focus on AI safety in particular could result in less attention being paid to other threats to humanity that turn out to be more pressing, but are neglected due to the overrepresentation of AI safety people in the existential threat field.
Point 2 is something that confused me at first as well. The reason they are different is that we are looking at the performance of the top apparent threat. If we were perfectly good estimators, this would be the same as the top actual threat, but we aren't: the threat of the top pick is generally going to be lower than the top actual threat due to uncertainty and the curse.
For the grounded estimator, the process of ranking threats gives useful information, and it means that the top threat picked is much higher than you would get from picking at random. Whereas in the speculative case, we are much closer to just picking at random, and thats reflected in the yellow curve which looks a lot like the power law sampling we are drawing from.
Here is an example run I did where I tuned down the alpha to 1.5 and tuned the lognormal standard deviation down to 1.5:
Here, the top actual threat really is 7 orders of magnitude more dangerous than the bottom evaluated threat. However, the top apparent threat is overestimated by a factor of 10,000 or so.
If I do a bunch of runs with these settings, the median overestimate is over 100x:
So even if I trust your vibes here (which do not seem to be based on anything), the curse can still hit quite badly. I personally believe that the spread of numbers that people make up is going to be higher than the spread of actual threats: when we look at actual surveys you get a highest-lowest estimate spread of 11 orders of magnitude for some questions.
One thing that might be confusing you is that the power law model assumes that only threats above a certain threshold of actual danger are considered (this is the xmin factor). Obviously nuclear risk is a much greater risk than stubbing your toe, but it's not going to show up the model.
This has not been my experience from 9 years of academia in physics and material science. Opinions published in scientific papers must be backed up with reference to actual evidence, not merely opinion. When deferral happens behind the scenes, it's usually justified by the person in question being an actual expert that knows their shit.
EA is far worse: I sometimes see people defer to random blog posters who have zero expertise in the subject they are talking about.
Interesting analysis! I'm somewhat skeptical that space based cells will perform as well as you find in this model. This seems like a classic case where something that looks good on paper will run into problems in real world applications for something as complicated as putting massive datacenters in space. I don't think your model is accounting for murphy's law here.
Also, on a technical note, do you have the source for the starlink solar cell specifications used for the phrase "Back-calculated from Starlink V3 specifications (50,400 W from 554 kg array, 257 m² area)"? This seems like one of the critical figures of your model but I couldn't find where you linked the source for it.
It is extremely difficult to determine base rates for something like sexual harassment, because it's an offence that allows for ambiguity and plausible deniability, because there's room for retaliation, etc, and it will strongly depend on how much people trust the bodies they are reporting to.
What we can do is look at the responses to the incidents that do get raised, and the experiences of victims, and judge whether or not they live up to the standards we want to see in a group that takes sexual harrasment seriously. I do not think the grades are very good on this front.
Only 3 months ago we had a writeup detailing a shockingly terrible response to sexual harrassment by one of the most prominent EA orgs out there. The response is far worse than anything I've ever seen at any organisation I've ever been in. This indicates to me that the environment is nowhere the high standards that should be aimed for.
Regardless of the actual base rates, the question that matters the most is whether there is room for improvement, and I think it's blindingly obvious that the answer is yes.
LLM disclosure in general is just a good idea to do. The internet is absolutely flooded with LLM-written spam at the moment, so if people detect LLM writing with no context it's natural to assume your post is spam as well. This is a shame when someone who is a non-native speaker has just used it for translation or whatnot.
Personally I'd recommend against using LLM-written text if you can help it, as in the age of spam the value of cultivating your own stylistic voice is increasing.
I think you would benefit from re-reading the article in question. For example, they directly adress your point 1 by pointing out that consumer diffusion figures are often misleading by expressing figures in terms of "percentage of people that use chatbots on occasion", rather than on frequency of use.
Point 3 is not even an argument, just a restatement of what they believe: yes, they think AI domination will take decades. They state the reasons they believe this very clearly in the section "Diffusion is limited by the speed of human, organizational, and institutional change": if you disagree with this, you have to present actual arguments. From what I know, most economists would agree with them.
Point 5 is not an argument either: they are not to blame for how you interpret their "vibes". If people interpret "AI will be akin to the internet" as anything other than "AI will be akin to the internet" that's their fault, not the authors.
As for point 6, I'm confused as to what your position is here. Do you think that AI systems are merely cheating on every single benchmark? In the section "benchmarks do not mention real-world utility", I took them as referring to benchmarks that are actually meaningful: saying that while they genuinely are good at taking law tests, even non-contaminated ones, that this doesn't translate into being a good lawyer because of the aspects that are not easily measurable. I don't see how this is a contradiction to any of their previous work?
I definitely agree that Bayesianism is overrated and overused. I do wonder if some of these pitfalls could be avoided if people started using actual Bayesian statistics, rather than this fake pop-bayes thing where people make single numerical estimates (like saying "10% chance of apocalypse") and nudge the number up and down a bit in response to news.
Maybe I'm being a grumbly math nerd here, but outside of toy examples, actual bayesianism involves updating distributions, not single numbers. And you need to actually run calculations using bayes formula, not guesstimate it.
Another point in favour of your position is that there are a lot of people like me, who think that extinction risk is massively overblown, but are seriously concerned about smaller-scale harms from AI. I would say this is a majority view among the people I know in real life, and is the dominant view on left-wing social media networks such as bluesky.
I will not join a movement that is solely focussed on human extinction risk, because I think the arguments for it are wrong, and I believe such a movement would backfire anyway by helping AI companies with their marketing hype. I would join a movement that cares about the negative effects of AI that are plainly visible right now.
I can add four more to the list of possible dangers:
Point 1 was a typo, thanks for pointing it out!
Point 2 is something that confused me at first as well. The reason they are different is that we are looking at the performance of the top apparent threat. If we were perfectly good estimators, this would be the same as the top actual threat, but we aren't: the threat of the top pick is generally going to be lower than the top actual threat due to uncertainty and the curse.
For the grounded estimator, the process of ranking threats gives useful information, and it means that the top threat picked is much higher than you would get from picking at random. Whereas in the speculative case, we are much closer to just picking at random, and thats reflected in the yellow curve which looks a lot like the power law sampling we are drawing from.
Here is an example run I did where I tuned down the alpha to 1.5 and tuned the lognormal standard deviation down to 1.5:
Here, the top actual threat really is 7 orders of magnitude more dangerous than the bottom evaluated threat. However, the top apparent threat is overestimated by a factor of 10,000 or so.
If I do a bunch of runs with these settings, the median overestimate is over 100x:
So even if I trust your vibes here (which do not seem to be based on anything), the curse can still hit quite badly. I personally believe that the spread of numbers that people make up is going to be higher than the spread of actual threats: when we look at actual surveys you get a highest-lowest estimate spread of 11 orders of magnitude for some questions.
One thing that might be confusing you is that the power law model assumes that only threats above a certain threshold of actual danger are considered (this is the xmin factor). Obviously nuclear risk is a much greater risk than stubbing your toe, but it's not going to show up the model.
This has not been my experience from 9 years of academia in physics and material science. Opinions published in scientific papers must be backed up with reference to actual evidence, not merely opinion. When deferral happens behind the scenes, it's usually justified by the person in question being an actual expert that knows their shit.
EA is far worse: I sometimes see people defer to random blog posters who have zero expertise in the subject they are talking about.
Interesting analysis! I'm somewhat skeptical that space based cells will perform as well as you find in this model. This seems like a classic case where something that looks good on paper will run into problems in real world applications for something as complicated as putting massive datacenters in space. I don't think your model is accounting for murphy's law here.
Also, on a technical note, do you have the source for the starlink solar cell specifications used for the phrase "Back-calculated from Starlink V3 specifications (50,400 W from 554 kg array, 257 m² area)"? This seems like one of the critical figures of your model but I couldn't find where you linked the source for it.
It is extremely difficult to determine base rates for something like sexual harassment, because it's an offence that allows for ambiguity and plausible deniability, because there's room for retaliation, etc, and it will strongly depend on how much people trust the bodies they are reporting to.
What we can do is look at the responses to the incidents that do get raised, and the experiences of victims, and judge whether or not they live up to the standards we want to see in a group that takes sexual harrasment seriously. I do not think the grades are very good on this front.
Only 3 months ago we had a writeup detailing a shockingly terrible response to sexual harrassment by one of the most prominent EA orgs out there. The response is far worse than anything I've ever seen at any organisation I've ever been in. This indicates to me that the environment is nowhere the high standards that should be aimed for.
Regardless of the actual base rates, the question that matters the most is whether there is room for improvement, and I think it's blindingly obvious that the answer is yes.
LLM disclosure in general is just a good idea to do. The internet is absolutely flooded with LLM-written spam at the moment, so if people detect LLM writing with no context it's natural to assume your post is spam as well. This is a shame when someone who is a non-native speaker has just used it for translation or whatnot.
Personally I'd recommend against using LLM-written text if you can help it, as in the age of spam the value of cultivating your own stylistic voice is increasing.
I think you would benefit from re-reading the article in question. For example, they directly adress your point 1 by pointing out that consumer diffusion figures are often misleading by expressing figures in terms of "percentage of people that use chatbots on occasion", rather than on frequency of use.
Point 3 is not even an argument, just a restatement of what they believe: yes, they think AI domination will take decades. They state the reasons they believe this very clearly in the section "Diffusion is limited by the speed of human, organizational, and institutional change": if you disagree with this, you have to present actual arguments. From what I know, most economists would agree with them.
Point 5 is not an argument either: they are not to blame for how you interpret their "vibes". If people interpret "AI will be akin to the internet" as anything other than "AI will be akin to the internet" that's their fault, not the authors.
As for point 6, I'm confused as to what your position is here. Do you think that AI systems are merely cheating on every single benchmark? In the section "benchmarks do not mention real-world utility", I took them as referring to benchmarks that are actually meaningful: saying that while they genuinely are good at taking law tests, even non-contaminated ones, that this doesn't translate into being a good lawyer because of the aspects that are not easily measurable. I don't see how this is a contradiction to any of their previous work?