Jack_S🔸
Programme Director at ML4Good.
I also have done work on China policy, AI governance, and animal advocacy in Asia.
Also interested in effective giving (mainly animal charities), economic development (and how AI will affect it), AI x Animals, wild animal welfare, cause prioritisation, and various meta-EA topics.
Good question, but I implore you not to take this post too seriously. It's a real phenomenon, but it's a real stretch to claim that this applies in the implied way to cause areas like AI safety.
The model in the article is just a toy model of a world where existential threats are randomly distributed according to a power law, where genuinely high-probability threats are, by assumption, basically absent from the space of possible threats, and where there's no process of updating based on evidence.
A more narrow claim like: "single narrowly defined x-risk estimates of genuinely speculative, unresearched causes are likely to be inflated" might be valid, but that doesn't seem to be what titotal is implying. He's making a claim way beyond anything implied by the model, even if the model were a valid representation of the phenomenon. He seems to believe that almost all of today's concern for AI risk is all downstream of a belief cultivated within a narrow subcommunity subject to the optimiser's curse - an extraordinary claim that requires a lot more evidence than that supplied in the article.
The claim being made is something like:
Note that this requires some pretty wild, difficult-to-justify assumptions on how this belief has spread.
The opposing narrative (which I would advocate for) is that:
And on your particular question of how to act, even given the optimiser's curse as stated in the toy model, working on the speculative thing could still be optimal. If a highly uncertain intervention seems exceptionally promising or x-risky, the value of information becomes incredibly high, because accurate or well-reasoned research will lead to this intervention being prioritised or not by far more people. If you have a research focus, it's therefore probably more recommended to focus on a more uncertain, "high-risk-high-reward" area. You could also draw up a toy model for explore/exploit based on the optimiser's curse.
Finally, you don't necessarily have to "pick" from a narrow set of pre-defined cause areas. You can also divide or merge risks, cause areas, skill-sets etc. to be more robust, precise, or coherent with your own world model (e.g. focusing on engineered pandemics because you realise this could interact with AI-related x-risk, GCBRs, and global health).
Your piece went kinda viral on Substack, and most of the comments on the comments thread, and elsewhere on Substack, were constructive and positive. This seems a moderate win for alternative/social media.
I would barely update at all on this paper. Effect sizes are very small and don't show a logical causal mechanism.
Looking at the full paper, in the fully adjusted matched model, smaller tattoos (under one hand palm) give you a higher risk (1.27) than larger ones (1.14). In the unmatched model, large tattoos had no effect. This seems contradictory with their suggested mechanism (ink affecting lymph nodes).
And their results imply that it's most dangerous getting tattoos before there's a plausible chemical pathway to harm (risk is highest for those who got tattoos under 2 years ago). They say this could be a result of antagonising existing tumours, but seems likely to be a statistical artifact or a non causal correlation.
If you have other evidence, though, I'd be happy to hear it.
I wrote a (draft amnesty) post on a very small subset of this - what global health charities you should donate to given different worldviews.
I agree with the "you don't have to debate on their terms" point here, but I think for 99% of your readers/listeners, it cuts far more strongly in a different way than that you're implying.
The debate has generally been set in terms of "Anthropic vs. DoW", and, while I know zero people in our community who have taken the government's side on this, I've seen many EAs and adjacent people become increasingly uncritical supporters of Anthropic, just because they're standing against the obviously bad actor in this situation.
I think it's important to remember:
Excellent post, very clever humour!
I'm relieved at the results.
Being forced by the data and our epistemic norms to say "well, actually..." would not have been a good look for public-facing EAs.
Thanks for this post, it's super valuable to get a better sense of this ecosystem.
On the apparent lack of Chinese companies, I think this is a methodological thing; a few possible blind spots:
I'd lean towards the World Happiness Report results here. IPSOS uses a fully online sample, which means you end up losing the "bottom half" of the population. World Happiness Report is phone and in-person.
Hi Klara, thanks for the response.
I don't think I am entering the abortion debate by assigning moral value to unborn lives any more than I'm entering any other debate that considers unborn or potential lives (e.g. the ethics of moderate drinking while pregnant, the ethics of having children in space, or the repugnant conclusion).
I think I'm comfortable with having mostly sidestepped the maternal health issues, given that I was focusing on interventions that are robustly good for the mother. If I were to do a stronger and more robust cost-effectiveness analysis, or tackle more controversial interventions where the interests of the mother and child clearly diverged, I would consider maternal health outcomes separately. I hope my piece makes it clear that we should prioritise uncontroversial and neglected interventions that treat or prevent painful conditions that women suffer from.
Although I do recognise that the ethics of pregnancy, lived experience of the mother, and autonomy trade-offs are important considerations, I'm afraid that attempting to tackle these here would have made this an impossibly long post!
When I say “the economics are looking good,” I mean that the conditions for capital allocation towards AGI-relevant work are strong. Enormous investment inflows, a bunch of well-capitalised competitors, and mass adoption of AI products means that, if someone has a good idea to build AGI within or around these labs, the money is there. It seems this is a trivial point - if there were significantly less capital, then labs couldn’t afford extensive R&D, hardware or large-scale training runs.
WRT Scaling vs. fundamental research, obviously "fundamental research" is a bit fuzzy, but it's pretty clear that labs are doing a bit of everything. DeepMind is the most transparent about this, they're doing Gemini-related model research, Fundamental science, AI theory and safety etc. and have published thousands of papers. But I'm sure a significant proportion of OpenAI & Anthropic's work can also be classed as fundamental research.