Aiming to finalise plans for Giving Season this week. I'm not running a donation election (for reasons I'll explain in the announcement), which gives us some more space to focus directly on having important conversations to do with funding and resource allocation in the movement.Â
I'll run a marginal funding week (week starting 16 Nov - message me if you'd like to take part), and a debate week (planning it for week starting Nov 30). I reckon I have capacity for two more themed weeks, as well as AMAs etc...Â
What would you like to see us run over Giving Season?Â
Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
PauseAI UK is holding a March Against AI Extinction on 21st November.
It will be the largest demonstration for AI safety in the world, by a substantial margin. I really hope that almost everyone in the London AI safety community will be there. If you know people in the UK, please ask them to go.
Anthropic just released its S-1 form, which contains the official financial information it must report to the SEC in order to go public. From the S-1, we now know that Anthropic's revenue grew more than 100x (specifically, 112x) from 2023 to 2025. Revenue was:
A few people in the EA community apparently believe this revenue growth rate will continue for the next several years at least. However, I am confident we won't see Anthropic bring in $514 billion in revenue in 2027, which is what extrapolation would imply.
Indeed, I'm confident that Anthropic won't hit half of what extrapolation would imply, which is $257 billion in 2027.
I'm willing to bet as much and would be delighted to wager $20 on it, to be donated to the charity of the winner's choice.
I also have an open offer for a more general AI bubble bet.
Update (Friday, October 30th at 5:45pm Eastern):
Here's a good article on this topic. Two economists look at Anthropic's S-1 and ask whether a $2 trillion valuation is reasonable: https://econjared.substack.com/p/the-fantastical-promises-of-a-2-trillion
If you have engaged with Anthony's sequence, and you hadn't thought seriously about cluelessness before, I'd be interested in knowing if it changed your mind on your cause prioritisation. You can answer as a reply, or just vote:
Note: I couldn't post this at the time because there was a bug, but I'd be keen to hear responses now as well.Â
(We've talked about most of this outside of the public view, but I can't restrain myself from commenting none-the-less)
I agree with you strategically that people who care about mental health need to make their case in a way that's legible to more traditional metrics like income, DALYs, etc. But the general thing this points to is greater legibility --> greater uptake --> impact blah blah.Â
On this general point, one thing we've been thinking about and piloting is just trying to make WELLBYs more legible, especially for MH interventions by converting them to years of depression averted (YODAs).Â
So I could say that a fictional charity Gaya Kuides generates 1,000 WELLBYs per $1k donated. Or given that the typical case of depression (moderate depression) is associated with a -1.3. lower wellbeing compared to low depression symptoms -- this implies that actually GK averts the equivalent of 1000 / 1.3 =769 years of depression per $1k donated. FWIW we explain the YODAs more here and we're looking for feedback on whether this seems like a useful communication device.Â
It just seems way easier to imagine the value of a year of depression averted compared to "WELLBY".Â
2. Long-run effects of psychotherapyÂ
This is very important and understudied. The effects of psychotherapy could last much longer, and we could maybe make these interventions benefit people for much longer than we. But we don't care much about LR effects in general in the research community and we do not have much understanding of the mechanisms that support long-term benefits. I obviously wish that we'd change on both accounts.Â
I've been chatting to Vasco via email about this. I guess I'll import some of my critiques here.
I feel like Brian's right that consciousness has some fuzzy edges (maybe it's a combination of many hard-to-recognise-in-isolation subphenomena), but his actual takes on this and most other philosophical subjects seem to me to reduce to an equivocation between nihilism and extreme egoism - they always seem to boil down to 'what "I" (the author) "care" about' as some kind of acceptance test. In the case of consciousness, this seems like you could understand it in a couple of ways, neither of which seem very informative to the reader:
Â
Brian subsequently wrote the essay My Confusions about the Hard Problem of Consciousness, which I think (though Vasco disagrees) basically contradicts this one.Â
Though to do that I need to digress into an extended caveat that any purported contradiction comes with the caveat that on interpretation two, Brian's words and all words are referentless. At the very least, if we abandon the notion of consciousness, I think we most likely lose our entire vocabulary of non-physically-descriptive nouns, including but not limited to
(not to say I would miss all of these) (also heads up so it doesn't catch Vasco off guard, I added a couple of key ones since our email discussion)
And I think Vasco is arguing for interpretation two. So if so I think there's no discussion to be had - we are both just flapping our meat. It might turn out somehow that one of us counterfactually changes their behaviour if in range of the others' meat-flapping, but we can't ascribe that to causality - since that's a non-physically-descriptive noun - and we certainly can't ascribe it to anything like semantic properties of the words.
So I have to proceed by assuming that the first interpretation was correct, or that I somehow misunderstand the nature of the absence of consciousness, which is a strange position to be in. Perhaps Vasco can help me understand a different way of 'thinking about' it, but I am sceptical.
On this first-and-a-halfth interpretation Brian has gone from defining consciousness as 'thing Brian cares about' to 'we need to know what other things have raw feelings'. If the first essay were correct, 'need' wouldn't mean much, and if it did, the answer would be 'whatever things Brian cares about'. So perhaps Brian is saying there he anticipates caring about future things and cares about that anticipation. But learning information about the world couldn't give him information about other consciousnesses, because that information would be entirely contained within the stochastic process of him caring about things or not.
In the latter essay, Brian writes:
Reductionism: The world ultimately reduces to physicsâquarks, leptons, etc., or probably something else more fundamental. "Consciousness" is not one of those physical primitives. Thus, consciousness must be an emergent phenomenonâa process that involves many parts working together in various ways. But if consciousness is emergent, then it must have fuzzy boundaries just like any concept does. There's no "true essence" underlying an emergent phenomenon. It just is what its components are, and we decide the boundaries of labeling which emergent phenomena belong to which categories that we make up. Consciousness must be like this.
Which Vasco approves of. I think these claims are far too confident:
Ultimately I think that consciousness, despite being a philosophical black box and a neuroscientifically grey one, is too powerful a high level theory to abandon without a much more solid replacement. If you were to put me on a rack for long enough, I would recant the positions I'm arguing for here. But my prior explanation of why I would recant refers to a relatively simple, localisable statement that some -currently but not inevitably hard to specify - measurable quantity had gone negative, and that 'caused' my action.Â
I won't put words into Vasco's mouth, but I wonder whether he thinks there's a way of describing that process without using 'reference classes' that's simpler than '{full mathematical derivation that} that's the all the interacting fields and forces in that local area'.
I also want to ask Vasco how he can be confident that, even if it's true that he has no consciousness (and it does seem to be a contingent and disablable property in humans), that lack of a light would generalise to those of us who profess to have one.
In the late 2000s and early 2010s, GiveWell would look at meta-analyses of randomized controlled trials and get lots of expert input on whether they were interpreting them correctly. This was the sort of approach that was supposed to typify effective altruism.
Fast forward to the mid-2020s and now subjective intuition plays an unreasonably large role in how effective altruism attempts to answer empirical, quantitative questions of a scientific or technical nature. What happened to effective altruism?
Worse, it's become a recurring problem that subjective intuitions are deceptively dressed up as objective empirical evidence. It's bad enough to leave behind more rigorous, scientific approaches in favour of subjective intuition. It's even worse to mislead people â either willingly or through carelessness â into thinking you're reporting objective, measured data when you're really just reporting your subjective guesses.
This shift away from science toward intuition feels even more troubling when you connect it to the hostility expressed by some members of the EA community toward academia, journalism, and liberal democracy. There is a sort of dark, conspiratorial, fringe worldview you get when you combine these troubling tendencies. When considering this, it should be remembered that there have been violent killings and threats of killings at the peripheries of the EA community. This does not normally happen with mainstream social movements of this size. What is fuelling these extremist acts?
More and more, I appreciate just how important are the intellectual cornerstones of modern, Enlightenment society like science and liberalism, and the institutions that support them, like academia and journalism. Extremism and darkness are only a few short steps away from the light. The barrier between them is as thin as air.
May those who carry the light win, and may those lost in the darkness find their way back.