Aimed at an audience newer to EA/AI safety
It seems like many people have been confused lately about why 1) AI safety orgs are in desperate need of good hires and 2) org hiring rates are very low. My understanding is that there are many reasons for this, but a 2021 post states it pretty well:
"A lot of EA content is subtle and detailed, and easy to accidentally compress into something misleading. For example, 80k might write a nuanced article saying 'You should focus on talent gaps, not funding gaps', but this gets translated into 'EA is talent constrained', and then people repeat that phrase without linking to the article, and then many people form an inaccurate belief that EA needs 'pretty talented people', rather than 'EA needs very specific talents that are missing.'"
The best career advice seems to be to focus on developing a rare and valuable skill that orgs currently need. In AI safety, this skill is a combination of specific context, agency, mission-alignment, and strategic vision. You can develop it by reading a lot, writing out your thoughts, talking to people at events, and completing real projects.
It's hard to do really impactful work. You need a plausible explanation for why your efforts are the best way your salary can be spent. This takes time, effort, and careful strategizing. Most of the world has high schools, colleges, and internships to train people for jobs, but a similar system is only recently becoming legible for AI safety and EA. The rare skills are rare for a reason. Still, I think the broader EA community could do a better job of communicating what it means for these "talent-gaps" to exist.
PauseAI UK is holding a March Against AI Extinction on 21st November.
It will be the largest demonstration for AI safety in the world, by a substantial margin. I really hope that almost everyone in the London AI safety community will be there. If you know people in the UK, please ask them to go.
I put a little poll here to follow up on whether minds were changed by the cluelessness critiques competition. If you read the posts during that week, check it out.
Alignment premium and misalignment tax
It’s interesting how it feels like in some ways there is or could be both of these;
Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.
Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope with, fix, or re-train misaligned models.
There's a quote in the post:
Some effective altruists expect artificial general intelligence (AGI) or transformative AI (TAI) to solve the taste and price challenges for alternative meats, and so they wonder: What's the point of work on alternative meats now?
I'd be interested to learn more about the reasoning behind this view. So far I'm only familiar with [1](https://www.lesswrong.com/posts/DuCWLYx4jAFPMXKjZ/paper-artificial-intelligence-in-life-extension-from-deep) and [2](https://forum.effectivealtruism.org/posts/TYhs8zehyybvMt5E4/).
And I have an intuition (possibly because I'm unintentionally making a strawman out of the actual position simply because I don't understand it) that the "we need AGI to solve this problem" relies on the assumption that "AGI is a thing that will solve basically all existing problems simply because it's generally much better at solving problems than we are". And the implicit conclusion is "so we don't need to think about how exactly AGI will solve this particular problem".
It seemed to me that the post was arguing against this approach.
And I actually want to argue even further:
I think we somewhat understand the problem at the gears-level where we can build models and test them against reality, given that we have narrow AI with today's SOTA capabilities at our disposal. Current models have already shown themselves to be effective in some kinds of scientific research (mostly when the task is formulated in a specific, AI-tractable way).
And overall I assign a higher probability to A than B.
And I assign a substantially higher probability to A than B. (And I'm not even talking about existential risks here, but never mind.)
Is it really true that my view is far from consensus? I thought pretty much everyone thought roughly along these lines, but after reading this post I'm somehow not so sure anymore.
If you have engaged with Anthony's sequence, and you hadn't thought seriously about cluelessness before, I'd be interested in knowing if it changed your mind on your cause prioritisation. You can answer as a reply, or just vote:
Note: I couldn't post this at the time because there was a bug, but I'd be keen to hear responses now as well.
(We've talked about most of this outside of the public view, but I can't restrain myself from commenting none-the-less)
I agree with you strategically that people who care about mental health need to make their case in a way that's legible to more traditional metrics like income, DALYs, etc. But the general thing this points to is greater legibility --> greater uptake --> impact blah blah.
On this general point, one thing we've been thinking about and piloting is just trying to make WELLBYs more legible, especially for MH interventions by converting them to years of depression averted (YODAs).
So I could say that a fictional charity Gaya Kuides generates 1,000 WELLBYs per $1k donated. Or given that the typical case of depression (moderate depression) is associated with a -1.3. lower wellbeing compared to low depression symptoms -- this implies that actually GK averts the equivalent of 1000 / 1.3 =769 years of depression per $1k donated. FWIW we explain the YODAs more here and we're looking for feedback on whether this seems like a useful communication device.
It just seems way easier to imagine the value of a year of depression averted compared to "WELLBY".
2. Long-run effects of psychotherapy
This is very important and understudied. The effects of psychotherapy could last much longer, and we could maybe make these interventions benefit people for much longer than we. But we don't care much about LR effects in general in the research community and we do not have much understanding of the mechanisms that support long-term benefits. I obviously wish that we'd change on both accounts.
It looks to me like OpenAI's latest model spec is the first version which explicitly asks its models to weigh animal welfare in their responses.
It’s encouraging to see OpenAI’s model spec joining Claude’s constitution in explicitly weighing animal welfare.