Currently doing local AI safety Movement Building in Australia and NZ.
a) The link to your post on defining alignment research is broken
b) "Governing with AI opens up this whole new expanse of different possibilities" - Agreed. This is part of the reason why my current research focus is on wise AI advisors.
c) Regarding malaria vaccines, I suspect it's because the people who focused on high-quality evidence preferred bet nets and folk who were interested in more speculative interventions were drawn more towards long-termism.
In retrospect, it seems that LLM's were initially successful because they allowed engineers to produce certain capabilities in a way that almost maximally leaned on crystallized knowledge and minimally leaned on fluid intelligence.
It appears that LLM's have continued to be successful because we've gradually been able to get them to rely more on fluid intelligence.
The AI Safety Fundamentals course has done a good job of building up the AI safety community and you might want to consider running something similar for moral alignment.
One advantage of developing on a broader moral alignment field is that you might be able to produce a course that would still appeal to folks who are skeptical of either the animal rights or AI sentience strands.
I can share a few comments on my thoughts here if this is something you'd consider pursuing.
(I also see possible intersections with my Wise AI advisor research direction).
Sorry to hear it didn't work out and thank you for your service.
For what it's worth, often it's valuable to take a step back rather than to just keep hitting your head against a wall. This can provide space to develop a better sense of perspective and why things went the way they did, whether you might have had a shot if you approached things in a different way or whether something else might be a better fit for you.
One thing I'd be much more excited about seeing rather than "quantifying post-training variables and their effects" (but which I'm not planning to pursue) would be to take an old model and then to try to map post-training enhancements discovered over time and see how the maximum elicitable capabilities change.
I'm worried that quantifying post-training variables directly has significant capabilities externalities and that there's no obvious limit to how far post-training can be pushed.
Comments:
Your model here seemed to be that you should only run a fellowship program if you already have a strong community, but running a fellowship program is a way to build up a community. It has a strong track record with university groups. Admittedly, it is slightly less suited to the country context, but it can still work either remotely or by running groups in the most major cities.
As an AI safety person who believes short timelines are very possible, I'm extremely glad to see this shift.
For those who are disappointed, I think it's worth mentioning that I just took a look at the Probably Good website and it seems much better than the last time I looked. I had previously been a bit reluctant to recommend it, but it now seems like a pretty good resource and I'm sure they'll be able to make it even better with more support.
Given that The 80,000 Hours Podcast is increasing its focus on AI, it's worth highlighting Asterisk Magazine as a good resource for exploring a broader set of EA-adjacent ideas.
For the record, I see the new field of "economics of transformative AI" as overrated.
Economics has some useful frames, but it also tilts people towards being too "normy" on the impacts of AI and it doesn't have a very good track record on advanced AI so far.
I'd much rather see multidisciplinary programs/conferences/research projects, including economics as just one of the perspectives represented, then economics of transformative AI qua economics of transformative AI.
(I'd be more enthusiastic about building economics of transformative AI as a field if we were starting five years ago, but these things take time and it's pretty late in the game now, so I'm less enthusiastic about investing field-building effort here and more enthusiastic about pragmatic projects combining a variety of frames).