Josh Landes shared that BlueDot Impact's application rate has been increasing by 4.7x/year.

Applications to MATS have been increasing by 70% per year. Is this the growth rate of AI safety?

As part of MATS' compensation reevaluation project, I scraped the publicly declared employee compensations from ProPublica's Nonprofit Explorer for many AI safety and EA organizations (data here) in 2019-2023. US nonprofits are required to disclose compensation information for certain highly paid employees and contractors on their annual Form 990 tax return, which becomes publicly available. This includes compensation for officers, directors, trustees, key employees, and highest compensated employees earning over $100k annually. Therefore, my data does not include many individuals earning under $100k, but this doesn't seem to affect the yearly medians much, as the data seems to follow a lognormal distribution, with mode ~$178k in 2023, for example.

I generally found that AI safety and EA organization employees are highly compensated, albeit inconsistently between similar-sized organizations within equivalent roles (e.g., Redwood and FAR AI). I speculate that this is primarily due to differences in organization funding, but inconsistent compensation policies may also play a role.

I'm sharing this data to promote healthy and fair compensation policies across the ecosystem. I believe that MATS salaries are quite fair and reasonably competitive after our recent salary reevaluation, where we also used Payfactors HR market data for comparison. If anyone wants to do a more detailed study of the data, I highly encourage this!

I decided to exclude OpenAI's nonprofit salaries as I didn't think they counted as an "AI safety nonprofit" and their highest paid current employees are definitely employed by the LLC. I decided to include Open Philanthropy's nonprofit employees, despite the fact that their most highly compensated employees are likely those under the Open Philanthropy LLC.

Hi Chinwe! We prefer applicants with US work authorization, but we can support US visas. Unfortunately, these roles are not remote.

If I were building a grantwriting bootcamp, my primary concerns would be:

• Where will successful grantees work?
  • I've found that independent researchers greatly benefit from a shared office space and community, for social connection, high-quality peer feedback, and centralizing operations costs.
  • Current AI safety offices seem to be overflowing. We likely need further, high-capacity AI safety offices to support the influx of independent researchers from Open Phil's RFPs.
  • I think that, in general, employment in a highly effective organization is more impactful than independent research for the majority of projects and researchers. While I greatly support the new Open Phil RFPs, I hope that more of their grants go towards setting up highly effective organizations, like nonprofit FROs, that can absorb and scale talent.
  • I see the primary benefit of the MATS extension program as a means of providing further research mentorship (albeit with more accountability and autonomy than the main program) with longer time horizons to complete research projects. The infrastructure we provide is quite significant and increasing the number of independent researchers without also scaling long-term support systems will likely not see optimal results.
• How will successful grantees obtain mentorship and high-quality feedback loops?
  • Even with the optimal project proposal, emerging researchers seem to benefit substantially from high quality mentorship, particularly over the course of a research project. I do not believe that all of this support should be front-loaded.
  • I would support an accompanying long-term peer support or mentorship program after the grantwriting bootcamp. I apologize if you were already planning this!
• Who will employ grantees on the conclusion of their research?
  • This is a significant question to MATS as well. I currently believe that high-quality research during the program is a strong enough output alone to justify the cost. However, ideally, most MATS alumni would find employment post-program. The main roadblocks to this employment seem to be software engineering skills (which points to ARENA-like coding bootcamps as a solution) and high-quality peer-reviewed publications (which usually need strong mentorship).
  • At the moment, I think "better grant proposals" is not a significant bottleneck to MATS alumni getting jobs. Rather, I think coding skills and high quality publications are the limiting factors. Also, I think there are far too few jobs to go around compared to the scale of the AI safety problem, so I also support more startup accelerators.

This seems like a great initiative! However, I don't expect this "grantwriting bootcamp" model to benefit the majority of MATS alumni compared to:

• Longer duration, more resourced independent research programs like the MATS extension program, Constellation Visiting Fellows Program, and the CHAI Research Fellowship that provide high-quality mentorship and job security for 6-12 months;
• Dedicated start-up incubator programs like Catalyze Impact, Entrepeneur First def/acc, and Y Combinator that provide seed funding, co-founder matching, and start-up advice;
• Longer duration, more established academic PhD/Postdoc programs that provide high-quality mentorship and legible credentials, e.g. via the Vitalik Buterin PhD Fellowship in AI Existential Safety and labs like UCB CHAI, NYU ARG, MIT AAG, Mila, KASL, MIT Tegmark Group, etc.

I can't speak for the other AI safety research programs on your list, but from my experience at MATS, alumni are principally worried about (in order):

1. Software engineering skills to pass AI lab coding tests;
2. High-quality peer-reviewed publications at top ML conferences to get into AI lab research jobs or top PhD programs;
3. Further high-quality mentorship;
4. Further connections to research collaborators.

In the past, acceptance into the MATS extension program literally required submitting a grant proposal and receiving external funding, generally from the LTFF or Open Phil. On average, 81% of alumni who apply have received LTFF or Open Phil grants (or equivalent AI lab contracting roles) for independent research post-program, even without the current (large) RFPs! Our Research Management staff are trained to help scholars submit AI safety grant proposals and we require all scholars to complete a Research Plan mid-program that doubles as a grant proposal. I think that the optimal program to help most MATS alumni looks less like a "grantwriting bootcamp" (though I'm sure some will benefit) and more like the programs I listed above. That said, I think the grantwriting bootcamp model will certainly benefit some early career researchers. I'm happy to call and chat more about this if you like!

Also, Apollo Research and Leap Labs grew out of the MATS London Office (what later became LISA). I realize this was an AI safety office, not an EA office, but it feels significant.

Yep, seems important. But I don't think this is particularly salient to the topic of the post: changes to AI safety priorities based on the new inference scaling paradigm.