People in AI safety/EA spheres should reorient now towards stopping continued AI capabilities escalation.
Historically, people have been unwilling to straightforwardly say “This AI situation is disgustingly dangerous, and we need to stop” and then actually work towards making this happen. I think a lot of this is downstream of deference to high status people who weren’t willing to take positions that seemed extreme.
The current situation is very, very bad, and there is no plan to make it better if we continue increasing AI capabilities at the current rate.
We need to stop things as soon as possible, so the community and the world can orient and work out what to do. We don't need to have the full plan yet, we just need to get to a state where we aren't in imminent danger.
I predict that this position will become increasingly obvious, and people will wish they reoriented earlier. I think this will be clear ex ante; when people look back they will think that they should have reoriented earlier, given the information they had at the time.
I wrote a quick post about why I think people committed to working on ASI+animals should be making sure we don't spread wild animal suffering throughout the universe.
Someone should write a good, linkable online resource describing the concept of the long reflection. It's very strange that there isn't a simple post/webpage that I can link to that gives a good, medium-depth description.
(Half baked and maybe just straight up incorrect about people's orientations)
I worry a bit about groups thinking about the post-AGI future (e.g., Forethought) will not want to push for something like super-optimized flourishing because this will seem weird and possibly uncooperative with factions that don't like the vibe of super-optimization. This might happen even if these groups thinking about the future do believe in their hearts that super-optimized flourishing is the best outcome.
It is very plausible to me that the situation is "convex" in the sense that it is better for the super-optimizers to optimize fully with their share of the universe, while the other groups do what they want with their share (with rules to prevent extreme suffering, pessimization etc). I think this approach might be better for all groups, rather than aiming for a more universal middle ground that leaves everyone disappointed. This bad middle ground might look like a universe that is both not very optimized for flourishing but is still super weird and unfamiliar.
It would be very sad if we miss out on the optimized flourishing because we were trying to not seem weird or uncooperative.
We may be running multiple smaller cohorts rather than one big one, if that's what maximizes the ability of strong candidates to participate.
The single most important factor in deciding the timing is the window in which strong candidates are available, and the target size for the cohort is small enough (5-20 depending on strength of applicants) that the availability of a single applicant is enough to sway the decision. It's specifically cases like yours that we're intending to accommodate. Please apply!
Announcing: 2026 MIRI Technical Governance Team Research Fellowship.
MIRI’s Technical Governance Team plans to run a small research fellowship program in early 2026. The program will run for 8 weeks, and include a $1200/week stipend. Fellows are expected to work on their projects 40 hours per week. The program is remote-by-default, with an in-person kickoff week in Berkeley, CA (flights and housing provided). Participants who already live in or near Berkeley are free to use our office for the duration of the program.
Fellows will spend the first week picking out scoped projects from a list provided by our team or designing independent research projects (related to our overall agenda), and then spend seven weeks working on that project under the guidance of our Technical Governance Team. One of the main goals of the program is to identify full-time hires for the team.
If you are interested in participating, please fill out this application as soon as possible (should take 45-60 minutes). We plan to setdates for participation based on applicant availability, but we expect the fellowship to begin after February 2, 2026 and end before August 31, 2026 (i.e., some 8 week period in spring/summer, 2026).
Strong applicants care deeply about existential risk, have existing experience in research or policy work, and are able to work autonomously for long stretches on topics that merge considerations from the technical and political worlds.
Unfortunately, we are not able to sponsor visas for this program.
Could/should big EA-ish coworking spaces like Constellation pay to have far-UV installed? (either on their floors specifically or for the whole building)
MATS has a very high bar these days, I'm pretty happy about there being "knock-off MATS" programs that allow people who missed the bar for MATS to demonstrate they can do valuable work.
People in AI safety/EA spheres should reorient now towards stopping continued AI capabilities escalation.
Historically, people have been unwilling to straightforwardly say “This AI situation is disgustingly dangerous, and we need to stop” and then actually work towards making this happen. I think a lot of this is downstream of deference to high status people who weren’t willing to take positions that seemed extreme.
The current situation is very, very bad, and there is no plan to make it better if we continue increasing AI capabilities at the current rate.
We need to stop things as soon as possible, so the community and the world can orient and work out what to do. We don't need to have the full plan yet, we just need to get to a state where we aren't in imminent danger.
I predict that this position will become increasingly obvious, and people will wish they reoriented earlier. I think this will be clear ex ante; when people look back they will think that they should have reoriented earlier, given the information they had at the time.
I wrote a quick post about why I think people committed to working on ASI+animals should be making sure we don't spread wild animal suffering throughout the universe.
Full post here: https://naiveconsequentialism.substack.com/p/dont-green-the-universe
Someone should write a good, linkable online resource describing the concept of the long reflection. It's very strange that there isn't a simple post/webpage that I can link to that gives a good, medium-depth description.
Currently the best things are probably the EA Forum Topic page, and this list of quotes.
I should read that piece. In general, I am very into the Long Reflection and I guess also the Viatopia stuff.
(Half baked and maybe just straight up incorrect about people's orientations)
I worry a bit about groups thinking about the post-AGI future (e.g., Forethought) will not want to push for something like super-optimized flourishing because this will seem weird and possibly uncooperative with factions that don't like the vibe of super-optimization. This might happen even if these groups thinking about the future do believe in their hearts that super-optimized flourishing is the best outcome.
It is very plausible to me that the situation is "convex" in the sense that it is better for the super-optimizers to optimize fully with their share of the universe, while the other groups do what they want with their share (with rules to prevent extreme suffering, pessimization etc). I think this approach might be better for all groups, rather than aiming for a more universal middle ground that leaves everyone disappointed. This bad middle ground might look like a universe that is both not very optimized for flourishing but is still super weird and unfamiliar.
It would be very sad if we miss out on the optimized flourishing because we were trying to not seem weird or uncooperative.
We may be running multiple smaller cohorts rather than one big one, if that's what maximizes the ability of strong candidates to participate.
The single most important factor in deciding the timing is the window in which strong candidates are available, and the target size for the cohort is small enough (5-20 depending on strength of applicants) that the availability of a single applicant is enough to sway the decision. It's specifically cases like yours that we're intending to accommodate. Please apply!
Announcing: 2026 MIRI Technical Governance Team Research Fellowship.
MIRI’s Technical Governance Team plans to run a small research fellowship program in early 2026. The program will run for 8 weeks, and include a $1200/week stipend. Fellows are expected to work on their projects 40 hours per week. The program is remote-by-default, with an in-person kickoff week in Berkeley, CA (flights and housing provided). Participants who already live in or near Berkeley are free to use our office for the duration of the program.
Fellows will spend the first week picking out scoped projects from a list provided by our team or designing independent research projects (related to our overall agenda), and then spend seven weeks working on that project under the guidance of our Technical Governance Team. One of the main goals of the program is to identify full-time hires for the team.
If you are interested in participating, please fill out this application as soon as possible (should take 45-60 minutes). We plan to set dates for participation based on applicant availability, but we expect the fellowship to begin after February 2, 2026 and end before August 31, 2026 (i.e., some 8 week period in spring/summer, 2026).
Strong applicants care deeply about existential risk, have existing experience in research or policy work, and are able to work autonomously for long stretches on topics that merge considerations from the technical and political worlds.
Unfortunately, we are not able to sponsor visas for this program.
See here for examples of potential projects
Could/should big EA-ish coworking spaces like Constellation pay to have far-UV installed? (either on their floors specifically or for the whole building)
MATS has a very high bar these days, I'm pretty happy about there being "knock-off MATS" programs that allow people who missed the bar for MATS to demonstrate they can do valuable work.
I still kinda feel this way about Asterisk (my opinion would change if I learned that the readership wasn't just EAs)