Quick takes

Set topic
Frontpage
Global health
Animal welfare
Existential risk
Biosecurity & pandemics
11 more

The AI Eval Singularity is Near

  • AI capabilities seem to be doubling every 4-7 months
  • Humanity's ability to measure capabilities is growing much more slowly
  • This implies an "eval singularity": a point at which capabilities grow faster than our ability to measure them
  • It seems like the singularity is ~here in cybersecurity, CBRN, and AI R&D (supporting quotes below)
  • It's possible that this is temporary, but the people involved seem pretty worried

Appendix - quotes on eval saturation

Opus 4.6

  • "For AI R&D capabilities, we found that Claude Opus 4.6 h
... (read more)

A bit sad to find out that Open Philanthropy’s (now Coefficient Giving) GCR Cause Prioritization team is no more. 

I heard it was removed/restructured mid-2025. Seems like most of the people were distributed to other parts of the org. I don't think there were public announcements of this, though it is quite possible I missed something. 

I imagine there must have been a bunch of other major changes around Coefficient that aren't yet well understood externally. This caught me a bit off guard. 

There don't seem to be many active online artifa... (read more)

4
anormative
Do you think this is evidence that OpenPhil's GCR staff/team is doing less cause prioritization now than they were before? The specific things you say don't seem to be much evidence either way about this (and also not much evidence about whether or not they actually need to be doing more cause prioritization on the margin). Maybe you have further reason to believe this is bad? What makes you expect this and why (assuming you do) do you expect these changes to be negative? 

I don't mean to sound too negative on this - I did just say "a bit sad" on that one specific point.

Do I think that CE is doing worse or better overall? It seems like Coefficient has been making a bunch of changes, and I don't feel like I have a good handle on the details. They've also been expanding a fair bit. I'd naively assume that a huge amount of work is going on behind the scenes to hire and grow, and that this is putting CE in a better place on average.

I would expect this (the GCR prio team change) to be some evidence that specific ambitious approac... (read more)

Should GiveWell offer Animal Welfare regrants on an opt-in basis?

The GiveWell FAQ (quoted below) suggests that GiveWell focuses exclusively on human-directed interventions primarily for reasons of specialization—i.e., avoiding duplication of work already done by Coefficient Giving and others—rather than due to a principled objection to recommending animal-focused charities. If GiveWell is willing to recommend these organizations when asked, why not reduce the friction a bit?

A major part of GiveWell’s appeal has been its role as an “index fund for charities... (read more)

TBH my sense is that GiveWell is just being polite.

A perhaps more realistic motivation is that admitting animal suffering into GiveWell's models would implicitly force them to specify moral weights for animals (versus humans), and there is no way to do that without inviting huge controversy leaving at least some groups very upset. Much easier to say "sorry, not our wheelhouse" and effectively set animal weights to zero.

FWIW I agree with this decision (of GiveWell's).

It seems like a worthwhile project to ask/pressure Anthropic's founders to make their pledges legally binding. 

Anthropic's founders have pledged to donate 80% of their wealth. Ozzie Gooen estimates that in a few years this could be worth >$40 billion.

As Ozzie writes, adherence to the Giving Pledge (the Gates one) is pretty low: only 36% of deceased original pledgers met the 50% commitment. It's hard to follow through on such commitments, even for (originally) highly morally motivated people.

Showing 3 of 4 replies (Click to show all)

I'm going to guess the total donated will be 30% of this by EA funders, and a low percentage by the rest. I think your conservative number is WAY too low based on previous pledge fulfillment rates. I get that it's just a claude generation though

But that's still 2 billion dollars at least, so I've updated positively on the amount of money that might go to good causes. Thanks for this @Ozzie Gooen strong upvote.

2
Ozzie Gooen
I had my Claude system do some brainstorming work on this.  https://www.longtermwiki.com/knowledge-base/models/intervention-models/anthropic-pledge-enforcement/ It generated some more specific interventions here.  
4
SiebeRozendal
Thanks, fixed the link

Mental health support for those working on AI risks and policy?

During the numerous projects I work on relating to AI risks, policies, and future threats/scenarios, I speak to a lot of people who bring exposed to issues of catastrophic and existential nature for the first time (or grappling with them for the first time in detail). This combined with the likelihood that things will get worse before they better, makes me frequently wonder: are we doing enough around mental health support?

Things that I don’t know exist but feel they should. Some may sound OTT ... (read more)

It is popular to hate on Swapcard, and yet Swapcard seems like the best available solution despite its flaws. Claude Code or other AI coding assistants are very good nowadays, and conceivably, someone could just Claude Code a better Swapcard that maintained feature parity while not having flaws.

Overall I'm guessing this would be too hard right now, but we do live in an age of mysteries and wonders. It gets easier every month. One reason for optimism is it seems like the Swapcard team is probably not focused on the somewhat odd use case of EAGs in general (... (read more)

@Yonatan Cale posted a demo last week of an app he’s building in the EAG Bay Area Slack.

What are people's favorite arguments/articles/essays trying to lay out the simplest possible case for AI risk/danger?

Every single argument for AI danger/risk/safety I’ve seen seems to overcomplicate things. Either they have too many extraneous details, or they appeal to overly complex analogies, or they seem to spend much of their time responding to insider debates.

I might want to try my hand at writing the simplest possible argument that is still rigorous and clear, without being trapped by common pitfalls. To do that, I want to quickly survey the field so I can learn from the best existing work as well as avoid the mistakes they make.

I've now written it here, thanks for all the feedback! :) https://linch.substack.com/p/simplest-case-ai-catastrophe

2
Will Aldred
my fave is @Duncan Sabien’s ‘Deadly by Default’
1
Jordan Arel
Max Tegmark explains it best I think. Very clear and compelling and you don’t need any technical background to understand what he’s saying. I believe his third or maybe it was second appearance on Lex Fridman’s podcast where I first heard his strongest arguments, although those are quite long with extraneous content, here is a version that is just the arguments. His solutions are somewhat specific, but overall his explanation is very good I think:

I've been experimenting recently with a longtermist wiki, written fully with LLMs.

Some key decisions/properties:
1. Fully LLM-generated, heavily relying on Claude Code.
2. Somewhat opinionated. Tries to represent something of a median longtermist/EA longview, with a focus on the implications of AI. All pages are rated for "importance".
3. Claude will estimates a lot of percentages and letter grades for things. If you see a percentage or grade, and there's no citation, it might well be a guess by Claude.
4. An emphasis on numeric estimates, models, and diagrams... (read more)

The next PauseAI UK protest will be (AFAIK) the first coalition protest between different AI activist groups, the main other group being Pull the Plug, a new organisation focused primarily on current AI harms. It will almost certainly be the largest protest focused exclusively on AI to date.

In my experience, the vast majority of people in AI safety are in favor of big-tent coalition protests on AI in theory. But when faced with the reality of working with other groups who don't emphasize existential risk, they have misgivings. So I'm curious what people he... (read more)

Consider adopting the term o-risk.

William MacAskill has recently been writing a bunch about how if you’re a Long-Termist, it’s not enough merely to avoid the catastrophic outcomes. Even if we get a decent long-term future, it may still fall far short of the best future we could have achieved. This outcome — of a merely okay future, when we could have had a great future — would still be quite tragic.

Which got me thinking: EAs already have terms like x-risk (for existential risks, or things which could cause human extinction) and s-risk (for suffering risks,... (read more)

might want to check out this (only indirectly related but maybe useful). 

https://forum.effectivealtruism.org/posts/zuQeTaqrjveSiSMYo/a-proposed-hierarchy-of-longtermist-concepts
 

Personally don't mind o-risk think it has some utility but s-risk ~somewhat seems like it still works here. An O-risk is just a smaller scale s-risk no?
 

Thanks to everyone who voted for our next debate week topic! Final votes were locked in at 9am this morning. 

We can’t announce a winner immediately, because the highest karma topic (and perhaps some of the others) touches on issues related to our politics on the EA Forum policy. Once we’ve clarified which topics we would be able to run, we’ll be able to announce a winner. 

Once we have, I’ll work on honing the exact wording. I’ll write a post with a few options, so that you can have input into the exact version we end up discussing. 

PS: ... (read more)

Showing 3 of 6 replies (Click to show all)
4
Toby Tremlett🔹
Thanks for the comments @Clara Torres Latorre 🔸 @NickLaing @Aaron Gertler 🔸 @Ben Stevenson. This is all useful to hear. I should have an update later this month.

Nice one @Toby Tremlett🔹 . If the forum dictators decide that the democratically selected topic of democratic backsliding is not allowed, I will genuinely be OK with that decision ;).

17
Clara Torres Latorre 🔸
I think allowing this debate to happen would be a fantastic opportunity to put our money where our mouth is regarding not ignoring systemic issues: https://80000hours.org/2020/08/misconceptions-effective-altruism/#misconception-3-effective-altruism-ignores-systemic-change On the other hand, deciding that democratic backsliding is off limits, and not even trying to have a conversation about it, could (rightfully, in my view) be treated as evidence of EA being in an ivory tower and disconnected from the real world.

Consultancy Opportunities – Biological Threat Reduction 📢📢📢


The World Organisation for Animal Health (WOAH) is looking for two consultants to support the implementation of the Fortifying Institutional Resilience Against Biological Threats (FIRABioT) Project in Africa.  Supported by Global Affairs Canada's Weapons Threat Reduction Program, this high-impact initiative aims to support WOAH Members in strengthening capacities to prevent, detect, prepare, respond and recover from biological threats. The project also supports the implementation of th... (read more)

EA Animal Welfare Fund almost as big as Coefficient Giving FAW now?

This job ad says they raised >$10M in 2025 and are targeting $20M in 2026. CG's public Farmed Animal Welfare 2025 grants are ~$35M.  

Is this right?

Cool to see the fund grow so much either way.

Agree that it’s really great to see the fund grow so much!

That said, I don’t think it’s right to say it’s almost as large as Coefficient Giving. At least not yet... :) 

The 2025 total appears to exclude a number of grants (including one to Rethink Priorities) and only runs through August of that year. By comparison, Coefficient Giving’s farmed animal welfare funding in 2024 was around $70M, based on the figures published on their website.

9
Jeff Kaufman 🔸
Specifically within animal welfare (this wasn't immediately clear to me, and I was very confused how CG's grants could be so low)
4
Ben_West🔸
Ah yeah good point, I updated the text.

A delightful thing happened a couple weeks ago, and it gives an example for why more people should comment on the forum. 

My forum profile is pretty scarce, less than a dozen comments, most of them are along the lines of 'I appreciate the work done here!'. Nevertheless, because I have linked in some social media profiles and set my city in the directory, a student from a nearby university reached out to ask about career advice after finding me on the forum. I gave her a personalised briefing on the local policy space and explained the details of how to... (read more)

Lots of “entry-level” jobs require applicants to have significant prior experience. This seems like a catch-22: if entry-level positions require experience, how are you supposed to get the experience in the first place? Needless to say, this can be frustrating. But we don’t think this is (quite) as paradoxical as it sounds, for two main reasons. 

1: Listed requirements usually aren't as rigid as they seem.

Employers usually expect that candidates won’t meet all of the “essential” criteria. These are often more of a wish list than an exhaustive list... (read more)

Anecdotally, it seems like many employers have become more selective about qualifications, particularly in tech where the market got really competitive in 2024 - junior engineers were suddenly competing with laid-off senior engineers and FAANG bros.

Also, per their FAQ, Capital One has a policy not to select candidates who don't meet the basic qualifications for a role. One Reddit thread says this is also true for government contractors. Obviously this may vary among employers - is there any empirical evidence on how often candidates get hired without meeti... (read more)

@Ryan Greenblatt and I are going to record another podcast together (see the previous one here). We'd love to hear topics that you'd like us to discuss. (The questions people proposed last time are here, for reference.) We're most likely to discuss issues related to AI, but a broad set of topics other than "preventing AI takeover" are on topic. E.g. last time we talked about the cost to the far future of humans making bad decisions about what to do with AI, and the risk of galactic scale wild animal suffering.

Showing 3 of 5 replies (Click to show all)

Much of the stuff that catches your interest on the 80,000 hours website's problem profiles would be something I'd like to watch you do a podcast on, or costly if I end up getting it from people whose work I'm less familiar with. Also, neurology, cogpsych/evopsych/epistasis (e.g. like this 80k podcast with Randy Neese, this 80k podcast with Athena Aktipis), and especially more quantitative modelling approaches to culture change/trends (e.g. 80k podcast with Cass Sunstein, 80k podcast with Tom Moynihan, 80k podcasts with David Duvenaud and Karnofsky). A lot... (read more)

2
Pablo
I’d be interested in seeing you guys elaborate on the comments you make here in response to Rob’s question that some control methods, such as AI boxing, may be “a bit of a dick move”.
7
Noah Birnbaum
I would love to hear any updated takes on this post from Ryan. 

Not sure who needs to hear this, but Hank Green has published two very good videos about AI safety this week: an interview with Nate Soares and a SciShow explainer on AI safety and superintelligence.

Incidentally, he appears to have also come up with the ITN framework from first principles (h/t @Mjreard).

Hopefully this is auspicious for things to come?

Showing 3 of 5 replies (Click to show all)
4
Arnold Beckham
Only if someone's inviting him perhaps? @akash 🔸 

I just emailed him, close to zero chance he will see it but if he does 🤞

14
Lorenzo Buonanno🔸
My understanding is that they already raise and donate millions of dollars per year to effective projects in global health (especially tuberculosis) For what it's worth, their subreddit seems a bit ambivalent about explicit "effective altruism" connections (see here or here)   Btw, I would be surprised if the ITN framework was independently developed from first principles: * He says exactly the same 3 things in the same order * They have known about effective altruism for at least 11 years (see the top comment here) * There have been many effective altruism themed videos in their "Project for Awesome" campaign several years * They have collaborated several times with 80,000 hours and Giving What We Can * There are many other reasonable things you can come up with (e.g. urgency)

I'm researching how safety frameworks of frontier labs (Anthropic RSP, OpenAI Preparedness Framework, DeepMind FSF) have changed between versions.

Before I finish the analysis, I'm collecting predictions to compare with actual findings later. 5 quick questions. Questions

Disclaimer: please take it with a grain of salt, questions drafted quickly with AI help, treating this as a casual experiment, not rigorous research.

Thanks if you have a moment

@Toby Tremlett🔹 and I will be repping the EA Forum Team at EAG SF in mid-Feb — stop by our office hours to ask questions, give us your hottest Forum takes, or just say hi and come get a surprise sweet! :)

Reminder: applications for EAG SF close soon (this Sunday!)

3
NickLaing
Unfortunately it's not a surprise sweet any more, you really messed that one up. 

The surprise is what kind of sweet! ^^

I’ve seen a few people in the LessWrong community congratulate the community on predicting or preparing for covid-19 earlier than others, but I haven’t actually seen the evidence that the LessWrong community was particularly early on covid or gave particularly wise advice on what to do about it. I looked into this, and as far as I can tell, this self-congratulatory narrative is a complete myth.

Many people were worried about and preparing for covid in early 2020 before everything finally snowballed in the second week of March 2020. I remember it personally.... (read more)

Showing 3 of 25 replies (Click to show all)

Following up a bit on this, @parconley. The second post in Zvi's covid-19 series is from 6pm Eastern on March 13, 2020. Let's remember where this is in the timeline. From my quick take above:

On March 8, 2020, Italy put a quarter of its population under lockdown, then put the whole country on lockdown on March 10. On March 11, the World Health Organization declared covid-19 a global pandemic. (The same day, the NBA suspended the season and Tom Hanks publicly disclosed he had covid.) On March 12, Ohio closed its schools statewide. The U.S. declared a nationa

... (read more)
2
Yarrow Bouchard 🔸
I spun this quick take out as a full post here. When I submitted the full post, there was no/almost no engagement on this quick take. In the future, I'll try to make sure to publish things only as a quick take or only as a full post, but not both. This was a fluke under unusual circumstances. Feel free to continue commenting here, cross-post comments from here onto the full post, make new comments on the post, or do whatever you want. Thanks to everyone who engaged and left interesting comments.
4
Jason
I like this comment. This topic is always at risk to devolving into a generalized debate between rationalists and their opponents, creating a lot of heat but not light. So it's helpful to keep a fairly tight focus on potentially action-relevant questions (of which the comment identifies one).
Load more