Benevolent_Rain's Quick takes

I now think principles-first EA is more important than I previously thought because it helps prevent effectiveness drift. My anecdotal evidence from AI Safety and especially biosecurity gives me the impression that without constant anchoring to EA and especially comparisons to the clear ToCs and tractability for e.g. AMF, it is easy to lose focus on the high demands of choosing x-risk as a cause area over others. I previously placed less value on having strong links between EA and the various cause areas but now think I should update to thinking strong, continuous engagement with EA is important to keep one's focus on each intervention's cause prioritization assumptions that make it comparable to e.g. AMF or cage free chicken campaigns. This is not to say that causes such as AI Safety and biosec are not important, but that unless constantly tied back to EA cause prioritization, there is risk of drifting away from what made the cause look good in the first place. An example from biosecurity is the very easy slippage away from human extinction scenarios to ones where nearly everone dies (the difference being a crux as it is the potentially enormous future one is saving, not the people living at the time of the catastrophe). That said, it is completely fine and commendable that there is AI Safety and biosecurity work that does not target existential threats, but for EAs such changes in the nature of the work means they should consider changing their career. I think this is also important for newcomers to EA: For those of us who were around when we discussed whether x-risks demanded attention we might take concepts such as Pascal's Mugging as obvious, but for newcomers it is important to engage with such concepts. Another observation I have made is that in animal welfare and global health one is constantly reminded by metrics of suffering alleviated per dollar, but such recurring reminders are lacking in x-risk focused cause areas.

Please list any new funding opportunities you can think of here on the Forum? I feel like we might already be in the early ramp-up to significantly more EA aligned funding. At the same time, the Forum's overview over funding opportunities feels like it is quickly getting outdated. I think as things move quickly, coordination might become looser and new promising interventions are identified, it is helpful for people to have a good overview over available funding sources and their priorities.

I have heard on the grapevine there is already funding on several fronts that might not be very public. I am a little bit uncertain if perhaps it is better these sources remain anonymous. At the same time, I think there might be several promising EA projects that are not sufficiently visible to people influencing funding decisions. 

Epistemic note: I am not listing these yet as I have not had time yet to verify how much they qualify as EA funding opportunities.

Here are a few recent developments I am considering listing on the funding opportunities page, but would like someone that knows these funds better to list them:

• Probably several I have missed - please list these here
• OpenAI Foundation - AI Resilience
• OpenAI Rosalind
• The Launch Sequence (not a fund in itself, but plausibly one can treat this kind of like a funding source?)
• Several of Renaissance Philanthropy's (RP) funds (several quite plugged in EAs do not even know of RP!)
• Astralis Foundation

The book "Careless People" starts as a critique of Facebook — a key EA funding source — and unexpectedly lands on AI safety, x-risk, and global institutional failure.

I just finished Sarah Wynn-Williams' recently published book. I had planned to post earlier — mainly about EA’s funding sources — but after reading the surprising epilogue, I now think both the book and the author might deserve even broader attention within EA and longtermist circles.

1. The harms associated with the origins of our funding

The early chapters examine the psychology and incentives behind extreme tech wealth — especially at Facebook/Meta. That made me reflect on EA’s deep reliance (although unclear how much as OllieBase helpfully pointed out after I first published this Quick Take) on money that ultimately came from:

• harms to adolescent mental health,
• cooperation with authoritarian regimes,
• and the erosion of democracy, even in the US and Europe.

These issues are not new (they weren’t to me), but the book’s specifics and firsthand insights reveal a shocking level of disregard for social responsibility — more than I thought possible from such a valuable and influential company.

To be clear: I don’t think Dustin Moskovitz reflects the culture Wynn-Williams critiques. He left Facebook early and seems unusually serious about ethics.
But the systems that generated that wealth — and shaped the broader tech landscape could still matter.

Especially post-FTX, it feels important to stay aware of where our money comes from. Not out of guilt or purity — but because if you don't occasionally check your blind spot you might cause damage.

2. Ongoing risk from the same culture

Meta is now a major player in the frontier AI race — aggressively releasing open-weight models with seemingly limited concern for cybersecurity, governance, or global risk.

Some of the same dynamics described in the book — greed, recklessness, detachment — could well still be at play. And it would not be completely surprising if such culture to some extent is being replicated across other labs and institutions involved in frontier AI.

3. Wynn-Williams is now focused on AI governance (e.g. risk of nuclear war)

In the final chapters, Wynn-Williams pivots toward global catastrophic risks: AI, great power conflict, and nuclear war.

Her framing is sober, high-context, and uncannily aligned with longtermist priorities. She seems to combine rare access (including relationships with heads of state), strategic clarity, and a grounded moral compass — the kind of person who can get in the room and speak truth to power. People recruiting for senior AI policy roles might want to reach out to her if they have not already.

I’m still not sure what the exact takeaway is. I just have a strong hunch this book matters more than I can currently articulate — and that Wynn-Williams herself may be an unusually valuable ally, mentor, or collaborator for those working on x-risk policy or institutional outreach.

If you’ve read it — or end up reading it — I’d be curious what it sparks for you. It works fantastically as an audiobook, a real page turner with lots of wit and vivid descriptions.

I’m pro-nuclear, but the commonly used EA framing of “nuclear is overregulated” seems net negative more often than not. Clearer Thinking’s new nuclear episode is one of the more epistemically rigorous discussions I’ve heard in EA-adjacent spaces (and Founders Pledge has also done nuanced work).

Nuclear is worth pursuing, but we should argue for it clear-eyed.

Was at EAG, "no seats left" on all AI talks, "57 seats left" on the biosec ones. Fully agree with others that biosec field building seems underserved. Anecdotal but perhaps illustrative. Thanks CEA for another event full of amazing people.

The electric grid is a powerful, current AI safety policy opportunity. ~25% of the cost of data centres is electricity and the electric grid is heavily regulated across government agencies. In fact, the very reason why many people claim nuclear energy is over regulated is exactly why we might be able to regulate AI strongly via electric grid regulation. This means "the grid" could be a strong contender for a space to make rapid and robust progress on AI safety policy. As an expert in the electric grid with a strong interest in AI safety and resilience, I see an electricity sector full of opportunities for quick wins and actually moving the needle. While "high-level" policy interventions such as SB 53 are important, a drawback of these types of policy interventions is twofold:
1 - They attract enormous public scrutiny. Electric regulation, on the other hand, hardly make it into local newspapers or industry press even.
2 - They don't really move the needle. They are more aspirational and rely on outcomes in court cases, enforcement, etc. The electric grid already has "kill switches" installed and integration with national security.

On the other hand, in the electric grid, one can plausibly pass very strong regulation with physical, actual AI safety levers. Some examples:
A - Large consumers of electricity are critical to the grid. It is not unforeseeable at all that gov't bodies could require a "kill switch" for data centres, not because of AI safety but due to grid health. 
B - The military is extremely focused on electrical grids. They are also a likely gov't body that would act quickly and decisively on perceived threats on the grid. They likely influence requirements for cyber security of the electrical grid. This can include monitoring of critical load (yes, data centres) and access to the above "kill switch". 

These are just two of many ideas I have around making rapid, robust progress on AI safety via much less public scrutiny, and using existing pathways and gov't focus areas around electric grid management and national defense. I have many more ideas and many years of working in and following the industry as well as a large professional network if anyone wants to talk - please DM me!

I’m skeptical that corporate AI safety commitments work like @Holden Karnofsky suggests. The “cage-free” analogy breaks: one temporary defector can erase ~all progress, unlike with chickens.

I'm less sure about corporate commitments to AI safety than Karnofsky. In the latest 80k hrs podcast episode, Karnofsky uses the cage free example of why it might be effective to push frontier AI companies on safety. I feel the analogy might fail in potentially a significant way in that the analogy breaks in terms of how many companies need to be convinced:
-For cage free chicken, convincing one company, even just for a few months, is a big win
-For frontier AI companies, you might not win until every single company is convinced, forever. One company not committing, perhaps only for a few months, and the risk reduction could evaporate, or at least take a significant hit

I do recognize that it might be more nuanced but felt the 80k interview overstated optimism on this front. For example, steel-manning his argument, maybe if one gets 60% "coverage" in a critical period, it still reduces risk significantly. But if it is to a large degree a "cat-out-of-the-bag" situation, the bag only needs to be open briefly.

Perhaps I am missing something obvious, so useful if people can correct me.

Could we please start propagating more nuanced and informed information about nuclear power? I have heard at least 2 times recently from prominent EAs on podcasts that nuclear is too expensive because it is over-regulated which is likely to be misguided at best and wrong at worst. The reason to have better epistemic hygiene around nuclear energy is because I imagine many people exploring EA for the first time to have a climate change background and it will likely seem both weird and off-putting to them if a movement that is supposedly evidence-driven throws out quite uninformed and controversial statements about a hot-button topic.  

There is an excellent post on LessWrong about nuclear power that shows not even the pro-nuclear lobby claim high costs due to over-regulation. There are many good reasons to invest in nuclear, and Johannes Ackva has written and talked about this many times and the biggest reason for nuclear is to have a back-up in case we run into problems converting to a renewable energy-dominated electric grid.

I am very happy to clarify topics around nuclear, coming from the energy industry myself. And for prominent EAs I am sure Johannes also has time to inform people before they make public statements on the topic.

Community > Epistemics
Community is more important to EA than epistemics. What drives EA's greater impact isn’t just reasoning, but collaboration. Twenty “90% smart” people are much more likely identify more impactful interventions than two “100% smart” people.

I may be biased by how I found EA—working alone on “finding most impactful work” before stumbling into the EA community—but this is the point: EA isn’t unique for asking, “How can I use reason to find the most impactful interventions?” Others ask that too. EA is unique because it gathers those people, and facilitates funding and coordination, enabling far more careful and comprehensive work.

Sales professionals might be able to meaningfully contribute to reducing bio x-risk. They could do so by working for germicidal UV companies in promoting their product and increasing sales. This is not my own idea, but I do not think I have seen this career track before and thought it might be useful to some - people with sales backgrounds might not easily find impactful roles (perhaps apart from fundraising and donor relations). If you need more details please just comment here and I will give as much detail as I have on this opportunity.

What I’ve learned from informal background checks in EA

I sometimes do informal background or reference checks on "semi-influential" people in and around EA. A couple of times I decided not to get too close — nothing dramatic, just enough small signals that stepping back felt wiser. (And to be fair, I had solid alternatives; with fewer options, one might reasonably accept more risk.)

I typically don’t ask for curated references, partly because it feels out of place outside formal hiring and partly because I’m lazy — it’s much quicker to ask a trusted friend what they think than to chase down a stranger who was pre-selected to say something nice.

Main takeaway: curated references tell you little. What actually helps is asking trusted mutuals or mutuals-of-mutuals who’ve worked with the person directly — ideally when things weren’t going perfectly. Ask what went wrong, how it was handled, and whether they’d recommend working closely again. Those short, candid conversations are gold.

People in EA are surprisingly open to act as such informal references if you approach them with integrity and transparency — they’ll tell you what they know, what they don’t, and often volunteer what to watch for. But you would need to build trust in advance. Thus, building trust in EA might be underrated. That said, I think if you consistently act with honesty, you’ll soon have access to genuinely useful informal information.

If you want to build strong collaborations here, earn trust by being open and careful — and don’t hesitate to cross-check before partnering or taking funding. On funding: it’s often wise to ask for concrete commitments quickly. Some people genuinely mean well but keep others waiting for months because they’re themselves over-committed.

(For context: informal reference checks not feeling right has only happened twice for me, and others might have seen the same things and made different judgment calls. That’s fine — I just tend to stay on the cautious side. Also, I’m no expert; but I’ve seen things go wrong and might have some biasing battle scars. Happy to hear suggestions or additional thoughts on how others approach this.)

Risk neutral grantmakers should, if they have not already, strongly consider modifying their position. If such a grantmaker has a choice of an intervention with 1000 utils of potential impact but only 1% chance of working out (10 utils in expectation), and an intervention with 10 utils of potential impact but 90% likely to work out (9 utils in expectation), I would suggest that one should go with the latter at this point where the x-risk community is hopefully still in its early days. 

The reason is that having wins has value in and of itself. I think this is especially true in the x-risk domain where the path to impact is uncertain and complex. At least now, in the hopefully early days of such work, there might be significant value by just demonstrating to ourselves, and perhaps major donors on the fence on whether to become "EA/x-risk donors" and also perhaps talent wondering if EA "is for real", that we can do something.

Metaculus does not seem to predict any significant reduction in risk from AI despite the recent movement on AI safety by the US and UK governments. If anything, it seems that for one reason or another, risks might have increased according to Metaculus. I am sharing as I would perhaps naively have thought that these government actions would significantly reduce risk, rather than being a tiny, hopeful first step in a long sequence of events that have to unfold to meaningfully reduce risk. However, I think my enthusiasm was correlated more with the amount of mainstream media coverage rather than any meaningful change in the risk landscape.

I looked at this, this, this and this question on Metaculus. However, I did not dig deep, I only looked visually at the very last tiny bit of the graphs where I did not have enough resolution for daily changes in probabilities so there could be something I missed. If anyone is using this to make big decisions, I recommend they do more rigorous work. It could also be that Metaculus forecasters are slower to update than I expect, even though more than 1 week has passed since the announcements by the UK and the US gov't - I would love to be proved wrong!

Tu Youyou might be a model for EAs. According to Wikipedia she has saved millions of lives due to her discovery of treatments for Malaria for which she received a Nobel prize. I am guessing, without having done research that at least hundred thousand of these lives might be counterfactually saved due to the time it would take until the next person made this discovery. I randomly came across her going down a GPT/Wikipedia rabbit hole and was surprised to see her not mentioned once on the Forum so far. That said, I am unsure how many people there are that might have counterfactually saved ~100k people or more.

Not well researched at all, but it seems there is some issue in the EA community with unwanted romantic/sexual advances. What I do not get is how people get into these situations. I thought it was common sense not to make a romantic/sexual advance until you have pretty strong evidence that it might be welcomed? Before one makes a sexual/romantic suggestion, at least one, and ideally several of the below should be true:

• The person repeatedly seeks you out and initiates contact
• The person either looks intently into your eyes or acts shy (not scared - it should not be hard to tell the difference!)
• The person touches you 
• The person has said yes to hang out on several occasions non-romantically (e.g. coffee, lunch, going for a walk during the day, etc.)
• You have asked a common friend if the person might be interested and gotten a probability >70%
• This is a non-exhaustive list, probably lots of other signs I am missing - this is just off the top of my head there is tons written about this (unsurprisingly)

Also, this person is not someone you have much power over. If you do have power, you need to act much more carefully and ideally not act at all - it might be net negative in expectation. This is because some of these signs might appear because of the power differential and not the attraction (although that can get more complicated!).

If you make a romantic/sexual advance despite none of the above being true you are asking for trouble. Especially in a mostly professional space like EA I think people should think hard about making advances without any of the "established" signs. And even if you get rejected if some of the signs are there, you are less likely to face backlash as the person probably likes you as a friends or something and will give you leeway.

This is inspired by a lot of recent discussion on "but isn't it ok for people to make an advance - if they didn't nobody would even be in romantic relationships". Others might have raised it but what I see missing is discussion of this crucial, pre-romantic stage and how that should update one's romantic priors.

Ambition is like fire. Too little and you go cold. But unmanaged it leaves you burnt.