All the c3 orgs I mentioned are funding-constrained (except perhaps MIRI). AI x-risk advocacy is unpopular among big funders (with a couple exceptions like SFF), which means both the c3s and c4s are funding-constrained. At least that's true currently; I'm not sure if that will still be true a year from now.

I agree with you about the structural funding disadvantage to c4s, but empirically it doesn't look obviously important for driving funding constraints.

I'm also uncertain about the relative value of lobbying vs. other forms of advocacy. My intuition is that direct lobbying is underfunded relative to its impact, partly because the structural barriers to funding it are so high, but I'd love to see more analysis on this.

Yeah this is a difficult question, I don't know. Another problem is that lobbying is opaque so it's harder to tell who's doing a good job. (ControlAI is my favorite lobbying org because they write a lot about what they do and I like what they write. But also I reached out to them and they said they were not accepting donations—this was a couple months ago.)

I'm not sure if there's any. My concerns are more theoretical than empirical, so it would take theoretical work to significantly change my mind.

Empirical work can provide a small amount of information, e.g. the fact that Claude expresses concern for ethics is a slight positive update relative to the world where Claude doesn't care about ethics, and I would feel slightly better about a Claude-based ASI than a ChatGPT-based ASI. But only slightly, because I don't think empirically observable behavior is that relevant to determining whether an AI is aligned. At least not using any empirical methods that we've devised so far.

For more on this, see e.g. A central AI alignment problem: capabilities generalization, and the sharp left turn, especially the part starting from 'How is the "capabilities generalize further than alignment" problem upstream of these problems?'

(ETA: On how various plans miss the hard bits of the alignment challenge is also kind of about this...I was looking around for some writings on why current empirical work isn't that relevant but it's hard to find anything that directly makes the argument)

From my POV we are deeply confused about what it would even mean to align ASI. If I could even describe what sort of theoretical work would be good evidence of progress on alignment, we'd be in a better place than we are currently.

A significant reason for my high P(doom) is that most safety researchers at AI companies are ignoring theoretical issues and pretending that alignment is purely an engineering problem. I don't think they are institutionally capable of solving alignment.

1. An LLM's expressed values are not the same thing as its actual values, insofar as it has any.
2. This paper doesn't really tell us anything about how ASI values will work. This paper is relevant to the immediate problem of making Claude commercially useful and non-destructive, but it's not relevant to ASI.

This isn't something I've thought a ton about but I think forecasting should plausibly still receive funding in a specific way:

Funders should either pay forecasters to make predictions on important questions, or subsidize prediction markets on those questions.

I don't think forecasting is a "solution seeking a problem." There are tons of important but hard-to-predict questions that I'd like better forecasts on! The problem is that the ecosystem hasn't done a great job of turning dollars into good forecasts.

For example, most of my Metaculus questions are things I wanted answers to, but I tended not to update on the results because the questions usually don't receive a lot of forecasts. If someone wanted to pay money to get more predictions on questions, I'd learn something useful!

I'm not sure how valuable this is compared to other uses of money (I wouldn't pay for it myself) but at least it's better than more general-purpose research on forecasting.

Thanks for this post! I broadly agree with your introduction but I'd say the "research vs. advocacy" distinction is more important than "501(c)(3) vs. 501(c)(4)". There are 501(c)(3) advocacy orgs including MIRI, Palisade, PauseAI US/PauseAI Global, Lightcone Infrastructure (sort of), etc. Lobbying on policy measures is one component of advocacy, but not the only component. I'm uncertain about how valuable it is compared to other kinds of advocacy that 501(c)(3)s can do.

AI systems are being built to do exactly what they were designed to do, which is to faithfully execute human preferences. And those are, in aggregate, to eat cheap meat, conduct research on living organisms when it's convenient, and prioritise cost and efficiency in agricultural supply chains. AI is reflecting the values of the humans. I don't think you can sneak those values in, unless there are specific opportunities to tweak things here and there before they get cemented.

This is true in a way. A deeper problem is that we don't know what values AI is reflecting. If you talk to an LLM, it will express some values, but it gives inconsistent answers depending on what questions you ask. We have no way of knowing whether its expressed values reflect its "true" values, if it has any. And we don't know how things will change as AI becomes increasingly powerful.

Is voting a Pascal's mugging?

Is searching for a breakthrough in cancer treatment a Pascal's mugging?

Marcus also nudged his friend Ege Erdil to start Mechanize, and offered them their first investment.

Mechanize's mission is to build AGI to replace human labor. They actively deny that AI misalignment risk is a concern. I would not list this as a positive under AI safety track record.

I keep seeing people use the word "dead" in this way and I don't get it. Today so far I see 8 posts and 2 shortforms. Dead would be 0 posts and 0 shortforms. Do you have some standard for how many posts the EA Forum should have per day?