AMA: Peter Wildeford (Co-CEO at Rethink Priorities)

Hi James,

Thanks for your thoughtful question, but I think you’re thinking about this incorrectly for a few reasons:

Firstly, while we raised $10.7M, most of that was earmarked for 2023 as we usually raise money in the current year for the following year. In 2022, we spent around $6.8M on RP core programs, not including special projects and operations to support special projects.

Secondly, we actually have published less than half of our 2022 research. My rough guess is that in 2022 we produced over 100 pieces of work, not ~64 as you estimate. This is for two reasons:

• Some research is confidential for whatever reason and is never intended to be published

• Some research is intended to be published but we haven’t had the resources or time to publish it yet because public outputs are not a priority for our clients and their funding does not cover it (this is actually something we’d love to get money from the EA public for).

To give a more clear substitute figure, we generally say that $20K-$40K pays for a typical short-term research project and $70K-$100K pays for a typical in-depth research project.

But more importantly I'd add that counting outputs per dollar is not a good way to view RP’s work. This is for a few reasons:

• “Outputs” are a vanity metric and we don’t want to take a quantity-focused approach to our work where we aim to produce as many outputs as cheaply as possible. Instead, the quest to quantify the impact – and the impact per dollar – of Rethink Priorities is much more difficult.

• “Outputs” vary a lot in size, scope, and funding and aren’t really apples-to-apples comparable in a way that would work for an aggregated metric/count. Some research reports take >12 months of full-time work whereas other research reports (especially internal ones not meant for publication) are completed in two weeks or less.

• A lot of the most important work that happens with our research isn’t the time spent actually producing the report, but also engaging with stakeholders, presenting findings, and/or providing feedback on others' research. We also spend a fair amount of time after we produce research reports engaging with the client — answering follow-up questions and/or otherwise helping people understand the research.

• Doing good research at scale requires a sizable operations budget, management budget, strategy work, etc., that doesn’t immediately translate into concrete publishable research outputs.

• We also produce other endpoints that aren’t research. We also incubate organizations and spend researcher time advising those organizations and we spend money to organize conferences and other events.

Love the question

Relatedly, how much of the funding (both for 2022 and for 2024) is for the production of research outputs, compared to how much it is for other operations (like fiscal sponsorships or incubation)?

I think this really depends on the research output. $100k for a report with roughly one person year's worth of effort seems about right. Or roughly one good academic paper or master's thesis. I suspect a lot of Rethink's reports are more valuable than that.

That's $100k all in cost, including costs that aren't specific to a project. Including salary, overheads, taxes, travel, any expenses, training, recruitment etc.

Do you have a sense of how much funding this informed?

I agree this is confusing. I get into this in my answer to Sebastian Schmidt.

It was an AMA similar to this one where Will Macaskill revealed that he took antidepressant medication and that actually had a large impact on me. I have historically struggled with anxiety and depression and Will’s response contributed a large portion of the reason why I chose to ask my doctor about SSRIs in 2019. Luckily they worked and hopefully by sharing my experience I can pay this forward.

Howie Lempel has also been very open about his experience. I think mental health concerns are common among EA “leaders” and I think they have been pretty open about it. I hope that continues and we could always use more.

I have been lucky to find antidepressants, talk therapy, regular exercise, and proactively engaging with a supportive friend group to be a great combination to alleviate the ways in which anxiety would otherwise derail my day. I encourage other people suffering from these conditions to explore these options.

Anxiety and depression will still be a lifelong struggle for me. Even with all of this there are still a few days a year where I am so anxious and depressed that I sleep for sixteen hours and barely get out of bed. But it’s much less worse due to me being lucky enough to have effective treatment.

I disagree; last I checked most AI safety research orgs think they could make more good hires with more money and see themselves as funding-constrained-- at least all 4 that I'm familiar with: RP, GovAI, FAR, and AI Impacts.

Edit: also see the recent Alignment Grantmaking is Funding-Limited Right Now (note that most alignment funding on the margin goes to paying and supporting researchers, in the general sense of the word).

I agree with Zach’s comment that other organizations are also underfunded and so this is not a unique view among RP. See also my comment to Aaron Bergman on donation opportunities. I think my comment to Sebastian Schmidt also helps answer this question and gives a bit more context about how and why RP has been less focused on talent gaps historically.

Honestly I love this question but I got asked a lot of real questions that I think were varied and challenging, so right now I don't currently feel like I need even more!

I do think RP, like nearly any other organization, has to “follow the money” and given that RP has historically relied a lot on restricted assets we do end up matching donor priorities. I think this could be good as it gives an independent check on our prioritization and encourages us to be responsive to the needs of the broader dollar-weighted EA community.

On the other hand, it is unlikely that donor priorities are indeed the best thing for us to work on and as we are funding constrained in all of our areas I do worry that we will be steered toward particular areas more than I’d like from impact assessment alone. This is one reason why we’ve been trying to make a large push to get more unrestricted funding for RP.

I feel a lot of cluelessness right now about how to work out cross-cause comparisons and what decision procedures to use. Luckily we hired a Worldview Investigations Team to work a lot more on this, so hopefully we will have some answers soon.

In the meantime, I currently am pretty focused on mitigating AI risk due to what I perceive as both an urgent and large threat, even among other existential risks. And contrary to last year, I think AI risk work is actually surprisingly underfunded and could grow. So I would be keen to donate to any credible AI risk group that seems to have important work and would be able to spend more marginal money now.

As Co-CEO of RP, I am obligated to say that our AI Governance and Strategy Department is doing this work and is actively seeking funding. Our work on Existential Security and doing surveys is also very focused on AI and are also funding constrained. You can donate to RP here.

…but given that you asked me specifically for non-RP work here is my ranked list of remaining organizations:

1. Centre for Long-Term Resilience (CLTR) does excellent work and appears to me to be exceptionally well-positioned and well-connected to meet the large UK AI policy window, especially with the UK AI Summit. My understanding is that they have funding gaps and that marginal funding could encourage CLTR to hire more AI-focused staff than they otherwise would and thus hopefully make more and faster progress. Donate here.
2. The Future Society (TFS) does excellent work on EU AI policy and in particular the EU AI Act. TFS is not an EA organization but that doesn’t mean they don’t do good work and in my personal opinion I think they are really underrated among the EA-affiliated AI governance community. They historically have not focused on fundraising as much as I think they should and they seem to now have a sizable funding gap that I think they could execute well on. Donate here.
3. Centre for Governance of AI (GovAI) is also doing great work on AI policy in both the US and UK and is very well-positioned. I think it’s plausible that they do the best work of any AI policy organization out there (including RP) – the reason I rank them third is mainly because I’m skeptical of the size of their funding gap and their plans for using marginal money. Donate here.
4. The Humane League (THL) is my honorable mention. I view AI risk mitigation work as more pressing than animal welfare work right now, but I still care a lot about the plight of animals and so I still support THL. They have a sizable funding gap, execute very competently, and do great work. I think the moral weight work that Rethink Priorities did made some in EA think that shrimp or insect work is more cost-effective than the kind of work that THL does but I don’t actually think that’s true insofar as readily available donation opportunities exist, though I’m unsure what other RP staff think and of course RP does research on shrimp and insects that we think is cost-effective. Donate here.

Here are my caveats for the above list:

• These are my own personal opinions. Your view may differ from mine even if you agree with me on the relevant facts due to differing values, differing risk tolerances, etc.
• I haven’t thought about this that much and I’m answering off-the-cuff for an AMA. This is definitely very subject to change as my own opinions change. I view my opinions on donation targets to be unusually in flux right now.
• Statements I have made about these organizations are my own opinions and have not been run by representatives of those organizations. Therefore, I may have misrepresented them.
• I don’t know details about room for more funding for these organizations which could change their prioritization in important ways.
• Note the question of where to donate as an individual and what to encourage RP to do as co-CEO of Rethink Priorities are very different, so this list shouldn’t necessarily be taken as indications of RP’s priorities.
• I focused on concrete “endpoint grants” but you may find more value in trusting a grant recommender and giving them money, such as via the Long-Term Future Fund, Manifund, Nonlinear, etc.
• I also value giving to political candidates and could view this as plausibly better than some options on my list above but due to US 501c3 law, I don’t want to solicit donations to such candidates.
• I know very little about the technical alignment landscape so perhaps there are good AI risk mitigation efforts there to support that would beat the options I recommend.
• A lot of information I am relying on to create this list is confidential and there’s also likely a lot of additional confidential information I don’t know about that could change my list.
   

In honor of this question and to put some skin in the game behind these recommendations, I have given each of the four organizations I listed $1000.

Thanks for the compliment! We have considered a few times but ultimately have declined to take the opportunity to review due to:

• There are capacity limitations on our end.

• We have concerns around how Rethink Priorities would be viewed by ACE’s audience given that we do a lot of research work in many different areas.

• We like the opportunity to be constructively critical of ACE’s research work and like that they are also willing to challenge and push back on our research work. We are concerned this dynamic might get complicated if we are in a clear reviewer-reviewee relationship.

We do work with ACE a lot and are excited to continue to work with them. We'd definitely consider doing an ACE review in future years if invited. We also hope that fans of our work will consider supporting us financially even if we don't have an ACE top charity designation!

Yes. I think animal welfare remains incredibly understudied and thus it is easier to have a novel insight, but also there is less literature to draw from and you can end up more fundamentally clueless. Whereas in global health and development work there is much more research to draw from, which makes it nicer to be able to do literature reviews to turn existing studies and evidence into grant recommendations, but also means that a lot of the low-hanging fruit has been done already.

Similarly, there is a lot more money available to chase top global health interventions relative to animal welfare or x-risk work, but it is also comparably harder to improve recommendations as a lot of the recommendations are already pretty well-known by foundations and policymakers.

AI has been an especially interesting place to work in because it has been rapidly mainstreaming this year. Previously, there was not much to draw on but now there is much more to draw from and many more people are open to being advised on work in the area. However, there are also many more people trying to get involved and work is being produced at a very rapid pace, which can make it harder to keep up and harder to contribute.

p(doom|AGI)

As for existential risk, my current very tentative forecast is that the world state at the end of 2100 to look something like:

73% - the world in 2100 looks broadly like it does now (in 2023) in the same sense that the current 2023 world looks broadly like it did in 1946. That is to say of course there will be a lot of technological and sociological change between now and then but by the end of 2100 there still won't be unprecedented explosive economic growth (e.g.., >30% GWP growth per year), no existential disaster, etc.

9% - the world is in a singleton state controlled by an unaligned rogue AI acting on its own initiative.

6% - the future is good for humans but our AI / post-AI society causes some other moral disaster (e.g., widespread abuse of digital minds, widespread factory farming)

5% - we get aligned AI, solve the time of perils, and have a really great future

4% - the world is in a singleton state controlled by an AI-enabled dictatorship that was initiated by some human actor misusing AI intentionally

1% - all humans are extinct due to an unaligned rogue AI acting on its own initiative

2% - all humans are extinct due to something else on this list (e.g., some other AI scenario, nukes, biorisk, unknown unknowns)

I think conditional on producing minimal menace AI by the end of 2070, there’s a 28% chance an existential risk would follow within the next 100 years that could be attributed to that AI system.

Though I don’t know how seriously you should take this, because forecasting >75 years into the future is very hard.

Also my views of this are very incomplete and in flux and I look forward to refining them and writing more about them publicly.

Re existential security, what are your AGI timelines

I have trouble understanding what “AGI” specifically refers to and I don’t think it’s the best way to think about risks from AI. As you may know, in addition to being co-CEO at Rethink Priorities, I take forecasting seriously as a hobby and people actually for some reason pay me to forecast, making me a professional forecaster. So I think a lot in terms of concrete resolution criteria for forecasting questions and my thinking on these questions has actually been meaningfully bottlenecked right now by not knowing what those concrete resolution criteria are.

That being said, being a good thinker also involves having to figure out how to operate in some sort of undefined grey space, and so I should be at least somewhat comfortable enough with compute trends, algorithmic progress, etc. to be able to give some sort of answer. And so I think for the type of AI that I struggle to define but am worried about – the kind that has the capability of autonomously causing existential risk – the kind of AI that AI researcher Caroline Jeanmaire refers to as the “minimal menace” – I am willing to tentatively put the following distribution on that:

• 5% probability of happening before 2035
• 20% probability of before 2041
• 50% probability of before 2054
• 80% probability of before 2400

(Though of course that's my opinion on my distribution, not saying that Caroline or others would agree.)

To be clear that’s an unconditional distribution, so it includes the possibility of us not producing “minimal menace” AI because we go extinct from something else first. It includes the possibility of AI development being severely delayed due to war or other disasters, the possibility of policy delaying AI development, etc.

I'm still actively working on refining this view so it may well change soon. But this is my current best guess.

do you support efforts calling for a global moratorium on AGI (to allow time for alignment research to catch up / establish the possibility of alignment of superintelligent AI)?

I'm definitely interested in seeing these ideas explored, but I want to be careful before getting super into it. My guess is that a global moratorium would not be politically feasible. But pushing for a global moratorium could still be worthwhile to pursue even if it is unlikely to happen as it could be a good galvanizing ask that brings more general attention to AI safety issues and make other policy asks seem more reasonable by comparison. I'd like to see more thinking about this.

On the merits of the actual policy, I am unsure whether a moratorium is a good idea. My concern is may just produce a larger compute overhang which could increase the likelihood of future discontinuous and hard-to-control AI progress.

Some people in our community have been convinced that an immediate and lengthy AI moratorium is a necessary condition for human survival, but I don't currently share that assessment.

I use both Manifold and Metaculus every day and it’s not really clear to me which I spend time on more. The answer is “a lot” to both.

For joy of use, I think Manifold has worked hard to make the forecasting process very seamless and I like that. I also like the gamification of the mana profit system. That being said, I think the questions on Metaculus tend to be more interesting. I personally like having rigorous resolution criteria and I personally prefer being able to give my true probabilities rather than bet up or down. So Metaculus might suit my personality better.

Surprisingly I don’t really have a clear read which platform is more accurate. So I think the value of information is optimized by using both platforms. I’m keen to see this researched more.

By answering this question I should disclose that Metaculus pays me money for being a forecaster. I suppose Manifold also indirectly pays me money because RP is part of their Manifold for Charity program. So my feelings towards them are not exactly unbiased.

I think it varies a lot by cause area but I think you would be unsurprised to hear me recommend more marginal thinking/research. I think we’re still pretty far from understanding how to best allocate a doing/action portfolio and there’d still be sizable returns from thinking more.

RP’s arrangement here is definitely not unique to EA, though I do agree we may be the largest EA-affiliated non-univeristy non-foundation research organization, as my guess is we are a little larger than GiveWell by FTE headcount. Though adding all those caveats ends up with me not saying very much, kinda like talking about being the largest private Catholic university in Vermont.

I think university affiliations definitely matter, especially for getting your work in front of policymakers. My guess is that research organizations choose to affiliate with a university when they can for this reason, and it’s a good one.

But I also like not having to worry about the bureaucracy that comes with interfacing with a university and I think this has historically allowed RP to be more agile and grow faster. I think it’s important that EA have both university and non-university research organizations.

(Obviously everyone would love to be attached to a multi-billion dollar foundation and if we can get more of those we obviously should, but I assume that’s not really an option.)

How do you decide on which research areas to focus on and, relatedly, how do you decide how to allocate money to them?

We do broadly aim to maximize the cost-effectiveness of our research work and so we focus on allocating money to opportunities that we think are most cost-effective on the margin.

Given that, it may be surprising that we work in multiple cause areas, but we face some interesting constraints and considerations:

• There is significant uncertainty about which priority area is most impactful. The general approach to RP has been that we can scale up multiple high-quality research teams in a variety of cause areas easier than we can figure out which cause area we ought to prioritize. Though we recently hired a Worldview Investigations Team to work a lot more on the broader question of how to allocate an EA portfolio. We also are investing a lot more in our own impact assessment. Together we hope that these will give us more insights into how to allocate our work going forward.

• There may be diminishing returns to RP focusing on any one priority area.

• A large amount of resources are not fungible across these different areas. The marginal opportunity cost to taking restricted funding is pretty low as we cannot easily allocate these resources to other areas, even if we were convinced they were higher impact.

• Work on any single area might gain from our working on multiple areas as teams have much greater access to centralized resources, staff, funding, and productive oversight than what they would receive if the team existed independently and solely focused on that priority. Relationships within an area could poten be useful for work in another area.

• Working across different priorities allows the organization to build capacity, reputation, and relations, and maintain option value for the future.

Most organizations within EA are relatively small (<20). Why do you think that's the case and why is RP different?

I’m not exactly sure and I think you’d have to ask some other smaller organizations. My best guess is that scaling organizations is genuinely hard and risky, and I can understand other organizations may feel that they work best and are more comfortable with being small. I think RP has been different by:

• Working in multiple different cause areas lets us tap into multiple different funding sources, thus increasing the amount of money we would take. It also increased the amount of work we wanted to do and the amount of people we wanted to hire.

• By being 100% remote-first from the beginning, we had a much larger talent pool to tap into. I think we’ve also been more willing to take chances on more junior-level researchers which has also broadened our talent pool. This allowed us to hire more.

• I think just a general willingness and aspiration to be a big research organization and take on this risk, rather than intentionally go it slow.

How do you decide whether something belongs to the longtermism department (i.e., whether it'll affect the long-term future)?

We haven’t had to make too many fine-grained decisions, so it hasn’t been something that has come up enough to merit a clear decision procedure. I think the trickiest decision was what to do with research aimed at understanding and mitigating the negative effects of climate change. The main considerations were questions like “how do our stakeholders classify this work” and “what is the probability of this issue leading to human extinction within the century” and both of those considerations led to climate change work falling into our “global health and development” portfolio.

This year we’ve made an intentional decision to focus nearly all our longtermist work on AI due to our assessment of AI risk as both unusually large and urgent, even among other existential risks. We will revisit this decision in future years and to be clear this does not mean that we think other people shouldn’t work on non-AI x-risk or non-xrisk longtermism.

What do you focus on within civilizational resilience?

This year we’ve made an intentional decision to focus nearly all our longtermist work on AI due to our assessment of AI risk as both unusually large and urgent, even among other existential risks. We will revisit this decision in future years and to be clear this does not mean that we think other people shouldn’t work on non-AI x-risk or longtermism-work not oriented towards existential risk reduction. But that does mean we don’t have any current work on civilizational resilience right now.

That being said, we do have some work on this in the past:

• Linch did a decent amount of research and coordination work around exploring civilizational refuges but RP is no longer working on this project.

• Jam has previously done work on far-UVC, for example by contributing to "Air Safety to Combat Global Catastrophic Biorisks".

• We co-supported Luisa in writing "What is the likelihood that civilizational collapse would directly lead to human extinction (within decades)?" while she was a researcher at both Rethink Priorities and Forethought Foundation.

I’ve personally liked it. There have been several times when I’ve talked with my co-CEO Marcus about whether one of us should just become CEO and it’s never really made sense. We work well together and the co-CEO dynamic creates a great balance between our pros and cons as leaders – Marcus leads the organization to be more deliberate and careful at the cost of potentially going too slowly and I lead the organization to be more visionary at the cost of potentially being too chaotic.

Right now we split the organization very well where Marcus handles the portfolios pertaining to Global Health and Development, Animal Welfare, and Worldview Investigations… and I handle the portfolios pertaining to AI Governance and Strategy, Existential Security (AI-focused incubation), and Surveys and Data Analysis (currently also mostly AI policy focused right now though you may know us mainly from the EA Survey).

I’m unsure if I’d recommend it to other orgs. I think most times it wouldn’t make sense. But I think it does make sense when there are two co-founders with an equally natural claim and desire to claim the CEO mantle, when they balance each other well, and when there is some sort of clear split and division of responsibility.

• I like pop music, like Ariana Grande and Olivia Rodriguo, though Taylor Swift is the Greatest of All Time. I went to the Eras Tour and loved it.

• I have strong opinions about the multiple types of pizza.

• I'm nowhere near as good at coming up with takes and opinions off-the-cuff in verbal conversations as I am in writing. I'm 10x smarter when I have access to the internet.

I haven’t actually thought much about the academic journal system, though I’m interested in what David Reinstein (former RP staff member) has been doing with his Unjournal.

I think this is a particularly good piece by Peter, though I am crying reading it. https://www.pasteurscube.com/for-samantha-a-eulogy/ 

This is so very sweet - thank you!

I like both! I developed squigglepy because I liked squiggle so much but RP really wanted to make use of the Python ecosystem. So now we get the best of both worlds! I am a strong supporter of QURI, especially because Rethink Priorities now provides them with fiscal sponsorship.

[This is 75% a joke, as Peter developed squigglepy based on QURI's squiggle]

(2) what are RP's plans with the Special Projects Program?

The plan for RP Special Projects is to continue to fiscally sponsor our existing portfolio of organizations and see how that goes and continue to build capacity to support additional organizations sometime in the future. Current marginal Special Projects time is going into exploring more incubation work with our Existential Security department.

(1) where do you think forecasting has its best use-cases? where do you think forecasting doesn't help, or could hurt?

I'm actually surprisingly unsure about this, especially given how interested I am in forecasting. I think when it comes to actual institutional decision making it is pretty rare for forecasts to be used in very decision-relevant ways and a lot of the challenge comes from asking the right questions in advance rather than the actual skill of creating a good forecast. And a lot of the solutions proposed can be expensive, overengineered, and focus far too much on forecasting and not enough on the initial question writing. Michael Story gets into this well in "Why I generally don't recommend internal prediction markets or forecasting tournaments to organisations".

I think something like "Can Policymakers Trust Forecasters?" from the Institute for Progress takes a healthier view about how to use forecasting. Basically, you need to take some humility about what forecasting can accomplish but explicit quantification of your views is a good thing and it is also really good for society generally to grade experts on their accuracy rather than their ability to manipulate the media system.

Additionally, I do think that knowing about the world ahead seems generally valuable and forecasting still seems like one of the best ways to do that. For example, everything we know about existential risk essentially comes down to various kinds of forecasting.

Lastly, my guess is that a lot of the potential of forecasting for institutional decision making is still untapped and merits further meta-research and exploration.

Do you think that promoting alternative proteins is (by far) the most tractable way to make conventional animal agriculture obsolete?

Evidence for alternative proteins being the most tractable way to make conventional animal agriculture obsolete is fairly weak. For example, similar products (eg, plant-based milk, margarine) have not made their respective categories obsolete.

Instead, we do have and we will continue to need a multi-pronged approach to transitioning conventional animal agriculture to a more just and humane system.

~

Do you think increasing public funding and support for alternative proteins is the most pressing challenge facing the industry?

Alternative proteins is a varied landscape so I imagine that the bottlenecks will be pretty different depending on the particular product, company, and approach. Unfortunately I am not up to date on details with regard to the funding gaps in this area.

~

Do you think there is expert consensus on these questions?

Unfortunately there is not. There also just aren't that many experts in this area in the first place.

Rethink Priorities does collaborate with academic institutions, mainly through hiring academics as contractors to either do novel research and/or to review our work. My sense is that these academics we work with are not unusually likely to be EA-affiliated though and I wouldn’t say we use any EA-specific academic infrastructure.