AMA: Owen Cotton-Barratt, RSP Director

For RSP, I think that:

• In starting RSP, I had an implicit theory of change in my head
  • There are quite a few facets of this (mechanisms for value produced, a continuum of hypotheses, etc.)
  • One important facet (particularly for early-RSP) was a sense of "pretty sure there's significant value available via something in this vicinity, let's try it and see if we can hone in"
• I explicitly share and communicate parts of this model to the extent that it's accessible for me to do so
  • This involved some conversations with people before RSP started, and some presenting thoughts to the research scholars as the programme started, and periodically returning to it
• As RSP has developed and other people have become major stakeholders, they've developed their own implicit theories of change
  • We make some space to discuss these / exchange models
• As RSP matures, it will make more sense to pin down a theory of change and have it explicit and shared
  • The facet of "let's work out what here is good" will naturally diminish, and we'll work out which other facets are best to lean on

Some general thoughts:

• Advantages of having an explicit theory of change:
  • Makes it easier to sync up about direction/priorities/reasons for doing things
  • Makes it easier for people to engage critically, or otherwise to notice mistakes and course-correct
• Disadvantages of having an explicit theory of change:
  • Easy to have the case where your best expression of something is dumber than your real internal sense of it
    • In this case it may be preferable to be guided by the internal sense rather than the explicit version
    • (this is at least some distant relative of Goodhart's law)
  • To the extent that you're going to be guided by your internal sense rather than an explicit version, sharing something as an explicit theory of change can be misleading
• In general I think it's good to encourage lots of explicit discussion about theories of change
  • Ideally without committing to reaching an "answer", but having that as a goal may be helpful for prompting the discussion

I think that I find the disadvantages quite emotionally resonant, which may pull me to err too far in the direction of not being explicit. I have appreciated some cases where people have pushed me towards "let's have a discussion where we're pretty explicit about best guesses".

FHI I think has an explicit theory of change even less than RSP does; my guess it that Nick Bostrom is also averse to incurring the costs of these disadvantages (and maybe more strongly so than me), but that's speculation.

A related question: which fraction of your and RSP's impact do you expect to come from direct and from community/field-building?

E.g.

• When working on a paper, do you think value consists of field-building or from a small personal chance of, say, coming up with a crucial consideration?
• Will most value of RSP will come from direct work done by scholars or by scholars [and program] indirectly influencing other people/organizations? [I would count consulting policy-makers as direct work.]

1. Does all of those claims seem true to you?

Yes, with some important comments:

• I don't think this is centrally about "researchers", but about "people-who-are-decent-at-working-out-what-to-do-amongst-the-innumerable-possibilities"
  • This is a class we need more of in EA (and particularly longtermist EA); research is one of the applications of the (major) applications of such people, but far from the only one
• Mentorship/management is more like a thousand small things than two big things
  • Often people will be better off learning from multiple strong mentors than one, because they'll be good at different subcomponents
• There are very substantial reasons beyond this to spend part of one's (research) career outside of explicitly EA orgs, particularly if you get an opportunity to work with outstanding people
  • Such as:
    • You can better learn the specialist knowledge belonging to the relevant domain by spending time working with top experts
      • Or idiosyncratic-but-excellent pieces of mentorship
    • To the extent that EA has important insights that are relevant in many domains, working closely with smart people is a good opportunity to share those insights
    • It's a powerful way to develop a network
  • I gave the reasons above in terms of "how this seems locally good", but it might be more natural to think about it globally, and notice that a version of EA which is very insular and just builds up its own stuff kind-of cut off from the rest of intellectual endeavour seems way worse (in expectation) than a version which has lots of surface area and good interfaces

2. If so, do you expect this to remain true for a long time, or do you think we're already moving rapidly towards fixing it?

Hmm, I think that I'm less conceiving of this as a problem-to-be-fixed than you are. Partially it's because I do see these substantial benefits of spending part of one's career outside of explicitly EA orgs -- I don't think it's important that everyone does this (and it doesn't have to be at the start of their career), but important that there's at least a solid fraction of people who have done so.

That said, I do think it's somewhat a problem, and there are people (whether or not they've already spent part of their career outside of explicitly EA orgs) who would be in a good position to contribute directly to EA work if only they had the right mentorship. I think maybe we're on the way to having the most acute versions of it fixed (though I'm not that confident about that), but I think the basic dynamic will remain true for a long time.

4. Do you think RSP, or things like it, are especially good ways to address this problem (if it exists)?

I think things like RSP are a good way to address a facet of this problem, of getting people towards "people-who-are-decent-at-working-out-what-to-do-amongst-the-innumerable-possibilities". I think that this can be significantly complementary to people spending part of their career outside of EA orgs.

(I think this last paragraph in particular may not be very clear. Feel free to poke at what doesn't make sense.)

Why not just have the people who need mentorship serve as "research personal assistants" to improve the productivity of people who are qualified to provide mentorship? (This describes something which occurs between professors and graduate students right?)

Interesting question!

Related: Suppose that, in 10 years, the RSP seems to have had no impact.* What signs would reveal this? And what seem the most likely explanations for the lack of impact?

*There already seem to be some indicators of impact, so feel free to interpret this as "seems to have had no impact after 2020", or as "seems to have had no impact after 2020, plus the apparent impacts by 2020 all ended up washing out over time".

Something like: it catalysed the creation of a whole stream of major new projects (led by scholars who used the space afforded by the programme to think seriously about possibilities, and who are well-networked with the broader x-risk ecosystem which makes coordination and recruitment easier).

We're doing a combination of:

• Looking at what people go on from RSP to do
  • For now this is just "where they're going next" for people leaving, but in the future I expect us to check back in a few years later
• Surveys (& conversations) asking research scholars how useful they have found RSP (and in what that value consists), and what they guess they would have done otherwise
• Comparison of the above with some people who narrowly didn't join RSP (for one reason or another)
• Looking at to what extent work done by research scholars while on the programme is directly useful
• Our (=RSP management's) independent impressions of whether / how much we've helped people

(I think we're still finding out feet with this.)

There are a relatively small number of individuals who have gone through the programme, and it's important to us to protect their privacy, so at the moment we don't have plans to publish any of this. When we have slightly more data I kind of like the idea of publishing some aggregate summaries, but I haven't thought seriously about whether this will be possible to do in a way which is properly privacy-preserving while also actually useful to readers.

That personal dietary choices are important on consequentialist effectiveness grounds. 

I actually think there are lots of legitimate and powerful reasons for EAs to consider veg*nism, such as:

• A ~deontological belief that it's wrong to eat animals
• A desire for lifestyle choices that help you connect with what you care about
• Signalling caring
• A desire for shared culture with people you share values with

... but it feels to me almost intellectually dishonest to have it be part of an answer to someone saying "OK, I'm bought into the idea that I should really go after what's important, what do I do now?"

(I'm not vegetarian, although I do try to only consume animals I think have had reasonable welfare levels, for reasons in the vicinity of the first two listed above. I still have some visceral unease about the idea of becoming vegetarian that is like "but this might be mistaken for being taken in by intellectually dishonest arguments".)

Estimating the value of research seems really hard to me (and this is significantly true even in retrospect).

That said, some candidates are:

• Work making the point that we should give outsized attention to mitigating risks that might manifest unexpectedly soon, since we're the only ones who can
  • At the time it didn't seem unusually valuable, but I think it was relatively soon after (a few months) that I saw some people changing behaviour in light of the point, which increased my sense of its importance
• Work on cost-effectiveness of research of unknown difficulty, particularly the principle of using log returns when you don't know where to start
  • Felt sort-of important at the time, although I think the kind of value I anticipated hasn't really manifested
  • I have felt like it's been useful for my thinking in a variety of domains, thinking about pragmatic prioritisation (and I've seen some others get some value from that); however logarithm is an obvious-enough functional form that maybe it didn't really add much
• Maybe something where it was more about dissemination of ideas than finding deep novel insights (I think it's very hard to draw a line between what counts as "research" or what doesn't), such as Prospecting for Gold, or How valuable is movement growth?
  • Quite a few people have told me that they got something out of one or both of those pieces, although it's extremely hard to assess the counterfactuals
  • I felt like I was doing something significant in these cases (particularly when writing the talk Prospecting for Gold)
• Overall I'd be hard pressed to decide between choosing one of the above, although I'd tend to guess these are more valuable than most other pieces I've done (excepting some recent work that I don't yet want to judge, and with the caveat that I'm surely forgetting some)
  • That said, some of the more policy-ish pieces of research might still turn out to be the most valuable, if they got picked up somewhere important, but so far I'll not count them

There's a class of things which feel majorly helpful, but it's hard to distinguish between whether I was helped by the background in pure mathematics, or whether I have some characteristics which both helped me in mathematics and help me now (I suspect it's some of both):

• Being good at framing things
  • Turning things over in my head, looking for the angle which makes them most parsimonious, and easiest to comprehend clearly
  • Relatedly, feeling happy to dive in and try to make up theory, but keep it grounded by "this has to actually explain the things we want to know about"
  • These are useful skills when faced with domains where we haven't yet settled on paradigms which we're satisfied capture the important parts of what we care about
• Generally keeping track of precisely what are the epistemic statuses of different claims, and how they interact
  • This is a useful skill for domains where we're projecting out beyond things we can easily check empirically

Then there are some cases where I was more directly applying some mathematical thinking, e.g.:

• Work on normative uncertainty (chiefly: variance normalization; bargaining)
• Theory of logarithmic returns

I'm not sure I really believe that "patient vs impatient longtermists" cleaves the world at its joins. I'll use the terms to mean something like resources aimed at reducing existential risk over the next fifty years or so, versus aiming to be helpful on a timescale of over a century?

In either case I think it depends a lot on the resource in question. Many resources (e.g. people's labour) are not fully fungible with one another, so it can depend quite a bit on comparative advantage.

If we're talking about financial resources, these are fairly fungible. There I tend to think (still applies to both "patient" and "impatient" flavours of longtermism):

• It doesn't make so much sense to analyse at the level of the individual donor
• Instead we should think about the portfolio we want longtermist capital as a whole to be spread across, and what are good ways to contribute to that portfolio at the margin
  • Sometimes particular donors will have comparative advantage in giving to certain places (e.g. they have high visibility on a giving opportunity so it's less overhead for them to assess it, and it makes sense for them to fill it)
  • Sometimes it's more about coordinating to have roughly the right amount total spent, and not fritter away too much on donor-of-last-resort type games
• Some particular opportunities around now look excellent, but the scale of them is such that it's hard to absorb a large fraction of longtermist capital over (say) the next five years, so it makes sense for some money to be held in traditional investments

Then more specialised statements:

• From a "patient" perspective, we want to invest in anything which grows the pool of informed longtermist resources faster than traditional investment, e.g. some versions of:
  • value-spreading
    • this is a lazy catch-all term which includes a ton of very different looking activities; I think getting better resolution on strategies that are worth employing here is pretty important
  • producing better intellectual material about what longtermists should do (includes research + dissemination, and works by increasing the degree people are informed plus by making the set of ideas seem more legit+solid, so easier to attract further people)
  • (specialised) education
  • research into what type of activities have good long-term returns for longtermists
• From an "inpatient" perspective, we want to invest in opportunities which are plausibly on critical paths to averting existential catastrophes, e.g. some versions of:
  • lots of the things mentioned for the "patient" perspective 
    • (there's a continuum here, and if you were so impatient you just wanted to work on averting existential risk that would manifest in the next five years, the patient strategies don't look so great)
  • research into understanding and characterising the nature and likelihood of imminent risks
    • + work which sets this out clearly, and can be usefully understood by many people (I think that broad understanding of risks is a big step towards reducing them)
  • work to develop careers of people who might be well-placed to do relevant work
    • analogously, work to develop institutions that could play a useful role

Overall, I don't think it makes sense to imagine that one of the "patient" or "impatient" perspectives is correct. I think that the correct longtermist portfolio certainly includes substantial amounts of both of these classes of investments. For financial resources, I think that over the next several years it's likely that the meaningful margins are all about which given opportunities rise above the bar of saving (rather than which class deserves more money).

For non-financial resources (e.g. the career of a specified individual) it's more plausible to ask about tradeoffs between patient and impatient perspectives. I think it may usually be better if decisions here come down to comparative advantage rather than high-level views of the tradeoffs between "patient" and "impatient".

If I pinned myself down and forced myself to name one bullet point above that I think I'd like to see slightly more of (at the expense of the other bullet points), then at-least-in-moment-writing-this, I'd say "research into what type of activities have good long-term returns for longtermists". But I think this is correctly quite a small slice of our portfolio: I just want it to be a slightly-less-small slice. (I'd have similar views about some particular subfields of various of the other activity-types.)

That in thinking about community/movement building, it's more important to consider something like how people should be -- e.g. what virtues should be cultivated/celebrated -- rather than just what people should do (although of course both matter). 

(That's in impression space. I have various drafts related to this, and I hope to get something public up in the next few months, so I'll leave it brief for now.)

I think the debate about ambiguity aversion mostly comes down to a bucket error about the meaning of "rational":

• I think that a fully rational actor would:
  • not exhibit ambiguity aversion
  • commit to a single joint probability distribution
• I think for boundedly rational actors:
  • ambiguity aversion is a (very) useful heuristic
    • particularly if you're in an environment which is or might be partially designed by other agents who could stand to benefit from your loss
  • it can make sense to hold onto ranges of probabilities
    • e.g. maybe you think event X has probability between 10% and 20%, then that's enough to determine what to do for lots of policy decisions; in cases where it doesn't determine what to do you can consider whether it's worth time investment to sharpen your probability estimate
• I think it's a bad (but frequently made, at least implicitly) assumption that boundedly rational actors should mimic the behaviour of fully rational actors in cases where they can work out what that is
  • For a particularly vivid example of (something at least strongly analogous to) this assumption breaking, see the theorem in the optimal taxation literature that the top marginal tax rate should be zero

One that comes to mind:

Theorem: Every finitely presented group is the fundamental group of some compact 4-manifold.

I like it because:

• It's a universal claim relating two very broad classes of objects, such that when I look at the statement I think "wow, how would you even start thinking about how to prove that?"
• There's a proof which is geometric, elegant, and short 
  • In fact there are multiple quite different geometric proofs!

[With apologies for the fact that this likely makes no sense to most readers.]

FWIW I agree with Owen. I agree the direction of effect supplies a pro tanto consideration which will typically lean in favour of other options, but it is not decisive (in addition to the scenarios he notes, some people have pursued higher degrees concurrently with RSP).

So I don't think you need to worry about potentially leading folks astray by suggesting this as an option for them to consider - although, naturally, they should carefully weigh their options up (including considerations around which sorts of career capital are most valuable for their longer term career planning).

I don't feel like I'm at all an expert in biosecurity careers, but I agree with that directionally they seem more credentialist.

I think this is a consideration against RSP, although it doesn't feel like an overwhelming one, since:

• It could be a reasonable option before a PhD
  • This is particularly relevant if taking the time to think about what you want to work on allows you to do a PhD in which your work is much closer to things you eventually care about
  • (similarly it could be a good option for some people after a PhD)
• There may well be some roles (now or in the future) which are less credential-locked

Good question.

Of the two options I'd be tempted to say it's more of a priority to spread the underlying arguments, but actually I think something more nuanced: it's a priority to keep engaging with people about the underlying arguments, finding where there seems to be the greatest discomfort and turning a critical eye on the arguments there, looking to see if we can develop stronger versions of them.

I think that talking about the tentative conclusions along with this is  important both for growing the network of people sympathetic to those, and for providing concrete instantiation of what is meant by the underlying philosophy (too much risk of talking past each other or getting lost in abstraction-land without this)

1. Did you make an active decision to shift your priorities somewhat from doing to facilitating research? If so, what factors drove that decision?

There was something of an active decision here. It was partly based on a sense that the returns had been good when I'd previously invested attention in mentoring junior researchers, and partly on a sense that there was a significant bottleneck here for the research community.

2. What do you think makes running RSP your comparative advantage (assuming you think that)? 

Overall I'm not sure what my comparative advantage is! (At least in the long term.) 

I think:

• Some things which makes me good at research mentoring are:
  • being able to get up to speed on different projects quickly
  • holding onto a sense of why we're doing things, and connecting to larger purposes
  •  finding that I'm often effective in 'reactive' mode rather than 'proactive' mode 
    • (e.g. I suspect this AMA has the highest ratio of public-written-words / time-invested of anything substantive I've ever done)
  • being able to also connect to where the researcher in front of me is, and what their challenges are
• There are definitely parts of running RSP which seem not my comparative advantage (and I'm fortunate enough to have excellent support from project managers who have taken ownership of a lot of the programme)

3. Any thoughts on how to test or build one's skills for that sort of role/pathway?

• Read a lot of research. Form views (and maybe talk to others) about which pieces are actually valuable, and how. Try to work out what seems bad even about good pieces, or what seems good even about bad pieces.
• Be generous with your time looking to help others with their projects. Check in with them afterwards to see if they found it useful. (Try to ask in a way which makes it safe for them to express that they did not.)
• Try your own hand at research. First-hand experience of challenges is helpful for this.

(I've focused on the pathway of "research mentorship"; I think there are other parts you were asking about which I've ignored.)

Malevolence seems potentially important to me, although I mostly haven't been thinking about it (except a bit about psychopathy and its absence). Things more like game-theoretic dynamics are where a good portion of my attention has been ... but I don't want to claim this means they're more important.

[meta: this is a short answer because while I might have things to say about crisper questions within this space, for saying things-in-general I think it makes more sense to wait until I have coherent enough ideas to publish something.]

I guess I think that "decision-making under deep uncertainty" is mostly too broad a category to be able to say useful things about (although maybe we can draw together useful lessons that seem to hold in a variety of more specialised contexts), and we're better trying to look at more particular setups and reason about those.

This is an interesting question, but I don't think there's a decent short-answer version; it's more like investing several hours or not at all.

So I'll take this as a prompt to consider the several-hour version, but won't answer for now.

Gee, this is really hard to measure.

I'd guess that somewhere between 10% and 30% is done as part of something that we'd naturally call the "standard academic process" ?

I think that there are some good reasons for deviation, and some things that academic norms provide that we may be missing out on.

I think academia is significantly set up as a competitive process, where part of the game is to polish your idea and present it in the best light. This means:

• It encourages you to care about getting credit, and people are discouraged from freely-sharing early stage ideas that they might turn into papers, for fear of being scooped
  • This seems broadly bad
• It encourages people to put in the time to properly investigate the ins and outs of an idea, and find the clearest framing of it, making it more efficient for later readers
  • This seems broadly good

I'd like it if we could work out how to get more of the good here with less of the bad. That could mean doing a larger proportion of things within some version of the academic process, or could mean working out other ways to get the benefits.

There's also a credentialing benefit to doing things within the academic process. I think this is non-negligible, but also that if you do really high-quality work anywhere, people will observe this and come, so I don't think it's necessary to rest on that credentialing.

I'm just using "deep uncertainty" to refer to a theme of situations where there are challenges about how you get going. I'm not thinking of it as a crisp referent. 

I guess that complex cluelessness would be a subclass of cases of deep uncertainty in my ontology, but I also mean to include e.g. normative uncertainty; Knightian uncertainty; heuristics for estimating probabilities when you don't really know where to start.

(I'm amused at the distribution of votes on this question).

Generally Oxford lectures are open to any university members, although:

• They wouldn't generally get "academic credit" for this
• They wouldn't necessarily be able to join accompanying classes (although we might be able to arrange this)
• I've no idea what the situation is now so many things are remote because of COVID-19