Comments
Readings and notes on how to do high-impact research
This shortform contains some links and notes related to various aspects of how to do high-impact research, including how to:
- come up with important research questions
- pick which ones to pursue
- come up with a "theory of change" for your research
- assess your impact
- be and stay motivated and productive
- manage an organisation, staff, or mentees to help them with the above
I've also delivered a workshop on the same topics, the slides from which can be found here.
The document has less of an emphasis on object-level things to do with just doing research well (as opposed to doing impactful research), though that’s of course important too. On that, see also Effective Thesis's collection of Resources, Advice for New Researchers - A collaborative EA doc, Resources to learn how to do research, and various non-EA resources (some are linked to from those links).
Epistemic status
This began as a Google Doc of notes to self. It's still pretty close to that status - i.e., I don't explain why each thing is relevant, haven't spent a long time thinking about the ideal way to organise this, and expect this shortform omits many great readings and tips. But several people indicated finding the doc useful, so I'm now sharing it more widely.
I’ve done ~6 FTE months of academic research (producing one paper) and ~1 FTE year of longtermist research at EA orgs.
I do not have excellent, one-size-fits-all, easy-win answers to how to do high-impact research; I just have various scraps and ideas, and this shortform is merely intend to collect those.
This shortform expresses my personal views only (and is based on a doc created before I started either of my current jobs).
Readings - misc
* Asterisks indicate sources I haven’t yet properly read myself.
Posts tagged Research methods
Posts tagged Org strategy
Ingredients for creating disruptive research teams (Forum post)
Ingredients for building disruptive research teams (EAG talk by the author of the post)
Can we intentionally improve the world? Planners vs. Hayekians
Building collaborative research teams — Jess Whittlestone
https://www.charityentrepreneurship.com/research.html (and the pages linked to under “OUR RESEARCH PROCESS”)
What can someone do to become a stronger fit for future Open Philanthropy generalist RA openings?
Tips On Doing Impactful Research - Effective Thesis, 2020
Hard problem? Hack away at the edges.
Some of Nuño Sempere’s recent work
Advice from 80,000 Hours: How to do high impact research
Rethink Priorities 2020 Impact and 2021 Strategy - EA Forum
Center on Long-Term Risk: 2021 Plans & 2020 Review - EA Forum
EAG talk on Aggregating Knowledge in EA
Literature Review for Academic Outsiders - LessWrong
Scholarship & Learning tag - LessWrong *
Readings - Primarily relevant to generating and picking questions
How to generate research proposals - EA Forum
Advice from Charity Entrepreneurship: How to do research that matters
Transcript: Karolina Sarek: How to do research that matters - EA Forum
Research as a stochastic decision process — Jacob Steinhardt (see also Should marginal longtermist donations support fundamental or intervention research?)
Potential benefits & downsides of making and/or sharing a research agenda [upcoming post by me, link will be added later]
Should marginal longtermist donations support fundamental or intervention research?
A case for strategy research: what it is and why we need more of it
Why EAs researching mainstream topics can be useful
Research project planning templates/resources [shared]
Readings - primarily relevant to theories of change
Theory of Change in Research [slides] (see also the accompanying worksheet)
Do research organisations make theory of change diagrams? Should they?
https://longtermrisk.org/identifying-plausible-paths-to-impact/
Modeling the impact of safety agendas
Readings - primarily relevant to assessing impact
Rethink Priorities Impact Survey - EA Forum
Should surveys about the quality/impact of research outputs be more common?
Posts tagged Impact assessment
Readings - primarily relevant to managing an organisation, staff, or mentees
Collection of collections of resources relevant to (research) management, mentorship, training, etc.
Notes
I’d guess that the best approaches to the first four points - coming up with important research questions, picking among them, coming up with a ToC, and assessing impact - will differ considerably for different topics/areas. They might be hardest for longtermism, as in that cause area goals are far away and sometimes unclear, and we get limited feedback loops.
On point 6 especially (regarding managing others), but also 1-5, I find it useful to think about the following interrelated points:
- Should we be more like planners or Hayekians?
- Should we be more top-down or bottom-up?
- How much control/guidance vs free rein should researchers be given?
- Should we value cohesion or not?
- Steve Jobs analogy:
- A biography of Jobs suggested that, instead of making the product the market wants, he leaned towards making the product he had strong inside-view reasons to think the market will want (even if they don't know it yet)
- Analogously, should research answer the questions decision-makers know they want answered, or questions we expect they would value answers to but that they haven't yet even noticed exist or understood the significance of?
- The latter might have tend to have a lower probability of success (including because you might just overestimate the importance), but might tend to lead to larger successes when it does succeed (because it causes a more fundamental shift and/or was less likely to have been done soon by someone else anyway).
Again, this was originally written like notes to self - let me know if I should clarify anything.
I'm grateful to Aleksandr Berezhnoi and Edo Arad for making useful comments on the Doc version of this shortform, and to Kat Woods for encouraging me to make a public post out of the Doc.
Notes from a call with someone who's a research assistant to a great researcher
(See also Matthew van der Merwe's thoughts. I'm sharing this because I think it might be useful to some people by itself, and so I can link to it from parts of my sequence on Improving the EA-Aligned Research Pipeline.)
For the last few years, I’ve been an RA in the general domain of ~economics at a major research university, and I think that while a lot of what you’re saying makes sense, it’s important to note that the quality of one’s experience as an RA will always depend to a very significant extent on one’s supervising researcher. In fact, I think this dependency might be just about the only thing every RA role has in common. Your data points/testimonials reasonably represent what it’s like to RA for a good supervisor, but bad supervisors abound (at least/especially in academia), and RAing for a bad supervisor can be positively nightmarish. Furthermore, it’s harder than you’d think to screen for this in advance of taking an RA job. I feel particularly lucky to be working for a great supervisor, but/because I am quite familiar with how much the alternative sucks.
On a separate note, regarding your comment about people potentially specializing in RAing as a career, I don’t really think this would yield much in the way of productivity gains relative to the current state of affairs in academia (where postdocs often already fill the role that I think you envision for career RAs). I do, however, think that it makes a lot of sense for some RAs to go into careers in research management. Though most RAs probably lack the requisite management aptitude, the ones who can effectively manage people, I think, can substantially increase the productivity of mid-to-large academic labs/research groups by working in management roles (I know J-PAL has employed former RAs in this capacity). A lot of academic research is severely management-constrained, in large part because management duties are often foisted upon PIs (and no one goes into academia because they want to be a manager, nor do PIs typically receive any management training, so the people responsible for management often enough lack both relevant interest and relevant skill). Moreover, productivity losses to bad management often go unrecognized because how well their research group is being managed is, like, literally at the very bottom of most PIs’ lists of things to think about (not just because they’re not interested in it, also because they’re often very busy and have many different things competing for their attention). Finally, one consequence of this is that bad RAs (at least in the social sciences) can unproductively consume a research group’s resources for extended periods of time without anyone taking much notice. On the other hand, even if the group tries to avoid this by employing a more active management approach, in that case a bad RA can meaningfully impede the group’s productivity by requiring more of their supervisor’s time to manage them than they save through their work. My sense is that fear of this situation pervades RA hiring processes in many corners of academia.
Thanks, I think this provides a useful counterpoint/nuance that I think should help people make informed decisions about whether to try to get RA roles, how to choose which roles to aim for/accept, and whether and how to facilitate/encourage other people to offer or seek RA roles.
Your second paragraph is also interesting. I hadn't previously thought about how there may be overlap between the skills/mindsets that are useful for RAs and those useful for research management, and that seems like an useful point to raise.
Minor point: That point was from the RA I spoke to, not from me. (But I do endorse the idea that such specialisation might be a good thing.)
More substantive point: It's worth noting is that, while a lot of the research and research training I particularly care about happens in traditional academia, a lot also happens in EA parts of academia (e.g., FHI, GPI), in EA orgs, in think tanks, among independent researchers, and maybe elsewhere. So even if this specialisation wouldn't yield much productivity gains compared to the current state of affairs in one of those "sectors", it could perhaps do so in others. (I don't know if it actually would, though - I haven't looked into it enough, and am just making the relatively weak claim that it might.)
Yeah, I think it’s very plausible that career RAs could yield meaningful productivity gains in organizations that differ structurally from “traditional” academic research groups, including, importantly, many EA research institutions. I think this depends a lot on the kinds of research that these organizations are conducting (in particular, the methods being employed and the intended audiences of published work), how the senior researchers’ jobs are designed, what the talent pipeline looks like, etc., but it’s certainly at least plausible that this could be the case.
On the parallels/overlap between what makes for a good RA and what makes for a good research manager, my view is actually probably weaker than I may have suggested in my initial comment. The reason why RAs are sometimes promoted into research management positions, as I understand it, is that effective research management is believed to require an understanding of what the research process, workflow, etc. look like in the relevant discipline and academic setting, and RAs are typically the only people without PhDs who have that context-specific understanding. Plus, they’ll also have relevant domain knowledge about the substance of the research, which is quite useful in a research manager, too. I think these are pretty much all of the reasons why RAs may make for good research managers. I don’t really think it’s a matter of skills or of mindset anywhere near as much as it’s about knowledge (both tacit and not). In fact, I think one difficulty with promoting RAs to research management roles is that often, being a successful RA seems to select for traits associated with not having good management skills (e.g., being happy spending one’s days reading academic papers alone with very limited opportunities for interpersonal contact). This is why I limited my original comment on this to RAs who can effectively manage people, who, as I suggested, I think are probably a small minority. Because good research managers are so rare, though, and because research is so management-constrained without them, if someone is such an RA and they have the opportunity, I would think that moving into research management could be quite an impactful path for them.
Ah, thanks for that clarification! Your comments here continue to be interesting food for thought :)
One idea that comes to mind is to set up an organization that hires RAs-as-a-service. Say, a nonprofit that works with multiple EA orgs and employees several RAs, some full-time and others part-time (think, a student job). This org can then handle recruiting, basic training, employment and some of the management. RAs could work on multiple projects with perhaps multiple different people, and tasks could be delegated to the organization as a whole to find the right RA to fit.
A financial model could be something like EA orgs pay 25-50% of the relevant salaries for projects they recruit RAs for, and the rest is complemented by donations to the non-profit itself.
Yeah, I definitely think this is worth someone spending at least a couple hours seriously thinking about doing, including maybe sending out a survey to or conducting interviews with non-junior researchers[1] to gauge interest in having an RA if it was arranged via this service.
I previously suggested a somewhat similar idea as a project to improve the long-term future:
And Daniel Eth replied there:
I'm going to now flag this idea to someone who I think might be able to actually make it happen.
Someone pointed out to me that BERI already do some amount of this. E.g., they recently hired for or are hiring for RAs for Anders Sandberg and Nick Bostrom at FHI.
It seems plausible that they're doing all the stuff that's worth doing, but also seems plausible (probable?) that there's room for more, or for trying out different models. I think anyone interested in potentially actually starting an initiative like this should probably touch base with BERI before investing lots of time into it.
Ah, right! There still might be a need outside of longtermist research, but I definitely agree that it'd be very useful to reach out to them to learn more.
For further context for people who might potentially go ahead with this, BERI is a nonprofit that supports researchers working on existential risk. I guess that Sawyer is the person to reach out to.
Btw, the other person I suggested this idea to today is apparently already considering doing this. So if someone else is interested, maybe contact both Sawyer and me, and I can put you in touch with this person.
And this person would do it for longtermist researchers, so yeah, it seems plausible/likely to me that there's more room for this for researchers focused on other cause area.
These feel like they should be obvious points and yet I hadn't thought about them before. So this was also an update for me! I've been considering PhDs, and your stated downsides don't seem like big downsides for me personally, so it could be relevant to me too.
Ok, so the imagine you/we (the EA community) successfully make the case and encourage demand for RA positions. Is there supply?
I actually think full-time RA roles are very commonly (probably more often than not?) publicly advertised. Some fields even have centralized job boards that aggregate RA roles across the discipline, and on top of that, there are a growing number of formalized predoctoral RA programs at major research universities in the U.S. I am actually currently working as an RA in an academic research group that has had roles posted on the 80,000 Hours job board. While I think it is common for students to approach professors in their academic program and request RA work, my sense is that non-students seeking full-time RA positions very rarely have success cold-emailing professors and asking if they need any help. Most professors do not have both ongoing need for an (additional) RA and the funding to hire one (whereas in the case of their own students, universities often have special funding set aside for students’ research training, and professors face an expectation that they help interested students to develop as researchers).
Separately, regarding the second bullet point, I think it is extremely common for even full-time RAs to only periodically be meaningfully useful and to spend the rest of their time working on relatively low-priority “back burner” projects. In general, my sense is that work for academic RAs often comes in waves; some weeks, your PI will hand you loads of things to do, and you’ll be working late, but some weeks, there will be very little for you to do at all. In many cases, I think RAs are hired at least to some extent for the value of having them effectively on call.
In regards to the third bullet point, there might be a nontrivial boost to the senior researchers' productivity and well-being.
Doing grunt-work can be disproportionally (to its time) tiring and demotivating, and most people have some type of work that they dislike or just not good at which could perhaps be delegated. Additionally, having a (strong and motivated) RA might just be more fun and help with making personal research projects more social and meaningful.
Regarding the salary, I've quickly checked GiveWell's salaries at Glassdoor
So from that I'd guess that an RA could cost about 60% as much as a senior researcher. (I'm sure that there is better and more relevant information out there)
I think you're asking "...encourage that people seek RA positions. Would there be enough demand for those aspiring RAs?"? Is that right? (I ask because I think I'm more used to thinking of demand for a type of worker, and supply of candidates for those positions.)
I don't have confident answers to those questions, but here are some quick, tentative thoughts:
But again, these are quick, tentative thoughts. I've neither worked as nor had an RA, haven't been closely involved with any RA hiring decisions, haven't done research into how RAing works and what value it provides in non-EA academia, etc.
See also 80k on the career idea "Be research manager or a PA for someone doing really valuable work".