Summary

• Effective Thesis is a project helping steer students’ research towards directions that are significantly more impactful than their previous plans.
• So far, Effective Thesis has helped steer students towards and find specific topics in directions such as animal advocacy, global priorities research, pandemic modelling, AI governance, AI safety, wild animal welfare, clean meat, food security and forecasting.
• So far, the project mainly attracts people from top 100 most prestigious universities, masters and undergraduates, across many countries and many study disciplines
• There might be a couple of ways to increase the impact of the project further, most notably focusing on non-EAs; focusing on talent pipelines in non-English speaking countries; and continuing to focus on a wide breadth of disciplines
• It is unclear whether helping students to find a specific research topic is the most useful service for them or whether there is something even more useful we can help them with within the process of forming the focus of their research careers
• Several other alternative interventions which can achieve the same goal as Effective Thesis are discussed

How you can help

• We seek for collaborators and funding to further develop project operations. If you’d like to join us or know of anyone who might be interested in either joining or funding us, please reach out at [email protected]. For more information see the Forthcoming plans section.

What is Effective Thesis?

Effective Thesis is a project that aims to assist students in their final thesis topic choice and direct their attention to areas that have the potential to greatly improve the world. It does this by offering students to engage with profiles of several high impact research directions for each discipline, offering to connect them with researchers working in these directions who can help with the specific topic choice, giving feedback on their existing ideas and research career tips, and inviting them to the online community of other students focusing on similar directions.

Here are the previous reports from 2017 when the project began and 2018 impact report with a description of project's history.

The hope is that this could be potentially high-leverage as we are focusing on a particularly crucial juncture in people’s trajectories. More generally, within the broader goal of generating new impactful research, this project focuses more on the junior side of research career trajectories, trying to create new junior researchers who are not necessarily dependent on EA funding. For other approaches see below.

My thinking about the project

In this section, I’ll describe in more detail some models and categorisations I use when thinking about this project.

Useful general model

I think that the situation of choosing a high impact research topic is in line with the “landscape model” described here. That is, we can imagine the space of potential research opportunities as a landscape with peaks of various heights symbolising how impactful various opportunities are. Our goal is then to locate the highest peaks in this landscape. For this image to work, there are some assumptions which I believe are met: there is a large space of opportunities (all possible specific research topics), these opportunities vary a lot by some metric (e.g. impact), it is hard to find the most impactful opportunities from the top-down perspective, and “local knowledge” carried by more experienced and senior people in various academic disciplines (localities) plays an important role in finding the most impactful topics.

The operating model of Effective Thesis is quite similar to The Good Technology Project, particularly in that Effective Thesis tries to advise individuals who are about to start their projects (dissertations) in a somewhat top-down manner (i.e. advising not from a position of experts in a specific field, but rather from a position of high-level prioritisation). However, there are some notable and important differences.

First, the ambition of the Effective Thesis is not to come up with specific research topics/opportunities for impact ourselves, but rather outsource this task to people who have already built some “local knowledge”, e.g. PhDs who have been thinking about problems in broader high impact cause areas a lot. I believe that mid- to late-PhDs are often able to spot very good topics/opportunities for impact within their disciplines and communicate them to younger students.

Second, since Effective Thesis targets students prior to choosing their PhD topic, I believe most of them haven’t built such a huge body of expertise within a specific topic/subfield to make them inflexible to transfer to some other.

How strong the institutional and incentive pressures are on students to do something mainstream, something their supervisors have expertise in or something that falls within the faculty requirements is unclear to me. However, again, I assume these pressures are weaker than the pressures to make a profit in the start-up scene.

Other interventions with the same goal

In this section, I will try to set Effective Thesis into the context of other interventions one can use to (support) generating more impactful research and give some examples of such projects in the EA ecosystem. This is just a convenience sample, not meant to be comprehensive. If you know about some useful classification in this space, I’d be very happy to hear about it. I might create a separate post from this section once I’ll deepen my thinking about it a bit more.

I would say the goal of Effective Thesis is “influencing which research is generated” (in comparison with e.g. “improving science as a whole”). If this is the goal, which other interventions could we use to achieve it?

1) influencing individuals by giving them information on what the potentially most impactful directions are and motivating them to pursue these directions

This is what the Effective Thesis does. The limitations are that it doesn’t help people to actually carry out the project either by giving them supervision/feedback or money to enable them to work on it. Anyway, most researchers are government-funded and they have pretty large freedom in which topic they decide to focus on.

2) providing funding for research directions that seem promising

This is what private foundations and grant agencies (like OpenPhil) do. This helps researchers already focusing on promising directions to generate more research. The limitation might be having the pool of researchers already focusing on promising directions.

3) setting up research organisations producing research in a specific direction

This is what FHI, GPI, Rethink Priorities, Faunalytics,... do. The limitation of this is that it is not well-scalable. However, having a couple of people working on questions of interest full time with a lot of feedback seems like the safest strategy to produce good quality research on a given topic.

4) organising research workshops

This is what the AI safety camp and AI safety research programme do (and outside EA e.g. Junior Research Programme). The goal of these workshops could be to provide participants with research experience (to better assess their fit with a given research mode), motivation, education in a specific research direction, mentoring and networking. Workshops don't have to be one-off events, there could be some follow up meeting in a couple months and remote teamwork in the meantime.

5) setting up prestigious prizes/awards

This is my understanding of what the Forethought undergraduate thesis prize does.

6) providing mentorship and space for exploration

This is what the Research Scholars Programme (RSP) does. RSP provides space for exploration and lets researchers work on research projects they probably couldn't work on elsewhere. Researchers in RSP are supported in their exploration in various ways, including research workshops being organised for them to let them familiarize with a range of research directions, having a community of other researchers around them and getting mentorship. As far as I understand, they are not expected to stay in FHI as researchers after completing this programme (though some do stay).

Approaches that don't aim to influence which research is generated

7) coordination - e.g. connecting students/researchers interested in the same topics

Organising conferences for a given research direction might be one example.
Effective Thesis does a little bit of this as well by connecting students interested in a similar direction via an online community.

There are also some other ways to generate more impactful research that I didn’t mention - some more conventional and smaller-scale like becoming a researcher yourself, taking a research management role in a group focusing on promising directions; and broader interventions like changing the incentives by influencing standards for evaluating science, improving research methods and tools or cognitive enhancement (see e.g. Bostrom’s paper). I’ll be happy to hear about other types of intervention that I missed in the comments.

Impact produced

Generally, feedback loops are pretty long in this project (up to 2-3 years to get information on how Effective Thesis intervention influenced students’ real career choices), so when tracking the impact Effective Thesis made in 2019, it is worth keeping in mind that these data are only preliminary and that many students haven’t finished their theses yet and haven’t yet made a subsequent career decision (most of them will likely do so by autumn 2020).

Best case studies

Based on my previous analysis and investigation of current data, I have created an assumption that most of the impact will come from a small number of cases. Summaries of the best case stories should thus be a good proxy for impact. I have chosen 3 anonymized examples of cases in which I think Effective Thesis had a significant impact. There are several more I would mention if I wanted to describe the full impact of the project, so please consider these only as an illustration.

Selected students who applied in 2018

Student 1

At the time of application, he studied PhD in environmental science, after finishing his masters with very good grades and having already published 2 papers. Effective Thesis suggested a couple of new directions that he hadn’t considered before, and he ended up collaborating with ALLFED being paid by the university he studied at for a subsequent postdoc position. He estimates that without Effective Thesis there would have been about (very roughly) a 30 % chance that he would have ended up doing similarly impactful work.

He said: “Before I contacted Effective Thesis I was quite unsure what to do as a next step. After reading 80k and other EA material I had realized that my PhD was not doing really important work and I wanted to change that. Right now I am mostly done with my PhD and started working with ALLFED a few months ago. As I probably have funding from my university for the next two years, I plan to use this funding to mainly work for ALLFED. Working with ALLFED was greatly influenced by effective thesis, as it allowed me to contact them in the first place. I might have come across ALLFED without effective thesis, but I am relatively sure that I would not have contacted them.”

Student 2

He applied for his master's thesis coaching while studying in a mathematics program. He was already planning to focus on AI alignment research but was unable to find a relevant supervisor at his university. After connecting him with one of our coaches, the coach became his supervisor which enabled him to pursue the topic he already had in mind. Now he is applying for PhD programmes planning to focus on AI safety research further. He also co-authored a paper on AI alignment for which he received feedback and guidance from his coach.

Student 3

He applied while studying a graduate degree in medicine, deciding between 2 directions he was considering (developing an alternative to QALY for measuring happiness vs biosecurity). Effective Thesis helped him via coaching to get inputs on how to set up a research project within these directions, frame his thesis as an important way of exploring his personal fit for research and finally decide which of the directions to focus on. He ended up focusing on biosecurity and as a part of his thesis, he then created a collaboration between Stanford, the US CDC, and Statens Serum Institut, where he was attempting to advance our understanding of the mortality attributable to seasonal flu and future influenza pandemics (however, he was planning to do this fellowship even before getting in touch with Effective Thesis). To support the research, he attracted more than 50,000 (USD) in funding, was a visiting researcher at Stanford for 6 months and visited the CDC. He is now in the process of publishing the results of this project and already presented it at the EuroMOMO conference and US CDC.

Selected students who applied in 2019

Student 4

She holds Honours in Pure Mathematics from the University of Sydney with 96.6/100 average mark, received several scholarships and also has published an academic paper already. She reached out to Effective Thesis to help her decide about her next steps to have the most impact with her research career. Based on a couple conversations with people in GPI and tasting AI safety research, she decided to pursue post-graduate study focusing on economics instead of pure maths in order to transfer to global priorities research. She said that if Effective Thesis didn't exist, there would be about an 80 % chance that she would have ended up continuing in pure maths.

She said that “..the main thing was that Effective Thesis forced me to spend a lot of time thinking about what the best path was, and if I hadn't spent that time, I wouldn't have made the decision, because it required quite a big push to convince myself to leave Pure Maths…
...Firstly, it just forced me to really spend time thinking about my options and their expected impact. Secondly, all the material I was given to read on AI Safety Research and Global Priorities Research was helpful in making me realise that I'm much more excited about Global Priorities Research. This wasn't at all clear to me coming into the coaching. Thirdly, being able to talk to people in Global Priorities Research was super helpful, especially Phil Trammell since he's an Economist.”

Student 5

At the time of application, she studied undergraduate in philosophy. She found a GPI research agenda on the Effective Thesis website and via reading it she found herself a topic. We then connected her with a GPI researcher who gave her feedback on her topic and advised her where to go next with it. She said that without ET she wouldn't have read the GPI agenda and probably would have ended up working on a different topic “guided by where I get the best grade (I would have chosen a different professor). ET got me thinking that I can maybe do something useful with my thesis”. Her topic idea was rated “very interesting” by one of the GPI researchers and she received very good grades by her supervisors.

Student 6

At the time of application, he was studying a masters in law. He said that Effective Thesis got him involved in EA which he hadn’t heard about before and also helped him with specific topic choice (counterfactually 40 %). His thesis topic is The Policy Diffusion of Localization in China, specifically how setting up of GDPR in the EU influenced Chinese data-protection policies. He is a US citizen studying masters of law at Yenching Academy of Peking University which is China’s top university for law. In the long-term, he is aiming to be a law professor helping the world in some practical aspect. That could be by practising law in addition to research or working in government.

There are also a couple of other students focusing on animal welfare, clean meat research and other causes that the EA community prioritises, but I haven’t included their stories so as not to make the post too long.

Aggregated descriptive data

Number of interested students

In 2019, there were 173 applications, out of which I wasn’t able to help (or had very low-quality applications from) about 92 (53%) of people, another 14 (8%) stopped communicating during the process and about 67 (39%) used the coaching advice and had an impact interview with me in the end.

Countries

People from 34 countries applied for coaching with most people applying from continental Europe (28 %), the UK (20 %), the US (12 %) and the rest of the world (most notably Australia, Israel, Canada, Russia). When considering only those who stayed in touch or had an impact interview with me, there were no clear differences in ratios (aside from the proportion of Europeans increasing to 36 % of the sample).

A comparison to the previous report (August 2018 - January 2019) would suggest that in 2019 there were fewer people applying from Europe (by 13 %).

Degree level

Most of the students who applied studied a master degree (42 %), then undergraduate degree (27 %) and then PhD degree 19 (11%) with 20% not reporting their degree levels. When considering only those who had stayed in touch or had an impact interview with me, the pattern holds the same with a slightly higher ratio of undergraduates.

A comparison to the previous report (August 2018 - January 2019) would suggest that the ratio of master and undergraduate students slightly increased on the expense of PhDs students.

University prestige

38 % of students who applied were from the top 100 universities rated by The Times Higher Education World University Ranking 2019. When considering only those who had stayed in touch or had an impact interview with me, this ratio increased to 48 %.

A comparison to the previous report (August 2018 - January 2019) would suggest that more people from prestigious universities applied (by 10 %).

Study disciplines

There was no clear pattern for which disciplines would be more likely to apply for our coaching. Disciplines included computer science (9.8 %), psychology (8.6 %), philosophy (7.5 %), mathematics (6.4 %), political science (6.4 %), communications/marketing, economics, environmental science and engineering (each 4.6 %) and most other disciplines (2.3 % or lower). When looking only at people who stayed in touch or had an impact interview with me or even at the best cases, there is no clear pattern either.

Cause areas

When looking at which cause areas people seem interested in working on, most people are interested in technical AI safety research (16 out of 79 known), closely followed by animal welfare (15 out of 79), AI governance (13 out of 79) and global priorities research (12 out of 79), other causes are referred to less (6 out of 79 or less). When looking only at the best case stories, there seem to be most people interested in working on global priorities research and animal welfare.

Sources of perceived value

I’ve tried to analyze what the main source of value was that students reported they received from interacting with Effective Thesis. Although pretty uncertain about this claim, it seems that in 2019, similarly to August 2018 - January 2019, students have found value in receiving suggestions for a general research direction they can take, in receiving feedback and help with refining the research topic they came up with themselves and in receiving academic career-related advice.

More students found value in receiving help with finding a specific thesis topic, which seems to be a positive improvement. However, given that many students identified sources of value other than help with the specific thesis topic choice, this leaves me in hesitation whether this is the main service which students need in order to start focusing on higher impact research.

Increasing the impact

When thinking about how to increase the value of the Effective Thesis project, I’ve gradually come to believe there might be several ways to do it - I will be glad for your opinion or other suggestions in the comments.

1) Focusing on students who are considering becoming researchers as one of their main long-term plans, since researchers might be influenced more directly by discussing their research direction than students planning to pursue other careers.

2) Focusing on non-EAs and people on the borders of the community rather than on EAs - it seems to me so far that many people who are highly involved in EA can find similarly good advice as we would be able to give them in their own circles so the counterfactual impact in this group is smaller.

3) Focusing on students choosing their PhD topic rather than their undergraduate or master thesis topic - future PhDs will devote more time to their research than undergraduates and masters and PhD also works as a "lock-in point" in many disciplines, making it harder for people to switch to a different research direction later in their research career. However, it is not clear which is the correct time point to influence people's PhD decisions - some of them are deciding in the last year of their undergraduate or master studies, so it actually might be a good strategy to keep focusing on undergraduates/masters but with an emphasis on choosing a PhD topic.

4) Focusing more on people from non-English speaking countries (or people not studying at the most prestigious universities in the UK and US) - even though getting into a prestigious university seems like a strong signal of talent, not getting there might not necessarily signal lack of talent - people who are talented might sometimes not get in because they have non-standard skills profile (e.g. high on one skill, but lower on another); are very skilled but don't want to move to some other country if they live outside the US or UK; are left out of the typical admission process for the prestigious universities because they lack a certain quality which is not correlated with research skills that much (e.g. not awesome extracurricular activities). I would expect there to be quite a lot of research-talented people (especially in non-English speaking countries because of the language and geographic barriers) who would be good to reach out to and who could produce very good research later on. I would expect outreach to these people to be significantly more neglected in comparison with outreach to people from prestigious unis, and thus it might be effective to focus on them. I've already identified potential talent pipelines in a couple of non-English speaking countries and am in the process of synthesizing it into a coherent strategy and possibly another EA forum post.

5) The value will likely come from fewer people who will be very skilled rather than from the average user. That being said, it might make more sense for the type of intervention that Effective Thesis provides not to focus on the very few researchers with transformative impact, but rather on the slightly higher number of (still very good) researchers who can work on the parts of the problems defined by other researchers.

6) Some of the value might come from more refined matching of people from various study disciplines with the important problems - both on the general level ("if you study math, you could consider biosecurity and pandemic modelling") and the specific level ("if you'd like help with figuring out which specific questions in global priorities research might be impactful, here is the research agenda and we can connect you with someone with the same background working on this problem"). This "specificity" can be achieved via harnessing and continuing to build a junior mentorship platform - connecting students looking for impactful topics with current PhD students who have already started working on some impactful projects. PhD level students might be knowledgeable enough to help pre-PhD students choose what to focus on, how to orient themselves in the academic world, etc.., it also minimizes the potential harm of wasting the scarce time of more senior researchers in the EA community and builds some mentorship capacity for the future of the EA movement. I have already built such a network of over 40 researchers who are happy to be connected with younger students and help them get on a more impactful research trajectory, most of them being PhD level or early post-docs. It seems pretty tractable to grow the network with more EA minded PhD-level students (e.g. from GPI fellowships, FHI research scholars, etc..) and we can also expect that a number of students we helped will be interested in becoming mentors once they've advanced a bit further in their academic career.

7) Focusing on a wide breadth of disciplines also seems valuable since as far as I know there is no single coordination research place for people from various study disciplines and having people from non-traditional disciplines increases the movement capacity for the future (e.g. if GPI decides to take on another discipline on the top of Philosophy and Economics, there will already be a couple of people from the other discipline who can give it a head start). A wider range of disciplines also contributes to the exploration of ways to do the most good.

8) Creating a signal of competence - motivating students who are considering becoming researchers to write a thesis on some EA related topic could also provide a specific signal of "how competent this person is in thinking about and researching this specific domain". Previous research work is already a factor that is assessed in almost all research hiring processes, but the question is how transferable is the assessed research quality in one domain (e.g. some traditional mainstream economics) to the other EA-related domain (e.g. global priorities research). I'm not sure about this, but if the transferability is not that high, the additional value of Effective Thesis might be in providing information about domain-specific research skills.

Forthcoming plans

Organisational plans

First, Effective Thesis is looking for collaborators: We would like to expand our team and take on a new team member who could help us carry out project operations. This would require an investment of at least 10 hours per week as we want to create a small and coherent team rather than a large and diffused one. The time investment might get paid for depending on the project receiving funding in the coming months, but so far it is volunteering-based. However, please reach out even if you'd consider joining us only in case your time was paid for. Specific responsibilities will be assigned based on project needs and personal fit, but generally, we are looking for someone proactive, comfortable with independently and creatively tackling various types of tasks and happy to become a core part of our team. The work is fully remote. If you’d like to join us or know of anyone who might be interested, please reach out at [email protected].

Second, Effective Thesis also has a funding gap of about $20.000 in 2020 and another $40.000 in 2021 (assuming 1-1,5 FTE + web design and maintenance cost). If you’d like to fund us or know of anyone who might be interested, please reach out at [email protected].

Strategy plans

First and foremost, I would like to finish developing and testing out a new outreach strategy, focusing mainly on talented research-oriented students from non-English speaking countries who have not ended up studying at the prestigious universities in the UK and US. As described above, I expect this to have a lot of direct value if successful and also a lot of learning value for the whole community.

Second, I would like to continue experimenting with and learning about which factors are important in students’ research topic choice and what kind of service might be good to offer them. This might also include searching for or putting together lists of potential supervisors for each research direction we prioritise or otherwise working with the supervision capacity.

Acknowledgements

I am very grateful to many people who enabled this project to exist. This project originated and in the first phases was supported by CZEA (most notably Jan Kulveit) and still uses support from some of its members (most notably Daniel Hnyk, Martin Račák, Vojtěch Veselý for their web development work). Many thanks go to all coaches involved (e.g. Philip Trammell, David Bernard, David Denkenberger, David Moss, Tomáš Gavenčiak to name some of the most active) and to a number of people helping with creating the content and improving research direction profiles. Caleb Huffman helped with organising the facebook online community for our students and many other students contributed by helping out each other. Many local group organisers also helped by promoting the project to their group members and helping me in exploring the potential new outreach strategy. Thanks Nicole Ross and Rose Hadshar for fruitful conversations about various aspects of the project. Thanks Hazel Browne and Ondřej Bajgar for proofreading and giving significant feedback on this post.
Many thanks also go to CEA and its grant managers for funding this project via EA Grants and EA Meta Fund.