Preliminary investigations on if STEM and EA communities could benefit from more overlap

TLDR

This post discusses the potential benefits of increased overlap between the EA and STEM communities, particularly for addressing existential risks.

We (at Foresight Institute, a non-profit working to advance technology for good) want to explore if introducing more EA ideas into scientific discussions and further informing the EA dialogue with novel scientific perspectives could increase awareness of potential risks with emerging technologies and identify new areas for differential technology development.

We conducted some very informal and preliminary investigations through conversations with 20 risk-researchers and a survey sent out to the Foresight STEM community, which suggested that most respondents saw a gap between the two communities and believed that more overlap would be beneficial.

Would the STEM and EA communities benefit from more bridge-building?

I currently work at an organization whose primary aim is to advance beneficial technology: the Foresight Institute. We focus on technologies that are too early-stage or interdisciplinary for legacy institutions to support and have a long history of community building within the STEM community. Our  network of spans various fields of science and technology that seem relevant to several EA cause areas, especially biotech/biorisk, neurotech/neurorisk, nanotech/nanorisk, and AI safety.

Although Foresight has significant overlap with EA, it is not an "EA-org" in itself. We receive very little funding from EA sources, and our approach to identifying worthwhile interventions is broader, encompassing beneficial applications in nanotech, neurotech, longevity, and space tech. However, we have participated in EA events since around 2015 and collaborated with many people and organizations within the EA ecosystem.

Our community at Foresight primarily consists of scientists and entrepreneurs working in the areas mentioned above. While many do not have a distinctly EA perspective, they deeply care about the long-term potential and risks associated with their technologies. Their mindset aligns well with the increasingly popular EA focus on differential technology development. As a result, we are considering whether it would be beneficial to work on building bridges between EA concepts, projects, and people—particularly those with a DTD focus—and our community.

This would involve:

• Introducing more EA ideas into scientific discussions,
• Further informing the EA dialogue with specific potential near- and long-term capabilities currently explored in the STEM community,
• Increasing long-term coordination amongst scientists/technologists and policy/risk researchers.

We believe that the key benefits of this bridge-building can be roughly summed up as follows:

• Increase awareness among hard science/tech actors regarding potential risks associated with their work, mitigation strategies, and distinctly beneficial applications to potentially prioritize.
• Increase awareness among risk and policy researchers regarding technological capabilities under development, as well as potential technological risk mitigation approaches.
• Foster long-term trust-building across these communities, which may facilitate easier collaboration on realizing risks and pursuing opportunities arising from these technologies as they mature.
• Identify potential areas for differential technology development and project collaborations in these areas.

Shallow Investigations

To explore this opportunity, we have conducted some informal preliminary investigations, including conversations I've had with approximately 20 relevant actors at EAG (identified as people doing or coordinating risk research on EA cause areas, mainly for AI and biorisk). We also sent out a survey to the Foresight community, which generated 41 responses from participants in our technical groups (computation, nanotech, neurotech, longevity, and space tech).

Conversations with risk-researchers

At the last two EAGs I attended, I had around 20 individual meetings with relevant actors working on reducing existential risk via technical research, mainly for AI and biorisk. These were a combination of mid and senior-level people conducting direct research on these topics and people coordinating programs aimed at reducing existential risk (for example, FHI, BERI, OpenPhil, SERI, as well as individual researchers with EA funding).

Very briefly summarized, I would interpret the results from these conversations to be something like:

• Everyone I talked to agreed that there is a gap between the communities. They believed that some level of collaboration was already happening but could still be improved. Everyone agreed that more overlap would be beneficial, except for one person who did not think additional collaboration would be useful for the field of AI, as they felt the existing cooperation was sufficient.
• Three people said that in working to bridge this gap, it seems important that these meetings need to be facilitated in a way that makes sure it does not lead to a (further) divide between communities. This mainly refers to STEM people interpreting a risk-reducing focus as an attempt to hinder progress. They suggested that open communication, mutual respect, and finding common ground should be prioritized to ensure a healthy exchange of ideas and prevent misunderstandings.

In the conversations, I aimed to ask questions in a way that made them open for discussion, but as the answers were quite homogenous, I believe there is a high risk that the way I phrased my questions during these conversations may have affected the responses and led to a bias of people agreeing with me on these topics.

Survey sent out to Foresight’s STEM community

In addition to these informal EA discussions, we also sent out a survey to the Foresight community, which mainly consists of people operating in the "hard" tech/science field. In terms of demographics, the majority were male and living in the US. A few respondents were based in the UK, and only one woman answered. The survey had the topic "Help Foresight improve our work on existential risks" and consisted of questions asking:

• If they are familiar with the concepts of Effective Altruism, Existential Risk and/or Longtermism
• If they think that EA/X-risk/longtermist ideas have far to go in being integrated into the mainstream science/tech community
• If they think that both communities (The EA/X-risk/longtermist community and the science/tech community) could benefit from more interaction.

You can access the actual survey sent out here.

Survey results

To protect the privacy of respondents, we will not be sharing the raw data here, but readers are welcome to reach out to me should they wish to dig deeper on this topic and the data we collected. However, we will paraphrase quotes from the survey.

In terms of who responded to this survey, there seemed to be a majority of respondents who were already aware of these concepts. 78% of respondents described themselves as familiar with these concepts, 19% as a little familiar, and only 3% were not aware at all, pointing to some selection bias in survey respondents.

86% self-identified as STEM professionals. 45% said they identified themselves as being "involved with the Effective Altruism community." This, in itself, could be somewhat interesting since the survey was sent out to about ~1800 people. As the survey had been given the topic "Help Foresight improve our work on existential risks," this could imply that the number of people aware of the idea of Existential Risks in our STEM community is potentially very low.

• 90% of respondents answered yes to the question of whether they agree that “EA/Xrisk/longtermist ideas have far to go in being integrated into the mainstream science/tech community”.
• 80% of respondents answered yes to the question of whether they agree that “both communities (The EA/Xrisk/longtermist community and the science/tech community) could benefit from more interaction”.

I will expand a bit more below on people’s reasoning for their answers, and divide the responses into two different categories, Category 1 being those saying “Yes, these communities would benefit from more overlap” and Category 2 saying “No, these communities would NOT benefit from more overlap”.

Category 1: Yes, these communities would benefit from more overlap

I’d categorize these answers into the following buckets:

1. Lack of awareness and need for education: Many respondents believed that people in STEM are not aware of effective altruism (EA) and existential risk (X-risk) ideas. Respondents suggested that more outreach and education are needed to raise awareness of these topics.
2. Technology is very important in solving X-risk challenges: Many respondents believe that technology has the potential to help mitigate X-risks. They feel that getting technologists involved in the conversation is essential to developing effective solutions.
3. Gap between ideas and execution: Some respondents stated that while STEM may not have a clear understanding of what is worthwhile, EAs may not have the skills to execute their ideas effectively. Bridging this gap could ensure that ideas can be more effectively executed.
4. Engagement with policy and long-term strategy: Respondents said there is a need for STEM to engage more with policy, philanthropy, and strategy to make long-term investments and risk mitigation possible. More EA integration would already be a step on the way towards more engagement on these topics.
5. Overly focused on short-term gains: Some respondents noted that most science and tech projects and their financial backers focus on short-term gains, rather than long-term interventions and risk mitigation. This could make it more difficult to invest in long-term solutions to X-risks. Something which more integration with EA could potentially help avoid.

Paraphrased quotes from the respondents:

• "Throughout my many years in the field, I've seldom encountered dedication to these principles. When I bring them up and advocate for awareness and action, it's typically from a conventional standpoint grounded in the essentials of the UN SDGs."
• "Technological advancements will be key in addressing existential risks, rather than depending on government or regulation. Involving technologists is crucial in this regard."
• "While effective altruists struggle with practical implementation, engineers often lack insight into what's truly valuable."
• "Many researchers, including myself, may not possess a clear or even basic understanding of these concepts. Nonetheless, it is important that we do, as all scientists aim to benefit humanity in some way."
• "It is essential for more individuals in STEM to grasp the importance of effective altruism, long-term thinking, and existential risk. At the same time, those in STEM can contribute significantly to addressing the challenges prioritized by effective altruism, long-term thinking, and existential risk management."

Category 2: No, these communities would NOT benefit from more overlap

I’d categorize these answers into the following buckets:

1. There is already significant overlap between the groups: Respondents felt that there is already a large portion of the EA/X-risk/longtermism community composed of STEM professionals, so a rigorous scientific worldview is not missing.
2. There is a risk of diluting the potency of energy in the EA community: Some believe that trying to cast too broad a net could weaken the effectiveness of the community by spreading its focus too thin.
3. Decentralized development and diversification is antifragile: Some stated that institutions like the Foresight Institute and the Center for Progress Studies are valuable because they are adjacent to but do not rigidly withinEA cause areas. In an uncertain world, diversification is antifragile, and strictly adhering to EA could make both communities more fragile.
4. The value of technological foresight is questionable: Some respondents believe that it is impossible to predict how far a technology will go and how it will be used, so discussing extremely long-term possibilities is not productive.
5. Trying to address existential risks can increase existential risk: Some respondents are not fans of trying to address existential risk because they believe that focusing too heavily on the issue can lead to actions that inadvertently increase the likelihood of existential risk.

Paraphrased quotes from the respondents:

• "A considerable portion of the effective altruism, existential risk, and long-term thinking community already consists of STEM professionals, ensuring that a rigorous scientific perspective is not absent. Furthermore, there is a potential risk of diluting the community's energy by trying to expand its scope too broadly."
• "I somewhat agree. However, organizations like the Foresight Institute occupy a unique position that allows for increased STEM involvement, discussion, and investigation beyond conventional effective altruism concerns. For instance, expertise in neurotechnology may not qualify someone for EA Global. Thus, it's valuable to have institutions like the Foresight Institute and the Center for Progress Studies that are related but do not strictly adhere to EA efforts. In an unpredictable world, decentralized development and diversification promote resilience; a strict adherence to effective altruism would make us more vulnerable."
• "I believe that there is already considerable overlap between these groups, and there may not be significant benefits to using resources to push for even greater integration."
• "If the Foresight Institute aims to contribute to the existential risk field, it would be more valuable to invest its efforts in debunking widely-accepted existential risk claims and strategies, especially those intending to restrict or slow down technological advancement."

Conclusions and recommended actions

Again, we want to emphasize that this was a very informal and preliminary investigation! But if we decide to work more on bridge building between these communities, here are a few avenues we would think are useful to pursue.

Broad ideas on what we can do to bridge the gap between EA and STEM:

1. Have more risk researchers participate in STEM events: Identify and invite risk researchers and invite them to relevant technical events and workshops to both inform and connect individuals from both communities
2. Build out DTD specific fellowships: Build out our existing fellowship, which already builds community across STEM and risk/policy researchers in these areas to facilitate more high-trust community-building across specific individuals, including grants for collaborative DTD research projects
3. Host intros on risk and mitigation strategies for the STEM audience: Host sessions on existential risks and EA cause areas for the STEM community
4. Determine areas for differential technology development and communicate to the STEM audience: Identify a few technologies identified in different STEM domains that would differentially increase safety in this domain, and encourage further work on it.

For all of these examples, we can see that this is something we have a good opportunity to integrate into the Foresight program. But we would be very interested in hearing if readers of this post have any thoughts on;

1. General criticisms of any of the points discussed above
2. Avenues for facilitating DTD discussions without increasing info hazards
3. Specific technological areas you deem promising for bridge-building
4. Additional mechanisms, such as prizes, fellowships, events that could be useful
5. Pointers to existing efforts with the goals discussed in this post

Please let us know in the comments if you have any thoughts or ideas you would like to share with us.