Announcing Athena - Women in AI Alignment Research

I'm really excited about this! :)

One further thought on pitching Athena: I think there is an additional, simpler, and possibly less contentious argument about why increasing diversity is valuable for AI safety research, which is basically "we need everyone we can get". If a large percentage of relevant people don't feel as welcome/able to work on AI safety because of, e.g., their gender, then that is a big problem. Moreover, it is a big problem even if one doesn't care about diversity intrinsically, or even if one is sceptical of the benefits of more diverse research teams.

To be clear, I think we should care about diversity intrinsically, but the argument above nicely sidesteps replies of the form "yes, diversity is important, but we need to prioritise reducing AI x-risk above that, and you haven't given me a detailed story for how diversity in-and-of-itself helps AI x-risk, e.g., one's gender does not, prima facie, seem very relevant to one's ability to conduct AI safety research". This also isn't to dispute any of your reasons in the post, by the way, merely to add to them :)

Thanks for sharing these studies explaining why you are doing this. Unfortunately, in general I am very skeptical of the sort of studies you are referencing. The researchers typically have a clear agenda - they know what conclusions they want to come to ahead of time, and what conclusions will most advantageous to their career - and the statistical rigour is often lacking, with small sample sizes, lack of pre-registration, p-hacking, and other issues. I took a closer look at the four sources you referenced to see if these issues applied.

When more women participate in traditionally male-dominated fields like the sciences, the breadth of knowledge in that area usually grows, a surge in female involvement directly correlates with advancements in understanding[1]. [emphasis added]

The link you provide here, to a 2014 article in National Geographic, has a lot of examples of cases where male researchers supposedly overlooked the needs of women (e.g. not adequately studying how women's biology affects how drugs and seat belts should work, or the importance of cleaning houses), and suggests that increasing number of female scientists helped address this. But female scientists being better at understanding women seems less relevant to AI technical alignment work, because AIs are not female or male. Maybe it is useful for understanding what distinctly female values we want AIs to promote, but it doesn't seem particularly relevant for things like Interpretability or most other current research agendas. The article also suggests that women are more communal and emotionally aware, vs men who are more agentic. But it doesn't really make any claims about overall levels of understanding 'directly correlating' with female involvement, especially in more abstract, less biological fields, and the word 'correlate' literally does not appear in the text. 

Cox & Fisher (2008) found that women in a single-sex environment in a software engineering course reported higher levels of enjoyment, fairness, motivation, support, and comfort and allowed them to perform at a level that exceeded that of the all-male groups in the class [1]. 

The first paper describes a n=7 study of a female group project, which apparently scored more highly than other group projects run by men. The study was not pre-registered, blinded or randomised, the researcher was an active participant, and there was no control. The author also obliquely references the need to avoid ''rigid marking schemes' if these might reveal the all-female group performing worse, which suggests a bias to me.

Kahveci (2008) explored a program for women in science, mathematics, and engineering and found that it helped marginalized women move towards legitimate participation in these fields and enhanced a sense of community and mutual engagement [2]. 

The second paper describes a n=74 study of a women-in-science program, where the positive result is basically that the participants gave positive reviews to the program and said it made them more likely to do science. The study was not pre-registered, blinded or randomised, the researcher was an active participant, and there was no control. The only concrete example provided of a student switching major was from Biology to Exercise Physiology, which seems like a move away from core science. 

“It is not about men against women, but there is evidence to show through research that when you have more women in public decision-making, you get policies that benefit women, children and families in general. When women are in sufficient numbers in parliaments they promote women’s rights legislation, children’s rights and they tend to speak up more for the interests of communities, local communities, because of their close involvement in community life. [2]

The link here goes to a web page with a quote from Oxfam. There are no links to the evidence or research that supposedly backs up the claim. 

Overall, my opinion of the linked research is it has very little scientific merit. They provide some interesting anecdotes, and the authors have some theories that someone else could test. But to the extent you are highlighting them because they are cruxes for your theory of change, they seem very weak. If your 'Why We Are Doing This' had been premised on 'well some women just like sex-segregated programs, so proving this option will help with recruitment' then I would have said fair enough. But if, as this post suggests, your theory of change is based on these sorts of dubious studies then that makes me significantly less optimistic about the project.

I’m strongly in support of this initiative, and hope to help out as my schedule permits.

I agree with Larks that the linked studies have poor methodology and don’t provide sufficient support for their claims. I wish that there was better empirical research on this topic, but I think that’s unlikely to happen for various reasons (specifying useful outcome metrics is extremely difficult, political and researcher bias pushes hard toward a particular conclusion, human studies are expensive, etc.).

In lieu of reliable large-scale data, I’m basing my opinion on personal experiences and observations from my 5 years as a full time (cis female) AIS researcher, as well as several years of advising junior and aspiring researchers. I want to be explicit that I’d like better data than this, but am using it because it’s the best I have available.

I see two distinct ways that this initiative could be valuable for AIS research:

1. It could help us to recruit and retain more promising researchers. As Lewis commented, we need all the help we can get. While this community tries hard to be meritocratic, and is much less overtly hostile to women than neighboring communities I’ve experienced, I have personally noticed and experienced unintentional-yet-systemic patterns of behavior that can make it particularly difficult to remain and advance in this field as a woman. I’d prefer not to get into an in-depth discussion of that on here, though I have written about a bit of it in a related comment.^[1] I believe that a more gender-balanced environment, and particularly more accessible senior female researchers and mentors, would likely reduce this.
   
   I also suspect that more balanced gender representation would make more people feel comfortable entering the field. I am often the only woman at lab meetings, research workshops, and other AIS events, and very often the only woman who isn’t married to a man who is also in attendance. This doesn’t bother me, but I think that’s just a random quirk of my personality. I think it’s totally reasonable and not uncommon for people to be reluctant to join a group where their demographics make them stand out, and we could be losing female entrants this way. (Though I have noticed much more gender diversity in AIS researchers who’ve joined in the past <2 years than in those who joined >=5 years ago, so it’s possible this problem is already going away!)
    
2. Women (or any member of an underrepresented group or background) could provide important perspective for some areas of AIS research. It's important to distinguish between different research areas here, so I’m gonna messily put AIS topics on a spectrum between “fundamental” and “applied”. By “fundamental”, I mean topics like interpretability, decision theory, science of deep learning, etc — work to understand, predict, and figure out how to control AI behavior at all. By “applied”, I mean topics like practical implications of RLHF when teachers have differing preferences, or constructing meaningful evaluations for foundation models — work to understand, predict, and dictate how AI interacts with the real world and groups of humans. 
   
   On the “fundamental” end of the spectrum, I don’t think that diversity in researcher background and life experience really matters either way. But in topics further toward the “applied” end of the spectrum, it can help a whole lot. There’s plausibly-important safety work happening all along this spectrum, especially now that surprisingly powerful AI systems are deployed in the real world, so there are areas where researchers with diverse backgrounds can be particularly valuable.

Overall, I think that this is an excellent thing to dedicate some resources to on the margin.

1. ^{^}

   A relevant excerpt: "most of these interactions were respectful, and grew to be a problem only because they happened so systematically -- for a while, it felt like every senior researcher I tried to get project mentorship from tried to date me instead, then avoided me after I turned them down, which has had serious career consequences."

Wow. This is exactly the type of opportunity I was looking for. I'm excited to apply!

I'm sceptical that there are substantial benefits to generating AI safety research ideas from gender diversity. I haven't read the literature here, but my prior on these types of interventions is that the effect size is small. 

I regardless think Athena is good for the same reasons Lewis put forward in his comment - the evidence that women are excluded from male-dominated work environments seems strong and it's very important that we get as many talented researchers into AI safety as possible. This also seems especially like a problem in the AIS community where anecdotal claims of difficulties from unwanted romantic/sexual advances are common. 

I think the intellectual benefits from gender diversity claims haven't been subjected to sufficient scrutiny because it's convenient to believe. For this kind of claim, I would need to see high-quality causal inference research to believe it and I haven't seen this research and the article linked doesn't cite such research. The linked NatGeo article doesn't seem to me to bring relevant evidence to bear on the question. I completely buy that having more women in the life sciences leads to better medical treatment for women, but that causal mechanism at work here doesn't seem like it would apply to AI safety research. 

I appreciate your feedback and comments! To clarify - my vision for this program is to emphasize that it's not a choice between x-risk and diversity; rather both can be pursued simultaneously and effectively. The core of this experiment is to integrate these two objectives. It's not about promoting diversity solely as a means to mitigate x-risk (though my intuition says that more diversity leads to a diverse range of ideas that will help with x-risk solutions). Instead, this program aims to address x-risk while concurrently supporting and helping retain qualified women in a predominantly male field that may otherwise consider leaving. This approach is based primarily on recent interviews I conducted with women in the alignment field and in EA, which revealed specific patterns and issues. 

It's important to note that this program isn't solely based on the research I mentioned here. In hindsight I regret emphasizing those papers, as it may have diverted attention from the program's primary objectives and reasoning for some. The cited studies were meant to provide additional insights rather than serve as definitive proof of the program's value, which I should have stated more clearly. I’m not very surprised that there are small sample sizes or limited studies on this, which should not be misinterpreted as a lack of such gender-based issues. Ultimately whether the program is a success or a learning experience, we'll just have to wait and let the results speak for themselves :)

I’ll have a think about which women I know whom I should suggest apply for this program. Do you have any more details about the kinds of candidates that you’re looking for?

This is really cool, thanks for organising it!