Following up on my earlier question post, The Unjournal now aims to evaluate at least one piece of research for each ~EA-funded research org in our domain, following the approach below. We'll be reaching out to these organizations directly soon, asking for their suggestions, and otherwise identifying research outputs. If you are at one such organization, please feel free to reach out to me in advance of this, if you like (here's the "submission form", but an email is good too). 

I've put together a brief list/database of the most relevant organizations here, focusing on those receiving substantial EA-aligned funding (esp. from Open Phil. or the SFF) that produce research that seems to be in our domain (~quantitative social science).  In many cases I took only a quick/moderate look in doing the prioritization, and there may be inaccuracies, so I'd love feedback. I also need to dig into more detail for some of the "funded university" research.

We've already evaluated (or are evaluating) some work funded by or linked to Open Philanthropy, GiveWell, GiveDirectly, the Gates Foundation, the Forecasting Research Institute, Founders Pledge, the Market Shaping Accelerator, the Global Priorities Institute, ALLFED,  Center for Effective Global Action (CEGA), Center for Global Development, JPAL, the World Bank, and Innovations for Poverty Action

Key organizations that we've not yet evaluated research from, that seem relevant include 

• Rethink Priorities
• Happier Lives Institute
• Global Catastrophic Risk Institute
• Animal Charity Evaluators
• Center for Security and Emerging Technology
• Ambitious Impact (formerly Charity Entrepreneurship)
• ID-Insight
• University of Chicago Existential Risk Laboratory
• The Center for Election Science
  Centre for Pesticide Suicide Prevention
• AidData
• Faunalytics
• Association for Long Term Existence and Resilience (ALTER)
• Centre for the Study of Existential Risk

• AI Impacts

   

Again see the database here for the full prioritization list.

 

Context (reprising earlier post)

 The Unjournal (unjournal.org) finds and solicits and prioritizes open-access research for impact and commissions it to be publicly evaluated. We focus on economics, quantitative social science, and impact assessment. See our output at unjournal.pubpub.org

For prominent/influential work, from senior authors, we ask for engagement, not permission (see our 'direct evaluation stream'). If the authors are ~junior/not prominent/less influential we ask and require their permission before we evaluate this work. 

We've mostly evaluated academic-oriented work, but we also have an Applied and Policy stream.  

We'd like to do more in the latter stream, engage with more EA-aligned research orgs, and add more value to the EA community.   

We're planning the following approach (reprise) 

1. Identify organizations linked/adjacent to EA &/or focusing on impact, that have been granted over ~$300k to do research (especially from Open Phil, EA Funds, SFF, etc. ). (See database above, will be refined).

2. Ask each organization to identify one piece of their research they would like to be evaluated (probably in our ~applied stream). For larger multi-division orgs, we might ask for one piece of research per major division (~per $300k).

We will otherwise evaluate the research that seems to be most heavily promoted/influential/driving funding. We could engage with the funding orgs/donors and this community to identify the best candidate research. We'll allow some exceptions especially if the funders suggest this (e.g., for info hazards)

Will reach out on the EA forum for further feedback; feel free to contact me directly.

3. Commission these for public evaluations, give the authors/organizations a chance to respond, post evaluations prominently (on EA forum etc.) We will be careful about COI with members of our management team, evaluators, etc., as always.

Why we are doing this

‘If you are getting funding to do impactful research you “should” be having some of it publicly evaluated ... unless there are strong reasons not to, such as info hazards’.

Why?  To help funders and practitioners know if and how to use the research, to provide building blocks for further research, to open productive communication, and for due diligence/quality control, so funders and donors can judge the value of the work. 

We hope these public evaluations will also be useful to budding researchers and research-users.

Disambiguation from 'Pivotal questions'

This has some connections to our "Pivotal Questions" initiative, with related goals. However, here

• here we're reaching out mainly to research-producing organizations
• and asking them to suggest their own research for our evaluation

In contrast, for Pivotal Questions we're reaching out to a range of stakeholders and organizations, not necessary those that produce research. We're asking them to identify and address the questions that could be informed by research that are high-value to them. See that post and this link on our knowledge base.

Evaluating the research, not the organizations themselves

We aim to commission some research from each organization for evaluations, often by academic experts. We don't intend these to be seen as evaluations of these organizations themselves, only as offering some feedback and assessment of some of their research. We also recognize that evaluator and academic feedback may be misguided, particularly if they misunderstand the applied research goals, etc. We will bear this in mind.  

But we hope these evaluations will generate some key value... 

• EA-unaffiliated academics (and others) may be generate useful critiques and suggestions for this work

• This can help us understand "how much should we trust EA-driven research?" in general and in different areas, and recalibrate, adapt and improve.

   

We're planning to push forward with this soon, but we have not finalized the approach or the plan. All feedback is welcome, of course.