[This is my best attempt at summarizing a reasonable outsider's view of the current state of affairs. Before publication, I had this sanity checked (though not necessarily endorsed) by an EA researcher with more context. Apologies in advance if it misrepresents the actual state of affairs, but that's precisely the thing I'm trying to clarify for myself and others.]
At GiveWell, the standard of evidence is relatively well understood. We can all see the Cost Effectiveness Analysis spreadsheet (even if it isn't taken 100% literally), compare QALYs and see that some charities are likely much more effective than others.
In contrast, Open Philanthropy is purposefully opaque. As Holden describes in "Anti-principles" for hits-based giving:
We don't: require a strong evidence base before funding something. Quality evidence is hard to come by, and usually requires a sustained and well-resourced effort. Requiring quality evidence would therefore be at odds with our interest in neglectedness.
And:
We don't: expect to be able to fully justify ourselves in writing... Process-wise, we've been trying to separate our decision-making process from our public writeup process. Typically, staffers recommend grants via internal writeups. Late in our process, after decision-makers have approved the basic ideas behind the grant, other staff take over and "translate" the internal writeups into writeups that are suitable to post publicly. One reason I've been eager to set up our process this way is that I believe it allows people to focus on making the best grants possible, without worrying at the same time about how the grants will be explained.
These are reasonable anti-principles. I'm not here to bemoan obfuscation or question the quality of evidence.
(Also note this recent post which clarifies a distinction within Open Phil between "causes focused on maximizing verifiable impact within our lifetimes" and "causes directly aimed at affecting the very long-run future". I'm primarily asking about the latter, which could be thought of as HoldenOpenPhil in contrast to the former AlexOpenPhil.)
My question is really: Given that so much of the decision making process for these causes is private, what are we actually debating when we talk about them on the EA Forum?
Of course there are specific points that could be made. Someone could, in relative isolation, estimate the cost of an intervention, or do some work towards estimating its impact.
But when it comes to actually arguing that X is a high priority cause, or even suggesting that it might be, it's totally unclear to me both:
- What level of evidence is required.
- What level of estimated impact is required.
To give some more specific examples, it's unclear to me how someone outside of Open Philanthropy could go about advocating for the importance of an organization like New Science or Qualia Research Institute.
Or in the more recent back-and-forth between Rethink Priorities and Mark Lutter on Charter Cities, Linch (of RP) wrote that:
I don't get why analyses from all sides [keep] skipping over detailed analysis of indirect effects.* To me by far the strongest argument for charter cities is the experimentation value/"laboratories of governance"angle, such that even if individual charter cities are in expectation negative, we'd still see outsized returns from studying and partially generalizing from the outsized successful charter cities that can be replicated elsewhere, host country or otherwise (I mean that's the whole selling point of the Shenzhen stylized example after all!).
At least, I think this is the best/strongest argument. Informally, I feel like this argument is practically received wisdom among EAs who think about growth. Yet it's pretty suspicious that nobody (to the best of my knowledge) has made this argument concrete and formal in a numeric way and thus exposed it to stress-testing.
I agree that this is a strong argument for charter cities. My (loose) impression is that it's been neglected precisely because it's harder to express in a formal and numeric way than the existing debate (from both sides) over economic growth rates and subsequent increases to time-discounted log consumption.
Again, I'm not here to complain about streetlight effects or express worry that EA tries too hard to quantify things. I understand the value of that approach. I'm specifically asking, as far as it concerns the Holden Open Phil world, which is expressly (as I understand it) more speculative, risk-neutral and non-transparent than some other EA grant makers, what is the role of public EA Forum discussion?
Some possibilities:
- Public discussion is meant to settle specific questions, but not address broader questions of grant-worthiness.
- Even in public, discussions can be productive as long as they have sufficient context on Holden Open Phil priorities, either through informal channels, or interfacing with HOP directly (perhaps as a consultancy).
- Absent that context, EA Forum serves more like training grounds on the path to work within a formal EA organization.
But isn't the GiveWell-style philanthropy exactly not applicable for your example of charter cities?
My sense is that the case for charter cities has some macro/systems process that is hard to measure (and that is why it is only now a new cause area and why the debate exists).
I specifically didn't want to pull out examples, but if it's helpful here's another example of a debate for an intervention that relies on difficult to measure outcomes and involves, hard to untangle, divergent worldviews between the respective proponents.
(This is somewhat of a tangent but honestly, your important question is inherently complex and there seems to be a lot of things going on, so clarity from smoothing out some of the points seems valuable.)
I don't understand why my answer in the previous post above, or these debates aren't object level responses to how you could discuss the value of these interventions.
I'm worried I'm talking past you and not being helpful.
Now, trying more vigorously / speculatively here:
At the risk of being prescriptive (which I dislike) I think this is a reasonable attitude on the forum, in the sense that "policing grants" or something, should be a very low priority for organic reasons for most people, and instead learning/communicating and a "scout mindset" is ultimately more productive. But such discussion cannot be proscribed and even a tacit norm against them would be bad.
I could see how HOP areas are harder but as in my first comment, I think it's inherently hard for anyone to criticize any well researched grant, especially if you account for social factors, as you do.
However, I think there are ways to indicate comfort or discomfort with grants or even major EA orgs.
There are specific examples of this, where people have individually started posts that were influential and drew enormous attention to their concerns.
If you go to the all posts page and select "Yearly" and "Top", you will find an remarkable example in the top 10 for 2021 (please do not link this post in any reply).
I'm pretty sure that some of these posts do influence Open Phil (but maybe not quite in the way we want).
Again, I choose a negative example because it's harder.
There's abundant posts that promote or talk about their work in a cause. In some sense. all blog posts from orgs promote their orgs and pretty much any such honest writing from an EA is welcome.
I think there's a causal chain where this can influence Open Phil or specific grant makers.
By the way, I'm 90% sure that two or more Open Phil members will read/has read your post already.