I previously gave a fair bit of feedback to this document. I wanted to quickly give my take on a few things.

Overall, I found the analysis interesting and useful. However, I overall have a somewhat different take than Nuno did.

On OP: 
-  Aaron Gertler / OP were given a previous version of this that was less carefully worded. To my surprise, he recommended going forward with publishing it, for the sake of community discourse. This surprised me and I’m really thankful. 
- This analysis didn’t get me to change my mind much about Open Philanthropy. I thought fairly highly of them before and after, and expect that many others who have been around would think similarly. I think they’re a fair bit away from being an “idealized utilitarian agent” (in part because they explicitly claim not to be), but still much better than most charitable foundations and the like.

On this particular issue: 
- My guess is that in the case of criminal justice reform, there were some key facts of the decision-making process that aren’t public and are unlikely to ever be public. It’s very common in large organizations for compromises to be made for various political or social reasons, for example. I’ve previously written a bit about similar things [here](https://twitter.com/ozziegooen/status/1456992079326978052).
- I think Nuno’s quantitative estimates were pretty interesting, but I wouldn’t be too surprised if other smart people would come up with numbers that are fairly different. For those reading this, I’d take the quantitative estimates with a lot of uncertainty.
- My guess is that a “highly intelligent idealized utilitarian agent” probably would have invested a fair bit less in criminal justice reform than OP did, if at all.

On evaluation, more broadly: 
- I’ve found OP to be a very intimidating target of critique or evaluation, mainly just because of their position. Many of us are likely to want funding from them in the future (or from people that listen to them), so the risk of getting people at OP upset is very high. From a cost-benefit position, publicly critiquing OP (or other high-status EA organizations)  seems pretty risky. This is obviously unfortunate; these groups are often appreciative of feedback, and of course, they are some of the most useful groups to get feedback. (Sometimes prestigious EAs complain about getting too little feedback, I think this is one reason why). 
- I really would hate for this post to be taken as “ammunition” by people with agendas against OP. I’m fairly paranoid about this. That wasn’t the point of this piece at all. If future evaluations are mainly used as “ammunition” by “groups with grudges”, then that makes it far more hazardous and costly to publish them. If we want lots of great evaluations, we’ll need an environment that doesn’t weaponize them.
- Similarly to the above point, I prefer these sorts of analysis and the resulting discussions to be fairly dispassionate and rational. When dealing with significant charity decisions I think it’s easy for some people to get emotional. “$200M could have saved X lives!”. But in the scheme of things, there are many decisions like this to make, and there will definitely be large mistakes made. Our main goals should be to learn quickly and continue to improve in our decisions going forward. 
- One huge set of missing information is OP’s internal judgements of specific grants. I’m sure they’re very critical now of some groups they’ve previously funded (in all causes, not just criminal justice). However, it would likely be very awkward and unprofessional to actually release this information publicly.
- For many of the reasons mentioned above, I think we can rarely fully trust the public reasons for large actions by large institutions. When a CEO leaves to “spend more time with family”, there’s almost always another good explanation. I think OP is much better than most organizations at being honest, but I’d expect that they still face this issue to an extent. As such, I think we shouldn’t be too surprised when some decisions they make seem strange when evaluating them based on their given public explanations.

(Writing from OP’s point of view here.)

We appreciate that Nuño reached out about an earlier draft of this piece and incorporated some of our feedback. Though we disagree with a number of his points, we welcome constructive criticism of our work and hope to see more of it.

We’ve left a few comments below.

*****

The importance of managed exits

We deliberately chose to spin off our CJR grantmaking in a careful, managed way. As a funder, we want to commit to the areas we enter and avoid sudden exits. This approach:

1. Helps grantees feel comfortable starting and scaling projects. We’ve seen grantees turn down increased funding because they were reluctant to invest in major initiatives; they were concerned that we might suddenly change our priorities and force them to downsize (firing staff, ending projects half-finished, etc.)
2. Helps us hire excellent program officers. The people we ask to lead our grantmaking often have many other good options. We don’t want a promising candidate to worry that they’ll suddenly lose their job if we stop supporting the program they work on.

Exiting a program requires balancing:

• the cost of additional below-the-bar spending during a slow exit;
• the risks from a faster exit (difficulty accessing grant opportunities or hiring the best program officers, as well as damage to the field itself).

We launched the CJR program early in our history. At the time, we knew that committing to causes was important, but we had no experience in setting expectations about a program’s longevity or what an exit might look like. When we decided to spin off CJR, we wanted to do so in a way that inspired trust from future grantees and program staff. In the end, we struck what felt to us like an appropriate balance between “slow” and “fast”.^[1]

It’s plausible that we could have achieved this trust by investing less money and more time/energy. But at the time, we were struggling to scale our organizational capacity to match our available funding; we decided that other capacity-strained projects were a priority.

*****

Open Phil is not a unitary agent

Running an organization involves making compromises between people with different points of view — especially in the case of Open Phil, which explicitly hires people with different worldviews to work on different causes. This is especially true for cases where an earlier decision has created potential implicit commitments that affect a later decision.

I would avoid trying to model Open Phil (or other organizations) as unitary agents whose actions will match a single utility function. The way we handle one situation may not carry over to other situations.

If this dynamic leads you to put less “trust” in our decisions, we think that’s a good thing! We try to make good decisions and often explain our thinking, but we don’t think others should be assuming that all of our decisions are “correct” (or would match the decisions you would make if you had access to all of the relevant info).

*****

“By working in this area, one could gain leverage, for instance [...] leverage over the grantmaking in the area, by seeding Just Impact.”

Indeed, part of our reason for seeding Just Impact was that it could go on to raise a lot more money, resulting in a lot of counterfactual impact. That kind of leverage can take funding from below the bar to above it.

*****

Open Philanthropy might gain experience in grantmaking, learn information, and acquire expertise that would be valuable for other types of giving. In the case of criminal justice reform, I would guess that the specific cause officers—rather than Open Philanthropy as an institution—would gain most of the information. I would also guess that the lessons learnt haven’t generalized to, for instance, pandemic prevention funding advocacy.

This doesn’t accord with our experience. Over six years of working closely with Chloe, we learned a lot about effective funding in policy and advocacy in ways we do expect to accrue to other focus areas. She was also a major factor when we updated our grantmaking process to emphasize the importance of an organization's leadership for the success of a grant. 

It’s possible that we would have learned these lessons otherwise, but given that Chloe was our first program officer, a disproportionate amount of organizational learning came from our early time working with her, and those experiences have informed our practices.

1. ^{^}

   Note that when we launched our programs in South Asian Air Quality and Global Aid Policy, we explicitly stated that we "expect to work in [these areas] for at least five years". This decision comes from the experience we’ve developed around setting expectations.

One context note that doesn't seem to be reflected here is that in 2014, there was a lot of optimism for a bipartisan political compromise on criminal justice reform in the US. The Koch network of charities and advocacy groups had, to some people's surprise, begun advocating for it in their conservative-libertarian circles, which in turn motivated Republican participation in negotiations on the hill. My recollection is that Open Phil's bet on criminal justice reform funding was not just a "bet on Chloe," but also a bet on tractability: i.e., that a relatively cheap investment could yield a big win on policy because the political conditions were such that only a small nudge might be needed. This seems to have been an important miscalculation in retrospect, as (unless I missed something) a limited-scope compromise bill took until the end of 2018 to get passed. I'm not aware of any significant other criminal justice legislation that has passed in that time period. [Edit: while this is true at the national level, arguably there has been a lot of progress on CJR at state and local levels since 2014, much of which could probably be traced back to advocacy by groups like those Open Phil funded.]

This information strongly supports the "Leverage Hypothesis," which was cited by Open Phil staff themselves, so I think it ought to be weighted pretty strongly in your updates.

Conversely, if sentences are reduced more than in the margin, common sense suggests that crime will increase, as observed in, for instance, San Francisco.

A bit of a nit since this is in your appendix, but there are serious issues with this reasoning and the linked evidence. Basically, this requires the claims that:
1. San Francisco reduced sentences
2. There was subsequently more crime

1. Shellenberger at the WSJ writes:

 the charging rate for theft by Mr. Boudin’s office declined from 62% in 2019 to 46% in 2021; for petty theft it fell from 58% to 35%. 

He doesn't provide a citation, but I'm fairly confident he's pulling these numbers from this SF Chronicle writeup, which is actually citing a change from 2018-2019 to 2020-2021. So right off the bat Shellenberger is fudging the data.

Second, the aggregated data is misleading because there were specific pandemic-effects in 2020 unrelated to Boudin's policies. If you look at the DA office's disaggregated data, there is a drop in filing rate in 2020, but it picks up dramatically in 2021. In fact, the 2021 rate is higher than the 2019 rate both for crime overall, and for the larceny/theft category. So not only is Shellenberger's claim misleading, it's entirely incorrect.

You can be skeptical of the DA office's data, but note that this is the same source used by the SF Chronicle, and thus by Shellenberger as well. 

2. Despite popular anecdotes, there's really no evidence that crime was actually up in San Francisco, or that it occurred as a result of Boudin's policies.
- Actual reported shoplifting was down from 2019-2020
- Reported shoplifting in adjacent countries was down less than in California as a whole, indicating a lack of "substitution effects" where criminals go where sentences are lighter
- The store closures cited by Shellenberger can't be pinned on increased crime under Boudin because:
A) Walgreens had already  announced a plan to close 200 stores back in 2019
B) Of the 8 stores that closed in 2019 and 2020, at least half closed in 2019, making the 2020 closures unexceptional
C) The 2021 store closure rate for Walgreens is actually much lower than comparable metrics, like the closures of sister company Duane Reader in NYC over the same year, or the dramatic drop in Walgreens stock price. It is also not much higher than the historical average of 3.7 store closures per year in SF.

I have a much more extensive writeup on all of this here:
https://applieddivinitystudies.com/sf-crime-2/

Finally, the problem with the "common sense" reasoning is that it goes both ways. Yes, it seems reasonable to think that less punishment would result in more crime,  but we can similarly intuit that spending time in prison and losing access to legal opportunities would result in more crime. Or that having your household's primary provider incarcerated would lead to more crime. Etc etc. Yes, we are lacking in high quality evidence, but that doesn't mean we can just pick which priors to put faith in.

Considering the size of these donations and the policy focus of many of these groups, it is useful to look at the campaigns these groups run. The issue is that political association for core EA institutions is not likely to be net positive unless very carefully managed and considered. It is also similarly the case that EA's should not support policy groups without clear rationale, express aims and an understanding that sponsorship can come with the reasonable assumption from general public, journalists, or future or current members, that EA is endorsing particular political views.

The Color of Change group, to which OP donated 50% of Color of Change's annual budget (at $2.5 million) for “increasing the salience of prosecutor and bail reform”, describes their work in a variety of mission statements. Some of these are very clearly not EA, such as:

• "Achieving meaningful diversity and inclusion behind the scenes in hollywood"
• "Ensuring Full and Fair Representation in the 2020 Census"
• "Protecting Net Neutrality"

Other mission statements are politically motivated to a degree which is simply unacceptable for a group receiving major funds from an EA org. Under a heading "Right Wing Politics and White Nationalism" - a clear elision of right wing politics with hard-line racism - there appears the mission statement:

• "Dismantling right-wing and white nationalist infrastructure/support"

This is a dishonest reading of politics, to be expected only of bad faith interpretations (akin to rendering left-wing politics as synonymous with Stalinism). Worse still than the rhetorical elision of RW and White Nationalism, is the mission itself - which taken at face value - commits the group to dismantling support and infrastructure for right-wing political beliefs. This is not an effective cause, and is - instead - a politically motivated one.

What, then, are the beliefs of the Color of Change? Under the title "Economic Justice", Color of Change advocates for "Building momentum for progressive tax, labor and education policies".

Their politics are all the clearer when looking to their current campaigns. These range from racial audits of Amazon, to advocating for the employment of a teacher fired for teaching critical race theory, to letters sent to prosecutors to stop "Anti-Trans Laws". Indeed, most of the campaigns appear to be letter writing. It is difficult to see how this organisation should be considered for $2.5 million of EA funds.

It should be clear at this point that Color of Change is a left-wing political pressure group designed to reduce right-wing infrastructure and support, and to smear right-wing beliefs (their political opponents) as synonymous with racism. What relevance has this to Effective Altruism?

These political beliefs are common across those funded. Mijente - given $255k in a grant to “support its work on criminal justice reform”  describes itself as believing "our people can't afford 4 more years of despair, fear and growing systematic criminalization. Our plan of attack is to win at the ballot box by mobilizing Latinx voters against Trump." Regardless of personal opinions on Trump, or his suitability for the office of President, this is not an EA cause area (arguments of x-risk prevention aside as this was apparently for a criminal justice reform grant). Furthermore, as a side note, from the Campaigns section of Mijente's website, it no longer appears they have an active criminal justice focus despite the $255k OP grant.

Similarly, LatinoJustice, given $500k from OP to encourage "Latinx activists to support criminal justice reform" does not advocate for criminal justice reform but rather states that  "The criminal justice system in the U.S. needs not only to be reformed; it needs to be dismantled. It needs to be de-structured. It needs to be decolonized". This is a political programme, disconnected to the original rationale for the OP grant.

Indeed, OP's reasoning itself was expressly political for funding of at least one of these groups. The $100k grant to ReFrame was described as a grant for the ReFrame Mentorship, which OP describes as "an intensive training and mentorship program in strategic communications for social justice movement organizers". This appears to be an expressly political rationale.

This is all to emphasise - not only as the post above does - that these groups were very likely far from effective uses of OP's grants, but that they were also given expressly to groups with very clear, particular politics. It is difficult to see how any of these groups met the bar of OP funding, but it is all the more concerning that these were essentially a pattern of political campaigning groups, with very particular (but coherent with one another) political beliefs, receiving significant OP funds. This appears more akin to political activism, and less to effective giving. It is surprising and disappointing that OP funds were apparently used in such an ineffective and expressly political direction. The above shifts my estimate strongly towards the Progressive Funders and Mind Killer hypotheses.

Thanks for putting this together! I think criticizing funders is quite valuable, and I commend your doing so. My main object-level thought here is I suspect much of the disagreement with the OP funding decision is based around the 1%-10% estimate of a $2B-$20B campaign leading to a 25%-75% decrease in incarceration. Since, per this article, incarceration rates in the U.S. have declined 20% (per person) between 2008 and 2019, your estimates here seem somewhat pessimistic to me. 

My guess is at the outset, OP would have predicted a different order of magnitude for both of those numbers (so I would have estimated something closer to $500M-$5B would produce a 5%-50% chance of a 25%-75% decrease), particularly since (as has been mentioned in other comments) it seemed a particularly tractable moment for criminal justice reform (since crime had declined for a long-time, there was seeming bipartisan support for reform, costs of incarceration were high, and U.S. was so far outside the global norm). By my quick read of the math, that change would put the numbers on par with your estimates for global health stuff. 

As someone who's worked on criminal justice reform (on a volunteer basis, not OP-funded though inspired by OP-funded work like Just Leadership's Close Rikers campaign), two features of the field that are striking to me are: 1. OP's original vision was to reduce incarceration while also reducing crime -- I don't think the "reduce crime" half ended up being a main goal of the work, which I think has probably made it less politically robust. 2. A lot of criminal justice reform work (including mine) has stemmed from the thesis that empowering the voices of current and formerly incarcerated people politically would be politically beneficial; I think in retrospect this may have been mistaken (or at least an incomplete hypothesis) and that, more broadly, left identity-politics based strategies of the 2010s have not been as politically (especially electorally) successful as I at least had hoped. 

On a meta-level, I think estimating impact of past grantmaking is very important, and EAs should do more of it. (I also think something along these lines could theoretically provide a scalable internship program for EA college students, since estimating impact teaches both cause prioritization skills and skills understanding the operations of organizations trying to achieve EA goals). 
 

Edit: I should clarify that I've received significant funding from OP (including from their US Policy side that covered their criminal justice work), so I'm naturally biased in its favor

Unlike poverty and disease, many of the harms of the criminal justice system are due to intentional cruelty. People are raped, beaten, and tortured every day in America's jails and prisons. There are smaller cruelties, too, like prohibiting detainees from seeing visitors in order to extort more money out of their families.

To most people, seeing people doing intentional evil (and even getting rich off it) seems viscerally worse than harm due to natural causes.

I think from a ruthless expected utility perspective, this probably is correct in the abstract, i.e. all else equal, murder is worse than equivalently painful accidental death. However I doubt taking it into account (and even being very generous about things like "illegible corrosion to the social fabric") would importantly change your conclusions about $/QALY in this case, because all else is not equal.

But, I think the distinction is probably worth making, as it's a major difference between criminal justice reform and the two baselines for comparison.

Excellent post. I have a strong prior that academic literature on criminology is biased, so I am more inclined than you to guess that consensus estimates for criminal justice reform not having net negative effects on crime are too optimistic. So my guess for second-order effects is that they make criminal justice reform even less valuable relative to other global health/wellness causes.

Putting that aside, I think one reason Open Phil might have been so favorably inclined to criminal justice reform was the bipartisan consensus that pursuing it was a good idea.  The 2010's were a uniquely good time to purse criminal justice reform (until ~ 2020, when increasing crime rates made criminal justice reform less bipartisan). 

Perhaps you could call this the "The  Hinge Hypothesis"-- during the years that Open Phil made large donations to criminal justice reform efforts, it was a uniquely good time to do so. I think this was a reasonable guess, though I don't think it brings the QALYs/$ to parity with other global health/wellness goals. 

Why did you take the mean $/QALY instead of mean QALY/$ (which expected value analysis would suggest)? When I do that I get $5000/QALY as the mean.

In this case, the "mistakes" is often a list of things like, "This specific organization was much worse than we thought.  The founders happened to have issues A, B, and C, which really hurt the organization's performance". 

Releasing such information publicly is a big pain and something our culture is not very well attuned to. If OP basically announced information like, "This small group we funded is terrible, in large part because their CEO, George, is very incompetent", that would be very unusual and there would likely be a large amount of resistance. I imagine OP would get a ton of heat for doing that. 

This issue is amplified by the fact that OP is a grantmaking organization much more than a grant recommendation organization. If it posted information publicly about an organization's competence, it's not clear which other orgs would actually use that information. 

My guess is that internally, people at OP admit a lot of mistakes. But making these mistakes public would create a lot of hazards, just because they contain a lot of information that specific organizations they work with would much prefer to be private. 

Hi Nuno, Great post. Really interesting. Well done!

I do a lot of estimating the impact of policy change in my work for Charity Entrepreneurship and I decided to have a look at your Squiggle models and oh boy, do your estimates of costs per policy change seem high to me.

DIFFERENCE IN ESTIMATES

CE have looked at case studies for 100s of different policy change campaigns in LMICs (see various reports here) and tend to find a decent chance of success, and estimates for the cost of policy change (LMIC and HIC) for a new EA charity tend to be around $1-1.5m, with like a 10-50% chance of success.

And existing EA charities seem even more effective than that. The APPG for Future Generations seemed to consistently drive a policy change for every ~$50k. LEEP  seems to have driven their first policy change for under ~$50k and seems on track to keep driving changes in that sort of ball park. 

Your costs are:

• For the Rikers policy change costs were $5m to $15m with a 7-50% chance of success. This is 1 order of magnitude above the CE estimates for new charities and 2 orders of above the track record of existing EA charities.
• For the prison reduction policy change costs were $2B to $20B with a 1-10% chance of success. This is 4 orders of magnitude above the CE estimates for new charities and 5 orders of magnitude above the track record of existing EA charities. This seems insanely expensive to me.

POSSIBLE REASONS 

The possible reasons for this difference are many: 

1. You think prison reform is a particularly untractable topic to make headway on.
2. Prison reform needs very very many small policy changes for a number of years to get anywhere, e.g. it needs changes in every state.
3. Policy work in the US is much much more expensive than policy work in the UK and LMICs.
4. CE estimates are too low as the data we have is affected by survivorship bias in case studies we can find.
5. Non-EA charities are many times less effective at driving change than EA run charities.

Even so 4-5 orders of magnitude is a big jump, so I thought it worth pointing out. I would love to hear your views on this. Especially if you think "2." is true as if so that affects some upcoming CEs decision making.

I do think it is possible you have significantly overestimated the costs here and that could be affecting your conclusions.

Edit: another way of looking at this in reading this your estimate of the effect of the Riker's change feels more indictive to me as to overall cost-effectiveness of good prison reform work, than your other estimate of spending $2-20bn to see 25-75% of prisoners released (which feels like made up numbers).  It seems plausible to me that a good campaign team can keep finding things as effective as the Rikers' closure, bail reform, etc, without significant diminishing marginal returns.

Good to hear it shifted your opinion!

> I can see why releasing information about personal incompetence for instance might be unusual in some cultures; I'm not sure why you can't build a culture where releasing such information is accepted.

I agree it's possible, but think it's a ton of work! Intense cultural change is really tough. 

Imagine an environment, for instance, where we had a public ledger of, for every single person:

• How much good and bad they've done.
• Their personal strengths and weaknesses.
• Their medical and personal issues. 

There would be many positives of having such a list. However, it would also create a lot of problems, especially in the short-term. It would be pretty radical.

I think it's good for people/orgs to experiment with radical honesty, though OP should probably be late to that. (You'd want to do experiments with smaller groups without so many commitments). 

The company Bridgewater is noted as being particularly honest. It seems to produce some good results, but also, it's an environment that seems terrible for most people. I recommend looking into reviews of it, it's pretty interesting. (Also, note that Elie and Holden both worked at Bridgewater). 

The baseline simple model is

estimateQALYs = initialPrisonPopulation * reductionInPrisonPopulation * badnessOfPrisonInQALYs * counterfactualAccelerationInYears * probabilityOfSuccess * counterfactualImpactOfGrant

cost = 2B to 20B

This seems to depend on:

Say that $2B to $20B, or 10x to 100x the amount that Open Philanthropy has already spent, would have a 1 to 10% chance of succeeding at that goal [5].

But shouldn't this be simplified to include fewer variables? In particular:

1. Why do we need cost as a variable on it's own?

cost is basically a choice variable. Presumably the more that is spent, the greater the probabilityOfSuccess. The uncertainty surrounds 'benefit per dollar spent'. But that 'slope of benefit in cost' is really only a single uncertainty, not two uncertainties. Wouldn't it be better to just pick a middle reasonable 'amount spent', perhaps the amount that seems ex-ante optimal?

2. Acceleration * reductionInPrisonPopulation * probabilityOfSuccess:

These seem likely to be highly (negatively?) correlated to each other, and positively correlated to cost. For a given expenditure, if we target a lower 'reduction in prison population', or we a slower rate-of-change, I expect a greater probabilityOfSuccess.

Would it make sense to instead think of something like 'reduction in total prison-years as a percent of current prison population?' Perhaps, feeding into this, some combinations of expenditure, acceleration, reduction percent, and prob(success) that jointly seem plausible?

Disclosure: I worked with Open Phil’s CJR team for ~4 months in 2020-2021 and was in touch with them for ~6 months before that. 

I’m very concerned by the way this post blends speculative personal attacks with legitimate cost effectiveness questions. 

Chloe and Jesse are competent and committed people working in a cause area that does not meet the 1000x threshold currently set by GiveWell top charities. If it were easy to cross that bar, these charities would not be the gold standard for neartermist, human-focused giving. Open Phil chose to bet on CJR as a cause area, conduct a search, and hire Chloe anyway. 

I genuinely believe policy- and politics-focused EAs could learn a lot from the CJR team’s movement building work. Their strengths in political coordination and movement strategy are underrepresented in EA. 

I bought the idea that we could synthesize knowledge from different fields and coordinate to solve the world’s most pressing problems. That won’t happen if we can’t respectfully engage with people who think or work differently from the community baseline.

We can’t significantly improve the world without asking hard questions. We can ask hard questions without dismissing others or assuming that difference implies inferiority. 

[I only got back on the forum to reply to this post.]

In July 2022, there still aren’t great forecasting systems that could deal with this problem. The closest might be Manifold Markets, which allows for the fast creation of different markets and the transfer of funds to charities, which gives some monetary value to their tokens. In any case, because setting up such a system might be laborious, one could instead just offer to set such a system up only upon request.

Manifold would be enthusiastic about setting up such a system for improving grant quality, through either internal or public prediction markets! Reach out ([email protected]) if you participate in any kind of grantmaking and would like to collaborate.

The primary blocker, in my view, is a lack of understanding on our team on how the grantmaking process operates - meaning we have less understanding of the the use case (and thus how to structure our product) than eg our normal consumer product. A few members of OpenPhil have previously spoken with us; we'd love to work more to understand how we could integrate.

A lesser blocker is the institutional lack of will to switch over to a mostly untested system. I think here "Prediction Markets in the Corporate Setting" is a little pessimistic wrt motives; my sense is that decisionmakers would happily delegate decisions, if the product felt "good enough" - so this kind of goes back to the above point.

My intuitions agree with you fwiw. I'm personally pretty skeptical of rational choice theory in criminology. I would guess criminology to be an unusually poor testing ground for microeconomics. Happy to be corrected otherwise with data.

I get the weak impression that worldview diversification (partially) started as an approximation to expected value, and ended up being more of a peace pact between different cause areas. This peace pact disincentivizes comparisons between giving in different cause areas, which then leads to getting their marginal values out of sync. 

Do you think there's an optimal 'exchange rate' between causes (eg. present vs future lives, animal vs human lives), and that we should just do our best to approximate it? 

Small suggestion: Could someone edit the post so that the hover footnotes work; makes it more readable? (Update: More importantly I'm not sure the footnote numbers are correct.)

Larger suggestion: This is a great post and provides a lot of methodological tools. I'd love to see it continue to be improved. E.g., detailed annotation, here or elsewhere, with the reasoning behind each element of the model.

It might help to turn the modeling part into a separate post linked to this one?

I have some 'point by point' questions and comments, particularly on the justification for the elements of the simple models. I don't want to add them all here because of the clutter, so I will add them as public https://hypothes.is/ notes. Unless anyone has a better suggestion for how to discuss this?

One component of this post was the quantified estimate of AMF's impact. My estimate was extremely rough. There is now a merely very rough estimate  here: <https://observablehq.com/@tanaerao/malaria-nets-analysis> , which looks as follows:

(I'm taking the estimate of lives saved, and then dividing them by life expectancy to get something resembling QALYs. I'm getting the life expectancy figures from Our World In Data, and then adding a bit of uncertainty).

Overall this update doesn't change this analysis much.

So I don't actually have to assume that criminals are rational actors; I can also assume that actions which are high expected value will spread through mimesis. See this short post: Unconscious Economics for an elaboration of this point.

But you are right that it smuggles many assumptions.

This is coming from my own understanding of what is politically feasible. I think that policies that make crime worth it in expectation are likely to be politically very unpopular and/or lead to more crime. So I think that restorative justice approaches would be stronger with a punitive component, even if they overall reduce that punitive component.

I speculate about why Open Philanthropy donated to criminal justice in the first place, and why it continued donating.

OPP, formerly GiveWell Labs, recommends grants. Good Ventures and other organizations that use these recommendations award the grants. The hypothesis that a funder was passionate about this cause and that OPP recommended effective solutions within this cause area rather than comparing the cause area to others seems prima facie plausible. However, I think that the influence is not as one-sided. OPP's leadership, represented by Holden Karnofsky, started considering (US) political advocacy early on. Since there are many ways to develop trusting relationships with political decisionmakers, following up on prominent conversations relating to justice and benevolence, criminal justice could have been actually a great idea.

You should also note that recently, OPP delegated criminal justice reform to Just Impact, led by Chloe Cockburn and Jesse Rothman,  the former OPP's grant advisors in this cause area. This grant should suffice for 3.5 years and perhaps the organization gains its own funding or solves the issue. So, perhaps this decision makes your argument outdated.

Maybe, are you criticizing that OPP commits to causes? There were some doubts^[1] about this in 2014. Currently, OPP conducts actually only shallow and medium-depth investigations, so the concern here can seem quite the opposite: where are the in-depth investigations?

I think that providing uncertain and perhaps flawed criticism is still worth it, in expectation

This should be the case if you are introducing new information in a way that is welcome by the decisionmakers and their informants. While you could rephrase some of your statements neutrally,^[2] the readers will probably go for content. You also agreed to not defame or cause rep loss of anyone involved,^[3] so you  wrote the entry with this intention. This could also imply that you would engage with any responses of these stakeholders constructively.

In the report, at the first glance, I would focus on the time on parole (p. 17). It can be that an increased time on parole increases criminality.^[4] Thus, increasing parole time may be a suboptimal solution.

The experiments' interpretations on pp. 22–26 seem horrifying, at the first glance.^[5] I am reading that deterrence, such as incarcerating someone for a day to discourage them, is bad while incapacitation (such as incarceration?) good.

On the second glance, I am also worried that the table shows the consistency of the studies with the report,^[6] not using the studies to inform conclusions. If this only shows that the writers had an extra challenge to fit external biases, that may be ok, but it can also be that the writers are biased toward incarceration and use the perceived readers' weaknesses and their appeal to authority. Thus, I suggest that the connections of the contributors to the profitable US prison system are overviewed.

On the third glance, the report simply analyzes various criminal justice factors that impact different measurable aspects of criminality, such as re-offences and their frequency/timing, but does not really discuss why people would commit crimes and how this could be prevented in ways other than those directly pertinent to criminal legislature and offenses.

In terms of the cost-effectiveness, reducing crime rate in the US can free significant capital,^[7] which can be used cost-effectively, including for deworming programs. When you consider that many US officials would love when they could attribute some crime reduction (people could elect them), then they could be attuned to certain budgetary recommendations from the crime reduction organization.

For example, if incarceration is reduced by 10% in the US,^[8] then, 600 million could be realistically annually spent on EA-related programs including deworming. ^[9]

The last one struck me as being both particularly bad and relatively easy to evaluate: A letter costs $2.5, about the same as deworming several kids at $0.35 to $0.97 per deworming treatment. But sending a letter intuitively seems significantly less impactful.

Have you considered that deworming may be a perpetual need while influencing a decision that motivates a sustainable systemic change a permanent solution? This could justify spending on advocacy, in general.

Furthermore, for change to happen, perhaps various media should be engaged. For example, letters should be complemented by local group support, by pro bono legal advisory that uses case studies to inform legal changes, by public awareness/electorate support that includes movies, by making sure that any new legislation is not discriminatory on the basis of race or otherwise, that judges are bought in, that any evidence-based appeal is complemented by US families' emotional advocacy, etc. So, it can be that grants awarded at any given time always complemented current activities to optimize for marginal cost-effectiveness.

the $415k salary of its president

Is it that in EA, the marginal cost-effectiveness^[10] is generally considered while factors such as the president's  pay are included in the analysis? For example, generally, a lawyer can be paid this salary. Legal manager who makes state and federal changes can be perceived to take a pay cut.

I apologize for I skipped your hypotheses.

It may be that at this point, the best course of action is reminding Just Impact to enter any conversations with the government with the awareness of favors return, the extent that reduced crime can increase budget, and the question on where they could see the greatest impact with some of this funding?^[11]

1. ^{^}

   It seems to me that the most common model in philanthropy ... is to have staff who specialize in a particular cause (for example, specializing in criminal justice policy).  ...  I think this model makes a great deal of sense, ... Yet I’m not convinced that this model is the right one for us. Depth comes at the price of breadth. ... I’ve been asking myself the question of whether there’s a way to be involved in many more causes at a much lower level of depth (October 30, 2014)

2. ^{^}

   For example, "and to what extent it was determined by other factors, such as the idiosyncratic preferences of Open Philanthropy's funders, human fallibility and slowness, paying too much to avoid social awkwardness," can be safely omitted since it is implied that if any reasoning is flawed, it is due to biases.

   Also, by stating "I end up uncertain about to what extent this was a sincere play based on considerations around the value of information and learning," you mean that the decision could have seemed valuable initially but with learning over time it showed as not so marginally cost-effective? Or that it should have been clear since the beginning that the org will learn little by focusing on criminal justice reform? Any hypothesis can be stated explicitly.

3. ^{^}

   5. D. vii. (6)

^{^}

This could show that any human bias was in favor of releasing criminals on parole and the report aimed to mitigate it.

^{^}

It may be because I am reading them wrong.

^{^}

In particular, see the assessment of Abrams (2012) (p. 23) that states "Strongly compatible after reanalysis." This could point out that an external standard was being met by the study interpretation.

^{^}

The societal cost of an inmate-year of incarceration is put at $92,000, with loss of liberty the largest item ($50,000) and the cost of incarceration next ($26,000 after netting out $5,000 in service transfers) (p. 135).

^{^}

From 2.3 million to 2.1 million.

^{^}

$0.23million\ast \$26,000/year\approx 6billion/year$. If the crime reduction organization influences 10%, additional 600 million can be spent.

^{^}

alongside with opportunity/timing, leverage, and other factors

^{^}

While deworming was used as an example, other expanded moral circle considerations can work. For example, I estimated, using references shared by a GiveDirectly employee, that targeting individuals at risk of criminal involvement who go through otherwise effective programs but turn to crime only because they do not have access to microfinance could multiply the cost-effectiveness many times. This also considers that the criminal activity of a single individual can reduce the wellbeing of entire communities while cash transfers targeted based on poverty rates only modest spillovers.

Show all footnotes