EA Forum Bot Site
EA Forum

Hide table of contents
Comment Permalink
Kyle Smith9
5
0

I think generally I agree with you, that people should be careful to not pull the trigger too early on closing down a project. However, I think in the general philanthropic landscape, organizations persist because of the lack of incentives to close them down, which is of course, inefficient. EA does a good job trying to correct this, but like with other areas of EA, it is possible that EA takes it "too far". 

I tend to think the people involved are most equipped to make this determination, and we have additional reason to trust their judgment because it likely goes against their self-interest to close a project down.

I think a related discussion could be had around funders making the decision to quit on projects too early, which is likely much more prevalent/an issue.

 

And as an aside - I am interested in this topic for a research project. I think doing some qualitative analysis (interviews?) with folks who have closed down projects would make for a fairly interesting research paper.

Reply
Showing 3 of 6 replies (Click to show all)
8
Ben Millwood🔸
Perhaps a minor point: the scout mindset encourages skepticism, but not deference. There's a big difference between deferring to a critique vs. listening to and agreeing with it. I think we should hesitate to describe people as deferring to others unless either (a) they say they are doing so or (b) we have some specific reason to think they can't be critically analysing the arguments for themselves.
1
Midtermist12
Thanks for the comment, Ben! You’re right that a perfectly applied scout mindset involves critically analyzing information and updating based on evidence, rather than deferring. In theory, someone applying the scout mindset would update the correct amount based on the fact that they have an interest in a certain outcome, without automatically yielding to critiques. However, in practice, I think there’s a tendency within EA to celebrate the relinquishing of positions, almost as a marker of intellectual humility or objectivity. This can create a culture where people may feel pressure to seem "scouty" by yielding more often than is optimal, even when the epistemological ecosystem might actually need them to advocate for the value of their intervention or program. In such cases, the desire to appear unbiased or intellectually humble could lead people to abandon or underplay their projects prematurely, which could be a loss for the broader system. It’s a subtle difference, but I think it’s worth considering how the scout mindset is applied in practice, especially when there’s a risk of overcorrecting in the direction of giving up rather than pushing for the potential value of one’s work.
titotal6
1
0

I think "scout mindset" vs "soldier mindset" in individuals is the wrong thing to be focusing on in general (. You will never succeed in making individuals perfectly unbiased. In science, plenty of people with "soldier mindset" do great work and make great discoveries. 

What matters is that the system as a whole is epistemologically healthy and has mechanisms to successfully counteract people's biases. A "soldier" in science is still meant to be honest and argue for their views with evidence and experimentation, and other scientists are incentivized to... (read more)

Reply
See in context
by Midtermist12
2 min read 17

66

CommunityOpinion
Frontpage

Reconsidering the Celebration of Project Cancellations: Have We Updated Too Far?

Epistemic status: Low certainty. These are tentative thoughts, and I’m open to alternative perspectives.

Posting from an alt account.

The Effective Altruism community has made significant strides in recognizing the importance of quitting projects that don’t deliver short-term results, which helps counteract the sunk cost fallacy and promotes the efficient use of resources. This mindset, in many cases, is a positive development. However, I wonder if we’ve over-updated in this direction. Recent posts about project cancellations, like the one regarding the Center for Effective Aid Policy (CEAP), have received considerable attention—CEAP’s closure post garnered 538 karma, for instance. While I don’t have a strong opinion on whether it was prudent to shutter CEAP, I am concerned that its closure, and the community's reaction to it, vibes in a direction where initial setbacks are seen as definitive reasons to quit, even when there might still be significant long-term potential.

From an outside perspective, it seemed that CEAP was building valuable relationships and developing expertise in a complex field—global aid policy—where results may take years to materialize. Yet, the organization was closed, seemingly because it wasn’t achieving short-term success. This raises a broader concern: are we in danger of quitting too early when projects encounter early challenges, rather than giving them the time they need to realize high expected value (EV) outcomes? There’s a tension here between sticking with projects that have a low probability of short-term success but could yield immense value in the long run, and the temptation to cut losses when things aren’t immediately working out.

High-EV projects often have low-impact modal outcomes, especially in the early stages. It’s entirely possible that a project with a 20% chance of success could still be worth pursuing if the potential upside is transformative. However, these projects can look like failures early on, and if we’re too quick to celebrate quitting, we may miss out on rare but important successes. This is particularly relevant in fields like AI safety, global aid policy, or other high-risk, high-reward areas, where expertise and relationships are slow to develop but crucial for long-term impact.

At the same time, it’s essential not to continue investing in clearly failing projects just because they might turn around. The ability to pivot is important, and I don’t want to downplay that. But I wonder if, as a community, we are at risk of overupdating based on short-term signals. Novel and complex projects often need more time to bear fruit, and shutting them down prematurely could mean forfeiting potentially transformative outcomes.

I don’t have an easy answer here, but it might be valuable to explore frameworks that help us better balance the tension between short-term setbacks and long-term EV. How can we better distinguish between projects that genuinely need to be ended and those that just need more time? Are there ways we can improve our evaluations to avoid missing out on projects with high potential because of an overemphasis on early performance metrics?

I’d love to hear thoughts from others working on long-term, high-risk projects—how do you manage this tension between the need to pivot and the potential upside of sticking with a challenging project?

66

0
3
1
1

Reactions

0
3
1
1

More posts like this

Comments17
Sorted by
New & upvoted
Click to highlight new comments since:
Joey🔸38
7
1
2

Personally, I do not think we have swung too far on this. Even in the AIM/CE cohorts (from which both of the recent public shutdown posts came), I still think people err too much on the side of keeping projects running when they are not performing.

I think how to think about shutting down projects comes down a lot to counterfactuals. It's not that a project could not work or definitely will fail; it's about how good a marginal bet it is. Every dollar and high-talent hour going into the project could, in theory, be going somewhere else—often somewhere with pretty high impact. When I think about a cohort of 6 charities founded out of AIM, after 2 years, I would expect shutting down the bottom 2 and having the staff/funding join the top 2 would result in far more impact for the world. That would result in a ~33% shutdown rate, which is way above what happens in the charity world and about double what happens in AIM right now. 

Reply
NickLaing10
2
0
1

Thanks for the input - you are certainly in a great position to judge this and I love this framework. I also agree shutting down earlier can be far easier, because if you are 3 years in with increasing funding, while realizing you aren't up to much it could be difficult socially and status wise to pull out at that point.

A slight caveat I would have is that it might depend on where follow-on funding was coming from. In a closed system with limited EA resources, then yes the only important Factor might be how good a marginal bet the org is.

If an organisation with a lower marginal chance of of success managed to get decent funding from NON- EA sources then it might be worth continuing with a worse bet. For example if maternal health initiative got a big foundation grant or if the recently closed policy org for funded through a thinktank or government grant. One issue here is that often orgs need 3-4 years of operation before people will grant them much money (outside of EA ecosystem)

But I would imagine getting big outside funding isn't super common for marginal bet CE orgs, if it has happened at all. I'm also not sure to what extent the source of funding should change our bet (this has been discussed on the forum but would love to see some math done - mine isn't good enough)

Another even less important caveat might be if talented people were so passionate about what they their initial org did, that they might not be able to easily maximize on their talent in another role.

I would be interested to get a vague idea of what CE founders usually end up doing after closing their orgs. Have many joined other higher performing CE orgs? This might be sensitive info though.

From the little I know of CE peeps I would imagine they are mostly doing pretty great stuff.

Reply
Joey🔸8
2
0

Joining another CE charity is pretty common, as is working at EA meta-orgs (AIM, GiveWell, etc.). I would guess that around 75% do something most people would regard as very high impact.

Agree regarding external marginal funding, but I would say, at least in AIM's case, this correlates with early-stage success.

Reply
Midtermist125
0
0

Thanks for your thoughtful response, Joey. I had originally approached this issue more from the perspective of founders and leaders being less "soldiery" for their projects, but I see that the funder’s viewpoint, especially regarding the counterfactual uses of money, is quite different and valid.

One key difference is that the counterfactual reallocation of talent from failed AIM charities may not be as impactful as the reallocation of funds by AIM-adjacent funders. As you mentioned, many people who worked at a failed AIM charity are likely to join another CE charity or work for an EA meta org, but these roles are in high demand and often attract top-tier talent regardless. It’s not clear that the movement of talent between these organizations would have as large an impact as reallocating funds to more successful initiatives.

This is where the dynamic between founders and funders diverges. From the leader’s perspective, it might make more sense to continue to pivot, seek out other funding sources, and keep the project alive, particularly if they still believe in the long-term potential. On the other hand, from the funder’s perspective, cutting their losses and focusing on capitalizing on wins may provide a much clearer path to maximizing impact. It seems that the optimal decisions for founders and funders could diverge, depending on their roles in the ecosystem.

I appreciate your insight into how marginal bets play into these decisions and how AIM's cohort-based structure could actually benefit from higher shutdown rates. It seems like there’s a balance between empowering founders to pursue potential breakthroughs while ensuring funders can make optimal reallocation decisions for broader impact.

Reply
Kyle Smith9
5
0

I think generally I agree with you, that people should be careful to not pull the trigger too early on closing down a project. However, I think in the general philanthropic landscape, organizations persist because of the lack of incentives to close them down, which is of course, inefficient. EA does a good job trying to correct this, but like with other areas of EA, it is possible that EA takes it "too far". 

I tend to think the people involved are most equipped to make this determination, and we have additional reason to trust their judgment because it likely goes against their self-interest to close a project down.

I think a related discussion could be had around funders making the decision to quit on projects too early, which is likely much more prevalent/an issue.

 

And as an aside - I am interested in this topic for a research project. I think doing some qualitative analysis (interviews?) with folks who have closed down projects would make for a fairly interesting research paper.

Reply
Midtermist128
1
1

I agree that funders can sometimes quit too early on promising projects. Founders and those directly involved often play a crucial role in pushing through early setbacks. However, I worry that in the EA community, there's an overemphasis on the “scout” mindset—being skeptical of one’s own work and too quick to defer to critiques from others.

I should acknowledge that I may be strawmanning the scout mindset a bit here. In its ideal form, the scout mindset encourages intellectual humility and a willingness to see things as they are, which is clearly valuable. The practical application, however, often leads people to focus on being extra vigilant about potential biases in projects they’re closely involved with. While this caution is important, I think there’s a risk that it prevents people from taking the necessary "inside view" to successfully lead new or risky projects.

Many successful endeavors require a degree of tenacity and advocacy from the inside—the willingness to believe in a project’s potential and push forward despite early doubts. In systems like the legal world or even competitive markets, having “soldiers” who advocate strongly for their side often leads to better overall outcomes. Within EA, founders and project leaders can play this same balancing role by fighting for their initiatives when external doubts arise. Without this advocacy, projects with high long-term potential might be abandoned too soon.

The soldier mindset may be particularly important for those leading high-risk, high-reward projects. Individuals like Elon Musk at SpaceX or Jeff Bezos at Amazon had to persist through early failures because they believed in their projects when others didn’t. Their success depended on taking the inside view and pushing forward despite setbacks. If founders in EA are too quick to adopt the scout mindset, we might lose out on similarly promising projects.

In short, while the scout mindset has its place, I think we need a balance. Founders and those deeply involved in a project should serve as a necessary counterweight to broader skepticism. A healthy epistemic system requires both scouts and soldiers, and right now, I think EA might benefit from more of the latter in some contexts. I’d also be interested in case studies of people who have quit projects to better understand these dynamics. Your research idea sounds like a great way to explore whether we’re underestimating the value of persistence in high-risk, high-reward initiatives.

Reply
Ben Millwood🔸8
2
0

However, I worry that in the EA community, there's an overemphasis on the “scout” mindset—being skeptical of one’s own work and too quick to defer to critiques from others.

Perhaps a minor point: the scout mindset encourages skepticism, but not deference. There's a big difference between deferring to a critique vs. listening to and agreeing with it. I think we should hesitate to describe people as deferring to others unless either (a) they say they are doing so or (b) we have some specific reason to think they can't be critically analysing the arguments for themselves.

Reply
Midtermist121
0
0

Thanks for the comment, Ben! You’re right that a perfectly applied scout mindset involves critically analyzing information and updating based on evidence, rather than deferring. In theory, someone applying the scout mindset would update the correct amount based on the fact that they have an interest in a certain outcome, without automatically yielding to critiques. However, in practice, I think there’s a tendency within EA to celebrate the relinquishing of positions, almost as a marker of intellectual humility or objectivity.

This can create a culture where people may feel pressure to seem "scouty" by yielding more often than is optimal, even when the epistemological ecosystem might actually need them to advocate for the value of their intervention or program. In such cases, the desire to appear unbiased or intellectually humble could lead people to abandon or underplay their projects prematurely, which could be a loss for the broader system.

It’s a subtle difference, but I think it’s worth considering how the scout mindset is applied in practice, especially when there’s a risk of overcorrecting in the direction of giving up rather than pushing for the potential value of one’s work.

Reply
titotal6
1
0

I think "scout mindset" vs "soldier mindset" in individuals is the wrong thing to be focusing on in general (. You will never succeed in making individuals perfectly unbiased. In science, plenty of people with "soldier mindset" do great work and make great discoveries. 

What matters is that the system as a whole is epistemologically healthy and has mechanisms to successfully counteract people's biases. A "soldier" in science is still meant to be honest and argue for their views with evidence and experimentation, and other scientists are incentivized to probe their arguments for weaknesses. 

A culture where less people quit of their own accord, but more people are successfully pressured to leave due to high levels of skeptical scrutiny might be superior. 

Reply
NickLaing7
3
0

I would go way further than this. Most (95 percent plus) charity organizations are more concerned with continuing than anything else, and closure is only forced by money running out or a big personality who drives it leaving. Effectiveness, let alone cost-effectiveness is very rarely a consideration.

Here in Gulu, Northern UgAnda I can't think of 1 org in 10 years that closed because they thought they weren't doing enough good. Our of hundreds operating.

Reply
Jason2
0
0

I think a related discussion could be had around funders making the decision to quit on projects too early, which is likely much more prevalent/an issue.

The lack of incentives to write posts criticizing one's former funders for pulling the plug early may be a challenge, though. After all, one may be looking to them for the next project. And writing such a post may not generate the positive community feeling that writing an auto-shutdown postmortem does.

Reply
John Salter4
1
0

I've had similar worries. Most extremely impactful projects look destined to fail half a dozen times before they blow up.

The right balance is hard to strike between incentivising determination and encouraging people to waste less money on ideas that prove weak.

Reply
titotal4
0
0

I've had similar worries. Most extremely impactful projects look destined to fail half a dozen times before they blow up.

Do you have any examples of this? 

Reply
John Salter7
1
0

Here's examples from six of list of top ten companies by market cap

Apple is worth $3 trillion despite being on the verge of bankruptcy in the mid-nineties. 
Google is now worth 1.9 trillion. The founders tried and failed to sell it for 1 million.
Amazon's stock price dropped 90% during dot com crash
Nvidia, recently world's most valuable business, had to lay off half its staff in 1997 and try to win a market with ~100 other startups all competing for same prize
Elon Musk: "I thought SpaceX and Tesla both had >90% chance of failure". He was sleeping on his friends coaches to avoid paying rent at that time.
Facebook's rise was so tumultuous they made a movie about it. Now worth 1.3 trillion.
Warren buffet regretted buying Berkhire Hathway, and almost sold it. Now worth 744 billion.

Talking about EA more specifically

~10 founders have spilled the details of their journeys to me. ~70% felt hopeless at least once. There's been at least four or five times I've been close to quitting. I had to go into credit card debt to finance our charity. I've volunteered full-time for >4 years to keep the costs lower, working evenings to pay rent. Things are now looking a lot better e.g. our funders doubled our budget last year and we're now successfully treating ~4-5x more people than this time last year.

Reply
Ian Turner3
0
0

The 2007 GiveWell marketing fiasco arguably came close to ending the project.

Reply
JamesN2
1
0

I tend to agree with you, though would rather people were more on the “close early” side of the coin than the “hold out”. Simply because the sunk cost fallacy and confirmation bias in your own idea is incredibly strong and I see no compelling reason for how current funders in the EA space help counteract these (beyond maybe being aware of them more than the average funder).

In an ideal system the funders should be driving most of these decisions by requiring clear milestones and evaluation processes for who they fund. If the funder did this they would be able to identify predictive signals of success and help avoid early or late closures (e.g. “we see on average policy advocacy groups that have been successful have met fewer/more comparable milestones and recommend continued/stopping funding”). This can still allow the organisation to pitch for why they are outside of the average, but the funder should be in the best position to know what is signalling success and what isn’t.

Unfortunately I don’t see such a system and I fear the incentives aren’t aligned in the EA ecosystem to create it. The organisations getting funded enjoy the looser, less funder involved setup. And funders de-risk their reputational risk by not properly evaluating what is working and why, and they can continue funding projects they are personally interested in but have questionable causal impact chains. *noting I think EA GHD has much less of this issue mainly because funders anchor on GiveWell assessments which is to a large degree delivering the mechanism I outline above.

Reply
Stephen Robcraft1
0
0

With my 'Good Governance' hat on, I think there's something in this. 

I personally celebrate that there is a culture of considering often and deeply whether an organisation/project should continue. Outside of EA, I've worked >50 mission-driven teams and have rarely encountered individuals, let along whole organisations, that are willing to ask this question of themselves. In my consulting work, I've recommended that organisations shut down or close a programme maybe a dozen or so times and can only think of one that took this recommendation seriously.

While I think it's appropriate to celebrate that there are cancellations/closures, I worry that how this happens is less than ideal. I need to engage with this a bit more deeply to order my thoughts (and will do if anyone is interested in discussing this!) but my take is that there's room to improve with regards to:

1. Defining who closes organisations - I think this is quite clearly a board responsibility (maybe the fundamental reason why good boards should exist) and think there should be more separation between a founder/exec team from this kind of decision than I understand to be the case

2. Defining when and why organisations should close - This post asks whether early-stage projects are closing too early but I also wonder if there are other questions to consider here

3. Articulating alternatives to closing/cancelling - Could projects continue in a different form? Might other people want to and be able to take on the work? Could the project continue to do good outside the EA community (with different funders, principles, values)?

Reply
More from Midtermist12
View more
Curated and popular this week
Relevant opportunities
View more