re: "I think Open AI are reading too much into the data", to be perfectly honest I don't think they're reading into anything, I just interpreted it as marketing and hence dismissed it as evidence pertaining to AI progress. I'm not even being cynical, I've just worked in big corporate marketing departments for many years.

Thanks Rasool. This is sad to read.

Yes definitely helpful, both for my own thinking and to be able to have something to point others to. With the caveat that learning from success stories requires some sort of survivorship bias adjustment, I think nuts-and-bolts writeups of technical policy reform success stories (as opposed to more high-level guides) are valuable and undersupplied, so if you ever get round to the more detailed writeup that would be great. 

Strong-upvoted, thank you for the detailed writeup and BOTEC. I currently work in global health policy and your takeaways seem broadly right to me, I would include your reflections on the work itself in that section for folks who jump straight to that.

You handwaved away the internal 1.5 FTE for 6 months * overhead costs of conducting the work, but I'd be remiss not to mention that ~$100M in benefits from your Guesstimate vs order-of-mag $100k internal costs is a 1,000x ROI, within spitting distance of Open Phil's funding bar.

The most surprising thing I got from your writeup was (emphasis mine)

I looked for some statistics on how many rule change requests end up resulting in rule changes to get a sense of the success rate. 548 rule change requests have been initiated. It’s hard to say how many of them were successful without spending more time, because AEMC doesn’t publish statistics on this, and some of the requests get merged into others. Also, not all rule change requests survive the consultation process intact. It could end up being quite different to what was originally proposed. A little over half of rule changes that have been initiated have commenced (incorporated in the live rules). Overall, I’m reminded of a hits-based giving approach.

Over half! I would've ballparked this at 10% give or take, so it's good to reorient my gut-feel on hit rate to empirics. 

I would indeed be keen for you to write more about what you specifically did during the project as per your offer, always good to have more case studies that go into the nuts and bolts for practitioners and folks wanting to test for fit in policy careers.

I think it's more so the latter. Scott Alexander's ACX Grants gives to a ton of systemic change-flavored stuff (see here), Charity Entrepreneurship / AIM has launched a fair number of orgs that aren't RCT-based direct delivery charities (policy, effective giving, evaluators, etc), etc to say nothing of longtermist and meta cause areas for which strict experimental control isn't possible at all.

I'm skeptical of the blindspot claim, e.g. there's a decade-old 80K article listing the wide variety of efforts by EAs working on systemic change even then.

There's an interesting variant of this if you generalise "small charities" to "small giving opportunities", cf. Nadia Asparouhova's Helium Grants. This doesn't so much address your "against" point but sidesteps it by focusing on individuals not orgs, which from having spoken with some meta-funders is standard. 

Open Phil / CoeffG has a fund focusing on air quality, so I suppose you're saying "no it's still relatively neglected, more resources should be allocated"?

Any interesting "adversarial" actions / interventions in GHD in your view?

From the article The Environmentalists Making Forest Fires Worse I learned about Denise Boggs, who seems to be a case study in what very effective altruistic execution looks like when it isn't grounded in evidence. First, adverse impact:

From 2010 to 2024, California endured more than 8000 wildfires a year that burned on average just under 1.1 million acres of land. In total, over that period, more than 16 million acres burned in California—about 16 percent of the total landmass of the state. These fires burned through endangered species’ habitats, killing millions of wild animals, and threatening endangerment of species like the long-toed salamander and dozens more.

The smoke produced by California’s fires also costs lives. Between 2008 and 2018, PM2.5 pollution from wildfires was responsible for between 52,480 and 55,710 premature deaths in California alone. But the smoke from California’s fires does not stop at the state border. In 2020, about 28,000 premature deaths were attributable to wildfire smoke across the United States, with the majority occurring in western States.

The cause:

To combat wildfires before they happen, the U.S. Forest Service (USFS), county conservation boards, and other stakeholders implement fuels reduction projects that can reduce excess dry wood and shrubs, and clear smaller vegetation that allows fires to grow faster and reach into the canopy of forests. Fuels reduction approaches like mechanical thinning and prescribed burns have proven to be effective mitigation strategies to reduce the damage from wildfires on ecosystems and to help firefighters stop fires. 

Yet, a small but loud environmentalist minority opposes fuels reduction, instead claiming that California’s forests must be left untouched. They use outdated environmental laws like the National Environmental Policy Act (NEPA), Endangered Species Act, Federal Land Policy and Management Act, and National Forest Management Act in courts to delay, and sometimes cancel, projects that would mitigate the wildfires that destroy the ecosystems they claim to protect, and threaten tens of thousands of lives.

During the period that about a sixth of California’s forests were going up in flames, one single group was busy suing the USFS 24 times. That group, Conservation Congress, was responsible for just under two fifths of the USFS’s NEPA-related lawsuits that were decided in federal circuit or appeals courts in California from 2010 to 2024, and spent $2 million on those lawsuits and 5 more in other western States.

Conservation Congress isn't just one group, it's one person, Denise Boggs:

What’s most remarkable about Conservation Congress is not their ability to single-handedly hamstring dozens of USFS projects, but that they are, in fact, single-handed: the organization effectively is just one person: Denise Boggs of Great Falls, Montana.

A long-time forest activist and veteran of the California “timber wars,” Boggs has taken the USFS to the mat on countless occasions, often coming up the loser. But she is determined. Boggs believes that the USFS, in bed with logging companies, is using fuels reduction programs and other fire management to create “loopholes big enough to drive logging trucks through.” It is Bogg’s mission to close those loopholes and save the northern spotted owl.

There are other such special interest groups, although they're much larger; the article names the Center for Biological Diversity (100+ staff) and the Sierra Club (700+!). In the case of Boggs, her mind-blowing cost-effectiveness, if not sign/direction, seems to be a testament to sheer tenacity + blinding focus + a theory of change that leverages US environmental law (NEPA litigation in particular) letting small groups delay or stop big federal projects with lots of local support (structural pendulum overswing from Jane Jacobs vs Bob Moses?):

The Center for Biological Diversity and the Sierra Club are national non-profits that advocate for and act on behalf of a specific ideological framework that places the abstract entity of “the environment” over all else. 

While the Sierra Club has a much longer history—the organization was founded in 1892 by legendary environmentalist and conservationist John Muir—the rest of these non-profits are relatively new projects. CBD was founded in the 1990s by a group of northern spotted owl biologists who sought to protect the species at all costs. Conservation Congress, Alliance for the Wild Rockies, and Native Ecosystems Council are all post-turn-of-the-21st-century organizations founded by activists who grew up—ideologically speaking—during the environmental protests of the late 20th century. 

With the exception of the Sierra Club, which has grown beyond just conservation and preservation, these groups are single-issue groups—protect endangered species, no matter their niche, or lack thereof, and ignore everything else.

(In different circles, there is another name for this kind of thing.)

I really like the outdoors, some of my most cherished youthful memories were the multiday hikes across the very SoCal wilderness Boggs' Conservation Congress aims to preserve. But man, this ain't it. 52,480 to 55,710 premature deaths between 2008-18 due to PM2.5 pollution from wildfires in California alone and 28,000 premature deaths in 2020 alone(!) attributable to wildfire smoke across the United States and 16%(!) of California's entire landmass ravaged by wildfires killing millions of wild animals and causing hundreds of billions of dollars in damage ain't it. Gargantuan impact and cost-effectiveness but negative sign. 

Evidence grounding matters a lot to ensure positive impact sign, and sometimes naively counterintuitive interventions like "regular targeted forest-thinning and burning are better than leaving forests untouched" are actually correct, and sometimes the evidence says that interventions need to change with the times (especially policy prescriptions) so you shouldn't get wedded to them let alone build ideologies and tie identities to them. Evidence grounding may not be enough however: your frame/prior for interpreting the evidence matters a lot too. The article argues that Boggs is trapped in a bad prior:

Ultimately, groups like Conservation Congress don’t really claim that fire prevention through mechanical thinning and prescribed burns—or other fuels management practices—are not effective. Boggs simply argues that any and all U.S. Forest Service projects are secretly logging projects. Fire management, to these groups, is a cover-up for a corrupt federal agency in bed with timber companies. To save old-growth forests, they must stop all projects all the time, and forests must go untouched.