Hide table of contents

Comment Permalink

Cool work! Props that you allow for people using their own discount rate, as your first footnote is a good point.

I think that for transparency and ease of reader understanding, you ought to link the 80K's article on grantmaking for most-pressing problems. Similarly, linking info on Squiggle would be good.

Also, best to clarify that this review is only about "working at a government agency that funds relevant research", and that this is only one out of their 3 mentioned highly-effective careers related to grantmaking (at the bottom), and that the other two would need a different analysis.

I also think that if the intent is to advise on careers, you need to do some analysis of the team of E-ARPA. Variables that come to mind are: size of team, how many of each role, how senior each person seems (for thoughts on how soon a person can get hired in such a role, and maybe even, in the case of junior employees, where they could go from there career-capital-wise), and a rough guesstimate of how much each role contributed to the overall annual grantmaking decisions of E-ARPA.

Also a minor wording note.. You say:
"We chose ARPA-E because other ARPA agencies are explicitly called out by 80,000 Hours profile of grantmaking (i.e., DARPA, IARPA)"

"Called out" has negative connotations, so I'd probably say "mentioned", "referred to", or "brought up" instead. That terminology confused me--I thought you were saying you only chose ARPA-E because the others had been essentially ruled out. I was a bit aghast thinking you'd chosen an example in a category where others had already been shown to be moot or something , and that's why I dug for and read the original 80K piece >.>

Phew, sorry that was so much seemingly-critical feedback, but to clarify I (not a researcher or data scientist) think what you did do is good and I'm happy you reviewed this career path, which tends to, I think, be unfortunately skipped in many career discussions. I strong upvoted the post.

See in context

Estimating the Average Impact of an ARPA-E Grantmaker

by charrin

Dec 1 20223 min read 8

22

ForecastingGrantmakingSquiggleClimate changeFermi estimateMeasuring and comparing valuePrizes and contests

Frontpage

Estimating the Average Impact of an ARPA-E Grantmaker

//BENEFIT-COST CALCS

8 comments

As part of the “$5k challenge to quantify the impact of 80,000 hours' top career paths,” we aim to estimate the average impact of a grantmaker focused on pressing world problems. To ground our inputs, we chose to focus specifically on one grantmaking organization -- ARPA-E. We chose ARPA-E because other ARPA agencies are explicitly called out by 80,000 Hours profile of grantmaking (i.e., DARPA, IARPA) and because ARPA-E has usable cost and benefit data.

The calculations below -- which can be plugged into Squiggle -- estimate the average impact of an ARPA-E grantmaker. Our inputs were primarily pulled from ARPA-E’s website. We then make a series of intermediate calculations to estimate the net present costs and benefits. Finally, we combine the cost and benefit estimates to make a benefit-to-cost ratio.

Our model has significant limitations, some of which are discussed in the “comments” below.

Our average impact estimate for ARPA-E grantmakers is a ~6 to 1 average benefit-to-cost ratio and ~$82 million average net benefit. These figures can be compared to other careers' estimated BCRs and net benefits to understand which career choice has higher estimated average impact. Benefits and costs are expressed in net present value of US dollars. Our calculations assume a 7% discount rate, but, if desired, users can adjust the discount rate in the code below.

//INPUTS

//Most inputs are expressed as point estimates instead of distributions since we pulled this data directly from ARPA-E’s website and other DOE resources.

arpae_budget_total = 3270000000 // from ARPA-E website

arpae_managers_annual = 17 // from ARPA-E website

projects_total = 1415 // from ARPA-E website

years_of_arpae = 13 // ARPA-E started in 2009

valuations = 21800000000 // value from ARPA-E website

discount_rate = 0.07 // Department of Energy discount rate

avg_proj_length = 1 to 3 // from ARPA-E FAQs

//PROJECT NUMBER CALCS

projects = projects_total/years_of_arpae // assumes equal number of projects per year

projects_per_manager = projects/arpae_managers_annual//= 6.4
projects_per_manager_dist = truncateLeft(normal(projects_per_manager,pow(6.4,1/2)), 1)

//COST CALCS

budget_per_project = arpae_budget_total/projects_total //= 2,310,954

budget_per_project_dist = .5M to 6M // Log normal with 2.3 mean + wikipedia estimates https://en.wikipedia.org/wiki/ARPA-E

npv_budget_per_project = (1+discount_rate)^avg_proj_length*budget_per_project //= 2,547,827

npv_budget_per_manager = projects_per_manager * npv_budget_per_project //= 6.4 * 2,547,827 = 16,306,091

//BENEFITS CALCS

benefit_per_project = valuations/projects_total //= 15,406,360

benefit_per_manager = benefit_per_project * projects_per_manager

___________________________________________________

We make two key assumptions here.

First, we assume all benefits are counterfactual since ARPA-E is meant to fund “white space” -- i.e., disruptive technology that is not mature enough and/or on too long of a time-horizon for VC. The implication of ARPA-E funding "white space," is that ARPA-E funds projects that counterfactually would not have been funded. Support for this claim can be found here and here. However, in reality, we think many of these technologies would be developed at some point -- ARPA-E simply speeds up the development. Thus, we believe that, all else equal, this first assumption leads to an overestimation of benefits. Future work could take advantage of other DOE BCR calculations that accounted for this consideration (links here, here, here, and here).
Secondly, we assume that all benefits -- including health benefits, reduction of monetary costs associated with climate change, reduction of existential risk associated with climate change, and spillover benefits into other industries -- are incorporated into the market valuation. We believe that markets likely price in the aforementioned externalities (e.g., health and enviro benefits) (support for this claim here). Furthermore, the valuations listed on ARPA-E’s site are from after the Inflation Reduction Act passed, which, itself, internalized a significant chunk of emission reductions associated with US tech. For these reasons, we believe it’s reasonable to assume a significant amount of external benefits have been internalized into markets, but, perhaps, not all benefits. Thus, we believe that, all else equal, this assumption leads to an underestimation of benefits.

//BENEFIT-COST CALCS

bcr_per_project = benefit_per_project/npv_budget_per_project = 98,600,704/16,306,091 = 6.04

net_benefits = benefit_per_project - npv_budget_per_project = 98,600,704 - 16,306,091 = 82,294,613

22 Reactions

Mentioned in

62Winners of the Squiggle Experimentation and 80,000 Hours Quantification Challenges

More posts like this

Comments8

Sorted by

New & upvoted

Click to highlight new comments since: Today at 7:54 PM

jackvaDec 1 20225

"2. Secondly, we assume that all benefits -- including health benefits, reduction of monetary costs associated with climate change, reduction of existential risk associated with climate change, and spillover benefits into other industries -- are incorporated into the market valuation. We believe that markets likely price in the aforementioned externalities (e.g., health and enviro benefits) (support for this claim here). Furthermore, the valuations listed on ARPA-E’s site are from after the Inflation Reduction Act passed, which, itself, internalized a significant chunk of US emissions. For these reasons, we believe it’s reasonable to assume a significant amount of external benefits have been internalized into markets, but, perhaps, not all benefits. Thus, we believe that, all else equal, this assumption leads to an underestimation of benefits. "

Thanks for this article! The above assumption feels quite wrong to me and, as such, I expect this makes your estimate a vast underestimate everything else being equal.

You seem to assume that climate risk and other externalities are priced in market valuations. Even if that were true for the US (which seems unlikely) it would certainly not be true for most places in the world that have very little in terms of pricing energy-related externalities. Given the role of US in the global energy innovation system, it seems reasonable to assume that the benefits that ARPA-E creates are way larger than market valuations, at least if market valuations are -- as you suggest -- reflective of expected policy returns.

ethaiDec 6 20222

Yeah, strong agree with this. [I used to work in VC and frequently diligenced ARPA-E grantees.] I don't think the cited study supports the claim that all externalities are priced in in the US, let alone globally.

I would also guess that the valuation of the 26 exited companies is an underestimation of overall impact for other reasons - top of mind: impact of non-exited companies, learning benefit to the field of a company that "fails" and exits at a very low valuation.

@charrin thanks for writing this, as a below commenter said it's nice to see an EA-style investigation of a potentially impactful career path outside the community!

charrinDec 12 20221

Appreciate the feedback. We definitely agree that not all benefits are captured in market valuation and that this leads to an underestimate. While working on this, we initially tried creating an "external benefits multiplier,” but ultimately couldn’t come up with a particularly sound way of estimating this multiplier prior to the submission deadline. Regardless, I think we were not focused enough on the fact that the US plays this significant role in the global energy innovation system, so thanks for bringing that up too. In retrospect, we should’ve just added something as a placeholder and then made updates later.

The general framework we could use to create this multiplier is to calculate the ratio of a standard carbon price recommended by environmental economists (e.g., ~$51 - 125 per ton) over an estimate of the “implicit US carbon price” (i.e., standard carbon price/implicit US carbon price). The implicit US carbon price could be derived by calculating an emissions-weighted carbon price associated with RGGI and the California cap-and-trade market, as well as the implied dollar paid per emission abated through current US subsidies for carbon reduction technologies. We could rely on work from UChicago’s EPIC to help guess the implied dollar paid per emission abated through US subsidies. However, as you can see in the aforementioned UChicago paper, US policies likely pay way more than $125/ton in some instances (e.g., with the weatherization assistance program) and miss cheap emissions reduction opportunities in other instances (e.g., nature-based solutions). This could make the “implicit US carbon price” look artificially high. Regardless, the implied emissions price may still be relatively low (e.g., <$30/ton). Thus, the external benefits multiplier might be something like 2 or 3.

If we further account for the US’ role in the global innovation system, potential existential risks associated with climate change, etc., we could increase the numerator.

Would love either of your thoughts on this approach!