EdoArad

Doing stuff @ Effective Altruism Israel
4906 karmaJoined Nov 2018Working (6-15 years)Tel Aviv-Yafo, Israel

Bio

Participation
1

Hey! I'm Edo, married + 2 cats, I live in Tel-Aviv, Israel, and I feel weird writing about myself so I go meta.

I'm a mathematician, I love solving problems and helping people. My LinkedIn profile has some more stuff.

I'm a forum moderator, which mostly means that I care about this forum and about you! So let me know if there's anything I can do to help.

I'm currently working full-time at EA Israel, doing independent research and project management. Currently mostly working on evaluating the impact of for-profit tech companies, but I have many projects and this changes rapidly. 

Comments
812

Topic contributions
32

Is the issue highly polarized/politicized in Germany? Say, do you expect some people to engage less with your work as a result of "choosing sides"?

In EA Israel we tend to take more politically neutral approaches when engaging publicly, including in relation to the current war or to the large "pro-democratic" public protests (wiki). I'm curious about your attitude to reputational risks and the potential problem of "painting EA with leftist colors".

Usually we use purchase-parity adjustments when we look at the value of money, so there might be a subtle circular reasoning problem here. One issue for example is that there is no value in innovation (say, by designing a cheaper transportation service/product, people could end up paying less for the service so the revenue may decrease [while the profits could get higher]). 

Another issue is that I don't think this could take into account counterfactual impact. 

Didn't read through the whole thing yet, sorry if I missed how you addressed this issues! I love the idea of trying to find simpler ways of estimating impact!

Also, CCC are using a discount rate of 8% (compared with 4% that's common at least for CE)

How should we compare their (CCC's) cost-benefit estimates to GiveWell's (GW's) results?

I quickly spot two differences:

  1. GW's benefit estimates are in units of doubling of consumption (interventions are compared to direct cash transfers through Give Directly, itself measured in terms of doubling of consumption), whereas CCC's estimates are in units of value of a statistical life.
  2. CCC's analyses are for large-scale government operations, so 
    1. they don't account for counterfactual investments,
    2. they encounter diminishing returns (compared to GW's focus on the marginal dollar).

Points 1. and 2.a. are both indications that CCC's cost-benefit ratios are higher than GW-esque's cost-effectiveness ratios, while 2.b. goes to the other direction.

From the Tuberculosis paper, regarding value of a statistical life (VSL):

The GDP growth in this group of countries outpaces the population growth, so that the VSL grows rapidly over time. In constant 2020 US$ values, the benefit of an averted death is US$ 98,700 (2020), US$ 149,800 (2025), US$ 212,000 (2030), US$ 276,300 (2035), US$ 338,100 (2040), US$ 396,800 (2045), and US$ 456,000 (2050).

In comparison, GW estimates that one can save a life (or have equivalent impact) at about $5000. That's a factor of ~20. [I think this is misleading, so I'd be interested in a more careful comparison here, directly comparing VSL to doubling of consumption].

I'm also not sure how exactly to account for the differences in 2.

Totally agree with the need for a more balanced and careful analysis!

The main point is that access management is more natively associated with the structure of the model in software settings. Say, you are less likely to release a model without its prerequisites.

But I agree that this could also be messed up in software environments, and that it's mainly an issue of UI and culture. I guess I generally argue for a modeling environment that is "modeling-first" rather than something like "explainable-results-first".

I have a very rough draft based on work with @Michael Latowicki and Charity Entrepreneurship (@Filip_Murar and @weeatquince) where we tried various tools for modeling uncertainty for use in their CEAs. Due to personal issues, I'm not sure when I'll finish it, so here it is (very unpolished and not finished).

Mainly, we list various tools we found with some comments on them and discuss the pros and cons of spreadsheet vs software solutions. The latter is mostly coherent so I'll paste it here:

Spreadsheets vs programming languages

The tools we found for probabilistic calculation come in two flavours: either the calculation is expressed in a spreadsheet, or it is written as text in some programming language that is customized by a dedicated interpreter or library for this purpose.

The spreadsheet-based solutions seem to have one salient advantage: they do not scare non-programmers away. This is, we think, a barrier-to-entry advantage, rather than a long-term productivity advantage. The kind of people who build cost-effectiveness model are not incapable of being productive in a simple, probability-dedicated programming language as well as with a probabilistic library within a general purpose language. 

The error-proneness of spreadsheets

We strongly suspect that using spreadsheets to organize formulae is more error-prone than using programming languages. We are not alone in this suspicion. Audits of spreadsheets generally find that something between a large fraction and nearly all spreadsheets contain errors. 

Some pitfalls that make spreadsheets more error-prone than programming languages:

  • In a spreadsheet, mistyping a cell address does not necessarily result in an error message. By contrast, mistyping a variable name in a programming language typically does.
  • In a spreadsheet, unintentionally leaving a cell empty is equivalent to setting it to zero, which often does not result in an error. In a programming language, using a variable without declaring it does trigger an error.
  • Since spreadsheet formulae typically reference values by cell address, erroneous references are not salient to the reader. If your formula is “power(1+interest_rate, -years_left)”, but it is encoded as “power(1+M7, $A$13)” then you may easily fail to notice that the “13” should actually be “12”.
  • Formulas are not automatically updated to account for added data, for example in sums of columns.
  • Copy-and-pasting formulae often results in erroneous cell references because the software adjusts treats cell addresses as relative by default and the spreadsheet author often fails to consider whether addresses should be relative or absolute.

There are other causes of errors in spreadsheets, but you get the point. Errors occur in every kind of programming environment, of course, but spreadsheets sport their own additional pitfalls, on top of those that exist in every programming language.

We are far from certain that writing cost-effectiveness analyses in an ordinary programming language would reduce the error rate compared to spreadsheets - quantitative estimates of the error rate in both spreadsheets and in non-spreadsheet programs find error rates on the same order of magnitude. The mix of problems that are typically approached using these two types of tools is different though, and we have not found an apples-to-apples study of those error rates.

It is apparently not easy to root out spreadsheet errors. In “Spreadsheet errors: What we know. What we think we can do”, Professor of IT management Ray Panko summarizes his findings:

Unfortunately, only one approach to error reduction has been demonstrated to be effective. This is code inspection, in which a group of spreadsheet developers checks a spreadsheet cell-by-cell to discover errors. Even this exhausting and expensive process will catch only about 80% of all errors. Other things can be done to reduce errors, but given the tenacity of spreadsheet error in the face of even cell-by-cell code inspection, we should be careful about expecting too much from most error-reducing approach.

Load more