Editing Festival: Results and Prizes

Thanks for this post :)

One thing I've been wondering about is how much impact will result (indirectly) from the wiki/tagging system, and, relatedly, from various types of individual contributions to it. 

I'm wondering this both to work out whether I'm spending more time contributing than I really should and to work out whether I should do more/less encouraging of other people to contribute. (This also seems relevant for "targeted solicitation toward Forum users we expect will be strong editors for specific articles" - having a clearer impact assessment would help both in deciding when it's worth people spending their time this way and in convincing them of that.)

Currently I - as you can probably tell - feel pretty excited about the wiki and tagging system, often send people links to entries, often suggest they edit or tag things, etc. But I don't actually have a clear sense of how good a use of people's time this is.

---

I think it'd help to create an explicit theory of change for the wiki and tagging system, or maybe one for each of those two components. Maybe ideally in diagram form. Then you/we could try to use this to identify assumptions, identify what proxies we should pay attention to and date we should gather, identify what forecasts we should make, etc. (See also.)

---

Here are some other proxy/impact assessment ideas, though they're kind-of putting the cart before the horse without having worked out a more explicit theory of change first. (This also overlaps with discussion we've previously had non-publicly, where Pablo mentioned some other ideas I won't mention here.)

• Look at total or average page views and work out how good or bad those results are
  • Obviously you've already got the page view data, but at least personally I don't know whether that's good or bad or what I wanted to see (except "more is better")
• Run something like Rethink Priorities' impact survey or my personal adaptation of that (see also), where either relatively "key" people or just a whole bunch of Forum users are asked how much they've engaged with the wiki/tags, how high-quality it seemed to be, and whether and how it affected their beliefs or behaviours
  • (I work at Rethink Priorities, but am - as usual - writing this comment in a personal capacity, and was a fan and referencer of their impact survey before joining)
  • What I'd have in mind as a quick and dirty option is basically just replacing "anything I've written" or "anything by Rethink Priorities on [topic]" with "any wiki entries"
  • It's less obvious how to use this to capture the benefits of the tag system, rather than the benefits of the wiki system. Maybe:
    • Similar questions, but for "have you found content to read via looking at tag pages or filtering by tags", "do you think you would've found that content anyway", "do you think that that changed your beliefs (relative to what you might've done with that time otherwise)", etc.
    • Also asking whether people have shared tag pages with people as collections of resources, whether they think that changed the recipients beliefs/behaviours, etc.
• See if the wiki/tags show up in a future EA Survey as a way people learned about or became more engaged with EA?
  • But I guess I feel like the wiki is relatively unlikely to serve that role? 
    • It seems better suited for getting people from participants to contributors or contributors to "core", or helping them do so faster and better, than getting people to enter the audience or become followers 

A caveat I should add: The wiki could be seen more as a product or website than a research output/project. And I'm more used to thinking about how to assess the impact of the latter (though I'm not an expert on that anyway).So maybe my suggestions are a bit off, and maybe it'd be useful to think more about things like:

• How 80k assess the impact of their services
• How the Forum and LessWrong assess their impact or performance or whatever
• How software companies assess how their products are doing (aside from just revenue)

Over the next few months, Pablo and I will be looking for ways to incentivize wiki edits from the wider community. A few things that come to mind:

• A feature allowing people to see their wiki edits on their own profiles
• An addition to the Forum Prize. awarding one or more wiki edits each month
• More targeted solicitation toward Forum users we expect will be strong editors for specific articles

These ideas seem good to me. I think another low-hanging fruit might be some way of displaying quantitative info about how much someone has contributed to this EA wiki, as is done on Wikipedia; see here for elaboration.

But it might be best to do all of these things after getting more sense of how much impact contributing to the wiki/tags has (see my other comment), since that'll inform how much time it'd actually be good to get people spending contributing to the wiki/tags. 

Oh wow, that's a fairly chunky prize! (For readers who don't follow the link to the festival announcement post, it seems worth noting that these prizes are "In the form of donations to an EA Funds-eligible charity of your choice", rather than e.g. deposits to my own bank account)

This seems like a good prompt to revisit and share my rough, interim notes-to-self on where I might donate in 2021 and why.  (I plan to do more thinking and share more details around November, like I did last year.)

• The EA Long-Term Future Fund
  • See also The Long-Term Future Fund has room for more funding, right now
  • As of April 2021, I have a positive impression of their fund managers
  • In April 2021, I skimmed through the LTFF's grants to date to see if there were any that I knew had later turned out quite well, and how this compares to what would've happened if I'd donated directly to things I'd heard of and thought were good
    • I felt unsure how to update on the results of that skimming and thinking
    • There were many cases where I do have a positive impression of the person or project that was funded, but where it wasn’t obvious that (a) LTFF was especially early in identifying this as good, or that (b) the LTFF money allowed the project to go much better than it would’ve gone otherwise
      • E.g., HIPE, All-Party Parliamentary Group for Future Generations, stuff Ozzie Gooen has done
      • In these cases, LTFF should get “points” for funding this, but it’s unclear that they’re choosing better funding opportunities than I’d choose if making decisions myself directly
        • (But note that I'm just saying this in relation to me, and things like work tests suggest that I may be a fairly good fit for grantmaking. Analogously, I should defer to others regarding good career pathways in the EU, but some other EAs need not defer much on that.)
    • But there was also at least 1-3 cases where the project seems to have gone really well or to still be early stage but have really good expected impact per dollar spent, and where I wasn’t aware of the project/plan at all when the grant was made
      • A grant to Nick Hollman to support the Legal Priorities Project
      • Pablo’s wiki work might count
        • (Note that I'm currently unsure how to think about the expected impact of this; I'll write another comment about that)
      • Nuno Sempere’s work might count
    • There was a perhaps surprising number of cases where, as of April 2021, I still hadn’t heard of the person or project at all, or where I knew the person but hadn’t seen any outputs of the project yet
      • But it’s hard to say what I should’ve expected here, since I of course shouldn’t expect to know about every nook and cranny of longtermism, and it’s good if LTFF finds good-but-obscure funding opportunities, and all of the grants were within the last couple years
  • I’d really like it if LTFF at least sometimes set or shared proxies of the impact they hope to see, made forecasts, and shared retrospectives
    • I feel the same way about the other EA Funds too
    • See discussion of this here and here
    • I will probably write more about this soon
  • I read LTFF's May 2021 payout report yesterday, and I don't think it was a positive or negative update for me (i.e., still seems basically quite good, but not obviously better than my donating directly, at least after accounting for the "diversity of viewpoints" benefit of me making my own, direct donation choices)
• Effective Thesis
  • My current, tentative impression is that they're filling a really important niche, and that it's really odd they haven't received funding at a somewhat higher level and with more stability
    • See also
  • I may think and write more about this in future
  • But I haven't yet tried to look closely at or make quantitative cost-effectiveness models or Fermi estimates or the link
• GCRI
  • This is one of the places I gave to in 2020, for the reasons explained here, and I still have roughly the views expressed there
• ALLFED
  • This is one of the places I gave to in 2020, for the reasons explained here, and I still have roughly the views expressed there
• Funding work on "Creating a central, editable database to help people choose and do research projects", of the kind I'll discuss in an upcoming post
  • This funding could be paying for contractors, for someone to "own" this project, for hosting fees, etc.
• All-Party Parliamentary Group for Future Generations
  • [I had some notes to self explaining why this stands out to me as one option, but will leave those of this comment. If I end up thinking more about whether to donate to them, I'll say more on this later, probably around November.]

It seems that Effective Thesis and APPGFG aren't available via the EA Funds platform, even though both have been funded by EA Funds before? Can they be added to the platform relatively easily? (That isn't a super big deal just for this particularly $2000 donation - I'm ok just picking between LTFF, GCRI, and ALLFED - but it seems like that'd be good for other donors too.)

Once I find out whether Effective Thesis and APPGFG can be options for me for this $2k, I'll decide on the resulting shortlist.