Peter Wildeford's Quick takes

For operations roles, and focusing on impact (rather than status), I notice that your view contrasts markedly with @abrahamrowe’s in his recent ‘Reflections on a decade of trying to have an impact’ post:

Impact Through Operations

• I don’t really think my ops work is particularly impactful, because I think ops staff are relatively easy to hire for compared to other roles. However I have spent a lot of my time in EA doing ops work.
  • I was RP’s COO for 4 years, overseeing its non-research work (fiscal sponsorship, finance, HR, communications, fundraising, etc), and helping the organization grow from around 10 to over 100 staff within its legal umbrella.
  • Worked on several advising and consulting projects for animal welfare and AI organizations
    • I think the advising work is likely the most impactful ops work I’ve done, though I overall don’t know if I think ops is particularly impactful.

I see both Abraham and yourself as strong thinkers with expertise in this area, which makes me curious about the apparent disagreement. Meanwhile, the ‘correct’ answer to the question of an ops role’s impact relative to that of a research role should presumably inform many EAs’ career decisions, which makes the disagreement here pretty consequential. I wonder if getting to the ground truth of the matter is tractable? (I’m not sure how best to operationalize the disagreement / one’s starting point on the matter, but maybe something like “On the current margin, I believe that the ratio of early-career EAs aiming for operations vs. research roles should be [number]:1.”)

(I understand that you and Abraham overlapped for multiple years at the same org—Rethink Priorities—which makes me all the more curious about how you appear to have reached fairly opposite conclusions.)

One question I often grapple with is the true benefit of having EAs fill certain roles, particularly compared to non-EAs. It would be valuable to see an analysis—perhaps there’s something like this on 80,000 Hours—of the types of roles where having an EA as opposed to a non-EA would significantly increase counterfactual impact. If an EA doesn’t outperform the counterfactual non-EA hire, their impact is neutralized. This is why I believe that earning to give should be a strong default for many EAs. If they choose a different path, they should consider whether:

1. They are providing specialized and scarce labor in a high-impact area where their contribution is genuinely advancing the field. This seems more applicable in specialized research than in general management or operations.
2. They are exceptionally competent, yet the market might not compensate them adequately, thus allowing highly effective organizations to benefit from their undercompensated talent.

I tend to agree more with you on the "doer" aspect—EAs who independently seek out opportunities to improve the world and act on these insights often have a significant impact.

Do you have any ideas or suggestions (even rough thoughts) regarding how to make this change,  or for interventions that would nudge peoples' behavior?

Off the top of my head: A subsidized bootcamp on core operations skills? Getting more EAG speakers/sessions focused on operations-type topics? Various respected and well-known EAs publicly stating that Operations is important and valuable? A syllabus (readings, MOOCs, tutorials) that people can work their way through independently?

I guess this is the same dynamic as why movie and sports stars are high status in society: they are highly visible compared to more valuable members of society (and more entertaining to watch). We don't really see much of highly skilled operations people compared to researchers

I do think that the marginal good of additional researchers, journalists, content creators and etc isn't exactly as high as it is thought to be. But there's an obvious rational-actor (collective action problem?) explanation: other people may not be needed, but me, with my idiosyncratic ideologies? Yep!

This also entails that the less representative an individual is of the general movement, the higher the marginal value for him in particular to choose a research role.

Which of these two things do you mean?

• operations/management/doer careers should be higher status than they currently are within EA
• operations/management/doer careers should be higher status than research careers within EA

I suspect it varies by cause area. In AI Safety, the pool of people who can do useful research is smaller than the pool of people who could do good ops work (which is more likely to involve EA’s who prefer a different cause area, but are happy to just have an EA ops job).

Just wanted to quickly say that I hold a similar opinion to the top paragraph and have had similar experiences on terms of where I felt I had most impact.

I think that the choice of whether to be a researcher or do operations is very context dependant.

If there are no other researchers doing something important your competitive advantage may be to do some research because that will probably outperform the counterfactual (no research) and may also catalyze interest and action within that research domain.

However if there are a lot of established organizations and experienced researchers, or just researchers who are more naturally skilled than you already involved in the research domain, then you can often have a more significant impact by helping to support those researchers or attract new researchers.

One way to navigate this is to have a what I call a research hybrid role where you work as researcher but allocate some flexible amount of time to more operations / field building activities depending on what seems most valuable.

Did the research experience help you be a better manager and operator from within research organizations?

I feel like getting an understanding by doing some research could be helpful and probably you could gain generalizable/transferable skills but I’m just speculating here.

I think it might be fine if people have genuine interest in research (though had to be intrinsic motivation), which will make their learning fast with more devoted energy. But overall generally I see a lot of value in operations/management/application work, as it gives people opportunities to learn how to land research into real impacts, and how tricky sometimes real world or applications can be.

Good reminder that you should red team your Theory of Change!

not sure if such a study would naturally also be helpful to potential attackers, perhaps even more helpful to attackers than defenders, so might need to be careful about whether / how you disseminate the information

My sense is that 100 is an underestimate for the number of OS libraries as important as that one. But I'm not sure if the correct number is 1k, 10k or 100k.

Relevant XKCD comic.

To further comment, this seems like it might be an intractable task, as the term "dependency hell" kind of implies. You'd have to scrap likely all of GitHub and calculate what libraries are used most frequently in all projects to get an accurate assessment. Then it's not clear to me how you'd identify their level of resourcing. Number of contributors? Frequency of commits?

Also, with your example of the XZ attack, it's not even clear who made the attack. If you suspect it was, say, the NSA, would you want to thwart them if their purpose was to protect American interests? (I'm assuming you're pro-American) Things like zero-days are frequently used by various state actors, and it's a morally grey question whether or not those uses are justified.

I also, as a comp sci and programmer, have doubts you'd ever be able to 100% prevent the risk of zero-days or something like the XZ attack from happening in open source code. Given how common zero-days seem to be, I suspect there are many in existing open source work that still haven't been discovered, and that XZ was just a rare exception where someone was caught. 

Yes, hardening these systems might somewhat mitigate the risk, but I wouldn't know how to evaluate how effective such an intervention would be, or even, how you'd harden them exactly. Even if you identify the at-risk projects, you'd need to do something about them. Would you hire software engineers to shore up the weaker projects? Given the cost of competent SWEs these days, that seems potentially expensive, and could compete for funding with actual AI safety work.

I'd be interested in exploring funding this and the broader question of ensuring funding stability and security robustness for critical OS infrastructure. @Peter Wildeford is this something you guys are considering looking at?

Something that I personally would find super valuable is to see you work through a forecasting problem "live" (in text). Take an AI question that you would like to forecast, and then describe how you actually go about making that forecast. The information you seek out, how you analyze it, and especially how you make it quantitative. That would

1. make the forecast process more transparent for someone who wanted to apply skepticism to your bottom line
2. help me "compare notes", ie work through the same forecasting question that you pose, come to a conclusion, and eventually see how my reasoning compares to yours.

This exercise does double duty as "substantive take about the world for readers who want an answer" and "guide to forecasting for readers who want to do the same".

I wonder if anyone else will getting a thinly veiled counterpart -- given that the lead character of the show seems somewhat based on MacKenzie Scott, this seems to be maybe a thing for the show.

Some people at FHI have had random conversations about this, but I don't think any serious work has been done to address the question.