I am deeply grateful to Aaron Gertler who reviewed this article. His comments were
very thorough (he didn't leave any hyperlink unclicked). He managed to
question several of my claims. I have updated parts of this post based
on his comments. I also want to thank Carrick Flynn, Peter Hurford,
EA Applicant, Jon Behar, Ben West, 80,000 hours for helping me
directly or indirectly. They provided valuable info in their
posts/comments/email which I have used in this article. That said,
again, they should not be viewed as endorsing anything in this. All
mistakes are mine. All views are mine.
Introduction
I have been using 80,000 Hours (80k) since 2017 and have read
almost all their posts, spent weeks after weeks reading them to figure
out what I should be doing in life. They seem to have done a ton of
research and put out many many posts, for us readers to benefit from.
In the process they have made a lot of claims which are hard and
time-consuming to verify as we don't have the insights, contacts or
the data that 80k is exposed to. For example they estimate "an
additional person working on the most effective issues will have over
100 times as much impact as an additional person working on a typical
issue". To verify this with one example, I would need estimates from
say Open Phil on the impact of an employee. I tried, but they are
unable to put effort into it at the moment.
Maybe 80k can be asked for clarification directly? Unfortunately, 80k
doesn't seem approachable other than through coaching (which is
only for the stellar). Comments sections seem to be deserted to ask
for help, and at the time, I didn't know of any other sources
doing this sort of research and coaching for people. Based on
reading 80k for years I formed the impression as shared by fellow EA
applicant:
Hey you! You know, all these ideas that you had about making the
world a better place, like working for Doctors without Borders?
They probably aren’t that great. The long-term future is what
matters. And that is not funding constrained, so earning to give is
kind of off the table as well. But the good news is, we really,
really need people working on these things. We are so talent
constrained...--- EA applicant in the EA Forum
And looking at the 277 karma this post got (the highest of any post
on the Forum), it might appear that a "lot of people" share(d)
this sentiment that EA orgs could potentially be seriously Talent
Constrained (TC).
A few weeks back I stumbled upon some articles in the EA forum
and to my surprise it appeared that some EA orgs were suggesting that
they were not TC. Until this point I don't think it occurred to me
that 80k' claims ("EA is TC") could be wrong or lost in
translation or that I should test it. Nevertheless, having seen orgs
say otherwise, it felt like a good idea to dig into it at least now.
The following article is my naive investigation on if EA is TC. Before
we start going deep into whether EA is TC or not, we must first state
the definition clearly.
Definitions
We are going to primarily deal with the term "Talent
Constrained" (TC). 80k defines TC in "Why you should work on
Talent gaps" (Nov 2015) as,
For some causes, additional money can buy substantial progress. In
others, the key bottleneck is finding people with a specific skill
set. This second set of causes are more “talent constrained” than
“funding constrained”; we say they have a “talent gap”.
So, a cause is TC if finding people with a specific skill set, proves
to be difficult. The difficulty I assume is in the lack of those
skilled people, and not some process/management constraint. "EA
Concepts", clears this confusion up with a better worded
"example":
Organization A: Has annual funding of $5m, so can fund more staff,
and has been actively hiring for a year, but has been unable to find
anyone suitable... Organization A is more talent constrained than
funding constrained...
In this post, discussions are focused on Orgs that are TC and not
Causes that are TC. When I read that AI strategy is TC with the
lack of "Disentanglement Research" (DR), I don't know what to do about
it. But if I know FHI and many other orgs are TC in DR, then I could
potentially upskill in DR, and close the talent gap. So looking at
causes for me, is less helpful, less concrete and is not what I have
set out to understand.
Why I think EA is TC
EA has been and is talent constrained, according to surveys based on
input from several EA orgs: 2017 survey, 2018 survey,
2019 survey. These surveys were conducted by 80k and CEA.
In all the surveys EAs on average claim to be more Talent Constrained
than Funding Constrained. For example, in 2019 EA orgs reported feeling
more Talent Constrained (3 out of 5 rating) and less Funding
Constrained (1 out of 5 rating).
80k doesn't seem to have changed it's position on this matter
since a while. In 2015, 80k suggested that we should focus on
providing talent to the community rather than ETG, in "Why you should
focus on talent gaps and not funding gaps". One of the examples
they give is about AI Safety where there are people who are ready to
donate even more funds, but think there isn't enough "talent
pool". More posts such as "Working at EA orgs (June 2017), "The
world desperately needs AI strategists (June 2017), "Why
operations management is the biggest bottleneck in EA" (March
2018), and High-Impact-Careers (Aug 2018), continue to make the
case for EA orgs being TC. Even in their recent post, "Key
Ideas" (October 2019)--which is mostly recycled from the 2018 article
on High-Impact-Careers--they continue to say that the bottleneck
to GPR for example, is researchers and operations people.
In Nov 2018, they wrote a post to clarify any misconceptions regarding
the understanding of the term TC: "Think twice before talking about
Talent gaps". 80k informs us that EA orgs are not TC in general
but are TC by specific skills. Some examples (according to them)
being, people capable of Disentanglement Research in Strategy and
Policy (FHI, OpenAI, Deepmind), dedicated people in influential
government positions etc... This is great, the claim is becoming
narrower: EA is TC in specifically X. So what is this X?
Where is the EA specifically TC
There seem to be a list of posts from 80k from which we can
gather where EA is specifically TC. They are:
The surveys from 2017 to 2019 that informed us that the EA Orgs
are TC, provide information on "what sort of talent the EA orgs and EA
as a whole would need more of, in the next 5 years?". This question
sounds like a proxy to "Where is EA specifically TC?". 80k seems
to agree with this proxy-approximation of the question as evidenced
here and here. The top 7 results (out of 20 or so) are
below:
|
2017 |
2017 (EA) |
2018 |
2018 (EA) |
2019 |
2019 (EA) |
| 1 |
GR |
G&P |
Oper. |
G&P |
GR |
G&P |
| 2 |
Good Calib. |
Good Calib. |
Mngment |
Oper. |
Oper. |
Mngment |
| 3 |
Mngment |
Mngment |
GR |
ML/AI |
Mngment |
GPR |
| 4 |
Off. mngers |
ML/AI |
ML/AI |
Mngment |
ML |
Founding |
| 5 |
Oper. |
Movt. build |
GPR |
GPR |
Econ/math |
Soc. Skill |
| 6 |
Math |
GR |
Founder |
GR |
HighEA* |
ML/AI |
| 7 |
ML/AI |
Oper. |
Soc. skill |
Founding |
GPR |
Movt. Build |
* High level overview of EA
*** Government and Policy
For the talents that are unclear, I am unable to do anything with
them at the moment. For the ones that I have clear examples for, I
proceed further.
Another way to arrive at or to supplement this list, is to look at the
top problem profiles and check what the bottlenecks are. For
example, in the profile on shaping AI (March 2017), we see that
80k calls for people to help in AI Technical research, AI
Strategy and Policy, Complimentary roles and, Advocacy and Capacity
building. So basically EVERYTHING IN AI except ETG, is TC (it
appears). In the problem profile on GPR (July 2018), 80k
suggests that they mainly need researchers trained in math, econ, phil
etc... Also needed are academic managers and operations staff. A very
similar story for working at EA orgs as well.
Is it just me or is EA TC in "general"? Like when researchers,
operations people and managers are in shortage at GPR orgs, AI orgs
and other EA orgs, then who else is left?
In the post on High Impact Careers (August 2018), 80k
suggests the following priority career paths and what they are
constrained by:
In brief, we think our list of top problems (AI safety, biorisk, EA,
GPR, nuclear security, institutional decision-making) are mainly
constrained by research insights, either those that directly solve
the issue or insights about policy solutions. --- High Impact
Careers
In focused bottleneck posts for Operations and AI Strategy just
the title already informs how TC the situation is:
In conclusion, the surveys say GRs, ML/AI people, GPR people and
movement building are TC (2019). The problem profiles seem to suggest
that GPR and AI are completely TC except for ETG (2017,2018). Whereas
the High-impact-careers post says that research insights (good
researchers) and policy solutions (good policy people) are the most
constrained (2018). It appears that there is some discrepancy between
different articles--every article doesn't seem to say the same
thing--but we move on with the key message that all these things
listed could be potentially TC. But are they really TC though?
The Evidence
GR in GPR
Researchers in GPR are claimed to be constrained. GR's also stand on
top of the survey lists shown before, for 2019. Yet, Open Phil seems
to paint a very different picture. For the recent hiring round by
Open Phil (started in Feb 2018 and ended in December 2018) they wanted
to hire 5 GRs. They report that more than a 100 strong resumes with
missions aligned to that of Open Phil were received. 59 of them were selected
after remote work tests and went into an interview. Of this, 17 of
them were offered a 3 month trial and 5 selected in the end. "Multiple
people" they met in this round are claimed to have potential to excel
in roles at Open Phil in the future. Open Phil does not seem to feel
that there is a lack of skilled people. It appears that they had
plenty to choose from and that they have found suitable candidates.
A similar case is observed with EAF. In EAF's November 2018 hiring
round they wanted to hire 1 GR (for grant evaluation) and 1
operations person. Within just 2 weeks, 66 people applied to this
EA org which was in a non-hub. These 66 trickled down to 10
interviews after work tests, 4 were offered trials and 2 were selected
in the end. No TC in GR here either.
Would Open Phil like to hire more GRs? For sure, but they don't have
the capability to deploy such a pool of available talent, they
say. They seem to be constrained by something else, something not
"talent".
AI Strategy and Policy
Researchers in AI Strategy and Policy are also claimed to be
constrained. The surveys echo the same as well. But Carrick from
FHI (Sep 2017) suggests that AI Policy implementation and research
work is essentially on hold until Disentanglement Research
progresses. And that even "extremely talented people" will not be
able to contribute directly until then. Similar to Open Phil,
institutional capacity to absorb and utilize more researchers in
Strategy is constrained, according to Carrick. It must be noted that
this is just one persons view on the matter and that a stronger
version of evidence for this would be if several AI orgs agreed with
Carrick's view.
Except for the TC in Disentanglement research (DR)---where there
seems to be large demand and if you meet the bar, you will get a
job---there seems to be no sign of TC in Strategy and Policy, at the
moment.
Once DR progresses, there would be a need for "a lot of AI
researchers", Carrick expects. It's been 2.5 years since the post by
Carrick, and as late as Nov 2018, 80k continues to cite Carrick's
article. This seems to suggest that not much might have changed. I
have tried requesting Carrick to write a reboot of his initial
post and hopefully he can further clarify the TC or lack there of.
Researchers and Management staff in other EA orgs
The co-founder and board member of Rethink Charity seems to suggest
that both senior and junior staff for Rethink Charity and Charity
Science were not hard to find, aka not TC.
I’ve certainly had no problem finding junior staff for Rethink
Priorities, Rethink Charity, or Charity Science (Note: Rethink
Priorities is part of Rethink Charity but both are entirely separate
from Charity Science)… and so far we’ve been lucky enough to have
enough strong senior staff applications that we’re still finding
ourselves turning down really strong applicants we would otherwise
really love to hire.---Peter Hurford says in the 2019 survey
The Life You Can Save's Jon Behar, agrees with Peter. He adds
that it's not the lack of talent but the lack of money to add new
staff which is the bottleneck for TLYCS.
Charity Entrepreneurship's incubation program has grown from 140
applications to ~2000 applications for 15-20 positions since last
year. It's plausibly not TC this year atleast.
Conclusion
An org is TC in Talent X, if it is not able to find "skilled" people
despite "hiring actively". So far we have seen that Open Phil, EAF,
Rethink Charity, Charity Science, TLYCS and FHI, are able to find the
skilled people they need--except for one concrete example of
Disentanglement Research in FHI (and possibly similar institutes).
Contrary to the claims from 80k, it appears that several orgs are
not TC.
I am really upset with 80k. First it was focusing too much on career
capital (CC) and positions like management consulting, and now
TC. Getting into the TC debate only opened a Pandora's box of more
issues. Recently, I discovered that their discussion on
replaceability is plausibly wrong. They have gone back and
forth on it in the past and currently have suggested that it
depends. They ended up inflating impact associated with people
working in EA orgs and have now taken it back. They severely
downplayed how competitive it is to get jobs in EA orgs. And
there are so many cases of people who feel the same way, not
without reason. I traveled with 80k on the CC hype and spent months on
identifying positions of "maximum CC". Then I did a 1 year course
of Data Science at Coursera. After that I jumped onto the
work-at-an-EA-org-because-TC hype and was just about to upskill in
statistics and apply for GR positions because they need me.
So many crucial mistakes that cost people like me and others a
lot of time, and the world a "lot of" dollars. And when someone
requests one of the members of 80k to not just serve the elite
and that perhaps maybe invest in a small conversation with the
non-elite EAs to save them years of wasting time, there is no
reply.
Thus, I find it very hard to trust the claims listed in 80k. And there
are so many of those claims in every post and it's just
impractical to verify each one of them. Rather than relying on the
interpretation of English and generalization of advice for
everyone, I find EA forums a much easier place to get information
from, challenge claims and get responses for (quickly). I found most
of the evidence against TC including the Pandora's box of issues,
there. A lot of the successful people from the EA world seem
approachable there with chats, comments and AMAs. Recently I was able
to chat with Ben West, Aaron Gertler, Peter Hurford, Jon Behar, Jeff
Kaufman and Stefan Torges. A bigger celebrity list of people can be
seen commenting in posts, such as Carrick Flynn and 80k's very own Rob
Wiblin.
Final message
Caution: Just because an org is not TC, it doesn't mean that you
should reject that org.
Why is this debate so important?
Whether an org is TC or not, has implications on the impact made. The
true impact you make when a job is TC at an EA org is (much) higher,
than when the job is not TC. A junior GR at GiveWell is expected to
move 2.4m$ if the job was TC. The same GR is expected to
move only 244k in the case that the hired GR is better than the
next-best-candidate by 10% (Not TC). Such is the distinction between
being TC and not.
The above example assumes no spillover effects. But is that
correct? Why is there no spillover? Should I work in EA or not? How
much value do people really get out of working at an EA org? What is
best path for my aspiring EA career?
Stay tuned...
I’m really sad to hear how upset you are with 80,000 Hours and how you feel it has made it harder rather than easier to find a role in which you can have impact.
It’s a real challenge for us to decide whether to share our views or not publish them until we’re more certain and clear. We hope that by getting more information out there, it will let people make better decisions, but unfortunately we’re going to continue to be uncertain and unable to explain all our evidence, and our views will change over time. It’s useful to hear your feedback that we might be getting the tradeoff wrong. We’ve been trying to do a better job communicating our uncertainty in the new key ideas series, for instance by releasing: advice on how to read our advice
Thank you for collecting together all this specific information about different organisations in EA. The question of whether the issues we focus on are ‘talent constrained’ or not (though I prefer not to use this term), is a complicated one. Unfortunately, I can’t give you a full response here, though I do hope to write about it more in the future.
I do just want to clarify that I do still believe that certain skill bottlenecks are very pressing in effective altruism. Here are a couple of additional points:
There’s a lot more I’d like to say about all of these topics. I hope that gives at least a little more sense of how I’m thinking about this. Unfortunately, I’ve been focusing on responding to covid-19 so won’t be able to respond to questions. I want to reiterate though how sad it is to hear that someone has found our advice so unhelpful, not just because of the negative effect on you, but also on those you’re working to help. Thank you for taking the time to tell us, and I hope that we can continue to improve not only our advice, but also the clarity with which we express our degree of certainty in it and evidence for it.
Thank You for acknowledging this post. I very much appreciate your reply.
I really wish you can put more of your evidence out there instead of sentences that are a summary of the evidence you have. "Another bottleneck to progress on GPR might be operations staff" (GPR Key-ideas). Is it a bottleneck or is it not? I don't know what to make of "might be". In this case if you presented your evidence that helps conclude this, say in a footnote, I think it will be more useful. People can then draw the conclusion for themselves.
I am glad you clarify about your position that you are focused on longtermism TC. I only know of two cases where longtermism positions are TC. Disentanglement research as informed by Carrick Flynn in Sep 2017 and AI Policy in US in Jan 2019 article). It still stands that Open Phil in GR seems to be not TC. ("The pool of available talent is strong, ... more than a hundred applicants had very strong resumes... but ... (to) deploy this base of available talent is weak")
I think what helps is to keep the TC debate focused on to specific cases. And this can be done with providing evidence as done in AI Policy in US.
Claims: Average Survey respondents feel they are TC more than RP because they have less funding needs than RP (and is "new").
Example: Open Phil is an average survey respondent (I presume). Open Phil has funding. Open Phil does not seem to feel TC in GR though.
It looks like the example does not satisfy the claim. So now I don't really know what you are talking about. I don't have one example of an org and a position that is skill-constrained in research in GPR. I keep hearing you saying that "research is the biggest need right now" (key-ideas post) but when I look in Open Phil it doesn't seem to be so. They are unable to absorb more researchers. So what exactly are you talking about?
You might wonder why I am quoting the same Open Phil example like a parrot. That is because that is one of the few hiring rounds available. And trying to ask companies like FHI or Open Phil etc., for more info on this or dollars moved by researcher or about replaceability does not seem to produce results unfortunately.
The definition for TC is that an org is unable to find "skilled people" despite hiring actively. I agree that number of people applied is not a measure for TC. But the number of people in the last round (after 4 other rounds) seems to suggest something regarding if orgs are able to find skilled people or not. Even if that is not the case --> When you look at what Open Phil says, I can't imagine that they are TC in GR based on the numbers of people who they thought had good resumes. In fact it seems like a bad idea to push for research at Open Phil (GPR) in GR considering replaceability atleast. And the more I talk to people like Peter Hurford (about replaceability) the more I feel like there is less point in being a GR.
About "successful applicants might have been still much better" (due to the potential log-normal distribution of candidates ability), I would also like one example for a case where this is true. I don't think that is the case with Open Phil in GR based on their hiring round.
Aaron also raised this point as well. Yes that is definitely a possibility that people would still be hired but the organization would continue to be TC. Seems like a reasonable hypothesis but still needs evidence (one example at least) to support it I think. Nevertheless, I don't think that is the case with Open Phil in GR based on their hiring round.
AI policy careers in the US seems to match the definition of TC. "80,000 Hours has attended, speakers have lamented the government’s lack of expertise on AI, and noted the substantial demand for such expertise within government. For example, DoD’s new Joint AI Center alone is apparently looking to hire up to 200 people.". I didn't know this before. This is so clear for me now, that I have an example for what you mean with "significant number of people". I wish the same was available for other top problem areas.
Thanks for this.
Thank you very much for taking the time to respond. I very much appreciate it. I would really appreciate more evidence displayed for claims and less generalization with 80khours blogs.
P.S
If you already know many opportunities are high-impact, I expect that you have looked at the value contributed by several people, and factored things like replaceability etc., before you came to a decision. Why not just publish it? Asking companies doesn't seem practical and no one seems to be giving out such information. One author even suggested that only if I am writing an academic paper he would be able to help otherwise he didn't find time for it.