Comments
I really appreciate how thoughtful you've been about this, including sensitivity to downside risks. Do you have any plans to monitor the downside risks? A lot of them seem quite verifiable/testable.
I really appreciate how thoughtful you've been about this, including sensitivity to downside risks. Do you have any plans to monitor the downside risks? A lot of them seem quite verifiable/testable.
Thanks Peter!
Yep; in the order I mentioned them in the post:
(Note: some of these are things I've done already, some are things I have vague plans to do at some point, and some are much firmer plans. You shouldn't take this as a commitment from me to take any of these particular actions, but more a sense of the kind of stuff I've thought about here. I'd greatly appreciate suggestions you or others have about how I could do better!)
I just want to second Peter's comments - your discussion of downside risks was really thoughtful and cogent.
Of the risks you identified, I'm personally most concerned about 80k contributing to a negative feedback loop on demographic diversity (public perception of EA becomes that it is demographically homogenous -> people who don't match those demographic characteristics are rightfully more skeptical of EA -> recruitment efforts target demographically skewed areas because engagement in them is higher -> public perception of EA as demographically homogenous solidifies -> ...). I look forward to seeing what you find in your investigation of methods of improving 80k's demographic reach!
I'm surprised to see how the book giveaway is more expensive than the costs of actually placing the ads to get eyes on the sites! Why did you decide to give away a physical book? What do you think the cost-effectiveness of that is compared to ebooks or not having a giveaway?
Nice, thanks for your question!
One relevant thing here is that I'm not thinking about the book giveaway as just (or even primarily) an incentive to get people to view our site — I think most of the value is probably in getting people in our target audience to read the books we give out, because I think the books contain a lot of important ideas. I think I'd be potentially excited about doing this without the connection to 80k at all (though the connection to 80k/incentive to engage with us seems like an added bonus).
Re: physical books versus ebook:
Edited cos it occured to me that maybe I didn't really answer your core question here, about cost-effectiveness —
I currently reckon the book giveaway is comparably cost-effective with other things we've done, but that's based on some fairly rough estimates. I didn't feel very satisfied with them, which is why we did the in-depth survey that I mentioned above. I haven't finished analysing the results yet, as I said, so I can't say anything more useful about it for now :)
I couldn't quickly find this data to cite my sources so I can't be more precise — so don't put too much stock in my claim here!
You should of course adjust down for sampling bias and self-report bias. I haven't finished analysing our data from this survey yet — it's hot off the press — so I don't have a view on how much to adjust down.
Interesting that the 80k book is so popular. Is this the book that came out in late 2016, or has the content been updated? If not, it may be worth doing a second edition, since I assume that 80k's thinking has evolved significantly since publication.
I'd look forward to seeing you post the results of the in-depth survey on the forum :-)
Pleased to hear this:
This year I plan to investigate ways to increase demographic diversity among the people we reach, and indeed it’s possible that our work here could really do quite a lot to move the needle in a positive direction for the community as a whole.
Do you have plans for increasing class diversity via 80k career advice / tailoring advice to those with less resources? If so, what are some strategies you have?
I've loved 80k career advice and have benefited a ton from it. But one frustration I've had (especially earlier in my career) is that it doesn't offer much advice for people starting with less resources. For example, non-profit jobs can be out-of-reach without relevant / outstanding credentials or money to do a Masters degree if moving into policy.
I also suspect there's working-class cultural factors some need to un-learn (this was true for me). Manual labor tends to reward putting your head down and doing the hard work. But professional + managerial jobs reward creativity, relationships, and questioning systems.
Thanks for your question & feedback about our advice!
Just to clarify, my responsibility is for outreach and promotion, and that’s all this post is intended to be about. The content of our advice/website/podcast is written & handled by others at 80k.
I passed along your question, and Arden from the content team asked me to share this:
We think a lot of our advice can be useful for people from a variety of backgrounds, but it’s true that most of our advice is geared toward people who have or are planning to get a college degree, because we think that’s our comparative advantage. There might also be cases where we’re unintentionally alienating people from working class backgrounds by not acknowledging the specific challenges they face — thanks for raising that.
If you have concrete ideas about how we could do better at these kinds of issues, we’d be really excited to hear it!
epistemic status: started reading the sequences while delivering food to people on a bike.
I never really felt excluded by problem profile pages. I felt like I could access the tone they take of "here's some broken stuff why don't you go fix it", I just had to discount or not read any sections that talked about status, top universities, etc., kind of assumed I'd have to write my own theory of change and have a thick skin about not always being taken seriously. (and then what happened was the actually-existing EA movement contained tons of more sr people doing actual work saying "yeah here take a crack at it", encouraging me to accept mentorship or jobs from them. If anyone's ever assumed I'm low potential it's been the more community builder types lol)
Age is a more likely vector of neglectedness for 80k than class. I'd be more excited about 80k targeting the frustration people older than 25 or 35 have with 80k, that seems more useful than thinking deeply about class. Related to both of these, we also have to talk about the risk intolerant: are people with obligations and responsibilities (family members with expensive diseases, childrearing projects) low EV? This again seems like a more potent surface for 80k criticism and improvement than class stuff. (But of course, both age and obligation load have some correlations with class!)
I never felt excluded either. And 80k does a lot of things right on this front. The messages of ambition and "here's some broken stuff why don't you go fix it" are good and certainly have pushed me to do things I wouldn't have done otherwise. I genuinely feel people from underprivileged backgrounds need to hear more of it and I try to promote them as much as I can.
>I just had to discount or not read any sections that talked about status, top universities, etc., kind of assumed I'd have to write my own theory of change and have a thick skin about not always being taken seriously.
I think this is where our experiences diverge. I also discounted these sections... by charging straight ahead into applying for my dream jobs and thinking I had the similar chances to Ivy League graduates. But (1) college brand name does matter if comparing two people with no experience, (2) the Ivy League graduate probably had more opportunities, networks and internships than me, and (3) many of my dream jobs were magnitudes more competitive than I initially thought.
Maybe 80k advice worked too well and I flew too close to the sun? I don't know. Messaging is tricky.
>I'd be more excited about 80k targeting the frustration people older than 25 or 35 have with 80k, that seems more useful than thinking deeply about class.
Agreed. To be clear, I'm not endorsing thinking deeply about class. The "business case" for class diversity doesn't seem great to me. But I saw "demographic diversity" mentioned in the original post and I was wondering if class was included in that.
Noted! Sorry for the misinterpretation.
One concrete idea could be an article centered on "class migrants". Perhaps it could be similar to the format of the anonymous interview series, or it could be like the imposter syndrome article where there's one personal profile and a few mini-profiles attached.
Partly, this is to help people feel less alone. But also, I think the strategy for developing your career differs based on where you're starting. Even between colleges, there's variation.
Beyond that, I'm not sure. I get that 80k's target audience is different from me and messaging is hard. So I'm hesitant to recommend huge strategic changes in content.
Thanks for sharing. Do you happen to have the number of books you sent out (as you've already given the cost). Just wondering if my current estimate of around 30000 is close.
Hi Bella, thank you for writing this up! Are you willing to share more granular performance data for the different marketing efforts to help other orgs estimate the expected cost and performance of paid advertising?
Hey Aidan!
I'm not sure — I didn't do this in this post/didn't have any plans to, mostly because I'm unsure how much our experiences would generalise to different contexts.
Performance of our ads within the same channel can vary by up to a couple orders of magnitude, so I'm just not sure how helpful it'd be for others.
That said, if you're considering a specific project, I'd probably be happy for me or one of my team to chat to you about it based on our experience?
Thanks for the prompt response! DM'd you
I'd like to hear more about why you think this causes people to take high impact jobs (what you're measuring, what you're observing) (or more cliche, "what do you think you know and why do you think you know it?").
I'm asking this in the spirit of "trust but verify": I do assume you did a good job here, and at the same time this seems to me like the main place a project like this might break, so it seems healthy to ask.
For reference of others, here's what was said in the post:
- We have various ways of measuring the inclinations of our new audience members, and almost all of them suggest that they are indeed significantly less inclined towards our advice, on average, than control groups of people that didn’t find us through our active outreach efforts.
- However, so long as we are still getting a good proportion of high-inclination traffic too (which it looks like we are), these efforts probably still look worth it.
- Our internal calculations of the value of this work take into account this expected decrease in inclination (and we think it looks good overall).
- All that said, the inclination of the new users we get from our outreach is a top priority, which we will continue to monitor.
Still, as I said elsewhere, seems really promising
Hey Yonatan —I think the more relevant part of my post is the following, which hopefully answers your question? Let me know if it doesn't.
There are some details I can't give because (as I said in the post) I don't have permission from the relevant people to talk about it publicly.
We can’t be sure how many additional people will change to a high-impact career as a result, in large part because we have found that “career plan changes” of this kind take, on average, about 2 years from first hearing about 80k.
Still, our current best guess is that these efforts will have been pretty effective at helping people switch careers to more impactful areas.
Partly this guess is based on the growth in new audience members that we’ve seen (plus 80k’s solid track record of getting new people to eventually switch to more impactful careers), and partly it’s based on a few “proof of concept” switches we’ve seen already.
For example, some small-scale social media ads which 80k ran in 2017 as an experiment led to at least one person switching to a career we’re especially excited about (and 70 people who reported changing their career plans due to 80k).[3] We’ve also already encountered[4] several people who found us via our marketing who seem likely to switch to a more impactful career.[5]
Thanks for writing this! This is cool to see.
I may have missed this but do you have a sense of whether this marketing push is leading to more people working on pressing problems that 80k endorses? I am curious if there is a direct correlation between money spent on marketing and people working on these problems.
Glad you liked the post!
I talk a bit about this in this section – basically, we don't know for sure how much this is causing people to work on pressing problems yet (because people changing their careers like this takes something like 2 years on average, some much longer — and we just haven't been at it that long yet!)
That said, we do have some preliminary evidence (at the very least, some "proofs of concept") that it's working. We also did a small-scale experiment in 2017, where we ran some social media ads and found 70 people who told us they changed their career plans 18 months later.
Any opinions about using this as a baseline to compare other EA outreach efforts?
I tend to be in favor, since this seems to be working, kind of tested, scalable, potentially high upside given some tweaks, and so on. I feel more reluctant to do my own outreach after reading this (but I'm saying this in a good way - as a compliment to you)
I'm not sure I understand what you mean by 'using this as a baseline to compare other EA outreach efforts'? Is there some specific outcome metric you'd want to use as a baseline?
In general, I think what works best in outreach can be pretty context-specific, and I wouldn't recommend everything I've done to people with different goals & constraints.
Being more reluctant to do your own outreach after learning about this makes sense if you think there’s some optimal growth rate in EA which we are at or nearly at. If you learn that I’m doing lots of outreach, then it decreases the value of additional outreach (unless we are not yet at or near the optimal rate of growth).
As one example, I have a tiny project where I publicly measure the effectiveness of Israeli startups (in a Facebook group where people can comment and ask questions), and a big part of my goal is to explain the Israeli tech ecosystem how one even measures impact, and that not all med-tech startups are equal.
I might try to estimate how much it would "cost" me to get one person to a high impact job, and if that "cost" is way more than what you're doing with the 80k outreach, I should maybe just dump the project. Or in other words, for my project to be worth while, it must be at least POTENTIALLY more cost effective than the 80k outreach. Similar to how if I'd want to run an intervention for global health and wellbeing, I'd want it to at least POTENTIALLY be better than AMF. (in both these examples, 80k and AMF are also tested and scalable, which are two advantages that new projects might not always have)
(Did this make more sense?)
The empirical reason is that being a newsletter subscriber correlates with having spent about twice the recorded time on our website, compared to non-subscribers.
I feel like the causality could be "people spend a lot of time on the website" --> "people subscribe to the newsletter" rather than the other way?
And how does 80k record people's time spent on the website?
I totally agree the causality could go that way.
In fact, I think it'd be really weird if there wasn't at least some effect in that direction, because people have to go on our website at some point in order to sign up to the newsletter!
I do think there is a significant effect in the direction 'people subscribe to the newsletter -> people spend time on the website,' though.
Why? Mostly cos of the big spikes in engagement we see on pages that we link to from the newsletter — it just does look like people click on and then sometimes read stuff on our site, if we put it in the newsletter.
As for how we record people's time spent on the website: we use the web analytics tools Google Analytics and Mixpanel. There are various caveats and systematic measurement errors that apply to both of them that we try to account for in our reporting on web engagement statistics.
I am excited to hear this. Thanks for your work.
Since the start of 2022, 80,000 Hours has been putting a lot more effort and money into getting more people to hear of us and engage with our advice.
This post aims to give some insight into what we’ve been up to and why.
80,000 Hours has, we think, been historically cost-effective at causing more people to aim their careers at tackling pressing world problems. We've built a system of resources (website, podcast, job board, advising) that many people have found helpful for this end — and so we want more people to find them.
Also, 80,000 Hours has historically been the biggest single source of people learning about the EA community. If we want to grow the community, increasing the number of people reached by 80k seems like one of the best available tools for doing that.
Thirdly, outreach at the “top of the funnel” (i.e. getting people to subscribe after they hear about 80k’s ideas for the very first time) has unusually good feedback mechanisms & is particularly easy to measure. For the most part, we can tell if what we’re doing isn’t working, and change tack pretty quickly.[1]
Another reason is that a lot of these activities take relatively little staff time, but can scale quite efficiently with more money.
Finally, based on our internal calculations, our outreach seems likely to be cost-effective as a means of getting more people into the kinds of careers we’re really excited about.
In 2020, 80k decided to invest more in outreach by moving one of their staff into a position focused on outreach, but it ended up not working out & that person left their role.
Then in mid-2021, 80k decided to hire someone new to work on outreach full-time. They hired 1 staff member (me!), and I started in mid-January 2022.
In mid-2022, we found that our initial pilots in this area looked pretty promising — by May we were on track to 4x our yearly rate of subscriber growth — and we decided to scale up the team and the resource investment. I ran a hiring round and made two hires, who started at the end of Nov 2022 and in Feb 2023; I now act as head of marketing.
We also decided to formalise a “marketing programme” for 80k, which is housed within the website team. Since this project spends money so differently from the rest of 80k, and in 2022 was a large proportion of our overall spending, last year we decided to approach funders specifically to support our marketing spend (rather than draw from our general funds). The marketing programme has a separate fundraising cycle and decisions are made on it somewhat independently from the rest of 80k.
In 2022, the marketing programme spent $2.65m (compared to ~$120k spent on marketing in 2021). The bulk of this spending was on sponsored placements with selected content creators ($910k), giving away free books to people who signed up to our newsletter ($1.04m), and digital ads ($338k). We expect to spend more in 2023, and are in conversation with funders about this.
As a result of our efforts, more than 5x as many people subscribed to our newsletter in 2022 (167k) than 2021 (30k), and we had more website visitors in Q4 2022 than any previous quarter (1.98m).[2]
We can’t be sure how many additional people will change to a high-impact career as a result, in large part because we have found that “career plan changes” of this kind take, on average, about 2 years from first hearing about 80k.
Still, our current best guess is that these efforts will have been pretty effective at helping people switch careers to more impactful areas.
Partly this guess is based on the growth in new audience members that we’ve seen (plus 80k’s solid track record of getting new people to eventually switch to more impactful careers), and partly it’s based on a few “proof of concept” switches we’ve seen already.
For example, some small-scale social media ads which 80k ran in 2017 as an experiment led to at least one person switching to a career we’re especially excited about (and 70 people who reported changing their career plans due to 80k).[3] We’ve also already encountered[4] several people who found us via our marketing who seem likely to switch to a more impactful career.[5]
(A bit more on audience inclination below.)
Why ask people to sign up to your newsletter?
Aren’t you going to get a lot of low-inclination traffic via these means (Where by ‘inclination’ I mean inclination to, eventually, change career plans to tackle one of the world’s most pressing problems)?
Are you worried about any negative externalities from this work?
Feel free to ask me any questions you have about this work in the comments. You can also email me at [email protected], or leave a comment on my admonymous.
Of course, some kinds of outreach activities are harder to measure, such as “awareness marketing” which aims to get people to hear about 80k, without optimising for them taking any particular action.
Our internal metrics make us feel confident that the fact that our website traffic reached its highest point this quarter was due to our outreach efforts, rather than because of the heightened media coverage on EA at this time.
By this, I mean that, 18 months after the ads were run, 70 people who first found out about us via these social media ads answered ‘yes’ to the question ‘Have your career plans changed in any way as result of engaging with 80,000 Hours?’
e.g. at EAGs, in our annual user survey, and in some user interviews I recently conducted.
I haven’t given any detail about the kind of careers they’re in or hope to switch to because I don’t have permission from them to mention them publicly. 80k publicly posts some people’s stories with their permission, which might include some of these people in future.
I’m not sure these are actually ‘frequently asked’ but more ‘questions I anticipate you might ask,’ based on the above.
Historically, people who found out about EA via 80k skew slightly more male, but slightly less white, than average EAs.
I like that you had the balls to scale 5-20x in a year, respect.