Hide table of contents

Comment Permalink

For what it's worth, I agree with the theoretical argument that people might be biased to overweight interestingness over importance.

But I think I disagree fairly strong with the following (implied?) points:

The incentive gradient within EA (always, or usually) points people towards work with longer feedback loops than shorter feedback loops.
The incentive gradient within EA rewards you for academia than for more direct work in EA.
The incentive gradient within EA rewards you for more theoretical work than more applied work

I think I agree with a limited version of:

People seem drawn to "bigger picture", more abstract work over narrow, concrete, specific problems.
People may naturally be overly inclined to do theoretical work over applied work
- But I think this is in spite of EA incentive gradients, not because of them.
Perhaps in some specific circumstances, EA social incentives need to be stronger to point people away from interestingness and towards importance.

Some collection of personal experiences:

On the forum: I feel like I get many more points (in the sense of recognition, but also literally EAF karma) per hour for commenting than for top-level posts. The former is faster and has locally better feedback loops.
On my day job (research manager): I feel like the incentive gradient (especially locally) points me to doing local activities like day to day management, giving reviews/feedback on people's research, and away from forming deeper clarity/high-level strategic thinking.
- This despite everyone involved generally being supportive and thinking the latter work is very important.
Grantmaking vs. conceptual research: I think per hour, I gain more recognition, encouragement, etc., from grantmaking than for forming conceptual views and research. The former also has much faster feedback loops.
- (This is ameliorated by me personally finding grantmaking much less interesting/fun than my other projects).
Friends in ML academia vs alignment labs: I feel like my friends in academia usually get less points when they want to stay in academia, as opposed to joining Anthropic/Redwood.
- And more locally, they seem to get more prestige points per hour for stuff like hosting podcasts, grantmaking, LW posts, etc, than from writing papers and deepening their understanding of ML.
  - (This isn't a clear misallocation. Naively you'd expect people staying in academia to get more prestige points from academia/the outside world, so EA giving them relatively less points seem like an appropriate redistribution).
How I got into EA research: I got into EA research in 2020 via one of the EAish projects with the fastest feedback loops ever (coronavirus forecasting). I suspect I'd be a lot slower to either a) find internal traction or b) external engagement/recognition if I were to try to start with, e.g., AI strategy or some of the other research projects I was excited about in early 2020 (e.g. heavy-tailedness of talent).

More broadly, I think the general vibe I get from the world is that you get locally rewarded more for things that have faster feedback loops and are more attention grabbing (related notes: gamification, contrast with Deep Work). So it should be surprising if EA is a stark contrast.

That said, I do think something about people finding "bigger picture" problems more interesting and doing them anyway, despite incentives pointing in the other discussion, is a relevant piece of the puzzle.

I think I've been in multiple positions where I strongly advised very junior researchers to focus on narrower, more concrete research questions, as it would be better for their career and (by my personal estimation) their personal learning to start small and concrete. Usually, they ended up tackling the "big picture" nebulous problems anyway. See also Nuno's notes.

(I've also been on the receiving end of such advice)

[epistemic status: idle, probably mean, speculation]

Still, there's something that needs to be explained via both your and Duncan's beliefs that people keep going up the longer feedback loop ladder and the abstraction ladder. If I were to hazard a guess for why people were to believe this, I'd probably go to:

Another important issue is that “it's easier to pass yourself off as a long-looper when you're really doing nothing,” but that’s a different discussion.

I think you treat that as a sidenote, but to me maybe this is the whole story. Speaking in a overly general way, we can imagine that EA prestige rewards people in roughly the following ways:

Theoretical wins (e.g. successfully champion a new cause area)> Concrete wins^[1](e.g. founding a company doing great work, distributing many bednets) >> Theoretical failure >> Concrete failure.

In this story, it's easier/much more legible to see somebody fail when they do very concrete work. People are scared of openly failing, so they if they think they can't get concrete wins (or they don't believe it has sufficiently high probability), they're drawn to doing theoretical work that is harder to pinpoint exactly if, or when, they've failed. This despite concrete work having perhaps higher EV (both altruistically and selfishly).

So if we celebrate legible failures more, and have less of an act/omission distinction, then perhaps the incentive gradient will point people who fail at conceptual/theoretical work to be more excited to take a shot at doing more concrete work.

Perhaps not, too. I'm far from confident that I pinpointed the exact problem or solution.

___

I also want to quickly note that your post conflates "theoretical/conceptual work" and "slow feedback loops." I think this is only sort of true.

^{^}
An important caveat here is that I think the rest of the world will privilege a concrete win over a theoretical win. So for example, most core EAs would consider "being one of the three most important people instrumental in causing EAs to work on AI alignment as a major cause area" to be more impressive than e.g. founding Wave. But I expect almost anybody else in the world to consider founding Wave a bigger deal.

See in context

Interesting vs. Important Work - A Place EA is Prioritizing Poorly

by Davidmanheim

Jul 28 20226 min read 52

118

Cause prioritizationBuilding effective altruismCriticism of effective altruist organizationsCriticism of the effective altruism communityCriticism of work in effective altruism

Frontpage

Interesting vs. Important Work - A Place EA is Prioritizing Poorly

What are the interesting things?

Motivating Theoretical Model

Towards fixing the problem[3]

52 comments

There are many important issues in the world, and many interesting topics. But these are not the same thing, and we should beware of suspicious convergence. Given that, our assumption should be that the most interesting topics we hear about are far less important than the attention they receive. Heeding Scott Alexander’s recent warning, I’ll therefore ask much more specifically, what are the most intellectually interesting topics in Effective Altruism, and then I’ll suggest that we should be doing less work on them - and list a few concrete suggestions for how to do that.

What are the interesting things?

Here are some of my concrete candidates for most interesting work: infinite ethics, theoretical AI safety, rationality techniques, and writing high-level critiques of EA^[1]. And to be clear, all of these ARE important. But the number of people we need working on them should probably be more limited than the current trajectory, and we should probably de-emphasize status for the most theoretical work^[2].

To be clear, I love GPI, FHI, CSER, MIRI, and many other orgs doing this work. The people I know at each org are great, and I think that many of the things they do are, in fact, really important. And I like the work they do - not only do I think it’s important, it’s also SUPER interesting, especially to people who like philosophy, math, and/or economics. But the convergence between important and interesting is exactly the problem I'm pointing towards.

Motivating Theoretical Model

Duncan Sabien talks about Monks of Magnitude, where different people work on things that have different feedback loop lengths, from 1 day, to 10-days, to people who spend 10,000 days thinking. He more recently mentioned that he noticed “people continuously vanishing higher into the tower,” that is, focusing on more abstract and harder to evaluate issues, and that very few people have done the opposite. One commenter, Ben Weinstein-Raun, suggested several reasons, among them that longer-loop work is more visible, and higher status. I think this critique fits the same model, where we should be suspicious that such long-loop work is over-produced. (Another important issue is that “it's easier to pass yourself off as a long-looper when you're really doing nothing,” but that’s a different discussion.)

The natural tendency to do work that is more conceptual and/or harder to pin to a concrete measurable outcome is one we should fight back against, since by default it is overproduced. The basic reason it is overproduced is because people who are even slightly affected by status or interesting research, i.e. everyone, will give it at least slightly more attention than warranted, and further, because others are already focused on it, the marginal value is lower.

This is not to say that the optimal amount of fun and interesting research is zero, nor that all fun and interesting work is unimportant. We do need 10,000 day monks - and lots of interesting questions exist for long-termism that make them significant moral priorities. And I agree with the argument for a form of long-termism. But this isn’t a contradiction - and work on long-termism can be concrete and visible, isn’t necessarily conceptual, and doesn’t necessarily involve slow feedback loops.

Towards fixing the problem^[3]

Effective Altruism needs to be effective, and that means we need evaluable outputs wherever possible.

First, anyone and everyone attempting to be impactful needs a theory of change^[4], and an output that has some way of impacting the world. That means everyone, especially academics and researchers, should make this model clear, at least to themselves, but ideally also to others.

If you’re writing forum posts, or publishing papers, you need to figure out who is reading them, and whether your input will be helpful to them^[5]. (Not whether it will be upvoted, or cited, because proxies are imperfect - but whether it will actually have an impact.) You should then look back and evaluate the work - did you accomplish something, or is it on the way to happening?

Lastly, I think that we as a community need to put status in better places. Philosophy and economics are fun, but there’s a reason they are seen as impractical, and belonging to the ivory-tower. Ceteris Paribus, academic work should get less status than concrete projects. And anyone on that path who is self-justifying it as their most impactful option needs to have a pretty strong story about impact to justify it^[6], and a really good reason to say they wouldn’t be more impactful elsewhere.

Thanks to Abi Olvera and Abie Rohrig for feedback

^{^}
Yes, I see the irony of suggesting less conceptual critical work - this is procrastivity on my part.
Also, feel free to suggest additional areas in the comments.
^{^}
To be clear, I’m guilty of some of this - I worked on a really fun paper with Anders Sandberg on the limits to value in a finite universe. During the same time period, I worked on introducing state legislation on lab accidents. And I will freely admit which one was more fun, and which was more impactful, and I think readers can guess which is which.
Also, in general, altruists need to push back against the exact opposite problem, where people do concrete things that have obvious immediate impact, instead of trying to optimize at all, either just to have done something, or to feel good about their work. However, I think the “purchase your fuzzies separately” criticism is already very clearly integrated into effective altruism, and if anything, the pendulum has swung too far away from that extreme.
^{^}
This won’t fix the problem - that takes actual concrete work, not critical forum posts. (In general, work which is useful doesn’t get titled “Towards X” - but I already had a theory section, and solving problems is hard, while writing papers, or blog posts, is much easier, more intellectually stimulating, and more fun. Which is exactly the point.)

^{^}

As an aside, focusing on areas that others are excited about as impactful, without a clear inside view about the topic, is a really bad heuristic. It means you bought the kool-aid, but likely aren’t capable of focusing on things that are neglected within the domain, because you don’t have your own model. That means your work might have an impact, but to make that happen, you’ll need to be incredibly careful and deferential to others - which is absolutely horrible for epistemic health.

Concretely, this suggests a few strategies. Don’t defer wholesale, and instead, give yourself slack and explore. Counter-intuitively, don’t aim for impact first. Working on community priorities before understanding them should be discouraged in favor of moving more slowly and building inside views - even if it takes quite a long time, and achieving clarity about why you disagree with others. (And if and where you agree with the consensus, or even some specific group’s position, again, beware of suspicious convergence between your views and the implicit social pressure to agree.)

^{^}

My hope for this post is that people who read the EA forum personally update towards more value for concrete work, and both mentally and interpersonally assign lower status to people who work at the coolest EA orgs.

Note to those people and orgs: I love you all, please don’t hate me. But the suggestions also apply to you. On the other hand, I think that much of this is a community dynamics issue rather than something the organizations themselves are doing wrong, so further explicitly signaling the value of concrete work, to push back against the tendency to emphasize the importance of your work, would be great.

^{^}

Concretely, ask other people in EA who are not in academia about whether they buy your argument for impact, or if they think you should be doing something different. To minimize distortion due to interpersonal dynamics, you probably want to ask a group of people to all give you feedback, probably via a google doc, with the option to do so anonymously.

Show all footnotes

118 Reactions

Mentioned in

124Some concerns about policy work funding and the Long Term Future Fund

47...but is increasing the value of futures tractable?

More posts like this

Comments52

Sorted by

New & upvoted

Click to highlight new comments since: Today at 12:44 PM

calebpJul 28 202260

I agree in principle with this argument but ....

Here are some of my concrete candidates for most interesting work: infinite ethics, theoretical AI safety, rationality techniques, and writing high-level critiques of EA[1].

I really don't think there are many people at all putting substantial resources into any of these areas.

Theoretical AIS work seems really important and depending on your definition of 'theoretical' there are probably 20-100 FTE working on this per year. I would happily have at least 10-100x this amount of work if the level of quality can be at least maintained.
Who is actually working on infinite ethics?? I don't think this has pulled a lot of EA talent, I'd guess fewer than 5 FTE.
Rat techniques - CFAR basically doesn't exist and has fewer than 5 FTE, I'd guess most of this work is done on the side now but it certainly does not seem to be over-done by EAs.
Criticisms - I think this is more reasonable, I don't think lots of time is spent here but a lot of attention is and I generally haven't found criticism posts to be particularly insight generating (although this one is more interesting).

My suspicion is that this "people are generally overprioritising interesting things" claim sounds nice but won't hold up to empirical investigation (at least on my world view).

[anonymous]Jul 28 202218

Those may seem like the wrong metrics to be looking at given that the proportion of people doing direct work in EA is small compared to all the people engaging with EA. The organizations you listed are also highly selective so only a few people will end up working at them. I think the bias reveals itself when opportunities such as MLAB come up and the number of applicants is overwhelming compared to the number of positions available, not to mention the additional people who may want to work in these areas but don't apply for various reasons. I think if one used engagement on things like forum posts like a proxy of total time and energy people put engaging with EA then I think it would turn out that people engage disproportionately more with the topics the OP listed. Though maybe that's just my bias given that's the content I engage with the most!

Jack O'BrienJul 29 20224

The overwhelming number of applicants to MLAB is not indicative of a surplus of theoretical AI alignment researchers. Redwood Research seems to be solving problems today which are analogous to future AI alignment problems. So, Redwood's work actually has decent feedback loops, as far as AI safety goes.

DavidmanheimJul 29 20223

Agreed - neither Redwood nor MLAB were the type of alignment work that was being referenced in the post.

calebpJul 29 20224

.... But the number of people we need working on them should probably be more limited than the current trajectory ....

I’ll therefore ask much more specifically, what are the most intellectually interesting topics in Effective Altruism, and then I’ll suggest that we should be doing less work on them - and list a few concrete suggestions for how to do that.

I feel like the op was mostly talking about direct work. Even if they weren't I think most of the impact that EA will have will eventually cash out as direct work so it would be a bit surprising if 'EA attention' and direct work were not very correlated AND we were losing a lot of impact because of problems in the attention bit and not the direct work bit.

DavidmanheimJul 29 20221

"I feel like the op was mostly talking about direct work."

No - see various other comment threads

DavidmanheimJul 28 20224

I noticed the same when attending GPI conferences, well attended by EA-adjacent academics, which is why I picked infinite ethics as an example.

calebpJul 29 20223

the organizations you listed are also highly selective so only a few people will end up working at them.

Which organisations? I think I only mentioned CFAR which I am not sure is very selective right now (due to not running hiring rounds).

DavidmanheimJul 28 202215

I claim that if you look at funding at what EA organizations are viewed as central - and again, GPI, FHI, CSER, and MIRI are all on the list, the emphasis on academic and intellectual work becomes clearer. I would claim the same is true for what types of work are easy to get funding for. Academic-like research into interesting areas of AI risk is far easier to get funded by many funders than direct research into, say, vaccine production pipelines.

[anonymous]Jul 28 202233

I don't see how this is a response to the comment. I think there is approximately ~1 FTE working on infinite ethics in EA . If infinite ethics is indeed, as you said in the main post, one of the four most interesting topics in the whole of EA and approximately no-one is working on it in EA, this is evidence that interestingness is not an important source of bias in the community.

Moreover, according to your argument we can know that fewer people should be working on infinite ethics in EA merely by knowing that the topic is interesting. This is very implausible.

Or take theoretical AI safety. I take it that your argument is that some (how many?) people should stop doing this work and we can know this only by virtue of knowing that the work is interesting. I can think of many arguments for not working on AI safety, but the fact that it is interesting seems a very weak one. I think interesting academic research on AI is easy to fund because the funders think (a) there is a decent chance we will all die due to AI in the next 20 years, and (b) this research might have some small chance of averting that. I find it hard to see how the fact that the research is interesting is an important source of bias in the decision to fund it.

GPI, FHI, CSER and MIRI are a small fraction of overall funding in EA. CSER doesn't get any EA money, and I think the budgets of FHI and GPI are in the low millions per year, compared to hundreds of millions of dollars per year in EA spending.

LinchAug 3 202215

GPI, FHI, CSER and MIRI are a small fraction of overall funding in EA. CSER doesn't get any EA money, and I think the budgets of FHI and GPI are in the low millions per year, compared to hundreds of millions of dollars per year in EA spending.

I agree with your other lines. But I think it's inaccurate to model the bulk of EA efforts, particularly in longtermism, in terms of funding (as opposed to e.g. people).

DavidmanheimJul 28 20228

"we can know that fewer people should be working on [each area I listed]"

I think you misread my claim. I said that "the number of people we need working on them should probably be more limited than the current trajectory" - EA is growing, and I think that it's on track to put far too much effort into theoretical work, and will send more people into academia than I think is optimal.

"I take it that your argument is that some (how many?) people should stop doing this work"

I had a section outlining what the concrete outcome I am advocating for looks like.

To address the question about AI safety directly, my claim is that of the many people interested in doing this work, a large fraction should at least consider doing something a step more concrete - as a few concrete examples, ML safety engineering instead of academic ML safety research, or applied ML safety research instead of mathematical work on AI safety, or policy activism instead of policy research, or public communication instead of survey research. And overall, I think that for each, the former is less prestigious within EA, and under-emphasized.

[anonymous]Jul 29 20222

I think the implications of your argument are (1) that these areas get too much interest already, and (2) these areas will get too much interest in the future, unless we make extra efforts relative to today, perhaps motivated by your post.

(1) doesn't seem true of the areas you mention and this is particularly clear in the case of infinite ethics, where there is only 1 FTE working on it. To give an instructive anecdote, the other person I know of who was working on this topic in her PhD (Amanda Askell) decided to go and work for Open AI to do AI policy stuff.

The point also seems clear in relation to rationality tools given that the main org working on that (CFAR) doesn't seem to operate any more.

There is more attention to theoretical AI stuff and to EA criticism. Taking your high-level EA criticism as an example, this is exclusively a side-hustle for people in the community spending almost all of their time doing other things. It is true that criticism gets lots of attention in EA (which is a strength of the community in my view) but it's still a very small fraction of overall effort.

And, the fact that these topics are interesting seems like a very weak steer as to how much resources should go into them.

DavidmanheimJul 29 20228

I'm explicitly saying that (1) is not my general claim - almost everything is under-resourced, and I don't think we want fewer people in any of these areas, but given limited resources, we may want to allocate differently. My point, as I tried to clarify, was (2).

Regarding infinite ethics , it came up in several different presentations at the recent GPI conference, but I agree it's getting limited attention, and on the other points, I don't think we disagree much. Given my perception that we barely disagree, I would be interested in whether you would disagree with any of my concrete suggestions at the end of the post.

[anonymous]Jul 29 20227

I know you are only claiming (2), but my point is that your argument implies (1). Simply, if there is a genuine bias towards interesting but not impactful work, why would it only kick in in the future but not so far after >10 years of EA?

If your claim is (2) only, this also seems false. The trajectory for infinite ethics is maybe 2-3FTE working on it 5 years or something? The trajectory for rationality tools seems like basically no-one will be working on that in the future; interest in that topic is declining over time.

I agree with the last section apart from the last paragraph - i think theoretical philosophy and economics are very important. I also think we have completely different reasons for accepting the conclusions we do agree on. I have not seen any evidence of an 'interestingness bias', and it plays no role in my thinking.

DavidmanheimAug 3 20227

First, biases are fare more critical in the tails of distributions. For example, if we should optimally have 1% of humans alive today work on ML-based AI safety and 0.01% of humanity work on mathematical approaches to AI risk, or 0.001% work on forecasting time scales, and 0.0000001% work on infinite ethics, but the interestingness heuristic leads to people doing 50x as much work as is optimal on the second area in each pair, the first ten thousand EAs won't end up overinvesting in any of them - but over time, if EA scales, we'll see a problem.

On the specific topics, I'm not saying that infinite ethics is literally worthless, I'm saying that even at 1 FTE, we're wasting time on it. Perhaps you view that as incorrect on the merits, but my claim is, tentatively, that it's already significantly less important than a marginal FTE on anything else on the GPI agenda.

Lastly, I think we as a community are spending lots of time discussing rationality. I agree it's no-one's full time job, but it's certainly a lot of words every month on lesswrong, and then far too little time actually creating ways of applying the insights, as CFAR did when building their curriculum, albeit not at all scalably. And the plan to develop a teachable curriculum for schools and groups, which I view as almost the epitome of the applied side of increasing the sanity waterline, was abandoned entirely. So we're doing / have done lots of interesting theory and writing on the topic, and much too little of value concretely. (With the slight exception of Julia's book, which is wonderful.) Maybe that's due to something other than how it was interesting to people, but having spent time on it personally, my inside view is that it's largely the dynamic I identified.

Sean_o_hJul 28 20224

A clarification that CSER gets some EA funds (combination of SFF, SoGive, BERI in kind, individual LTFF projects) but likely 1/3 or less of its budget at any given time. The overall point (all these are a small fraction of overall EA funds) is not affected.

DavidmanheimJul 31 20221

I'll just note that lots of what CSER does is much more policy relevant and less philosophical compared to the other orgs mentioned, and it's harder to show impact for more practical policy work than it is to claim impact for conceptual work. That seems to be part of the reason EA funding orgs haven't been funding as much of their budget.

calebpJul 28 202216

(I think CSER has struggled to get funding for a some of its work, but this seems like a special case so I don't think it's much of a counter argument)

I think if this claim is true it's less because of motivated reasoning arguments/status of interesting work, and more because object level research is correlated with a bunch of things that make it harder to fund.

I still don't think I actually buy this claim though, it seems if anything easier to get funding to do prosaic alignment/strategy type work than theory (for example).

Howie_LempelJul 28 20229

I agree with Caleb that theoretical AIS, infinite ethics, and rationality techniques don't currently seem to be overprioritized. I don't think there are all that many people working full-time on theoretical AIS (I would have guessed less than 20). I'd guess less than 1 FTE on infinite ethics. And not a ton on rationality, either.

Maybe your point is more about academic or theoretical research in general? I think FHI and MIRI have both gotten smaller over the last couple of years and CSER's work seems less theoretical. But you might still think there's too much overall?

My impression is that there's much more of a supply of empirical AI safety research and, maybe, theoretical AI safety research written by part-time researchers on LessWrong. My impression is that this isn't the kind of thing you're talking about though.

There's a nearby claim I agree with, which is that object level work on specific cause areas seems undervalued relative to "meta" work.

Academic-like research into interesting areas of AI risk is far easier to get funded by many funders than direct research into, say, vaccine production pipelines.

My guess is that this has less to do with valuing theory or interestingness over practical work, and more to do with funders prioritizing AI over bio. Curious if you disagree.

DavidmanheimJul 28 20226

First, yes, my overall point was about academic and theoretical work in general, and yes, as you pointed out, in large part this relates to how object level work on specific cause areas is undervalued relative to "meta" work - but I tried to pick even more concrete areas and organizations because I think that being more concrete was critical, even though it was nearly certain to have more contentious specific objections. And perhaps I'm wrong, and the examples I chose aren't actually overvalued - though that was not my impression. I also want to note that I'm more concerned about trajectory rather than numbers - putting aside intra-EA allocation of effort, if all areas of EA continue to grow, I think many get less attention than they deserve at a societal level, I think that the theoretical work should grow less than other areas, and far less than they seem poised to grow.

And as noted in another thread, regarding work on infinite ethics and other theoretical work, I got a very different impression at the recent GPI conference - though I clearly have a somewhat different view of what EAs work on compared to many others since I don't ever manage to go to EAG. (Which they only ever have over the weekend, unfortunately.) Relatedly, on rationality techniques, I see tons of people writing about them, and have seen people who have general funding pending lots of time thinking and writing about it, though I will agree there is less recently, but (despite knowing people who looked for funding,) no-one seems interested in funding more applied work on building out rationality techniques in curricula, or even analysis of what works.

Lastly, on your final point, my example was across the domains, but I do see the same when talking to people about funding for theoretical work on biosafety, compared to applied policy or safety work. But I am hesitant to give specific examples because the ones I would provide are things other people have applied for funding on, whereas the two I listed were things I directly worked on and looked for funding for.

Chris LeongJul 29 20225

"Who is actually working on infinite ethics?"- I'm actually very interested in this question myself. If you know anyone working on infinite ethics, please connect me :-).

kokotajlodJul 28 202241

He more recently mentioned that he noticed “people continuously vanishing higher into the tower,” that is, focusing on more abstract and harder to evaluate issues, and that very few people have done the opposite. One commenter, Ben Weinstein-Raun, suggested several reasons, among them that longer-loop work is more visible, and higher status.

I disagree that longer-loop work is more visible and higher status, I think the opposite is true. In AI, agent foundations researchers are less visible and lower status than prosaic AI alignment researchers, who are less visible and lower status than capabilities researchers. In my own life, I got a huge boost of status & visibility when I did less agent foundationsy stuff and more forecasting stuff (timelines, takeoff speeds, predicting ML benchmarks, etc.).

MichaelDickensJul 29 20228

If you are working with fast feedback loops, you can make things and then show people the things. If you're working with slow feedback loops, you have nothing to show and people don't really know what you're doing. The former intuitively seems much better if your goal is status-seeking (which is somewhat my goal in practice, even if ideally it shouldn't be).

DavidmanheimJul 28 20223

Yes, in AI safety that seems correct - it's still probably more interesting to do more theoretical work, but it is less prestigious or visible.

kokotajlodJul 29 20223

I don't even know that it's more interesting. What's interesting is different for different people, but if I'm honest with myself I probably find timelines forecasting more interesting than decision theory, even though I find decision theory pretty damn interesting.

DavidmanheimJul 29 20225

If it is any indication, I'm plenty skeptical of the value of marginal timelines work - it's a very busy area, and one that probably needs less work than it gets - I suspect partly because it has lots of visibility.

kokotajlodJul 29 20222

Fair enough!

LinchAug 3 202220

The incentive gradient within EA (always, or usually) points people towards work with longer feedback loops than shorter feedback loops.
The incentive gradient within EA rewards you for academia than for more direct work in EA.
The incentive gradient within EA rewards you for more theoretical work than more applied work

I think I agree with a limited version of:

People seem drawn to "bigger picture", more abstract work over narrow, concrete, specific problems.
People may naturally be overly inclined to do theoretical work over applied work
- But I think this is in spite of EA incentive gradients, not because of them.
Perhaps in some specific circumstances, EA social incentives need to be stronger to point people away from interestingness and towards importance.

Some collection of personal experiences:

On the forum: I feel like I get many more points (in the sense of recognition, but also literally EAF karma) per hour for commenting than for top-level posts. The former is faster and has locally better feedback loops.
On my day job (research manager): I feel like the incentive gradient (especially locally) points me to doing local activities like day to day management, giving reviews/feedback on people's research, and away from forming deeper clarity/high-level strategic thinking.
- This despite everyone involved generally being supportive and thinking the latter work is very important.
Grantmaking vs. conceptual research: I think per hour, I gain more recognition, encouragement, etc., from grantmaking than for forming conceptual views and research. The former also has much faster feedback loops.
- (This is ameliorated by me personally finding grantmaking much less interesting/fun than my other projects).
Friends in ML academia vs alignment labs: I feel like my friends in academia usually get less points when they want to stay in academia, as opposed to joining Anthropic/Redwood.
- And more locally, they seem to get more prestige points per hour for stuff like hosting podcasts, grantmaking, LW posts, etc, than from writing papers and deepening their understanding of ML.
  - (This isn't a clear misallocation. Naively you'd expect people staying in academia to get more prestige points from academia/the outside world, so EA giving them relatively less points seem like an appropriate redistribution).
How I got into EA research: I got into EA research in 2020 via one of the EAish projects with the fastest feedback loops ever (coronavirus forecasting). I suspect I'd be a lot slower to either a) find internal traction or b) external engagement/recognition if I were to try to start with, e.g., AI strategy or some of the other research projects I was excited about in early 2020 (e.g. heavy-tailedness of talent).

(I've also been on the receiving end of such advice)

[epistemic status: idle, probably mean, speculation]

Another important issue is that “it's easier to pass yourself off as a long-looper when you're really doing nothing,” but that’s a different discussion.

Perhaps not, too. I'm far from confident that I pinpointed the exact problem or solution.

___

I also want to quickly note that your post conflates "theoretical/conceptual work" and "slow feedback loops." I think this is only sort of true.

^{^}
An important caveat here is that I think the rest of the world will privilege a concrete win over a theoretical win. So for example, most core EAs would consider "being one of the three most important people instrumental in causing EAs to work on AI alignment as a major cause area" to be more impressive than e.g. founding Wave. But I expect almost anybody else in the world to consider founding Wave a bigger deal.

Stefan_SchubertJul 28 202217

I think that we as a community need to put status in better places.

My sense is that it's relatively hard (though not completely infeasible) to change the status of particular jobs by just trying to agree to value them more.

An alternative strategy is to pay more for impactful roles that you can't otherwise fill. That would directly incentivise people to take them, and probably increase their status as well.

DavidmanheimJul 28 20223

I think that as individuals, the practice of noticing that we have accorded more status than our values imply is correct is itself worthwhile, and pushes in the correct direction.

That said, I certainly agree that it's hard to coordinate about status, especially directly, and indirect signals like funding and salaries are critical - though I'd argue that it should happen at the level of choosing how generously to fund different types of organizations, rather than how to set individual salaries within those organizations. (But separately, substantively I mostly agree with you about salaries and compensation variance.)

Stefan_SchubertJul 28 20222

I'd argue that it should happen at the level of choosing how generously to fund different types of organizations, rather than how to set individual salaries within those organizations.

I'm not sure I follow. My argument is (in part) that organisations may want to pay more for impactful roles they otherwise can't fill. Do you object to that, and if so, why?

DavidmanheimJul 28 20223

I agree that orgs should pay more of their budget for impactful roles they can't otherwise fill, but I also think that orgs overall should get relatively less funding to do more abstract work, and relatively more to do things that are directly impactful.

EmrikAug 21 202214

I'm generally skeptical of arguments of the form "we probably have a bias in favour of X, so we should do less X" without having an underlying model that lets you understand why you should deprioritise it. It's like when you walk with a limp in one leg and decide to start limping with your other leg to balance it out, instead of figuring out how to heal your leg in the first place (ht. Eliezer for this metaphor). Moves like that are shortsighted, and doesn't take us to a greater theoretical understanding of how to walk faster.

If the reason you're biased in favour of X (interesting) is because you don't intuitively care about Y (impactfwl) enough, then the solution is to figure out how to intuitively care more about Y. This can involve trying therapy, or it can involve growing a community that helps shape your intuitions to be more in line with what you reflectively care about.

"longer-loop work is more visible, and higher status"

Well, depends. I think legible, technical, concrete, results-producing work is sometimes overweighted because it often looks impressive and more like "doing actual work". Whereas I think backchaining is incredibly important, and working on nodes far away from our current technical frontier is almost always going to look wishy-washy and fail to produce concrete results. Unfortunately, what you call "longer-loop" work is often hard to verify, so there will be unaligned people just having fun for fun's sake, but that's not an indictment of the activity itself, or of work that just looks superficially similar.

As a deconfusion researcher, and as part of a notoriously un-paradigmatic field lacking a clear formalization, I feel like that regularly. From where I come from, math just looks more like research and actually doing work, instead of simply talking about stuff. And there is definitely a lot of value in formalization and its use to unveil confusing parts of what we’re investigating.
-- Adam Shimi

In conclusion, we have a lot of biases in all kinds of directions, and we should be wary of Goodharting on them. But the way to do that is to learn to see and understand those biases so we can optimise more purely. The solution is not to artificially add a constant weight in the opposite direction of whatever biases we happen to notice exists sometimes.

DavidmanheimAug 21 20223

This seems exactly right, and worth thinking much more about - thanks!

EmrikAug 21 20221

Oi, I didn't expect that response. Nice of you to say. Thank you!

Stefan_SchubertJul 28 202211

A third strategy is selecting for people who are (somewhat) resilient to status and monetary incentives

Thanks for raising this possibility. However, I think EA has already effectively selected for quite low sensitivity to monetary incentives at least. Also, selecting more for insensitivity to status and monetary incentives than we do now would likely make EA worse along other important dimensions (since they're likely not much correlated with such insensitivity). I also think it's relatively hard to be strongly selective with respect to status insensitivity. For these reasons, my sense is that selecting more for these kinds of incentive insensitivity isn't the right strategy.

Yonatan CaleJul 28 20225

I suggest that [pointing out a bias, like "interesting", that leads more people to some direction] and [suggesting that we should put slightly less focus on that direction because of that bias] isn't such good decision making

DavidmanheimJul 28 202211

I don't understand your model here. You think that it's wrong because it's bad to actively work to counteract a bias, or because you don't think the bias exists, or because it will predictably lead to worse outcomes?

Yonatan CaleJul 29 20223

Because [actively working to correct for a bias] is less good than [figure out what the correct unbiased answer should be]

Especially when the bias is "do X a bit more"

(there are probably some other ways I'd pick where I would or wouldn't use this strategy, but TL;DR: Deciding how many people should do something like ai-safety seems like a "figure out the correct solution" and not "adjust slightly for biases" situation. Do you agree with that part?)

Yonatan CaleJul 29 20224

As a naive example to make my point more clear:

"People are biased not to work on AI Safety because it often seems weird to their families, so we should push more people to work on it" - I don't actually believe this, but I am saying that we can find biases like these to many many directions (and so it's probably not a good way to make decisions)

What do you think?

DavidmanheimJul 29 20222

I think that it's reasonable to think about which biases are relevant, and consider whether they matter and what should be done to account for them. More specifically, AI safety being weird-sounding is definitely something that people in EA have spent significant effort working to counteract.

Yonatan CaleJul 28 20226

Also, "interesting" is subjective. Different people find different things to be interesting. I was surprised you called "theoretical research" interesting (but then reminded myself that "interesting" is subjective)

Jack RJul 29 20223

"Interesting" is subjective, but there can still be areas that a population tends to find interesting. I find David's proposals of what the EA population tends to find interesting plausible, though ultimately the question could be resolved with a survey

DavidmanheimJul 28 20222

Any given person should look at what they find most interesting, and make sure to double-check that they aren't claiming it's impactful because they enjoy doing it. This was the point of the conclusion, especially footnote 6.

konradJul 29 20224

Maybe a helpful reframe that avoids some of the complications of "interesting vs important" by being a bit more concrete is "pushing the knowledge frontier vs applied work"?

Many of us get into EA because we're excited about crucial considerations type things and too many get stuck there because you can currently think about it ~forever but it practically contributes 0 to securing posterity. Most problems I see beyond AGI safety aren't bottlenecked by new intellectual insights (though sometimes those can still help). And even AGI safety might turn out in practice to come down to a leadership and governance problem.

oh54321Jul 29 20223

"Academic work should get less status than concrete projects."

This seems like a pretty strong claim and I'm not sure I agree with it. While yeah, sure, I'm guessing people working on theory sometimes end up doing too much of what's interesting and not enough of what's the most useful, I'm guessing projects are also mostly things that people find interesting but might not be useful .

Theory = not useful, projects = useful doesn't seem to exactly hit what I think the problem is. I'm guessing theory researchers specifically gravitate towards the bits of theory that seem the most interesting, projects people gravitate towards the most flashy/interesting projects, and these are the problems.

DavidmanheimJul 31 20222

I didn't say don't do academic work, I said that across different areas, there is more emphasis on interesting things, and therefore I think everyone should push the other direction - and one way to do that is less status for academic-style work.

oh54321Jul 29 20221

Although to be clear, I do think it's probably correct that this tends to happen more with academic work.

NickGabsAug 4 20221

I think the key thing here is that the criterion by which EA intellectuals decide whether something is interesting is significantly related to it being useful. Firstly, because a lot of EA's are intellectually interested in things that are at least somewhat relevant to EA, lots of these fields seem useful at least at a high level; moral philosophy, rationality, and AI alignment are all clearly important things for EA. Moreover, many people actually don't find these topics interesting at all, and they are thus actually highly neglected. This is compounded by the fact that they are very hard, and thus probably only quite smart people with good epistemics can make lots of progress on them. These two features in turn contribute to the work being more suspiciously theoretical than it would be if the broad domains in question (formal ethics, applied rationality, alignment) were less neglected, as fields become increasingly applied as they become better theorized. In other words, it seems prima facie plausible that highly talented people should work in domains that are initially selected partially for relevance to EA and that are highly neglected due to being quite difficult and also not as interesting to people who aren’t interested in topics related to EA, and thus more theoretical than they would be if more people worked on them.

Yonatan CaleJul 28 20221

I'm considering working at arxiv and think that would be really interesting. Any pushbacks?

DavidmanheimJul 28 20225

"To minimize distortion due to interpersonal dynamics, you probably want to ask a group of people to all give you feedback, probably via a google doc, with the option to do so anonymously."