Against EA-Community-Received-Wisdom on Practical Sociological Questions

Michael_Cohen

Against EA-Community-Received-Wisdom on Practical Sociological Questions

Michael_Cohen

19 min readMar 9, 2023

202

Comments 35

Sorted by

New & upvoted

Rubi J. Hudson

How does Rationalist Community Attention/Consensus compare? I'd like to mention a paper of mine published at the top AI theory conference which proves that when a certain parameter of a certain agent is set sufficiently high, the agent will not aim to kill everyone, while still achieving at least human-level intelligence. This follows from Corollary 14 and Corollary 6. I am quite sure most AI safety researchers would have confidently predicted no such theorems ever appearing in the academic literature. And yet there are no traces of any minds being blown. The associated Alignment Forum post only has 22 upvotes and one comment, and I bet you've never heard any of your EA friends discuss it. It hasn't appeared, to my knowledge, in any AI safety syllabuses. People don't seem to bother investigating or discussing whether their concerns with the proposal are surmountable. I'm reluctant to bring up this example since it has the air of a personal grievance, but I think the disinterest from the Rationality Community is erroneous enough that it calls for an autopsy. (To be clear, I'm not saying everyone should be hailing this as an answer to AI existential risk, only that it should definitely be of significant interest.)

I'm someone who has read your work (this paper and FGOIL, the latter of which I have included in a syllabus), and who would like to see more work in similar vein, as well as more formalism in AI safety. I say this to establish my bona fides, the way you established your AI safety bona fides.

I don't think this paper is mind-blowing, and I would call it representative of one of the ways in which tailoring theoretical work for the peer-review process can go wrong. In particular, you don't show that "when a certain parameter of a certain agent is set sufficiently high, the agent will not aim to kill everyone", you show something more like "when you can design and implement an agent that acts and updates its beliefs in a certain way and can restrict the initial beliefs to a set containing the desired ones and incorporate a human into the process who has access to the ground truth of the universe, then you can set a parameter high enough that the agent will not aim to kill everyone" [edit: Michael disputes this last point, see his comment below and my response], which is not at all the same thing. The standard academic failure mode is to make a number of assumptions for tractability that severely lower the relevance of the results (and the more pernicious failure mode is to hide those assumptions).

You'd be right if you said that most AI safety people did not read the paper and come to that conclusion themselves, and even if you said that most weren't even aware of it. Very little of the community has the relevant background for it (and I would like to see a shift in that direction), especially the newcomers that are the targets of syllabi. All that said, I'm confident that you got enough qualified eyes on it that if you had shown what you said in your summary, it would have had an impact similar in scale to what you think is appropriate.

This comment is somewhat of a digression from the main post, but I am concerned that if someone took your comments about the paper at face value, they would come away with an overly negative perception of how the AI safety community engages with academic work.

JoshuaBlake

The standard academic failure mode is to make a number of assumptions for tractability that severely lower the relevance of the results (and the more pernicious failure mode is to hide those assumptions).

Perhaps, but at least these assumptions are stated. Most work leans on similarly strong assumptions (for tractability, brevity, or lack of rigour meaning you don't even realise you are doing it) but doesn't state them.

Michael_Cohen

I'm someone who has read your work (this paper and FGOIL, the latter of which I have included in a syllabus), and who would like to see more work in similar vein, as well as more formalism in AI safety. I say this to establish my bona fides, the way you established your AI safety bona fides.

Thanks! I should have clarified it has received some interest from some people.

you don't show that "when a certain parameter of a certain agent is set sufficiently high, the agent will not aim to kill everyone", you show something more like "when you can design and implement an agent that acts and updates its beliefs in a certain way and can restrict the initial beliefs to a set containing the desired ones and incorporate a human into the process who has access to the ground truth of the universe, then you can set a parameter high enough that the agent will not aim to kill everyone"

"When you can design and implement an agent that acts and updates its beliefs in a certain way and can restrict the initial beliefs to a set containing the desired ones". That is the "certain agent" I am talking about. "Restrict" is an odd word choice, since the set can be as large as you like as long as it contains the truth. "and incorporate a human into the process who has access to the ground truth of the universe." This is incorrect; can I ask you to edit your comment? Absolutely nothing is assumed about the human mentor, certainly no access to the ground truth of the universe; it could be a two-year-old or a corpse! That would just make the Mentor-Level Performance Corollary less impressive.

I don't deny that certain choices about the agent's design make it intractable. This is why my main criticism was "People don't seem to bother investigating or discussing whether their concerns with the proposal are surmountable." Algorithm design for improved tractability is the bread and butter of computer science.

Rubi J. Hudson

I'll edit to comment to note that you dispute it, but I stand by the comment. The AI system trained is only as safe as the mentor, so the system is only safe if the mentor knows what is safe. By "restrict", I meant for performance reasons, so that it's feasible to train and deploy in new environments.

Again, I like your work and would like to see more similar work from you and others. I am just disputing the way you summarized it in this post, because I think that portrayal makes its lack of splash in the alignment community a much stronger point against the community's epistemics than it deserves.

Michael_Cohen

Thank you for the edit, and thank you again for your interest. I'm still not sure what you mean by a person "having access to the ground truth of the universe". There's just no sense I can think of where it is true that this a requirement for the mentor.

"The system is only safe if the mentor knows what is safe." It's true that if the mentor kills everyone, then the combined mentor-agent system would kill everyone, but surely that fact doesn't weight against this proposal at all. In any case, more importantly a) the agent will not aim to kill everyone regardless of whether the mentor would (Corollary 14), which I think refutes your comment. And b) for no theorem in the paper does the mentor need to know what is safe; for Theorem 11 to be interesting, he just needs to act safely (an important difference for a concept so tricky to articulate!). But I decided these details were beside the point for this post, which is why I only cited Corollary 14 in the OP, not Theorem 11.

Michael_Cohen

Do you have a minute to react to this? Are you satisfied with my response?

Aaron Gertler 🔸

I think that most Forum users would agree with most of what you've written here, and I don't see much in the post that I consider controversial (maybe the peer review claim?) .

I gave the post an upvote, and I'm glad someone wrote down all these sensible points in one place, but I found the conventional wisdom and the vibe of "this will surprise you" to be an odd mix. (Though I might be more normie relative to the EA community than I realize?)

*****

As is the nature of comments, these will mostly be pushback/nitpicks, but the main thrust of the post reads well to me. You generally shouldn't take Forum posts as seriously as peer-reviewed papers in top journals, and you shouldn't expect your philosophy friends to take the place of a strong professional network unless there are unusual circumstances in play.

(That said, I love my philosophy friends, and I'm glad I have a tight-knit set of interesting people to hang out with whenever I'm in the right cities; that's not an easy thing to find these days. I also think the people I've met in EA are unusually good at many conventional virtues — honesty, empathy, and curiosity come to mind.)

If you want to be sure that a job done is well, don't hire an EA fresh out of college. Hire someone with a strong track record that a conventional HR department would judge as demonstrably competent. Companies hire people all the time who are not "aligned" with them, i.e. not philosophically motivated to maximize the company's profit, and it works out fine.

I've seen this debate play out many times online, but empirically, it seems to me like EA-ish orgs with a lot of hiring power (large budgets, strong brands) are more likely than other EA-ish orgs to hire people with strong track records and relevant experience.^[1]

My guess is that other orgs EA-ish would also like to hire such people in many cases, but find them to be in short supply!

People with a lot of experience and competence command high salaries and can be very selective about what they do. Naively, it seems hard to convince such people to take jobs at small charities with low budgets and no brand recognition.

When I put up a job application for a copywriting contractor at CEA , my applicants were something like:

60% people with weak-to-no track records and no EA experience/alignment
30% people with weak-to-no track records and some EA experience/alignment
10% people with good track records (some had EA experience, others didn't)

Fortunately, I had a bunch of things going for me:

CEA was a relatively established org with a secure budget; we gave a reliable impression and I could afford to pay competitive rates.
Writing and editing are common abilities that are easy to test.

This means I had plenty of people to choose from (10% was 20 applicants), and it was easy to see whether they had skills to go with their track records. I eventually picked an experienced professional from outside the EA community, and she did excellent work.

A lot of jobs aren't like this. If you're trying to find someone for "general operations", that's really hard to measure. If you're trying to find a fish welfare researcher, the talent pool is thin. I'd guess that a lot of orgs just don't have many applicants a conventional HR department would love, so they fall back on young people with good grades and gumption who seem devoted to their missions (in other words, "weak-to-no track records and some EA experience/alignment").

*****

I also think that (some) EA orgs do more to filter for competence than most companies; I've applied to many conventional jobs, and been accepted to some of those, but none of them put me through work testing anywhere near as rigorous and realistic as CEA's.

*****

If "grown-ups" had been involved at the officer level at FTX, I claim fraud probably would not have occurred. I can't say I predicted FTX's collapse, but I didn't know it was being run by people with no experience.

I'm not sure about the "probably" here: there are many counterexamples. Ken Lay and Jeff Skilling are the first who come to mind — lots of experience, lots of fraud.

If you think that FTX was fraud borne of the need to cover up dumb mistakes, that could point more toward "experience would have prevented the dumb mistakes, and thus the fraud". And lack of experience is surely correlated with bad business outcomes...

...but also, a lot of experienced businesspeople drove their own crypto companies to ruin — I grabbed the first non-FTX crypto fraud I found for comparison, and found Alex Mashinsky at the helm. (Not an experienced financial services guy by any means, but he had decades of business experience and was the CEO of a company with nine-figure revenue well before founding Celsius.)

Many small organizations are founded by EAs who would not succeed at being hired to run an analogous non-EA organization of similar scope and size. I (tentatively) think that these organizations, which are sometimes given the outrageously unsubstantiated denomination "effective organization", are mostly ineffective.

As people I've edited can attest, I've been fighting against the "effective" label (for orgs, cause areas, etc.) for a long time. I'm glad to have others alongside me in the fight!

Better alternatives: "Promising", "working in a promising area", "potentially highly impactful", "high-potential"... you can't just assume success before you've started.

On the "would not succeed" point: I think that this is true of EA orgs, and also all other types of orgs. Most new businesses fail, and I'm certain the same is true of new charities. This implies that most founders are bad at running organizations, relative to the standard required for success.

(It could also imply that EA should have fewer and larger orgs, but that's a question too complicated for this comment to cover.)

^{^}
Open Philanthropy once ran a research hiring round with something like a thousand applicants; by contrast, when I applied to be Stuart Russell's personal assistant at the relatively new CHAI in 2018, I think I was one of three applicants.

Tiresias

"I've seen this debate play out many times online, but empirically, it seems to me like EA-ish orgs with a lot of hiring power (large budgets, strong brands) are more likely than other EA-ish orgs to hire people with strong track records and relevant experience."

Based on speaking to people at EA orgs (and looking at the orgs' staff lists), I disagree with this. When I have spoken to employees at CEA and Open Phil, the people I've spoken to have either (a) expressed frustration about how focused their org is on hiring EA people for roles that seem to not need it or (b) defended hiring EAs for roles that seem to not need it. (I'm talking about roles in ops, personal assistants, events, finance, etc.)

Maybe I agree with your claim that large EA orgs hire more "diversely" than small EA orgs, but what I read as your implication (large EA orgs do not prioritize value-alignment over experience), I disagree with. I read this as your implication since the point you're responding to isn't focusing on large vs. small orgs.

I could point to specific teams/roles at these orgs which are held by EAs even though they don't seem like they obviously need to be held by EAs. But that feels a little mean and targeted, like I'm implying those people are not good for their jobs or something (which is not my intent for any specific person). And I think there are cases for wanting value-alignment in non-obvious roles, but the question is whether the tradeoff in experience is worth it.

Michael_Cohen

Upvoted this.

You generally shouldn't take Forum posts as seriously as peer-reviewed papers in top journals

I suspect I would advise taking them less seriously than you would advise, but I'm not sure.

It could also imply that EA should have fewer and larger orgs, but that's a question too complicated for this comment to cover

I think there might be a weak conventional consensus in that direction, yes. By looking at the conventional wisdom on this point, we don't have deal with the complicatedness of the question--that's kind of my whole point. But even more importantly, perhaps fewer EA orgs that are not any larger; perhaps only two EA orgs (I'm thinking of 80k and OpenPhil; I'm not counting CHAI as an EA org). There is not some fixed quantity of people that need to be employed in EA orgs! Conventional wisdom would suggest, I think, that EAs should mostly be working at normal, high-quality organizations/universities, getting experience under the mentorship of highly qualified (probably non-EA) people.

Aaron Gertler 🔸

I suspect I would advise taking them less seriously than you would advise, but I'm not sure.

The range of quality in Forum posts is... wide, so it's hard to say anything about them as a group. I thought for a while about how to phrase that sentence and could only come up with the mealy-mouthed version you read.

But even more importantly, perhaps fewer EA orgs that are not any larger.

Maybe? I'd be happy to see a huge number of additional charities at the "median GiveWell grantee" level, and someone has to start those charities. Doesn't have to be people in EA — maybe the talent pool is simply too thin right now — but there's plenty of room for people to create organizations focused on important causes.

(But maybe you're talking about meta orgs only, in which case I'd need a lot more community data to know how I feel.)

Conventional wisdom would suggest, I think, that EAs should mostly be working at normal, high-quality organizations/universities, getting experience under the mentorship of highly qualified (probably non-EA) people.

I agree, and I also think this is what EA people are mostly doing.

When I open Swapcard for the most recent EA Global, and look at the first 20 attendees alphabetically (with jobs listed), I see:

Seven people in academia (students or professors); one is at the Global Priorities Institute, but it still seems like "studying econ at Oxford" would be a good conventional-wisdom thing to do (I'd be happy to yield on this, though)
Six people working in conventional jobs (this includes one each from Wave and Momentum, but despite being linked to EA, both are normal tech companies, and Wave at least has done very well by conventional standards)
One person in policy
Six people at nonprofit orgs focused on EA things

Glancing through the rest of the list, I'd say it leans toward more "EA jobs" than not, but this is a group that is vastly skewed in favor of "doing EA stuff" compared to the broad EA community as a whole, and it's still not obvious that the people with EA jobs/headed for EA jobs are a majority.

(The data gets even messier if you're willing to count, say, an Open Philanthropy researcher as someone doing a conventionally wise thing, since you seem to think OP should keep existing.)

Overall, I'd guess that most people trying to maximize their impact with EA in mind are doing so via policy work, earning-to-give,^[1] or other conventional-looking strategies; this just gets hidden by the greater visibility of people in EA roles.

I'd love to hear counterarguments to this — I've held this belief for a long time, it feels uncommon, and there's a good chance I'm just wrong.

^{^}
This isn't a conventional way to use money, but the part where you earn money is probably very conventional (get professional skill, use professional skill in expected way, climb the ladder of your discipline).

Michael_Cohen

This is very high-quality. No disputes just clarifications.

I don’t just mean meta-orgs.

I think working for a well-financed grantmaking organization is not outrageously unconventional, although I suspect most lean on part-time work from well-respected academics more than OpenPhil does.

And I think 80k may just be an exception (a minor one, to some extent), borne out of an unusually clear gap in the market. I think some of their work should be done in academia instead (basically whatever work it’s possible to do), but some of the very specific stuff like the jobs board wouldn’t fit there.

Also, if we imagine an Area Dad from an Onion Local News article, I don’t think he’s skepticism would be quite as pronounced for 80k as for other orgs like, e.g., an AI Safety camp.

Tiresias

Yeah, I'm not sure that people prioritizing the Forum over journal articles is a majority view, but it is definitely something that happens, and there are currents in EA that encourage this sort of thinking.

I'm not saying we should not be somewhat skeptical of journal articles. There are huge problems in the peer-review world. But forum/blogs posts, what your friends say, are not more reliable. And it is concerning that some elements of EA culture encourage you to think that they are.

Evidence for my claim, based on replies to some of Ineffective Altruism's tweets (who makes a similar critique).

1: https://twitter.com/IneffectiveAlt4/status/1630853478053560321?s=20 Look at replies in this thread

2: https://twitter.com/NathanpmYoung/status/1630637375205576704?s=20 Look at all the various replies in this thread

(If it is inappropriate for me to link to people's Twitter replies in a critical way, let me know. I feel a little uncomfortable doing this, because my point is not to name and shame any particular person. But I'm doing it because it seems worth pushing back against the claim that "this doesn't happen here." I do not want to post a name-blurred screenshot because I think all replies in the thread are valuable information, not just the replies I share, so I want to enable people to click through.)

JeremiahJohnson

-4

>I also think that (some) EA orgs do more to filter for competence than most companies; I've applied to many conventional jobs, and been accepted to some of those, but none of them put me through work testing anywhere near as rigorous and realistic as CEA's.

I want to push back here based on recent experience. I recently applied for a job at CEA and was essentially told that my background and experience was a perfect fit and that I aced the application but that I was not 'tightly value aligned' and thus would not be getting the role.

CEA certainly had an extensive, rigorous application process. They just proceeded to not value the results of that application process. I expect they will hire someone with demonstrably less competence and experience for the role I applied for, but who is more 'value aligned'.

I would normally hesitate to air this sort of thing in public, but I feel this point needs to be pushed back against. As far as I can tell this sort of thing is fairly endemic within EA orgs - there seems to be a strong, strong preference for valuing ideological purity as opposed to competence. I've heard similar stories from others. My small sample size is not just that this happens, but that it happens *openly*.

To relate this to the OP's main thesis - this is a problem other areas (for instance, politics) have already seen and confronted and we know how it plays out. It's fairly easy to spot a political campaign staffed with 'true believers' as opposed to one with seasoned, hardened campaign veterans. True Believer campaigns crash and burn at a much higher rate, controlling for other factors, because they don't value basic competence. A common feature of long time pols who win over and over is that they don't staff for agreement, they staff for experience and ability to win.

EA's going to learn the same lesson at some point, it's just a matter of how painful that learning is going to be. There are things EA as a movement is simply not good at, and they'd be far better off bringing in non-aligned outsiders with extensive experience than hiring internally and trying to re-invent the wheel.

Shakeel Hashim

Hi Jeremiah. I was the hiring manager here and I think there's been something of a misunderstanding here: I don't think this is an accurate summary of why we made the decision we did. It feels weird to discuss this in public, but I consent to you publishing the full rejection email we sent, if you would like.

JeremiahJohnson

-3

I don't particularly feel it would be a valuable use of anyone's time to get into a drawn out public back-and-forth debate where we both nitpick the implications of various word choices. I'll just say that if your intention was to communicate something other than "We prefer a candidate who is tightly value aligned" then there was a significant failure of communication and you shouldn't have specifically used the phrase "tightly aligned" in the same sentence as the rejection.

Jeff Kaufman 🔸

If the issue is that CEA communicated poorly or you misunderstood the rejection, I agree that's not necessarily worth getting into. But you've made a strong claim about how CEA makes decisions based on the contents of a message, whose author is willing to make public. It looks to me like you essentially have two choices:

Agree to make the message public, or
Onlookers interpret this as an admission that your claim was exaggerated.

Lumpyproletariat

I'm strongly downvoting the parent comment for now, since I don't think it should be particularly visible. I'll reverse the downvote if you release the rejection letter and it is as you've represented.

Aaron Gertler 🔸

I'm sorry you had such a frustrating experience. The work of yours I've seen has been excellent, and I hope you find a place to use your skills within EA (or keep crushing it in other places where you have the chance to make an impact).

Some hiring processes definitely revolve around value alignment, but I also know a lot of people hired at major orgs who didn't have any particular connection to EA, and who seem to just be really good at what they do. "Hire good people, alignment is secondary" still feels like the most common approach based on the processes I've seen and been involved with, but my data is anecdata (and may be less applicable to meta-focused positions, I suppose).

[anonymous]

The consensus of most people is that conventional wisdom is pretty good when it comes to designing institutions; at least compared to what a first-principles-reasoner could come up with.

I think your characterization of conventional answers to practical sociological questions is much too charitable, and your conclusion ("there is a rot in the EA community that is so consequential that it inclines me to discourage effective altruists from putting much, if any, trust in EA community members, EA "leaders", the EA Forum, or LessWrong") is correspondingly much too strong.

Indeed, EA grew out of a recognition that many conventional answers to practical sociological questions are bad. Many of us were socialized to think our major life goals should include getting a lucrative job, buying a nice house in the suburbs, and leaving as much money as possible to our kids. Peter Singer very reasonably contested this by pointing out that the conventional wisdom is probably wrong here: the world is deeply unjust, most people and animals live very hard lives, and many of us can improve their lives while making minimal sacrifices ourselves. (And this is to say nothing about the really bad answers societies have developed for certain practical sociological questions: e.g., slavery; patrilineal inheritance; mass incarceration; factory farming.)

More generally, our social institutions are not designed to make sentient creatures' lives go well; they are designed to, for instance, maximize profits for corporations. Amazon is not usually cited as an example of an actor using evidence and reason to do as much good as possible, and company solutions are not developed with this aim in mind. (Think of another practical solution Amazon workers came up with: peeing in bottles because they risked missing their performance targets if they used the bathroom.)

I agree with many of the critiques you make of EA here, and I agree that EA could be improved by adopting conventional wisdom on some of the issues you cite. But I would suggest that your examples are cherrypicked, and that "the rot" you refer to is at least as prevalent in the broader world as it is within EA. Just because EA errs on certain practical sociological questions (e.g., peer review; undervaluing experience) does not mean that conventional answers are systematically better.

Tiresias

Yeah, I strongly agree and endorse Michael's post, but this line you're drawing out is also where I struggle. Michael has made better progress on teasing out the boundaries of this line than I have, but I'm still unclear. Clearly there are cases where conventional wisdom is wrong -- EA is predicated on these cases existing.

Michael is saying on questions of philosophy, we should not accept conventional wisdom, but on questions of sociology, we should. I agree with you that the distinction between sociological and philosophical are not quite clear. I think you're example of "what should you do with your life" is a good example of where the boundaries blur.

Maybe, I think "sociological" is not quite the right framing, but something along the lines of "good governance." The peer review point Michael brings up doesn't fit into the dynamic. Even though I agree with him, I think "how much should I trust peer review" is an epistemic question, and epistemics does fall into the category where Michael thinks EAs might have an edge over conventional wisdom. That being said, even if I thought there was reason to distrust conventional wisdom on this point, I would still trust professional epistemic philosophers over the average EA here and I would find it hard to believe that professional epistemic philosophers think forums/blogs are more reliable than peer reviewed journals.

Michael_Cohen

What "major life goals should include (emphasis added)" is not a sociological question. It is not a topic that a sociology department would study. See my comment that I agree "conventional wisdom is wrong" in dismissing the philosophy of effective altruism (including the work of Peter Singer). And my remark immediately thereafter: "Yes, these are philosophical positions, not sociological ones, so it is not so outrageous to have a group of philosophers and philosophically-minded college students outperform conventional wisdom by doing first-principles reasoning".

I am not citing Amazon as an example of an actor using evidence and reason to do as much good as possible. I am citing it as an example of an organization that is effective at what it aims to do.

Ben Millwood🔸

Maybe I'm just missing something, but I don't get why EAs have enough standing in philosophy to dispute the experts, but not in sociology. I'm not sure I could reliably predict which other fields you think conventional wisdom is or isn't adequate in.

Michael_Cohen

In fields where it's possible to make progress with first-principles arguments/armchair reasoning, I think smart non-experts stand a chance of outperforming. I don't want to make strong claims about the likelihood of success here; I just want to say that it's a live possibility. I am much more comfortable saying that outperforming conventional wisdom is extremely unlikely on topics where first-principles arguments/armchair reasoning are insufficient.

(As it happens, EAs aren't really disputing the experts in philosophy, but that's beside the point...)

Michael_Cohen

So basically, just philosophy, math, and some very simple applied math (like, say, the exponential growth of an epidemic), but already that last example is quite shaky.

[anonymous]

I think the crux of the disagreement is this: you can't disentangle the practical sociological questions from the normative questions this easily. E.g., the practical solution to "how do we feed everyone" is "torture lots of animals" because our society cares too much about having cheap, tasty food and too little about animals' suffering. The practical solution to "what do we do about crime" is "throw people in prison for absolutely trivial stuff" because our society cares too much about retribution and too little about the suffering of disadvantaged populations. And so on. Practical sociological solutions are always accompanied by normative baggage, and much of this normative baggage is bad.

EA wouldn't be effective if it just made normative critiques ("the world is extremely unjust") but didn't generate its own practical solutions ("donate to GiveWell"). EA has more impact than most philosophy departments because it criticizes many conventional philosophical positions while also generating its own practical sociological solutions. This doesn't mean all of those solutions are right—I agree that many aren't—but EA wouldn't be EA if it didn't challenge conventional sociological wisdom.

(Separately, I'd contest that this is not a topic of interest to sociologists. Most sociology PhD curricula devote substantial time to social theory, and a large portion of sociologists are critical theorists; i.e., they believe that "social problems stem more from social structures and cultural assumptions than from individuals... [social theory] argues that ideology is the principal obstacle to human liberation.")

Jeff Kaufman 🔸

I think an exaggerated version of this is a big part of what went wrong with Leverage.

Chris Leong

I wasn’t really a fan of framing this as a “rot”. I worry that this tilts people towards engaging with this topic more emotionally rather than rationally.

I thought you made some good points, however: Regarding peer review, I expect that one of the major cruxes here is timelines and whether engaging more with the peer-review system would slow down work too much. Regarding hiring value-aligned people, I thought that you didn’t engage much with the reasons why people tend to think this is important (ability to set people ill-defined tasks which you can’t easily evaluate + worry about mission drift over longer periods of time).

Tiresias

One thing I struggle with in discourse, is expressing agreement. Agreeing seems less generative, since I often don't have much more to say than "I agree with this and think you explain it well." I strongly agree with this post, and am very happy you made it. I have some questions/minor points of disagreement, but want to focus on what I agree with before I get to that, since I overwhelmingly agree and don't want to detract from your point.

The sentiment "we are smarter than everyone and therefore we distrust non-EA sources" seems pervasive in EA. I love a lot about EA, I am a highly engaged member. But that sentiment is one of the worst parts about EA (if not the worst). I believe it is highly destructive to our ability to achieve our aims of doing good effectively.

Some sub-communities within EA seem to do better at this than others. That being said, I think every element of EA engages in this kind of thinking to some extent. I don't know if I've ever met any EA who didn't think it on some level. I definitely have a stream of this within me.

But, there is a much softer, more reasonable version of that sentiment. Something like "EA has an edge in some domains, but other groups also have worthwhile contributions." And I've met plenty of EAs who operate much more on this more reasonable line than the excessively superior sentiment described above. Still, it's easy to slip into the excessively superior sentiment and I think we should be vigilant to avoid it.

------

Onto some more critical questions/thoughts.

My previous epistemics used to center around "expert consensus." The COVID-19 pandemic changed that. Expert consensus seemed to frequently be wrong, and I ended up relying much more on individuals with a proven track record, like Zeynep Tufekci. I'm still not sure what my epistemics are, but I've moved towards a forecasting based model. Where I most trust people with a proven track record of getting things right, rather than experts. But it's hard to find people with this proven track record, so I almost always still default to trusting experts. I certainly don't think forum/blog posts fit into this "proven track record" category, unless it's the blog of someone with a proven track record. But "proven track record" is still a very high standard. Zeynep is literally the only person I know who fits the bill. My worry with people using a "forecaster > expert" model is they won't have a high enough standard for what qualifies someone as a trust worthy forecaster. And it's not like I trust her on everything. I'm wondering what your thoughts are on a forecaster model.

Another question I have is that the slowness of peer-review does strike me as a legitimate issue. But I am not in the AI field at all so I have very little knowledge. I still would like to see AI researchers make more efforts to get their work peer-reviewed, but I'm wondering if there might be some dual system, where less time sensitive reports get peer reviewed and are treated with a high-level of trust, and more time-sensitive reports do not go through as rigorous of a process, but are still shared, albeit with a lower level of trust. I'm really not sure, but some sort of dual system seems necessary to me. It can't be we totally disregard all non peer-reviewed work?

Ben Millwood🔸

I upvoted this post and am strongly in favour of more institutional expertise in the community.

However, to take the example of peer review specifically, my sense is that:

academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc
people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,
the peer review system didn't obviously arise from a really intentional and thoughtful design effort (though maybe this is just my historical ignorance?), and there are institutional-inertia reasons why it would be hard to replace with something better even once we had a good proposal,
at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didn't exist, so it really arose in an environment quite different from the one it's now in.

It feels to me both like the consensus view isn't as strongly in favour of peer review as you suggest, and that there are some structural reasons to think that the dominance of peer review in academic contexts isn't so strong an indicator of its fitness.

I recognise my above criticisms have holes in them, but they seemed worth airing anyway, to at least gesture at why people might end up where they are on this topic.

(Also, obviously I've done nothing to demonstrate that forum posts aren't worse on every axis. I just think that if we're really in the situation of "we should use this moderately terrible system instead of this extremely terrible system", we need to acknowledge that if we're going to get people who can see the terribleness on board.)

Michael_Cohen

academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc

I think we could quibble on the scale and importance of all of these points, but I'm not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?

people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,

Again, compared to what alternative? I'm guessing they would say that at a company, the "idea people" get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to "productize" it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).

the peer review system didn't obviously arise from a really intentional and thoughtful design effort

This is definitely not a precondition for a successful social institution.

at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didn't exist, so it really arose in an environment quite different from the one it's now in.

But conventional wisdom persists in endorsing peer-review anyway!

Ben Millwood🔸

This is definitely not a precondition for a successful social institution.

I want to differentiate two kinds of success for a social institution:

"reproductive" success, by analogy with evolution: how well the institution establishes and maintains itself as dominant,
success at stated goals: for peer review, success at finding the truth, producing high quality research, etc.

Your argument seems to be (at least in part) that because peer review has achieved success 1, that is strong evidence that it's better than its alternatives at success 2. My argument (in part) is that this is only true if the two kinds of success have some mechanism tying them together. Some example mechanisms could be:

the institution achieved reproductive success by means of being pushed really hard by people motivated by the desire to build and maintain a really high quality system,
the institution is easy to replace with better systems, and better systems are easy to try, so the fact that it hasn't been replaced must mean better systems are hard to find.

I don't think either of these things are true of peer review. (The second is true of AWS, for example.) So what's the mechanism that established peer review as the consensus system that relates to it being a high quality system?

(I'm not saying I have alternatives, just that "consensus means a thing is good" is only sometimes a good argument.)

Anonymous EA Forum user

For practical sociological questions that many people want to get right, if there is a conventional answer, you should go with the conventional answer

I'm not confident that this is right. I'm thinking about conventional answers to common questions (although I suppose it depends on what you define as sociological) that tend to be not great. Ideas such as

If you want to find a romantic partner, convention wisdom says to make yourself as appealing to as wide an audience as possible; remove quirks and be bland; don't be weird. Whereas I think that it would be far better to filter out the people that wouldn't be a good match.
If you want to be happy, convention wisdom says that you should get a well-paid and respectable job. Whereas much of the of the literature (both psychological research and Buddhist psychology) suggest other methods, such as this paper, or Happiness: Lessons from a New Science.

I'm sure there are others, and I don't want to make the claim that conventional answers are never right (for example, for finding a romantic partner having good conversational skills and grooming yourself are convention wisdom that I agree with). But I would be wary of the claim that "if many people want to get right and there is a conventional answer, you should go with the conventional answer."

Vasco Grilo🔸

Hi Michael,

Nice post!

Claim 1: If there is a large group of people trying to answer a question correctly, and they are trying (almost) as hard as you to answer it correctly, any consensus of theirs is more likely to be correct than you.

In general, I think the consensus of a group of experts is more accurate than that of the most knowledgable person or the whole crowd. However, the optimal size of the group of experts depends on the specific question, and it may often be much smaller than one thinks. From Mannes 2014:

Through both simulation and an analysis of 90 archival data sets, we show that select crowds of 5 knowledgeable judges yield very accurate judgments across a wide range of possible settings—the strategy is both accurate and robust.

Douglas Knight

So if you intuit that we can do better than peer review, I would recommend getting a PhD in economics under a highly-respected supervisor, and learn how to investigate institutions like peer review (against proposed alternatives!) with a level of rigor that satisfies high-prestige economists.

Wow, that's really specific. Are you trying to evoke Robin Hanson? Before following him, ask him if it's a good idea. I think he regrets his path.

Michael_Cohen

Robin Hanson didn't occur to me when I wrote it or any of the times I read it! I was just trying to channel what I thought conventional advice would be.

Comments

Curated and popular this week

Cultivating hope: calibrating the expectations for cultivated meat to end factory farming

PabloAMC 🔸·1w ago·Curated 4d ago·22m read

109

Maybe do the thing you wish CEA would do

alejoacelas 🔸·3d ago·2m read

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts: * Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I've found who knows them is really excited about what they do * NEST is an opinionated ideas-fi...

GWWC's 2025 impact evaluation (executive summary)

Aidan Whitfield🔸, Giving What We Can🔸·6d ago·2m read

This post presents the executive summary from Giving What We Can’s impact evaluation for 2025. At the end of this post we share links to more information, including the full report and...

Recent opportunities to take action

New round of digital minds funding opportunities at Longview

zdgroff, Longview Philanthropy·15h ago·2m read

We've Just Launched Our Grant Readiness Course

Deena Englander·4h ago·1m read

RP is looking for project founders in neglected animal areas

Rethink Priorities·4d ago·7m read

Aaron Gertler 🔸

I think that most Forum users would agree with most of what you've written here, and I don't see much in the post that I consider controversial (maybe the peer review claim?) .

*****

If you want to be sure that a job done is well, don't hire an EA fresh out of college. Hire someone with a strong track record that a conventional HR department would judge as demonstrably competent. Companies hire people all the time who are not "aligned" with them, i.e. not philosophically motivated to maximize the company's profit, and it works out fine.

My guess is that other orgs EA-ish would also like to hire such people in many cases, but find them to be in short supply!

When I put up a job application for a copywriting contractor at CEA , my applicants were something like:

60% people with weak-to-no track records and no EA experience/alignment
30% people with weak-to-no track records and some EA experience/alignment
10% people with good track records (some had EA experience, others didn't)

Fortunately, I had a bunch of things going for me:

CEA was a relatively established org with a secure budget; we gave a reliable impression and I could afford to pay competitive rates.
Writing and editing are common abilities that are easy to test.

*****

If "grown-ups" had been involved at the officer level at FTX, I claim fraud probably would not have occurred. I can't say I predicted FTX's collapse, but I didn't know it was being run by people with no experience.

I'm not sure about the "probably" here: there are many counterexamples. Ken Lay and Jeff Skilling are the first who come to mind — lots of experience, lots of fraud.

Many small organizations are founded by EAs who would not succeed at being hired to run an analogous non-EA organization of similar scope and size. I (tentatively) think that these organizations, which are sometimes given the outrageously unsubstantiated denomination "effective organization", are mostly ineffective.

As people I've edited can attest, I've been fighting against the "effective" label (for orgs, cause areas, etc.) for a long time. I'm glad to have others alongside me in the fight!

Better alternatives: "Promising", "working in a promising area", "potentially highly impactful", "high-potential"... you can't just assume success before you've started.

(It could also imply that EA should have fewer and larger orgs, but that's a question too complicated for this comment to cover.)

^{^}
Open Philanthropy once ran a research hiring round with something like a thousand applicants; by contrast, when I applied to be Stuart Russell's personal assistant at the relatively new CHAI in 2018, I think I was one of three applicants.

^{^}

This isn't a conventional way to use money, but the part where you earn money is probably very conventional (get professional skill, use professional skill in expected way, climb the ladder of your discipline).

Against EA-Community-Received-Wisdom on Practical Sociological Questions

Against EA-Community-Received-Wisdom on Practical Sociological Questions

High-Level Claims

EA Trust-Networks

Peer Review

Conventional Experience

Objections

Consequences

Gratitude to EA

Alternatives