Added 2022-10-20: To clarify up-front, the main point of this article is not to give hiring advice. More nuance is required for that. The main point is to tell you that you have to be careful with hiring and especially with outsourcing hiring.

Who makes up an organization, a community? People. People do the work. People make the decisions. Choosing people well is the most important thing we do. People choice is self-reinforcing, too. Choose one bad COO and he’ll hire more bad people. Have a few bad people in an organization and nobody good will want to work there anymore. Hiring, therefore, is vital.

What do you want from a hiring process? A good hire. Crucially, no bad hire. And for those people whom you haven’t hired to be mostly happy with how things went. Because you care for them.

Sadly, you’re at risk of making a bad hire and disgruntling your other applicants if you don’t know what you’re doing. If you don’t know what you’re doing, outsourcing isn’t a solution, either, because you don’t know how to judge the actions of those you’re outsourcing to. I will demonstrate this by example of a hiring process I’ve observed as an outsider, in which the hiring firm (call them Hirely) acted in a way that would have seemed sensible to the average founder who knows little about hiring, but to me looked like blundering. Even if you don’t plan to outsource hiring, the following points are worth thinking about.

Added 2022-10-16: I won't be arguing every point fully. One commenter even wrote that I make ‘lots of general assertions without a clear explanation as to why people should believe [me]’. (I appreciate this comment.) That's because doing otherwise would have made the article ten times longer. Hiring is a wide field and I've only tilled a small patch of it myself. I encourage you to follow the links to Manager Tools podcasts/whitepapers that I've included in the article. They argue many of the claims properly. I will also be glad to explain more in the comments.

Added 2022-10-22: A commenter points out, and I agree, that there are ways to outsource something safely without knowing about it. In the case of hiring, one way is to find strong evidence that the hiring processes run by a firm have resulted in many good hires, vanishingly few bad hires and the people who weren't hired being mostly happy with how things went. Although if you're paying attention to these three things, you already know something about hiring. (The above is not exactly what the commenter wrote. His comment is here. If you read it, please note that he wrote it based on ‘casually listening’ to my post and also read my reply, which counters many of his claims.)

Aside: Manager Tools is a decent authority

In the two main sections I will repeatedly reference Manager Tools in support of my arguments. Why is Manager Tools (MT) a decent authority to reference? (By the way, I’m in no way affiliated with or paid by MT. They have no idea that I’m writing this.)

Their core guidance is based on data

As far as I know, MT measure how well their core guidance works for their clients. And they adjust accordingly. Since they’ve presented their data on one-on-ones (mto3s) and feedback (mtfeedback), you can judge for yourself whether their research makes sense. They haven’t presented their data on hiring behaviours, but I suspect that they do have a lot of it. Additionally, the authors of MT’s hiring guidance, Wendii Lord and Mark Horstman, were recruiters themselves and claim to have in sum spent thousands (tens of thousands?) of hours interviewing. If this is true (and I have little reason to doubt it), they must have learned a thing or two about what works and what doesn’t.

Their guidance leads to success

Applying MT guidance has worked out well for many people. I conducted the selection phase of AI Safety Camp #5 (aisc5) based on MT hiring guidance (mthiringfeed). The participants whom my volunteers and I selected did well. One person thanked me when I rejected his application. And participants gave the selection phase the highest rating of all AISC selection phases so far.

I also followed MT guidance (mtinterv) for my last job search, writing résumés, preparing for the interview etc. It helped me get my current job at Spark Wave. On the other end of the job lifecycle, my wife used MT guidance (mtresign) to resign and she is still in good standing with her company, unlike others before her.

More generally, MT are a small company that has been training and advising organizations for a long time. They have been podcasting for seventeen years and get > 100 k downloads per week. And they appear to have made good hiring decisions themselves (mtsuccession).

Hirely puts you at risk of making a bad hire

They might not give you many (good) candidates

The more good applicants you have, the higher the likelihood that one of them will meet your standard. If you don’t have enough and none meets your standard, you will be tempted to hire the best of those who don’t meet your standard. This would be bad. Hirely, on the one hand, is good at getting many eyes on a job ad. On the other hand, the format of their job ad is a turn-off, especially for busy high-caliber people.

A good thing I’ve heard about Hirely is that they use marketing channels skillfully to get a job ad in front of many people. So they might bring in a decent number of candidates. The conversion from job ad and application form, however, likely won’t be as good as it could be. The drop-out throughout the hiring process would also be high because it’s intransparent and wastes people’s time.

At least the one job ad I’ve seen was almost three pages, with long sentences and big, empty words. Busy people don’t have time to read this. And good candidates tend to be busy because they’re good. Besides, we in the EA community care about imposter syndrome, right? I don’t have imposter syndrome, but to me,

Convert strategy into executable steps for growth, putting into practice operations planning, organization-wide goal setting, and performance management to scale the business effectively (quote rewritten to be not Googleable)

sounds much less like ‘I can do it’ than this:

Project planning and management: Take rough, high-level guidance from the CEO, turn it into concrete plans, then make sure that they’re carried out.

Further time wasters/points of intransparency:

• Irrelevant questions on the application form, such as this for an operations job: ‘What do you think the field of AI Alignment most needs?’ Would you rule out a good ops person because they haven’t thought about whole-field strategy? I hope not. If a question doesn’t help you make a hiring decision, why do you ask it?
• They ask applicants to take a Big Five assessment. At least the Big Five version I know can be gamed easily. Even integrity and conscientiousness tests that can't easily be gamed appear not predictive enough of job performance to be worth the applicants' time (selval). I concede that some assessments make sense in some hiring processes and I don't have space here to discuss that properly. The minimum viable point is: A Big Five assessment didn't make sense for this role. And Hirely's hiring process draft didn't mention taking it into account for any decision. It would have been a waste of applicant time. (Rewritten 2022-10-19. See also discussion.)
• They don’t outline the whole hiring process up-front. Most egregiously, the process draft I saw didn’t tell candidates about a work trial (1-2 weeks of full-time work?) until after some test task, the Big Five test and two interviews. (By the way, work trials are a nice idea, but in many cases don’t make sense practically. I don’t have space to discuss this here.)
• They haven’t figured out the whole process up-front, leading to delays between steps, which causes candidates to drop out. (Remember that people need a job and giving someone a job quickly is a competitive advantage. And you’re competing for good people.)

Their filters don’t work

What is bad? To turn down a candidate who did badly in the interview because of nervousness, only to later see them kill it at a different (possibly non-EA) org. What is far worse? To hire someone who did great in the interview, but then sucks at the job itself. Sadly, with Hirely you’re at risk to get both the bad and the worse.

How do you do better? By having filters that let nervous-but-awesome people shine and that expose the flaws in people who only seemed awesome. Call the former nervous aces and the latter confident duds. Sounds impossible? Not impossible, but of course difficult. (Added 2022-10-16: Another way to look at this is to think about spreading the field of applicants. (‘Spreading’ in the sense of ‘widening’, not ‘distributing’.) I'm too lazy to revise this section based on that frame. So just an example. – If I was interviewing for a senior developer position at GuidedTrack, I could ask: ‘Tell me about a time you successfully added two numbers using a pocket calculator.’ Since almost everyone can use a pocket calculator, the answers would barely distinguish candidates. If, however, I ask, ‘Tell me about a time you successfully fixed a defect in an interpreter’, it would open a wide gap between candidates who likely are and aren't fit for the job.)

I will first describe a way to do it right, then how Hirely does it wrong. Basically this is an extremely compressed selection of MT guidance. It is much too short to be a guide to hiring well. If you want to hire well, go to the source (mthiringfeed, mtehm). (Added 2022-10-21: Of course, MT is not the only valid source about hiring. Some of their guidance might not apply to your case. Some of it might be outperformed by more EA-fashionable approaches. But I think they're an excellent starting point, and you will do decently if you rely on their material and common sense alone.)

An inkling of how to filter well

(I’m focussing on structured behavioural interviews as the main filter. Further filters that work are structured technical interviews, work samples and IQ tests (selval). I have thoughts on those, but no space to include them here.)

1. Set a standard. A high standard. In the rest of the process, compare people against this standard. Do not compare them against one another. Comparing people against one another feeds your biases (‘I like that guy better’). Do not rank them. The best of ten candidates who don’t meet the standard still doesn’t meet the standard. (You may rank people at the very end when you have to decide whom to offer first.) In the whole process look for reasons to say no. Because hiring the wrong person results in Hell on Earth. (mtbarhigh)
2. Narrow the field of applicants by sorting résumés (mtresumes). (Yes, résumés are a flawed filter. If you care about false negatives, you need to modify the approach. I don’t have space to discuss this here.)
3. Design a structured, behavioural interview (mtcreation, mtbehavioural) that helps you compare candidates against the standard you set. Every question must help you find out whether a candidate meets the standard. In particular, it must dig up any reasons to say no. If it doesn’t help you make a decision, it’s a waste of time.
   • A structured interview is one with the same set of questions for each candidate and a fixed procedure for scoring each answer. See also the appendix. MT doesn’t go as far as assigning numeric scores, but their interview creation tool (mtcreation) at least provides criteria for each question.
   • A behavioural question sounds like this: ‘Tell me about a time when you successfully planned an event with more than twenty people.’ Ie. you ask an open-ended question about past behaviour. Past behaviour because that’s a hard fact: A confident dud can’t confabulate a shiny ‘I would do this and that‘ answer like he can for a scenario question. And even if he tells you about a past ‘success’, you can probe it until you get to its empty bottom. In contrast, the nervous ace gets more confident as you probe into the awesome details of her past behaviour, which she forgot to tell you about because she was nervous. (With confident duds I don’t mean people who are simply unfit for a certain job. They will do well somewhere else. And I don’t mean those who are confident and talk a lot, but still get the job done. I mean those people who get jobs easily because of their confidence and then cause harm. They are rare, but destructive.)
4. Narrow the field further by doing phone screens.
   • Here you decide whom you will spend hours of interviewing time on in the next step. Do you want to outsource this decision to a hiring consultant or HR? Only if you can trust them to know what you’re looking for. Often you can’t. (mthrscreen)
   • The way to do this is similar to a behavioural interview that is cut off after the second or third question. For details see mtphonescr.
5. Interview the remaining few candidates intensely.
   • Reduce stress. Be clear before the interview about what will happen how. Be as friendly as you can be. Smile. Start with small talk. See also mtstress. Reserve matters of salary until you make an offer. I don’t have data on this, but ‘salary negotiation’ must be one of the most dreaded parts of an interview. So leave it out. It has no bearing on whether the candidate can do the job.
   • Ask behavioural questions (see item 3) and probe (mtprobe). A candidate will seldom tell you everything you need to know. You must to find it out by asking follow-up questions.
   • Several people in your organization interview each candidate. Everyone uses the same set of questions, with exceptions for technical interviews (mtehm, p. 111).
     • Yes, this means that the candidate has to answer the same question repeatedly. But it won’t be the same because every interviewer will hear the answer differently and ask different follow-up questions.
     • You might say that this takes a lot of time from the candidate (and the organization) and you’re right. But you would only do this with a few candidates. And it’s in the interest of the candidate: Wouldn’t you be glad to see a manager take hiring seriously and and know that your potential colleagues will all be up to par? Wouldn’t you want to be listened to in detail before it’s decided whether you’ll be allowed to contribute the next two, five, ten years of productivity to an organization? (And mind you, a few hours is still mighty short for judging whether someone can do a job.) Finally, would you want to be hired by mistake into a role that doesn’t fit you? I would rather spend an hour more being interviewed than three months suffering in a job that wasn’t for me, only to be fired or resign and then be asked by every potential employer about that three-month engagement on my résumé.
     • This is also a way to reduce bias. Having diverse interviewers is an easy way to increase diversity on your team without introducing quotas or lowering standards for certain groups of people.
6. After a series of interviews hold an interview results capture meeting (mtcapture). Each interviewer has to say ‘hire’ or ‘don’t hire’ and justify this recommendation. One ‘don’t hire’ is generally enough to rule out the candidate, but you as the hiring manager make the final decision.

There is much more to know around offers, turning down candidates, onboarding etc. No space for that here.

How Hirely doesn’t filter well

Now how does Hirely do it wrong? In short, they are mediocre at setting a standard and many of their questions aren’t dispositive, ie. they don’t help find out whether the candidate meets the standard. They do the phone screen themselves and plan on only one further interview by the hiring manager. And they increase interviewee stress. I’m assessing this based on a draft hiring process document from Hirely. Since I was advising the hiring manager in parallel to Hirely, I’m familiar with the requirements for the role, which is a ‘COO’/high-level ops role.

To get into the worst detail first: scenario questions. A good candidate will have a lot to tell about past successes that match the requirements for the role you’re hiring for. (These don’t have to be suit-and-tie, commended-by-CEO ‘professional’ successes. If you’re hiring an intern and are looking for planning skills, having built a weather-proof tree house in ninth grade can be an adequate accomplishment to talk about.) Detailed information about past successes is highly useful, but Hirely barely asks about it. (Talk about disgruntling applicants!) Instead, their main interview appears to focus on scenario questions, which put the nervous ace on the spot while being a perfect opportunity for the confident dud to confabulate something that sounds great. For each follow-up you ask, she’ll invent a good answer. How do you know it’s the right answer? You don’t because it’s all hypothetical! And what if you’re dealing with a clever interviewee who asks you a question back? I’m being facetious. Case study interviewing is a well-known technique. But it’s hard. There is a lot more to a case study than asking a three-sentence scenario question. Stick to behavioural questions about the past.

Let me backtrack and point out that Hirely is using fixed criteria and scoring rubrics. This is good. Unfortunately, the criteria I can see in the draft are only mediocre. Some are irrelevant, such as ‘Strong familiarity with AI Safety and Existential Risks’. – This is nice to have. But you wouldn’t reject an application from a highly successful ops person who only heard about AI safety last week. – They are not specific, either. For example, when they ask for ‘high levels of conscientiousness’, every interviewer has to make up their own idea of where ‘high levels’ are.

I also have to point out that Hirely has good intentions. They try to save the hiring manager time by doing the phone screen themselves. And they plan one main interview with the hiring manager. This is internally consistent, but doesn’t make sense if you want to hire well. I’ve established above that one main interview is not enough (see also mtmult). You need to dig deep. You need to have multiple perspectives. Even if you don’t buy the diversity thing, wouldn’t you want to give your team a say about whom they have to work with for several years? If you’re going to spend a lot of time interviewing, you need to choose well whom to spend that time on. You can only outsource this decision if the consultants are intimately familiar with your hiring needs. And the previous paragraph hints that they are not.

Now, Hirely might say: ‘That’s why we record the phone screens.’ And later in the draft it sounds as if the hiring manager is supposed to watch all those recordings. First of all, if I have to spend time watching all the recordings, why would I not do the phone screens myself and ask the questions I want to ask? Second, recording stresses people. What else stresses people? Panel interviews (mtpanel). And questions about compensation! Both Hirely interviews are panel interviews (with good intentions: they want to assist the hiring manager). And they plan to discuss compensation at the end of the main interview. Once more, they’re putting the nervous ace at a disadvantage.

Coming back to questions. If a question doesn’t help you decide whether or not to hire, it’s a waste of both your and the candidate’s time. Additionally, it takes time away from questions that do help you decide. A Big Five assessment is one group of questions that don’t work. I’ve commented on this in the section about time wasters. Most questions in the phone screen aren’t good at digging up information, either. Example: ‘What are you really good at that might apply to this position?’ Candidate: ‘Er, I’m good at project management.’ Interviewer: ‘What else are you good at?’ Candidate: ‘Hm, event planning and, uh, accounting.’ Interviewer: ‘Give me an example of a time you used that skill.’ Candidate: ‘I used it for organizing last year’s orchid auction.’ – You’ve asked three questions, gotten three answers and still don’t know much more about the candidate’s ability. Of course, you could now start probing into that orchid auction piece. But Hirely doesn’t mention probing once. And you could have circumvented that awkward back-and-forth by instead asking an open-ended question about a particular skill you’re interested in: ‘Tell me about a time you successfully ran an event, including managing its finances.’

Hirely puts you at risk of disgruntling applicants

(This section is much shorter than the previous only because I’ve addressed many points above. It is important nevertheless.)

When I ran the applications process for AI Safety Camp, one applicant whom I had to turn down replied: ‘Thank you for your thoughtful and graceful rejection email.’ How do we treat candidates in a way that keeps them happy even when we have to reject their applications? We don’t waste their time during the process. We don’t increase their anxiety and imposter syndrome. And ideally we give them something useful, such as an option to get feedback. This is an aspect of hiring which Manuel Allgaier of EA Germany has made me especially aware of. There have been at least two popular posts on this forum about this, too: eaf1a, eafcost I don’t agree with all of that, but it is important feedback.

Unfortunately again, Hirely doesn’t appear to take it into account. Above I’ve addressed time wasters, anxiety (nervousness) and imposter syndrome. And you know what makes you as a company look even worse? Rejecting a candidate, who has gone through application form, test task, personality test, two interviews and a work trial … by email! That’s what Hirely is planning to do. Rejection hurts. It hurts only a little bit when it comes from a warm voice on the other end of the telephone (mtturndown). Show them that you care.

(I mention above that I sent rejection emails. That’s acceptable at earlier stages when the candidate hasn’t had to spend much time yet.)

My last gasp: In their rejection email, at least the one after the second interview, they preclude feedback. I grant that feedback in application processes is a difficult thing. It allows people to argue and, in the worst case, to sue you. So you don’t have to offer it explicitly. But precluding it outright makes you look bad. And if a candidate asks you and you do it right (mtdemo), you provide value even to someone you’ve had to turn down.

Added 2022-10-20: Let me remind you one more time, the main point of this article is not to give hiring advice. More nuance is required for that. The main point is to tell you that you have to be careful with hiring and especially with outsourcing hiring.

Appendix: On structured vs. unstructured interviews

Unstructured interviews have no fixed format or set of questions to be answered. In fact, the same interviewer often asks different applicants different questions. Nor is there a fixed procedure for scoring responses; in fact, responses to individual questions are usually not scored, and only an overall evaluation (or rating) is given to each applicant, based on summary impressions and judgments. Structured interviews are exactly the opposite on all counts. In addition, the questions to be asked are usually determined by a careful analysis of the job in question. As a result, structured interviews are more costly to construct and use, but are also more valid. – selval

References

• aisc5: https://aisafety.camp/2021/06/23/aisc5-research-summaries/
• eaf1a: https://forum.effectivealtruism.org/posts/jmbP9rwXncfa32seH/after-one-year-of-applying-for-ea-jobs-it-is-really-really
• eafcost: https://forum.effectivealtruism.org/posts/Khon9Bhmad7v4dNKe/the-cost-of-rejection
• lwspec: https://www.lesswrong.com/posts/XosKB3mkvmXMZ3fBQ/specificity-your-brain-s-superpower
• mtbarhigh: https://www.manager-tools.com/2007/04/effective-hiring-set-the-bar-high
• mtbehavioural: https://www.manager-tools.com/2008/06/how-create-simple-behavioral-interview-question
• mtcapture: https://www.manager-tools.com/2008/04/the-interview-results-capture-meeting
• mtcreation: https://www.manager-tools.com/products/interview-creation-tool
• mtdemo: https://www.manager-tools.com/2013/07/you-did-not-demonstrate-part-1-hall-fame-guidance
• mtehm: https://www.manager-tools.com/products/effective-hiring-manager-book
• mtfeedback: https://www.manager-tools.com/2021/03/manager-tools-data-feedback-part-1
• mthiringfeed: https://www.manager-tools.com/podcasts/important-topic-feeds/hiring-feed
• mthrscreen: https://www.manager-tools.com/2019/01/should-hr-do-my-phone-screens-i-interview
• mtinterv: https://www.manager-tools.com/products/interview-series
• mtmult: https://www.manager-tools.com/2011/01/conduct-multiple-interviews-chapter-1-part-1
• mto3s: https://www.manager-tools.com/2019/01/manager-tools-data-one-ones-part-1-hall-fame-guidance
• mtpanel: https://www.manager-tools.com/2016/08/no-panel-interviews
• mtphonescr: https://www.manager-tools.com/2015/12/how-do-phone-screen-interview-part-1
• mtprobe: https://www.manager-tools.com/2012/01/first-rule-probing-interview
• mtresign: https://www.manager-tools.com/2006/07/how-to-resign-part-1-of-3
• mtresumes: https://www.manager-tools.com/2016/05/how-scan-resume-part-1
• mtstress: https://www.manager-tools.com/2019/12/effective-hiring-manager-missing-chapters-reducing-interviewee-stress-part-1
• mtsuccession: https://www.manager-tools.com/2022/07/special-cast-succession-planning
• mtturndown: https://www.manager-tools.com/2014/11/how-turn-down-job-candidate-part-1
• selval: https://web.archive.org/web/20160227075513/http://lab4.psico.unimib.it/nettuno/forum/free_download/articolo_114.pdf

Background: This is a repurposed article with a history

(Expanded on 2022-10-16 from the last paragraph of the originally published introduction. Moved to the end of the article on 2023-08-22. This describes the article's history in boring detail.)

This article is strange because it's repurposed from a direct critique of the organization behind ‘Hirely’, which is a fictional name. (Please don't try to find out who is behind that name.) This is the first part of the article's history step-by-step:

1. I observe the hiring process in which Hirely is advising the hiring manager. (I'm also advising the hiring manager, mostly telling him to be more involved and listen to Manager Tools.)
2. I think Hirely is giving harmful advice.
3. I write this article as a direct critique.
4. I give the article to Hirely to react to. I tell them that I will edit it to be more general and not point the finger at them (meaning I won't out the organization by name) if they convince me that they're on a better trajectory. You may view this as being kind or you may view it as blackmail.
5. Hirely deliberates internally.
6. Hirely responds to me with the improvements they've made and are making, and asks me to deliver on my promise to edit the critique before publishing. Their response does make me think they're on a better path. (So again, please don't try to find out who they are.)
7. Since I'm too lazy to rewrite the whole damn article, I search and replace the organization name with ‘Hirely’, and rewrite only the introduction.

Despite this laziness, I was happy with the way it demonstrated the new main point with specificity (lwspec): If you know very little about X, you can’t safely outsource X. If X = hiring, it’s especially bad. So you better learn something about hiring.

You're already yawning, but the story isn't over. To explain all the strangenesses of this article, I have to describe the rest:

8. I publish the article.
9. It receives a lot of downvotes in addition to upvotes.
10. I add a prescript (opposite of postscript) asking for the downvoters to at least hint at what they don't like about this article.
11. alexrjl and Kirsten helpfully comment with their speculations why people might be downvoting. (Thank you, Alex and Kirsten!) None of the downvoters comment.
12. More downvotes and upvotes, steadying at 3 total karma from 41 votes.
13. It turns out that this article was mass-downvoted by anonymous accounts created for this purpose. The EA Forum team confirms this to me (and you'll see a note from them at the top of the comments).
14. The EA Forum team reverts the mass-downvoting.
15. The EA Forum team graciously reruns the article, which might be how you came to read it.