The waste of human resources caused by poor selection procedures should pain the professional conscience of I–O psychologists.^[1]

Short Summary: I think that effective altruist organizations can do hiring a lot better.^[2] If you are involved in hiring, you should do these three things: avoid unstructured interviews, train your interviewers to do structured interviews, and have a clear idea of what your criteria are before you start to review and filter applicants. Feel free to jump to the “Easy steps you should implement” section if you don’t want to go through all the details.

Intro

If we view a hiring process as an attempt to predict which applicant will be successful in a role, then we want to be able to predict this as accurately as we can. That is what this post will be about: how can we increase the accuracy of our predictions for something that is as complex as human beings.^[3] Whenever I write about predicting success in this post, it is about trying to predict whether (and to what extent) a job applicant would be successful if hired.

This post will be full of suggestions about hiring. Like all suggestions anyone gives you about how to do things, feel free to ignore aspects that you think aren’t applicable to your situation.

What this article is not, and epistemic status

This isn’t original research. This is mostly just me summarizing things I’ve read, with a dash of my own perspectives sprinkled in here and there. I think it should be quite clear from context which ideas are my own, but if you have any questions about that please do make a comment and I will clarify.

This also isn’t legal advice for employment law, for avoiding adverse impact, etc. 

This isn’t an operational plan for how to implement a hiring system. While I have designed hiring systems in the past and most good hiring systems should make use of the techniques/tools described here, describing how to put these ideas into practice would be a separate post.

And of course, the standard caveat that these general recommendations might not fit the particular situation you face.

Although I asked a few strangers to review this, this is not a professionally edited piece of writing: all errors are my own.

My background

I ran a good deal of operations and a good deal of hiring (as well as some other HR functions) for a couple of small organizations (headcount 20-60), and I had some insight into the hiring practices of a larger organization (headcount of about 2500) when I worked as a director in the HR department there. I've read some books that are relevant to hiring; this includes books on mistakes/biases in decision making, books specifically about human resources, as well general business books which have given me relevant tidbits.^[4] I am wrapping up a graduate level course in Organizational Behavior^[5] as part of my effort to earn a master's degree in management. I've recently become aware of the Scientist–Practitioner Gap (sometimes called the The Research-Practice Gap), and I'm interested in it. I've discovered a few academic papers recently by following and exploring the sources cited in my textbook and in my professors lectures, and Stubborn Reliance on Intuition and Subjectivity in Employee Selection has inspired me to write this post.

I want to emphasize that I am not an expert on personnel selection. I am a man who has a few years of work experience in operations and HR at small organizations, and who designed hiring processes for a couple of startup-y teams. Over the past few months I've read some relevant journal articles, over the past several years I've read a lot of relevant popular press articles and books, and I am about 1/12 of the way through a relevant master's degree.^[6] I'd encourage you to view this not as an expert advising you on how to do something, but rather as advice from someone who is interested in this topic, is learning more about this topic, and wants to share some knowledge.

What the research says about hiring

How to do Interviews

Use Structured Interviews

I'm going to quote from my textbook, because I think it does a fairly good job of describing the problem with unstructured interviews.

The popular unstructured interview—short, casual, and made up of random questions—is simply not a very effective selection device. The data it gathers are typically biased and often only modestly related to future job performance... Without structure, interviewers tend to favor applicants who share their attitudes, give undue weight to negative information, and allow the order in which applicants are interviewed to influence their evaluations.

You want to avoid bias, right? You want to have something better than the unstructured interview, right? Fortunately, there is a simple solution readily available: use a structured interview. Again, quoting from from my textbook:

To reduce such bias and improve the validity of interviews, managers should adopt a standardized set of questions, a uniform method of recording information, and standardized ratings of applicants’ qualifications. Training interviewers to focus on specific dimensions of job performance, practicing evaluation of sample applicants, and giving interviewers feedback on how well they were focused on job-relevant characteristics significantly improves the accuracy of their ratings.

One thing that I want to emphasize is that you need to have this all prepared before you interview the first applicant. Ideally, you should have clear criteria set before you start fielding applicants. If you value qualities/traits A, B, and C in applicants, you should define what minimum score on a scale of A, or B, or C is required in order for an applicant to do the job well.

If you want to read about the research, you can start with the paper A Counterintuitive Hypothesis About Employment Interview Validity and Some Supporting Evidence and Is more structure really better? A comparison of frame‐of‐reference training and descriptively anchored rating scales to improve interviewers’ rating quality, but the short version is that structured employment interviews have higher validity for predicting job performance than do unstructured interviews.^[7] There is a lot more research that I want to read about structured, unstructured, and behavioral interviews, but I haven't gone through all of it yet.

As a small note, I also recommend that you avoid asking brainteaser questions. Google was famous for these, but has since disavowed them.^[8]

Use Behavioral interviews

Behavioral interview questions are a particularly effective type of interview question. I spent about four years working for a company at which a major part of the product/service was conducting behavioral interviews, where I conducted several hundred of them and served as operations manager. Once every few weeks I would fly to a different city and conduct a day or two’s worth of behavioral interviews. Although it has been a few years since I worked there, I do consider myself quite familiar with behavioral interviews.

Behavioral interviews are simple and easy: ask about past behavior.

• Instead of asking applicants what would you do in situation X? ask them tell me about a time you faced situation X.
• Instead of asking applicants how do you tend to deal with Y? ask them The last time you dealt with Y, how did you handle it?
• Rather than asking tell me about how to design Z, ask last time you designed Z, how did you do it?

My former colleagues and I used to joke that we could just stick a "tell me about a time when" at the beginning of any question to make it behavioral. The textbook definition of a behavioral interview question is to require "applicants to describe how they handled specific problems and situations in previous jobs, based on the assumption that past behavior offers the best predictor of future behavior."

Not all questions can be made behavioral. If you want to ask an applicant if they would be willing to move to a new city, or when they are able to start or if they have a particular certification, you should just ask those directly. But to the extent possible I encourage you to try and make many questions behavioral.

Edit: here is a list of behavioral questions that I put together, adapted from the US Department of Veteran's Affairs Performance Based Interviewing. Feel free to use and adapt these as you see fit.

Note: I've updated my opinion regarding behavioral questions. While a 2014 meta-analysis suggests that situational questions have slightly lower criterion-validity than behavioral questions for high-complexity jobs, this does not mean that only behavioral questions should be used, as both situational questions and behavioral questions have good validity for most jobs. I haven't yet delved into the research about what counts as a "high-complexity job." This section on behavioral interviews should be viewed as mostly personal opinion rather than a summary of the research.

Applicant–organization fit

You might want to use an interview to evaluate how well a person would mesh with other people at your organization. People often describe this as cultural fit or as a cultural interview, but the academics seem to call it applicant–organization fit. While in theory I do think there is value in working with people that you enjoy working with, in practice I'm wary of this type of filtering, because it sounds like we are just using motivated reasoning to find a way to let our biases creep in. Paul Green (who taught my Organizational Behavior class) said that if he designed a good hiring system he would be happy to hire people without worrying about fit at all, instead just using an algorithm to accept or reject. I know that people tend to be resistant to outsourcing decisions to an algorithm,^[9] but I have also read so much about how we tend to be good at deceiving ourselves that I am starting to give it more consideration. However, I haven't read much about this yet, so as of now I still don't have much to say on this topic. I’d love to hear some perspectives from other people, though.

I also think that people influenced by the culture of EA are particularly susceptible to a “hire people similar to me” bias. From what I’ve observed, EA organizations ask questions in interviews that have no relevance to an applicant’s ability to perform the job. These types of questions will create a more homogeneous workforce, but rather than diving into that issue more here, I’ll consider writing a separate post about it.

Use Work Sample Tests

Using work sample tests seems to already be very common within the culture of EA, so I won't spend a lot of time on them here. While accurately predicting applicant success is difficult, work sample tests seem to be one of the best predictors. I'll quote from Work Rules, by Lazlo Bock:

Unstructured interviews... can explain only 14% of an employee’s performance. The best predictor of how someone will perform in a job is a work sample test (29 percent).^[10] This entails giving candidates a sample piece of work, similar to that which they would do in the job, and assessing their performance at it.

Thus, if the nature of the job makes it feasible to do a work sample test, you should have that as part of your hiring process.

Intelligence Tests & Personality Tests

Regarding personality tests, it is pretty widely accepted among researchers that conscientiousness is a predictor of work success across a wide variety of contexts,^[11] but that the details of the job matter a lot for using personality to predict success. It is also well established that cognitive ability is quite predictive of work success in a wide variety of contexts, but I am concerned about the many biases involved in testing cognitive ability.^[12]

I’m intrigued by this topic, and there is quite a bit of research on it. But I haven't read much of the research yet and I don't have any experience in using these types of tests in a hiring process. My starting point is that I am very wary of them in terms of the legal risks, in terms of ethical concerns, and in terms of whether or not they are actually effective in predicting success. Scientists/researchers and job applicants tend to have very divergent views about these assessments.^[13] There is also a plethora of companies pushing pseudoscience “snake oil” solutions involving tests and a few buzzwords (AI, automate). Thus, rather than typing a few paragraphs of conjecture I’ll simply state that I am cautious about them, that I urge EA organizations to only implement them if you have very strong evidence that they are valid, and that I hope to write a different post about intelligence tests and personality tests after I'm more familiar with the research on their use in hiring.

Additional thoughts

I've put some ideas here that I like and that I think are helpful for thinking about hiring. However, as far as I am aware there isn't much scientific research to support these ideas. If you know of some research related to these ideas, please do let me know.

Tailoring the hiring process to the role

Like so many things in life, the details of the particular situation matter. Deciding how to design a hiring campaign will change a lot depending on the nature of the job and of the organization. It is hard to give clear advice in a piece of writing like this, because I don’t know what the specific context of your open positions are. One helpful step that is relatively easy is to think about which of these two scenarios your organization is in for the particular role you are hiring:

• It is easy for us to find people to do the job. We have too many qualified applicants, I don’t want to take the time to review them all, so I am going to filter out a bunch of them and then look at the ones that remain. I need a more strict hiring process. This gives a very low chance to make bad hires, but at the cost of missing out on good hires, thus you will get some false negatives.
• We need more qualified applicants. We are having trouble finding people who are able to do the job. I need to be more lenient in filtering out and eliminating candidates. This gives a very low chance to miss out on good hires, but at the cost of a higher probability of hiring some bad fits as well, thus you will get some false positives.

My rough impression is that most roles in EA fall into the first category (a strict hiring process, accepting of false negatives). One method I’ve used in the past to deal with this kind of situation is what I call an auto-filtering quiz^[14]. Depending on how the respondent answers the questions in the quiz, the applicant will see either a rejection page or a link for the next step in the process (such as a link to upload a resume and a calendly link to book an interview). The auto-filtering quiz can automate some very clear/strict filtering, allowing less time to be spent manually reviewing applications. The upfront cost is that the quiz would have to be designed for each distinct role, because the traits we want to filter for are different for each role.

As an example, for a position for a Senior Front End Developer only applicants that answer “yes” to all of the following three questions would progress to the next stage, and any applicant that does not answer “yes” to all three of these questions would be shown the rejection page:

• “Do you have three or more years of full-time work experience in a Front End role?”
• “Are you able to use HTML, CSS, and JavaScript?”
• “Do you have consistent and reliable internet access?”

I do think that such a quiz should be used with caution, as it can make the hiring process feel fairly impersonal for the applicant. Depending on what population you want to recruit from, some applicants may strongly prefer human contact rather than to use a quiz, so much then when prompted to do the quiz they choose to drop out of the application process.^[15] But I have found it to be quite effective in preventing me from manually reviewing dozens and dozens of applications that don’t meet minimum requirements. If you do use a quiz, I recommend only using it for initial screening, not for secondary or tertiary screening.

Irreducible Unpredictability

I like the idea of irreducible unpredictability that Scott Highhouse describes in Stubborn Reliance on Intuition and Subjectivity in Employee Selection. He writes that most of the variance in success is simply not predictable prior to employment. I think that he described it fairly well, so I'll quote him rather than putting it into my own words.

The business of assessment and selection involves considerable irreducible unpredictability; yet, many seem to believe that all failures in prediction are because of mistakes in the assessment process. Put another way, people seem to believe that, as long as the applicant is the right person for the job and the applicant is accurately assessed, success is certain. The ‘‘validity ceiling’’ has been a continually vexing problem for I–O psychology (see Campbell, 1990; Rundquist, 1969). Enormous resources and effort are focused on the quixotic quest for new and better predictors that will explain more and more variance in performance. This represents a refusal, by knowledgeable people, to recognize that many determinants of performance are not knowable at the time of hire. The notion that it is still possible to achieve large gains in the prediction of employee success reflects a failure to accept that there is no such thing as perfect prediction in this domain.

Scott Highhouse included the below Figure 2 in the paper which I think helps to solidify the idea in memory. 

My own summary of the idea is that people think hiring can be done so well as to have a very high success rate, and refuse to believe that the process involves so much uncertainty. This meshes well with a lot of what I read regarding people having difficulty thinking probabilistically and people refusing to accept that high rates of failure are inherent to a particular task.

In a similar vein, here is my paraphrasing of the same idea from Personnel Selection: Adding Value Through People: In hiring, the very best methods of predicting performance only account for about 25% to 33% of the variance in performance. It may not be realistic to expect more. This is primarily because performance at work is influenced by other factors that are orthogonal to the individual (management, organizational climate, co‐workers, economic climate, the working environment).

So you should be aware that if you do everything optimally, using all the best tools and all the best processes, you will be able to predict less than 50% of the variability in performance. If I was more of a marketer, I would encourage people to adopt humble hiring: being aware that we are flawed and biased decision-makers, and even when using all the right tools we generally make suboptimal decisions.

A holistic hiring process

This is an idea that I was introduced to by Paul Green. I had a vague idea of it previously, but I like how explicitly he described it.^[16]

He described two approaches that one can use in hiring. The approach on the left is a linear if-else linear model, in which applicants are eliminated at each stage if they don't meet the requirement. After taking applicants through these stages, you are left with a pool of applicants who meet all your requirements. The approach on the right is what he describes as more like a regression model, in which you give weights to various traits depending how important they are. I view a major benefit of the regression model is if an applicant is abnormally strong in one area, this can make up for a slightly weaker ability in another area. He suggests that using a holistic regression model rather than an if-else model, will result in more successful hiring, although I haven't seen research about this yet.

Rather than a hiring process in which applicants are eliminated step by step for not meeting the standard A, or for not meeting standard B, Paul Green encourages a more holistic approach, in which various aspects are considered in order to evaluate the applicant.

The benefit of the if-else model is its simplicity: it is relatively easy to design and execute. It also has a relatively low chance of making a bad hire (assuming that the criteria you use to filter are good predictors of success). The downside of the if-else model is that it is can be overly strict in filtering out and eliminating applicants, meaning that you will miss out on good hires (potentially the best hires). The benefit of the regression model is that you have a very low chance of missing out on a good hire. However, the cost is that it requires extra resources to design a more complicated hiring process.

Get feedback from applicants

This is another idea that I first learned from Lazlo Bock’s book, Work Rules. Creating a survey to send out to every applicant that you interview is probably the easiest thing to do among all of the recommendations here. You can use the template that is provided at re:Work or you can make your own survey.

I’d encourage you to make it anonymous so that applicants are clear that this survey doesn’t influence whether or not they progress through the selection process. I’d also encourage you to send the link to both applicants that “pass” the interview as well as those that “fail” the interview. You want to better understand applicants’ experiences in the hiring process, regardless of whether they received a job offer. You can use this feedback to improve the hiring process.

Easy steps you should implement

Easy is a relative term.^[17] Although these are some of the easier actions to take for improving your hiring process, they will still take effort to implement.

Use work sample tests

My impression is that this seems to be a common practice within EA culture, so I won't spend much time on this. If the nature of the job allows you to use a work sample test, I encourage you to use one.

Use structured interviews, avoid unstructured interviews

Unstructured interviews make it difficult to compare applicant A to applicant B, they allow biases to creep in, and they are not as predictive of success as structured interviews. Create a set of questions addressing the traits/qualities you want to evaluate in applicants, create a rubric, train your interviewers, and score each answer an applicant gives according to the rubric. If you need some help to start doing structured interviews, start with re:Work.

Get feedback from applicants 

You can use a survey such as Google Forms, Survey Money, Airtable, or whatever other system you prefer. In order to improve the hiring process, you want to get feedback from the people involved in the hiring process.

Have a plan before you start the hiring campaign

This is a bit more vague, and is more of a personal opinion rather than a summary of research-supported best practices. This is also somewhat related to the idea of a holistic hiring process that I mentioned above.

I think that anybody running hiring should be really clear about what knowledge, skills, abilities and other characteristics (KSAOs) are actually important, and what level of each KSAOs you need. You can figure this out by looking at the people who are already doing the job successfully and figuring out what KSAOs lead to their success. This is called job analysis, and while doing a full job analysis can be very tedious and time-consuming, sometimes a quick and dirty version is sufficient to get some benefits. As a simplistic example, rather than simply stating that a successful applicant needs “good excel skills” you could instead specify that a successful applicant “is able to complete at least 6 out of the 10 tasks on our internal excel test without referring to any resources.” Anything above 6 makes the applicant more competitive, but anything below 6 disqualifies him/her.^[18]

Be clear about must haves, and nice to haves. When I was hiring a junior operations manager for a former employer, the only real must haves were good communication skills, basic computer literacy, and an optimization mindset. There were lots and lots of nice to haves (strong spreadsheet skills, experience with data analysis, familiarity with Asana and Google Sheets, excellent attention to detail, comfort giving and accepting feedback, experience managing projects and people, etc.). For that hiring campaign, we weren't considering applicants that only had the must haves; they needed the must haves plus a good combination of as many nice to haves as possible.^[19]

I also think that you should decide in advance how to select among the qualified applicants. Thus, if you have multiple applicants that all pass your requirements and all seem that they would be good fits but you only want to hire one person, which one should you hire? Should you hire the overall best one? The cheapest one that can do the job to an acceptable level of quality^[20]? One that gives you the most bang for buck (maybe he costs 80% of the best applicant but can produce 95% of his workload)? The answer will vary depending on your specific circumstances, but I recommend that you consciously make the decision. This is something that I have opinions about, but very little reading about. Maybe I'll have more to say about this after reading a more of the research on personnel selection.

Questions, concerns, and topics for further investigation

If you have answers for any of these items or relevant readings you can point me to (even just a nudge in the right direction), I'd love to receive your suggestions. It is possible that some of these are common knowledge among practitioners or among researchers, and perhaps I'll find answers to some of these questions during the coming months as I continue to work my way through my to-read list.

• Certainly there is research on personnel selection at non-profit or impact-focused organizations, and I just haven't read this research yet, right?
• For some jobs, it is easy to create work sample tests, but for a lot of knowledge work jobs it is hard to think up what a good work sample test could be. I'd love to see some kind of a work sample test example book, in which example work samples for many different jobs are written out. Does something like this exist?
• How to create good structured interviews? The gap between this is more effective than unstructured interviews and here are the details of how to implement this seem fairly large. Are there resources that go more in-depth than re:Work?
• How can an organization effectively use intelligence tests & personality tests in hiring, while avoiding/minimizing legal risks? I know that many are rubbish, but are they all varying degrees of rubbish, or are some of them noticeably better than the rest? How good is Wonderlic? The gap between this can help you select better applicants and here are the details of how to implement this seem fairly large. What research has been done in the recent two decades? Are there any intelligence tests that aren't rife with cultural bias? I plan to explore this, but if you have suggestions I'd love to hear them. As of now, I would like to read Assessment, Measurement, and Prediction for Personnel Decisions, and also to explore papers related to and papers citing The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.
• I'm considering reading Essentials of Personnel Assessment and Selection or Personnel Selection: Adding Value Through People - A Changing Picture in order to learn more of the research on hiring. Are there any better textbooks that you can recommend to me?

Sources and further reading

• Organizational Behavior, by Stephen P. Robbins & Timothy A. Judge. This is the textbook for my Organizational Behavior class.
• Work Rules!: Insights from Inside Google That Will Transform How You Live and Lead. I first read this book in 2017 and I really enjoyed it. It was one of the things that really opened my eyes to how impactful good design of an organization's systems/processes can be. I got a lot of good ideas about how to run an organization in general, and there are some good concepts specifically related to hiring in Chapter 3 through Chapter 5. It isn't peer reviewed, but it does cite sources so you can track down the research if you want. If you are brand new to HR work or to hiring and you want something that is more easily digestible than an academic paper or a textbook, then I recommend this book.
• Stubborn Reliance on Intuition and Subjectivity in Employee Selection, an academic paper written by Scott Highhouse which got me thinking about why so many organizations have such bad hiring processes, and which gave me the inspiration to write this post.

1. ^{^}

   This is a quote from the paper Stubborn Reliance on Intuition and Subjectivity in Employee Selection, written by Scott Highhouse.

2. ^{^}

   Actually, I think that most organizations (both EA and non-EA) do hiring fairly poorly and could do a lot better.

3. ^{^}

   I want to emphasize the complexity. I used to think of social sciences as more fuzzy than the hard sciences, but now I simply view them as having more variables. The simple example I think of is dropping a rock. If I am a physicist and I want to know what will happen to the sand when I drop the rock, then I can figure that out by running some trials. But what if the shape of the rock is different each trial? What if the size of the sand grains vary each time? What if the rock's relationship with her mother and how much sleep she got last night  and her genetics all affect how fast it falls? I think of social sciences sort of like that: theoretically we could isolate and control all the variables, but realistically it isn't feasible. I wish I could just do randomized control trials, having each applicant do the job one hundred times in one hundred different identical universes, and then see which applicant was successful in the highest number of trials. But as one of my favorite quotes in all of the social sciences says, "god gave all the easy problems to the physicists."

4. ^{^}

   My partner tells me that I read an impressive number of books each year and that I should brag about it, but I haven't yet found a way to do that without making myself cringe due to awkward boasting. Thus, I'll just leave it at "quite a few." However, if you want recommendations for books on a particular topic I am happy to share recommendations.

5. ^{^}

   Organizational Behavior has a lot of overlap with I/O Psychology, and I generally view them as almost the same field. However, I am quite new to both of them and I don't feel qualified to give a good description of how they differ, nor of the exact borders of each field.

6. ^{^}

   I am required to take twelve courses to get a full degree and I am just finishing up my first course, so we'll see how far I get. Doing a degree as a working adult while maintaining all the other aspects of life (household, job, relationships, etc.) feels very different from being a full-time student living on campus.

7. ^{^}

   Paraphrased from A Counterintuitive Hypothesis About Employment Interview Validity and Some Supporting Evidence.

8. ^{^}

   These are questions such as: “Your client is a paper manufacturer that is considering building a second plant. Should they?” or “Estimate how many gas stations there are in Manhattan” or “How many golf balls would fit inside a 747?” There’s no correlation between fluid intelligence (which is predictive of job performance) and insight problems like brainteasers. (excepted from  Work Rules, by Lazlo Bock)

9. ^{^}

   You can read more about this topic in the paper Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them 

10. ^{^}

    The data that Lazlo Bock refers to here comes from The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.

11. ^{^}

    Although there is at least one paper that suggests there is a negative relationship between conscientiousness and managerial performance. This is something I want to look into more.

12. ^{^}

    If you want a quick description of the biases, this excerpt from Work Rules is a quick overview: "The problem, however, is that most standardized tests of this type discriminate against non-white, non-male test takers (at least in the United States). The SAT consistently underpredicts how women and non-whites will perform in college. Phyllis Rosser’s 1989 study of the SAT compared high school girls and boys of comparable ability and college performance, and found that the girls scored lower on the SAT than the boys. Reasons why include the test format (there is no gender gap on Advanced Placement tests, which use short answers and essays instead of multiple choice); test scoring (boys are more likely to guess after eliminating one possible answer, which improves their scores); and even the content of questions (“females did better on SAT questions about relationships, aesthetics, and the humanities, while males did better on questions about sports, the physical sciences, and business”). These kinds of studies have been repeated multiple times, and while standardized tests of this sort have gotten better, they are still not great."

13. ^{^}

    Some interesting comments and discussions here.

14. ^{^}

    Here is an example from a company I used to work for. They’ve changed it since I was there, but it still serves as a decent demonstration of a simple auto-filtering survey.

15. ^{^}

    A Chinese company I did some consulting for described to me how non-Chinese applicants are fairly comfortable filling out a form to submit their information, but Chinese applicants don't want to fill out a form to submit their info; Chinese applicants want to talk to an HR person via a phone call for a “quick interview.”

16. ^{^}

    To my knowledge, Paul Green doesn't use the word "holistic" to describe this. The use of the word "holistic" to describe this is my own choice.

17. ^{^}

    Easy in the way that learning Spanish or German is easy for a monolingual English speaker: It still takes a lot of effort, but far less than learning Arabic or Mandarin or Basque.

18. ^{^}

    Note that this is nothing more than a simplistic example. In reality I wouldn’t recommend rejecting an applicant simply because she/he doesn’t know how to use INDEX MATCH in excel. Skills that can be learned to a satisfactory level of performance in less than an hour or that can be easily referenced should, in my opinion, generally not be used to filter applicants out.

19. ^{^}

    I think that the analogy of classes for a degree in an American university is helpful here. In order to graduate with a particular major you have to take some specific classes within that major/department. You also have to take some electives, but you get to choose which electives you take from a list of approved classes. I was looking for somebody that has all the core classes, plus a good combination of electives.

20. ^{^}

    The idea that the resource (in this context the resources is a person) able to complete the task to a sufficient quality should do the task is described by Manager Tools as Managerial Economics 101.