Comment Permalink

Do you know what these researchers are measuring when looking at the "results" level?

If I'm understanding correctly, they are claiming that training increases some sort of result by 0.6 standard deviations, which seems huge. E.g. if some corporate training increased quarterly revenue by 0.6 sd's that would be quite shocking.

(I tried to read through the meta-analyses but I could only find their descriptions of how the four levels differ, and nothing about what the results level looks like.)

See in context

Research on developing management and leadership expertise

by Jamie_Harris

Mar 5 20201 min read 9

45

Career choiceAnimal Advocacy CareersEntrepreneurshipManagement & mentoringPracticalExpertiseOrganization strategy

Frontpage

We recently announced the launch of Animal Advocacy Careers, a new nonprofit that seeks to address the career and talent bottlenecks in the effective animal advocacy community. Since then, we have released the following brief research reports:

Following on from this, we have created a document to support individuals to identify management and leadership resources for self-development.

This research has been conducted in order to inform our first intervention; helping individuals in existing animal advocacy organisations develop their management and leadership skills. We also hope it may be of some use to organisations in the wider effective altruism community.

Stay up to date with our work by signing up to our newsletter.

45 Reactions

Mentioned in

36Are there robustly good and disputable leadership practices?

29EA Updates for March 2020

Comments9

Sorted by

New & upvoted

Click to highlight new comments since: Today at 2:48 PM

Ben_West🔸Mar 7 202010

Thanks so much for sharing this and doing this research!

Regarding this:

That high performance on measures of leadership effectiveness causes organisational success, rather than organisational success inspiring high performance on (or at least more positive evaluations of) measures of leadership effectiveness. Given that the research is almost exclusively correlational,[36] we cannot be confident that this assumption is correct. However, this seems to me to be intuitively likely.

The Halo Effect is a compendium of evidence to the contrary. Basically, leaders who are good at one thing (e.g. maximizing revenue) are considered to be good at everything else (e.g. being humble). It has great examples of how the exact same CEO behavior is described positively versus negatively as the company's stock price fluctuates.

I would recommend at least skimming the book – it has really helped me differentiate useful from less useful business research.

Jamie_HarrisMar 14 20203

Sounds good, I'll have a look at some point, thanks for the recommendation. To clarify, the implication is that the causal chain might be from good organisational outcomes to good evaluations on leadership evaluation instruments, rather than the other way round?

Ben_West🔸Mar 17 20202

To clarify, the implication is that the causal chain might be from good organisational outcomes to good evaluations on leadership evaluation instruments, rather than the other way round?

Yep.

Ben_West🔸Mar 7 20204

One thing I found really interesting about this research is statements like these:

Therefore, though transformational leadership has been contrasted to transactional leadership (with the former being suggested to be superior), the use of contingent reward behaviours seems similarly effective to transformational leadership.

It sounds very believable to me that ~0% of "nonobvious" leadership recommendations don't outperform a "placebo". (Or, as you suggest, are only good subject to contingencies like personal fit.)

I would be curious if doing this review gave you a sense of what the "control group" for leadership could be?

I'm imagining something like:

Your team has reasonably well defined goals
Your team has the ability to make progress towards those goals
Your team is not distracted from those goals by some major problem (e.g. morale, bureaucracy)

We might hypothesize that any team which meets 1-3 will not have its performance improved by "transformational" leadership etc.

Do you know if anyone has studied or hypothesized such a thing? If not, do you have a sense from your research of what this might look like?

Jamie_HarrisMar 14 20203

<<Do you know if anyone has studied or hypothesized such a thing?>>

No, but my research was very brief and focused mostly on reviews and meta-analyses rather than looking at the primary studies. So a more thorough research project (or interviews with the authors of the reviews and meta-analyses?) might reveal this sort of thing better.

<<If not, do you have a sense from your research of what this might look like?>>

I'm not sure I'm quite following, but if you are asking what the control group looks like currently:

Usually the research is just correlational. So implicitly, it's comparing high scores to low scores on the same scales.

Some training evaluations have used experimental or quasi-experimental designs, randomising participants to receive training or receive no training, I believe.

If you're asking if I can imagine there being more targeted sorts of research that address your specific hypothesis ("any team which meets 1-3 will not have its performance improved by 'transformational' leadership etc"), I can certainly imagine it. E.g. you run a similar experiment across lots of different organisations, and beforehand, you assign the organisation some sort of score (subjective rankings out of 10?) for each of those variables and see whether there are correlations. But 1) this sounds like a very intensive research programme, 2) I don't usually see social scientists and academics use subjective rankings as variables, perhaps because it seems less replicable by other researchers and less rigorous?

Ben_West🔸Mar 7 20204

Do you know what these researchers are measuring when looking at the "results" level?

(I tried to read through the meta-analyses but I could only find their descriptions of how the four levels differ, and nothing about what the results level looks like.)

DirectedEvolutionMar 6 20203

Do the book and other resource recommendations especially apply to people interested in working on animal welfare?

Jamie_HarrisMar 6 20201

Not really! It's just that some of the recommendations come from our conversations with managers and leaders who work in animal advocacy. The third tab on the spreadsheet has some animal-advocacy-specific resources, but most of them are generic.

Vilfredo's GhostMar 8 20201

Most of these approaches all sound the same to me. At least in practice, as applied by a busy boss trying to make real day-to-day decisions. Transformational vs. transactional makes sense intuitively as involving different things, but transformational vs. servant vs. ethical leadership, I'd never be able to keep straight. I think good research on leadership would be a lot smaller than what's been done. Less Grand Theory of Leadership, more individualized testing of specific behaviors. Successful leaders would probably not agree to be subjected to RCTs because you'd risk making their performance worse. But if you could take e.g. leaders who've just received a bad performance review, or people with no prior leadership experience, experimenting offers a lot of upside for them. And if you can turn bad or inexperienced leaders into good ones, or at least better than a control group, then you're really onto something.