Reducing long-term risks from malevolent actors

David_Althaus; Tobias_Baumann

Hi, interesting article. Thank you for writing.

I felt that this article could have said more about possible policy interventions and that it dismisses policy and political interventions as crowded too quickly. Having thought a bit about this area in the past I thought I would chip in.

MORE ON POLICY INTERVENTIONS

Even within established democracies, we could try to identify measures that avoid excessive polarization and instead reward cross-party cooperation and compromise. ... (For example, effective altruists have discussed electoral reform as a possible lever that could help achieve this.)

There are many things that could be done to prevent malevolent leaders within established democracies. Reducing excessive polarization (or electoral reform) are two minor ones. Other ideas you do not discuss include:

Better mechanisms for judging individuals. Eg ensuring 360 feedback mechanisms are used routinely to guide hiring and promotion decisions as people climb political ladders. (I may do work on this in the not too distant future)
Less power to individuals. Eg having elections for parties rather than leaders. (The Conservative MPs in the UK could at any time decide that Boris Johnson is no longer fit to be a leader and replace him with someone else, Republicans cannot do this with Trump, Labour MPs in the UK cannot do this with a Labour leader to the same extent).
Reduce the extent to which corruption / malevolence is beneficial for success. There are many ways to do this. In particular removing the extent to which individuals raising money is a key factor for their political success (in the UK most political fundraising is for parties not for individuals). Also removing the extent to which dishonesty pays, for example with better fact-checking services.
More checks and balances on power. A second house. A constitution. More independent government institutions (central banks, regulators, etc – I may do some work in this space soon too). More transparency of political decision making. Better complaint and whistle-blowing mechanisms. Limits on use of emergency powers. Etc.

ARE POLITICAL INTERVENTIONS CROWDED?

Alternatively, we could influence political background factors that make malevolent leaders more or less likely... interventions to promote democracy and reduce political instability seem valuable—though this area seems rather crowded.

You might be correct, but this feels a bit like saying the AI safety space is crowded because lots of groups are trying to develop AI. However it may not be the case that those groups are focusing as much on safety as you would like. Although there are many groups (especially nation states) that want to promote democracy there may be very specific interventions that prevent malevolent leaders that are significantly under-discussed, such as elections for parties rather than leaders, or other points listed above. It seems plausible that academics and practitioners in this space may be able to make valuable shifts in the way fledgling democracies are developing that are not otherwise being considered.

And as someone in the improving government institutions space in the UK is is not evident to me that there is much focus on the kinds of interventions that would limit malevolent leaders.

David_Althaus

Thank you, excellent points!

I will probably add some of your intervention ideas to the article (I'll let you know in that case).

I felt that this article could have said more about possible policy interventions and that it dismisses policy and political interventions as crowded too quickly.

Sorry about that. It certainly wasn’t our intention to dismiss political interventions out of hand. The main reason for not writing more was our lack of knowledge in this space; which is why our discussion ends with “We nevertheless encourage interested readers to further explore these topics”. In fact, a comment like yours—containing novel intervention ideas written by someone with experience in policy—is pretty much what we were hoping to see when writing that sentence.

Better mechanisms for judging individuals. Eg ensuring 360 feedback mechanisms are used routinely to guide hiring and promotion decisions as people climb political ladders. (I may do work on this in the not too distant future)

Very cool! This is partly what we had in mind when discussing manipulation-proof measures to prevent malevolent humans from rising to power (where we also briefly mention 360 degree assessments).

For what it's worth, Babiak et al. (2010) seemed to have some success with using 360 degree assessments to measure psychopathic traits in a corporate setting. See also Mathieu et al. (2013).

weeatquince🔸

Relevant policy report from the UK Parliament on enforcing the Ministerial Code of good behaviour, (from 2006): https://publications.parliament.uk/pa/cm200506/cmselect/cmpubadm/1457/1457.pdf

(I wasn't sure what to do with this when I found it, I might add other policy reports I find to this thread too until I have the capacity to actually work on this in any detail)

Less directly relevant but somewhat interesting too: https://publications.parliament.uk/pa/cm200607/cmselect/cmpubadm/121/121i.pdf

MichaelA🔸

Thanks for sharing these additional ideas and insights!

Also removing the extent to which dishonesty pays, for example with better fact-checking services.

I ran (and published a paper on) an experiment on fact-checking for my Psychology Honours in 2017, so I have a smidge of knowledge here, though it's a bit rusty. Some brief thoughts:

I do suspect "better fact-checking services", in the sense of more accuracy or checking more facts, would be somewhat beneficial
But I think there's some reason for pessimism about just how much people really "care" about the results of fact-checks, even when they see those fact-checks. Here's my study's abstract:

In the ‘post-truth era’, political fact-checking has become an issue of considerable significance. A recent study in the context of the 2016 US election found that fact-checks of statements by Donald Trump changed participants' beliefs about those statements—regardless of whether participants supported Trump—but not their feelings towards Trump or voting intentions. However, the study balanced corrections of inaccurate statements with an equal number of affirmations of accurate statements. Therefore, the null effect of fact-checks on participants’ voting intentions and feelings may have arisen because of this artificially created balance. Moreover, Trump's statements were not contrasted with statements from an opposing politician, and Trump's perceived veracity was not measured. The present study (N = 370) examined the issue further, manipulating the ratio of corrections to affirmations, and using Australian politicians (and Australian participants) from both sides of the political spectrum. We hypothesized that fact-checks would correct beliefs and that fact-checks would affect voters’ support (i.e. voting intentions, feelings and perceptions of veracity), but only when corrections outnumbered affirmations. Both hypotheses were supported, suggesting that a politician's veracity does sometimes matter to voters. The effects of fact-checking were similar on both sides of the political spectrum, suggesting little motivated reasoning in the processing of fact-checks.

This is also relevant to the following sentence from the original post: "While the notion of Dark Tetrad traits is not foremost in most people’s minds, one could argue that much political debate is about related concepts like the trustworthiness or honesty of candidates, and voters do value those attributes." I think this is true, but probably less true than many might think (or at least than they would've thought pre-2016).

And then there's also the matter of whether people come to encounter fact-checks in the first place. In my study's conclusion, I wrote that "participants were unable to avoid fact-checks or to select which ones they received. In reality, some people may not encounter any fact-checks at all [9], and the sample of fact-checks which others encounter is often influenced by selective exposure and selective sharing [65,66]." (I feel weird about quoting myself, but 2017 Michael knew more about this than 2020 Michael does!)
So I'd tentatively see more value in making fact-checking services "better" in the sense of being clearer, more attention-grabbing, better publicised, or things like that (as long as this doesn't cost too much accuracy, nuance, etc.), rather than in e.g. making more or more accurate fact-checks.

And there may be still more value in somehow "shifting norms" towards valuing truth more highly, or things like that, though I don't know how one would actually do that. (I'm guessing this post is relevant, but I haven't read it yet.)

Jc_Mourrat

I've just read the results of an interesting new study on the effect of red-flagging some information on social media, with flags such as "Multiple fact-checking journalists dispute the credibility of this news", and variations with "Multiple fact-checking journalists" replaced by, alternatively, "Major news outlets", "A majority of Americans", or "Computer algorithms using AI". The researchers tested the effect this had on the propensity of people to share the content. The effect of the "fact-checking" phrasing was the most pronounced, and very significant (a reduction of about 40% of the probability to share content; which jumps to 60% for people who identify as Democrats). Overall the effect of the "AI" phrasing was also very significant, but quite counterintuitively it has the effect of increasing the probability of sharing content for people who identify as Republicans! (By about 8%; it decreases that same probability by 40% for people who identify as Democrats.)
https://engineering.nyu.edu/news/researchers-find-red-flagging-misinformation-could-slow-spread-fake-news-social-media

weeatquince🔸

Thank you for the insight. I really have no strong view on how useful each / any of the ideas I suggested were. They were just ideas.

I would add on this point that narcissistic politicians I have encountered worried about appearance and bad press. I am pretty sure that transparency and fact checking etc discouraged them from making harmful decisions. Not every narcissistic leader is like Trump.

MichaelA🔸

Yeah, that sounds right to me. And reminds me of a paper I read when working on that experiment, the abstract of which was:

Does external monitoring improve democratic performance? Fact-checking has come to play an increasingly important role in political coverage in the United States, but some research suggests it may be ineffective at reducing public misperceptions about controversial issues. However, fact-checking might instead help improve political discourse by increasing the reputational costs or risks of spreading misinformation for political elites. To evaluate this deterrent hypothesis, we conducted a field experiment on a diverse group of state legislators from nine U.S. states in the months before the November 2012 election. In the experiment, a randomly assigned subset of state legislators was sent a series of letters about the risks to their reputation and electoral security if they were caught making questionable statements. The legislators who were sent these letters were substantially less likely to receive a negative fact-checking rating or to have their accuracy questioned publicly, suggesting that fact-checking can reduce inaccuracy when it poses a salient threat.

Relatedly, it could be that "more" or "better" fact-checking would lead to better actions by or discourse from politicians, even if voters "don't really care much" about fact-checks or never really see them, due to politicians overestimating what impact fact-checks would have on voters' perceptions.

(To be clear, I do think fact-checks probably have at least some impact via the more obvious route too; I wonder mostly about the magnitude of the effect, not whether it exists.)

Richard Möhn

Just a quick thought: Supporting (investigative) journalism is also a possible intervention. I think journalism is considered a pillar of democracy? Currently, Kelsey Piper's work on OpenAI/Sam Altman is a good example.

David_Althaus

2y*

Yes, I think investigative journalism (e.g., Kelsey Piper's work on OpenAI) is immensely valuable.

In general, I've become more pessimistic about technology-centric/ "galaxy-brained" interventions in this area and more optimistic about "down-to-earth" interventions like, for example, investigative journalism, encouraging whistleblowing (e.g. setting up prizes or funding legal costs), or perhaps psychoeducation / workshops on how to detect malevolent traits and what do when this happens (which requires, in part, courage / the ability to endure social conflict and being socially savvy, arguably not something that most EAs excel in).

Richard Möhn

Down-to-earth interventions sounds better to me, too. I like all of the examples.

tommasoquerini

11mo

Hello. Could you explain a bit more what you mean?

Intuitively I imagine journalism is becoming less influential as people are increasingly trapped in echo chambers and their ability to process complex information and think critically is reducing.

What do you make of the arguments in Christopher H. Achen and Larry M. Bartels' Democracy for Realists: Why Elections Do Not Produce Responsive Government?

Namely that voters are driven by group identity and tend to evaluate political leaders and parties based on recent events or outcomes in their lives—especially economic conditions or crises—after they’ve happened, rather than making decisions based on informed policy preferences or long-term considerations.

If that's the case, investigative journalism may have a smaller than expected impact.

David_Althaus

11mo*

Hi, makes sense, thanks.

I certainly don't think investigative journalism is a panacea or tremendously effective. Definitely agree that most voters are motivated by tribalism and economic conditions.

Still, even in this age, investigative journalism sometimes seems to have substantial effects. For example, the book on the Biden's cognitive decline and cover-up made some waves and hopefully has some positive effects.

More generally, I'm also not sure to what extent our age is highly unusual. It's not like people in the 1930s and 1940s (in say, Germany) weren't subject to echo chambers and didn't exhibit a few "skill issues" in critical thinking.

My sense is that many elections are decided by a few percent of voters (with around 20-40% on each side (the "base") essentially being completely immovable by any argument) so even shifting a few percent of voters journalism could have an impact.

I think investigative journalism may be particularly effective during more "hingey" moments, before a political party has rallied around a certain candidate and before tribalism kicks in.

I agree that, say, changing the underlying structure of the information eco-system is probably more effective (e.g., changing (social media) recommendation algorithms (e.g., on Youtube, Facebook, or Twitter) to promote truth-seeking and improve the quality of general discourse). (But that's probably almost impossible to pull off for any given individual unless you happen to be, say, Zuckerberg or Musk). Also, interventions in the AI for epistemics space may be much more scalable and high-leverage.

Richard Möhn

11mo

I agree with David's response.

Ramiro

Personally, I think this is gold.

On the other hand, I’m not so sure that sadism is particularly worse for the long-term future than the dark triad traits. Yes, sadistic leaders may pose a special S-risk, but I am not sure they increase other x-risks so much – e.g., Hitler was kind of cautious concerning the risk of nuclear weapons igniting the atmosphere, and Stalin was partially useful in WW II and avoided WW III.

Contrast:

Some dark traits, such as Machiavellianism, can be beneficial under certain circumstances. It might be better if one could single out a dark trait, such as sadism, and only select against it while leaving other dark traits unchanged.

with these parts of the Appendix A:

Dark Triad traits predict increased intention to engage in political violence (Gøtzsche-Astrup, 2019)…

Historical evidence suggests that even many of their political adversaries—at least for some time—did not realize that Hitler, Mao, and Stalin were malevolent

Actually, open sadism often raises opposition; and I think the big 3 above weren't salient because they were sadistic – would they have killed less people if they were just functional psychopaths?

It might be hard to spot dark tetrad individuals, but it’s not so hard to realize an individual is narcissistic or manipulative. I don’t think people contend that Stalin and Mao were so – they may argue that they were beneficial, or that they were excused because they were “great man”.

So why do such guys acquire power? Why do people support it? We often dislike acquaintances that exhibit one of the dark traits; then why do people tolerate it when it comes from the alpha male boss?

My point is that the problem might be precisely that we often claim things like “Some dark traits, such as Machiavellianism, can be beneficial under certain circumstances. It might be better if one could single out a dark trait, such as sadism, and only select against it while leaving other dark traits unchanged.” If we could just acknowledge that dark triad traits individuals are very dangerous, even if they’re sometimes useful (like a necessary evil), then perhaps we could avoid (or at least be particularly cautious with) malevolent leaders.

David_Althaus

Thank you, good points.

I agree that it’s not clear whether sadism is really worse than the Dark Triad traits.

e.g., Hitler was kind of cautious concerning the risk of nuclear weapons igniting the atmosphere,

I’m not sure whether Hitler and Stalin were more sadistic than they were Machiavellian or narcissistic. If I had to decide, I’d say that their Machiavellianism and narcissism were more pronounced than their sadism.

Regarding Hitler being “kind of cautious”: It seems plausible to me that Hitler was less cautious than a less malevolent counterfactual German leader.

What is more, it seems not too unlikely that Hitler would have used nuclear weapons in 1945 if he had had access to them. For instance, see this quote from Glad (2002, p. 32):

As the Allies closed in on Berlin, [Hitler] wanted churches, schools, hospitals, livestock, marriage records, and almost anything else that occurred to him to be destroyed. In April 1945 he wanted the entire leadership of the Luftwaffe to be summarily hanged. He considered bringing about the destruction of German cities by announcing the execution of all Royal Air Force war prisoners, so that there would be massive bombing in reprisal. He may also have given orders for all wounded German soldiers to be killed. His aim became the destruction of Germany in the greatest Gotterdaemmerung of history. As Albert Speer said about Hitler's obsession with architecture: "Long before the end I knew that Hitler was not destroying to build, he was building to destroy.

You write:

and Stalin was partially useful in WW II and avoided WW III.

I'm not sure about that. Stalin was kind of Hitler’s ally until Hitler invaded Russia. In other words, if it weren’t for Hitler (and his overconfidence), Stalin might have even helped Hitler to win WW II (though Hitler might have lost WW II even with Stalin's support, I don't know). I’m no historian but Stalin’s response to Hitler’s invasion also seems to have been rather disastrous (see also Glad, 2002, p.8) and might have ended up in Russia losing WW II.

Resources like Khlevniuk (2015) also suggest that Stalin, if anything, was detrimental to Russia’ military success, at least for the first years of the war.

It might be hard to spot dark tetrad individuals, but it’s not so hard to realize an individual is narcissistic or manipulative.

I generally agree but there also seem to exist exceptions. The most strategic individuals know that narcissism and manipulativeness can be very off-putting so they will actively try to hide these traits. Stalin, for example, often pretended to be humble and modest, in public as well as in his private life, at least to some success. My impression from reading the various books listed in the references is that Stalin and Mao managed to rise to the top, partly because they successfully deceived others into believing that they are more modest and less capable, ruthless, and strategic than they actually are.

Also, it’s often not clear where justified confidence ends and narcissism begins—certainly not to the admirers of the leader in question. Similarly, what some would see as benign and even necessary instances of being realistic and strategic, others describe as unprincipled and malicious forms of manipulativeness. (The Sanders supporters who don't want to vote for Biden come to mind here even though this example is only tangentially related and confounded by other factors.)

So why do such guys acquire power? Why do people support it? We often dislike acquaintances that exhibit one of the dark traits; then why do people tolerate it when it comes from the alpha male boss?

Good questions. Indeed, some people do not only seem to tolerate these traits in leaders, they seem to be actively drawn to such "strong men" who display certain Dark Tetrad characteristics. I have some theories for why this is but I don't fully understand the phenomenon.

If we could just acknowledge that dark triad traits individuals are very dangerous, even if they’re sometimes useful (like a necessary evil), then perhaps we could avoid (or at least be particularly cautious with) malevolent leaders.

I’m definitely sympathetic to this perspective.

One important worry here is that different movements/political parties seem to be in a social dilemma. Let’s assume—which seems relatively plausible—that leaders with a healthy dose of narcissism and Machiavellianism are, on average, really better at things like motivating members of their movement to work harder, inspiring onlookers to join their movement, creating and taking advantage of high-value opportunities, and so on. If one movement decides to exclude members who show even small signs of Dark Triad traits, it seems plausible that this movement will be at a disadvantage compared to other movements that are more tolerant regarding such traits. Conversely, movements that tolerate (or even value) Dark Triad traits might be even more likely to rise to power. It seems very important to avoid such outcomes.

I’m also somewhat worried about "witch hunts" fueled by simplistic conceptions of malevolence. I know of several individuals who exhibit at least some level of Machiavellian and narcissistic traits—many even say so themselves—but whom I’d still love to give more influence because I believe that they are “good people” who are aware of these traits and who channel these traits towards the greater good. (Admittedly, a supporter of, say, Mao might have said the same thing back in the days.)

Ramiro

Thanks for the remarks concerning Hitler and Stalin.

I think it might be quite valuable, for the project as a whole, to better understand why people are drawn to leaders with features they would not tolerate in peers, such as dark traits.

For one, it’s very plausible that, as you mentioned, the explanation is (a) dark traits are very useful – these individuals are (almost) the only ones with incentives enough to take risks, get things done, innovate, etc. Particularly, if we do need things like strategies of mutually assured destruction, then we need someone credibly capable of “playing hawk”, and it's arguably hard to believe nice people would do that. This hypothesis really lowers my credence in us decreasing x-risks by screening for dark traits; malevolent people would be analogous to nukes, and it’s hard to unilaterally get rid of them.

A competing explanation is that (b) they’re not that useful, they’re parasitical. Dark traits are uncorrelated with achievement, they just make someone better at outcompeting useful pro-social people, by, e.g., occupying their corresponding niches, or getting more publicity - and so making people think (due to representative bias) that bad guys are more useful than they are. That's plausible, too; for instance, almost no one outside EA and LW communities knows about Arkhipov and Petrov. If that’s the case, then a group could indeed unilaterally benefit from getting rid of malevolent / dark trait leaders.

(Maybe I should make clear that I don't have anything against dark triad traits individuals per se, and I'm as afraid of the possibility of witch hunts and other abuses as everyone else. And even if dark traits were uncorrelated with capacity for achievement, a group might deprise itself from a scarce resource by selecting against very useful dark trait individuals, like scientists and entrepreneurs)

Kaj_Sotala

Malevolent humans with access to advanced technology—such as whole brain emulation or other forms of transformative AI—could cause serious existential risks and suffering risks.

Possibly relevant: Machiavellians Approve of Mind Upload Technology Directly and Through Utilitarianism (Laakasuo et al. 2020), though it mainly tested whether machiavellians express moral condemnation of mind uploading, rather than their interest directly.

In this preregistered study, we have two novel findings: 1) Utilitarian moral preferences are strongly and psychopathy is mildly associated with positive approval of MindUpload; and 2) that Machiavellianism – essentially a calculative self-interest related trait – is strongly associated with positive approval of Mind Upload, even after controlling for Utilitarianism and the previously known predictor of Sexual Disgust (and conservatism). In our preregistration, we had assumed that the effect would be dependent on Psychopathy (another Dark Triad personality dimension), rather than Machiavellianism. However, given how closely related Machiavellianism and Psychopathy are, we argue that the results match our hypothesis closely. Our results suggest that the perceived risk of callous and selfish individuals preferring Mind Upload should be taken seriously, as previously speculated by Sotala & Yampolskiy (2015)

Emanuele_Ascani

This is one of the best posts I've read here, wow.

One of the main things that concern me is that malevolent people could appropriate the concept of malevolence itself and start a witch hunt for people who have nothing to do with malevolence. This was passingly mentioned when acknowledging that political leaders could brand their opponents as malevolent. Overall I think this post makes a good job of outlining the pros and cons, but I just wanted to write this consideration in a comment because it has been somewhat prominent in my mind.

Tobias_Baumann

Thanks for the comment!

I would guess that having better tests of malevolence, or even just a better understanding of it, may help with this problem. Perhaps a takeaway is that we should not just raise awareness (which can backfire via “witch hunts”), but instead try to improve our scientific understanding and communicate that to the public, which hopefully makes it harder to falsely accuse people.

In general, I don’t know what can be done about people using any means necessary to smear political opponents. It seems that the way to address this is to have good norms favoring “clean” political discourse, and good processes to find out whether allegations are true; but it’s not clear what can be done to establish such norms.

Pablo

I'm surprised to see that this post hasn't yet been reviewed. In my opinion, it embodies many of the attributes I like to see in EA reports, including reasoning transparency, intellectual rigor, good scholarship, and focus on an important and neglected topic.

sbehmer

Thanks, it's a very nice article on an important topic. If you're interested, there's a small literature in political economy called "political selection" (here's an older survey article) . As far as I know they don't focus specifically on the extreme lower tail of bad leaders, but they do discuss how different institutional features can lead to different types of people gaining power.

David_Althaus

Thanks for the pointer!

Ben_West🔸

Thanks for this interesting article. Regarding malevolence among business leaders: my impression is that corporations have rewarded malevolence less over time.

E.g. in the early 1900s you had Frederick Taylor (arguably the most influential manager of the 20th century) describing his employees like:

one of the very first requirements for a man who is fit to handle pig iron as a regular occupation is that he shall be so stupid and so phlegmatic that he more nearly resembles in his mental make-up the ox than any other type.

Modern executives would never say this about their staff, and no doubt this is partly because what's said in the boardroom is different from what's said in public, but there is a serious sense in which credibly signaling prosocial behaviors towards your employees is useful. E.g. 80 years later you have Paul O'Neill, in almost exactly the same industry as Taylor, putting worker safety as his key metric, because he felt that people would work harder if they felt taken care of by the company.

My guess is that corporations which rely on highly skilled workers benefit more from prosocial executives, and that it's hard to pretend to be prosocial over a decades-long career, though certainly not impossible. So possibly one hard-to-fake measure of malevolence is whether you repeatedly succeed in a corporation where success requires prosociality.

David_Althaus

Thanks, that's a good example.

my impression is that corporations have rewarded malevolence less over time.

Yeah, I think that's probably true.

Just to push back a little bit, the pessimistic take would be that corporate executives simply have become better at signalling and public relations. Maybe also partly because the downsides of having bad PR are worse today compared to, say, the 1920s—back then, people were poorer and consumers didn't have the luxury to boycott companies whose bosses said something egregious; workers often didn't have the option to look for another job if they hated their boss, et cetera. Generally, it seems plausible to me that "humans seem to have evolved to emphasize signaling more in good times than in bad." (Hanson, 2009).

I wonder if one could find more credible signals of things like "caring for your employers", ideally in statistical form. Money invested in worker safety might be one such metric. Salary discrepancies between employees and corporate executives might be another one (which seems to have gotten way worse since at least the 1970s) though there are obviously many confounders here.

The decline in child labor might be another example of how corporations have rewarded malevolence less over time. In the 19th century, when child labor was common, some amount of malevolence (or at least indifference) was arguably beneficial if you wanted to run a profitable company. Companies run by people who refused to employ children for ethical reasons presumably went bankrupt more often given that they could not compete with companies that used such cheap labor. (On the other hand, it's not super clear what an altruistic company owner should have done. Many children also needed jobs in order to be able to buy various necessities—I don't know.)

Maybe this is simply an example of a more general pattern: Periods of history marked by poverty, scarcity, instability, conflict, and inadequate norms & laws will tend to reward or even require more malicious behavior, and the least ruthless will tend to be outcompeted (compare again Hanson's "This is the Dream Time", especially point 4).

Ben_West🔸

I wonder if one could find more credible signals of things like "caring for your employers", ideally in statistical form. Money invested in worker safety might be one such metric.

That seems reasonable. Another possibility is looking at benefits, which have grown rapidly (though there are also many confounders here).

Something which I can't easily measure but seems more robust is the fraction of "iterated games". E.g. I would expect enterprise salespeople to be less malevolent than B2C ones (at least towards their customers), because successful enterprise sales relies on building relationships over years or decades. Similarly managers are often recruited and paid well because they have a loyal team who will go with them, and so screwing over that team is not in their self-interest.

Caspar Oesterheld

Probably you're already aware of this, but the APA's Goldwater rule seems relevant. It states:

On occasion psychiatrists are asked for an opinion about an individual who is in the light of public attention or who has disclosed information about himself/herself through public media. In such circumstances, a psychiatrist may share with the public his or her expertise about psychiatric issues in general. However, it is unethical for a psychiatrist to offer a professional opinion unless he or she has conducted an examination and has been granted proper authorization for such a statement.

From the perspective of this article, this rule is problematic when applied to politicians and harmful traits. (This is similar to how the right to confidentiality has the Duty to Warn exception.) A quick Google Scholar search gives a couple of articles since 2016 that basically make this point. For example, see Lilienfeld et al. (2018): The Goldwater Rule: Perspectives From, and Implications for, Psychological Science.

Of course, the other important (more empirical than ethical) question regarding the Goldwater rule is whether "conducting an examination" is a necessary prerequisite for gaining insight into a person's alleged pathology. Lilienfeld et al. also address this issue at length.

Pivocajs

Some thoughts that occured to me while reading:

1) Research suggestion: From afar, malevolence-detection techniques seem like a better version of the already-existing tool of top-secret security clearance (or tests similar to it). I am not confident about this, but it already seems that if top-secret security clearance was a requirement for holding important posts, a lot of grief would be avoided (at least where I am from). Yet we generally do not use this tool. Why is this? I suspect that whatever the answer is, it will apply to malevolence-detection techniques as well.

2) Potential bottleneck: Suppose you succeed and develop 100% accurate malevolence-detection technique. I think that, by default, you would have trouble convincing people to use it. ("I mean, what if I score high on it? You know, I am keeping my dark side in check and I don't plan to become too influential either, so my malevolence doesn't really hurt anybody. But the other people don't know that! If I get branded as malevolent, nobody will talk to me ever, or hire me, or anything!") I conjecture that the impact of this agenda will be bottlenecked on figuring out how to leave the malevolent people a line of retreat; making sure that if you score high on this, the implications aren't that bad. I see three reasons for this:

a) non-malevolent people might not know they are non-malevolent, and hence be afraid of this,

b) malevolent-and-know-it people might have enough power to hinder this,

c) reasonable general concerns about any test like this getting out of hand.

3) Relatedly to (2), would it make sense to consider some alternative branding that more accurately suggests what you intend to do with the concept (and doesn't suggest other things)? Unwieldly suggestion, to illustrate what I mean: Being publicly known as "potentially too risky to be in a position of great power" indicates that you shouldn't be a president, but you might still have friends, a spouse, and a prestigeous job. Being publicly known as "malevolent", however, ... . (Also, it seems plausible that there are people who are malevolent, but do not endorse being so, similarly to how, I think, there are paedophiles who wish they weren't so.)

(Also, it might not be obvious from my nitpicking, but I really like the post, thanks for it :-).)

David_Althaus

(Also, it might not be obvious from my nitpicking, but I really like the post, thanks for it :-).)

Thank you. :) No worries, I didn't think you were nitpicking. I agree with many of your points.

[...] if top-secret security clearance was a requirement for holding important posts, a lot of grief would be avoided (at least where I am from). Yet we generally do not use this tool. Why is this? I suspect that whatever the answer is, it will apply to malevolence-detection techniques as well.

One worry with security clearances is that they tend to mostly screen for impulsive behaviors such as crime and drug use (at least, according to my limited understanding of how these security clearances work) and would thus often fail to detect more strategic malevolent individuals.

Also, your claim that “we generally do not use this tool [i.e., security clearances]” feels too strong. For example, 5.1 million Americans seem to have a security clearance. Sounds like a lot to me. (Maybe you had a different country in mind.)

I conjecture that the impact of this agenda will be bottlenecked on figuring out how to leave the malevolent people a line of retreat; making sure that if you score high on this, the implications aren't that bad.

Good point. I guess we weren’t sufficiently clear in the post about how we envision the usage of manipulation-proof measures of malevolence. My view is that their results should, as a general rule, not be made public and that individuals who are diagnosed as malevolent should not be publicly branded as such. (Similarly, my understanding is that if someone doesn’t get a top level security clearance because they, for instance, have a serious psychiatric disorder, they only don't get the job requiring the security clearance—it's not like the government makes their mental health problems public knowledge.)

My sense is that malevolent individuals should only be prevented from reaching highly influential positions like becoming senators, members of Congress, majors, CEOs of leading AGI companies, et cetera. In other words, the great majority of jobs would still be open to them.

Of course, my views on this issue are by no means set in stone and still evolving. I’m happy to elaborate on my reasons for preferring this more modest usage if you are interested.

Pivocajs

Of course, my views on this issue are by no means set in stone and still evolving. I’m happy to elaborate on my reasons for preferring this more modest usage if you are interested.

I think the more modest usage is reasonable choice.

Maybe you had a different country in mind. [regarding top-secret security clearance]

I am Czech. We do have the institute, and use it. But, as far as I know, our president doesn't have it, and a bunch of other people don't have it. (I.e., it seems that people who need secret information on a daily basis have it. But you don't need it for many other positions from which you could put pressure on people who have the clearance.)

MichaelPlant

Thanks for this write-up, I thought it was really interesting and not something I'd ever considered - kudos!

I'll now hone in on the bit of this I think needs most attention. :)

It seems you think that one of the essential things is developing and using manipulation-proof measures of malevolence. If you were very confident we couldn't do this, how much of an issue would that be? I raise this because it's not clear to me how such measures could be created or deployed. It seems you have (1) self-reports, (2) other-reports, (3) objective metrics, e.g. brain scans. If I were really sneaky, I would just lie or not take the test. If I were really sneaky, I would be able to con others, at least for a long-time - perhaps until I was in power. Regarding objective measures, there will be 'Minority Report' style objections to actually using them in advance, even if they have high predictive power (which might be tricky as it relies on collecting good data, which seems to require the consent of the malevolent).

The area where I see this sort of stuff working best is in large organisations, such as civil services, where the organisations have control over who gets promoted. I'm less optimistic this could work for the most important cases, political elections, where there is not a system that can enforce the use of such measures. But it's not clear to me how much of an innovation malevolence tests are over the normal feedback processes used in large organisations. Even if they could be introduced in politics somehow, it's unclear how much of an innovation this would be: the public already try to assess politicians for these negative traits.

It might be worth adding that the reason the Myers-Brigg style personality tests are, so I hear, more popular in large organisations than the (more predictive) "Big 5" personality test is that Myers-Briggs has no ostensibly negative dimensions. If you pass round a Big-5 test, people might score highly on neuroticism or low on openness and get annoyed. If this is the case, which seems likely, I find it hard e.g. Google will insist that staff take a test they know will assess them on their malevolence!

As a test for the plausibility of introducing and using malevolence tests, notice that we could already test for psychopathy but we don't. That suggests there are strong barriers to overcome.

David_Althaus

Thank you, great points.

It seems you think that one of the essential things is developing and using manipulation-proof measures of malevolence. If you were very confident we couldn't do this, how much of an issue would that be?

I wouldn't say it's "essential"—influencing genetic enhancement would still be feasible—though it would certainly be a big problem.

Regarding objective measures, there will be 'Minority Report' style objections to actually using them in advance, even if they have high predictive power

Yes, that would be an issue.

The area where I see this sort of stuff working best is in large organisations, such as civil services, where the organisations have control over who gets promoted. I'm less optimistic this could work for the most important cases, political elections, where there is not a system that can enforce the use of such measures.

That seems true though it seems at least conceivable that voters will demand such measures in the future. (As an aside, you mention large organisations but it seems such measures could also be valuable when used in smaller (non-profit) organizations?)

But it's not clear to me how much of an innovation malevolence tests are over the normal feedback processes used in large organisations.

Yeah, true. I guess it's also a matter of how much (negative) weight you put on malevolent traits, how much of an effort you make to detect them, and how attentive you are to potential signs of malevolence—most people seem to overestimate their ability to detect (strategic) malevolence (at least I did so before reality taught me a lesson).

It might be worth adding that the reason the Myers-Brigg style personality tests are, so I hear, more popular in large organisations than the (more predictive) "Big 5" personality test is that Myers-Briggs has no ostensibly negative dimensions.

Interesting, that seems plausible! I've always been somewhat bewildered by its popularity.

If this is the case, which seems likely, I find it hard e.g. Google will insist that staff take a test they know will assess them on their malevolence!

True. I guess measures of malevolence would work best as part of the hiring process (i.e., before one has formed close relationships).

As a test for the plausibility of introducing and using malevolence tests, notice that we could already test for psychopathy but we don't. That suggests there are strong barriers to overcome.

I agree that there are probably substantial barriers to be overcome. On the other hand, it seems that many companies are using "integrity tests" which go in a similar direction. According to Sacket and Harris (1984), at least 5,000 companies used "honesty tests" in 1984. Companies were also often using polygraph examinations—in 1985, for example, about 1.7 million such tests were administered to (prospective) employees (Dalton & Metzger, 1993, p. 149)—until they became illegal in 1988. And this even though polygraph tests and integrity tests (as well as psychopathy tests) can be gamed (rather easily).

I could thus imagine that at least some companies and organizations would start using manipulation-proof measures of malevolence (which is somewhat similar to the inverse of integrity) if it was common knowledge that such tests actually had high predictive validity and could not be gamed.

Milan Griffes

Regarding objective measures, there will be 'Minority Report' style objections to actually using them in advance, even if they have high predictive power (which might be tricky as it relies on collecting good data, which seems to require the consent of the malevolent).

You could imagine scenarios where we apply objective measures to children, embryos, and/or couples who are considering having children.

This would avoid some of the free-choice problem around who takes the test, though it doesn't really get around the Minority Report problem. Also there'd still be selection at the parent level (some would-be parents would decide to not get tested).

Interesting that we don't do anything like this for psychopathy currently, as far as I know. (Psychopathy appears to be somewhat genetic.)

JustinShovelain

Nice post!

Here are a couple additional posts that I think are worth checking out by Gwern:

https://www.lesswrong.com/posts/ktr39MFWpTqmzuKxQ/notes-on-psychopathy

https://www.lesswrong.com/posts/Ft2Cm9tWtcLNFLrMw/notes-on-the-psychology-of-power

David_Althaus

Thank you!

Agree, these posts are excellent. For what it's worth, I share Gwern's pessimistic conclusion about the treatment of psychopathy. Other Dark Tetrad traits—especially if they are less pronounced—might be more amenable to treatment though I'm not especially optimistic.

However, even if effective treatment options existed, the problem remains that the most dangerous individuals are unlikely to ever be motivated to seek treatment (or be forced to do so).

L.F.D Olivares

Amazing article, It is a great insight into the malevolent person.

Yet I don't see the problem in the malevolent, but the system that allows their rise by making those malevolent traits praisable in every ideology, even in commerce.

There is no definition of ideology in the text, yet one is needed, because every ideology is dangerous. For an ideology seeks to impose a single maxim over other human endeavours, nazism was about racial supremacy, marxism positions the class struggle as an idea over all, religious fundamentalism puts god above everything, capitalism puts commerce above all, logic positivist wanted to reduce everything to epistemology as such classical physics uber alles.

Ideologies, then have a tendency to be holistic and reductionist, they attempt to put everything inside their maxim and reduce all other bodies of knowledge to such maxims, they are dogmatic, thus making it favorable for those with malevolent personalities to thrive as they are the most dogmatic of all.

The reason why some autocratic rulers were no malevolent such as Marcus Aurelius, Atatürk, and others is because they followed no ideology. They were not idealist but realists, in the case of Aurelius we have his meditations and as such we can see that he holded no maxim, he made no reduction to a singular knowledge, Stoicism was a physicalist philosophy, a realist belief system.

How can we fight ideologies then? Well Ideology are always opposed by other ideologies and we can keep that fight eternally, but also by science, as the relentless pursuit of operational knowledge always ends up destroying the maxims' ideologies hold. Examples of these are; Galileo when incarcerated by proving the copernican system, a maxim of a race above all of nazims is absurd when biology proves there are only phenotypic differences between humans, the materialist beginning of civilization proposed by marxist and social darwinist both erroneos by the excavations of Gobekli Tepe and the other Tepe cites, etc.

In essence to stop the malevolent is by generating a system that does not allow for their rise, a system without ideologies or at least where they are contained into their own battle ground, internet forums. Such system then must disregard arguments by maxims, in essence make the validation by science alone.

If the malevolent can rise still in the system of science it is to be seen, but for that to happen they would need to transform science into an ideology, but that would be no science.

David_Althaus

Thanks.

I guess I agree with the gist of your comment. I'm very worried about extremist / fanatical ideologies but more on this below.

because every ideology is dangerous

I guess it depends on how you define "ideology". Let's say "a system of ideas and ideals". Then it seems evident that some ideologies are less dangerous than others and some seem actually beneficial (e.g., secular humanism, the Enlightenment, or EA). (Arguably, the scientific method itself is an ideology.)

I'd argue that ideologies are dangerous if they are fanatical and extreme. The main characteristics of such fanatical ideologies include dogmatism (extreme irrationality and epistemic & moral certainty), having a dualistic/Manichean worldview that views in-group members as good and everyone who disagrees as irredeemably evil, advocating for the use of violence and unwillingness to compromise, blindly following authoritarian leaders or scriptures (which is necessary since debate, evidence and reason are not allowed), and promising utopia or heaven. Of course, all of this is a continuum. (There is much more that could be said here; I'm working on a post on the subject).

The reason why some autocratic rulers were no malevolent such as Marcus Aurelius, Atatürk, and others is because they followed no ideology. [...] Stoicism was a physicalist philosophy, a realist belief system.

Sounds like an ideology to me but ok. :)

Milan Griffes

In this section, we explore how possible future technologies could be used to reduce the influence of malevolent actors.

Psychedelics probably help with this as well (when carefully administered).

Anecdotally, a lot of Western contemplative teachers got started on that path because of psychedelic experiences (Zen, Tibetan Vajrayana, Vipassana, Advaita Vedanta, Kashmiri Shaivism). These traditions are extremely prosocial & anti-malevolent.

Less anecdotally, Hendricks et al. 2018 found lifetime psychedelic use correlated with reduced criminality (survey with n = 480,000). Lyons & Carhart-Harris 2018 found psilocybin decreased authoritarian political beliefs (small, open-label, prospective study).

There hasn't been a proper RCT yet though.

David_Althaus

There hasn't been a proper RCT yet though.

Pokorny et al. (2017) seems like a relevant RCT. They found that psilocybin significantly increased empathy.

However, even such results don’t make me very optimistic about the use of psychedelics for reducing malevolence.

The kind of individuals that seem most dangerous (people with highly elevated Dark Tetrad traits who are also ambitious, productive and strategic) seem less likely to be interested in taking psychedelics—such folks don't seem interested in increasing their empathy, becoming less judgmental or having spiritual experiences. In contrast, the participants of the Pokorny et al. study—like most participants in current psychedelics studies (I think)—wanted to take psychedelics which is why they signed up for the study.

Moreover, my sense is that psychedelics are most likely to increase openness and compassion in those who already started out with some modicum of these traits and who would like to increase them further. I’m somewhat pessimistic that giving psychedelics to highly malevolent individuals would make them substantially more compassionate. That being said, I'm certainly not confident in that assessment.

My intuition is partly based on personal experience and anecdotes but also more objective evidence like the somewhat disappointing results of the Concord Prison Experiment. However, due to various methodological flaws, I'd be hesitant to draw strong conclusions from this experiment.

Overall, I’d nevertheless welcome further research into psychedelics and MDMA. It would still be valuable if these pharmaceutical agents “only" increase empathy in individuals who are already somewhat empathic.

Anecdotally, a lot of Western contemplative teachers got started on that path because of psychedelic experiences (Zen, Tibetan Vajrayana, Vipassana, Advaita Vedanta, Kashmiri Shaivism). These traditions are extremely prosocial & anti-malevolent.

My guess is that most Western contemplative teachers who, as a result of taking psychedelics, got interested in Buddhism and meditation (broadly defined) were, on average, already considerably more compassionate, idealistic, and interested in spiritual questions than the type of practically oriented, ambitious, malevolent people I worry about.

As an aside, I'm much more optimistic about the use of psychedelics, empathogens, and entactogens for treating other issues such as depression or PTSD. For example, the early results on using MDMA for treatment-resistant PTSD seem extremely encouraging (and Doblin's work in general seems promising).

Aside from the obvious dangers relating to bad trips, psychosis, neurotoxicity (which seems only relevant for MDMA), et cetera^[1], my main worry is that psychedelics sometimes seem to decrease people’s epistemic and instrumental rationality. I also observed that they sometimes seem to have shifted people’s interests towards more esoteric matters and led to “spiritual navel-gazing”—of course, this can be beneficial for people whose life goals are comparatively uninformed.

Though my impression is that these risks can be reduced to tolerable levels by taking psychedelics only in appropriate settings and with the right safety mechanisms in place. ↩︎

Milan Griffes

Thanks for this great comment!

I had forgotten about Pokorny et al. 2017 – that's definitely relevant work.

The kind of individuals that seem most dangerous (people with highly elevated Dark Tetrad traits who are also ambitious, productive and strategic) seem less likely to be interested in taking psychedelics—such folks don't seem interested in increasing their empathy, becoming less judgmental or having spiritual experiences.

A lot of Nazis were interested in the occult, and Mao wrote poetry.

I’m somewhat pessimistic that giving psychedelics to highly malevolent individuals would make them substantially more compassionate. That being said, I'm certainly not confident in that assessment.

More research needed here.

As far as I know, there hasn't been any work done on how psychedelics affect individuals with high Dark Tetrad scores.

... the Concord Prison Experiment. However, due to various methodological flaws, I'd be hesitant to draw strong conclusions from this experiment.

Yeah, the Concord Prison Experiment is pretty bad. Hendricks et al. 2014 is better work on this, though it's just a survey.

my main worry is that psychedelics sometimes seem to decrease people’s epistemic and instrumental rationality.

Do you know where this worry comes from?

Gabay et al. 2019 found that MDMA boosted people's cooperation with trustworthy players in an iterated prisoner's dilemma, but not with untrustworthy players. I take that as some evidence that MDMA doesn't acutely harm one's rationality. I don't think there's been similar work done for other psychedelics.

I also observed that they sometimes seem to have shifted people’s interests towards more esoteric matters and led to “spiritual navel-gazing”—of course, this can be beneficial for people whose life goals are comparatively uninformed.

I think spiritual bypassing is a real problem, and unwise psychedelic use can definitely facilitate it. Context & integration seems very important.

David_Althaus

A lot of Nazis were interested in the occult, and Mao wrote poetry.

Good point, my comment was worded too strongly. I’d still guess that malevolent individuals are, on average, less interested in things like Buddhism, meditation, or psychedelics.

Do you know where this worry [that psychedelics sometimes seem to decrease people’s epistemic and instrumental rationality] comes from?

Gabay et al. 2019 found that MDMA boosted people's cooperation with trustworthy players in an iterated prisoner's dilemma, but not with untrustworthy players. I take that as some evidence that MDMA doesn't acutely harm one's rationality.

Interesting paper! Though I didn’t have MDMA in mind; with “psychedelics” I meant substances like LSD, DMT, and psilocybin. I also had long-term effects in mind, not immediate effects. Sorry about the misunderstanding.

One reason for my worry is that people who take psychedelics seem more likely to believe in paranormal phenomena (Luke, 2008, p. 79-82). Of course, correlation is not causation. However, it seems plausible that at least some of this correlation is due to the fact that consuming psychedelics occasionally induces paranormal experiences (Luke, 2008, p. 82 ff.) which presumably makes one more likely to believe in the paranormal. This would also be in line with my personal experience.

Coming back to MDMA. I agree that the immediate, short-term effects of MDMA are usually extremely positive—potentially enormous increases in compassion, empathy, and self-reflection. However, MDMA’s long-term effects on those variables seem much weaker, though potentially still positive (see Carlyle et al. (2019, p. 15).

Overall, my sense is that MDMA and psychedelics might have a chance to substantially decrease malevolent traits if these substances are taken with the right intentions and in a good setting—ideally in a therapeutic setting with an experienced guide. The biggest problem I see is that most malevolent people likely won’t be interested in taking MDMA and psychedelics in this way.

Milan Griffes

Overall, my sense is that MDMA and psychedelics might have a chance to substantially decrease malevolent traits if these substances are taken with the right intentions and in a good setting—ideally in a therapeutic setting with an experienced guide. The biggest problem I see is that most malevolent people likely won’t be interested in taking MDMA and psychedelics in this way.

Our estimates of the likelihood of malevolent people being interested probably hinge on our theory of where malevolence comes from.

e.g. if we think malevolence mostly arises as a maladaptive coping response to early trauma, you could imagine interventions that resolve the trauma and replace the maladaptive response with a more prosocial & equally fit response (and malevolent people being interested in those interventions).

But if we think malevolence is mostly a genetically-mediated trait, it's probably harder to change.

I haven't poked the literature on this yet.

Timothy_Liptrot

I am skeptical of this line of reasoning because I see little reason to believe that malevolence determined the policies in question. Game theory political scientists argue that different institutional structures make it rational or irrational for leaders to distribute public goods or targeted goods, practice repression, allow political parties. For a more in depth treatment, see the Dictator's Handbook by Bruce Bueno De Meqsuita and Alistair Smith. Their core argument is that because dictators must appease a very small group of powerful interest leaders (generals for Mussolini, members of the centcom for Stalin, tribal and military leaders for the Abdullah II) they can protect their power by rewarding only that small group at the expense of the masses.

Here is an illustrative example of political phenomena that is difficult to explain from the leaders personality. Torture is more common in multi-party autocracies than in one-party states. If the leaders narcissism strongly influences policies and narcissism and sadism are strongly correlated, then we would expect torture to be more common in states that ban dissent. Suppose that torture is not about satisfying the personal desire of the dictator and is instead about policing dissent. Now it makes sense that if some dissent (like resistance to a new "non-security" policy) is allowed, there must be some boundary into banned dissent. Then the occurrence of torture in multi-party states makes more sense and the rarity of torture in the most severe autocracies makes sense.

Opposing personality-of-dictator explanations to ideological explanations surprised me because it ignored the strongest explanations in institutional structures of states and in political cultures. Possibly you emphasized ideology because your samples are older. While the early modernist dictators were authentically ideological, most modern autocrats espouse a bland, centrist, syncretic corporatism. Dictators like Chavez and Castro are the exception today (although their ideology does influence behavior). Here is an article which argues most dictatorships are interest driven, not ideologically driven https://sci-hub.tw/10.1080/13510347.2017.1307823

MichaelA🔸

I'd definitely agree that leaders' personalities are not the sole factor in determining a countries' policies, levels of repression, etc., even in autocracies. And I'd tentatively guess that leaders' personalities tend not to be a larger factor than other influences combined. And it's very plausible to me that institutional structures tend to be a larger factor.

But it seems to me that leaders personalities are very likely at least sometimes a substantial factor. And, from memory, I don't think this post makes or requires stronger claims than that. I.e., I don't recall it making confident claims that other factors like institutional structures are usually smaller factors than personality. Are there passages you thought were overly strong?

I think this post was largely arguing that, on the margin, it could be quite valuable to think about reducing risks from malevolence - not that other factors like institutional structures don't also matter.

Torture is more common in multi-party autocracies than in one-party states.

This claim seems plausible to me, but I've never seen the claim before or seen specific evidence on it. Could you cite a source for that?

I also know of some data points that seem to push against that claim. For example, I believe there was a lot of torture during the one-party states of Hitler, Stalin, and Mao. (Here are some sources that seem to support this, though I only read the titles, as I was already pretty sure this was true: 1, 2, 3.)

And a handful of historical accounts I've read make it seem quite plausible that the personality of the leaders played a role in this. In particular, those accounts suggest these leaders (or at least Hitler and Stalin; I know less about Mao) took pleasure in things like ordering tortures or gruesome deaths, that they regularly personally ordered these things, and that they sometimes ordered these things at times when it seems it wasn't useful or logical to do so. Here's one paper that supports those claims (though I'm mildly skeptical of the paper in some ways): Why Tyrants Go Too Far: Malignant Narcissism and Absolute Power.

(I've just started doing research following on from this post, and collaborating somewhat with its lead author, but all of the above opinions are my own, as always.)

Timothy_Liptrot

Source: https://edisciplinas.usp.br/pluginfile.php/103226/mod_resource/content/1/James%20Vreeland%202008.PDF

Concerns with causation:

I worry about the underlying assumption that democracies don't encourage malevolent traits. So we observe less mass killings and rival killings in democracies than in dictatorships. One explanation is that democracies are selecting for anti-killing leaders. Another explanation is that malevolent leaders in democracies see little gain from killing while malevolent leaders in certain types of dictatorships see much gain from killing. For example, Abraham Lincoln and Richard Nixon seem to have a lot of malevolent traits, but they mostly refrained from political killings. So the fourth section suggests too much causation on the personalities of leaders.

But it seems to me that leaders personalities are very likely at least sometimes a substantial factor.

Fair - even if most of the difference in political killing is from institutional incentives, preventing malevolent actors from becoming dictators or presidents is big gains.

Secondly, this research agenda must recognize that authoritarian cultures may accept or encourage violence against non-comformists or "disloyal" people. It's a deeply sad fact, but important to understand. If torturing dissidents is an expected and approved behavior it is weaker evidence of a malevolent personality (fundamental attribution error).

Targeting your interventions:

You should pay more attention to how autocrats are actually selected. There are a few good models, my favorite is selectorate theory which (in dictatorships) emphasizes trust between top lieutenants. To continue with your preferred 40's references, Herman Goring, Himler, Keitel and Borman all hate and distrust one another. But they each trust Hitler because Hitler selected them to receive stolen wealth in exchange for support. So as long as the autocrat is alive the alliance is "stable".

In this model they care most about malevolence when the ruling coalition/launching org selects the next dictator. This famously happened after Stalin died, when Beria (head of secret police) got overthrown and replaced with Khrushchev who promised to stop political killings. So target regimes in or near transitions.

For example, the politburo of Cuba might be interested in this research. Saudi Arabia, now that MBS has solidified control, will be less fruitful.

MichaelA🔸

https://edisciplinas.usp.br/pluginfile.php/103226/mod_resource/content/1/James%20Vreeland%202008.PDF

Thanks for sharing that source. I tried to skim through to find the part most relevant to your claim, and found this:

What is observed regarding patterns of torture under dictatorship? The Hathaway torture data include 967 country-year observations of from 1985 to 1996 covering 109 separate dictatorships. The mean rate of torture is 3.0 with a standard deviation of 1.1; the median is 3. My principal explanatory variable is Gandhi's "parties," a dummy variable coded 1 if more than one party exists legally and 0 otherwise. For the Hathaway data, nearly 52 percent of the observations have multiple political parties. Average rates of torture under dictatorships without political parties is 2.8 (standard deviation 1.1), but rates of torture under dictatorships with political parties is 3.1 (standard deviation 1.0). A t-test indicates that the difference in average rates of torture is statistically significant beyond the 0.01 confidence level (t = -4.44). Rates of torture are ironically higher under open dictatorships that allow multiple political parties

So I do think this source supports your claim. But I'd note that:

That data just covers 1985-1996. Hitler, Stalin, and Mao seem the most salient examples of potentially very malevolent individuals wielding great power and causing great harm, and each would be excluded from that data set.
- I'm not saying that the study should've focused on an earlier period, or that there's some reason our predictions about the future would be better informed by an earlier 11 year period.
- But I'd be more confident about extrapolations from the data set if it spanned a larger period of time.
- And I think it would make sense for our predictions to draw on both that data and the observation of extreme harms under the one-party states of Hitler, Stalin, and Mao. (The latter is just a sample of 3, but I believe it accounted for a large portion of all deaths in mass murders, famines, etc.)
The difference in averages doesn't look hugely substantial - ~30% of 1 standard deviation - even if it was statistically significant.
- This seems like a further reason to also pay substantial attention to other observations, like harms under Hitler, Stalin, and Mao.

(Again, this doesn't mean your claim was false, just that its evidence base and implications may be more limited than one might have thought.)

Timothy_Liptrot

I'm compelled by this. That the difference is only 30% of 1 standard deviation means that lots of variation could be explained by other factors. Personality of the dictator could still explain lots of variation, even a majority. There could also be a relationship between dictator personality and allowances for dissent. Thanks for explaining that!

Aside, you would be more compelling if you talked about autocrats other than Hitler, Stalin and Mao.

MichaelA🔸

Aside, you would be more compelling if you talked about autocrats other than Hitler, Stalin and Mao.

Do you mean talking about other autocrats as well, or instead of, talking about Hitler, Stalin, and Mao?

If you mean "as well", I'd agree. I've already started looking at some others (as did David and Tobias, the authors of this post), and will hopefully do more of that. The reason for focusing mostly on those three so far is just that it takes time to learn about more, and those three were (I'd argue) huge factors in a large portion of all harms from political atrocities in the 20th century.

If you mean "instead of", could you explain further why you say that?

Timothy_Liptrot

Hmmm, that is a good question. Let me dig in more. Here are reasons to talk about others than Hitler Stalin and Mao

Coup Proofing is a common practice of dictators for political survival

Some behaviors of Hitler, Stalin and Mao have compelling institutional explanations that have become repeated behavior of long-ruling dictators. I'm thinking of coup proofing in particular. Coup proofing is a set of policies dictators enact to prevent a single small group from seizing power; rotatring or purging officers (Tukachevsky/Rommel) splitting the army into multiple factions/militias (NKVD/SS/Revolutionary Guard). We've since observed that lots of dictators (and coup-threatened democracies) practice coup-proofing. So I would be careful about attributing the *intra-elite* violence by particular regimes to the personalities of the leaders. Coup-proofing cannot explain violence against non-elites by those regimes.

Change in ideological motivation between early dictators and modern dictators

Secondly, Hitler, Stalin and Mao were much more ideological than most modern dictators. The Mussolini model of a "moderate" or "synarchic" authoritarianism spread more after WWII. By moderate I mean without a narrow, extreme vision of state-society interaction, not that dissent or economic freedom were allowed. Particularly today ideology structures the behavior of dictators much less. So one could argue that both Hitler and Stalin faithfully followed the visions laid out in their (terrible and warped) ideologies. If you buy that argument you would update downward on future dictators committing similar violence against non-elites.

For example, Mohammad Bin Salman has killed and tortured elite rivals, dissenters and starved many thousands of Yemenis to death. But he does not seem interested in any state project that would involve violence on the scale of Hitler/Stalin/Pol Pot.

All that said, I personally do not put much weight on my ideology argument. Firstly, the ideological explanation of Hitler and Stalin's behavior is not that strong. Secondly just because most dictators, like MBS and the Ethiopians, have "moderate" visions of state-society relations does not prevent future radical dictators from taking power (tail risk). Note that ISIS is a deathcult that took over half of two middle income countries, and that even in the 20's and 30's Mein Kampf was such ridiculous nonsense that Hitler should not have risen to power.

But on a first read this is a real hole in your argument. Almost all modern dictators do not look like Hitler, Stalin and Mao due to their weak ideoligical commitments. If you include a few more modern examples you can evade this argument. Off the top of my head I would suggest:

Khomeini and Ahmedinajad - they did not murder lots of people but their stubborn refusal to compromise with the world order has impoverished Iran without causing a regime change (they changed a bit recently but 30 years of bad decisions before).

Pol Pot - 1980's, only stopped by Vietnamese intervention

Modern Burma

MichaelA🔸

Your comments about coup-proofing seem interesting and useful.

We've since observed that lots of dictators (and coup-threatened democracies) practice coup-proofing. So I would be careful about attributing the *intra-elite* violence by particular regimes to the personalities of the leaders.

I think the fact that more leaders engaged in violent coup-proofing (rather than it just being Hitler, Stalin, and Mao) should indeed provide weak evidence against the theory that the unusual (compared to the population as a whole) personalities of leaders plays a key role in whether violent coop-proofing occurs. This is because that theory would now need to claim that a larger number of leaders have personalities that are unusual in the relevant way, or that a lack of such unusual personalities was "made up for" by other factors in some cases.

But I think that this fact would only serve as weak evidence, because it doesn't seem very implausible to claim that a fairly large number of dictators, or leaders of coup-threatened democracies, have personalities that are unusual in the relevant way. These are people in unusual positions which are arguably easier to get into if one is ruthlessly self-interested, so it wouldn't seem surprising (prima facie) if their average levels of ruthlessness-relevant traits was notably above population averages.

Additionally, it seems worth distinguishing violent coup-proofing from coup-proofing as a whole. In terms of how well they might evidence malevolent traits, "rotating [...] officers" and "splitting the army into multiple factions/militias" seem quite different from the sorts of violent purges engaged in by e.g. Stalin. (It may well be that violent coup-proofing is very common as well; I'm just flagging that the distinction seems relevant for my purposes.)

You also seem to imply that (a) these coup-proofing behaviours may have been rational things for a self-interested leader to do in those situations, and (b) this is reason to be careful in assuming that this is about personality. I think (a) is a good point. And I think there's some merit to (b), in the sense that this pushes against thinking something like "These leaders are just crazy and evil."

But overall, I don't think this is a question of personality vs incentive structures. I think neither determines a person's behaviour by itself. And I think whether or not one would do even horrific things if it's in one's self-interest is partly a matter of personality. E.g., I'd be willing to bet that a very large portion of people wouldn't engage in violent coup-proofing, even if they were in a situation where doing so would help them keep power. (Another framing of this is that a person's personality and values helps determine their incentives, in the sense that it influences what we like and dislike, including how much we dislike harming people.)

Coup-proofing cannot explain violence against non-elites by those regimes.

I think this is key; I think the massive-scale violence against non-elites is probably a larger part of why Hitler, Stalin, and Mao seem like interesting case studies of potential harm from malevolence than is the potentially "rational" coup-proofing.

That said, it's good to independently consider the merits of a even non-central arguments/points, and to update incrementally, so your arguments against making naive inferences from coup-proofing behaviours are still useful.

Timothy_Liptrot

I have one criticism of the argument that coup-proofing prevalence is evidence for personality factors. Suppose that if people observe a game being played multiple times they are more likely to set aside their personal preferences and "play to win". So if I were the first dictator of Iraq I might say "no I'm not going to kill generals who come from different towns, that would be evil". And then get killed for it. And maybe the second dictator says the same thing. But by the time the third or fourth dictator rises to power he'll either be selected for willingness to use violence or he will decide his preference for living is stronger than his preference for not killing. While I agree that many people would not commit inter-elite violence as the first leader, I suspect a much larger number would as the 5th leader. So an argument for point B.

Saddam Hussein was the 5th Iraqi leader to take power by coup within 21 years.

But on the other hand, there are lots of leaders that just stepped down when they lost the support of their ruling coalitions. And those heroes do not become famous. This is strong evidence of the importance of personality.

MichaelA🔸

I have one criticism of the argument that coup-proofing prevalence is evidence for personality factors.

To be clear, my argument was more like "coup-proofing prevalence doesn't seem like strong evidence against personality playing an important role". I.e., I don't think that it should reduce our belief that personality plays an important role.

It is true that I think I'd see these behaviours as evidence for personality playing an important role. But I'm not sure, and I'm not seeing it as key evidence.

While I agree that many people would not commit inter-elite violence as the first leader, I suspect a much larger number would as the 5th leader. So an argument for point B.

I'd agree that a much larger number would as the 5th leader than as the 1st leader, in the scenario you describe. And I think this is a valuable point.

But, in line with your final paragraph, I'd still bet that many people wouldn't; I think many people would simply step down, flee, or accept radical changes to the nature of their regime.

And perhaps more importantly, I think personality influences whether someone tries to become a leader in the first place, and whether they succeed in that. So I expect a lot of people to not want to "do horrible things", recognise that pursuing this leadership position may require them to "do horrible things" along the way or to stay in power, and thus just not pursue those positions.

(That said, I did say "I'd be willing to bet that a very large portion of people wouldn't engage in violent coup-proofing, even if they were in a situation where doing so would help them keep power." So there's a valid reason why you focused on how people would behave if they somehow landed in the leadership position, rather than how likely they are to enter those positions to begin with.)

MichaelA🔸

Thanks for this comment.

Your points (and counterpoints!) about changes in ideological motivation are very interesting. And I think it'd probably be good for me to spend some time engaging with evidence/arguments about how much ideology influenced Hitler, Stalin, and Mao's most "extreme" behaviours and whether/how much the influence of ideology has waned.

And it does seem wise to think about that, and about more modern examples, if one is planning to communicate with the public, policymakers, or academics about this topic in a way that leans substantially on historical examples of dictators. (I'm not sure if anyone will actually do such communications, or emphasise those cases when doing so. It may, for example, make more sense to just focus on the psychological studies, or on examples from business.)

MichaelA🔸

Another explanation is that malevolent leaders in democracies see little gain from killing while malevolent leaders in certain types of dictatorships see much gain from killing. For example, Abraham Lincoln and Richard Nixon seem to have a lot of malevolent traits, but they mostly refrained from political killings. So the fourth section suggests too much causation on the personalities of leaders.

I did recently think it might be interesting to look into Nixon as a case study in how, and how well, democratic institutions can mitigate the harm caused by leaders with high levels of dark tetrad traits. (I just think it might be such a case study, because I haven't yet really looked into evidence on Nixon's personality - this is just a guess so far.) Thanks for highlighting Lincoln too - I wouldn't have guessed he had high levels of dark traits, but I'll look into it.

I'd definitely guess that the reasons there is less harm from malevolent leaders in democracies are both that democracies select for malevolence less, and that democracies don't allow/incentivise malevolent behaviours as much.

In my head, I currently break intervention options in this cause area into:

Reducing how malevolent people are (via, e.g., very cautious and well thought-out and not-rushed genetic engineering)
Reducing the chances of malevolent people getting into positions where those traits create major risks (via, e.g., electoral reform, reducing instability and conflict)
Reducing the risks created when malevolent people get into those positions (via, e.g., checks and balances, maybe reducing centralisation of power)

I had felt like this post implied all three of those categories, not just the first two. But now that I re-skim the Political interventions section, I see that that might not have been made very explicit. So that critique of yours may be valid. (And I definitely agree with your point, separate from how it relates to potential oversights of this post.)

this research agenda must recognize that authoritarian cultures may accept or encourage violence against non-comformists or "disloyal" people. It's a deeply sad fact, but important to understand. If torturing dissidents is an expected and approved behavior it is weaker evidence of a malevolent personality (fundamental attribution error).

I definitely agree that:

it's important to consider multiple explanations of the various horrific or troubling behaviours
it can be easy to psychoanalyse/diagnose from a distance in a foolish way
the fundamental attribution error is worth keeping in mind here

I'm hoping to look a bit into how much we can trust speculative psychological profiles in general, maybe "best practices" for that shaky endeavour, and maybe how this relates to people like Hitler in particular. (Before getting into EA, I might've dismissed such psychological profiles. But I've moved towards thinking that, for many questions, it's hard to do better than weak evidence, and it can be best to just gather diverse kinds of weak evidence, update incrementally, and remain aware of how uncertain we are.)

(Also, by the way, I read The Dictator's Handbook last year and found it interesting. I agree that the models/ideas in that book seem relevant here.)

MichaelA🔸

Great post, thanks for writing it!

Baron-Cohen (2012) argues that the defining feature of human evil is “zero degrees of empathy.” However, some psychopaths can read other people extremely well and would thus score highly on certain items of the empathy questionnaires Baron-Cohen describes in his book. Furthermore, as Baron-Cohen acknowledges, people on the autism spectrum tend to have less empathy—at least certain forms of it—but they are not more malevolent than the population average. Therefore, reducing malevolence to “zero degrees of empathy” could be problematic or at least crucially depends on how we define and operationalize empathy.

Wikipedia notes that "Empathy is generally divided into two major components":

"Affective empathy, also called emotional empathy:[28] the capacity to respond with an appropriate emotion to another's mental states.[27] Our ability to empathize emotionally is based on emotional contagion:[28] being affected by another's emotional or arousal state."
"Cognitive empathy: the capacity to understand another's perspective or mental state."

And that article further notes:

Although science has not yet agreed upon a precise definition of these constructs, there is consensus about this distinction.[36][37] Affective and cognitive empathy are also independent from one another; someone who strongly empathizes emotionally is not necessarily good in understanding another's perspective.

And I remember being told during my psych undergrad that "psychopaths" have low levels of affective empathy but roughly average levels of cognitive empathy, while people on the autism spectrum have low levels of cognitive empathy but roughly average levels of affective empathy. (I haven't fact-checked this, though, or at least not for years.)

In the context of this post's arguments, I'd guess it's low affective empathy that is especially concerning, and perhaps especially when paired with high cognitive empathy.

(I assume you already know this stuff, but thought it might be useful for other readers.)

David_Althaus

Thank you! I agree that the distinction between affective and cognitive empathy is relevant, and that low affective empathy (especially combined with high cognitive empathy) seems particularly concerning. I should have mentioned this, at least in the footnote you quote.

And I remember being told during my psych undergrad that "psychopaths" have low levels of affective empathy but roughly average levels of cognitive empathy, while people on the autism spectrum have low levels of cognitive empathy but roughly average levels of affective empathy. (I haven't fact-checked this, though, or at least not for years.)

That sounds right. According to my cursory reading of the literature, psychopathy and all other Dark Tetrad traits are characterized by low affective empathy. While all Dark Tetrad traits except for narcissism also seem to correlate with low cognitive empathy, the correlation with diminished affective empathy seems substantially more pronounced (Pajevicc et al., 2017, Table 1; Wai & Tiliopoulos, 2012, Table 1).^[1] As you write, people on the autism spectrum basically show the opposite pattern (normal affective empathy, lower cognitive empathy (Rogers et al., 2006; Rueda et al., 2015).

We focused on the Dark Tetrad traits because they overall seem to better capture the personality characteristics we find most worrisome. Low affective empathy seems a bit too broad of a category as there are several other psychiatric disorders which don’t seem to pose any substantial dangers to others but which apparently involve lower affective empathy: schizophrenia (Bonfils et al., 2016), schizotypal personality disorder (Henry et al., 2007, Table 2), and ADHD (Groen et al., 2018, Table 2).^[2]

Of course, the simplicity of a unidimensional construct has its advantages. My tentative conclusion is that the D-factor (Moshagen et al., 2018) captures the most dangerous personalities a bit better than low affective empathy—though this probably depends on the precise operationalizations of these constructs. In any case, more research on (diminished) affective empathy seems definitely valuable as well.

Though Jonason and Krause (2013) found that narcissism actually correlates with lower cognitive empathy and showed no correlation with affective empathy. ↩︎
This list is not necessarily exhaustive. ↩︎

MichaelA🔸

Interesting!

Reading your comment, I realised one statement of mine may have been unclear. I wrote:

In the context of this post's arguments, I'd guess it's low affective empathy that is especially concerning, and perhaps especially when paired with high cognitive empathy

By that I just meant that, in terms of those two components of empathy, that might be the most concerning pattern. I didn't meant to imply that that's the key thing we should be concerned about in relation to personality traits more broadly - the idea that Dark Tetrad traits are a better focus seems reasonable to me (though I don't bring much independent knowledge to the table on that point).

Alexander Herwix 🔸

Thanks for writing this post, very interesting! I haven't read all of the comments but wanted to share one point that came to me over and over again while reading the post. Apologies if it has already been mentioned in another comment.

It seems like you assume a strong (and relatively simple) causal relationship from genetics to malevolent traits to bad behavior. I think this view might make the problem seem more tractable than it actually might be. Humans are complex systems that are nested in other complex systems and everything is driven by complex and interacting feedback loops. Thus, to me it seems very difficult to untangle causality here. To me it would be much more intuitive to think about malevolence as a dynamic phenomenon that is emergent based on a history of interactions rather than a static personality trait. If you accept this characterization as plausible the task of screening for malevolence in a valid and reliable way seems much more difficult than just designing a better personality test. I think the main difference between those two perspectives is that in the simple case you have a lot of corner cases to keep in mind (e.g., what is if people have malevolent traits but actually want to be good people?) whereas the complex case is more holistic but also much more, well, complex and likely less tractable.

Nevertheless, I agree with the general premise of the post that mental health is an important aspect in the context of X/S-risk related activities. I would go even further than this post and argue that mental health in the context of X/S-risk related activities in general is a very pressing cause area that would score quite well in terms of ITN-Analysis. Thus, I would really love to see an organization or network being set up dedicated to the serious exploration of this area because existing efforts in the mental health space seem only to be focus on happiness in the context of global development. If some interested in this topic reads this, don't hesitate to reach out, I would love to support such efforts.

Neil_Dullaghan🔹

Thanks for this!
I see you referenced Matthews et al. (2018), which I haven't read, and wondered if you had also seen the Authoritarian Ruling Elites Database, compiled by Matthews (2019): “a collection of biographical and professional information on the individuals who constitute the top elite of authoritarian regimes.” Each of the project’s 18 datasets focuses on a particular regime, such as the military dictatorship that ruled Chile from 1973 to 1990. The biographical data-points include gender, occupation, dates of birth and death, tenure among the elite, and more. (This came to me via the Data Is Plural mailing list fwiw). Apologies if you mentioned it and I missed it.

peter_janicki

applause. thx for this highly interesting (and important) article.

while reading i thought about a lot of commenting, but you already considered most of these things…

still some minor comments:

as to narcissism (maybe (?) the least important "dark tetrad trait"): as for narcistic personality disorder, there is a reason why (some) of these people are trying to gain power. a huge lack of self-esteem etc. and some narcistic people are trying to fill this lack with a successful career etc., still it does not gets filled this way, so (some of them) are trying to become even more successful... a vicious circle. but maybe even more important for narcissism usually has it´s origin in childhood/ growing up. i don´t believe, that genetic (epigenetic?) factors attribute a lot to this trait. (I only found that one here, and I don’t know, whether it is good or representative: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3973692/ - still, it seems, that those twins grew up with the same parents …).

as for scanning for dark tetrad-traits: smart malevolent people might deceive tests/scanning procedures, once they are smart enough, once they know what they want. they usually don´t do so, when still in kindergarten, in elementary school. And it is sure (unprecise) to scan for example for those kids, who torture animals (or peers) etc. i am not saying: “let´s do this”, only saying: “looking at kids might makes sense”.

as for proposing to cut f. e. only the 1% with the highest polygenetic score for dark tetrad-traits: it be interesting, if this would include persons like hitler, stalin, mao, … they sure had some luck with timing, being at the right place, living in instable times, … maybe they would only be in the top 10th percentile? still, it sure be a bad idea, to make big cuts without more reasoning and knowledge.

and: actually, even though (or because) i just read some things about dictators and sadism... i am not convinced. most journalists argue, that being responsible for the deaths of millions makes one a sadist. or because one makes up some torture-routines. but from my understanding, sadism is to gain direct benefit through direct torturing, humuliating (and seeing/getting feedback on their reaction) of humans (or animals). if someone lets millions of people getting killed or tortured - i guess this is for other reasons (economic profit, consolidating power - even if through terror or because someone simply does not care). having a lack of empathy, it might easily sound like a reasonable idea to send millions to deaths, if there are other things to gain. actually my understanding of sadism is not important. but if those dictators haven´t been sadists, then it might be a good idea, to exclude sadism out of this topic.

still: up until now in this comment, there are 3 lines of praise for this article, and some other 20 lines... It should be the other way round. thx again.

Tobias_Baumann

Thanks for commenting!

I agree that early detection in children is an interesting idea. If certain childhood behaviours can be shown to reliably predict malevolence, then this could be part of a manipulation-proof test. However, as you say, there are many pitfalls to be avoided.

I am not well versed in the literature but my impression is that things like torturing animals, bullying, general violence, or callous-unemotional personality traits (as assessed by others) are somewhat predictive of malevolence. But the problem is that you'll probably also get many false positives from those indicators.

Regarding environmental or developmental interventions, we write this in Appendix B:

Malevolent personality traits are plausibly exacerbated by adverse (childhood) environments—e.g. ones rife with abuse, bullying, violence or poverty (cf. Walsh & Wu, 2008). Thus, research to identify interventions to improve such environmental factors could be valuable. (However, the relevant areas appear to be very crowded. Also, the shared environment appears to have a rather small effect on personality, including personality disorders (Knopik et al., 2018, ch. 16; Johnson et al., 2008; Plomin, 2019; Torgersen, 2009).)

Perhaps improving parenting standards and childhood environments could actually be a fairly promising EA cause. For instance, early advocacy against hitting children may have been a pretty effective lever to make society more civilised and less violent in general.

EdoArad🔸

Thanks for a very thorough and interesting report!

It seem plausible that institutional mechanisms that prevent malevolent use of power may work well today in democracies. I think that the comparison is very important for understanding the value of the suggested interventions. You have briefly touched this -

Overall, it seems plausible that many promising political interventions to prevent malevolent humans from rising to power have already been identified and implemented—such as, e.g., checks and balances, the separation of powers, and democracy itself. After all, much of political science and political philosophy is about preventing the concentration of power in the wrong hands.[26] We nevertheless encourage interested readers to further explore these topics.

If these mechanisms are actually working quite well today, this somewhat lowers the importance of the suggested interventions. The analysis given above is mostly for non-modern institutions, but perhaps the court system, democracy and transparency has evolved so that malevolent actors can not really do much harm (or that it will be harder for them to get in power).

Also, the major alternative to reducing the influence of malevolent actors may be in the institutional decision making itself, or some structural interventions. AI Governance as a field seems to mostly go in that route, for example.

That said, I think that efforts going into your suggested interventions are largely orthogonal to these alternatives (and might actually be supportive of one another). Also, I intuitively find your arguments quite compelling.

David_Althaus

6y*

It seem plausible that institutional mechanisms that prevent malevolent use of power may work well today in democracies.

I agree that but there still seems to be room for improvement.

Of course, I agree that stable and well-functioning democracies with good cultural norms would benefit substantially less from many of our suggested interventions.

Also, the major alternative to reducing the influence of malevolent actors may be in the institutional decision making itself, or some structural interventions. AI Governance as a field seems to mostly go in that route, for example.

Just to be clear, I'm very much in favor of such "structural interventions". In fact, they overall seem more promising to me. However, it might not be everyone's comparative advantage to contribute to them which is why I thought it valuable to explore potentially more neglected alternatives where lower-hanging fruits are still to be picked.

That said, I think that efforts going into your suggested interventions are largely orthogonal to these alternatives (and might actually be supportive of one another).

Yes, my sense is that they should be mutually supportive—I don't see why they shouldn't. I'm glad you share this impression (at least to some extent)!

Milan Griffes

Somewhat related – US college students have been growing more narcissistic (or maybe just gaining higher self-esteem) since the 1980s: https://flightfromperfection.com/college-students-more-narcissistic.html

Max Harris

Here is an interesting paper related to using machine learning to identify language patterns in psychopaths if people are interested. https://egrove.olemiss.edu/cgi/viewcontent.cgi?article=1445&context=etd

Aaron Bergman

Another type of intervention that could plausibly reduce the influence of malevolent actors is to decrease intergenerational transfer of wealth and power. If competent malevolence both (i) increases one's capacity to gain wealth and/or power and (ii) is heritable, then we should expect malevolent families amass increasing wealth and power. This could be one reason that the global shift away from hereditary monarchies is associated with global peace (I sense that both of these things are correct, but am not positive).

For example, North Korea's Kim family is almost certainly malevolent in the way that this post describes, and the country's political structure enables this family to continually keep power.

On a broader scale, larger estate taxes and other economic policies to decrease wealth transfer might have a similar effect.

Dave Cortright 🔸

Big Think just posted a video about improving leader selection by screening for psychopathy including narcissism…

edcon

Thanks for the post! Really well reasoned on the broad impact of malovelence.

1. It seems that any research on manipulation proof measures for detection for malevolence, would help the development of tools that would be useful for a totalitarian state.

2. I'm sceptical of further research on malevolence being helpful in stopping these people being in positions of power. At first glance I don't think a really well developed literature on malevolence, would of changed leaders coming to power in 20th century.

3. In terms of Public engagement, I am also sceptical (eg. jon ronson the psychopath test) as I suspect that making people want more altruistic people in charge, is hard to move needle on. (Interesting piece on malevelonce in leaders https://theconversation.com/narcissists-and-psychopaths-how-some-societies-ensure-these-dangerous-people-never-wield-power-118854)

4. In terms of development of genetically engineered individuals, I think it really matters more who is in charge. I doubt bio-ethicists saying dark triad traits are bad will have much of an effect. In terms of further GWAS studies, I suspect by the time this becomes feasible more GWAS on desirable personality traits will have been undertaken. And it seems that work done ahead of time will be less useful, due to earlier studies being superseded by studies with more advanced technology .

David_Althaus

Thanks, these are valid concerns.

It seems that any research on manipulation proof measures for detection for malevolence, would help the development of tools that would be useful for a totalitarian state.

My guess is that not all research on manipulation-proof measures of malevolence would pose such dangers but it’s certainly a risk to be aware of, I agree.

I'm sceptical of further research on malevolence being helpful in stopping these people being in positions of power. At first glance I don't think a really well developed literature on malevolence, would of changed leaders coming to power in 20th century.

In itself, a better scientific understanding of malevolence would not have helped, agreed. However, more reliable and objective ways to detect malevolence might have helped iff there also had existed relevant norms to use such measures and place at least some weight on them.

I think it really matters more who is in charge. I doubt bio-ethicists saying dark triad traits are bad will have much of an effect.

Bioethicists sometimes influence policy though I generally agree with your sentiment. This is also why we have emphasized the value of acquiring career capital in fields like bioinformatics.

In terms of further GWAS studies, I suspect by the time this becomes feasible more GWAS on desirable personality traits will have been undertaken.

I agree that this is plausible—though also far from certain. I’d also like to note that (very rudimentary) forms of embryo selection are already feasible, so the issue might be a bit time-sensitive (especially if you take into account that it might take decades to acquire the necessary expertise and career capital to influence the relevant decision makers).

Vasco Grilo🔸

Hi David and Tobias,

I just came across Who would destroy the world? Omnicidal agents and related phenomena, which relates to your analysis. For reference, here is the abstract:

The capacity for small groups and even single individuals to wreak unprecedented havoc on civilization is growing as a result of dual-use emerging technologies. This means that scholars should be increasingly concerned about individuals who express omnicidal, mass genocidal, anti-civilizational, or apocalyptic beliefs/desires. The present article offers a comprehensive and systematic survey of actual individuals who have harbored a death wish for humanity or destruction wish for civilization. This paper thus provides a strong foundation for future research on “agential risks” and related issues. It could also serve as a helpful resource for counterterrorism experts and global risk scholars who wish to better understand our evolving threat environment.

However, the analysis is quite qualitative. Are you aware of research trying to quantify the number of omnicidal actors?

David_Althaus

I'm not aware of quantitative estimates of omnicidal actors. Personally, I'm less interested in omnicidal actors and more interested in actors that would decrease the quality of the long-term future if they had substantial levels of influence. This is partly because the latter type of category is plausibly much larger (e.g., Hitler, Mao, and Stalin wouldn't have wanted to destroy the world but would have been bad news regardless).

FWIW, I've done a few pilots on how common various "s-risk conducive" values and attitudes are (e.g., extreme retributivism, sadism, wanting to create hell) and may publish these results at some point.

Dvir Caspi

Regarding neurological measurements - there is actually really significant fMRI work which demonstrates lack of empathy and fearlessness. I would recommend reading some of the work by Prof. Abigail Marsh

ben.smith

I think the main solution is to develop strong and resilient institutions. Areas for improvement could be:

Distributing power over more individuals rather than less
Making office-holding unappealing for people with narcissistic or sadistic intentions or tendencies by increasing penalties for abuses of office
More transparency in government to make it harder to abuse the office
More checks and balances
Educating the electorate and building a healthier society so that people don’t want to elect a narcissist

[anonymous]

It seems like a strange claim that both the atrocities committed by Hitler, Stalin and Mao were substantially more likely because they had dark triad traits and that when doing genetic selection we're interested in removing the upper tail, in the article it was the top 1%. To take this somewhat naively, if we think that the Holocaust, and Mao and Stalin's terror-famines wouldn't have happened unless all three leaders exhibited dark tetrad traits in the top 1%, this implies we're living in a world world that comes about with probability 1/10^6, i.e 1 in a million, assuming the atrocities were independent events. This implies a need to come up with a better model.

Edit 2: this is also wrong. Assuming independence the number of atrocious should be binomially distributed with p=1/100 and n=#of leaders in authoritarian regimes with sufficiently high state capacity or something. Should probably be a markov-chain model.

If we adjust the parameters to top 10% and say that the atrocities were 10% more likely to happen if this condition is met, this implies we're living in a world that's come about with probability (p/P(Dark triad|Atrocity)^3, where p is the probability of that the atrocity would have occurred without Hitler, Stalin and Mao having dark triad traits. The interpretation of P(Dark triad|Atrocity) is what's the probability that a leader has a dark triad traits given they've committed an atrocity. If you have p as 0.25 and P(Dark|Atrocity) as 0.75 this means we're living in a 1/9 world, which is much more reasonable. But, this makes this intervention look much less good.

Edit: the maths in this section is wrong because I did a 10% probability increase of p as 1.1*p rather than p having an elasticity of 0.1 with respect to the resources put into the intervention or something. I will edit this later.

Excluding 10% of population from politcal power seems like a big ask. If the intervention reduced the probability that someone with dark triad traits coming to power (in a system where they could commit an atrocity) by 10%, which seems ambitious to me, this reduces the probability of an atrocity by 1% (if the above model is correct). Given this requires excluding 10% of the population from politcal power, which I'd say is generously 10%, this means that EV of the intervention is reducing the probability of an atrocity by 0.1%. Although this would increase if the intervention could be used multiple times, which seems likely.

Thomas Kwa🔹

I think an implicit assumption of the article is that people with dark triad traits are more likely to gain power. If unstable politics can create a 99:1 selection effect toward the Hitlers and Maos in the top %ile of dark triad, then they come to power in half of the nations with unstable politics. I can imagine this being true if people with dark triad traits are way more power-seeking and this is what matters in unstable political climates.

[anonymous]

So the model is more like, during the Russian revolution for instance it's a 50/50 chance that whichever leader came out of that is very strongly selected to have dark traid traits, but this is not the case for the contemporary CCP.

Yeah seems plausible. 99:1 seems very very strong. If it were 9:1 means we're in a 1/1000 world, 1:2 means an approx 1/10^5. Yeah, I don't have a good enough knowledge of rulers before they gained close to absolute power to be able to evaluate that claim. Off the top of my head, Lenin, Prince Lvov (the latter led the provisional govt' after Feb revolution) were not dark triady.

The definition of unstable also looks important here. If we count Stalin and Hitler, both of whom came to power during peacetime, then it seems like also should count Soviet leaders who succeeded Stalin, CCP leaders who succeeded Mao, Bashar al-Assad, Pinochet, Mussolini. Sanity check from that group makes it seem more much like a 1:5 than 1:99. Deng definitely not Dark Triad, nor Bashar, don't know enough about the others but they don't seem like it?

If we're only counting Mao, then the selection effect looks a lot stronger off the top of my head, but should also probably be adjusted because the mean of sadism seems likely much higher after a period of sustained fighting given the effect of prison guards for instance becoming more sadistic over time, and gennerally violence being normalised.

Don't know enough about psychopathy or machivallianism.

It's also not completely clear to me that Stalin and Mao were in the top 10% for sadism at least. Both came from very poor peasant societies. I know at least Russian peasant life in 1910 was unbelievably violent and they reguarly did things which we sort of can't imagine. My general knowledge of European peasant societies - e.g crowds at public executions - makes me think that it's likely that the average Chinese peasant in 1910 would have scored very highly on sadism. If you look at the response of the Chinese police/army to the 1927 Communist insurgency it was unbelievably cruel.

Makes screening for malicious actors seem worse and genetic selection seem better.

Apologies that this is so scattered.

MikeJohnson

A core 'hole' here is metrics for malevolence (and related traits) visible to present-day or near-future neuroimaging.

Briefly -- Qualia Research Institute's work around connectome-specific harmonic waves (CSHW) suggests a couple angles:

(1) proxying malevolence via the degree to which the consonance/harmony in your brain is correlated with the dissonance in nearby brains;
(2) proxying empathy (lack of psychopathy) by the degree to which your CSHWs show integration/coupling with the CSHWs around you.

Both of these analyses could be done today, given sufficient resource investment. We have all the algorithms and in-house expertise.

Background about the paradigm: https://opentheory.net/2018/08/a-future-for-neuroscience/

Ben_West🔸

A minor copyediting suggestion (adding the words in bold):

Factor 1—characterized by cruelty, grandiosity, manipulativeness, and a lack of guilt—arguably represents the core personality traits of psychopathy. However, scoring highly on factor 2—characterized by impulsivity, reactive anger, and lack of realistic goals—is less problematic from our perspective. In fact, humans scoring high on factor 1 but low on factor 2 are probably more dangerous than humans scoring high on both factors (more on this below).

It's not a big deal, but it took me a minute to understand why you were saying it's both less problematic and more dangerous.

David_Althaus

Thanks! Added your suggestion.

brb243

Thank you very much for the breathtaking analysis. I think that malevolent humans will best be prevented from gaining influence if benevolent humans outcompete them at influential posts.

David_Althaus

Thank you.

The section “What about traits other than malevolence?” in Appendix B briefly discusses this.

brb243

Thank you. I see. Then, I wish you the very best of luck in making an effective impact with your actions. I hope that you will succeed in preventing many decision-makers from promoting malevolent leaders.

Phil Tanny

Presumably, the same argument applies to banning scientific progress.

In case it's needed, I haven't made that argument, but have instead made this argument.

https://www.facebook.com/phil.tanny/posts/pfbid028hLRWeuJsDymwb3sgn4m8i1HLiA6YGbqAPNLgirFWEx4aBjXUp8t5E9pqBEYEjpsl

Phil Tanny

It's easy to say you're not calling for violence, but I don't see any way you could adopt this without violence, even if you had the technology to allow for it.

Ok, I accept your report that YOU don't see this. That's completely normal. There's a lot of it I don't see either. But once we TRY to see, some possibilities can emerge.

I agree that within the current group consensus, a "world without men" proposal will be widely, if not universally, rejected. I've been writing about this for years now, and that is what the evidence does show. But the current group consensus is not locked in concrete, and I propose that it will be edited by large real world events which are probably coming.

The problem is that the potential collapse of civilization is currently an abstraction, and most people are unable to engage abstractions emotionally. Real world events are likely to transform such an abstraction in to a form that large numbers of people can engage emotionally.

To pick an example randomly out of the hat, imagine even a limited nuclear exchange between Pakistan and India. More people will die in a few minutes than died in all of WWII. Global media will ABSOLUTELY OBSESS over this event for months or longer. Graphic scenes of mass destruction and vast suffering will be pumped in to every home across the civilized world, day after day after day after day. And then...

Large numbers of people across the world will finally grasp that this could happen to them too. They'll grasp it where it matters, emotionally, not just intellectually.

What happens next is anybody's guess, but it seems fair to claim the group consensus will then be up for grabs.

Human beings don't learn things this large through reason alone. We learn primarily through pain. And lots of pain is most likely coming.

Phil Tanny

Many longtermist effective altruists think that shaping transformative artificial intelligence, and in particular solving the alignment problem, is a particularly good lever to improve the long-term future.

What will happen is what always happens. The good people will use emerging technologies for well intended purposes, and the bad people will use emerging technologies in service to their agendas.

What seems to be missing from commentary on these subjects is the insight that as the scale of such emerging powers grows, the room for error steadily shrinks, and the bad (or stupid) people are increasingly in a position to bring down the entire system, thus erasing whatever benefits the well intended good people have managed to create.

Shouldn't we have learned this in 1945??

Nuclear power is a powerful energy generation technology which, when perfected, can go a long way in addressing climate change challenges.
But that doesn't really matter, because sooner or later a very long standing pattern of all out fight to the death conflict which has been routine for 10,000 years will reappear (as it's trying to now in Ukraine) and nuclear weapons will erase the benefits of nuclear power.

Right now, as we speak, we may be on the very edge of the largest nuclear power plant in Europe poisoning all of Europe. And that's because Putin can invade his neighbor with impunity. And that's because he has nuclear weapons which protect him from any truly effective response to his invasion.

There is no science magic which will transform every psychopathic power tripper on the planet (almost all of whom are men) in to nice guys who aren't a threat.

Hitler understood the laws of nature, and was very honest about it. He said, "We are the predator, and you are the prey." Hitler didn't invent that, he just understood that this has been the law of nature for a billion years. The big fish eat the little fish.

The science clergy is working as hard as they can, to make the big fish bigger. From the perspective of we the little fish, not such a great plan!

brb243

Prima facie, this looks like a thoroughly researched innovative piece recommending an important focus area. The notions of preventing malevolent actors from causing harm if they rise to power by developing robust institutions, advancing benevolent frameworks to enable actors join alternatives as they are gaining decisionmaking leverage, and creating narratives which could inspire malevolent actors to contribute positively while keeping true to their beliefs are not discussed. Thus, using this framework for addressing malevolence can constitute an existential risk.

Michael St Jules 🔸

Could you expand on this? Are you suggesting that it could take attention away from solutions ("inspiring malevolent actors to contribute positively while keeping true to their beliefs") that would reduce x-risk more?

By "keeping true to their beliefs", do you mean their malevolent views? E.g. we want a non harmful or less harmful outlet for their malevolence? Or, just working with their beliefs without triggering their malevolence on a large scale? If the latter, what kinds of beliefs do you have in mind?

brb243

Yes, if the community focuses on preventing people with the specified personality traits from gaining influence, it may become unpopular among prominent circles (not because leaders would necessarily possess these traits but because they may seek to present in agreement with the thinking, in order to keep their authority). Non-decisionmakers could be hesitant to publicly support frameworks which target actors which can be understood as threats, due to the fear of negative consequences for them. Thus, I am suggesting that while it is important to advance global good, this should be presented in a less conflicting manner.

By "keeping true to their beliefs" I mean fundamental righteousness/honorable behavior convictions. Yes, we would want an opportunity where they can be truly welcome and acknowledged for their actions. Or, working with their beliefs without triggering their malevolence on a large scale, since all decisions may have some costs in addition to their benefits – this is also meant by developing robust institutions that prevent powerful malevolent actors from causing harm.

An example can be a dictator who favors their own tribe. Then, if they can agree to their fundamental conviction as benefiting their own persons, and be excited about the modern legal personhood research, then they can gain satisfaction from skillfully manipulating their forces to achieve their objectives, being admired for that, and hurting their interlocutors in debates. If the dictator does not agree with interpreting their beliefs in a modern way, then prospects of the economic and geopolitical benefits of taking measures which consider different individuals to their own tribe can be introduced and less harm is caused.

Phil Tanny

-2

I find a far simpler insight to be useful.

The marriage between violent men and an ever accelerating knowledge explosion is unsustainable.

Violent men
Knowledge explosion
Pick one

I'm afraid the solution may require us simply making a choice between men and knowledge. I realize that academics may find such unsophisticated simplicity unsatisfying, but the reality may actually be simple. That's worth considering at any rate. Sometimes we invent complexity in the attempt to avoid hard choices that we don't wish to make.

To my knowledge, no society in history has figured out how to keep the peaceful men while losing the violent men. If true, then there's not really any compelling evidence that some magic technology will somehow solve the problem. If true, then to have men is to have violent men.

And to have violent men empowered by an ever accelerating knowledge explosion sounds like a recipe for civilization collapse to me. If that sounds like hysterical speculation then we might recall that Putin already has the ability to destroy modern civilization in just minutes. In a few more years the North Korean regime will be able to destroy at least America. And the 21st century is still young.

I believe there is a way out of the box, but the evidence I've seen from years of writing about this, including today on this forum, strongly suggests we are simply too attached to memorized dogmas and the past to access the way out. That's sad, but it's real.

Almost all the violence in the world is committed by men.

A simple fact showing the way forward.

If we have the vision and courage to do what is needed to save our civilization.

(PLEASE NOTE THERE ARE EXACTLY NO CALLS TO VIOLENCE IN THIS POST. THANK YOU.)

Jay Bailey🔸

The argument you're making seems strange to me. I'm going to talk purely about the logistics here. You say that no society in history has figured out how to keep peaceful men without the violent ones, and therefore this means it probably can't be done. Presumably, the same argument applies to banning scientific progress.

But the same argument can be applied to a society without men at all! No society in history has managed to do it. In fact, the obstacles are far greater - men are still biologically necessary, there are billions of men already on the planet, and most of them wouldn't agree with you that their gender should be phased out. It's easy to say you're not calling for violence, but I don't see any way you could adopt this without violence, even if you had the technology to allow for it.

Finally, it occurs to me that if your ideal-but-untenable solution is to keep peaceful men and not violent men, then your solution is anti-correlated with what you want - what kind of man would willingly permit his own gender's extinction for the greater good of humanity? Certainly not a violent one.

It seems to me like you've correctly identified a large problem that lurks behind several EA causes (The exponentially advancing state of knowledge resulting in existential risk) and a compelling analogy for it (The ever-increasing speed of a conveyor belt) which is a very useful question worth exploring. But then, having successfully identified this as a hard problem, you've then shied away from the difficulty of the problem, which is actually solving the problem. If you propose an unworkable solution, and then people ignore it, you at least get to say that you tried, and other people are to blame for denying your vision. In other words, you are Trying To Try.

In short - you've proposed a solution that would almost certainly fail in real life. It is in fact much more difficult than lesser impossibilities like "Remove the tendency for violence in men/humans" or "Halt all scientific progress in any field that is deemed unsafe". It might not even solve the problem if it succeeded. And it also happens to be outside the Overton window to boot. And then, when people downvote your posts, you assume that they disagree with the solution because it's outside the Overton window and not because it's just...not going to work. And even if it could work, the solutions I mentioned above are better across every possible dimension. They're slightly less impossible, morally more reasonable, societally more tenable, and equally likely to solve the problem if you could somehow pull them off. And even then, those aren't good solutions because they STILL don't bring the problem into a state where a solution can realistically be seen.

The problem you mention is real, but you've cut your search for a solution short way too fast.

Phil Tanny

Thank you very much for engaging my comment Jay. I almost missed your reply, and am thankful I didn't.

Given that ideas widely accepted by the group consensus have utterly failed to cure the rampant human (mostly male) violence which has long permeated our world, my goal is to encourage explorations in to ideas outside of the current group consensus. The reasoning here is simple. If ideas within the group consensus could solve this existential threat, the problem would likely already be solved.

One of the challenges we will face in conducting such "out of the box" investigations is that many or most of those who make their livings doing intellectual work are not in a position to participate, at least not publicly. As example, if I was a philosophy professor working at a university, I would not be able to post about a "world without men" without fear of damage to my reputation, and thus my salary, my mortgage, and my children's college fund. As evidence, I've already been threatened with being banned from this forum simply for mentioning a "world without men" might be an option worth exploring. Sorry to say, fear abounds in academia.

You are right. The challenges involved in a "world without men" proposal are vast. But then so is every other plan for saving our civilization from male violence. To me, it seems useful to be simple and clear minded in identifying the source of the threat, men.

Men will soon no longer be biologically necessary, as science is learning how to make sperm and egg cells from skin cells. To my limited knowledge such work is already well underway in mice, but not in humans.

In any case, it's already true that a tiny number of men can impregnate a vast number of women. So while a world totally without men may not yet be an option, a world with far fewer men is. Critics of my posts never seem to get this far, because they invest all of their energy in to rejection, rejection, rejection, which obscures their own common sense analysis.

Also, why should men resist the idea of their being fewer men??? The fewer men there are, the more valuable the remaining men become in the dating pool. This is another obvious point critics always ignore in their rush to rejection.

But then, having successfully identified this as a hard problem, you've then shied away from the difficulty of the problem, which is actually solving the problem.

This is an understandable critique, which is welcomed. But to be more precise, what is really being said is that I've shied away from solving the problem in the manner preferred by those who make their living serving an out of control knowledge explosion. If such folks can solve the problem in the manner which they prefer, that's great, I'm all for solving the problem. Let's hear their solution....

As I see it, the underlying problem is a failure to think holistically, and acknowledge that the human condition with all it's many weaknesses is one component of the knowledge development system. My reasoning is that, given that human beings are limited (like every other species) then knowledge development will have to be as well. It simply doesn't make sense to keep piling more and more and more power on to a species that can't manage the power we already have (thousands of hydrogen bombs aimed down our own throat).

Yes, the argument I'm making seems strange to you. It seems strange to most people. That's what I like about it. We're exploring outside the limits of a group consensus which has a proven record of failing to successfully address these existential threats.

Being outside the box doesn't guarantee success, agree completely there. But where else would you have me look?

Thanks again for engaging. I was on the edge of retiring from the forum, so you've brought me back from the edge, for now.

This is obviously a huge topic and I'm attempting to not write a book in each of my posts. If there is some angle you feel I've not adequately addressed, please remind me of it and I'll try again.

Jay Bailey🔸

The reasoning here is simple. If ideas within the group consensus could solve this existential threat, the problem would likely already be solved.

First off - this isn't necessarily true. There are ideas within the group consensus that could at least take steps towards solving nuclear war (advocate for nuclear disarmament), climate change (advance the state of carbon capture technology), and reduce global poverty. (Basically everything GiveWell does) The reason we haven't solved them yet is that executing these ideas is still very hard due to problems around logistics, co-ordination, and they simply require a hell of a lot of work.

That said, I do agree with you that looking at the root cause of exponentially increasing x-risk capable technologies is not an idea that is currently discussed much. But that should actually move you towards thinking inside the box - if people aren't thinking about it, there's low-hanging fruit to be picked, and you don't need to go straight to the most radical idea you can think of. Maybe there are some good ideas within the Overton window that haven't been popularised yet, just waiting to be found!

This is an understandable critique, which is welcomed. But to be more precise, what is really being said is that I've shied away from solving the problem in the manner preferred by those who make their living serving an out of control knowledge explosion. If such folks can solve the problem in the manner which they prefer, that's great, I'm all for solving the problem. Let's hear their solution...

No, that's not what's being said. What I am saying is that you've shied away from solving the problem. "Solving the problem" means coming up with a solution that has a chance of actually working giving the constraints of the real world. In the real world, I cannot see any sort of realistic path towards your solution, and coming up with such a path is a requirement for a valid solution. It is not enough to describe a possible state of the world that solves a problem - you have to have some path to get from our current world to that world.

I also gave you two ideas for the limitation of knowledge development - ban all research in any areas deemed to be prone to existential risk, and find a way to remove the tendency for highly aggressive men to seek power in world-threatening ways. Don't get me wrong, these still don't count as solutions in my definition above, but they seem at least as plausible as removing the male gender entirely. It seems "probably impossible" rather than "definitely impossible", and limiting research even suggests some possible next steps, like banning biological gain-of-function research (even though these are still very hard and I've heard people make decent arguments for why it can't be done, largely around the difficulty of international co-operation - but this is also an argument that applies equally well to your proposal)

Yes, the argument I'm making seems strange to you. It seems strange to most people. That's what I like about it. We're exploring outside the limits of a group consensus which has a proven record of failing to successfully address these existential threats.
Being outside the box doesn't guarantee success, agree completely there. But where else would you have me look?

You seem to have the idea that people are disagreeing with you because your ideas are strange and outside the box. This is from the same forum that has wild animal suffering as a small but real cause area. When I said your argument seemed strange, I was being polite. What I should have said was that your argument seems bad. Not because it's outside the box. Not because I disagree with it morally. Because I don't see how it could actually work, and there seem to me to be options that are simultaneously less radical and also closer to the set of things I would describe as possible solutions. The reason to think out of the box is because nothing else has worked - you should be aiming for the least radical solution that is necessary to solve the problem.

Finally, while I don't think this point is necessary for my core argument, I'll address it anyway:

Also, why should men resist the idea of their being fewer men??? The fewer men there are, the more valuable the remaining men become in the dating pool. This is another obvious point critics always ignore in their rush to rejection.

Firstly - a society that wants to eliminate the male gender probably isn't going to be very nice for men to live in. You can't have a society dedicated to the extinction of men that also respects, appreciates, and is kind to them. Any attitudes society adapted that would allow them to even consider such a radical act are going to be opposed by men for precisely this reason - which, by the way, is an extremely good reason and has my full sympathy. There are people right now who advocate for the extinction of men, and they are not the kind of people I would feel safe around, let alone being ruled by.

Secondly - let's say I don't care about the future of men, and I don't care what people think of me, and I'm certain that society's newfound attitudes that allow for the intentional extinction of men definitely won't turn into violence, and I'm willing to throw away my entire gender to increase my chance of getting laid. Even then, I'm thirty years old now. How does this help me? If you're not going to commit violence, which you've previously claimed, the only way you can reduce the number of men is to stop new ones from being born. That means that if you could start having statistically relevant effects a mere ten years from now, I'm still going to be in my sixties by the time the dating pool starts shifting in any sort of meaningful way, and there's still just as many 30+ year olds as there ever was. That doesn't seem like it benefits me.

Finally, if most humans were willing to consider twenty years in the future when deciding on what policies they support, we might not be in this mess in the first place.

Phil Tanny

Hi again Jay, I'm enjoying your responses, thanks.

There are ideas within the group consensus that could at least take steps towards solving nuclear war

Your point is taken on climate change and poverty. Nuclear war seems an example of conventional thinking which has failed.

That said, I do agree with you that looking at the root cause of exponentially increasing x-risk capable technologies is not an idea that is currently discussed much.

Thanks for seeing that. If I can change that a little bit, it's worth a try. Like everybody else, I'm just doing what little I know how to do.

you don't need to go straight to the most radical idea you can think of

That's true. But it is that radical idea that engaged your interest, so sometimes it works. I'm not claiming to have the perfect engagement system etc, again, I'm just doing what I know how to do.

"Solving the problem" means coming up with a solution that has a chance of actually working giving the constraints of the real world.

What I see is that many of my readers often come to hard fixed opinions on what is possible after reading two paragraphs of one of my posts, on a subject they've never really thought about. And then all their further effort goes in to defending those positions.

You've helped me think of an example from the real world that seems somewhat relevant to the "world without men" idea. China's one child policy. To the best of my knowledge that was pursued with financial carrots and sticks, not violence (not totally sure about this).

In the real world a great many people around the world have been prioritizing male babies over female babies for centuries. This demonstrates that when people have some population or gender agenda, they can take steps that move them towards their goals.

In the real world, I cannot see any sort of realistic path towards your solution,

How long have you been trying? Today and yesterday? I do agree that the world without men proposal faces a very large pile of challenges, I really do. But what proposal for radically reducing human violence doesn't?

What you, and most readers, are not focusing on are the enormous game changing benefits that a world without men can credibly be said to offer. That's not a good enough reason to immediately agree, but is does seem a good enough reason to explore further. That's all I want. Let's explore further.

I also gave you two ideas for the limitation of knowledge development - ban all research in any areas deemed to be prone to existential risk, and find a way to remove the tendency for highly aggressive men to seek power in world-threatening ways.

I'm receptive to that discussion. Let's explore that too. I'm modestly optimistic on the first, less so on the second. My optimism will increase at the point of some real world catastrophe highlighting the problem in a manner many more people can access. It looks like it will have to be catastrophe of significant scale though. And probably something easy to understand, like a nuclear event for instance.

You seem to have the idea that people are disagreeing with you because your ideas are strange and outside the box.

I've been floating this idea for years all over the net. I could indeed be wrong, but that is my current working theory. The theory comes from the observation that the overwhelming majority of respondents have been focused on automated rejection, not analysis. They typically never try to solve the puzzle, preferring the easier route of immediately declaring it impossible. It's human nature. We want radical improvements, by doing all the same stuff we've always done.

Firstly - a society that wants to eliminate the male gender probably isn't going to be very nice for men to live in. You can't have a society dedicated to the extinction of men that also respects, appreciates, and is kind to them.

I must respectfully disagree here. Everybody wants to look at this through the "you must hate men" lens. As I see it, a society that wants to remove the male gender is taking the necessary bold action to create a near utopia (compared to today) for the human race.

What is so important about the male gender that our existence justifies unspeakable suffering on a vast scale over thousands of years? Why is such a concept rational???

The women I envision are human beings capable of making a bold, rational, altruistic decision which benefits the future of humanity in a revolutionary manner. It's not about hate. It's about reason. Civilization survival. A brighter future for humanity. It's about adapting to the radical new environment the knowledge explosion has created.

NATURE'S RULE: Those who don't adapt must die.

That doesn't seem like it benefits me.

You think you have problems, I'm 70! I still look at the gals. They stopped looking back years ago. :-) Why do you think I'm trying to save the world?? I got nothing else to do dude! :-)

Good conversation! You're personally making my attendance here worth the trouble, and I thank you for that. If you start a thread on the site and want engagement, let me know, I'll do my best.

Jay Bailey🔸

That's true. But it is that radical idea that engaged your interest, so sometimes it works. I'm not claiming to have the perfect engagement system etc, again, I'm just doing what I know how to do.

The idea that engaged my interest was that of the exponential knowledge explosion. I thought there was a good idea there, and I replied since I had seen a lot of your posts recently and nobody else was replying to them. I replied in spite of, not because of, the proposed solution. I imagine it's quite likely that others decided not to reply because of the proposed solution as well.

I apologise for misleading you there - I'll focus just on the exponential knowledge explosion side of things to help correct that.

Me: In the real world, I cannot see any sort of realistic path towards your solution,

You: How long have you been trying? Today and yesterday?

The above line by me was an invitation for you to share a path. You're right - my inability to see a path after five minutes isn't a strong indication that no such path exists. On the other hand, you've had this idea for years - if you don't have a path forward, that's much stronger evidence for the idea's untenability. If you do have a path forward, this would be a good thing to add to your argument.

You seem to have the idea that people are disagreeing with you because your ideas are strange and outside the box.

This sentence was referring entirely to the EA Forum - people tend to be more tolerant of weird ideas here than in most places on the internet.

Phil Tanny

The idea that engaged my interest was that of the exponential knowledge explosion

Ok, great, thanks. As you can now see, I'm obsessed with this.

The knowledge explosion idea and world without men idea are somewhat linked in my mind. Meaning, if violent men weren't creating havoc all over the planet, there would be less reason to worry about the knowledge explosion. If there was no knowledge explosion, there'd be less reason to worry about violent men.

I do have a plan for moving forward on the world without men concept. The plan is to recognize that no single human being, certainly not me, can meet such an enormous challenge, so it's necessary to engage many bright minds to work on the issue. That is, I have faith that once many bright minds are engaged, progress can be made. I'm asking readers to have as much faith in themselves as I have in them. Instead of chanting "we can't" we might instead choose to start believing that we can.

I have a few detail ideas, such as financial incentives to reward female babies etc. Admittedly I am no where near a complete solution. I'm puzzled as to why anyone would think that I was, should be, or could be.

My main contribution, as I see it, is to boil down all the endless complexity in to a simple verifiable truth. The overwhelming vast majority of violence is committed by men. It's not human violence that we're talking about, it's male violence. Being clear on this creates a target of opportunity.

The other thing I hope to contribute is to articulate the revolutionary positive benefits that would flow from radically reducing male violence. If this problem could be solved, the pay day would be almost beyond imagination.

The bumper sticker version is this:

The marriage between violent men and an ever accelerating knowledge explosion is unsustainable. One of them has to go.

Well, you'll probably need to buy a big truck to get all that on the bumper.

Jay Bailey🔸

The marriage between violent men and an ever accelerating knowledge explosion is unsustainable. One of them has to go.

I think our core crux here is that if this is true, I would rather tackle it from the "ever accelerating knowledge explosion" side or the "violent" side rather than the "men" side.

Good luck with your ideas, man. You've certainly given me a new idea to think about (knowledge explosion) and I hope I've done the same.

Phil Tanny

I think our core crux here is that if this is true, I would rather tackle it from the "ever accelerating knowledge explosion" side or the "violent" side rather than the "men" side.

Ok, that's cool. To each their own of course. But we don't really have to choose, we can investigate both at the same time.

Should you start writing about the knowledge explosion side of things you will find it just as challenging as the "world without men" side. Search for my article "Our Relationship With Knowledge" on this site, it's been down voted like crazy, no engagement at all. That's pretty normal.

The science community in particular (those who have the most cultural authority on such issues) will view your writing on limiting the knowledge explosion much the same way the Catholic Church viewed atheists in the 12th century. You will be dismissed, ridiculed, scorned, deleted, banned etc. But probably not burned at the stake, so that's good. :-)

If I was pressed to choose, I'd probably agree the "world without men" idea makes people more hysterical. But a notion that we can't have never ending unlimited goodies from the knowledge explosion does not exactly delight people, to say the least.

Both of these ideas are wildly ambitious. Neither has much of a chance until MAYBE after some large historical event which undermines the assumption of the group consensus that we can somehow magically have our cake and eat it too, radical change for the better, without radical change.

Of course, assessing other people’s personality is always fraught with uncertainty, especially if they are long dead. ↩︎
Other researchers have suggested similar constructs aimed to represent the common core of “evil” (e.g., Book et al., 2015; Jones & Figueredo, 2013; Marcus et al., 2018). ↩︎
Baron-Cohen (2012) argues that the defining feature of human evil is “zero degrees of empathy.” However, some psychopaths can read other people extremely well and would thus score highly on certain items of the empathy questionnaires Baron-Cohen describes in his book. Furthermore, as Baron-Cohen acknowledges, people on the autism spectrum tend to have less empathy—at least certain forms of it—but they are not more malevolent than the population average. Therefore, reducing malevolence to “zero degrees of empathy” could be problematic or at least crucially depends on how we define and operationalize empathy. ↩︎
As of now, there is no established treatment of malevolence. Harris and Rice (2006) review the empirical findings on the treatment of psychopathy but are quite pessimistic about their effectiveness. ↩︎
A more rigorous analysis would be valuable, though it would also be methodologically challenging—assessing the personality traits of historical figures, for example, is rather difficult. ↩︎
One reason to be hesitant here is that it seems plausible, for instance, that we, as well as journalists and historians, see more signs of malevolent personality traits in leaders who have caused great harm, and will tend to overlook malevolent personality traits in leaders who have done more good. ↩︎
Again, we refer to Appendix A for more details. ↩︎
For example, without Mao and Stalin the probability of a communist China is smaller. A non-communist China may have better relations with the U.S., and the probability of great power wars and (AI) arms races may be reduced. However, such claims are necessarily very speculative. For instance, one could also argue that World War II may have increased longer-term stability by leading to the formation of the UN. ↩︎
Or, as Robert Hare, one of the most well-known researchers of psychopathy, puts it: “Serial killer psychopaths ruin families. Corporate, political and religious psychopaths ruin economies [and] societies.” (Ronson, 2012, p. 117). ↩︎
Gallagher & Allen (2014) found that U.S. presidents scoring higher on the Big Five facet “altruism” were less likely to employ military force. ↩︎
Also compare MacAskill: “I still endorse the view of pushing resources into the future. The biggest caveat actually I’d have is about the rise of fascism and Stalinism as the thing to push on [...] even though you might not think that a particular ideology will last forever, well, if it lasts as long until you get like some eternal lock-in event, then it lasts forever. [...] I kind of think the rise of fascism and Stalinism was a bigger deal in the 20th century than the invention of nuclear weapons.” (MacAskill, 2020). ↩︎
Dark Triad traits in political candidates correlate with more negative campaigns and fear appeals (Nai, 2019). ↩︎
Since Dark Triad traits correlate with social dominance orientation (SDO, Jones & Figueredo, 2013), malevolent leaders will, on average, exhibit higher SDO and prefer policies resulting in higher social and economic inequality. ↩︎
Azizli et al. (2016) find that psychopathy and Machiavellianism are associated with a greater propensity to lie and engage in high-stakes deception. ↩︎
Bowler & Karp (2004) find that scandals involving politicians tend to lower political trust. It seems plausible that malevolent political leaders are more likely to be involved in scandals. ↩︎
According to Ulrich et al. (2001), the rate of antisocial personality disorder among criminal offenders, 45% of whom were convicted of “robbery or extortion,” is more than 10 times higher than that of the control sample. Jonason et al. (2012) also find that Machiavellianism, narcissism, and psychopathy all correlate with the use of “hard tactics” in the workplace, including “threats of punishment” (see Table 1, p. 451). ↩︎
MacAskill (2020, emphasis added): “[...] when you look at history of well what are the worst catastrophes ever? They fall into three main camps: pandemics, war and totalitarianism. Also, totalitarianism or, well, autocracy has been the default mode for almost everyone in history. And I get quite worried about that. So even if you don’t think that AI is going to take over the world, well it still could be some individual. And if it is a new growth mode, I do think that very significantly increases the chance of lock-in technology.” ↩︎
Kaja Perina on the Manifold podcast (Perina, 2020): “[M]ost of the studies on psychopaths [...] are done on inmates. For that reason, we’re forced to conjecture about the really successful ones because I think the more successful, the more they evade detection, perhaps, lifelong. So there is this disconnect wherein a lot of them, the violent ones, the less intelligent ones, really end up in jail, and these are the ones who are studied, but these are not the ones who are highly Machiavellian, necessarily, these are not the ones who are brilliantly manipulative. These are the ones who are committing violent crimes and get caught.” ↩︎
Extensive background checks, for example with the help of private investigators, would be another promising possibility. Intelligence agencies do this already for somewhat related purposes. Generally, the competitive nature of the political process can often uncover past immoral behavior—though swaying partisan views seems to require evidential strength that is difficult to achieve. Thanks to Mojmír Stehlík for raising these points. ↩︎
Yet another possibility would be to use “objective” personality tests that don’t rely on self- or other-report but use actual performance tests to evaluate personality traits (without the test-taker knowing which trait is supposed to be measured). However, according to our cursory reading of the literature, the few “objective” personality tests that exist seem to have low validity (e.g., Kline & Cooper, 1984). ↩︎
However, one needs examples to train such predictors in the first place. One could start by looking for differences in the brains of normal people and, say, diagnosed psychopaths, but this metric will be biased towards diagnosed psychopaths who are at least somewhat unrepresentative of non-diagnosed malevolent humans (as explained above). One needs to correct for this ascertainment bias. ↩︎
Relatedly, neuroscience research is often underpowered, resulting in low reproducibility of the accumulated findings (Button et al., 2013). ↩︎
However, the education system also involves a lot of tests and grading; and is at least somewhat related to career advancement. Such tests are also common for military entry and sometimes civil service. In the context of elections, the key question is why voters do not generally demand such tests (including related objective measures, such as tax returns). ↩︎
However, there exists the frightening possibility that some voters want their political leaders to be at least moderately malevolent. Most Russians and Chinese, for example, seem to think highly of Stalin and Mao, respectively—though this is likely at least partly due to propaganda. Generally, many voters seem to like “strong men” like Putin and overlook or even appreciate elevated Dark Tetrad traits in their political leaders. Also, according to the (potentially biased) assessment of “experts”, politicians with autocratic tendencies—many of whom nonetheless received the majority of votes—score significantly higher on Dark Triad traits than the average politician (Nai & Toros, 2020). ↩︎
It is also worth noting that in some forms of government, such as allocating political positions to randomly selected individuals or hereditary monarchy, those in positions in power are exactly as likely to be malevolent as the population at large. This may be better than fierce competition for positions of power if the latter advantages the most ruthless and malevolent individuals. On the other hand, good selection procedures could also reduce malevolence in positions of power below the baseline; and of course this is only one consideration among many when evaluating different forms of government. ↩︎
Thanks to Richard Ngo for making this point. ↩︎
However, initial distributions may change due to competitive pressures or other factors. Even if none of the first ems are malevolent, there is no guarantee that malevolence will remain absent in the long run. ↩︎
Extreme events like severe abuse or violence can make a huge difference for the victims, but such events are relatively rare and therefore do not explain much variance in the general population (Plomin, 2019). ↩︎
Also note that shifting personality traits by more than this would likely be very difficult even if one wanted to do this. ↩︎
Some dark traits, such as Machiavellianism, can be beneficial under certain circumstances. It might be better if one could single out a dark trait, such as sadism, and only select against it while leaving other dark traits unchanged. ↩︎
However, GWA studies of personality are still fairly weak even at such scales. Even higher sample sizes might be achieved by identifying proxy variables of malevolence, such as public records on crime. However, this could easily backfire and cause great harm in numerous ways, so one would have to be very careful. ↩︎
Currently, Dark Tetrad traits seem to be neglected in many relevant areas. Most sperm banks measure traits such as height, attractiveness, physical and mental health but not Dark Tetrad traits. Services that offer pre-implementation diagnostics screen for all sorts of genetic diseases and some even for IQ but not for Dark Tetrad traits. Test batteries of enormous government projects like the UK Biobank measure thousands of variables—including physical health, height, and preferred coffee and cereal type—but they don’t measure Dark Tetrad traits or even most personality traits in any sort of rigorous manner. ↩︎
Generally, reducing the influence of malevolent actors can be done in myriad (often mutually reinforcing) ways. For instance, the more widespread belief systems are that put great value on non-malevolent traits such as compassion and altruism, the more parents might demand screening for malevolent traits, and the more future (government) projects will include measures of malevolence in their test batteries. ↩︎
Adopting certain ideologies could also make one more malevolent. However, we think it’s plausible that most of the correlation is explained by causation from malevolent traits to dangerous ideologies, partly because personality traits seem less amenable to change than beliefs. ↩︎
The Dark Triad also predicts sexism (Gluck et al., 2020; O'Connell & Marcus, 2016), nationalism (Matthews et al., 2018) as well as cognitive and affective prejudice (Koehn et al., 2019). Psychopathic traits predict opposition towards free speech and animal rights as well as support for using war as a tool for diplomacy (Preston & Anestis, 2018, Table 3). Machiavellianism and narcissism also seem to correlate with overconfidence (Campbell et al., 2002; 2004; Macenczak et al., 2016; Jain & Bearden, 2011), and thus plausibly negatively correlate with epistemic humility which should serve a protective function against all kinds of extremism and fanaticism. (Note that we don’t intend to convey that all these associations are equally dangerous.) ↩︎
Many of these and similar quotes could have been made solely for political reasons, e.g., to strengthen alliances. ↩︎

Reducing long-term risks from malevolent actors

Coup Proofing is a common practice of dictators for political survival

Change in ideological motivation between early dictators and modern dictators

Reducing long-term risks from malevolent actors

Summary

What do we mean by malevolence?

Malevolent humans in power pose serious long-term risks

Malevolent humans often rise to power

History suggests that malevolent leaders have caused enormous harm

Malevolent leaders have the potential to corrupt humanity’s long-term future

Broad risk factors due to malevolent leaders

Existential and suffering risks due to malevolent leaders

Interventions to reduce the influence of malevolent actors

Advancing the science of malevolence

Developing better constructs and measures of malevolence

The most dangerous individuals tend to go undiagnosed

Manipulation-proof measures of malevolence

Potential misuse and negative consequences

How valuable would manipulation-proof measures of malevolence be in practice?

Political interventions

Future technologies and malevolence

Whole brain emulation

Transformative AI

Genetic enhancement

Overview of genetic enhancement technologies

Dangers

Interventions

Concluding remarks

Appendix A

How important are situational factors and ideologies compared to personality traits?

How well can people detect malevolent traits?

Appendix B

Acknowledgements

References