Comment Permalink

Thank you for sharing, but I've read your post and am not convinced (either in this instance or in general). I think it was a fine comment to which you reacted with unwarranted negativity. Or, in short: no, you're wrong.

Scouts need soldiers for their work to be worth anything

by Holly Elmore ⏸️ 🔸

Nov 7 20243 min read 22

116

Building effective altruismCommunityCommunity epistemic healthLess-discussed causesRationalityScout mindset

Frontpage

Show all topics

Since I started PauseAI, I’ve encountered a wall of paranoid fear from EAs and rationalists that the slightest amount of wrongthink or willingness to use persuasive speech as an intervention will taint the person’s mind for life with self-deception-- that “politics” will kill their mind. I saw people shake in fear to join a protest of an industry they thought would destroy the world if unchecked because they didn’t want to be photographed next to an “unnuanced” sign. They were afraid of sinning by saying something wrong. They were afraid of sinning by even trying to talk persuasively!

The worry about destroying one’s objectivity was often phrased to me as “being a scout/not being a soldier”, referring to Julia Galef’s book Scout Mindset. I think we have all the info we need to contradict the fear of not being a scout in her metaphor. Scouts are important for success in battle because accurate information is important to draw up a good battle plan. But those battle plans are worthless without soldiers to fight the battle! “Everyone Should be a Mapmaker and Fear that Using the Map to Actually Do Something Could Make Them a Worse Mapmaker” would be a much less rousing title, but this is how many EAs and rationalists have chosen to interpret the book.

Even a scout can’t be only a scout. If a scout reports what they found to a superior officer, and the officer wants to pretend they didn’t hear it, a good scout doesn’t just stay curious about the situation or note that the superior officer has chosen a narrative. They fight to be heard! Because the truth of what they saw matters to the war effort. The success of the scout and the officer and the soldier is all ultimately measured in the outcome of the war. Accurate intel is important for something larger than the map— for the battle.

Imagine if the insecticide-treated bednets hemmed and hawed about the slight chance of harm from their use in anti-malaria interventions. Would that help one bit? No! What helps is working through foreseeable issues ahead of time at the war table, then actually trying the intervention with each component fully committed. Bednets are soldiers, and all our thinking about the best interventions would be useless if there were no soldiers to actually carry the interventions out. Advocating for the PauseAI proposal and opposing companies who are building AGI through protests is an intervention, much like spreading insecticide-treated bednets, but instead of bednets the soldiers are people armed with facts and arguments that we hope will persuade the public and government officials.

Interventions that involve talking, thinking, persuasion, and winning hearts and minds require commitment to the intervention and not simply to the accuracy of your map or your reputation for accurate predictions. To be a soldier in this intervention, you have to be willing to be part of the action itself and not just part of the zoomed out thinking. This is very scary for a contingent of EAs and rationalists today who treat thinking and talking as sacred activities that must follow the rules of science or lesswrong and not be used for anything else. Some of them would like to entirely forbid "politics" (by which they generally mean trying to persuade people of your position and get them on your side) or "being a [rhetorical] soldier" out of the fear that people cannot compartmentalize persuasive speech acts from scout thinking and will lose their ability to earnestly truth-seek.

I think these concerns are wildly overblown. What are the chances that amplifying the message of an org you trust in a way the public will understand undermines your ability to think critically? That's just contamination thinking. I developed the PauseAI US interventions with my scout hat on. When planning a protest, I'm an officer. At the protest, I'm a soldier. Lo and behold, I am not mindkilled. In fact, it's illuminating to serve in all of those roles-- I feel I have a better and more accurate map because of it. Even if I didn't, a highly accurate map simply isn't necessary for all interventions. Advocating for more time for technical safety work and for regulations to be established is kind of a no-brainer.

It's noble to serve as a soldier when we need humans as bednets to carry out the interventions that scouts have identified and officers have chosen to execute. Soldiers win wars. The most accurate map made by the most virtuous scout is worth nothing without soldiers to do something with it.

116 Reactions

More posts like this

Comments22

Sorted by

New & upvoted

Click to highlight new comments since: Today at 12:19 PM

Tao8mo43

This is a valuable post, but I don't think it engages with a lot of the concern about PauseAI advocacy. I have two main reasons why I broadly disagree:

Pausing AI development could be the wrong move, even if you don't care about benefits and only care about risks

AI safety is an area with a lot of uncertainty. Importantly, this uncertainty isn't merely about the nature of the risks but about the impact of potential interventions.

Of all interventions, pausing AI development is, some think, a particularly risky one. There are dangers like:

Falling behind China
Creating a compute overhang with subsequent rapid catch-up development
Polarizing the AI discourse before risks are clearer (and discrediting concerned AI experts), turning AI into a politically intractable problem, and
Causing AI lab regulatory flight to countries with lower state capacity, less robust democracies, fewer safety guardrails, and a lesser ability to mandate security standards to prevent model exfiltration

People at PauseAI are probably less concerned about the above (or more concerned about model autonomy, catastrophic risks, and short timelines).

Although you may have felt that you did your "scouting" work and arrived at a position worth defending as a warrior, others' comparably thorough scouting work has led them to a different position. Their opposition to your warrior-like advocacy, then, may not come (as your post suggests) from a purist notion that we should preserve elite epistemics at the cost of impact, but from a fundamental disagreement about the desirability of the consequences of a pause (or other policies), or of advocacy for a pause.

If our shared goal is the clichéd securing-benefits-and-minimizing-risks, or even just minimizing risks, one should be open to thoughtful colleagues' input that one's actions may be counterproductive to that end-goal.

2. Fighting does not necessarily get one closer to winning.

Although the analogy of war is compelling and lends itself well to your post's argument, in politics fighting often does not get one closer to winning. Putting up a bad fight may be worse than putting up no fight at all. If the goal is winning (instead of just putting up a fight), then taking criticism to your fighting style seriously should be paramount.

I still concede that a lot of people dismiss PauseAI merely because they see it as cringe. But I don't think this is the core of most thoughtful people's criticism.

To be very clear, I'm not saying that PauseAI people are wrong, or that a pause will always be undesirable, or that they are using the wrong methods. I am answering to

(1) the feeling that this post dismissed criticism of PauseAI without engaging with object-level arguments, and the feeing that this post wrongly ascribed outside criticism to epistemic purism and a reluctance to "do the dirty work," and

(2) the idea that the scout-work is "done" already and an AI pause is currently desirable. (I'm not sure I'm right here at all, but I have reasons [above] to think that PauseAI shouldn't be so sure either.)

Sorry for not editing this better, I wanted to write it quickly. I welcome people's responses though I may not be able to answer to them!

Holly Elmore ⏸️ 🔸4mo-5

OscarD🔸8mo20

This analysis seems roughly right to me. Another piece of it I think is that being a 'soldier' or a 'bednet-equivalent' probably feels low status to many people (sometimes me included) because:

people might feel soldiering is generally easier than scouting, and they are more replaceable/less special
protesting feels more 'normal' and less 'EA' and people want to be EA-coded

To be clear I don't endorse this, I am just pointing out something I notice within myself/others. I think the second one is mostly just bad, and we should do things that are good regardless of whether they have 'EA vibes'. The first one I think is somewhat reasonable (e.g. I wouldn't want to pay someone to be a fulltime protest attendee to bring up the numbers) but I think soldiering can be quite challenging and laudable and part of a portfolio of types of actions one takes.

Holly Elmore ⏸️ 🔸8mo21

Yes, this matches what potential attendees report to me. They are also afraid of being “cringe” and don’t want to be associated with noob-friendly messaging, which I interpret as status-related.

This deeply saddens me because one of the things I most admired about early EA and found inspirational was the willingness to do unglamorous work. It’s often neglected so it can be very high leverage to do it!

MichaelDickens8mo16

I feel this way—I recently watched some footage of a PauseAI protest and it made me cringe, and I would hate participating in one. But also I think there are good rational arguments for doing protests, and I think AI pause protests are among the highest-EV interventions right now.

Paul_Lang8mo9

I'd like to add another bullet point
- personal fit

I think that protests play an important role in the political landscape, so I joined a few, but but walking through streets in large crowds and chanting made me feel uncomfortable. Maybe I'd get used to it if I tried more often.

ASuchy8mo8

Love this!

Soldiers win wars. The most accurate map made by the most virtuous scout is worth nothing without soldiers to do something with it.

My experience in animal protection has shown me the immense value of soldiers and FWIW I think some of the most resolute soldiers I know are also the scouts I most look up to. Campaigning is probably the most mentally challenging work I have ever done. I think part of that is constantly iterating through the OODA loop, which is cycling through scout and soldier mindsets.

Most animal activists I know in the EA world, were activists first and EA second. It would be interesting to see more EAs tapping into activist actions, which often are a relatively low lift. And I think embracing the soldier mindset is part of that happening.

Paul Z W3mo1

Setting aside the concrete example of Pause AI (haven't given it enough thought), I totally agree with the statement in the title.
Also, if I may: to some extent, you can accomplish things even when your soldiers aren't as smart, or as ideologically aligned with you, as your scouts; same thing holds for officers. The historical example that comes to mind is the army of the Soviet Union: for some years at least, an important fraction of the officers were former officers of the imperial army; they were called "voenspetsy", which means "military specialists".

From the Wikipedia page on the Red Army:

"In June 1918, Leon Trotsky abolished workers' control over the Red Army, replacing the election of officers with traditional army hierarchies and criminalizing dissent with the death penalty. Simultaneously, Trotsky carried out a mass recruitment of officers from the old Imperial Russian Army, who were employed as military advisors (voenspetsy).^[19]^[20] The Bolsheviks occasionally enforced the loyalty of such recruits by holding their families as hostages.^[21]^[^page needed^] As a result of this initiative, in 1918, 75% of the officers were former tsarists.^[22] By mid-August 1920 the Red Army's former tsarist personnel included 48,000 officers, 10,300 administrators, and 214,000 non-commissioned officers.^[23] When the civil war ended in 1922, ex-tsarists constituted 83% of the Red Army's divisional and corps commanders."

[anonymous]8mo1

I think we have all the info we need to contradict the fear of not being a scout in her metaphor. Scouts are important for success in battle because accurate information is important to draw up a good battle plan. But those battle plans are worthless without soldiers to fight the battle! “Everyone Should be a Mapmaker and Fear that Using the Map to Actually Do Something Could Make Them a Worse Mapmaker” would be a much less rousing title, but this is how many EAs and rationalists have chosen to interpret the book.

seems locally invalid.^[1]

argues from the meaning of terms in a metaphor
"Everyone Should be a Mapmaker and Fear that Using the Map to Actually Do Something Could Make Them a Worse Mapmaker" is not a description of the position you want to argue against, because you can do things with information other than optimizing what you say to persuade people.

^{^}
'locally invalid' means 'this is not a valid argument', separate from the truth of the premises or conclusion

Linch5mo*11

At the risk of being pedantic, I reread your comment several times^[1] and I still don't see why it's locally invalid. I can see why it's externally/globally invalid, but I don't think you actually speak to the local validity here?

^{^}
And the comment is pretty short so I don't think I'm missing something.

Holly Elmore ⏸️ 🔸8mo-14

I’m answering this obnoxiously posed criticism because I have something interesting to say in response. You could have said “Maybe just persuasion is off-limits to them and not other actions?”

I’ll answer this better criticism by saying, yes, there are other actions besides persuasion. I did not mention it here but this same crowd is afraid of most actions because of “nth order effects”. I think they suffer from serious perfectionism, catastrophizing, and action/inaction distinction and that’s motivating them to see dire problems with all actions.

Holly Elmore ⏸️ 🔸8mo-2

I've had a request to explain why this comment is obnoxious. It's obnoxious because of the way it's like grading my paper instead of saying anything important. I thought there was a nugget of a good question in there so I polished it up and answered that. If quilia really cares about where the scout mindset metaphor falls apart they could have probed that instead of just dinging me as they are the referee. This is the exact dynamic I railed about here.

I'm aware it's considered gauche to do anything but graciously accept and thank someone for even the most pedantic criticism. To thank people for reflexive gotchas is 1) a lie, in my case, and 2) training them to be lazy and shallow and negative in their interactions with me and my writing. I found it very annoying to write something thoughtful and have someone lazily mark it up in red pen without adding anything, thoughts seemingly having terminated after locating the nitpick. I'm not grateful for that, and it's not a contribution. It's making this place terrible and needs to be disincentivized.

dirk8mo8

I think if your arguments are locally invalid, that is something important about your post. High standards of accuracy and quality are something I value about Less Wrong and EA, and to me part of having high standards is trying to avoid even small mistakes.

Holly Elmore ⏸️ 🔸8mo-21

You're wrong. It sounds good naively to say this but it's destructive in practice. https://forum.effectivealtruism.org/posts/tuSQBGgnoxvsXwXJ3/criticism-is-sanctified-in-ea-but-like-any-intervention

Also the comment was phrased very unhelpfully for getting to the bottom of the error, if there was one. The attitude that the poster puts an idea out there and the commenters just shoot it down from afar without even needing to be comprehensible is so counterproductive. It's treating this like a test instead of a conversation. If it is a test, why is only the poster being tested? The commenters do routinely do an awful job, leaving confusing, discouraging, and rude replies. Why is it only the poster who has to worry about the truth or usefulness of what they say if we're trying to find the truth? I thought the comment was bad-- that's at least as important to share as what they thought was invalid about my post.

dirk8mo1

dirk8mo4

(Also, I understand the comment was not phrased helpfully to you, but for my part I felt that it communicated the errors clearly enough that I could understand them easily, and appreciate having the false dichotomy especially pointed out without having to discover it myself).

dirk8mo6

(Also also, it isn't only the poster who has to worry about the truth of what they say? It's everyone? Comments also receive criticism all the time. I don't think this poster/commenter divide cuts reality at the joints.)

Brad West🔸8mo6

I agree with your post overall and think that EA can be very pedantic, professorial, and overly averse to persuasion. I am very glad that you wrote this post and believe that EAs should credit more the importance of persuasion (and probably be more susceptible to positive persuasion as against criticism).

However, the title of your post suggested that the scout mindset is valuable only as a servant of persuasion. I think that it is important to note that scout mindset has other valuable applications.

Holly Elmore ⏸️ 🔸8mo4

Persuasion is my intervention at PauseAI US-- never meant to imply it was the only intervention. Bednets, mentioned in the post, are soldiers that are not persuasion.

For a map (generated by a scout) to have value, you have to be able to do something with it ("doing" here being analogized to soldiers). It doesn't have to be persuasion, but it also can't be pure scoutly knowledge-seeking.

Holly Elmore ⏸️ 🔸8mo0

You're not the worst, quilia, and my frustration goes so much deeper than your comment. Id on't want to put you on blast. You're just an example at a time when I have had it with the perverse norms we've accepted in this community.

[anonymous]8mo14

i'm not bothered by your comments.

your first reply seemed to be about how i worded the point (you wrote "obnoxiously posed", and reworded it) rather than pedanticness/irrelevance. i mentally replaced "this is obnoxious" with "this makes me feel annoyed", which i think is okay to say. i also considered letting you know i'm autistic, which makes me word things differently or more literally^[1] or in ways that can seem to have unintended emotional content. (i wonder if that's what made it feel like "marking it up in red pen")

onto object-level: what i wrote actually seemed substantive to me, i.e. it really did seem to me that the quote in point 2 was strongly misrepresenting the position the post intended to argue against, so i wouldn't consider it pedantic. (it could separately be false, of course)

If quila really cares about where the scout mindset metaphor falls apart they could have probed that instead of just dinging me as they are the referee

it did not occur to me that you might endorse the scout/soldier metaphor, and just be using the existence of scout/soldier in 'scout/soldier mindset' to bring it up; so yes, if that's actually the case, it would have been better to notice that and then either not comment on it or probe it as you say. using a metaphor is not invalid.

here's how i perceived it at the time: 'scout mindset' and 'soldier mindset' have particular meanings, so whether traditional soldiers are necessary for traditional scouts is a different topic. writing about them instead seemed 'opportunistic' in some sense, as if the text was using the terminological overlap to sneak through an argument about one as about the other.

i wonder if this thread could have been mitigated if i were more clear about that in my initial comment. if anyone has advice it is welcome.

^{^}
maybe 'more structured like the thought is structured internally'

Holly Elmore ⏸️ 🔸8mo5

here's how i perceived it at the time: 'scout mindset' and 'soldier mindset' have particular meanings, so whether traditional soldiers are necessary for traditional scouts is a different topic. writing about them instead seemed 'opportunistic' in some sense, as if the text was using the terminological overlap to sneak through an argument about one as about the other.

I mean, no more than when Julia Galef wrote it? Have you read the book? There's a long discussion of this metaphor and my analysis would totally fit there. Julia says there are important times and places for soldier mindset, but everyone seems to have forgotten this and just remembers scout mindset as "the good one".