Comments
I ran a successful protest at OpenAI yesterday. Before the night was over, Mikhail Samin, who attended the protest, sent me a document to review that accused me of what sounds like a bait and switch and deceptive practices because I made an error in my original press release (which got copied as a description on other materials) and apparently didn't address it to his satisfaction because I didn't change the theme of the event more radically or cancel it.
My error: I made the stupidest kind of mistake when writing the press release weeks before the event. The event was planned as a generic OpenAI protest a ~month and half ahead of time. Then the story about the mysteriously revised usage policy and subsequent Pentagon contract arose and we decided to make rolling it back the "small ask" of this protest, which is usually a news peg and goes in media outreach materials like the press release. (The "big ask" is always "Pause AI" and that's all that most onlookers will ever know about the messaging.) I quoted the OpenAI charter early on when drafting it, and then, in a kind of word mistake that is unfortunately common for me, started using the word “charter” for both the actual charter document and the usage policy document. It was unfortunately a semantic mistake, so proofreaders didn’t catch it. I also subsequently did this verbally in several places. I even kind of convinced myself from hearing my own mistaken language that OpenAI had violated a much more serious boundary– their actual guiding document– than they had. Making this kind of mistake is unfortunately a characteristic sort of error for me and making it in this kind of situation is one of my worst fears.
How I handled it: I was horrified when I discovered the mistake because it conveyed a significantly different meaning than the true story, and, were it intentional, could have slandered OpenAI. I spent hours trying to track down every place I had said it and people who may have repeated it so it could be corrected. Where I corrected "charter" to "usage policy", I left an asterisk explaining that an earlier version of the document had said "charter" in error. I told the protesters in the protester group chat (some protesters just see flyers or public events, so I couldn't reach them all) right away about my mistake, which is when Mikhail heard about it, and explained changing the usage policy is a lot less bad than changing the charter, but the protest was still on, as it had been before the military story arose as the “small ask”. I have a lot of volunteer help, but I'm still a one woman show on stuff like media communications. I resolved to have my first employee go over all communications, not just volunteer proofreaders, so that someone super familiar with what we are doing can catch brainfart content errors that my brain was somehow blind to.
So, Mikhail seems to think I should have done more or not kept the no-military small ask. He's going to publish something that really hurt my feelings because it reads as an accusation of lying or manipulation and calls for EA community level "mechanisms" to make sure that "unilateral action" (i.e. protests where I had to correct the description) can't be taken because I am "a high status EA".
This is silly. I made a very unfortunate mistake that I feel terrible about and tried really hard to fix. That's all. To be clear, PauseAI US is not an EA org and it wouldn't submit to EA community mechanisms because that would not be appropriate. Our programming does not need EA approval and is not seeking it. I failed in my own personal standards of accuracy, the standards I will hold PauseAI US to, and I will not be offering any kind of EA guarantees on what PauseAI does. Just because I am an EA doesn't sign my organization up for EA norms or make me especially accountable to Mikhail. I'm particularly done with Mikhail, in fact, after spending hours assisting him with his projects and trying to show him a good time when he visits Berkeley and answering to his endless nitpicks on Messenger and getting what feels like zero charity in return. During what should have been a celebration for a successful event, I was sobbing in the bathroom at the accusation that I (at least somewhat deliberately-- not sure how exactly he would characterize his claim) misled the press and the EA community.
I made suggestions on his document and told him he can post it, so you may see it soon. I think it's inflammatory and it will create pointless drama and he should know better, but I also know he would make a big deal and say I was suppressing the truth if I told him not to post it. I think coming at me with accusations of bad faith and loaded words like "deceptive" and "misleading" is shitty and I really do not want to be part of my own special struggle session on this forum. It wounds me because I feel the weight of what I'm trying to do, and my mistake scared me that I could stumble and cause harm. It's a lot of stress to carry this mission forward against intense disagreement, and carrying it forward in the face of personal failure is even tougher for me. But I also know I handled my error honorably and did my best.
There's a serious courage problem in Effective Altruism. I am so profoundly disappointed in this community. It's not for having a different theory of change-- it's for the fear I see in people's eyes when considering going against AI companies or losing "legitimacy" by not being associated with them. The squeamishness I see when considering talking about AI danger in a way people can understand, and the fear of losing face within the inner circle. A lot of you value being part of a tech elite more than you do what happens to the world. Full stop. And it does bother me that you have this mutual pact to think of yourselves as good for your corrupt relationships with the industry that's most likely to get us all killed.
Big +1 to the gist of what you are saying here re: courage. It's something I've struggled with a lot myself, especially while I was at OpenAI. The companies have successfully made it cool to race forward and uncool to say e.g. that there's a good chance doing so will kill everyone. And peer pressure, financial incentives, status incentives, etc. really do have strong effects, including on people who think they are unaffected.
As an EA who is profoundly disappointed at the level of Bay Area EA collaboration with the labs, I must agree. As a technical researcher, I work with them when it improves safety, but I'm a big proponent of Dan Hendrycks when he says that regulation shouldn't be pleasing for companies since they are the antagonists to any regulation in this field.
And every time I meet policy professionals in the field, they are infinitely scared to ever say extinction risk, pause, or anything like this (happy to hear retorts to this from other insiders), which is incredibly critical. In that sense, it becomes a power-seeking move, but once you are in power, if you've never had an opinion on your way to the top, no one will listen to you.
Very relevant to mention that there are definitely some EAs that are honest and pragmatic, but it's too rare.
Looking forward to be proven wrong but this seems like a profoundly misguided strategy.
This topic is painful for me because I opened my heart to EA as fellow travelers who could share my moral burden. And I feel betrayed by my former community when they let themselves be charmed and intimidated by the AI industry. And then turn all their rhetorical tricks against me not to listen to me ("You're not being nice and/or smart enough, so it's actually you who has the problem.").
I realize now the rationalists were always cowards who wanted the Singularity and to imagine colonizing Galaxies more than to protect people. But I can't quite accept EA's failure here. I'm like a spurned lover who can't believe the love wasn't real. I thought, at its core, EA really got it about protecting beings.
Have you seen any polls on this? I would guess that the majority of EAs and rationalists would not support a pause (because they think it would increase overall risk, at least at this point, e.g. Carl Shulman), but they would also generally not be supportive of what the labs are doing (racing, opposing regulations, etc).
That’s not what Carl Shulman said, and the fact that people want to take it that way is telling. He messaged me recently to clarify that he meant unilateral pauses would be bad, something I still kind of disagree with but which isn’t something PauseAI advocates, and he said it way at the beginning of Pause talk. EAs just don’t want to arrest the tech momentum bc they see themselves as technocracy elite, not as humble grassroots organizers. They are disappointed at the chance we have to rally the public and want to find some way they don’t have to contribute to it.
People who identify as EAs in other countries might not be supportive of the AI companies, but they aren’t the ones on the ground in the Bay Area and DC that are letting me down so much. They aren’t the ones working for Anthropic or sitting on their fake board, listening to Dario’s claptrap that justifies what they want to do and believe anyway. They aren’t the ones denying my grant applications bc protests aren’t to their tastes. They aren’t the ones terrified of not having the vision to go for the Singularity, of being seen as “Luddites” for opposing a dangerous and recklessly pursued technology. Frankly they aren’t the influential ones.
Though Carl said that an unilateral pause would be riskier, I'm pretty sure he is not supporting a universal pause now. He said "To the extent you have a willingness to do a pause, it’s going to be much more impactful later on. And even worse, it’s possible that a pause, especially a voluntary pause, then is disproportionately giving up the opportunity to do pauses at that later stage when things are more important....Now, I might have a different view if we were talking about a binding international agreement that all the great powers were behind. That seems much more suitable. And I’m enthusiastic about measures like the recent US executive order, which requires reporting of information about the training of new powerful models to the government, and provides the opportunity to see what’s happening and then intervene with regulation as evidence of more imminent dangers appear. Those seem like things that are not giving up the pace of AI progress in a significant way, or compromising the ability to do things later, including a later pause...Why didn’t I sign the pause AI letter for a six-month pause around now?
But in terms of expending political capital or what asks would I have of policymakers, indeed, this is going to be quite far down the list, because its political costs and downsides are relatively large for the amount of benefit — or harm. At the object level, when I think it’s probably bad on the merits, it doesn’t arise. But if it were beneficial, I think that the benefit would be smaller than other moves that are possible — like intense work on alignment, like getting the ability of governments to supervise and at least limit disastrous corner-cutting in a race between private companies: that’s something that is much more clearly in the interest of governments that want to be able to steer where this thing is going. And yeah, the space of overlap of things that help to avoid risks of things like AI coups, AI misinformation, or use in bioterrorism, there are just any number of things that we are not currently doing that are helpful on multiple perspectives — and that are, I think, more helpful to pursue at the margin than an early pause."
So he says he might be supportive of a universal pause, but it sounds like he would rather have it later than now.
I see where you are coming from, but it think it would be more accurate to say that you are disappointed in (or potentially even betrayed by[1]) the minority of EAs who are accelerationists, rather than characterizing it as being betrayed by the community as a whole (which is not accelerationist).
Though I think this is too harsh as early thinking in AI Safety included Bostrom's differential technological development and MIRI's seed (safe) AI, the former of which is similar to people trying to shape Anthropic's work and the latter of which could be characterized as accelerationist.
To get a pause at any time you have to start asking now. It’s totally academic to ask about when exactly to pause and it’s not robust to try to wait until the last possible minute. Anyone taking pause advocacy seriously realizes this pretty quickly.
But honestly all I hear are excuses. You wouldn’t want to help me if Carl said it was the right thing to do or you’d have already realized what I said yourself. You wouldn’t be waiting for Carl’s permission or anyone else’s. What you’re looking for is permission to stay on this corrupt be-the-problem strategy and it shows.
As Carl says, society may only get one shot at a pause. So if we got it now, and not when we have a 10x speed up in AI development because of AI, I think that would be worse. It could certainly make sense now to build the field and to draft legislation. But it's also possible to advocate for pausing when some threshold or trigger is hit, and not now. It's also possible that advocating for an early pause burns bridges with people who might have supported a pause later.
I personally still have significant uncertainty as to the best course of action, and I understand you are under a lot of stress, but I don't think this characterization is an effective way of shifting people towards your point of view.
This is so out of touch with the realities of opinion change. It sounds smart and it lets EAs and rationalists keep doing what they’re doing, which is why people repeat it. This claim that we would only get one shot at a pause is asinine— pause would become more popular as an option the more people were familiar with it. It’s only the AI industry and EA that do not like the idea of pausing and pretend like they’re gonna withdraw support that we actually never had if we do something they don’t like.
The main thing we can do as a movement is gain popular support by talking about the message. There is no reliable way to “time” asks. None of that makes any sense. Honestly, most people who give this argument are industry apologists who just want you feel out of your league if you do anything against their interests. Hardware overhang was the same shit.
Ah, the EA “fuck you”.
I see, so other people (😉) might not agree with me bc of how I say it, and rather than it being their responsibility to seek truth, that’s my fault for not behaving the way you— oops, I mean “they”— like. It’s just ineffective, so nobody listening has any responsibilities and should probably go back to working at an AI company.
Hey Holly, it sounds like you’re frustrated by how people in EA are engaging with the idea of a pause. I’m sure that’s really hard, and I’m sure I don’t know even a fraction of what you’ve gone through. I know you’re doing this advocacy work because you care a lot, and I really appreciate that. You know that I personally support your work.
However, I’m worried that this thread is becoming unproductive, and risks making the Forum feel like less of a safe space[1].
In particular, my concern is that you are criticizing @Denkenberger🔸 directly in a way that appears to come from nowhere. @Denkenberger🔸 doesn’t say anything about their personal views on a pause before you respond with:
In my opinion, this sort of accusation without evidence erodes the Forum’s ability to be a safe community space for important discussions. If it's the case that I'm missing some context and you have personal beef with @Denkenberger🔸, then I don’t think the Forum is the appropriate place to hash that out.
To be clear, I think the following are broadly fine and often good (assuming adherence to our Forum norms):
So I’d like to suggest that you take a step back from this thread. If you find yourself getting frustrated at others on the Forum elsewhere, I’d suggest taking a break from those as well.
You’ve written some of the best posts on this Forum, and the Forum Team greatly values your contributions. At the same time, I also think it’s important that the Forum continues to be a productive discussion space.
A relevant quote from our “Moderation principles” post:
I think you hit the nail on the head— this forum is not a safe space for me. Like you said, I’m an all-time top poster, and yet I get snobby discouragement on everything I write since I started working on Pause, with the general theme that advocacy is not smart enough for EAs (and a secondary theme of wanting to work for AI companies).
This is a serious problem given what the EA Forum was supposed to be. It’s not a problem with following your rules for polite posts, but it’s against something more important— the purpose of the Forum and of the EA community.
But, I’ve clearly reached the end of my rope, and since I’d like to keep my account and be able to post new stuff here, I’ll just stop commenting.
I see that you are getting some downvotes, and while I disagree with this comment in particular I'm glad that you are there in the arena strongly pushing your vision of the good :)
I agree with this problem, that there are many folk who lack courage, particularly within AI governance, but I don't really see these kinds of comments as being that likely to change things.
There are times when it can make sense to shout at people, but it normally works better when there's a clear and realistic path that's been sketched out[1]. And when this isn't the case, it tends to simply create strife for very little gain.
Someone really needs to write an article along the lines of "The case for being bold".
It should hurt to work with AI companies. You should be looking for excuses not to even if you think there’s an important reason to maintain the relationship. But instead it’s always the other way around because working with corrupt tech elites who “joke” about being evil is ego-syntonic to you. That’s an extremely serious problem.