Critiques of non-existent AI safety labs: Yours

I am confused about what your claims are, exactly (or what you’re trying to say). 

One interpretation, which makes sense to me, is the following

“Starting an AI safety lab is really hard and we should have a lot of appreciation for people who are doing it. We should also cut them some more slack when they make mistakes because it is really hard and some of the things they are trying to do have never been done before.” (This isn’t a direct quote)

I really like and appreciate this point. Speaking for me personally, I too often fall into the trap of criticising someone for doing something not perfectly and not 1. Appreciating that they have tried at all and that it was potentially really hard, and 2. Criticising all the people who didn’t do anything and chose the safe route. There is a good post about this: Invisible impact loss (and why we can be too error-averse).

In addition, I think it could be a valid point to say that we should be more understanding if e.g. the research agendas of AIS labs are/were off in the past as this is a problem that no one really knows how to solve and that is just very hard. I don’t really feel qualified to comment on that.  

Your post could also be claiming something else:

“We should not criticise / should have a very high bar for criticizing AI safety labs and their founders (especially not if you yourself have not started an AIS lab). They are doing something that no one else has done before, and when they make mistakes, that is way understandable because they don’t have anyone to learn from.” (This isn’t a direct quote)

For instance, you seem to claim that the reference class of people who can advise people working on AI safety is some group whose size is the number of AI safety labs multiplied by 3. (This is what I understand your point to be if I look at the passage that starts with “Some new organizations are very similar to existing organizations. The founders of the new org can go look at all the previous closeby examples, learn from them, copy their playbook and avoid their mistakes.” and ends in “That is the roughly the number of people who are not the subject of this post.”)

If this is what you want to say, I think the message is wrong in important ways. In brief: 

1. I agree that when people work on hard and important things, we should appreciate them, but I disagree that we should avoid criticism of work like this. Criticism is important precisely when the work matters. Criticism is important when the problems are strange and people are probably making mistakes. 
2. The strong version of “they’re doing something that no one else has done before … they don’t have anyone to learn from” seems to take a very narrow reference class for a broad set of ways to learn from people. You can learn from people who aren’t doing the exact thing that you’re doing.

1. A claim like: “We should not criticise / should have a very high bar for criticizing AI safety labs / their founders (especially not if you yourself have not started an AIS lab).”

As stated above, I think it is important to appreciate people for trying at all, and it’s useful to notice that work not getting done is a loss. That being said, criticism is still useful. People are making mistakes that others can notice. Some organizations are less promising than others, and it’s useful to make those distinctions so that we know which to work in or donate to. 

In a healthy EA/LT/AIS community, I want people to criticise other organisations, even if what they are doing is very hard and has never been done before. E.g. you could make the case that what OP, GiveWell, and ACE are doing has never been done before (although it is slightly unclear to me what exactly “doing something that has never been done before” means), and I don’t think anyone would say that those organisations should be beyond criticism. 

This ties nicely into the second point I think is wrong: 

2. A claim like: “they’re doing something that no one else has done before … they don’t have anyone to learn from”

A quote from your post:

The founders of the new org can go look at all the previous closeby examples, learn from them, copy their playbook and avoid their mistakes.  If your org is shaped like a Y-combinator company, you can spend dozens of hours absorbing high-quality, expert-crafted content which has been tested and tweaked and improved over hundreds of companies and more than a decade. You can do a 15 minute interview to go work next to a bunch of the best people who are also building your type of org, and learn by looking over their shoulder and troubleshooting together. You get to talk to a bunch of people who have actually succeeded  building an org-like-yours.  … How does this look for AI safety? … Apply these updates to our starting reference class success rate of ONE. IN. TWENTY. Now count the AI safety labs. Multiply by ~3.  

A point I think you’re making:  

“They are doing something that no one else has done before [build a successful AI safety lab], and therefore, if they make mistakes, that is way understandable because they don’t have anyone to learn from.”

It is true that the closer your organisation is to an already existing org/cluster of orgs, the more you will be able to copy. But just because you’re working on something new that no one has worked on (or your work is different in other important aspects), it doesn’t mean that you cannot learn from other organisations, their successes and failures. For things like having a healthy work culture, talent retention, and good governance structures, there are examples in the world that even AIS labs can learn from. 

I don’t understand the research side of things well enough to comment on whether/how much AIS labs could learn from e.g. academic research or for-profit research labs working on problems different from AIS. 

Seems like academic research groups would be a better reference class than YC companies for most alignment labs.

If they're trying to build an org that scales a lot, and is funded by selling products, YC companies is a good reference class, but if they're an org of researchers working somewhat independently or collaborating on hard technical problems, funded by grants, that sounds much more similar to an academic research group.

Unsure how to define success for an academic research group, any ideas? They seem to more often be exploratory and less goal-oriented.

As someone who did recently set up an AI safety lab, success rates have certainly been on my mind. It's certainly challenging, but I think the reference class we're in might be better than it seems at first.

I think a big part of what makes succeeding as a for-profit tech start-up challenging is that so many other talented individuals are chasing the same, good ideas. For every Amazon there are 1000s of failed e-commerce start-ups. Clearly, Amazon did something much better than the competition. But what if Amazon didn't exist? What if there was a company that was a little more expensive, and had longer shipping times? I'd wager that company would still be highly successful.

Far fewer people are working on AI safety. That's a bad thing, but it does at least mean that there's more low-hanging fruit to be tapped. I agree with [Adam Binks](https://forum.effectivealtruism.org/posts/PJLx7CwB4mtaDgmFc/critiques-of-non-existent-ai-safety-labs-yours?commentId=eLarcd8no5iKqFaNQ) that academic labs might be a better reference class. But even there, AI safety has had far less attention paid to it than e.g. developing treatments for cancer or unifying quantum mechanics and general relativity. 

So overall it's far from clear to me that it's harder to make progress on AI safety than solve outstanding challenge problems in academia, or in trying to make a $1 bn+ company.

Thanks for writing this. It felt a bit like an AI safety version of Roosevelt's Man in the arena

It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.

I'm honestly not sure whether this is an argument in support of AI labs or against?

~65% of charity entrepreneurship charities are at least moderately successful, with half of those being very successful. They're probably a closer reference class, being donor-funded organisations run by EAs for impact.

One way in which AI safety labs are different than the reference class of Y-combinator startups is in their impact. Conditioned on the median Forum user's assessment of X-risk from AI, the leader of a major AI safety lab probably has more impact that the median U.S. senator, Fortune 500 CEO, or chief executive of smaller regional or even national governments, etc. Those jobs are hard in their own ways, but we expect and even encourage an extremely high amount of criticism. 

I am not suggesting that is the proper reference class for leaders of AI labs that have raised at least $10MM . . . and I don't think it is. But I think the proper scope of criticism is significantly higher than for (e.g.) the median CEO whose company went through Y Combinator.^[1] If a startup CEO messes up and their company explodes, the pain is generally going to be concentrated in the company's investors, lenders, and employees . . . a small number of people, each of whom who consented to bearing that risk to a significant extent. If I'm not one of those people, my standing to complain about the startup CEO's mistakes is significantly constrained.

In contrast, if an AI safety lab goes off the rails and becomes net-negative, that affects us all (and futute generations). Even if the lab is merely ineffective, its existence would have drained fairly scarce resources (potential alignment researchers and EA funding) from others in the field. 

I definitively agree that people need to be sensitive to how hard running an AI safety lab is, but also want to affirm that the idea of criticism is legitimate.

1. ^{^}

   To be clear, I don't think Anneal's post suggests that this is the reference class for deciding how much criticism of AI lab leaders is warranted. However, since I didn't see a clear reference class, I thought it was worthwhile to discuss this one.

Fail early, fail often. Many little dooms are good. One big doom is not so good.