Sure, "hiring managers being bad at marketing is the bottleneck, not funding" is at least partially true. It still implies that if you happen to stumble across a poorly advertised position, you shouldn't expect the acceptance rate to be low!

My version of Matt's critique that you quoted is something like:

Imagine you're running a mining company, and you want to start mining Venus. You could either embark on a massive terraforming project to make Venus habitable by biological humans who can work in your mining colony, or you could just build a bunch of robots who can naturally withstand Venus' climate, think faster than a human, make better decisions than a human, etc. etc. What do you do?

Obviously you are going to choose to send the robots, and the robots aren't going to want to eat meat, so you don't need to worry about factory farming on Venus.

I don't think this argument is bulletproof. For example, ports in the U.S. are required to pay human dock workers to sit around and do nothing after their jobs had been automated. I could imagine some sort of analogous regulatory capture in the future which would require mining companies to send humans to other planets even when robots would be more efficient. Preventing this kind of lock-in is one of the few interventions targeting a post-singularity world that I feel positive about.

this claim feels at odds with what I understood your perspective to be from the shallow review you did a while ago

Yeah, I think I did a bad job explaining my views there. Could you say more about what you thought I believed? I should maybe update the post.

However, it seems like this would fall into the bucket of "not actionable unless you work directly on AI," so it seems like it might be practically useful to act as if this wasn't going to be the case? 

Good question and I claim the answer is "no" because you can work on AI! E.g. The Midas Project (founded by a former THL campaigner) is bringing corporate campaign tactics to AI safety. See also e.g. AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists.

I think there's a decent chance that AI won't actually be very transformative (or at least won't be transformative soon) and therefore it's reasonable to bet your career plans on that assumption. But, to the extent you think AI is actually going to be a big deal, I would suggest considering working on it! 

Fair. "In good timelines some humans continue to exist despite not really being a major force analogous to how the Amish exist today but aren't really a major force" is probably the modal view amongst people I talk to.

I shouldn't have used the term "TAI" here; I've clarified. Thanks!

What I said was unclear; I probably should have just quoted Holden Karnofsky:

I think there is a good chance that:

1. During the century we're in right now, we will develop technologies that cause us to transition to a state in which humans as we know them are no longer the main force in world events. This is our last chance to shape how that transition happens.
2. Whatever the main force in world events is (perhaps digital people, misaligned AI, or something else) will create highly stable civilizations that populate our entire galaxy for billions of years to come. The transition taking place this century could shape all of that.

I think it's very unclear whether this would be a good or bad thing. 

There are some people (e.g. Evitable) who are trying to stop this transition, but I think more AI safety people would self-describe their work as "shap[ing] how that transition happens."

What Matt calls efficient many would call “weird” and hence undesirable. I’m not sure we have a reason to think AI will mitigate this rather than amplify it.

In case you haven't seen it, one argument for why AI might amplify (or at least lock-in) traditions was given by Buck here: Christian homeschoolers in the year 3000. 

Thanks for writing this! I appreciate you flagging:

Here, I’m conditioning on humans as we know them continuing to exist. I take no position on how likely this is to happen, but it seems to me like this is the outcome that EAs are pushing for when trying to reduce existential risk from AI. 

I am not aware of anyone who works on AI safety who would say that this is what they are pushing for, with the exception of people who are pushing for a complete pause in AI development. 

The rest of us are generally resigned to biological humans disappearing once we have transformative AI [edit: this was unclear, see update], even under the most optimistic scenarios. I expect that this is a major way in which the AI safety people I know (including Matt) find arguments like this and Bentham's Bulldogs' uncompelling.

Interesting post, thanks! On this:

It’s possible that the lack of evidence has been accounted for in other ways. Perhaps someone who initially guesses a 20% chance of extinction is subtly dropping that down to 5% on the grounds of epistemic modesty. But it’s unlikely they are doing so in the exact right way to counteract the effect of the optimizer’s curse.

My understanding is that worldview diversification partially addresses things like this (see this old critique from Holden Karnofsky, which makes a similar point to yours and I think is intellectually upstream of CG's later thinking), in addition to accounting for e.g. normative uncertainty.

I'm not exactly sure how worldview diversification works (maybe someone from CG can comment) but I share your skepticism that it's being done in exactly the right way to counteract these effects.