This is a special post for quick takes by Roman Ross. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.
I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:
- A lot of the most viral pro-AIS people (especially the ones that are relatively new to the field) seem to not be reasoning especially carefully
- The field’s hiring process is quite insular
- Not enough time is spent engaging with a lot of good faith disagreements (imo)
- Fieldbuilding strategy doesn’t optimize for careful reasoning amongst those it recruits
- People receive lots of social/material benefits for continuing to believe that AI risk is the most important thing and work on it
- Even under some of the shortest timelines and most speculative capabilities that superintelligence might have, I really don’t extinction within the next year should be assigned any notable probability mass, due to the need for data centers to be replenished
I also see a lot of disagreements that I think are poorly reasoned. For example:
- Poor application of/thinking about EV when conceptualizing risks
- Psychoanalysis that doesn’t explain the safety community’s behaviors well
- Comments significantly overstating the case for something
- Arguments without the slightest amount of research into what the other side actually believes
Tbh though, most of these critiques apply to both sides.
I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.
My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.
Quick math: In terms of total expected suffering averted, inviting a group of friends to a barbecue where the main meat is beef is probably more impactful than eating a vegan meal by yourself.
According to Faunalytics' Animal Product Impact Scales, a serving of chicken requires approximately 10 times as many days of animal life to produce as a serving of beef (not adjusted for quality of life or moral weights). Additionally, most seafood is much, much worse than this. If this meal prevents your friends from going and eating those other foods, tricking them into eating beef will likely do more good than eating a vegan meal by yourself.
Still, this may not be as impactful as saving your money and donating it directly to impactful charities. Saving a single dollar on food and donating it to the SWP will probably do more to reduce suffering than hosting a barbecue.
Also, you might expose yourself to various forms of value drift by hosting a barbecue, but I could see this happening in a net positive (hosting more barbecues in the future) or a net negative (giving up on veganism entirely/permanently failing to convince any of your friends to give up meat) direction.
At my particular place, I think that improving AI safety community epistemics (writing thorough and careful arguments against people both in the community and outside of it who make silly mistakes) is, by most reasonable estimates, probably less impactful than helping to run MAIA, which is what I currently spend most of my time doing.
I don’t think I’m the only person in this situation, which is quite scary to me, as I think there are a lot of things are epistemically iffy in the AI safety community. For example:
I also see a lot of disagreements that I think are poorly reasoned. For example:
Tbh though, most of these critiques apply to both sides.
I think this primarily arises because the people working in AI safety who are the most careful reasoners also have the highest opportunity costs, because they tend to be good at other things. This means it’s less valuable for them to write out their thoughts online carefully.
My main point, though, is that it’s a little scary to see so much going wrong with the discourse and not being able to do anything about it (or even be worth trying to do anything about it). It feels like there’s not much worth trying except praying for Forethought to do a good job and reach everyone.