The hygiene hypothesis (especially the autoimmune disease variant, brief 2-paragraph summary here if you Ctrl+F "Before we go") could be another example.
On a somewhat related note, Section V of this SlateStarCodex post goes through some similar examples where humans departing from long-lived tradition has negative effects that don't become visible for a long time.
I'm curious about what's the original source of the funding you're giving out here. According to this Nonlinear received $250k from Future Fund and $600k from Survival and Flourishing Fund. Is the funding being distributed here coming solely from the SFF grant? Does Nonlinear have other funding sources besides Future Fund and SFF?
(I didn't do any deeper dive than looking at Nonlinear's website, where I couldn't find anything about funding sources.)
Thanks for writing this--even though I've been familiar with AI x-risk for a while, it didn't really hit me on an emotional level that dying from misaligned AI would happen to me too, and not just "humanity" in the abstract. This post changed that.
Might eventually be useful to have one of these that accounts for biorisk too, although biorisk "timelines" aren't as straightforward as trying to estimate the date that humanity builds the first AGI.
If you are a community builder (especially one with a lot of social status), be loudly transparent with what you are building your corner of the movement into and what tradeoffs you are/aren’t willing to make.
I like this suggestion--what do you imagine this transparency looks like? Do you think, e.g., EA groups should have pages outlining their community-building philosophies on their websites? Should university groups should write public Forum posts about their plans and reasoning before every semester/quarter or academic year? Would you advocate for more community-building roundtables at EAGs? (These are just a few possible example modalities of transparency that just came to my head, very interested in hearing more.)
Hey Jordan! Great to see another USC person here. The best writing advice I've gotten (that I have yet to implement) is to identify a theory of change for each potential piece--something to keep in mind!
6 sounds interesting, if you can make a strong case for it. Aligning humans isn't an easy task (as most parents, employers, governments, and activists know very well), so I'm curious to hear if you have tractable proposals.
7 sounds important given that a decent number of EAs are vegan, and I'm quite surprised I haven't heard of this before. 15 IQ points is a whole standard deviation, so I'd love to see the evidence for that.
8 might be interesting. I suspect most people are already aware of groupthink, but it could be good to be aware of other relevant phenomena that might not be as widely-known (if there are any).
From what I can tell, 11 proposes a somewhat major reconsideration of how we should approach improving the long-term future. If you have a good argument, I'm always in favor of more people challenging the EA community's current approach. I'm interested in 21 for the same reason.
(In my experience, the answer to 19 is no, probably because there isn't a clear, easy-to-calculate metric to use for longtermist projects in the way that GiveWell uses cost-effectiveness estimates.)
Out of all of these, I think you could whip up a draft post for 7 pretty quickly, and I'd be interested to read it!
Thanks Linch! This list is really helpful. One clarifying question on this point:
Relatedly, what does the learning/exploration value of this project look like?
To the researcher/entrepreneur?
To the institution? (if they're working in an EA-institutional context)
To the EA or longtermist ecosystem as a whole?
For 1) and 2), I assume you're referring to the skills gained by the person/institution completing the project, which they could then apply to future projects.
For 3), are you referring to the possibility of "ruling out intervention X as a feasible way to tackle x-risks"? That's what I'm assuming, but I'm just asking to make sure I understand properly.
This thinking has come up in a few separate intro fellowship cohorts I’ve facilitated. Usually, somebody tries to flesh it out by asking whether it’s “more effective” to save one doctor (who could then be expected to save five more lives) or two mechanics (who wouldn’t save any other lives) in trolley-problem scenarios. This discussion often gets muddled, and many people have the impression that “EAs” would think it’s better to save the doctor, even though I doubt that’s a consensus opinion among EAs. I’ve found this to be a surprisingly large snag point that isn’t discussed much in community-building circles.
I think it would be worth it to clarify the difference between intrinsic and instrumental value in career advice/intro fellowships/other first interactions with the EA community, because there are some people who might agree with other EA ideas but find that this argument undermines our basic principles (as well as the claim that you don’t need to be utilitarian to be an EA). Maybe we could extend current messaging about ideological diversity within EA.
That said, I read Objection 4 differently. Many people (especially in cultures that glorify work) tie their sense of self-worth to their jobs. I don’t know how universal this is, but at least in my middle-class American upbringing, there was a strong sense that your career choice and achievement is a large part of your value as a person.
As a result, some people feel personally judged when their intended careers aren’t branded as “effective”. If you equate your career value with your personal value, you won’t feel very good if someone tells you that your career isn’t very valuable, and so you’ll resist that judgment.
I don’t think that this feeling precludes people from being EAs. It takes time to separate yourself from your current or intended career, and Objection 4 strikes me as a knee-jerk defensive reaction. Students planning to work in shipping logistics won’t immediately like the idea that the job they’ve been working hard to prepare for is “ineffective,” but they might come around to it after some deeper reflection.
I could be misreading Objection 4, though. It could also mean something like “shipping logistics is valuable because the world would grind to a halt if nobody worked in shipping logistics,” but then that’s just a variant of Objection 5.
I’m very curious to know more about the sense in which these students gave Objection 4.
I recently learned about Training for Good, a Charity Entrepreneurship-incubated project, which seems to address some of these problems. They might be worth checking out.
I think this is a great exercise to think about, especially in light of somewhat-recent discussion on how competitive jobs at EA orgs are. There seems to be plenty of room for more people working on EA projects, and I agree that it’s probably good to fill that opportunity. Some loose thoughts:
There seem to be two basic ways of getting skilled people working on EA cause areas: 1. Selectively recruiting people who already have skills. 2. Recruiting promising people who might not yet have needed skills and train them.
Individual organizations can choose both options, depending on their level of resources. But if most organizations choose option 1, the EA community might be underutilizing its potential pool of human resources. So we might want the community in general to use option 2, so that everyone who wants to be involved with EA can have a role—even if individual EA organizations still choose option 1. For this to happen, the EA community would probably need a program whereby motivated people can choose a skillset to learn, are taught that skillset, and are matched with a job at the end of the process.
Currently, motivated people who don’t yet possess skills are placed into a jumble of 1-on-1 conversations, 80k advising calls, and fellowship and internship listings. Having those calls and filling out internship and fellowship applications takes a ton of time and mental energy, and might leave people more confused than they were initially. A well-run training program could eliminate many of these inefficiencies and reduce the risk that interested people won’t be able to find a job in EA.
We can roughly rank skill-building methods by the number of people they reach (“scale”), and the depth of training that they provide. In the list below, “high depth” skill development could lead to being hired for that skill (when one would not have been hired for that skill otherwise), “medium depth” as warranting a promotion or increase in seniority level, and “low depth” as an enhancement of knowledge that can help someone perform their job better, but probably won’t lead to new positions or higher status.
Internal development within organizations, like Aaron Gertler mentioned (small scale, medium depth)
Internship/fellowship programs (medium scale, medium depth)
One-off workshops and lectures (small scale, low depth)
A training program like the one I described above (large scale, high depth)
An EA university, as proposed here (large scale, high depth)
If we choose option 2, we probably want large scale, high depth ways to train people. I’m interested in hearing people’s thoughts on whether this is a good way to evaluate skill-building methods.
One caveat: there’s a lot more interest in working for the military than there is in working for EA orgs. Since this interest already exists, the military just needs to capitalize on it (although they still spend lots of money on recruitment ads and programs like ROTC). The EA community doesn’t even have great name recognition, so it’s probably premature to assume that we’d have waves of people signing up for such a training program—but it’s possible that we could get to that point with time.
Thanks for the post, this is an important and under-researched topic.
Examples include some well-known conditions (chronic migraine, fibromyalgia, non-specific low-back pain), as well as many lesser-known ones (trigeminal neuralgia, cluster headache, complex regionary pain syndrome)
Some of these well-known chronic pain conditions can be hard to diagnose, too. Chronic pain conditions like fibromyalgia, ME/CFS, rheumatoid arthritis, and irritable bowel syndrome are frequently comorbid with each other, and may also be related to depression and mental health disorders. This overlap probably makes it harder for doctors to tease out the root cause of patients’ symptoms.
As an anecdote, a close relative spent around a year bouncing around various doctors before she got a useful diagnosis, and even then the recommended therapies didn’t help much. So far, her pain is managed best by a diet she found on the internet herself.
I speculate that conventional medicine’s relative lack of machinery for identifying and treating some of these chronic illnesses may cause some patients to turn to pseudoscience instead—which could be another downstream harm of neglecting chronic pain treatments. (I haven’t tried to look for evidence for/against this conclusion.)
This is an interesting idea. I'm trying to think of it in terms of analogues: you could feasibly replace "digital minds" with "animals" and achieve a somewhat similar conclusion. It doesn't seem that hard to create vast amounts of animal suffering (the animal agriculture industry has this figured out quite well), so some agent could feasibly threaten all vegans with large-scale animal suffering. And as you say, occasionally following through might help make that threat more credible.
Perhaps the reason we don't see this happening is that nobody really wants to influence vegans alone. There aren't many strategic reasons to target an unorganized group of people whose sole common characteristic is that they care about animals. There isn't much that an agent could gain from a threat.
I imagine the same might be true of digital minds. If it's anything similar to the animal case, moral circle expansion to digital minds will likely occur in the same haphazard, unorganized way--and so there wouldn't be much of a reason to specifically target people who care about digital minds. That said, if this moral circle expansion caught on predominantly in one country (or maybe within one powerful company), a competitor or opponent might then have a real use for threatening the digital mind-welfarists. Such an unequal distribution of digital mind-welfarists seems quite unlikely, though.
At any rate, this might be a relevant consideration for other types of moral circle expansion, too.
The hygiene hypothesis (especially the autoimmune disease variant, brief 2-paragraph summary here if you Ctrl+F "Before we go") could be another example.
On a somewhat related note, Section V of this SlateStarCodex post goes through some similar examples where humans departing from long-lived tradition has negative effects that don't become visible for a long time.
I'm curious about what's the original source of the funding you're giving out here. According to this Nonlinear received $250k from Future Fund and $600k from Survival and Flourishing Fund. Is the funding being distributed here coming solely from the SFF grant? Does Nonlinear have other funding sources besides Future Fund and SFF?
(I didn't do any deeper dive than looking at Nonlinear's website, where I couldn't find anything about funding sources.)
Thanks for writing this--even though I've been familiar with AI x-risk for a while, it didn't really hit me on an emotional level that dying from misaligned AI would happen to me too, and not just "humanity" in the abstract. This post changed that.
Might eventually be useful to have one of these that accounts for biorisk too, although biorisk "timelines" aren't as straightforward as trying to estimate the date that humanity builds the first AGI.
Thanks, great points (and counterpoints)!
I like this suggestion--what do you imagine this transparency looks like? Do you think, e.g., EA groups should have pages outlining their community-building philosophies on their websites? Should university groups should write public Forum posts about their plans and reasoning before every semester/quarter or academic year? Would you advocate for more community-building roundtables at EAGs? (These are just a few possible example modalities of transparency that just came to my head, very interested in hearing more.)
Hey Jordan! Great to see another USC person here. The best writing advice I've gotten (that I have yet to implement) is to identify a theory of change for each potential piece--something to keep in mind!
6 sounds interesting, if you can make a strong case for it. Aligning humans isn't an easy task (as most parents, employers, governments, and activists know very well), so I'm curious to hear if you have tractable proposals.
7 sounds important given that a decent number of EAs are vegan, and I'm quite surprised I haven't heard of this before. 15 IQ points is a whole standard deviation, so I'd love to see the evidence for that.
8 might be interesting. I suspect most people are already aware of groupthink, but it could be good to be aware of other relevant phenomena that might not be as widely-known (if there are any).
From what I can tell, 11 proposes a somewhat major reconsideration of how we should approach improving the long-term future. If you have a good argument, I'm always in favor of more people challenging the EA community's current approach. I'm interested in 21 for the same reason.
(In my experience, the answer to 19 is no, probably because there isn't a clear, easy-to-calculate metric to use for longtermist projects in the way that GiveWell uses cost-effectiveness estimates.)
Out of all of these, I think you could whip up a draft post for 7 pretty quickly, and I'd be interested to read it!
Thanks Linch! This list is really helpful. One clarifying question on this point:
For 1) and 2), I assume you're referring to the skills gained by the person/institution completing the project, which they could then apply to future projects.
For 3), are you referring to the possibility of "ruling out intervention X as a feasible way to tackle x-risks"? That's what I'm assuming, but I'm just asking to make sure I understand properly.
Thanks again!
This thinking has come up in a few separate intro fellowship cohorts I’ve facilitated. Usually, somebody tries to flesh it out by asking whether it’s “more effective” to save one doctor (who could then be expected to save five more lives) or two mechanics (who wouldn’t save any other lives) in trolley-problem scenarios. This discussion often gets muddled, and many people have the impression that “EAs” would think it’s better to save the doctor, even though I doubt that’s a consensus opinion among EAs. I’ve found this to be a surprisingly large snag point that isn’t discussed much in community-building circles.
I think it would be worth it to clarify the difference between intrinsic and instrumental value in career advice/intro fellowships/other first interactions with the EA community, because there are some people who might agree with other EA ideas but find that this argument undermines our basic principles (as well as the claim that you don’t need to be utilitarian to be an EA). Maybe we could extend current messaging about ideological diversity within EA.
That said, I read Objection 4 differently. Many people (especially in cultures that glorify work) tie their sense of self-worth to their jobs. I don’t know how universal this is, but at least in my middle-class American upbringing, there was a strong sense that your career choice and achievement is a large part of your value as a person.
As a result, some people feel personally judged when their intended careers aren’t branded as “effective”. If you equate your career value with your personal value, you won’t feel very good if someone tells you that your career isn’t very valuable, and so you’ll resist that judgment.
I don’t think that this feeling precludes people from being EAs. It takes time to separate yourself from your current or intended career, and Objection 4 strikes me as a knee-jerk defensive reaction. Students planning to work in shipping logistics won’t immediately like the idea that the job they’ve been working hard to prepare for is “ineffective,” but they might come around to it after some deeper reflection.
I could be misreading Objection 4, though. It could also mean something like “shipping logistics is valuable because the world would grind to a halt if nobody worked in shipping logistics,” but then that’s just a variant of Objection 5.
I’m very curious to know more about the sense in which these students gave Objection 4.
I recently learned about Training for Good, a Charity Entrepreneurship-incubated project, which seems to address some of these problems. They might be worth checking out.
I think this is a great exercise to think about, especially in light of somewhat-recent discussion on how competitive jobs at EA orgs are. There seems to be plenty of room for more people working on EA projects, and I agree that it’s probably good to fill that opportunity. Some loose thoughts:
There seem to be two basic ways of getting skilled people working on EA cause areas:
1. Selectively recruiting people who already have skills.
2. Recruiting promising people who might not yet have needed skills and train them.
Individual organizations can choose both options, depending on their level of resources. But if most organizations choose option 1, the EA community might be underutilizing its potential pool of human resources. So we might want the community in general to use option 2, so that everyone who wants to be involved with EA can have a role—even if individual EA organizations still choose option 1. For this to happen, the EA community would probably need a program whereby motivated people can choose a skillset to learn, are taught that skillset, and are matched with a job at the end of the process.
Currently, motivated people who don’t yet possess skills are placed into a jumble of 1-on-1 conversations, 80k advising calls, and fellowship and internship listings. Having those calls and filling out internship and fellowship applications takes a ton of time and mental energy, and might leave people more confused than they were initially. A well-run training program could eliminate many of these inefficiencies and reduce the risk that interested people won’t be able to find a job in EA.
We can roughly rank skill-building methods by the number of people they reach (“scale”), and the depth of training that they provide. In the list below, “high depth” skill development could lead to being hired for that skill (when one would not have been hired for that skill otherwise), “medium depth” as warranting a promotion or increase in seniority level, and “low depth” as an enhancement of knowledge that can help someone perform their job better, but probably won’t lead to new positions or higher status.
If we choose option 2, we probably want large scale, high depth ways to train people. I’m interested in hearing people’s thoughts on whether this is a good way to evaluate skill-building methods.
One caveat: there’s a lot more interest in working for the military than there is in working for EA orgs. Since this interest already exists, the military just needs to capitalize on it (although they still spend lots of money on recruitment ads and programs like ROTC). The EA community doesn’t even have great name recognition, so it’s probably premature to assume that we’d have waves of people signing up for such a training program—but it’s possible that we could get to that point with time.
Thanks for the post, this is an important and under-researched topic.
Some of these well-known chronic pain conditions can be hard to diagnose, too. Chronic pain conditions like fibromyalgia, ME/CFS, rheumatoid arthritis, and irritable bowel syndrome are frequently comorbid with each other, and may also be related to depression and mental health disorders. This overlap probably makes it harder for doctors to tease out the root cause of patients’ symptoms.
As an anecdote, a close relative spent around a year bouncing around various doctors before she got a useful diagnosis, and even then the recommended therapies didn’t help much. So far, her pain is managed best by a diet she found on the internet herself.
I speculate that conventional medicine’s relative lack of machinery for identifying and treating some of these chronic illnesses may cause some patients to turn to pseudoscience instead—which could be another downstream harm of neglecting chronic pain treatments. (I haven’t tried to look for evidence for/against this conclusion.)
This is an interesting idea. I'm trying to think of it in terms of analogues: you could feasibly replace "digital minds" with "animals" and achieve a somewhat similar conclusion. It doesn't seem that hard to create vast amounts of animal suffering (the animal agriculture industry has this figured out quite well), so some agent could feasibly threaten all vegans with large-scale animal suffering. And as you say, occasionally following through might help make that threat more credible.
Perhaps the reason we don't see this happening is that nobody really wants to influence vegans alone. There aren't many strategic reasons to target an unorganized group of people whose sole common characteristic is that they care about animals. There isn't much that an agent could gain from a threat.
I imagine the same might be true of digital minds. If it's anything similar to the animal case, moral circle expansion to digital minds will likely occur in the same haphazard, unorganized way--and so there wouldn't be much of a reason to specifically target people who care about digital minds. That said, if this moral circle expansion caught on predominantly in one country (or maybe within one powerful company), a competitor or opponent might then have a real use for threatening the digital mind-welfarists. Such an unequal distribution of digital mind-welfarists seems quite unlikely, though.
At any rate, this might be a relevant consideration for other types of moral circle expansion, too.