We should be reluctant to make statements which could be taken as "scientific" justification for ignoring either of the previous bullet points
Thank you for stating plainly what I suspect the original doc was trying to hint at.
That said, now that it's plainly stated, I disagree with it. The world is too connected for that.
Taken literally, "could be taken" is a ridiculously broad standard. I'm sure a sufficiently motivated reasoner could take "2+2=4" as justification for racism. This is not as silly a concern as it sounds, since we're mostly worried about motivated reasoners, and it's unclear how motivated a reasoner we should be reluctant to offer comfort to. But let's look at some more concrete examples:
In early 2020, people were reluctant to warn about covid-19 because it could be taken as justification for anti-chinese racism. I can't actually follow the logic that goes from "A dangerous new disease emerged in China" to "I should go beat up someone of Chinese ancestry" but it seems a few people who had been itching for an excuse did. Nevertheless, given the relative death tolls, we clearly should have had more warnings and more preparations. The next pandemic will likely also emerge in a place containing people against whom racism is possible (base rate, if nothing else), and pandemic preparedness people need to be ready to act anyway.
Similarly, many people tried to bury the fact that monkeypox was sexually transmitted because it could lead to homophobia. So instead they warned of a coming pandemic. False warnings are extremely bad for preparedness, draining both our energy and our credibility.
Political and Economic Institutions are a potentially high-impact cause area in both near- and far-term (albeit, dubiously tractable). Investigating them is pretty much going to require looking at history, and at least sometimes saying that western institutions are better than others.
Going back to Bostrom's original letter, many anti-racists have taken to denying the very idea of intelligence in order to reject it. Hard to work on super-intelligence-based x-risk (or many other things) without that concept.
Another class of pressures require me to recapitulate the great covid lesson of social reality vs. physical reality (I forget which rationalist citation goes here, I'm building off of a discussion at NYC Solstice):
One challenge will be the uncertainty in models. It's (comparatively) easy to say "this will reshuffle climate so that our system which assumes current climate zones will be in trouble" and much harder to say which areas will flood and which will burn.
There may be work with doing refining those models.
There may also be things we can do to increase flexibility and resiliency without knowing exactly what's coming.
I remember a program specifically for young women (possibly in Bangladesh, possibly linked from slatestarcodex) that specifically listed "ambition" as one of the things it wanted to foster. But participants went in wanting to be doctors and left wanting to be administrative assistants. They did not show improvement on any measured axis. Can't seem to find the link
I'd like to introduce a setup that's a little different from these arbitrary axes and feels truer to life...
To avoid object-level politics, I'll use Scott's (or was it Nick's?) example: * Party A wants to increase taxes and social services 5%, and to require everyone to electrocute themselves 8 hours a day. * Party B wants to decrease taxes and social services 5%, and to require everyone to electrocute themselves 8 hours a day. * Party C wants to leave taxes and social services as they are, and stop the electrocutions.
"Everyone" knows that Party C isn't serious. They get no media coverage, except as a punchline. Only people with no popularity to lose will come out openly as Party C'ers. And rather then break the dam, they make Party C association a mark of stigma.
Under FPTP, we need roughly a third of the people to *believe party C has a chance*, with no way to build momentum. Naturally, the electrocutions continue.
Under IRV (and I think any ordered ranking), we need roughly a third of people to pay attention. Then they can easily vote C>A>B or C>B>A and end the electrocutions. And if it's less than a third, it still shows a nice clear signal that Not Electrocuting Ourselves is an idea to be taken seriously.
Under Approval, people won't want to vote "C" because that gives up the chance to effect the taxes/services tradeoff which is the only thing they expect to be up for grabs. So they vote "A,C" or "B,C". And feel bad about it, because they don't actually *approve* of A or B. Which means they're voting against themselves. Now we need half of people to pay attention, and with a much weaker take-this-seriously signal. After all, "A,C" could just be intented as a hardcore vote against B (some people take the A-B rivalry very seriously).
This misses senses in which resources can run out.
Simplestly, there's locked-in-use. Consider Rhenium. It's about 1ppb in Earth's crust and about 1000 tonnes of it have been refined in all of history. How much can be produced without implausibly destructive mining techniques is hard to estimate. It's essentially indestructable and uncreateable. It's used in jet engines and other high-temperature high-pressure applications. The number of jet engines in service at any time is bounded by available Rhenium. After that limit, new engines can only be made by melting down old ones. If you try to stick a trillion people on Earth, the jet/person ratio may get awfully low.
A more subtle locked-when-in-use resource is surface area. Especially temperate land surface. It can either be providing humans with psychologically-needed sky access or be covered in solar panels. (Or be used for agriculture or left as wilderness for other species, but in the extreme case those will be abandoned as inefficient.)
Another failure mode is that we may fail to solve the technological problems in using resources so efficiently. I mentioned above that agriculture was an inefficient way of converting sunlight and CHON into consumable food. But if we replace it with photovoltic cells and chemical plants, we risk missing a vital micronutrient and suffering widespread health issues.
A subtler version of this is failing to solve the social problems. Imagine living in Alberta Canada in a world where electricity comes from a solar plant thousands of miles to the south and water from a desalinization plant thousands of miles to the west. And if either breaks down even briefly you and your neighbors all die. Likewise if anything goes wrong in the transmission systems anywhere along the route. Can we run infrastructure that reliably? Can we prevent terrorism in such a situation? Can we cope with the problems caused by the solutions to the preceding problems?
Just make sure no AI seizes control of its eye-color channel:
Buy your grandmother proper dentures and she eats a balanced diet, teach her to suck eggs and...
(Yes, I propose the new proverb trail off)
Thank you for stating plainly what I suspect the original doc was trying to hint at.
That said, now that it's plainly stated, I disagree with it. The world is too connected for that.
Taken literally, "could be taken" is a ridiculously broad standard. I'm sure a sufficiently motivated reasoner could take "2+2=4" as justification for racism. This is not as silly a concern as it sounds, since we're mostly worried about motivated reasoners, and it's unclear how motivated a reasoner we should be reluctant to offer comfort to. But let's look at some more concrete examples:
The citation you're looking for is https://putanumonit.com/2021/04/03/monastery-and-throne/ particularly the section titled "Coordinating Social Reality"
One challenge will be the uncertainty in models. It's (comparatively) easy to say "this will reshuffle climate so that our system which assumes current climate zones will be in trouble" and much harder to say which areas will flood and which will burn.
There may be work with doing refining those models.
There may also be things we can do to increase flexibility and resiliency without knowing exactly what's coming.
But it'll be tricky.
How did you get your employer to approve "half time for half pay"?
I remember a program specifically for young women (possibly in Bangladesh, possibly linked from slatestarcodex) that specifically listed "ambition" as one of the things it wanted to foster. But participants went in wanting to be doctors and left wanting to be administrative assistants. They did not show improvement on any measured axis. Can't seem to find the link
DARE belongs in the same paragraph as Scared Straight, FWIW
[Repost from my FB]
I'd like to introduce a setup that's a little different from these arbitrary axes and feels truer to life...
To avoid object-level politics, I'll use Scott's (or was it Nick's?) example:
* Party A wants to increase taxes and social services 5%, and to require everyone to electrocute themselves 8 hours a day.
* Party B wants to decrease taxes and social services 5%, and to require everyone to electrocute themselves 8 hours a day.
* Party C wants to leave taxes and social services as they are, and stop the electrocutions.
"Everyone" knows that Party C isn't serious. They get no media coverage, except as a punchline. Only people with no popularity to lose will come out openly as Party C'ers. And rather then break the dam, they make Party C association a mark of stigma.
Under FPTP, we need roughly a third of the people to *believe party C has a chance*, with no way to build momentum. Naturally, the electrocutions continue.
Under IRV (and I think any ordered ranking), we need roughly a third of people to pay attention. Then they can easily vote C>A>B or C>B>A and end the electrocutions. And if it's less than a third, it still shows a nice clear signal that Not Electrocuting Ourselves is an idea to be taken seriously.
Under Approval, people won't want to vote "C" because that gives up the chance to effect the taxes/services tradeoff which is the only thing they expect to be up for grabs. So they vote "A,C" or "B,C". And feel bad about it, because they don't actually *approve* of A or B. Which means they're voting against themselves. Now we need half of people to pay attention, and with a much weaker take-this-seriously signal. After all, "A,C" could just be intented as a hardcore vote against B (some people take the A-B rivalry very seriously).
This misses senses in which resources can run out.
Simplestly, there's locked-in-use. Consider Rhenium. It's about 1ppb in Earth's crust and about 1000 tonnes of it have been refined in all of history. How much can be produced without implausibly destructive mining techniques is hard to estimate. It's essentially indestructable and uncreateable. It's used in jet engines and other high-temperature high-pressure applications. The number of jet engines in service at any time is bounded by available Rhenium. After that limit, new engines can only be made by melting down old ones. If you try to stick a trillion people on Earth, the jet/person ratio may get awfully low.
A more subtle locked-when-in-use resource is surface area. Especially temperate land surface. It can either be providing humans with psychologically-needed sky access or be covered in solar panels. (Or be used for agriculture or left as wilderness for other species, but in the extreme case those will be abandoned as inefficient.)
Another failure mode is that we may fail to solve the technological problems in using resources so efficiently. I mentioned above that agriculture was an inefficient way of converting sunlight and CHON into consumable food. But if we replace it with photovoltic cells and chemical plants, we risk missing a vital micronutrient and suffering widespread health issues.
A subtler version of this is failing to solve the social problems. Imagine living in Alberta Canada in a world where electricity comes from a solar plant thousands of miles to the south and water from a desalinization plant thousands of miles to the west. And if either breaks down even briefly you and your neighbors all die. Likewise if anything goes wrong in the transmission systems anywhere along the route. Can we run infrastructure that reliably? Can we prevent terrorism in such a situation? Can we cope with the problems caused by the solutions to the preceding problems?