Low probabilities doesn't seem like the appropriate crux. They can be generated by e.g. considering tiny time intervals. The issue seems more like "reality is underpowered" or "we don't have well tested or believable models". For airplanes we've flown the huge numbers of miles and crashed the planes needed to find most common failure modes and developed models/checklists/practices to mitigate this. We intentionally simulate the low probability events to maximize our chances in surviving them. For asteroids and voting we have statistical models that give us reasonable confidence in the chances of impact.
The challenge is how do we model existential risks without incurring the risk itself? We can't crash humanity thousands of times to figure out how to not do it as often. In some cases (e.g. pandemics) we may have a small sample to look at to find the more frequent low-impact events to build a swiss cheese model from (pandemic causes, reactions to pandemics), but in others (AI, unknown risks) modeling without any data seems very hard.
Low probabilities doesn't seem like the appropriate crux. They can be generated by e.g. considering tiny time intervals. The issue seems more like "reality is underpowered" or "we don't have well tested or believable models". For airplanes we've flown the huge numbers of miles and crashed the planes needed to find most common failure modes and developed models/checklists/practices to mitigate this. We intentionally simulate the low probability events to maximize our chances in surviving them. For asteroids and voting we have statistical models that give us reasonable confidence in the chances of impact.
The challenge is how do we model existential risks without incurring the risk itself? We can't crash humanity thousands of times to figure out how to not do it as often. In some cases (e.g. pandemics) we may have a small sample to look at to find the more frequent low-impact events to build a swiss cheese model from (pandemic causes, reactions to pandemics), but in others (AI, unknown risks) modeling without any data seems very hard.