I research and train expertise and decision-making in various high stake domains. I work with Gary Klein who founded the field of Naturalistic Decision Making.
Figuring out who to involve is half of figuring out how to manage a crisis, but this cannot be entirely protocolized because each crisis is different. As such, your crisis management team should be very aware of what various people in the organization do so that they can know who to bring in and when.
Vague verbiage about how urgent and how large a crisis is can be a major confusion, especially for non-technical staff who may not fully understand the nature of the crisis. Teams with experience managing multiple crises might come up with some sort of idiosyncratic terminology for communicating these things, but teams that do not have regular crises will likely not have such a terminology.
Do regular scenario exercises. There are many valid ways to do them, and the important thing to consider is what types of fidelity actually matter. For most exercises the physical fidelity is not going to matter a lot, but maintaining the cognitive fidelity (ie the actual cognitive challenges you would face in a real crisis) will be very important.
I think it is also generally important to maintain some of the stress of a real situation (this is called psychological fidelity), but without making exercises something to be dreaded. Stressful exercises can be fun if managed right.
Tunnel vision is very common in crises, even when evidence starts mounting up that things are different. You need to train an anomaly detection stance in which people notice and mention when their expectations have been been proven wrong.
Maintaining 'common ground' is very difficult. One thing we have found useful is having people play different roles in the organization than their actual role (eg getting the researchers to be in charge of supply chain, and supply chain in charge of facilities) as it gives everyone a more in depth understanding of how the organization as a whole responds to crises. This is one way to ensure bullet point 1 above, and also helps people better understand prioritization.
A lot of political and organisation decision making is not based on how accurately you’ve predicted the world. Sad, but true.
I do not consider this sad, but just a fact about how decision-making works in the real world. Economists model humans as-if they use probabilities to make choices between well-defined options. If you model humans as making decisions in this way, then forecasting makes sense.
But the real world is not a multiple choice test offering pre-defined options which need probabilistic estimates. In real world decision-making, problem formulation and problem solving are the same cognitive process. There are no problems out there waiting for you to find them, rather, you have to define what is even a problem in the first place and how to think about it. Figuring out how to solve a problem is a process of sensemaking until you feel like you have a grip on the situation; like you know what is relevant, how it is relevant, what the moving parts are, how they interact, what your leverage point is, and what your values are. A probabilistic estimate can never offer that. A probabilistic estimate abstracts away so much that it actually leaves you feeling like you don't know the space at all.
Dominic Cummings has mentioned this
During the pandemic, Dominic Cummings said some of the most useful stuff that he received and circulated in the British government was not forecasting, it was qualitative information explaining the general model of what’s going on, which enabled decision-makers to think more clearly about their options for action and the likely consequences. If you’re worried about a new disease outbreak, you don’t just want a percentage probability estimate about future case numbers, you want an explanation of how the virus is likely to spread, what you can do about it, how you can prevent it. Not the best estimate for how many COVID cases there will be in a month, but why forecasters believe there will be X COVID cases in a month. https://www.samstack.io/p/five-questions-for-michael-story
Forecasting is based on an as-if (read: wrong) model of decision-making. You wouldn't decide when to make a left-hand-turn at a busy intersection based on some narrow probabilistic estimate (there is a 99% chance you won't get t-boned if you turn now) because you want to understand the decision yourself, and that probabilistic estimate is missing so much (Will I T-bone someone? Are there people in the cross walk? Are the cars slowing down because the light already changed colors? Or are they speeding up because it already changed colors?)
In real world decision-making, framing/modeling is everything. How you take large problems and turn them into something tractable that the human mind can comprehend and reason about. But superforecasting assumes that problem away. It assumes away the most important aspect of decision-making; understanding.
A couple years ago I was wondering why all the focus is on Superforecasters when really we should be emphasizing the best arguments or the best persuaders. It seems like knowing who is best at forecasting is is less useful to me that knowing what (or who) would persuade me to change my mind (since I only care about forecasts in so far as they change my mind, anyways).
The incentive system for this seems simple enough. Imagine instead of upvoting a comment, the comment has a "update your forecast" button. Comments that are persuasive get boosted by the algorithm. Authors who create convincing arguments can get prestige. Authors who create convincing argument that, on balance, lead to people making better forecasts, get even more prestige.
It could even be a widget that you embed at the beginning and end of off-site articles. That way we could find the "super-bloggers" or "super-journalists" or whatever you want to call them.
Heck, you could even create another incentive system for the people who are best at finding arguments worth updating on.
The point is, you need to incentivize more than good forecasts. You need an entire knowledge generation economy.
There is probably all kinds of ways this gets gamed. But it seems at least worth exploring. Forecasts by themselves are just not that useful. Explanations, not probabilities, are what expert decision-makers rely on. At least that is the case within my field of Naturalistic Decision Making, and also seems true in Managerial Decision Making - managers don't seem to use probabilities in order to do Expected Utility calculations, but rather to try and understand the situation and its uncertainties.
This is the conclusion Dominic Cummings came to during the pandemic, as well. Summarized here
> During the pandemic, Dominic Cummings said some of the most useful stuff that he received and circulated in the British government was not forecasting, it was qualitative information explaining the general model of what’s going on, which enabled decision-makers to think more clearly about their options for action and the likely consequences. If you’re worried about a new disease outbreak, you don’t just want a percentage probability estimate about future case numbers, you want an explanation of how the virus is likely to spread, what you can do about it, how you can prevent it. Not the best estimate for how many COVID cases there will be in a month, but why forecasters believe there will be X COVID cases in a month. https://www.samstack.io/p/five-questions-for-michael-story
A few thoughts on crisis management
I do not consider this sad, but just a fact about how decision-making works in the real world. Economists model humans as-if they use probabilities to make choices between well-defined options. If you model humans as making decisions in this way, then forecasting makes sense.
But the real world is not a multiple choice test offering pre-defined options which need probabilistic estimates. In real world decision-making, problem formulation and problem solving are the same cognitive process. There are no problems out there waiting for you to find them, rather, you have to define what is even a problem in the first place and how to think about it. Figuring out how to solve a problem is a process of sensemaking until you feel like you have a grip on the situation; like you know what is relevant, how it is relevant, what the moving parts are, how they interact, what your leverage point is, and what your values are. A probabilistic estimate can never offer that. A probabilistic estimate abstracts away so much that it actually leaves you feeling like you don't know the space at all.
Dominic Cummings has mentioned this
Forecasting is based on an as-if (read: wrong) model of decision-making. You wouldn't decide when to make a left-hand-turn at a busy intersection based on some narrow probabilistic estimate (there is a 99% chance you won't get t-boned if you turn now) because you want to understand the decision yourself, and that probabilistic estimate is missing so much (Will I T-bone someone? Are there people in the cross walk? Are the cars slowing down because the light already changed colors? Or are they speeding up because it already changed colors?)
In real world decision-making, framing/modeling is everything. How you take large problems and turn them into something tractable that the human mind can comprehend and reason about. But superforecasting assumes that problem away. It assumes away the most important aspect of decision-making; understanding.
A couple years ago I was wondering why all the focus is on Superforecasters when really we should be emphasizing the best arguments or the best persuaders. It seems like knowing who is best at forecasting is is less useful to me that knowing what (or who) would persuade me to change my mind (since I only care about forecasts in so far as they change my mind, anyways).
The incentive system for this seems simple enough. Imagine instead of upvoting a comment, the comment has a "update your forecast" button. Comments that are persuasive get boosted by the algorithm. Authors who create convincing arguments can get prestige. Authors who create convincing argument that, on balance, lead to people making better forecasts, get even more prestige.
It could even be a widget that you embed at the beginning and end of off-site articles. That way we could find the "super-bloggers" or "super-journalists" or whatever you want to call them.
Heck, you could even create another incentive system for the people who are best at finding arguments worth updating on.
The point is, you need to incentivize more than good forecasts. You need an entire knowledge generation economy.
There is probably all kinds of ways this gets gamed. But it seems at least worth exploring. Forecasts by themselves are just not that useful. Explanations, not probabilities, are what expert decision-makers rely on. At least that is the case within my field of Naturalistic Decision Making, and also seems true in Managerial Decision Making - managers don't seem to use probabilities in order to do Expected Utility calculations, but rather to try and understand the situation and its uncertainties.
This is the conclusion Dominic Cummings came to during the pandemic, as well. Summarized here
> During the pandemic, Dominic Cummings said some of the most useful stuff that he received and circulated in the British government was not forecasting, it was qualitative information explaining the general model of what’s going on, which enabled decision-makers to think more clearly about their options for action and the likely consequences. If you’re worried about a new disease outbreak, you don’t just want a percentage probability estimate about future case numbers, you want an explanation of how the virus is likely to spread, what you can do about it, how you can prevent it. Not the best estimate for how many COVID cases there will be in a month, but why forecasters believe there will be X COVID cases in a month.
https://www.samstack.io/p/five-questions-for-michael-story