Warning: AI is being used to formulate this response, it is necessary for my participation.
The measurement problem you ran into with the questionnaire seems like the actual finding here, not a limitation of the finding, and I think it deserves more weight than a footnote in the lessons learned section. Three to five raters per organization, working off whatever the org's own website says about itself, and even then you couldn't reliably compare internal to external ratings. That's not a small sample problem. That's evidence that nobody, including the organizations themselves, currently has a reliable way to see what's true about how a system is actually functioning. And if that's true, every systemic change effort discussed in this sequence, and honestly most collective decision-making anywhere, is being built on the same missing foundation.
Zoom out from just organizational self-assessment for a second. The same failure shows up everywhere systemic change actually needs to happen. Public opinion is measured through polls that are small, leading, and filtered through whoever's paying for them. Political mandates get inferred from elections that compress millions of distinct views into a single binary choice every few years. Markets get treated as a signal of what people want, when they're really a signal of what people can afford and what's been marketed to them. In every one of these cases, the actual thing, what people genuinely think, need, and would choose given honest information, never gets recorded anywhere. It gets inferred, aggregated, spun, or guessed at, and every layer of that process is a place where power dynamics and self-interest quietly rewrite the data before anyone sees it.
I've spent the past while building something aimed at that root problem directly: a public, permanent, unranked and unfiltered record of what people actually think and report, with nobody in the middle scoring, interpreting, or deleting any of it. Not another survey instrument, an actual open record anyone can check and add to. If reliable ground-truth data about what's actually happening, in organizations, in institutions, in public sentiment generally, doesn't exist anywhere right now, that's not a side issue to systemic change work, it's the blocker underneath all of it, and underneath a lot more than this sequence covers. I'd rather this get real scrutiny than sit as a footnote. kaosnow.com
Warning: AI is being used to formulate this response, it is necessary for my participation.
The measurement problem you ran into with the questionnaire seems like the actual finding here, not a limitation of the finding, and I think it deserves more weight than a footnote in the lessons learned section. Three to five raters per organization, working off whatever the org's own website says about itself, and even then you couldn't reliably compare internal to external ratings. That's not a small sample problem. That's evidence that nobody, including the organizations themselves, currently has a reliable way to see what's true about how a system is actually functioning. And if that's true, every systemic change effort discussed in this sequence, and honestly most collective decision-making anywhere, is being built on the same missing foundation.
Zoom out from just organizational self-assessment for a second. The same failure shows up everywhere systemic change actually needs to happen. Public opinion is measured through polls that are small, leading, and filtered through whoever's paying for them. Political mandates get inferred from elections that compress millions of distinct views into a single binary choice every few years. Markets get treated as a signal of what people want, when they're really a signal of what people can afford and what's been marketed to them. In every one of these cases, the actual thing, what people genuinely think, need, and would choose given honest information, never gets recorded anywhere. It gets inferred, aggregated, spun, or guessed at, and every layer of that process is a place where power dynamics and self-interest quietly rewrite the data before anyone sees it.
I've spent the past while building something aimed at that root problem directly: a public, permanent, unranked and unfiltered record of what people actually think and report, with nobody in the middle scoring, interpreting, or deleting any of it. Not another survey instrument, an actual open record anyone can check and add to. If reliable ground-truth data about what's actually happening, in organizations, in institutions, in public sentiment generally, doesn't exist anywhere right now, that's not a side issue to systemic change work, it's the blocker underneath all of it, and underneath a lot more than this sequence covers. I'd rather this get real scrutiny than sit as a footnote. kaosnow.com