I completely agree. That's the core issue I'm trying to untangle. Commitments erode under pressure; we saw it this week in action and quickly going downhill, and I don't think it will be the first or last time, given the stakes. But for me, the deeper issue is that even when commitments hold (perfect world), we don't have tools to verify whether they're actually being met.
Look at Karnofsky's post. He's remarkably blunt about how Anthropic's own RSP created pressure to declare systems below capability thresholds to avoid triggering pause requirements. The commitments existed on paper (good). The institutional incentives worked against honest evaluation (good). If they couldn't even verify whether their own commitments were being met honestly, that tells you the verification tools don't exist yet (failed).
On leverage, I think the answer isn't promises but actual infrastructure that makes it costly to lie. Take for examole how financial markets work and regulate themselves. They don't rely on companies' goodwill. They rely on auditing standards and independent verification. AI governance has none of that. The Symmetrian Index is a first step toward building that auditing layer.
Great article! The point about measurement enabling governance is right (100% agree); we can't regulate what we can't measure. I've been thinking/working on a related gap.
I've been working on measuring not just AI behavior, but whether AI systems are actually governable. By governable, I mean: can institutions trace how a decision was made, contest the reasoning, and audit it after the fact?
I ran evaluations on GPT-4 and Claude Opus and found a pattern I've started calling "comprehension decay." Both models score nearly perfect on comprehensibility = clear, readable outputs. But both score very low on reversibility = almost no mechanisms to trace reasoning, contest decisions, or audit what happened.
The outputs are readable. The reasoning is unverifiable.
This matters for the measurement-enables-governance thesis. Even if we build great behavioral benchmarks (sycophancy, deception, etc.), we face a deeper problem: when you ask an LLM (using this system as an example, but it applies to all) why it said something, the answer is just more generated text. There's no trace of what actually happened inside the model. You can't verify it, contest it, or audit against it.
So I'd add something to the framework: we need measurement of governability itself, not just behavior. Can decisions be traced to computations? Can those traces be challenged? Are there audit hooks?
Without this, one of the biggest risks is building infrastructure that creates the appearance of accountability without the substance of it; we have fluent explanations that don't connect to anything you can actually govern.
I posted last week about if you want to check it out in more detail!
I am so happy to see this! As someone who lives in South America and sees the cultural barriers to accessing mental health, it's great to see an initiative like this!
Thanks for this post, Will. I believe you've touched on many points that many of us have been pondering. I've translated it into Spanish, as I feel it's relevant to the entire community.
Thanks for this post. I found it incredibly insightful, especially the part discussing how crucial it is to ensure that AI goals align with human values. The risks, ranging from competitive pressures to the potential for advanced AI deception, are authentic and require immediate attention. In particular, the concept of a 'treacherous turn,' where an AI pretends to align with our goals until it gains enough power to pursue its own, is both captivating and alarming. It emphasizes the need for advancements in research on AI transparency, model honesty, and adversarial robustness.
Understanding the fast-paced and impactful nature of the AI field, I've taken the liberty to translate this critical content into Spanish to make it more accessible. In this rapidly evolving field, fostering inclusivity and accessibility of information is crucial.
Great post. As someone who's always keeping an eye on Rethink Priorities, I enjoyed learning about your journey, the bumps along the way, and the changes you're looking to make. I've taken the liberty of translating the post into Spanish as part of my initiative to make important content more available to Spanish speakers. I think many would be interested in learning about RP and your journey as a founder.
Hello Jesse! Great article. I felt the need to translate the article into Spanish to make it accessible for Spanish-speaking individuals! That being said, I'm from Florida, where my home has perennially battled with high humidity. Since childhood, I've dealt with allergies and frequent respiratory illnesses and even saw my mother develop rhinitis. We were under the impression that our HVAC system was functioning optimally, but after reviewing this article, I've come to understand that there's considerable scope for improvement and proactive maintenance. As you mentioned, the workforce isn't very instructed on what to do or has the best interests in installing a safe and reliable system. Thanks again for this. I learned a lot and will put it to use.
Hey Joey! Thank you so much for this thoughtful and comprehensive piece on the complexities of starting and managing a nonprofit incubator. I think this can be very useful for Spanish speakers too. I took the liberty of translating it! Hope that's not a problem! Thanks again for this amazing piece!
Hi Richard! I truly appreciate your enlightening post! It struck me as highly informative, and I believe others will feel the same. In order to reach out to our Spanish-speaking individuals, I have proactively translated it into Spanish, as I'm sure they will also see the immense value in this information.
I completely agree. That's the core issue I'm trying to untangle. Commitments erode under pressure; we saw it this week in action and quickly going downhill, and I don't think it will be the first or last time, given the stakes. But for me, the deeper issue is that even when commitments hold (perfect world), we don't have tools to verify whether they're actually being met.
Look at Karnofsky's post. He's remarkably blunt about how Anthropic's own RSP created pressure to declare systems below capability thresholds to avoid triggering pause requirements. The commitments existed on paper (good). The institutional incentives worked against honest evaluation (good). If they couldn't even verify whether their own commitments were being met honestly, that tells you the verification tools don't exist yet (failed).
On leverage, I think the answer isn't promises but actual infrastructure that makes it costly to lie. Take for examole how financial markets work and regulate themselves. They don't rely on companies' goodwill. They rely on auditing standards and independent verification. AI governance has none of that. The Symmetrian Index is a first step toward building that auditing layer.
Great article! The point about measurement enabling governance is right (100% agree); we can't regulate what we can't measure. I've been thinking/working on a related gap.
I've been working on measuring not just AI behavior, but whether AI systems are actually governable. By governable, I mean: can institutions trace how a decision was made, contest the reasoning, and audit it after the fact?
I ran evaluations on GPT-4 and Claude Opus and found a pattern I've started calling "comprehension decay." Both models score nearly perfect on comprehensibility = clear, readable outputs. But both score very low on reversibility = almost no mechanisms to trace reasoning, contest decisions, or audit what happened.
The outputs are readable. The reasoning is unverifiable.
This matters for the measurement-enables-governance thesis. Even if we build great behavioral benchmarks (sycophancy, deception, etc.), we face a deeper problem: when you ask an LLM (using this system as an example, but it applies to all) why it said something, the answer is just more generated text. There's no trace of what actually happened inside the model. You can't verify it, contest it, or audit against it.
So I'd add something to the framework: we need measurement of governability itself, not just behavior. Can decisions be traced to computations? Can those traces be challenged? Are there audit hooks?
Without this, one of the biggest risks is building infrastructure that creates the appearance of accountability without the substance of it; we have fluent explanations that don't connect to anything you can actually govern.
I posted last week about if you want to check it out in more detail!
I am so happy to see this! As someone who lives in South America and sees the cultural barriers to accessing mental health, it's great to see an initiative like this!
Thanks for this post, Will. I believe you've touched on many points that many of us have been pondering. I've translated it into Spanish, as I feel it's relevant to the entire community.
Thanks for this post. I found it incredibly insightful, especially the part discussing how crucial it is to ensure that AI goals align with human values. The risks, ranging from competitive pressures to the potential for advanced AI deception, are authentic and require immediate attention. In particular, the concept of a 'treacherous turn,' where an AI pretends to align with our goals until it gains enough power to pursue its own, is both captivating and alarming. It emphasizes the need for advancements in research on AI transparency, model honesty, and adversarial robustness.
Understanding the fast-paced and impactful nature of the AI field, I've taken the liberty to translate this critical content into Spanish to make it more accessible. In this rapidly evolving field, fostering inclusivity and accessibility of information is crucial.
Great post. As someone who's always keeping an eye on Rethink Priorities, I enjoyed learning about your journey, the bumps along the way, and the changes you're looking to make. I've taken the liberty of translating the post into Spanish as part of my initiative to make important content more available to Spanish speakers. I think many would be interested in learning about RP and your journey as a founder.
Hello Jesse! Great article. I felt the need to translate the article into Spanish to make it accessible for Spanish-speaking individuals! That being said, I'm from Florida, where my home has perennially battled with high humidity. Since childhood, I've dealt with allergies and frequent respiratory illnesses and even saw my mother develop rhinitis. We were under the impression that our HVAC system was functioning optimally, but after reviewing this article, I've come to understand that there's considerable scope for improvement and proactive maintenance. As you mentioned, the workforce isn't very instructed on what to do or has the best interests in installing a safe and reliable system. Thanks again for this. I learned a lot and will put it to use.
Hey Joey! Thank you so much for this thoughtful and comprehensive piece on the complexities of starting and managing a nonprofit incubator. I think this can be very useful for Spanish speakers too. I took the liberty of translating it! Hope that's not a problem! Thanks again for this amazing piece!
Hi Richard! I truly appreciate your enlightening post! It struck me as highly informative, and I believe others will feel the same. In order to reach out to our Spanish-speaking individuals, I have proactively translated it into Spanish, as I'm sure they will also see the immense value in this information.
Thanks again!