Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

In this edition: California’s legislature sent SB-53—the ‘Transparency in Frontier Artificial Intelligence Act’—to Governor Newsom’s desk. If signed into law, California would become the first US state to regulate catastrophic risk.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

A note from Corin: I’m leaving the AI Safety Newsletter soon to start law school—but if you’d like to hear more from me, I’m planning to continue to write about AI in a new personal newsletter, Conditionals. On a related note, we’re also hiring a writer for the newsletter.

California’s SB-53 Passes the Legislature

SB-53 is the Legislature’s weaker sequel to last year’s vetoed SB-1047. After Governor Gavin Newsom vetoed SB-1047 last year, he convened the Joint California Policy Working Group on AI Frontier Models. The group’s June report recommended transparency, incident reporting, and whistleblower protections as near-term priorities for governing AI systems. SB-53 (the “Transparency in Frontier Artificial Intelligence Act”) is an attempt to codify those recommendations. The California Legislature passed SB-53 on September 17th.

Transparency. To track and respond to the risks involved in frontier AI development, governments need frontier developers to disclose the capabilities of their systems and how they assess and mitigate catastrophic risk. The bill defines a “catastrophic risk” as a foreseeable, material risk that a foundation model’s development, storage, use, or deployment will result in death or serious injury to more than 50 people, or more than $1 billion in property damages arising from a single incident involving a foundation model:

• Providing expert-level assistance in the creation or release of CBRN weapons.
• Autonomous cyberattack, murder, assault, extortion, or theft.
• Evading the control of its frontier developer or user.

With these risks in mind, SB-53 requires frontier developers to:

• Publish a frontier AI framework that includes (among other things) the developer’s capability thresholds for catastrophic risks, risk mitigations, and internal governance practices.
• Review and update the framework once a year, and publish modifications within 30 days.
• Publish transparency reports for each new frontier model, including technical specifications and catastrophic risk assessments.
• Share assessments of catastrophic risks from internal use of frontier models with California’s Office of Emergency Services (OES) every 3 months.
• Refrain from lying about catastrophic risks from its frontier models, its management of catastrophic risks, or its compliance with its frontier AI framework.

Incident reporting. Governments need to be alerted to critical safety incidents involving frontier AI systems—such as harms resulting from unauthorized access to model weights or loss of control of an agent—to intervene before they escalate into catastrophic outcomes. SB-53 provides that:

• The OES will establish a hotline for reporting critical safety incidents.
• Frontier developers are required to report critical safety incidents to within 15 days, or 24 hours if there is an imminent threat of death or serious injury.
• Each year, the OES will produce a report with anonymized and aggregated information about critical safety incidents.

The bill’s incident reporting requirements are also designed to accommodate future federal requirements. In the case that federal requirement for critical safety incident reporting becomes equivalent to, or stricter than, those required by SB-53, then OES can defer to those federal requirements.

Whistleblower protection. California state authorities will need to rely on whistleblowers to report whether frontier AI companies are complying with SB-53’s requirements. Given the industry’s mixed history regarding whistleblowers, the bill provides that:

• Frontier developers are prohibited from preventing or retaliating against covered employees (employees responsible for assessing, managing, or addressing risk of critical safety incidents) from reporting activities that they have reason to believe pose a specific and substantial catastrophic risk. (Existing whistleblower protections cover all employees and any violation of law—which includes SB-53’s transparency and incident-reporting requirements.)
• Each year, the Attorney General will publish a report with anonymized and aggregated information about reports from covered employees.

Covered employees can sue frontier developers for noncompliance with whistleblower protections, and the Attorney General is empowered to enforce the bill’s transparency and incident reporting requirements by punishing violations with civil penalties of up to $1 million per violation.

How we got here, and what happens next. SB-1047 required frontier AI developers to implement specific controls to reduce catastrophic risk (such as shutdown controls and prohibitions on releasing unreasonably risky models), and Governor Newsom vetoed the bill under pressure from national Democratic leadership and industry lobbying. Since SB-53 only implements transparency requirements—and relies on the recommendations made by the Governor’s working group—SB-53 seems more likely to be signed into law. Anthropic has also publicly endorsed the bill.

Governor Newsom has until October 12th to sign SB-53. If he does, SB-53 will be the first significant AI legislation to become law since Senator Ted Cruz pushed (and narrowly failed) to attach a 10-year moratorium on state and local AI enforcement to federal budget legislation. He has since picked up the idea again in a new proposal—which, if it gains traction, might set up a conflict between California and Washington.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Subscribe to receive future versions.

In Other News

Government

• The Cyberspace Administration of China banned Chinese companies from buying Nvidia chips.
• Italy approved a law regulating the use of artificial intelligence that includes criminal penalties for misuse.
• Dozens of UK lawmakers accused Google of violating its AI safety commitments.

Industry

• OpenAI and Anthropic published reports tracking economic patterns in how people use AI.
• OpenAI and DeepMind both claimed gold-medal performance at the International Collegiate Programming Contest World Finals.
• Nvidia is investing up to $100 billion in OpenAI. It’s also investing $5 billion in Intel.
• Anthropic published a report discussing how AI is being used for cybercrime.

Civil Society

• An open letter signed by former heads of state, nobel laureates, and other prominent figures calls for an international agreement on clear and verifiable red lines to prevent AI risks.
• Stanford researchers published a paper finding that employment of early-career workers in exposed industries has declined 13%.
• AI safety activists have begun hunger strikes outside of AI company headquarters in London and San Francisco.
• Dan Hendrycks and Adam Khoja respond to critiques of Mutually Assured AI Malfunction (MAIM).
• Rosario Mastrogiacomo discusses how AI agents are eroding the foundations of cybersecurity.
• Ben Brooks argues that keeping frontier AI behind paywalls could create a new form of digital feudalism.
• Oscar Delaney and Ashwin Acharya discuss ‘the hidden frontier’ of internal models at AI companies.

See also: CAIS’ X account, our paper on superintelligence strategy, our AI safety course, and AI Frontiers, a new platform for expert commentary and analysis.