California Proposes AI 'Kill Switch' and Mandatory Safety Oversight

California’s regulatory landscape for artificial intelligence is poised for a significant shift as Governor Gavin Newsom’s administration introduces a framework for emergency intervention and heightened oversight. Released on Wednesday, November 18, by the California Government Operations Agency, the proposal seeks to grant state regulators the authority to mandate the immediate shutdown of AI systems that pose an imminent threat of catastrophic harm or loss of control.

The move comes amid growing concerns regarding the pace of AI development and the potential for systems to operate beyond human control.

Emergency Shutdowns and Regulatory Authority

Under the proposed powers, state officials could order a developer to deactivate an AI system if an imminent threat cannot be mitigated through less restrictive measures. The recommendations call for mandatory safety standards and continuous independent evaluations to ensure developers maintain control over their models.

The proposal also addresses the speed of incident reporting. While the existing 24-hour requirement for reporting imminent catastrophic risks would remain, the deadline for reporting other critical safety incidents would be tightened from 15 days to 48 hours.

Independent Oversight and Compliance

The framework emphasizes the role of independent organizations in verifying safety protocols. Large AI developers would be subject to continuous assessments, with evaluators granted access to technical information and testing environments. If an evaluator determines that a developer is failing to meet state safety standards, regulators would have the authority to mandate corrective actions.

Industry Context

The push for these regulations occurs alongside broader industry discussions. Notably, Anthropic, the company behind the Claude AI chatbot, has privately pitched lawmakers its own federal AI safety bill. This proposal would establish new oversight requirements for the industry while simultaneously restricting when the government can block dangerous artificial intelligence models.

For the California proposals to become law, they must be approved by the state legislature. If enacted, they would represent a significant expansion of state authority, filling what officials describe as a critical gap in federal AI regulation.