Compliance

Critical safety incident

Incident like unauthorized weight access or loss of model control.

How the laws define this

Regulators agree that a critical safety incident involves severe harms or loss of control associated with frontier AI models, triggering mandatory, timely reporting obligations to oversight offices such as Cal OES. However, the instruments diverge on the exact criteria that trigger an incident and whether potential harm is sufficient. Under An Act to amend the general business law, in relation to transparency and safety requirements for developers of artificial intelligence frontier models, an incident encompasses events that merely have the potential to result in significant harm or pose a substantial threat. Conversely, the Transparency in Frontier Artificial Intelligence Act and California SB 53 — Transparency in Frontier Artificial Intelligence Act (TFAIA) focus on specific materialized conditions, requiring unauthorized access, modification, or exfiltration of model weights resulting in death or injury, deceptive model behavior that subverts controls, loss of model control, or the actual manifestation of a catastrophic risk.

Synthesised from the 3 statutory definitions below. Each one is quoted in full, with its source.

How jurisdictions differ

What is distinctive in each definition, binding law in force first.

Jurisdiction	Instrument	Status	What is distinctive
California, United States	California SB 53 — Transparency in Frontier Artificial Intelligence Act (TFAIA)	In Force	Adds deceptive model behavior that subverts developer controls alongside unauthorized access, catastrophic risk manifestation, and loss of model control.
New York, United States	An Act to amend the general business law, in relation to transparency and safety requirements for developers of artificial intelligence frontier models	In Force	Broadens to any events with frontier AI models posing potential for significant harm or substantial threat, without specifying types like weight access.
California, United States	Transparency in Frontier Artificial Intelligence Act	In Force	Limits to unauthorized access, modification, or exfiltration of model weights causing death or injury, or harm from catastrophic risk or loss of control.
California, United States	California Executive Order N-9-26 — Independent Oversight of Frontier AI	In Force	Expands to include loss-of-control incidents beyond harmful output or security events, considering examples like the Hugging Face attack.

What it triggers

Under California SB 53, such incidents trigger specified reporting timelines to Cal OES and other obligations for developers of frontier AI models.source
Under California's SB 53, such incidents trigger safety framework obligations and incident reporting for developers of advanced AI models.
Under New York S8828, these incidents necessitate mandatory and timely reporting to the oversight office.

Definitions (5)

An event such as unauthorized access to unreleased model weights causing injury, the manifestation of a catastrophic risk, loss of model control, or deceptive model behavior that subverts developer controls; such incidents trigger specified reporting timelines to Cal OES and other obligations.

California SB 53 — Transparency in Frontier Artificial Intelligence Act (TFAIA)Definition 1 of 5

A critical safety incident is defined as unauthorized access, modification, or exfiltration of a frontier model's model weights resulting in death or bodily injury. It also includes harm stemming from the materialization of a catastrophic risk or a loss of control over the AI system by its developers or users.

Transparency in Frontier Artificial Intelligence ActDefinition 2 of 5

Events involving frontier AI models that result in, or have the potential to result in, significant harm or pose a substantial threat. These incidents necessitate mandatory and timely reporting to the oversight office.

An Act to amend the general business law, in relation to transparency and safety requirements for developers of artificial intelligence frontier modelsDefinition 3 of 5

The order may expand this concept to include loss-of-control incidents, not just conventional harmful-output or security events. The official release specifically mentions considering an update to include incidents like the Hugging Face attack.

Executive Order N-9-26 — Independent Oversight of Frontier AIDefinition 4 of 5

A critical safety incident is an event involving AI that raises material safety or security concerns and may require reporting. The order suggests expanding this concept to cover broader loss-of-control incidents, not just narrower failure events.

Executive Order N-9-26 — Independent Oversight of Frontier AIDefinition 5 of 5

Not to be confused with

Incident

Critical safety incident focuses on severe events with frontier models like weight access or control loss, while incident covers any adverse effect or material risk to rights from AI operation.

Unsafe AI System

Critical safety incident refers to specific events, whereas unsafe AI system describes an AI posing unreasonable risks to health, safety, or rights.

Questions and answers

What triggers reporting for critical safety incidents in California?

Incidents like unauthorized access to model weights causing injury, catastrophic risk manifestation, loss of model control, or deceptive behavior trigger reporting timelines to Cal OES. This applies to developers of frontier AI models under SB 53.

Does a critical safety incident always require death or injury?

No, under some definitions it includes potential harm or threats without requiring actual death or injury, but one narrows it to cases resulting in death or bodily injury from weight access.

What distinguishes loss of model control in critical safety incidents?

Loss of model control is explicitly included in California definitions as a trigger, and one expands to such incidents beyond conventional harms, potentially including events like the Hugging Face attack.