Compliance

Catastrophic Risk

Risk of severe, large-scale, potentially irreversible negative impacts from AI systems on health, safety, or fundamental rights.

How the laws define this

Regulators broadly align on treating catastrophic risk as the threat of severe, widespread harm arising from advanced or frontier artificial intelligence systems, particularly through weaponization, cyberattacks, or loss of model control. However, these instruments split sharply on how to measure that threat. Quantitative frameworks—such as California SB 53 — Transparency in Frontier Artificial Intelligence Act (TFAIA), the Artificial Intelligence Safety Measures Act, and the Transparency in Frontier Artificial Intelligence Act—establish explicit numerical thresholds, requiring a foreseeable, material risk of causing death or serious injury to 50 or more people or property damage exceeding $1 billion from a single incident. In contrast, qualitative frameworks—including The Great American AI Act and An Act to amend the general business law, in relation to transparency and safety requirements for developers of artificial intelligence frontier models—omit hard numbers entirely. Instead, they define catastrophic risk through broader, thematic harms like critical infrastructure failure, threats to national security, or general economic and public safety disruptions.

Synthesised from the 5 statutory definitions below. Each one is quoted in full, with its source.

How jurisdictions differ

What is distinctive in each definition, binding law in force first.

Jurisdiction	Instrument	Status	What is distinctive
California, United States	California SB 53 — Transparency in Frontier Artificial Intelligence Act (TFAIA)	In Force	Specifies exact thresholds of 50 deaths/injuries or $1 billion damage, lists three precise scenarios including CBRN weapons and excludes public information, lawful federal actions, or non-material model contributions.
New York, United States	An Act to amend the general business law, in relation to transparency and safety requirements for developers of artificial intelligence frontier models	In Force	Broadly limits term to potential harms from misuse, malfunction, or unintended properties of frontier models, focusing on public safety, economic stability, or national security without thresholds or scenarios.
Illinois, United States	Artificial Intelligence Safety Measures Act	Adopted	Uses $1,000,000,000 threshold explicitly and ties risk to developer's activities materially contributing to harms, including misuse for weapons, cyberattacks, or control evasion.
United States	The Great American AI Act	Proposed	Broadly describes harms like critical infrastructure failure, mass casualties, or economic disruption without thresholds, scenarios, or exclusions.
California, United States	Transparency in Frontier Artificial Intelligence Act	In Force	Summarizes scenarios including death/injury to 50+ or $1 billion damages, CBRN assistance, autonomous crimes/cyberattacks, or control evasion, without detailing exclusions.

What it triggers

In California under SB 53, developers of frontier AI models must publish safety frameworks.source
Under New York S8828, developers of frontier AI models face transparency and safety requirements including risk management and incident reporting.
Under the Illinois Artificial Intelligence Safety Measures Act, frontier developers must conduct independent third-party audits for models presenting such risks.source

Definitions (5)

A foreseeable, material risk of causing death or serious injury to 50 or more people, or property damage of $1 billion or more, from a single incident involving (1) expert‑level assistance in creating/releasing/using CBRN weapons, (2) conduct without meaningful human oversight constituting cyberattacks or serious crimes, or (3) the model evading developer control; excludes harms from public information, lawful federal actions, or where the model did not materially contribute.

California SB 53 — Transparency in Frontier Artificial Intelligence Act (TFAIA)Definition 1 of 5

A foreseeable and material risk that a frontier developer's activities related to a frontier model could materially contribute to the death or serious injury of more than 50 people, or over $1,000,000,000 in property damage or loss from a single incident. This includes misuse for weapons, cyberattacks, or evasion of control.

Artificial Intelligence Safety Measures ActDefinition 2 of 5

Refers to potential harms from AI systems that could lead to widespread severe negative consequences, such as critical infrastructure failure, mass casualties, or significant economic disruption.

The Great American AI ActDefinition 3 of 5

As defined by the Act, catastrophic risk includes scenarios where an AI model could cause death or serious injury to 50 or more individuals, or result in damages exceeding $1 billion. It also covers situations where an AI system provides expert assistance in creating or releasing chemical, biological, radiological, or nuclear weapons, autonomously commits major crimes or cyberattacks, or evades developer control.

Transparency in Frontier Artificial Intelligence ActDefinition 4 of 5

Potential harms that could result from the misuse, malfunction, or unintended emergent properties of frontier AI models. This includes risks to public safety, economic stability, or national security.

An Act to amend the general business law, in relation to transparency and safety requirements for developers of artificial intelligence frontier modelsDefinition 5 of 5

Not to be confused with

High‑risk AI

High-risk AI refers to uses with potential for severe or large-scale harm, while catastrophic risk definitions specify thresholds like 50 deaths or $1 billion damage or particular severe scenarios.

Risk

Risk is the general combination of harm probability and severity, whereas catastrophic risk definitions narrow to extreme thresholds or scenarios like mass casualties or CBRN weapons.

Questions and answers

What counts as catastrophic risk under California SB 53?

It includes foreseeable risks of 50+ deaths/injuries or $1 billion damage from CBRN weapons, autonomous cyberattacks/crimes, or model evasion, excluding public info, lawful federal acts, or non-material contributions.

Does New York law define catastrophic risk with specific thresholds?

No, it broadly covers potential harms from misuse, malfunction, or emergent properties affecting public safety, economic stability, or national security.

What obligations arise for catastrophic risks in Illinois?

Frontier developers must perform independent third-party audits for models with such risks, defined by material contributions to 50+ deaths/injuries or $1B damage.