Technical

Model Drift

Degradation or change in model performance over time.

How the laws define this

Regulators and standard-setting bodies agree that model drift centers on the degradation or change of an artificial intelligence model's performance or behavior over time due to evolving underlying data, context, or environment shifts. The key divergence lies in whether the concept is strictly a descriptive measure of performance or extends to mandate operational remedies and encompass fairness metrics. ISO/IEC 23053:2022 - Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML) takes the narrowest approach, defining drift purely as performance degradation caused by data distribution changes without detailing required actions. In contrast, the Implementation Guide for Managers of Artificial Intelligence Systems (Innovation, Science and Economic Development Canada) and the NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers explicitly connect drift to mandatory corrective responses such as continuous monitoring, revalidation, retraining, or mitigation. Furthermore, the Nigerian Bar Association Guidelines for the Use of Artificial Intelligence in the Legal Profession uniquely broadens drift to include alterations in fairness characteristics.

Synthesised from the 4 statutory definitions below. Each one is quoted in full, with its source.

How jurisdictions differ

What is distinctive in each definition, binding law in force first.

Jurisdiction	Instrument	Status	What is distinctive
Canada	Implementation Guide for Managers of Artificial Intelligence Systems (Innovation, Science and Economic Development Canada)	In Force	Explicitly identifies shifts in input data distributions, environment, or use patterns as causes and highlights revalidation, retraining, or other mitigation actions.
United States	NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers	In Force	Specifies changes in underlying data or relationships as causes and emphasizes continuous monitoring and validation as responses.
ISO	ISO/IEC 23053:2022 - Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)	In Force	Limits causes solely to changes in the underlying data distribution, omitting behavioral changes or specific mitigation measures.
Nigeria	Nigerian Bar Association Guidelines for the Use of Artificial Intelligence in the Legal Profession	Adopted	Includes changes in behavior, accuracy, or fairness characteristics and stresses periodic reassessment, testing, and remediation.

What it triggers

In Canada, the Implementation Guide for Managers of Artificial Intelligence Systems recommends revalidation, retraining, or other mitigation actions.source
The NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers calls for continuous monitoring and validation.source
ISO/IEC 23053:2022 sets out a framework addressing degradation due to data distribution changes within the AI lifecycle.source
The Nigerian Bar Association Guidelines for the Use of Artificial Intelligence in the Legal Profession recommend periodic reassessment, testing, and remediation.source

Definitions (4)

The phenomenon where an AI model’s behaviour, accuracy or fairness characteristics change over time due to evolving data, context or usage patterns, necessitating periodic reassessment, testing and remediation to maintain reliability.

Nigerian Bar Association Guidelines for the Use of Artificial Intelligence in the Legal ProfessionDefinition 1 of 4

The phenomenon where an AI model's performance degrades or its behavior changes over time due to shifts in input data distributions, environment, or use patterns, prompting revalidation, retraining, or other mitigation actions.

Implementation Guide for Managers of Artificial Intelligence Systems (Innovation, Science and Economic Development Canada)Definition 2 of 4

The degradation of a model's performance over time due to changes in the underlying data distribution.

ISO/IEC 23053:2022 - Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)Definition 3 of 4

The degradation of a model's performance over time, often due to changes in the underlying data or relationships, necessitating continuous monitoring and validation.

NAIC Model Bulletin on the Use of Artificial Intelligence Systems by InsurersDefinition 4 of 4

Not to be confused with

Model robustness

Model robustness concerns resistance to errors, attacks, and unexpected inputs, while model drift addresses performance degradation from data or environmental shifts over time.

Model lifecycle

Model lifecycle covers all phases from design to decommissioning, whereas model drift specifically refers to performance changes occurring during deployment.

Questions and answers

What causes model drift according to these definitions?

Causes include shifts in input data distributions, environment, or use patterns (Canada); changes in underlying data or relationships (US NAIC); changes in data distribution (ISO); and evolving data, context, or usage patterns affecting behavior, accuracy, or fairness (Nigeria).

What actions are recommended for model drift?

Recommendations involve revalidation, retraining, or mitigation (Canada); continuous monitoring and validation (US NAIC); framework guidance across the AI lifecycle (ISO); and periodic reassessment, testing, and remediation (Nigeria).

Does model drift apply to all AI systems or specific sectors?

It applies to AI systems managed in Canada generally; to AI systems used by US insurers; within machine learning systems under the ISO framework; and to AI used by lawyers in Nigeria.