Misuse Risk
The potential for AI systems to be intentionally used for harmful purposes beyond their intended function.
Definition
Misuse Risk refers to the potential for AI systems, models, or capabilities to be deliberately used for purposes that cause harm, violate rights, or undermine safety—distinct from unintentional malfunctions or errors. It encompasses both malicious use by bad actors and inappropriate use by authorized users.
Regulatory Recognition: The EU AI Act addresses misuse risk through multiple mechanisms including prohibited practices (Article 5), high-risk system requirements, and obligations for GPAI models with systemic risk. Providers must assess reasonably foreseeable misuse when conducting risk assessments.
Categories of Misuse:
- Malicious use: Deliberate harm by bad actors (cyberattacks, fraud, weapons)
- Unauthorized use: Use beyond licensed scope or terms of service
- Inappropriate use: Authorized users applying AI to unsuitable contexts
- Manipulation: Exploiting AI systems through adversarial inputs
Risk Assessment Requirements:
- EU AI Act Article 9 requires identification of "reasonably foreseeable misuse"
- GPAI providers must evaluate misuse risks under Article 55
- NIST AI RMF MAP function includes misuse scenario analysis
Mitigation Strategies: Access controls, usage monitoring, terms of service, technical safeguards (output filtering, watermarking), red teaming to identify misuse vectors, and incident reporting mechanisms.
Related concepts: Dual-Use AI, AI Safety, Red Teaming, Adversarial Testing, Risk Management System
Sources
- •NIST AI RMF
- •AI Safety Standards
Related Terms
Criminal misuse
Uses of AI that enable or materially facilitate unlawful activity causing public harm....
high-risk use
Uses with significant impacts on health, safety or fundamental rights....
high-risk uses
AI uses with significant potential harm to rights, safety or wellbeing....
user harm
Harms to users including privacy invasion, defamation, discrimination, illegal facilitation....
Catastrophic Risk
Risk of severe, large-scale, potentially irreversible negative impacts from AI systems on health, safety, or fundamental rights....