Emergent Capabilities

Emergent Capabilities (or emergent behaviors) refer to abilities or behaviors that appear in AI systems—particularly large language models and foundation models—that were not explicitly programmed, trained for, or anticipated by developers.

Characteristics of Emergence:

• Scale-Dependent: Often only appear at certain model sizes or compute thresholds
• Unpredictable: Cannot be reliably predicted from smaller-scale experiments
• Non-Linear: May appear suddenly with incremental increases in scale
• Diverse: Can include both beneficial and harmful capabilities

Regulatory Significance:

• Risk Assessment Challenges: Traditional risk assessment may not capture emergent risks
• GPAI with Systemic Risk: The EU AI Act's provisions for GPAI models with systemic risk are partly motivated by concerns about emergent capabilities
• Continuous Monitoring: Requires ongoing evaluation even after deployment
• Pre-deployment Testing: Comprehensive red-teaming and evaluation needed

Examples of Emergent Capabilities:

• In-context learning (learning from examples in prompts)
• Chain-of-thought reasoning
• Code generation and execution
• Multilingual transfer
• Potentially harmful capabilities (deception, manipulation)

Policy Implications: The unpredictable nature of emergent capabilities supports arguments for:

• Compute thresholds as regulatory triggers (e.g., 10^25 FLOPS in EU AI Act)
• Mandatory pre-deployment evaluations for frontier models
• Ongoing monitoring and incident reporting requirements
• International coordination on frontier AI safety