NIST Generative AI Risk Profile

Overview

The NIST AI 600-1, officially titled "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile," serves as a crucial companion resource to the broader NIST AI Risk Management Framework (AI RMF 1.0). Published by the National Institute of Standards and Technology (NIST) on July 26, 2024, this document specifically addresses the unique and exacerbated risks associated with generative artificial intelligence (GAI) technologies. Its development was undertaken in part to fulfill the directives outlined in President Biden's Executive Order 14110 on Safe, Secure, and Trustworthy Artificial Intelligence, underscoring a national commitment to responsible AI development and deployment. The Generative AI Profile is designed for voluntary use, aiming to equip organizations with the necessary tools and insights to incorporate trustworthiness considerations into the entire lifecycle of GAI products, services, and systems, from design and development to use and evaluation.

This profile is cross-sectoral, meaning its guidance is applicable across various industries and use cases that leverage generative AI, such as large language models (LLMs) and cloud-based AI services. It outlines a structured approach to identifying, assessing, and mitigating GAI-specific risks, helping organizations balance innovation with responsibility. By adapting the core functions of the AI RMF—Govern, Map, Measure, and Manage—to the context of generative AI, NIST AI 600-1 provides actionable steps for organizations to manage challenges like hallucinations, data leakage, intellectual property concerns, and harmful bias. The framework emphasizes a proactive and continuous risk management culture, encouraging integration into existing governance, risk, and security programs rather than being treated as a standalone compliance checklist.

Definitions

For the purposes of the NIST AI 600-1 Generative AI Profile, several key terms are central to understanding its guidance. "Artificial Intelligence Risk Management Framework" (AI RMF) refers to the overarching voluntary framework published by NIST in January 2023, which provides a flexible, risk-based approach to managing AI across its lifecycle, built on four core functions: Govern, Map, Measure, and Manage. The AI RMF aims to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. The "Generative Artificial Intelligence Profile" (NIST AI 600-1) is a companion resource that specifically adapts the AI RMF to the unique risks and challenges presented by generative AI systems, which are capable of creating new content, code, or designs autonomously.

The profile also defines or addresses specific risks inherent to generative AI. "Confabulation," often referred to as "hallucinations," describes the phenomenon where an AI system generates inaccurate, false, or misleading information with high confidence, despite lacking factual basis. "Harmful Bias and Homogenization" relates to the presence of stereotypes, unfair outcomes, or a lack of diversity in AI outputs, often stemming from non-representative training data. "Data Privacy" concerns the unauthorized leakage, exposure, or de-anonymization of sensitive personal information by GAI systems. "Intellectual Property" risks involve the potential for GAI to infringe on copyrights, patents, or trade secrets, or to misuse personal identity, likeness, or voice without permission. These definitions are critical for organizations to accurately identify and categorize the risks they face when developing or deploying generative AI.

Governance and Institutional Framework

The governance and institutional framework for NIST AI 600-1 is rooted in the broader mission of the National Institute of Standards and Technology (NIST) to develop measurements, technology, tools, and standards that advance reliable, safe, transparent, explainable, privacy-enhanced, and fair artificial intelligence. As a non-regulatory agency of the U.S. Department of Commerce, NIST’s role is to provide voluntary guidance and frameworks, rather than binding regulations. The AI RMF, and by extension its Generative AI Profile, is intended for voluntary adoption by organizations in both the public and private sectors to improve their capacity to manage AI-related risks. This voluntary approach allows for flexibility, enabling organizations to tailor the framework's implementation to their specific operational context, risk tolerance, and resources.

The NIST AI RMF, which AI 600-1 profiles, is structured around four core functions: Govern, Map, Measure, and Manage. The "Govern" function is foundational, establishing a comprehensive risk management culture within an organization. It involves defining organizational accountability, aligning with legal and regulatory requirements, setting AI usage policies, and establishing clear escalation paths for AI incidents. For generative AI, this specifically means establishing policies and oversight mechanisms that address the unique risks of GAI, ensuring transparent decision-making, and fostering a diverse workforce involved in AI development and deployment. NIST also supports the Trustworthy and Responsible AI Resource Center (AIRC), which facilitates the implementation of and international alignment with the AI RMF, providing a platform for sharing use cases and resources. This institutional framework emphasizes collaboration, transparency, and continuous improvement in AI risk management practices.

Key Focus Areas

The NIST AI 600-1 Generative AI Profile identifies twelve specific risks that are either unique to or significantly exacerbated by the development and use of generative AI systems. These risks are categorized to provide a structured approach for organizations to understand and mitigate potential harms. One major category includes technical or model risks, which are issues directly related to the design, operation, and output of AI systems. This encompasses "Confabulation" (hallucinations), where GAI produces confidently stated but erroneous information; "Dangerous or Violent Recommendations," where AI facilitates access to or creation of harmful content; "Data Privacy" concerns, involving unauthorized data leakage or exposure of sensitive information; and "Harmful Bias and Homogenization," arising from non-representative training data leading to unfair or stereotypical outputs. "Value Chain and Component Integration" also falls here, addressing risks from non-transparent third-party components or improperly obtained data.

Another critical focus area is "Misuse by Humans," which addresses how individuals might exploit GAI technologies. This includes risks related to "CBRN Information or Capabilities," where GAI could enable easier access to knowledge for creating chemical, biological, radiological, or nuclear weapons; "Obscene, Degrading, and/or Abusive Content," concerning the production and dissemination of harmful or illegal imagery; and "Intellectual Property" issues, such as copyright infringement or unauthorized use of personal identity. Finally, "Ecosystem / Societal Risks" pertain to broader impacts. These include "Environmental Impacts" from resource-intensive AI operations; "Information Security" risks, such as increased vulnerability to cyberattacks facilitated by AI; and "Human-AI Configuration" issues, which encompass problems arising from human interaction with AI, including overreliance, automation bias, or inappropriate anthropomorphizing. By detailing these diverse risk areas, NIST AI 600-1 provides a comprehensive roadmap for organizations to anticipate and address the multifaceted challenges of generative AI.

Implementation Framework

The implementation framework for NIST AI 600-1 is built upon the foundational four core functions of the NIST AI Risk Management Framework (AI RMF 1.0): Govern, Map, Measure, and Manage. Organizations are encouraged to integrate the Generative AI Profile into their existing governance, risk, and security programs, rather than treating it as a separate compliance burden. The "Govern" function involves establishing a culture of AI risk management, defining accountability, and setting clear policies for GAI use. This includes aligning with legal and regulatory requirements, establishing clear escalation paths for incidents, and setting thresholds for acceptable and unacceptable risks associated with generative AI. Effective governance also emphasizes transparent policies, clear accountability structures, and robust engagement with relevant AI actors throughout the GAI lifecycle.

The "Map" function focuses on establishing the context for framing GAI risks. This involves documenting AI use cases, intended purposes, stakeholders, and data sources, with a specific emphasis on explicitly mapping generative AI risks such as misinformation, bias, intellectual property concerns, and model supply chain dependencies. The "Measure" function is dedicated to assessing identified risks, which includes testing GAI systems for issues like hallucinations, bias, privacy leaks, and environmental impacts. This often involves both internal evaluations and external red-teaming exercises to thoroughly vet the system's trustworthiness characteristics. Finally, the "Manage" function involves allocating resources to address mapped and measured risks, implementing response and recovery plans, and communicating about incidents. This includes implementing safeguards like content provenance, robust incident disclosure mechanisms, fallback plans for third-party dependencies, and structured decommissioning processes for GAI systems. The integrated application of these four functions ensures a holistic and continuous approach to managing generative AI risks.

Monitoring and Evaluation

Monitoring and evaluation are integral components of the NIST AI 600-1 framework, primarily addressed within the "Measure" function of the underlying AI RMF. This function is designed to ensure that organizations can effectively assess the risks associated with their generative AI systems and determine the efficacy of their mitigation strategies. It involves identifying and applying appropriate methods and metrics to evaluate AI systems for trustworthiness characteristics such as validity, reliability, safety, security, fairness, and privacy-enhancement. For generative AI, this means conducting rigorous testing to detect and quantify issues like confabulation (hallucinations), harmful bias, data privacy leaks, and the potential for generating dangerous or abusive content. Evaluation methods can include both automated tools and human-in-the-loop assessments, as well as red-teaming exercises to proactively identify vulnerabilities and potential misuse.

Beyond initial assessments, the "Measure" function also mandates the establishment of mechanisms for tracking identified AI risks over time. This continuous monitoring is crucial for generative AI, as models can drift, new vulnerabilities may emerge, or their impacts can change with evolving usage patterns and data inputs. Organizations are expected to gather and assess feedback about the efficacy of their measurement activities, allowing for iterative improvements to their risk management processes. This feedback loop ensures that monitoring and evaluation efforts remain relevant and effective in an rapidly evolving AI landscape. By systematically measuring and tracking risks, organizations can maintain a dynamic understanding of their GAI systems' performance and trustworthiness, enabling timely adjustments and fostering ongoing confidence in their AI deployments.

Penalties, Liability, and Appeals

As a voluntary framework and guidance document, NIST AI 600-1 does not prescribe direct penalties, liability provisions, or appeal mechanisms in the manner of legally binding regulations. Its purpose is to provide best practices and a structured approach for organizations to manage the risks associated with generative AI. Therefore, non-adherence to the guidelines outlined in NIST AI 600-1 does not automatically trigger legal sanctions or fines from a regulatory body. However, while voluntary, the framework's adoption is increasingly seen as crucial for organizations, especially those handling sensitive data or making high-stakes decisions with AI. Failure to implement robust AI risk management practices, even in the absence of direct regulatory penalties, can expose organizations to significant indirect consequences.

These indirect consequences can include substantial reputational damage, loss of public trust, and potential legal liabilities under existing laws (e.g., consumer protection, data privacy, intellectual property, or anti-discrimination laws) if generative AI systems cause harm. For instance, if a GAI system produces biased outputs leading to discrimination, or infringes on intellectual property rights, organizations could face lawsuits, regulatory investigations under other statutes, and significant financial repercussions. Adopting NIST AI 600-1 can serve as evidence of an organization's due diligence and commitment to responsible AI, potentially mitigating the severity of such outcomes in legal or public scrutiny. The framework helps organizations create a governance program that balances innovation with responsibility, thereby reducing exposure to legal and reputational harm and aligning with widely recognized standards.

Relationship to Other Instruments

NIST AI 600-1 is explicitly designed as a "companion resource" and a "cross-sectoral profile" for the foundational NIST AI Risk Management Framework (AI RMF 1.0). The AI RMF 1.0, released in January 2023, provides the overarching voluntary, flexible, and risk-based approach to managing AI across its lifecycle, built on the four core functions of Govern, Map, Measure, and Manage. The Generative AI Profile then adapts these general principles and functions to the specific risks and challenges presented by generative AI technologies. It extends the AI RMF by providing detailed insights into how risk can be managed for GAI as a technology, and across various stages of the AI lifecycle, particularly for activities common across sectors like the use of large language models. Therefore, AI 600-1 should be used in conjunction with, and layered on top of, the broader AI RMF 1.0 for comprehensive generative AI risk management.

Furthermore, the development of NIST AI 600-1 was undertaken to fulfill specific mandates from President Biden's Executive Order (EO) 14110 on Safe, Secure, and Trustworthy Artificial Intelligence, issued on October 30, 2023. The EO directed NIST to develop guidance for generative AI, and this profile represents a key deliverable in response to that directive. This connection highlights the profile's alignment with national strategic priorities for AI governance in the United States. NIST also collaborates with other entities, such as the National Cybersecurity Center of Excellence (NCCOE), to develop AI Community Profiles that adapt existing frameworks like the Cybersecurity Framework and Privacy Framework to AI, further demonstrating its interconnectedness within a broader ecosystem of guidance. The framework is also intended to build on, align with, and support AI risk management efforts by others, including potential international standards, showcasing its role within a wider landscape of AI governance instruments.

International Alignment

While NIST AI 600-1 is a U.S.-specific document issued by the National Institute of Standards and Technology, its underlying principles and the broader NIST AI Risk Management Framework (AI RMF) are developed with an eye towards international alignment and collaboration. NIST actively engages with international partners and stakeholders to foster a common understanding and approach to AI risk management. The AI RMF itself is intended to build on, align with, and support AI risk management efforts by others globally, and NIST has even published Japanese and Arabic translations of the main AI RMF document to facilitate broader understanding and adoption. This commitment to international cooperation is driven by the understanding that AI development and deployment are global endeavors, and harmonized approaches to trustworthiness and risk management are beneficial for all.

The Generative AI Profile, by providing a detailed methodology for addressing the specific risks of generative AI, contributes to this global dialogue by offering a robust, practical framework that can inform similar efforts in other jurisdictions. Although AI 600-1 focuses on the United States, the nature of generative AI risks—such as hallucinations, data privacy, and intellectual property issues—are universal concerns that transcend national borders. Therefore, the structured approach and suggested actions within NIST AI 600-1 can serve as a valuable reference for international organizations and governments developing their own guidelines for responsible generative AI. The NIST Trustworthy and Responsible AI Resource Center also plays a role in facilitating international alignment with the AI RMF, providing a platform for sharing insights and best practices that can contribute to a globally coherent approach to AI safety and trustworthiness.

Implementation Timeline

Sources and References