European Union - Text and Data Mining Guidelines (2024/1689)

Overview

The European Commission is running a stakeholder-driven process to identify and agree on machine-readable protocols that allow rightsholders to reserve rights and opt out from Text and Data Mining (TDM) uses, in the specific context of the obligations set by the Artificial Intelligence Act (Regulation (EU) 2024/1689) and the Copyright chapter of the General-Purpose AI Code of Practice (GPAI CoP). The initiative is implemented through a public consultation and an information session organised by the European AI Office, with technical support from the European Union Intellectual Property Office (EUIPO) study on generative AI and copyright. The consultation invites rightsholders, providers, standardisation bodies and civil society to comment on candidate TDM opt-out solutions (including robots.txt variants, manifest files, metadata schemes and provenance standards), and to express interest in participating in follow-up workshops. The aim is to publish a list of generally-agreed opt-out protocols that GPAI Code signatories and other providers may rely on to demonstrate compliance with Article 53(c) of the AI Act.

Definitions

Key terms used in the process include: "Text and Data Mining (TDM)" — automated techniques to analyze large bodies of text or data; "Reservation of rights" — a rightsholder declaration to withhold authorisation for TDM (Article 4(3) of Directive (EU) 2019/790); "GPAI Code of Practice" — the General-Purpose AI Code of Practice (Copyright chapter) approved by the Commission as an adequate voluntary compliance pathway; "GPAI provider" — a provider of general-purpose AI models within the meaning of the AI Act; "Machine-readable opt-out protocol" — an agreed technical standard or convention (protocol, metadata field, robots-like file, manifest or API) that signals a rights reservation in a format that automated crawlers, indexers, or training pipelines can detect and respect. For legal definitions and obligations, refer to the DSM Directive (Directive (EU) 2019/790) and the AI Act text on EUR-Lex.

Governance and Institutional Framework

Implementation is led by the European Commission's AI Office (European AI Office) with operational support from DG CONNECT and the EUIPO. The AI Act vests the Commission (AI Office) with supervisory powers for compliance by providers of general-purpose AI models, including powers to request information, conduct evaluations and, where warranted, impose fines under Article 101 of the AI Act. The EUIPO contributes technical and sectoral expertise on copyright, having published a comprehensive study on GenAI and copyright issues. National competent authorities remain responsible for enforcement of many AI Act provisions and for setting national penalty regimes in line with Article 99. The GPAI Code of Practice, as a voluntary instrument, functions as an approved code under Article 56 of the AI Act: adherence to an approved code can be used by providers to demonstrate compliance until harmonised standards are published. The process uses EUSurvey for formal stakeholder inputs and Commission-facilitated workshops to reach broad consensus among rights holders, providers and standards bodies.

Key Focus Areas

The initiative focuses on three interlinked domains: legal clarity, technical feasibility and operational uptake. Legally, the project seeks to operationalise Article 53(c) of the AI Act (providers must identify and comply with reservations expressed under Article 4(3) of Directive (EU) 2019/790) by defining acceptable machine-readable opt-out methods. Technically, the Commission and EUIPO have catalogued candidate methods in the EUIPO study: robots.txt-like mechanisms, manifest files at dataset level, C2PA provenance metadata, content labels, centralized registries, signature-based manifests and API-based consent endpoints; each option has trade-offs on discoverability, reliability, spoofing risk and cross-border applicability. Operational uptake requires solutions to be implementable at scale, interoperable with existing web infrastructure, resilient to circumvention, and compatible with rights-management and licensing markets. Accessibility for small rights holders and international interoperability (non-EU websites and datasets) are flagged as critical concerns. The consultation solicits evidence on technical constraints, adoption barriers, verification methods and the relative merits of distributed versus centralized opt-out registries.

Implementation Framework

The Commission's two-track approach combines an open consultation (EUSurvey) to collect technical and policy evidence with follow-up facilitated workshops where stakeholders (default-invited: GPAI Code signatories) can negotiate and reach broad consensus on a short-list of opt-out protocols. The EUIPO provides technical briefings and a study that maps each option's capability to express TDM reservations, its integration with existing standards (IETF robots.txt, C2PA, W3C metadata), and mitigation of spoofing or abuse. After workshops, the Commission will publish a list of generally-agreed machine-readable opt-out protocols and guidance explaining expected provider behaviour, including how GPAI providers should detect and honor opt-outs in their data collection, data curation and training pipelines. The list will be maintained and reviewed at least every two years. Implementation materials (templates, detection guidelines, sample manifests and QA checks) will be provided by the AI Office and EUIPO to support adoption.

Monitoring and Evaluation

Monitoring combines periodic reporting, stakeholder feedback and technical evaluations. The AI Office is empowered by the AI Act to request information and to evaluate general-purpose models; Article 101 grants the Commission sanctioning powers for provider non-compliance. The Commission will assess uptake metrics (number of rights-reservations registered, number of GPAI providers adopting detection workflows, results from interoperability tests) and report publicly. The EUIPO will support technical testing (sample crawlers and validators) and may propose further standardisation work. Reviews are scheduled at least biennially in coordination with Code updates; the AI Act mandates Commission evaluations (Article 112) and a four-year review cycle for the Regulation. Stakeholder inputs from the EUSurvey and workshop outputs will feed into the monitoring dashboard and updates to the agreed list of protocols.

Penalties, Liability, and Appeals

Sanctions for non-compliance fall under the AI Act enforcement regime. Member States must adopt effective, proportionate and dissuasive penalties for infringements (Article 99). For providers of general-purpose AI models, the Commission may impose fines up to 3% of annual worldwide turnover or EUR 15,000,000 for intentional or negligent failures to comply with obligations including to respond to Commission information requests or to comply with measures requested under Article 93 (see Article 101). Administrative fines for other infringements are set out in Article 99 (up to EUR 15,000,000 or 3% of turnover for a set of specified provider obligations; up to EUR 35,000,000 or 7% of turnover for the prohibition of AI practices). The AI Act also guarantees procedural safeguards, right to be heard and judicial review by the Court of Justice of the EU for Commission decisions under Article 101. Liability for copyright infringement (civil remedies) remains governed by EU and national copyright law; the protocol work seeks to reduce inadvertent infringement risks by improving discoverability of opt-outs.

Relationship to Other Instruments

The consultation and protocol work explicitly bridge several EU instruments: the AI Act (Regulation (EU) 2024/1689) sets provider obligations on training data transparency and respect for copyright reservations; Directive (EU) 2019/790 (DSM Directive) establishes the legal basis for TDM exceptions and rights reservations (Article 3 and Article 4). The EUIPO study provides technical and legal analysis linking these instruments. The GPAI Code of Practice is recognised by the Commission as an adequate voluntary pathway for compliance with Article 53 obligations and is therefore recognised within the AI Act implementation ecosystem (Article 56 on codes of practice). The activity also interacts with standards work (IETF robots.txt, W3C, C2PA, ETSI) and with national copyright transpositions of the DSM Directive. The Commission's outputs (guidelines, templates and the agreed opt-out list) will not alter substantive copyright rights but will seek to make expression and detection of reservations practicable in real-world data pipelines.

International Alignment

Because AI training pipelines and datasets are global, the Commission emphasises international alignment. The GPAI Code of Practice is an international instrument with governmental signatories and the Commission is coordinating with GPAI stakeholders to encourage uptake beyond the EU. The EU will engage standards organisations (IETF, W3C, C2PA, ETSI) and international partners through GPAI workstreams and bilateral regulatory dialogues to encourage interoperable opt-out mechanisms. The EUIPO study warns that non-EU actors and jurisdictions with different copyright exceptions may limit the unilateral effectiveness of EU opt-outs, so international engagement and standardisation are central to achieving broad technical uptake and minimizing circumvention risks.

Implementation Timeline

Sources and References