Use-case guide

AI in Children's Products & Family-Facing AI

AI for children is governed by the strictest privacy and safety regimes anywhere — separate from the Education guide because this covers consumer-product use, not curricular use. The core hooks: COPPA (US), GDPR-K (EU), the UK Age-Appropriate Design Code, California's Age-Appropriate Design Code (AB 2273), and a wave of state laws (NY SAFE for Kids Act, Maryland AADC, Connecticut). Plus FTC enforcement that's intensified since 2023 — Epic Games' $520M COPPA settlement set a new ceiling. AI features layered on top: chatbots aimed at kids, recommendation algorithms, biometric data collection from age verification. The compliance design choice — gate or strip — drives the entire product.

For: Toy makers, EdTech vendors selling to families, children's app developers, family-streaming platforms, child-safety compliance officers

What's at stake

COPPA targets under-13s and is being aggressively enforced

Verified parental consent before data collection from under-13s. The FTC's 2023 COPPA Rule update is in active rulemaking; the proposed amendments would extend to AI-personalisation features by default. Epic Games settlement ($275M COPPA + $245M dark-pattern) established the modern ceiling.

California AADC took effect 2024

California AB 2273 (Age-Appropriate Design Code) requires Data Protection Impact Assessments for any feature 'likely to be accessed by children'. Default-on privacy protections. Enforcement by California AG with civil penalties up to $7,500/affected child.

UK AADC + new EU rules expand the perimeter

ICO's UK AADC (in force 2021) requires 15 standards including data minimisation, no nudge-to-share, no profiling by default. The EU DSA has minor-specific provisions (no profiled ads for under-18s). The EU AI Act bans certain emotion-recognition uses on children.

Generative AI + minors triggers ABA / state-AG attention

AI chatbots that maintain emotionally-attentive 'companion' relationships with minors are under direct FTC and state-AG scrutiny — Character.AI litigation and Replika EU bans are the live examples.

Regulations that apply

COPPA + FTC enforcement

Law
→

Verified parental consent for data collection from under-13s. AI personalisation, recommendation, and avatar features are in scope. FTC enforcement-by-settlement is the active mechanism.

Where in the text: 15 U.S.C. §§ 6501-6506; 16 C.F.R. Part 312.

California Age-Appropriate Design Code (AB 2273)

Law
→

DPIA required for any service likely to be accessed by children. Default-on privacy protections. Effective for new features as of mid-2024.

Where in the text: Cal. Civ. Code §§ 1798.99.28-1798.99.40.

EU AI Act (Article 5 + Article 50)

Law
→

Bans emotion-recognition AI in 'educational institutions' (Article 5(1)(f)); transparency duties for child-facing chatbots (Article 50). Manipulative AI targeting child vulnerability is prohibited (Article 5(1)(b)).

Where in the text: Article 5(1)(b), 5(1)(f); Article 50.

UK Age-Appropriate Design Code

Law
→

15 standards for online services likely to be accessed by children. Data minimisation, no profiling-by-default, no nudge patterns. ICO has audit + fine authority.

Where in the text: UK ICO Age-Appropriate Design Code (2020); UK Data Protection Act 2018 Part 5.

Do

  • ✓Design age-verification with proportionality — over-collection of biometric or government-ID data to verify age can itself be a violation. Self-declared age with a strong default-down policy is often the lowest-risk path.
  • ✓Run a DPIA per California AADC standards on every feature likely to be accessed by children before launch — this is now a documented expectation.
  • ✓Default to data minimisation for under-18s: no behavioural advertising, no profiling, no engagement-maximisation features.
  • ✓Test your AI features for child-safety failure modes (grooming responses, harmful content, age-inappropriate recommendations) on red-team data sets BEFORE shipping.
  • ✓Maintain a parental dashboard that shows what AI is doing — required by several state AADC laws and de-facto required by FTC since Epic.

Don't

  • ✗Don't ship a 'companion' chatbot for minors without robust safety guardrails, distress-detection, and human-escalation. Character.AI litigation is the live cautionary tale.
  • ✗Don't use parental-consent flows that are pre-checked, hidden, or buried — dark-pattern enforcement carried the second-largest Epic penalty.
  • ✗Don't deploy emotion-recognition AI in an EU children's educational product. Article 5(1)(f) of the AI Act bans it.
  • ✗Don't sell or share children's data with third-party advertisers, including 'lookalike' modelling. This is the COPPA tripwire that produces seven-figure settlements.
  • ✗Don't claim AI-feature 'safe for kids' without a documented test methodology. The FTC has called out general 'safety' claims as deceptive when not substantiated.

Also worth knowing

If you operate an EdTech platform that's also a consumer product (sold to families AND schools): you need BOTH the school-FERPA contract architecture AND COPPA verified-parental-consent flows. The two regimes don't substitute. For YouTube Kids-style platforms: the FTC's 2019 YouTube COPPA settlement ($170M) established that 'general-audience' platform claims don't shield child-directed content from COPPA enforcement.

Want a tailored answer?

The wizard takes your jurisdiction, AI use case, and data types and gives you the top-3 regulations to focus on — in 60 seconds.

Start the wizard →

Educational guide. Not legal advice. For specific compliance decisions, consult qualified counsel in the relevant jurisdiction.

Note: this guide was drafted with AI assistance — Anthropic Claude.