# regulations.ai — crawler policy # # This file is kept deliberately IN SYNC with ALLOWED_BOTS in middleware.ts. # The middleware hard-403s any user-agent containing "bot", "crawler" or # "spider" unless it is named there, so a crawler allowed here but missing # there gets a 403 instead of the page — which is worse than an honest # Disallow. If you add an agent to one list, add it to the other. # # Policy: this is a free, public reference on AI regulation. Being cited by # search engines and AI assistants IS the distribution model, so reputable # search, answer-engine and AI-training crawlers are all allowed. SEO/backlink # scrapers (Ahrefs, Semrush, MJ12, BLEXBot, DotBot) are not — they consume # crawl budget and origin cost and return nothing. # # Last reviewed: 2026-09-29 -- /auth/ disallowed for every crawler: each gated page links to # /auth/login?redirect=, so crawlers saw thousands of login URLs (Googlebot timed out # on them 18 times on 2026-09-29). Login pages have nothing to index. # ---------------------------------------------------------------- search ---- User-agent: Googlebot Allow: / Disallow: /auth/ User-agent: Googlebot-Image Allow: / Disallow: /auth/ User-agent: Googlebot-News Allow: / Disallow: /auth/ User-agent: Googlebot-Video Allow: / Disallow: /auth/ User-agent: Google-InspectionTool Allow: / Disallow: /auth/ User-agent: GoogleOther Allow: / Disallow: /auth/ # Google-Extended controls whether Google may ground Gemini and AI Overviews # answers in this content. Blocking it makes the site invisible to Google's AI # surfaces while still being indexed for classic search. User-agent: Google-Extended Allow: / Disallow: /auth/ User-agent: Storebot-Google Allow: / Disallow: /auth/ User-agent: Bingbot Allow: / Disallow: /auth/ # BingPreview renders the page snapshots shown in Bing results and Copilot. User-agent: BingPreview Allow: / Disallow: /auth/ User-agent: msnbot Allow: / Disallow: /auth/ User-agent: DuckDuckBot Allow: / Disallow: /auth/ User-agent: Applebot Allow: / Disallow: /auth/ # Applebot-Extended governs Apple Intelligence / Siri summarisation. User-agent: Applebot-Extended Allow: / Disallow: /auth/ User-agent: Slurp Allow: / Disallow: /auth/ User-agent: YandexBot Allow: / Disallow: /auth/ Crawl-delay: 2 User-agent: Baiduspider Allow: / Disallow: /auth/ Crawl-delay: 2 # ------------------------------------------- AI assistants / answer engines -- # ChatGPT-User is the live fetch made when a person asks ChatGPT a question. # It is already the largest single consumer of this site and follows a human # weekday rhythm — it is readers, arriving through an assistant. User-agent: ChatGPT-User Allow: / Disallow: /auth/ User-agent: OAI-SearchBot Allow: / Disallow: /auth/ User-agent: GPTBot Allow: / Disallow: /auth/ User-agent: ClaudeBot Allow: / Disallow: /auth/ User-agent: Claude-User Allow: / Disallow: /auth/ User-agent: Claude-SearchBot Allow: / Disallow: /auth/ User-agent: anthropic-ai Allow: / Disallow: /auth/ User-agent: PerplexityBot Allow: / Disallow: /auth/ User-agent: Perplexity-User Allow: / Disallow: /auth/ User-agent: xAI-Bot Allow: / Disallow: /auth/ # Grok: xAI's assistant fetching a page for a person who asked it a question. User-agent: Grok Allow: / Disallow: /auth/ User-agent: meta-externalagent Allow: / Disallow: /auth/ User-agent: meta-externalfetcher Allow: / Disallow: /auth/ User-agent: facebookexternalhit Allow: / Disallow: /auth/ User-agent: Amazonbot Allow: / Disallow: /auth/ Crawl-delay: 2 User-agent: MistralAI-User Allow: / Disallow: /auth/ User-agent: cohere-ai Allow: / Disallow: /auth/ User-agent: YouBot Allow: / Disallow: /auth/ User-agent: CCBot Allow: / Disallow: /auth/ Crawl-delay: 2 # ------------------------------------------------------ everything else ----- # Politely declined here; also 403'd by middleware.ts. User-agent: * Disallow: / Sitemap: https://regulations.ai/sitemap.xml