AI Companies Back Shared Publisher Scraping Controls

The proposed directive, developed by Anthropic standards lead Kevin Kelley through the Internet Engineering Task Force, adds an AI-training signal to a website’s robots.txt file. The companies say their crawlers will respect it, giving publishers one way to block or permit AI-training access across multiple services.

Website owners can also allow access to selected pages while blocking it elsewhere. The signal is intended for crawlers gathering material for generative AI training; it does not stop all crawling or remove a site from Google search results. Microsoft says it will offer separate controls for AI training and AI grounding, which uses verified sources to inform responses.

Anthropic has already begun supporting the directive. OpenAI plans to implement it in the coming months, while Google expects to support it in early 2027. Microsoft has set February 1, 2027, as its start date. The directive applies to future access and cannot undo content the companies have already collected.

The proposal comes as publishers contend with AI crawlers collecting material from their sites. Other responses have included the Really Simple Licensing standard, which lets publishers charge crawlers, and Cloudflare’s default blocking of AI crawlers.