AI Companies Back Shared Publisher Scraping Controls
OpenAI, Google, Microsoft and Anthropic have committed to a shared website signal for limiting AI training crawlers, announced on October 30. Publishers will be able to set preferences in robots.txt rather than configuring rules separately for each company.