Leading AI companies including Google, Microsoft, Anthropic, and OpenAI have jointly announced support for an upcoming standardized website crawling protocol. This standard aims to streamline how website operators can block AI developers from using their content to train generative AI models, consolidating multiple individual settings into a single, effective opt-out entry within the robots.txt file.
- Unified robots.txt entry to block AI training data scraping
- Microsoft and OpenAI to offer AI usage controls beyond training data scraping
- Meta supports the concept but has not committed to implementation
What happened
Google, Microsoft, Anthropic, and OpenAI have unveiled plans to implement a new crawling standard designed to enhance website control over AI data scraping. This standard will integrate into the existing Robots Exclusion Protocol, used globally via robots.txt files, to allow website owners to block all AI model training scrapers with a single directive instead of multiple entries for each AI provider or scraper.
The organizations have submitted a draft of this new standard to the Internet Engineering Task Force, the group responsible for maintaining the Robots Exclusion Protocol. This effort addresses the growing need for website owners to manage how AI developers collect data, reflecting the current complexities of adding separate entries for each entity and their various scrapers.
Why it matters
As generative AI models advance, the volume of web content scraped for training has increased substantially, causing concerns among content creators regarding data use and copyright implications. The existing process is cumbersome since site operators must maintain numerous individual allowances or blocks for different AI companies and their multiple scrapers, complicating compliance and enforcement.
This new crawling opt-out standard streamlines the process, empowering websites to conveniently and uniformly deny permission for training-related scraping. Additionally, Microsoft's 'AI Grounding' setting and similar anticipated controls from OpenAI will expand site owners' capability to limit AI systems from accessing their content for real-time query answering—addressing a broader scope of AI-related content access beyond training.
What to watch next
Google, Microsoft, Anthropic, and OpenAI plan staggered rollouts of support for the new crawling standard, with Microsoft and OpenAI targeting implementation within the coming months and Google expected to follow next year. The adoption pace among other tech companies, including Meta which has expressed conceptual backing but no implementation commitment, will be important to monitor.
The broader industry response and potential updates to the Robots Exclusion Protocol to fully integrate AI-specific controls will influence how website operators manage their content in an evolving AI ecosystem. Further developments around enforcement and compliance mechanisms for AI data usage are also anticipated as the standard gains traction.