Google, Microsoft, OpenAI, and Anthropic have endorsed a shared AI training opt-out mechanism embedded within the standard robots.txt file, simplifying how sites can exclude their content from generative AI training.
- One robots.txt rule blocks Google, Microsoft, OpenAI, and Anthropic AI training.
- Implementation starts 2027 with phased rollout and complementary crawler controls.
- New standard aligns with EU AI Act recommendations and publisher concerns.
What happened
Google, Microsoft, OpenAI, and Anthropic have jointly committed to honoring a shared AI training opt-out directive via the robots.txt standard. This initiative comes from an Internet Engineering Task Force (IETF) draft proposed by Anthropic’s Kevin Kelley that introduces a new policy token named __usage_ai-training. By placing this token in their robots.txt file along with specified paths, website owners can instruct these companies not to use that content for training generative AI models.
Previously, sites had to create separate blocking rules for each company's crawler to prevent content from being included in AI training datasets. The unified approach greatly simplifies the process. While not a formal IETF standard yet, major industry players plan to support the protocol beginning in early 2027. Microsoft, Google, and OpenAI have all announced timelines for implementing the shared robot exclusion, ensuring broad industry adoption.
Why it matters
This development addresses growing concerns from publishers and content creators about control and compensation related to the use of their materials in AI training. Without a shared signal, sites faced fragmented and inconsistent approaches that complicated content management and enforcement. The new robots.txt token empowers them to express a clear, machine-readable preference limiting the inclusion of their content in AI model datasets.
The consensus among these leading AI companies also signals maturation in the responsible AI ecosystem, showing responsiveness to regulatory and market pressures. The approach aligns with the EU AI Act’s Code of Practice, which references robots.txt as a key tool for machine-readable content usage preferences. While this does not establish any licensing or payment agreements, it is a foundational step towards clearer digital content boundaries in the AI era.
What to watch next
Industry observers should monitor how quickly and comprehensively the AI companies implement support for the new token across their services. Anthropic and OpenAI have committed to rolling it out in the coming months, while Google and Microsoft target early 2027. Detailed rules and edge cases will likely emerge as websites adopt the new standard and seek clarifications on scope and enforcement.
It is also important to track whether other major tech firms, such as Meta, follow suit in recognizing this shared AI training opt-out signal. Additionally, legal and licensing discussions around AI content usage and compensation will continue evolving, with the new standard serving as a technical foundation but not resolving broader economic questions. Publishers and platform operators might further develop complementary tools or agreements based on this architecture.