Google, Microsoft and OpenAI Back One Shared AI-Training Opt-Out
A single robots.txt line would let a site say no to AI training at all four big labs at once. It is voluntary, it is not a standard yet, and it does not switch on everywhere at the same time.
Google, Microsoft and OpenAI said on Friday they will honor a single shared signal for AI training. A website would state its preference once, in its robots.txt file, instead of writing a separate rule for each company's crawler.
The signal is a policy token called __usage_ai-training. It goes in the User-Agent line of an ordinary robots.txt group. A site that wants out of training writes Disallow: / under that token. It can carve out exceptions with Allow lines. The draft's own example opts out a whole site except for a press-kit folder.
The idea came from Anthropic. Kevin Kelley, a technical lead for standards at Anthropic, submitted it on October 30 as an Internet-Draft to the Internet Engineering Task Force's AI Preferences working group. Anthropic said it has already started implementing support. Google, Microsoft and OpenAI announced their backing the same day.
What it covers, and what it does not
The token names a use, not a crawler. Under the working group's vocabulary, AI training means using content to change a generative model's learned parameters. Models that only do non-generative work, such as classification or ranking, fall outside that definition.
It is not a block on crawling. The draft states plainly that the token is not a directive for crawling behavior. A bot may still visit a page. It also does not remove content that has already been collected or already trained into a model.
Two traps matter for site owners. A catch-all User-Agent: * rule does not count as a training preference. And a robots.txt file with no __usage_ai-training group expresses no preference at all, which is true of every robots.txt file published today.
Different timelines at each company
Nothing changes yet. None of the three companies supports the line today, on the timelines they gave.
Microsoft gave the firmest dates. It will start honoring the new preferences as it crawls from February 1, 2027. It is also adding a separate page-level AI Grounding preference, which can be set by meta tag or HTTP header. After March 1, 2027, Microsoft will stop treating its older NoArchive and NoCache tags as training or grounding signals. Sites that still rely on those tags have until then to switch.
Google aims to support the shared line in 2027, with its crawler documentation saying early 2027. If a site opts out through the shared line and does not specify Google-Extended, Google plans to treat it as if the publisher had blocked Google-Extended, its own AI training control since 2023. That control covers future Gemini model training and grounding in Gemini Apps.
OpenAI said it aims to implement the signal in the coming months. Its existing crawler controls will stay in place rather than being replaced. As of October 30, OpenAI had not said what happens when a shared rule conflicts with its existing GPTBot rule.
Not a standard yet
The draft is an individual submission. The IETF tracker lists it as such, which means it is not official IETF standardization activity and the body has not endorsed it. It expires on May 3, 2027, unless a new version replaces it.
It also binds only the companies that choose to follow it. Scrapers that ignore robots.txt will ignore this too. Meta has expressed support for the idea but has not committed to implementing it, according to reports of the announcement.
For publishers, the change is still a simplification. One line, written once, would carry the same preference to four of the largest AI labs. Whether it becomes the norm depends on whether the draft survives the standards process, and on whether the companies meet the dates they just gave.
