Google, Microsoft, and OpenAI each said on Friday, Oct. 30, that they plan to read a single robots.txt group as a site’s AI training opt-out. None of the three supports it today. Publishers who maintain separate rules for every AI crawler could eventually collapse them into one entry, but the companies attach different consequences to it.
Search Engine Journal’s Matt G. Southern detailed the plans. The mechanism comes from an individual Internet Draft that Kevin Kelley of Anthropic submitted to the IETF’s AI Preferences working group. An individual draft is not official IETF standardization. Kelley told the group’s mailing list that Anthropic has started building support, and Anthropic’s newsroom had published nothing on it as of Oct. 30.
The draft creates no new robots.txt directive. A site writes an ordinary rule group addressed to a policy token named __usage_ai-training, using standard Allow and Disallow lines. Kelley’s own example blocks the whole site except a press-kit folder. Training, in the working group’s vocabulary, means using content to change the learned parameters of a generative model. Models that only classify or rank fall outside that definition.
Existing files do not opt anyone out. Under the draft, a wildcard group never applies to this token, and a robots.txt with no group addressed to it states no preference, which describes every file published today. The token also does not control crawling. A site with a blanket block for all user agents must therefore add an explicit group to say anything about training.
Google’s announcement came from John Mueller. If a site opts out via the token and leaves Google-Extended unspecified, Google plans to treat the site as though it had blocked Google-Extended. Google targets 2027, and the crawler page it maintains narrows that to the first part of the year. Google-Extended governs training of future Gemini models, plus grounding for Gemini Apps and Vertex AI. Grounding means handing a model material drawn from the search index while it composes an answer.
That bundling is the catch. Google’s documentation describes no option that withholds content from Gemini model training yet permits Gemini Apps to ground answers on it. Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode answer to a separate opt-out in Search Console.
Microsoft takes a different route. Krishna Madhavan, a Bing principal product manager, wrote the post that adopts the token for training and introduces a distinct AI Grounding preference, available per page only. A page can be set to Allow (the default), Disallow, or DisallowExtended, which permits grounding in Microsoft’s own tools such as Copilot and the Bing AI Mode but blocks it in grounding services Microsoft supplies to other businesses. Microsoft says either restriction can reduce brand visibility in AI answers and leaves Bing search listings untouched. Individual pages can also carry a training preference in a meta tag or HTTP header, which overrides robots.txt for that page.
Microsoft is the only company replacing controls that sites already use. It begins honoring the new options on Feb. 1, 2027, and stops treating NoArchive and NoCache as training or grounding signals after March 1, 2027. Sites that still carry those tags on Dec. 15 can register a stopgap preference through Bing Webmaster Tools. According to Microsoft’s FAQ, pages lacking a preference at the March 1 cutoff will permit grounding until Microsoft crawls them again.
OpenAI aims to support the signal within the next few months and says its current robots.txt policy stays in effect. Neither its announcement nor its bot documentation, as reviewed on Oct. 30, says what happens when the token and a GPTBot rule conflict. Anthropic’s help page, also reviewed that day, offers three controls: ClaudeBot, Claude-User, and Claude-SearchBot.
The unanswered conflict question matters most for sites with mixed policies. A publisher that blocks the token but allows GPTBot has no documented outcome. The announcements also include no independent testing of how any crawler will behave once support ships.
Teams relying on NoArchive or NoCache to restrict Copilot should settle their grounding stance before Dec. 15, because Microsoft’s deadline arrives first. Everyone else should hold existing per-crawler rules in place, since Google, OpenAI, and Anthropic’s Kelley say existing controls keep working.
Search Engine Journal, Matt G. Southern, published Nov. 1, 2026, with additional reporting from Search Engine Roundtable.
