Google, Microsoft, OpenAI and Anthropic plan to honor one robots.txt line for opting out of AI training
Search Engine Journal compares each company's announcement and timeline side by side, including what the shared line won't do.
Google, Microsoft and OpenAI announced on Oct. 30 that they plan to honor a shared robots.txt group for AI-training preferences, so a website can state one choice to every participating provider instead of writing rules for each company's crawler. The token, __usage_ai-training, comes from an individual IETF Internet-Draft by Anthropic's Kevin Kelley, who wrote that Anthropic has started building support, according to Search Engine Journal.
None of the companies supports it yet. Google says it aims to in early 2027 and will treat the opt-out as a Google-Extended block when a site doesn't set Google-Extended itself; Microsoft will start honoring it Feb. 1, 2027, adds a separate page-level control for AI grounding, and will stop treating its old NoArchive/NoCache tags as training signals after March 1; OpenAI says "in the coming months."
Engadget notes the line can't undo past scraping, and it doesn't block crawling or affect appearing in search results. Existing robots.txt rules aren't read as a training preference.
-
RFC 9309: Robots Exclusion Protocol
After exploring Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out, the standard the new token builds on is RFC 9309: Robots Exclusion Protocol.
The 2022 IETF standard that formally specifies robots.txt, the method Martijn Koster first defined in 1994 for site owners to tell crawlers what they may access.
IETF (RFC Editor) · Plunge · an evening
0