Google, Microsoft, OpenAI and Anthropic have backed a shared robots.txt signal that lets website owners opt out of AI training without blocking crawler access. The token expresses a preference about how content is used, not whether a bot may fetch it, so sites can stay visible in search while declining to feed model training.
Microsoft said its support starts on 1 February 2027. It is also adding a separate AI Grounding preference that decides whether a page can inform AI-generated answers in Bing, Copilot and Microsoft 365, so a site can opt out of training yet still be used as a source for answers. Google is listed for 2027, and OpenAI has said only that it will implement the signal in the coming months; reports differ on how far Anthropic has got.
Why it matters: until the signal is honored, publishers who want to opt out should keep their existing crawler-specific blocks.