AI Giants Back One Shared Opt-Out for AI Training Scraping

OpenAI, Google, Microsoft and Anthropic said on October 30, 2026 that they will honor a single robots.txt signal letting publishers opt out of AI training scraping, replacing the need to write separate rules for every provider.

AI TrainingAnthropicGoogle

Four of the largest AI companies have agreed to honor a single, shared signal that lets website publishers opt out of having their content scraped to train AI models. OpenAI, Google, Microsoft and Anthropic announced their support for the common robots.txt preference on October 30, 2026, in a coordinated move that turns a patchwork of per-company blocking rules into one standard instruction.

The change matters because of how robots.txt works today. A publisher that wants to block AI training scraping must add a separate entry for every provider, and for every scraper each provider runs. The new approach replaces that tangle with one entry that participating companies all recognize. The signal is defined in an Internet-Draft submitted to the Internet Engineering Task Force (IETF), titled Machine-Readable Training Preference Signal and authored by Kevin Kelley, a technical lead for standards at Anthropic. The draft is dated October 30, 2026, and builds on the work of the IETF AI Preferences working group. Reporting on the proposal identifies the shared token as usage_ai-training.

OpenAI said it will honor the common signal in robots.txt, the file websites use to express preferences about how their content is used for foundational generative AI training. The company aims to implement the signal in the coming months and says it will share more detail about its implementation when it does. OpenAI stressed that the new signal will not replace the crawler controls it already provides, which will continue to apply and be respected.

Microsoft gave the most detailed rollout plan. The company said it will start honoring the shared AI Training preference on February 1, 2027, and that it is extending the same preference to the page level, so a publisher can override a robots.txt setting for an individual page. Microsoft is also adding a separate AI Grounding preference, which covers a different use of content: when a model accesses a page to inform a generated answer rather than to collect training data. Alongside Disallow, which prevents grounding use, Microsoft is introducing a more granular option called DisallowExtended, which limits grounding use to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry. Microsoft said these preferences will replace its existing NoArchive and NoCache tags for training and grounding, and that choosing them will not affect eligibility for Bing search results.

Anthropic has already started honoring the signal, according to press reports of the announcement, making it the first of the four to move. Google plans to implement support sometime in 2027. Meta Platforms expressed support for the idea but has not committed to implementing it, according to SiliconANGLE.

The scope of the agreement has clear limits, and publishers should read them carefully. The shared signal applies only to future training of generative AI models. It does not cover models that perform exclusively non-generative tasks, and it does not stop bots from crawling a website for other purposes. Scraping remains allowed by default until a publisher changes the setting, Microsoft said. The proposal itself is still an individual Internet-Draft, an early stage of IETF standardization: it is documented and open for review, but it has not been adopted as a formal standard.

The announcement lands after years of friction between AI companies and publishers. News outlets and other website owners have complained about a surge in bots collecting their articles to feed AI models, and several publishers have pursued legal claims over training data. Until now, each company offered its own opt-out mechanism, so a publisher had to track and maintain a separate rule for each one. A single signal, if widely honored, lowers that burden substantially. Whether it changes much in practice will depend on how quickly the companies ship support, how completely their crawlers respect the token, and whether other major model developers, including Meta and open-weight labs, join in.

The effort behind the signal is not new. Microsoft noted that the announcement builds on more than a year of work by the IETF AI Preferences working group toward a global, harmonized practice for AI preferences, and that the group continues to work toward a final standard. Microsoft was also the first company to offer publisher controls for AI training and grounding, introducing its NoArchive and NoCache tags in 2023. Those tags are now being replaced by the clearer training and grounding preferences, a sign of how quickly the tooling around publisher choice is evolving.

Verification remains an open question. The companies have not yet published a shared mechanism for publishers to check whether every crawler has applied their preferences, and the technical specification of the token is still described only in outline in press coverage. Publishers that adopt the token early will be relying on each company to implement it faithfully, and on the IETF process to turn an individual draft into a stable standard that smaller model developers can also adopt.

For publishers, the practical advice in the meantime is unchanged: existing per-provider controls still apply, and the new shared token is an addition, not a replacement. For the AI industry, the joint announcement is a rare moment of alignment on a contested issue, and a sign that the fight over training data is moving from courtrooms and crawler blocklists toward shared standards.