One robots.txt line for AI training opt-out, from 2027

1. What happened

On October 30, 2026, Kevin Kelley, a standards representative at Anthropic, submitted an Internet-Draft to the IETF describing a shared way to state AI training preferences in robots.txt. A site writes the token __usage_ai-training on a User-agent line; Disallow under it means "please do not train on this" and Allow means training is fine. No new directives or grammar are added (IETF draft). ITmedia NEWS reports that Google, Microsoft and OpenAI announced plans to support it the same day.

In the draft's example, putting Disallow: / and Allow: /press-kit/ under User-Agent: __usage_ai-training says that everything except /press-kit/ should not be used for training. The setting is not a crawl ban and is independent of the per-crawler rules already in use. A site that does not write the token is treated as having stated no preference (ITmedia NEWS). The draft was submitted in an individual capacity and has no formal standing as an IETF standard at present.

2. When each company will support it

Google's official blog says it aims to support the control in 2027 along with other AI providers. If a site opts out of training with the shared token and has not explicitly specified Google-Extended, Google plans to treat that as an opt-out under Google-Extended too. Google-Extended keeps working alongside it (Google Search Central Blog).

Microsoft will start reading the new settings as it crawls from February 1, 2027. Besides the shared robots.txt token, sites can set training and "grounding" (using web pages to inform generated answers) separately, page by page, and the page-level training setting overrides robots.txt. The older NoArchive and NoCache tags are replaced, and sites that use them must move to the new settings by March 1, 2027. A site that wants to keep restricting grounding must also record a temporary preference in Bing Webmaster Tools, which opens on December 15, 2026 (Microsoft Bing Blogs).

OpenAI is reported to be aiming for support within a few months and to publish details when it ships. This comes from ITmedia NEWS; OpenAI's own page could not be opened from our environment, so we have not confirmed it against the original (ITmedia NEWS). The same article says Anthropic is also working on an implementation.

3. Why it matters to small businesses

Until now, a site had to look up each AI company's crawler name and write separate rules for each. With the shared token, a few added lines in robots.txt can state the preference to every company that supports it. Google says the approach is consistent with the EU AI Act code of practice for general-purpose AI model providers (Google Search Central Blog). Microsoft also cautions that blocking grounding as well reduces brand visibility in AI experiences (Microsoft Bing Blogs).

Our view Refusing training and appearing in AI answers are separate choices. We think even a company that does not want its site used for training will still want its name and products to show up in search and AI answers, so the two should be decided separately. Left undecided, price lists and original articles may be used for training without anyone intending it. Refusing everything without understanding it, on the other hand, risks dropping out of the picture when AI compares supplier candidates. Only companies that have not decided bear either loss. Start by listing which information you want shown and which you want protected.

4. What small businesses should check now

Open your site's robots.txt (https://your-domain/robots.txt) and read what is there now. WordPress and shop platforms, or your web agency, sometimes write it automatically
List what you want shown (company profile, products, prices) separately from what you do not want used for training (original articles, photos, members-only pages)
Ask your web agency whether any pages carry NoArchive or NoCache tags. If they do, March 1, 2027 is the migration deadline for Microsoft
Support is expected in 2027 and the draft may or may not become a standard, so do not rush the wording; prepare on the basis that per-company controls such as Google-Extended are the current tool
Confirm who updates robots.txt and whether that is covered by your maintenance contract with the agency

Our outlook

Over the next six months to two years, we expect stating an AI training preference in robots.txt to become a basic site setting, on a par with a company profile or the legal notices Japanese shops must display. As more people use AI to look up and compare products and suppliers, a website becomes both a document for people and a declaration to AI of how it may be used. A site with no stated preference is left to each company's defaults. Companies that separate what to keep out of training from what to show in AI answers early, and reflect that in their site structure and robots.txt, will be the ones deciding how their information is used. Now is the time to lay the groundwork.

Questions this article answers

What is the shared robots.txt way to refuse AI training?

In a draft submitted to the IETF on October 30, 2026, writing Disallow: / under User-agent: __usage_ai-training in robots.txt states that you do not want your content used for AI training. It is not a crawl ban, and a site without the token is treated as having stated nothing. Google, Microsoft and OpenAI have announced plans to support it, with Google aiming for 2027.

Do I need to rewrite robots.txt right now?

There is no need to hurry. Google plans support in 2027 and Microsoft starts reading the new settings on February 1, 2027, and the draft may not become a formal standard. Sites that use NoArchive or NoCache, however, must migrate by March 1, 2027 for Microsoft. First check your current robots.txt and decide which information you want shown and which protected.

If I refuse AI training, will my company vanish from AI answers?

Refusing training and being referenced when an answer is generated (grounding) are separate settings. Microsoft lets sites set them independently and cautions that restricting grounding as well reduces brand visibility in AI answers. Decide whether to refuse training only or to stop referencing too after you have settled which information you want shown.

Sources

Related Q&A

Related services

If you want to try this in your own business

To see how the topics in this article would apply to your own operations, and what it would cost, send us a message through the contact form.