robots.txt AI Training Opt-Out: Keep Your Search Traffic

On October 30, 2026, Google, Microsoft, OpenAI, and Anthropic said they will honor one shared robots.txt rule for opting out of AI training: a group that starts with User-Agent: __usage_ai-training. It states a training preference only. It does not block crawling, and it does not change Google or Bing search rankings. Most providers have not switched it on yet, so the safe setup for a blog today is the new group plus your existing per-crawler training rules, while leaving search crawlers alone.

Last verified: November 2, 2026. Sources: IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00 (published October 30, 2026), OpenAI crawler documentation, and the October 30, 2026 announcements from Google, Microsoft Bing, and OpenAI. Rollout dates are vendor plans and may change.

What is the __usage_ai-training rule in robots.txt?

__usage_ai-training is a shared “policy token” you put on a User-Agent line in robots.txt. A Disallow rule under it tells every participating AI provider that you prefer those paths not be used to train generative AI models. It is a usage preference, not a crawler name, and it does not stop any bot from visiting your pages.

The rule comes from an individual IETF Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Kevin Kelley of Anthropic and published on October 30, 2026. The draft expires on May 3, 2027. Its key design choice is that it adds no new robots.txt grammar. You write an ordinary group, and the same path matching and longest-match precedence from RFC 9309 (the robots.txt standard) apply.

Here is the example group from the draft. It states a preference against AI training for the whole site, except a press-kit folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Three rules in the draft matter most for bloggers:

• Wildcards do not count. A User-Agent: * group must not be read as matching the token. An old blanket rule does not opt you out.
• No group means no preference. If your file has no User-Agent: __usage_ai-training line, you have expressed nothing either way.
• It is not a crawl directive. The draft says content-usage preferences and traffic-management rules are “independent and orthogonal.” To limit crawling, you still use crawler-specific groups.

The draft borrows its meaning of “AI training” from the AIPREF working group vocabulary (draft-ietf-aipref-vocab-08): using content “to modify the learned parameters of a generative AI model.” Training models used only for classification, ranking, or scoring is not included. That exclusion is one reason the token is safe for search: ranking systems are outside its scope.

Training vs. search vs. AI answers: three different switches

“Blocking AI” now means three separate decisions: whether your content trains models, whether it appears in classic search results, and whether AI answers can quote or ground on it. The new token covers only the first. Mixing them up is how bloggers lose traffic by accident.

Each vendor splits these jobs a little differently. The table below maps the main controls as of November 2, 2026.

How to read the table: rows marked “No” are safe to set if your only goal is to keep content out of training. Rows marked “Yes” or “removes you” are visibility switches. Touch those only if you accept losing traffic from that surface.

OpenAI’s crawler documentation states the split directly: each setting is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot for training. OpenAI also notes it can take about 24 hours after a robots.txt change for its search systems to adjust.

Copy-paste robots.txt: opt out of AI training, keep search traffic

Add the shared training group, then keep explicit groups for the training-only tokens that work today. Do not add Disallow rules for Googlebot, Bingbot, or OAI-SearchBot. Your existing User-Agent: * rules stay as they are.

This file is a starting point for a typical blog. Put it at the root of your domain (for example, https://example.com/robots.txt), merged with whatever rules you already have.

# 1) Shared AI-training preference (IETF draft, rolling out 2026-2027)
User-Agent: __usage_ai-training
Disallow: /

# 2) Vendor training tokens that work today
User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# 3) Search crawlers: no Disallow rules, so traffic is unaffected
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Why keep the vendor groups if the shared token exists? Because the shared token is not live everywhere yet. Google says it aims to support it in 2027, and Microsoft starts on February 1, 2027. Until then, Google-Extended and GPTBot are the controls those companies actually read. The draft also lets providers treat their own tokens as more specific than the shared one, so explicit vendor groups keep your intent clear.

Variation: allow training on some folders

Path rules work exactly like normal robots.txt. To keep posts out of training but allow a public docs folder, write:

User-Agent: __usage_ai-training
Disallow: /
Allow: /docs/

Variation: stay in Gemini answers but not Gemini training

This one is not possible with Google-Extended alone, because that token covers both Gemini training and Gemini grounding. Google’s announcement says websites that specifically allow Google-Extended stay eligible to appear and receive clicks via grounding in Gemini Apps. So the choice is a trade-off: disallow Google-Extended to stay out of Gemini training and grounding, or allow it to keep Gemini grounding traffic.

Who supports the AI training opt-out, and when?

Four providers committed on October 30, 2026, but on different schedules. Microsoft has a firm date of February 1, 2027. Google says 2027. OpenAI said it will honor the signal without a published date. Anthropic is reported to have begun rolling it out.

Google’s Search Central post, written by John Mueller, explains the mapping: when a site uses the policy token to opt out of training “and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended.” Google adds that company-specific controls “will continue to function alongside this new joint signal.”

The draft itself has no formal IETF standing yet. It is an individual draft discussed in the AIPREF working group, and Internet-Drafts can be revised or replaced. Treat the syntax as stable enough to publish now, and recheck it when the draft is revised.

Bing’s new page-level training and grounding tags

Microsoft went further than the shared token. From February 1, 2027, Bing reads separate page-level preferences for AI training and AI grounding (using your page to build AI answers), and classic Bing rankings are not affected by either one.

Microsoft described three main options in its announcement: an AI Training setting, an AI Grounding setting, and a Limited Grounding setting that lets your content support Microsoft’s own AI products but not its third-party grounding services. That means a blogger can block training while still letting Copilot cite posts.

The page-level preferences are set with a meta tag or an X-Robots-Tag HTTP header aimed at Bingbot. Published guides covering Microsoft’s documentation show this form:

<meta name="Bingbot" content="ai-training:disallow">

X-Robots-Tag: Bingbot: ai-training:disallow

Check Bing’s own AI preferences help page for the final value names before deploying, since this rollout has not started yet.

If you used NoArchive or NoCache

Since 2023, Bing has treated NoArchive and NoCache as its AI controls. That ends. Here are the dates Microsoft announced:

• December 15, 2026: temporary grounding controls open in Bing Webmaster Tools for eligible sites using NoArchive or NoCache.
• February 1, 2027: Bing begins honoring the new preferences as it crawls.
• March 1, 2027: deadline to configure the new preferences if you want to keep your restrictions.
• September 30, 2027: temporary grounding preferences expire.

If you added NoArchive years ago for an unrelated reason, this is a good moment to decide whether you still want it at all.

Common robots.txt mistakes that cost search traffic

The costly mistakes all come from blocking a search crawler when you meant to block training. Googlebot, Bingbot, and OAI-SearchBot are visibility crawlers. Disallowing any of them removes you from that search surface, not just from AI.

What the new AI training rule cannot do

The token is a voluntary preference for future collection. It does not delete content already in datasets, it does not stop scrapers that ignore robots.txt, and it does not cover AI answers that fetch or quote your pages.

• Not retroactive. The draft ties preferences to the robots.txt file “in effect when the asset was retrieved.” Pages fetched before you added the rule were collected under the old file.
• Voluntary. robots.txt is honored by companies that choose to honor it. A bot that ignores it is not stopped. For enforcement, you need a firewall or bot-management service.
• Training only. Live grounding and user-triggered fetches are separate. OpenAI notes that ChatGPT-User visits are started by users and “robots.txt rules may not apply.”
• Private deals win. The draft says private agreements may override preferences expressed through this mechanism.
• Still a draft. The token name or rules could change before any standard is published.

10-minute setup checklist for bloggers

Open your live robots.txt, add the shared group and the vendor training groups, confirm no search crawler is disallowed, then retest. Most blogs need five lines of change and no plugin.

1. Open https://yourdomain.com/robots.txt in a browser and save a copy of the current file.
2. Add the User-Agent: __usage_ai-training group with Disallow: / (plus any Allow paths you want to share).
3. Add or confirm Disallow groups for Google-Extended, GPTBot, and ClaudeBot.
4. Search the file for Googlebot, Bingbot, and OAI-SearchBot. Make sure none has Disallow: /.
5. On WordPress, check whether an SEO plugin generates robots.txt. Edit it there so your change is not overwritten.
6. If you use NoArchive or NoCache for Bing, put March 1, 2027 on your calendar.
7. Decide separately whether you want to appear in AI Overviews, ChatGPT search, and Copilot answers. Leave those on if AI referrals matter to you.
8. Run Google Search Console’s robots.txt report after publishing to confirm Google fetched the new file.