---
title: "OpenAI, Google, Microsoft and Anthropic agree on a single robots.txt line to opt out of AI training"
url: https://hernandez.systems/en/articles/openai-google-microsoft-and-anthropic-agree-on-a-single-robots-txt-line-to-opt-out-of-ai-training
lang: en
type: NewsArticle
section: AI
published: "2026-11-04T14:34:16+00:00"
modified: "2026-11-05T15:00:24+00:00"
publisher: Osky Networks
image: https://oskydj.sfo3.cdn.digitaloceanspaces.com/calamardo/news_item/2026-11-04/e45-portada-f33de472.png
alternate_es: https://hernandez.systems/es/articulos/openai-google-microsoft-y-anthropic-acuerdan-una-sola-linea-de-robotstxt
---

# OpenAI, Google, Microsoft and Anthropic agree on a single robots.txt line to opt out of AI training

> OpenAI announced on October 30, 2026 that it will honor a common signal in robots.txt so sites can say whether or not they accept their content being used to train generative AI. It wasn't alone: that same day Google and Microsoft published equivalent commitments, and an Anthropic engineer submitted the draft that defines the signal to the IETF. The idea is simple: instead of maintaining one set of rules for GPTBot, another for Google-Extended and another for every bot that shows up, you write one single line that every participating provider reads. For anyone who runs a site, a blog or a client's website, this is the first time the four companies that crawl the web the most have agreed on something like this. But there's fine print, and quite a lot of it.

OpenAI announced on October 30, 2026 that it will honor a common signal in `robots.txt` so sites can say whether or not they accept their content being used to train generative AI. It wasn't alone: that same day Google and Microsoft published equivalent commitments, and an Anthropic engineer submitted the draft that defines the signal to the IETF. The idea is simple: instead of maintaining one set of rules for GPTBot, another for Google-Extended and another for every bot that shows up, you write **one single** line that every participating provider reads. For anyone who runs a site, a blog or a client's website, this is the first time the four companies that crawl the web the most have agreed on something like this. But there's fine print, and quite a lot of it.

## What OpenAI announced, exactly

OpenAI's announcement is short. It says it will honor "a common signal" in `robots.txt` to express preferences about the use of content "for foundational generative AI training," and that it is a "voluntary effort." The specifics:

- **No date.** OpenAI says it aims to implement it "in the coming months" and that it will publish another notice with details when it does.
- **It replaces nothing.** Its current per-bot controls remain in force. According to its crawler documentation, blocking `GPTBot` still indicates that content should not be used to train its foundation models, and `OAI-SearchBot` remains a separate control for appearing in ChatGPT search.
- **It comes from the IETF.** OpenAI ties the signal to the work of the IETF's AI Preferences (AIPREF) group and asks other providers to adopt it.

## How the signal works

The signal is defined in the draft `draft-kelley-aipref-training-preference-signal-00`, written by Kevin Kelley of Anthropic and published on October 30. Its beauty is that it doesn't invent new syntax: it uses a normal `robots.txt` group, but the `User-Agent` line doesn't hold a bot's name, but rather a "policy token," `__usage_ai-training`:

```
User-Agent: __usage_ai-training
Disallow: /
Allow: /prensa/
```

That group expresses a preference against training across the whole site, except for `/prensa/`, where the preference is in favor. Path matching and the longest-match-wins rule are the same as in RFC 9309, the `robots.txt` standard. Parsers that don't know the token ignore it like any unknown bot, so nothing breaks.

Three details of the draft that matter more than they seem:

1. **The wildcard doesn't count.** A `User-agent: *` group is never interpreted as speaking about the new token. If your `robots.txt` blocks everything with `*`, as far as this signal is concerned you **said nothing** about training. Kelley explained it on the IETF list: today no published `robots.txt` expresses any preference at all.
2. **It's a usage preference, not an access one.** The token doesn't tell anyone what to crawl. For that you still need groups with each bot's name. The draft only says that a consumer *may* stop downloading what it won't be able to use.
3. **Private agreements win.** A licensing contract between a media outlet and an AI company can override whatever the `robots.txt` says.

What counts as "training" isn't invented by the draft either: it takes it from the AIPREF group's vocabulary (`draft-ietf-aipref-vocab-08`). It means using content to modify the learned parameters of a generative AI model. Models that only classify, rank or score are left out. Heads up: the document itself warns that its text doesn't yet reflect working group consensus.

![Diagrama de un robots.txt con tres grupos: el comodín para tráfico, el token __usage_ai-training para la preferencia de entrenamiento y una excepción con Google-Extended](https://oskydj.sfo3.cdn.digitaloceanspaces.com/calamardo/news_item/2026-11-04/e45-robots-6dd075fc.png)

## Four companies, four calendars

They all back the same line, but each will read it whenever it wants and in its own way:

| Company | When | What it adds or changes |
|---|---|---|
| Microsoft (Bing) | February 1, 2027 | Training preference also per page and a separate *grounding* preference; retires `NoArchive` and `NoCache` as an AI control |
| Google | 2027 | A shared `Disallow` with no mention of Google-Extended will be read as a Google-Extended `Disallow` |
| OpenAI | "In the coming months" | Nothing yet; keeps GPTBot and its controls |
| Anthropic | No date | Kelley says they've already started implementing it and that covering all their services will take time |

**Microsoft** is the only one with a firm date and the one that changes the most. It will honor the signal starting February 1, 2027, as it crawls. It also adds, on its own, a *grounding* preference, that is, whether your page can feed generated answers in Bing, Copilot or Microsoft 365. That one is set page by page with a meta tag or HTTP header (`ai-grounding:disallow` or `ai-grounding:disallowextended`). The most urgent part: if you use `NoArchive` or `NoCache` today to stay out of its AI, Microsoft will stop reading them as a training or grounding preference after **March 1, 2027**. Before that date you have to publish the new preferences.

**Google** doesn't create a new system: it translates. If your `robots.txt` denies training with the shared token and says nothing about Google-Extended, it will treat it as if you had blocked Google-Extended. The nuance to understand: Google-Extended doesn't only cover training. According to Google's documentation, it also decides whether your content is used for Gemini *grounding*. Google puts it the other way around in its post: sites that explicitly allow Google-Extended can appear in Gemini Apps answers and receive clicks through grounding. My reading, and PPC Land's, is that with Microsoft that line only takes you out of training, and with Google it also takes you out of Gemini citations, unless you add an explicit `Allow` for Google-Extended.

![Tarjetas con lo que dijo cada empresa: Microsoft el 1 de febrero de 2027, Google a principios de 2027, OpenAI en los próximos meses y Anthropic sin fecha](https://oskydj.sfo3.cdn.digitaloceanspaces.com/calamardo/news_item/2026-11-04/e45-calendario-08ea64ee.png)

## What to do with your robots.txt

I haven't tested it; this comes from the official documents from Google, Microsoft, OpenAI and the IETF draft. The practical stuff:

- **If you don't want your site used for training**, add the `__usage_ai-training` group with `Disallow: /`. Adding it today does no harm: no parser that respects RFC 9309 breaks, and it'll be ready for when each provider turns it on.
- **Don't delete your current rules.** As long as nobody reads the signal, what works today is still blocking `GPTBot`, `Google-Extended` and company by name. All four companies say those controls remain in force.
- **Don't confuse usage with crawling.** The token doesn't stop a bot from visiting you. If you're worried about server load, that's still a matter for per-bot groups or your firewall.
- **Decide what to do about Gemini.** If you want out of training but still want to show up cited in Gemini, Google indicates that an explicit `Allow` for Google-Extended keeps you eligible.
- **If you use `NoArchive` or `NoCache` for Bing**, mark March 1, 2027 on your calendar.
- **Use paths.** An `Allow` for a press folder or public documentation lets you give up what you do want used.

## What this means (and what it doesn't)

My take: it's a useful and pretty well-designed step. The fact that it doesn't invent syntax and that the wildcard doesn't count is what keeps millions of old `robots.txt` files from being reinterpreted all at once. But it needs the right label. **It's a preference, not a lock.** The IETF vocabulary says it bluntly: the specification doesn't guarantee that preferences will be honored. OpenAI calls it voluntary. The draft is an individual submission, has no formal IETF backing and expires on May 3, 2027 if it isn't renewed. And the list of those who have committed has absences: none of the announcements names Meta, Apple or Amazon among those who will read the token.

What does change is the conversation. Until now, opting out of training meant chasing a list of bots that grew every month. If the signal catches on, one line is enough for those who respect it, and it becomes clearer who doesn't.

## Sources

- [OpenAI: Making it easier for websites to express AI training preferences](https://openai.com/index/helping-websites-express-ai-training-preferences/)
- [IETF: draft-kelley-aipref-training-preference-signal-00](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/)
- [Kevin Kelley en la lista ai-control del IETF](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w)
- [IETF: A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08)](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/)
- [Google Search Central: Supporting a shared AI training control for websites](https://developers.google.com/search/blog/2026/10/ai-training-control)
- [Microsoft Bing: Your content, your choice](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)
- [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)
- [PPC Land: Google, Microsoft, OpenAI and Anthropic back one AI training opt-out](https://ppc.land/google-microsoft-openai-and-anthropic-back-one-ai-training-opt-out/)