OpenAI, Google, Microsoft and Anthropic agree on a single robots.txt line to opt out of AI training

OpenAI announced on October 30, 2026 that it will honor a common signal in robots.txt so sites can say whether or not they accept their content being used to train generative AI. It wasn't alone: that same day Google and Microsoft published equivalent commitments, and an Anthropic engineer submitted the draft that defines the signal to the IETF. The idea is simple: instead of maintaining one set of rules for GPTBot, another for Google-Extended and another for every bot that shows up, you write one single line that every participating provider reads. For anyone who runs a site, a blog or a client's website, this is the first time the four companies that crawl the web the most have agreed on something like this. But there's fine print, and quite a lot of it.

What OpenAI announced, exactly

OpenAI's announcement is short. It says it will honor "a common signal" in robots.txt to express preferences about the use of content "for foundational generative AI training," and that it is a "voluntary effort." The specifics:

No date. OpenAI says it aims to implement it "in the coming months" and that it will publish another notice with details when it does.
It replaces nothing. Its current per-bot controls remain in force. According to its crawler documentation, blocking GPTBot still indicates that content should not be used to train its foundation models, and OAI-SearchBot remains a separate control for appearing in ChatGPT search.
It comes from the IETF. OpenAI ties the signal to the work of the IETF's AI Preferences (AIPREF) group and asks other providers to adopt it.

How the signal works

The signal is defined in the draft draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic and published on October 30. Its beauty is that it doesn't invent new syntax: it uses a normal robots.txt group, but the User-Agent line doesn't hold a bot's name, but rather a "policy token," __usage_ai-training:

User-Agent: __usage_ai-training
Disallow: /
Allow: /prensa/

That group expresses a preference against training across the whole site, except for /prensa/, where the preference is in favor. Path matching and the longest-match-wins rule are the same as in RFC 9309, the robots.txt standard. Parsers that don't know the token ignore it like any unknown bot, so nothing breaks.

Three details of the draft that matter more than they seem:

The wildcard doesn't count. A User-agent: * group is never interpreted as speaking about the new token. If your robots.txt blocks everything with *, as far as this signal is concerned you said nothing about training. Kelley explained it on the IETF list: today no published robots.txt expresses any preference at all.
It's a usage preference, not an access one. The token doesn't tell anyone what to crawl. For that you still need groups with each bot's name. The draft only says that a consumer may stop downloading what it won't be able to use.
Private agreements win. A licensing contract between a media outlet and an AI company can override whatever the robots.txt says.

What counts as "training" isn't invented by the draft either: it takes it from the AIPREF group's vocabulary (draft-ietf-aipref-vocab-08). It means using content to modify the learned parameters of a generative AI model. Models that only classify, rank or score are left out. Heads up: the document itself warns that its text doesn't yet reflect working group consensus.

Diagrama de un robots.txt con tres grupos: el comodín para tráfico, el token __usage_ai-training para la preferencia de entrenamiento y una excepción con Google-Extended

Four companies, four calendars

They all back the same line, but each will read it whenever it wants and in its own way:

Company	When	What it adds or changes
Microsoft (Bing)	February 1, 2027	Training preference also per page and a separate grounding preference; retires `NoArchive` and `NoCache` as an AI control
Google	2027	A shared `Disallow` with no mention of Google-Extended will be read as a Google-Extended `Disallow`
OpenAI	"In the coming months"	Nothing yet; keeps GPTBot and its controls
Anthropic	No date	Kelley says they've already started implementing it and that covering all their services will take time

Microsoft is the only one with a firm date and the one that changes the most. It will honor the signal starting February 1, 2027, as it crawls. It also adds, on its own, a grounding preference, that is, whether your page can feed generated answers in Bing, Copilot or Microsoft 365. That one is set page by page with a meta tag or HTTP header (ai-grounding:disallow or ai-grounding:disallowextended). The most urgent part: if you use NoArchive or NoCache today to stay out of its AI, Microsoft will stop reading them as a training or grounding preference after March 1, 2027. Before that date you have to publish the new preferences.

Google doesn't create a new system: it translates. If your robots.txt denies training with the shared token and says nothing about Google-Extended, it will treat it as if you had blocked Google-Extended. The nuance to understand: Google-Extended doesn't only cover training. According to Google's documentation, it also decides whether your content is used for Gemini grounding. Google puts it the other way around in its post: sites that explicitly allow Google-Extended can appear in Gemini Apps answers and receive clicks through grounding. My reading, and PPC Land's, is that with Microsoft that line only takes you out of training, and with Google it also takes you out of Gemini citations, unless you add an explicit Allow for Google-Extended.

Tarjetas con lo que dijo cada empresa: Microsoft el 1 de febrero de 2027, Google a principios de 2027, OpenAI en los próximos meses y Anthropic sin fecha

What to do with your robots.txt

I haven't tested it; this comes from the official documents from Google, Microsoft, OpenAI and the IETF draft. The practical stuff:

If you don't want your site used for training, add the __usage_ai-training group with Disallow: /. Adding it today does no harm: no parser that respects RFC 9309 breaks, and it'll be ready for when each provider turns it on.
Don't delete your current rules. As long as nobody reads the signal, what works today is still blocking GPTBot, Google-Extended and company by name. All four companies say those controls remain in force.
Don't confuse usage with crawling. The token doesn't stop a bot from visiting you. If you're worried about server load, that's still a matter for per-bot groups or your firewall.
Decide what to do about Gemini. If you want out of training but still want to show up cited in Gemini, Google indicates that an explicit Allow for Google-Extended keeps you eligible.
If you use NoArchive or NoCache for Bing, mark March 1, 2027 on your calendar.
Use paths. An Allow for a press folder or public documentation lets you give up what you do want used.

What this means (and what it doesn't)

My take: it's a useful and pretty well-designed step. The fact that it doesn't invent syntax and that the wildcard doesn't count is what keeps millions of old robots.txt files from being reinterpreted all at once. But it needs the right label. It's a preference, not a lock. The IETF vocabulary says it bluntly: the specification doesn't guarantee that preferences will be honored. OpenAI calls it voluntary. The draft is an individual submission, has no formal IETF backing and expires on May 3, 2027 if it isn't renewed. And the list of those who have committed has absences: none of the announcements names Meta, Apple or Amazon among those who will read the token.

What does change is the conversation. Until now, opting out of training meant chasing a list of bots that grew every month. If the signal catches on, one line is enough for those who respect it, and it becomes clearer who doesn't.