One robots.txt line to opt out of AI training

The one-line change

On 30 October 2026 four of the largest Western AI companies said they will honour a single shared signal in robots.txt for AI training preferences. OpenAI, Google, Microsoft and Anthropic each backed the same policy token. A website owner can now state one training preference that every participating company reads in the same way instead of maintaining a separate rule for each company. [S1][S3][S4][S5][1][2][3][4][5]

The token is `__usage_ai-training`. It sits in the User-Agent line of an ordinary robots.txt group where a crawler name normally goes. A site that wants to opt out of training for everything writes two lines. A site that wants to keep one area open adds an Allow line for that path. Nothing about the file format changes. The draft introduces no new directive names and no new grammar. [S2][S3][S4][1][2][3][4][5]

Why publishers asked for one signal

Until now a publisher that wanted to refuse training had to learn each company's control and name it separately. Google offered Google-Extended. OpenAI named GPTBot. Anthropic named ClaudeBot. Other companies ran their own tokens. A site with several rules had to keep them all in step and check each company's documentation when anything changed. [S1][S3][S4][1][3][4][5][8][9]

That burden fell hardest on small publishers. A large newsroom has staff who track crawler documentation. A small blog or an independent journal does not. One shared token lowers the cost of saying no from a research project to a single edit. [S1][S5][1][3][4][5][8][9]

The complaint behind the change is about traffic as much as training. Publishers whose content gets answered directly by AI systems have seen search referrals fall. Reddit said on 29 October 2026 that it will test login-first browsing after referrals fell by more than one million per week on average in October. A shared training signal does not fix that traffic problem on its own. [S8][S9][1][3][4][5][8][9]

How the token works in robots.txt

The mechanism comes from an Internet-Draft published on 30 October 2026 by Kevin Kelley of Anthropic. Its title is "Machine-Readable Training Preference Signal" and its number is draft-kelley-aipref-training-preference-signal-00. It is an individual submission with no IETF endorsement yet and it expires on 3 May 2027 unless revised or adopted. [S2][S3][S4][2][3][4]

A site expresses a preference by writing a group keyed to the token. The draft example opts out of training for all paths except a press kit, written as three lines: User-Agent: __usage_ai-training / Disallow: / / Allow: /press-kit/[2][3][4]

That group says the site prefers no AI training on any path except pages under the press kit path. The preference is path specific. A publisher can refuse training for a whole site, for one section or for everything except a section it wants to share. The token must match exactly apart from letter case. The two leading underscores make accidental matches unlikely. [S2][S3][2][3][4]

What counts as training under the draft

The draft defines AI training narrowly. It means using an asset to modify the learned parameters of a generative AI model. A generative AI model is one used or made available to generate synthetic content in one or more modalities. Models used only for classification, ranking or scoring fall outside the definition even when they generate a rationale for their output. [S2][S3][2][3][4]

That narrow definition matters for site owners who read the token as a general ban on AI use. It is not one. The token speaks to training of generative models. It does not speak to every AI system that touches a page. A publisher that wants to refuse other uses still needs the controls that cover those uses. [S2][S4][2][3][4]

What it does not do: crawling

The most common misunderstanding is also the most important to correct. The policy token is not a directive for crawling behaviour. The draft says so in plain terms. A consumer may use a preference to inform crawling. It may refrain from retrieving resources for which a preference against a covered purpose has been expressed if it crawls only for that purpose. But the token itself does not block a crawler from fetching a page. [S2][S3][2][3][4]

A site that wants to limit crawling still needs crawler rules. It writes groups for the crawlers product tokens in robots.txt to do that job. Content-usage preferences and traffic management signals can both live in robots.txt. They are independent and orthogonal. [S2][S3][2][3][4]

Wildcard groups do not match the token. A group headed `User-Agent: *` must not be interpreted as matching `__usage_ai-training`. A file with no group for the token expresses no training preference at all whatever its catch-all rules say. A site that relies on a catch-all block has not opted out of training until it adds the token group explicitly. [S2][S4][2][3][4]

What it does not do: search and answers

Opting out of training does not remove a page from search results. Google has long separated its training control from search inclusion. Google-Extended does not affect a site's inclusion in Google Search and it is not a ranking signal. The shared token follows the same separation. A publisher can refuse training while staying fully visible in ordinary search. [S3][S4][3][4][5]

The same separation applies to AI answers that quote pages. A training opt-out does not keep a page out of AI-written search answers. Those answers draw on content through grounding: the step where a model is pointed at a trusted source to answer a question. [S3][S4][S5][3][4][5]

Microsoft makes the distinction explicit with a separate control. It will offer a page-level AI Grounding preference alongside the shared training token. That control covers whether content can help shape AI answers in products such as Copilot and Microsoft 365. The two preferences answer two different questions. [S4][S5][3][4][5]

What it does not do: data already collected

Updating robots.txt does not remove material already collected into existing datasets or trained into model weights. OpenAI said the new signal will not replace the controls it already provides. Its existing crawler rules stay in force and will continue to be respected. The shared token governs future training use by participating companies. It does not unwind past collection. [S1][S3][S4][1][2][3][4]

That limit shapes honest expectations. A publisher that adds the token today should not expect older content to disappear from models already trained. The value of the token is forward looking. It sets the terms for the next training run rather than rewriting the last one. Publishers with serious concerns about past collection still need direct routes to companies and to regulators. [S3][S4][1][2][3][4]

Private agreements may also override preferences expressed through the mechanism. A licensing deal or a partnership term between a publisher and a company can set different terms. The draft says so directly. The token is a public default for companies that choose to honour it. It does not override a bargain the two sides struck separately. [S2][1][2][3][4]

Who ships when

Support is promised rather than live at most companies. The timelines are uneven and publishers should plan around the slowest one that matters to them. Anthropic has already started implementing support. Rollout across its services will take time. [S3][S4][S5][1][3][4][5]

OpenAI aims to implement the signal in the coming months. Its announcement gave no firm date and promised another update with implementation details when it does ship. Until that update lands, OpenAI's existing crawler controls remain the working mechanism for its systems. Publishers should keep those rules in place rather than replacing them with the token alone. [S1][S3][S4][1][3][4][5]

Google aims to support the control alongside other AI providers in 2027. Its crawler documentation narrows that window to early 2027. Microsoft gave the firmest date of the four. Its Bing crawler will start honouring the new preferences as it crawls from 1 February 2027. A page's setting updates only when it is recrawled. [S3][S4][S5][1][3][4][5]

Google's mapping to Google-Extended

Google's announcement carries one mapping rule that every site owner should understand before editing a file. If a site uses the shared token to opt out of training and does not explicitly mention Google-Extended, Google plans to treat that as if the publisher had disallowed Google-Extended. Company-specific controls will continue to function alongside the new joint signal. An explicit Google-Extended rule still wins where one exists. [S3][S4][2][3][4][5][10]

The mapping works in the other direction too. Sites that explicitly allow Google-Extended remain eligible to appear in Gemini Apps responses and receive clicks through grounding. A publisher that wants grounding traffic can say yes to Google-Extended while saying no to the shared token or the reverse. Google confirmed it will work that way. [S2][S3][2][3][4][5][10]

Google also links the proposal to European regulation. Its post calls the proposal consistent with the EU Code of Practice for general-purpose AI models under the AI Act. That code recognises robots.txt as a main way to state machine-readable preferences. The link does not turn the token into law. [S5][S10][2][3][4][5][10]

Microsoft's separate grounding control

Microsoft went further than a single token. Its Bing announcement pairs the shared training preference with a separate page-level AI Grounding setting. Temporary transition settings open in Bing Webmaster Tools on 15 December 2026. Those temporary settings expire on 30 September 2027. [S5][S10][4][5][10]

The grounding control answers a question publishers have asked since AI answers started citing their pages. Training changes a model permanently. Grounding shapes one answer at a time. A publisher might accept the second while refusing the first or the reverse. Microsoft's split lets a site express that choice page by page rather than site by site. It also confirms that opting out of training does not affect eligibility for regular Bing results. [S4][S5][4][5][10]

Why it is still a draft not a standard

The document behind the announcement is an individual Internet-Draft. The IETF datatracker states that an individual draft is not endorsed by the IETF and has no formal standing in the standards process. Internet-Drafts are working documents valid for a maximum of six months. They may be updated, replaced or obsoleted at any time. [S2][S3][S4][2][3][4][5][6][11][12]

That status is how internet standards begin. The AI Preferences working group is chaired by Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco. Participants include Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons. [S5][S6][2][3][4][5][6][11][12]

Meta's position shows how far consensus still has to travel. Meta communications head Andy Stone called a common signal an important step. The Verge reported that Meta did not confirm whether it will honour the token. One of the largest crawlers on the web has therefore welcomed the idea without committing to it. [S11][S12][2][3][4][5][6][11][12]

What a site owner should do now

The practical advice follows directly from the uneven timelines. Keep every existing per-company rule in place today. The company-specific rules are what works now. The shared token does nothing yet at most of the companies that backed it. [S3][S4][1][2][3][4]

Add the shared token group alongside the old rules so coverage arrives as each company ships support. A site that wants a full opt-out adds the token group with a Disallow for all paths. A site that wants a partial opt-out adds Allow lines for the sections it wants to share. Test the file after editing it. A single mistyped underscore breaks the match because consumers must match the token exactly apart from case. [S2][S3][1][2][3][4]

Then set a calendar reminder for each company's ship date. Check Anthropic's rollout as it proceeds. Watch for OpenAI's implementation update in the coming months. Recheck Google in early 2027 and Microsoft from 1 February 2027. Verify after each date that the company's crawler documentation still says what its announcement said. [S1][S3][S4][1][2][3][4]

Site owners should also separate their decisions. Decide training first. Decide grounding second. Decide search visibility third. A publisher that wants to refuse all three needs the training token plus the grounding controls plus the search controls for each product. (For the basics of how names reach servers, see our explainer: [DNS explained: how domain names find websites](https://technogalaxy.tech/blogposts/dns-how-domain-names-find-websites/). For the wider security posture behind trusting no single control, see [Zero trust starts with the resource—not the office network](https://technogalaxy.tech/blogposts/zero-trust-resource-access-explained/). For how retrieval changes what AI answers can do, see [Why retrieval makes AI more useful](https://technogalaxy.tech/blogposts/why-retrieval-makes-ai-more-useful/).) [S2][S3][S4][1][2][3][4]

What remains unconfirmed

Several questions have no answer in the 30 October 2026 announcements. Anthropic has published no firm completion date for its rollout. OpenAI has published no implementation details beyond its coming-months window. Google has published no exact day inside early 2027. [S1][S3][S4][S5][1][2][3][4][5][7]

Enforcement is the largest open question. Robots.txt has always worked on trust. The AI Preferences working group charter excludes technical enforcement of preferences. A published preference is not a technical barrier that prevents unwanted use. It binds only the companies that choose to follow it. Publishers who need legal force still need contracts, licences and regulation. The token is a coordination tool. It is not a lock. [S3][S7][1][2][3][4][5][7]

The draft itself may change before it becomes a standard. Its expiry date of 3 May 2027 forces a decision: revise it, adopt it or let it lapse. Publishers who build tooling around the token should build it to accept revisions. The token name, the precedence rules and the vocabulary may all move as the working group reaches consensus. Recheck the file in May 2027 against the current draft version. [S2][S3][1][2][3][4][5][7]