One robots.txt Line to Stop AI Training: What the Shared Opt-Out Means

At the end of October 2026, Google, Microsoft and OpenAI said they would honour one shared signal in the robots.txt file that lets a website say it does not want its content used to train AI models. Several reports also list Anthropic, which proposed the idea; coverage is not consistent on that point, so treat Anthropic's commitment as unconfirmed for now.

What the signal looks like

Instead of blocking each company's crawler by name, a site owner adds one rule for a shared token, written as User-Agent: __usage_ai-training, followed by Disallow: /. That expresses a whole-site preference against AI training in a single place. Anthropic put the proposal to the IETF, the body that writes internet standards, but it is currently an individual draft and not an official standard.

What it cannot do

• It cannot remove content from a model that has already been trained.
• robots.txt is a request, not a lock. It works only for companies that choose to respect it.
• It covers training only. Search features such as AI Overviews are controlled separately, through a setting in Google Search Console.

When it starts to matter

Timing differs by company. Microsoft has said it will take the new entries into account when crawling from 1 February 2027, and has talked about a separate setting for AI grounding, the step where live pages are pulled into chat answers. OpenAI has said it aims to honour the signal within months. Google's timeline is described as 2027. Meta has not committed.

What a site owner can do today

Decide first what you want. Blocking training does not stop your pages appearing in normal search results, and many publishers want to be quoted in AI answers while refusing training. If you do want to opt out, add the rule, keep your existing Googlebot and Bingbot rules unchanged, and check the file loads at yourdomain.com/robots.txt. Prompt Mandi keeps its own robots.txt simple for the same reason: search visibility and AI training are separate choices.

These details come from press reports of the announcement; the draft itself is the place to confirm exact wording before you rely on it.