Block AI Training, Allow AI Search: A robots.txt Guide

You can stay in AI search results and still opt out of the AI training uses that OpenAI and Google name, because each company gives the two jobs separate controls. In robots.txt, allow OAI-SearchBot and disallow GPTBot. For Google, keep Googlebot allowed and disallow Google-Extended. Google's AI Overviews and AI Mode have their own switch in Search Console.

Here is each control as its owner documents it today, so you can block AI training, allow AI search and know which switch does which. Smarter Revolution is a US team that builds AI into a company's real workflows and then runs it for them. This post covers OpenAI and Google only, because those are the crawler pages we checked for it. Whether to opt out of training is a business decision for you, and nothing here is legal advice.

Why are AI search and AI training separate choices?

Because each company gives them separate controls. OpenAI's crawler overview describes two robots.txt tags that work independently. Where a site allows both, OpenAI says it may run a single crawl and use the results for both purposes.

Google draws the line elsewhere. Google's list of common crawlers, last updated October 30, 2026, says rules addressed to Googlebot "affect Google Search (including Discover and all Google Search features)". Google-Extended is a separate token for Gemini, and the same page is explicit: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

That leaves three controls to set on purpose, not two:

• OAI-SearchBot and GPTBot in robots.txt, for OpenAI.
• Google-Extended in robots.txt, for Gemini training and grounding.
• The Search generative AI control in Search Console, for AI features inside Google Search.

What do OpenAI's crawlers each do?

OAI-SearchBot crawls for ChatGPT's search features, and GPTBot is the one that gathers content which may go into training OpenAI's foundation models. Blocking one leaves the other untouched. If you disallow OAI-SearchBot, OpenAI says your site will not appear in ChatGPT search answers, although a plain navigational link to it can still show up. If you disallow GPTBot, OpenAI reads that as a statement that your content should not be used to train its generative AI foundation models. So to keep search and refuse training, write one group that allows OAI-SearchBot and a second group that disallows GPTBot. OpenAI also recommends allowing requests from the IP ranges it publishes for OAI-SearchBot, so check that a firewall is not blocking what robots.txt allows. After a robots.txt edit, OpenAI puts the delay for search results at roughly a day.

Two more OpenAI agents matter for this decision:

• ChatGPT-User may open a page when a person asks ChatGPT, or a custom GPT, a question. It does not crawl the web on its own. Because a user starts these visits, OpenAI says robots.txt might not govern them, and that this agent plays no part in deciding what appears in search.
• OAI-AdsBot goes only to the landing pages of ads placed on ChatGPT, and OpenAI says the data it collects is not used to train its foundation models.

What does Google-Extended control, and what does it leave alone?

The common crawlers page gives Google-Extended two jobs for content Google crawls from your site. One is training future generations of the Gemini models behind Gemini Apps and the Vertex AI API for Gemini. The other is grounding, which Google defines as handing the model content from the Google Search index when it is prompted, in Gemini Apps and in Grounding with Google Search on Vertex AI.

In our view, the grounding part is the one to weigh. Disallowing Google-Extended is not only a training opt-out. It also keeps your content out of grounding in Gemini Apps, where your pages could otherwise feed an answer. Decide whether that trade is worth it.

Google-Extended has no user agent string of its own. Google crawls with its existing user agents and treats the robots.txt line as a control, so no Google-Extended visitor will appear in your logs.

The page also carries a note: "Google intends to support the AIPREF shared machine-readable training preference signal" in early 2027. That is a stated plan, not a control you can use today, so Google-Extended remains the token the page documents now.

Where does the Search Console switch fit?

Google updated its guide to how you control what you share with Google Search on October 21, 2026. It now lists a site-wide option: "You can manage your site's inclusion in generative AI features on Google Search, including AI Overviews and AI Mode, by using the Search generative AI control in Search Console."

Google's help page for the control says it reached every website on August 31, 2026, and that it also covers generative AI features in Discover. It sits under Settings in Search Console. Excluding your site removes its links and content from those features, stops them grounding answers in your pages, and ends traffic and impressions from them. Google says the setting plays no part in ranking or inclusion anywhere else in Search.

Three points on that help page shape the decision:

• It is not a training control. Google says the switch has no effect on AI training, and it points to Google-Extended for limiting training of the models that write those answers.
• Search may still use your content in other ways. Google says your pages can still feed the systems behind Search in general, and gives language understanding as one example.
• Leaving Search entirely is a different tool. For that, Google names noindex.

So the Google half of the setup is: Googlebot able to crawl, Google-Extended disallowed, and the Search Console control set to include your site. Include is Google's default. A property that sits under a parent property takes the parent's setting instead. Our post on the Generative AI performance report shows how to see whether those features list your pages at all.

What does the robots.txt look like?

Here is an example. Each row is its own group in the file:

In Google's robots.txt specification, last updated October 23, 2026, a path of a single slash "Matches the root and any lower level URL." Two more rules matter when a file already has groups. First, "Only one group is valid for a particular crawler." Google's crawlers pick the group with the most specific matching user agent and ignore the others. Second, "User agent specific groups and global groups (*) are not combined."

The example leaves Googlebot out on purpose. Giving it a group of its own would mean rules you wrote under User-agent: stop applying to it. Instead, check that whichever group covers Googlebot today does not block pages you want in Search. That specification covers Google's crawlers only; OpenAI's page does not say how its crawlers resolve groups. If your User-agent: group keeps crawlers out of private folders, our suggestion is to repeat those lines inside the OAI-SearchBot group too.

How long until a change takes effect?

• OpenAI search: about 24 hours after a robots.txt edit, per OpenAI.
• Google's crawlers: Google "generally caches the contents of robots.txt file for up to 24 hours", per its specification, and can keep a copy longer when it cannot fetch a fresh one.
• The Search Console switch: excluding a site generally takes a few days, per Google's help page, and some content can take longer because of caching across Google's systems.

How do you check the rules are working?

Open your own domain followed by /robots.txt in a browser and read what is actually live. Then treat log entries with care. Google warns: "The HTTP user agent string can be spoofed." Google's crawler page links to a way to verify a visitor. OpenAI publishes an IP address list for each of its agents, so you can compare a visit against it.

Robots.txt is not the only gate. Our guide on getting cited by ChatGPT covers other places a crawler can be stopped, such as a host or CDN setting. Our free website authority check fetches your robots.txt and sitemap along with your homepage, and robots directives are among its technical checks.

How does Smarter Revolution help?

Through LLM Search Optimization. The work is an audit, schema, answer-first content and a monthly citation scoreboard, and it does not guarantee placement. The training decision stays yours. If you want to know what AI models say about your company before changing anything, our free AI visibility audit puts two questions to up to three models: which companies they recommend in your industry, and what they know about you.

Frequently asked questions

Will blocking GPTBot remove my site from ChatGPT search?

No. OpenAI treats GPTBot and OAI-SearchBot as independent settings. Search depends on OAI-SearchBot, so a site can disallow GPTBot and still be eligible for ChatGPT search answers.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode have their own switch, the Search generative AI control in Search Console.

Can I stop ChatGPT opening a page when a user asks for it?

Not reliably with robots.txt. OpenAI says ChatGPT-User visits start with a person, so robots.txt might not govern them.

Does the Search Console switch stop Google training on my content?

No. Google's help page says the switch does not affect AI training, and it points to Google-Extended for that.

The bottom line

OpenAI and Google both let you separate AI search from AI training, but the switches sit in different places. For OpenAI, allow OAI-SearchBot and disallow GPTBot. For Google, keep Googlebot able to crawl, disallow Google-Extended knowing it also covers grounding in Gemini Apps, and leave the Search Console control on include. Reread the pages linked here before you edit the file, because Google updated three of them in October 2026.

If you want help making your website easier for AI tools to cite, a free 30-minute scoping call comes first, and you then get the exact scope in writing. We charge one fixed price to build your system and one monthly fee to run and improve it, both agreed in writing before work starts. Timelines run in weeks, not quarters, and the count starts once we have the access and approvals we need. To talk it through, book a call.

Sources

• OpenAI, "Overview of OpenAI Crawlers", no date shown, opened November 13, 2026: developers.openai.com
• Google Crawling Infrastructure, "Google's common crawlers", last updated October 30, 2026, opened November 13, 2026: developers.google.com
• Google Crawling Infrastructure, "How Google interprets the robots.txt specification", last updated October 23, 2026, opened November 13, 2026: developers.google.com
• Google Search Central, "Control what you share with Google Search", last updated October 21, 2026, opened November 13, 2026: developers.google.com
• Google Search Console Help, "Search generative AI control", no date shown, opened November 13, 2026: support.google.com