If you want to appear in AI answers, allow the search/citation crawlers: OAI-SearchBot (ChatGPT search)[1], Claude-SearchBot (Claude)[2], PerplexityBot (Perplexity)[3], Bingbot (Copilot) and Googlebot (AI Overviews and AI Mode). Training crawlers (GPTBot, ClaudeBot, CCBot) are a separate decision; blocking them limits use of your content in model training but doesn’t directly cut visibility in search answers. Two exceptions: Google-Extended[4] covers grounding in Gemini apps as well as Gemini training, and meta-externalagent[5] does direct indexing as well as training. For the shared __usage_ai-training token announced on October 30, 2026, see the section below.
What does each crawler do?
| Crawler | Company | Purpose | Our advice |
|---|---|---|---|
OAI-SearchBot | OpenAI[1] | ChatGPT search results | Allow |
ChatGPT-User | OpenAI | User-requested visits | Allow |
GPTBot | OpenAI | Model training | Your choice |
Claude-SearchBot | Anthropic[2] | Claude search index | Allow |
Claude-User | Anthropic | User-requested visits | Allow |
ClaudeBot | Anthropic | Model training | Your choice |
PerplexityBot | Perplexity[3] | Perplexity search index | Allow |
Perplexity-User | Perplexity | User-requested visits | Allow |
Googlebot | Search + AI Overviews / AI Mode | Allow | |
Google-Extended | Google[4] | Gemini training and grounding in Gemini apps | Your choice |
Bingbot | Microsoft | Bing index → Copilot | Allow |
meta-webindexer | Meta[5] | Meta AI search quality | Allow |
meta-externalagent | Meta | Training / direct indexing | Your choice |
CCBot | Common Crawl | Open web archive (training data for many models) | Your choice |
Opting out of AI Overviews and AI Mode isn’t done in robots.txt but with the Search generative AI control in Search Console; according to Google, it doesn’t affect ranking elsewhere in Search or model training[10].
Sample robots.txt
A configuration open to search and citation crawlers and closed to training crawlers (adjust to your own preference):
# Search and citation crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /
# Model training crawlers: blocked (optional)
# Note: blocking Google-Extended also removes you as a source in Gemini apps
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /
# Everything else
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlThis site’s own file is open to all AI crawlers: robots.txt — because being quoted is the point.
Frequently asked questions
01Should I block GPTBot?
It’s a choice. GPTBot is for model training; according to OpenAI, ChatGPT search visibility is governed by OAI-SearchBot. If you don’t want your content used for training, block GPTBot and allow OAI-SearchBot.
02Does robots.txt block user-initiated bots?
Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says all three of its bots, including Claude-User, honor robots.txt.
03How do I remove my site from AI Overviews and AI Mode?
Google-Extended in robots.txt doesn’t do this; it doesn’t affect Google Search. Since August 31, 2026, Search Console has a Search generative AI control under Settings: it removes the site from AI Overviews, AI Mode and generative AI features in Discover, usually within a few days. According to Google, it doesn’t affect ranking elsewhere in Search or model training. If visibility is your goal, leave it on.
Sources
- OpenAI — Overview of OpenAI crawlers · accessed 27 September 2026
- Anthropic — Claude crawlers and robots.txt · accessed 27 September 2026
- Perplexity — Perplexity Crawlers · accessed 27 September 2026
- Google — Google’s common crawlers (Google-Extended) · accessed 27 September 2026
- Meta — Meta Web Crawlers · accessed 27 September 2026
- Google Search Central Blog — Supporting a shared AI training control for websites (Oct 30, 2026) · accessed 6 November 2026
- Bing Blog — Your content, your choice: clearer preferences for training and grounding (Oct 2026) · accessed 6 November 2026
- OpenAI — Making it easier for websites to express AI training preferences (Oct 30, 2026) · accessed 6 November 2026
- IETF ai-control — Kevin Kelley (Anthropic), training preference signal draft (Oct 30, 2026) · accessed 6 November 2026
- Search Console Help — Search generative AI control · accessed 6 November 2026