GUIDE № 06LAST UPDATED · 7 MIN READ

Technical

AI crawlers and robots.txt: which to allow, which to block?

AI companies run separate crawlers for separate purposes: one to train models, one to cite sources in search answers, one for when a user asks to open a link. Blocking them all in one line can quietly remove you from AI search too.

Written by: Vextoo Geo Agency Team · Last updated:

✳ Short answer

If you want to appear in AI answers, allow the search/citation crawlers: OAI-SearchBot (ChatGPT search)[1], Claude-SearchBot (Claude)[2], PerplexityBot (Perplexity)[3], Bingbot (Copilot) and Googlebot (AI Overviews and AI Mode). Training crawlers (GPTBot, ClaudeBot, CCBot) are a separate decision; blocking them limits use of your content in model training but doesn’t directly cut visibility in search answers. Two exceptions: Google-Extended[4] covers grounding in Gemini apps as well as Gemini training, and meta-externalagent[5] does direct indexing as well as training. For the shared __usage_ai-training token announced on October 30, 2026, see the section below.

What does each crawler do?

CrawlerCompanyPurposeOur advice
OAI-SearchBotOpenAI[1]ChatGPT search resultsAllow
ChatGPT-UserOpenAIUser-requested visitsAllow
GPTBotOpenAIModel trainingYour choice
Claude-SearchBotAnthropic[2]Claude search indexAllow
Claude-UserAnthropicUser-requested visitsAllow
ClaudeBotAnthropicModel trainingYour choice
PerplexityBotPerplexity[3]Perplexity search indexAllow
Perplexity-UserPerplexityUser-requested visitsAllow
GooglebotGoogleSearch + AI Overviews / AI ModeAllow
Google-ExtendedGoogle[4]Gemini training and grounding in Gemini appsYour choice
BingbotMicrosoftBing index → CopilotAllow
meta-webindexerMeta[5]Meta AI search qualityAllow
meta-externalagentMetaTraining / direct indexingYour choice
CCBotCommon CrawlOpen web archive (training data for many models)Your choice

Opting out of AI Overviews and AI Mode isn’t done in robots.txt but with the Search generative AI control in Search Console; according to Google, it doesn’t affect ranking elsewhere in Search or model training[10].

Sample robots.txt

A configuration open to search and citation crawlers and closed to training crawlers (adjust to your own preference):

# Search and citation crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /

# Model training crawlers: blocked (optional)
# Note: blocking Google-Extended also removes you as a source in Gemini apps
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# Everything else
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

This site’s own file is open to all AI crawlers: robots.txt — because being quoted is the point.

New: one line for your training preference (__usage_ai-training)

On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will support stating a training preference to several providers at once with a single robots.txt group. __usage_ai-training is not a crawler name but a policy token, based on the IETF AIPREF work[6][7][8][9].

# To participating providers: do not use my content for model training
User-agent: __usage_ai-training
Disallow: /
  • Status (November 6, 2026): Anthropic says it has started implementing support[9]; Microsoft starts on February 1, 2027[7]; Google is aiming for 2027[6] and OpenAI for the coming months[8]. Existing GPTBot, ClaudeBot and Google-Extended rules still apply.
  • Watch out on Google: if you opt out of training with the token and don’t name Google-Extended, Google plans to treat it as if you had disallowed Google-Extended. To stay eligible as a source in Gemini apps, you need to allow Google-Extended explicitly. Because Google-Extended covers both Gemini training and grounding in Gemini apps, you can’t choose the two separately on Google[4][6].
  • A separate preference on Microsoft: Microsoft adds a page-level preference for whether your content is used as a source (grounding) in AI answers. Microsoft itself says turning it off will reduce brand visibility in AI experiences; it doesn’t change eligibility for traditional Bing results. After March 1, 2027, NoArchive and NoCache will no longer be read as a training or grounding preference[7].

robots.txt is clean, but the bot may still be locked out

  • CDN / firewall: “block AI bots” settings and bot-management rules at providers like Cloudflare work independently of robots.txt. Check your dashboard.
  • JavaScript-dependent content: even when the bot gets in, text loaded by JavaScript often looks like an empty page.
  • Login walls and cookie barriers: pop-ups that fully cover content can break parsing.
  • Spoofed bot traffic: bad actors can fake these names; companies like OpenAI and Perplexity publish IP ranges for verification[1][3].

Check all of it in seconds: GEO Readiness Check.

Frequently asked questions

01Should I block GPTBot?

It’s a choice. GPTBot is for model training; according to OpenAI, ChatGPT search visibility is governed by OAI-SearchBot. If you don’t want your content used for training, block GPTBot and allow OAI-SearchBot.

02Does robots.txt block user-initiated bots?

Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says all three of its bots, including Claude-User, honor robots.txt.

03How do I remove my site from AI Overviews and AI Mode?

Google-Extended in robots.txt doesn’t do this; it doesn’t affect Google Search. Since August 31, 2026, Search Console has a Search generative AI control under Settings: it removes the site from AI Overviews, AI Mode and generative AI features in Discover, usually within a few days. According to Google, it doesn’t affect ranking elsewhere in Search or model training. If visibility is your goal, leave it on.

Sources

  1. OpenAI — Overview of OpenAI crawlers · accessed 27 September 2026
  2. Anthropic — Claude crawlers and robots.txt · accessed 27 September 2026
  3. Perplexity — Perplexity Crawlers · accessed 27 September 2026
  4. Google — Google’s common crawlers (Google-Extended) · accessed 27 September 2026
  5. Meta — Meta Web Crawlers · accessed 27 September 2026
  6. Google Search Central Blog — Supporting a shared AI training control for websites (Oct 30, 2026) · accessed 6 November 2026
  7. Bing Blog — Your content, your choice: clearer preferences for training and grounding (Oct 2026) · accessed 6 November 2026
  8. OpenAI — Making it easier for websites to express AI training preferences (Oct 30, 2026) · accessed 6 November 2026
  9. IETF ai-control — Kevin Kelley (Anthropic), training preference signal draft (Oct 30, 2026) · accessed 6 November 2026
  10. Search Console Help — Search generative AI control · accessed 6 November 2026

Next step

Where does your brand stand inside AI answers?

Our free preliminary scan maps your brand across 6 AI engines — we come back within 48 hours.