AI Crawlers & robots.txt: GPTBot, ClaudeBot, PerplexityBot

✳ Short answer

If you want to appear in AI answers, allow the search/citation crawlers: OAI-SearchBot (ChatGPT search)^[1], Claude-SearchBot (Claude)^[2], PerplexityBot (Perplexity)^[3], Bingbot (Copilot) and Googlebot (AI Overviews). Training crawlers (GPTBot, ClaudeBot, Google-Extended^[4], meta-externalagent^[5], CCBot) are a separate decision; blocking them limits use of your content in model training but doesn’t directly cut visibility in search answers.

What does each crawler do?

Crawler	Company	Purpose	Our advice
`OAI-SearchBot`	OpenAI^[1]	ChatGPT search results	Allow
`ChatGPT-User`	OpenAI	User-requested visits	Allow
`GPTBot`	OpenAI	Model training	Your choice
`Claude-SearchBot`	Anthropic^[2]	Claude search index	Allow
`Claude-User`	Anthropic	User-requested visits	Allow
`ClaudeBot`	Anthropic	Model training	Your choice
`PerplexityBot`	Perplexity^[3]	Perplexity search index	Allow
`Perplexity-User`	Perplexity	User-requested visits	Allow
`Googlebot`	Google	Search + AI Overviews / AI Mode	Allow
`Google-Extended`	Google^[4]	Gemini training and grounding in Gemini apps	Your choice
`Bingbot`	Microsoft	Bing index → Copilot	Allow
`meta-webindexer`	Meta^[5]	Meta AI search quality	Allow
`meta-externalagent`	Meta	Training / direct indexing	Your choice
`CCBot`	Common Crawl	Open web archive (training data for many models)	Your choice

Sample robots.txt

A configuration open to search and citation crawlers and closed to training crawlers (adjust to your own preference):

# Search and citation crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: meta-webindexer
Allow: /

# Model training crawlers: blocked (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# Everything else
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

This site’s own file is open to all AI crawlers: robots.txt — because being quoted is the point.

robots.txt is clean, but the bot may still be locked out

CDN / firewall: “block AI bots” settings and bot-management rules at providers like Cloudflare work independently of robots.txt. Check your dashboard.
JavaScript-dependent content: even when the bot gets in, text loaded by JavaScript often looks like an empty page.
Login walls and cookie barriers: pop-ups that fully cover content can break parsing.
Spoofed bot traffic: bad actors can fake these names; companies like OpenAI and Perplexity publish IP ranges for verification^[1]^[3].

Check all of it in seconds: GEO Readiness Check.

Frequently asked questions

01Should I block GPTBot?

It’s a choice. GPTBot is for model training; according to OpenAI, ChatGPT search visibility is governed by OAI-SearchBot. If you don’t want your content used for training, block GPTBot and allow OAI-SearchBot.

02Does robots.txt block user-initiated bots?

Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says all three of its bots, including Claude-User, honor robots.txt.

Sources

OpenAI — Overview of OpenAI crawlers · accessed 27 September 2026
Anthropic — Claude crawlers and robots.txt · accessed 27 September 2026
Perplexity — Perplexity Crawlers · accessed 27 September 2026
Google — Google’s common crawlers (Google-Extended) · accessed 27 September 2026
Meta — Meta Web Crawlers · accessed 27 September 2026

AI crawlers and robots.txt: which to allow, which to block?