ENGINE FILE · E-05MONITORED ENGINE
- Developer
- Anthropic
- Crawlers
Claude-SearchBot(search index) ·ClaudeBot(model training) ·Claude-User(user-requested visits)- robots.txt
- Per Anthropic, all three respect robots.txt (including Crawl-delay)
- Citations
- Source links on answers that used web search
- Traffic trace
claude.aireferral in GA4
✳ Short answer
To appear in Claude’s search answers, don’t block Claude-SearchBot in robots.txt — Anthropic says blocking it may reduce visibility and accuracy in Claude-powered search answers[1]. ClaudeBot is for model training; Claude-User fetches a page when a user asks about it. According to Anthropic, all three honor robots.txt[1].
What do Claude’s three crawlers do?
| Crawler | Purpose | If you block it |
|---|---|---|
Claude-SearchBot | Indexes content to improve Claude’s search results | Visibility in Claude-powered search answers may drop |
ClaudeBot | Collects public content for model training | Your future content is excluded from training data |
Claude-User | Fetches your page when a user asks a question that needs it | Claude can’t retrieve your page for users |
What should you do to appear in Claude?
- Allow
Claude-SearchBotandClaude-Userin robots.txt; decide separately about training viaClaudeBot. - Write concepts as clear definition sentences: “X is a … used for …”. Claude often reuses definition sentences.
- Build focused sections per question instead of long, sprawling pages; keep the heading hierarchy clean.
- Keep brand and service details consistent across the web (entity SEO).
Frequently asked questions
01If I block ClaudeBot, will I disappear from Claude?
ClaudeBot is for model training. Visibility in Claude’s search answers relies on Claude-SearchBot, and reading pages on a user’s request relies on Claude-User. You can block the training bot and allow the other two[1].
02Do Anthropic’s bots respect robots.txt?
According to Anthropic’s help center, all three bots honor robots.txt directives, including the non-standard Crawl-delay[1].
Sources
- Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler? · accessed 27 September 2026