ENGINE FILE · E-08GUIDE COVERAGE
- Developer
- Meta
- Surfaces
- WhatsApp, Instagram, Facebook, Messenger and the Meta AI app
- Crawlers
meta-webindexer(Meta AI search quality) ·meta-externalagent(training/indexing) ·meta-externalfetcher(user request)- Special signal
- Facebook page and Instagram profile details
- Monitoring
- Guide coverage — added to measurement on request
✳ Short answer
Meta uses several crawlers for different purposes: meta-webindexer navigates the web to improve Meta AI search result quality, meta-externalagent crawls for uses such as training foundation models or indexing content directly, and meta-externalfetcher fetches individual links at a user’s request[1]. For Meta AI visibility, don’t block meta-webindexer and keep your Facebook/Instagram profiles consistent with your site.
What should you do for Meta AI?
- Allow
meta-webindexerin robots.txt[1]; decide separately about training viameta-externalagent(AI crawler guide). - Make your Facebook page and Instagram profile name, category, address, hours and website link identical to your site.
- Link the profiles to your site’s Organization schema via
sameAs. - Answer frequent questions in clear sentences both on your site and in profile descriptions.
Frequently asked questions
01If I block meta-externalagent, will I disappear from Meta AI?
Per Meta’s documentation, meta-externalagent crawls for uses such as training foundation AI models or improving products by indexing content directly, while meta-webindexer is used to improve Meta AI search result quality[1]. Which bot you block determines which use you opt out of.
Sources
- Meta for Developers — Meta Web Crawlers · accessed 27 September 2026