meta-externalagent
Meta's crawler for AI training and indexing. Blocking it excludes your content from Meta AI training and direct indexing.
| Operator | Meta |
|---|---|
| What it does | Model training |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | AI model training |
| Has a crawler | Yes |
| Last verified | 13 August 2026 |
What blocking it costs you
What to know
Meta publishes no IP address list for its crawlers, which means nobody — us included — can verify a request claiming this user agent actually came from Meta. Historical guidance points at their autonomous system, which is weaker evidence than the signed IP lists OpenAI, Anthropic and Google publish.
There is a related token worth knowing about: meta-externalfetcher, which fetches links on a user's request for Meta's agentic AI, and which Meta's own documentation says may bypass robots.txt.
The robots.txt rules
To block meta-externalagent:
User-agent: meta-externalagent
Disallow: /To allow it explicitly — worth doing when your file also contains a blanket User-agent: * disallow, since the most specific matching group wins and a token with no group of its own falls back to the wildcard:
User-agent: meta-externalagent
Allow: /If neither rule is present, meta-externalagent is allowed. That is the default, and it is why an accidental block is nearly always something that was added rather than something that was forgotten — usually a snippet copied from a blog post that listed more tokens than the author intended to block.
Is it blocked on your site?
Your robots.txt is only the half you can read. A CDN or bot-management rule can turn meta-externalagent away while your file still says it is welcome, and that mismatch is invisible from the front of the site. We check both layers separately and tell you which one is doing what.
We request two things: your robots.txt and your homepage. No other pages, ever.