meta-externalagent: what it does, and what blocking it means
Meta's crawler for AI training and indexing. Blocking it excludes your content from Meta AI training and direct indexing.
| Operator | Meta |
|---|---|
| What it does | Model training |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | AI model training |
| Has a crawler | Yes |
| Last verified | 19 September 2026 |
| Next review | 19 December 2026. We re-check every entry against the operator’s documentation each quarter. |
What happens if you block meta-externalagent?
Blocking it affects
- Meta AI training and direct indexing
It does not affect
- Link previews when people share you
- The landing pages behind your Meta ads
Can you tell a real meta-externalagent request from a fake one?
Meta publishes no IP list for its crawlers, so nobody, us included, can verify that a request carrying this user agent really came from Meta. The historical guidance points at their autonomous system, which is weaker evidence than the signed IP lists OpenAI, Anthropic and Google publish.
Two related tokens are worth knowing about. meta-externalfetcher fetches links on a user's request for Meta's agentic AI, and Meta's own documentation says it may bypass robots.txt. meta-externalads reads the landing pages behind ads you are paying for. Neither is a training crawler, and a rule written to stop training will catch both if it matches on the meta- prefix.
How do you spot meta-externalagent in your logs?
To find meta-externalagent in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)Anyone can send that string, and Meta publishes no address ranges for meta-externalagent, so there is no way to confirm from outside that a request carrying it is really theirs.
Worth knowing about meta-externalagent
Without a published IP list, the usual fallback is checking that a request comes from Meta's own network, AS32934. That is weaker evidence than a published list, so we treat it as a medium-confidence check.
How do you block or allow meta-externalagent in robots.txt?
To block meta-externalagent:
User-agent: meta-externalagent
Disallow: /To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:
User-agent: meta-externalagent
Allow: /With neither rule, meta-externalagent is allowed by default.
Not sure how meta-externalagent fits with the others? Build the whole file, or see which AI crawlers to allow.
How many sites block meta-externalagent?
In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 13.6% disallowed meta-externalagent in robots.txt, and 12.4% refused a request identifying as meta-externalagent. For 9.8%, the refusal came from a server whose robots.txt allowed it: a CDN or bot rule deciding something the site’s own file does not say.
Is your site blocking meta-externalagent?
A CDN or bot-management rule can still block meta-externalagent even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Guides about meta-externalagent
- Do the top 1,000 sites block AI crawlers?
What the most visited sites say in robots.txt, what their servers actually do, and how often the two disagree.
Official sources
How is meta-externalagent different from Meta’s other crawlers?
| meta-externalagent | meta-webindexer | meta-externalfetcher | meta-externalads | facebookexternalhit | |
|---|---|---|---|---|---|
| Its job | Model training | Search & citations | Fetches when a user asks | Reference | Reference |
| Blocking it affects | Meta AI training and direct indexing | Meta AI answers and citations | Meta AI opening a link someone asks about (it may ignore the rule) | Meta reading the landing pages behind your own ads | Link previews on Facebook, Instagram, WhatsApp and Messenger |
| Blocking it leaves alone | Link previews when people share you; The landing pages behind your Meta ads | Meta AI training | Meta AI training | Meta AI answers and citations; Meta AI training | Meta AI answers and citations; Meta AI training |
| Obeys robots.txt | Yes, per its operator's docs | Yes, per its operator's docs | May bypass, per its operator's docs | Yes, per its operator's docs | Generally, with caveats |
| Verifiable by IP | No, no published ranges | No, no published ranges | No, no published ranges | No, no published ranges | No, no published ranges |
| Last verified | 19 September 2026 | 15 September 2026 | 19 September 2026 | 19 September 2026 | 19 September 2026 |
Which other crawlers do the same job as meta-externalagent?
- GPTBot: Model training, run by OpenAI
- ClaudeBot: Model training, run by Anthropic
- Google-Extended: Model training, run by Google
- Applebot-Extended: Model training, run by Apple
- Bytespider: Model training, run by ByteDance
- CCBot: Open training corpus, run by Common Crawl
- MistralAI-Training: Model training, run by Mistral