meta-externalfetcher: what it does, and what blocking it means

The fetcher behind Meta's agentic AI, for links a user asks about. Meta's own documentation says it may ignore what you write here.

What happens if you block meta-externalfetcher?

Does meta-externalfetcher obey robots.txt?

This token fetches one page at a time, when somebody asks Meta AI about that page. It does not crawl a site, and it is not collecting training data.

Meta documents it as a fetcher that may bypass robots.txt, on the reasoning that the request came from a person rather than from a crawl. So a rule naming this token records what you want; it does not guarantee what happens. We report it that way rather than letting a line in a file stand in for a control, the same thing Perplexity says about Perplexity-User, and the same reason we say no rule reaches Grok at all.

If your intent is to stay out of Meta's training data, meta-externalagent is the token that does that, and it is documented as honouring robots.txt.

How do you spot meta-externalfetcher in your logs?

To find meta-externalfetcher in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.

meta-externalfetcher/1.1 (+https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)

Anyone can send that string, and Meta publishes no address ranges for meta-externalfetcher, so there is no way to confirm from outside that a request carrying it is really theirs.

How do you block or allow meta-externalfetcher in robots.txt?

To block meta-externalfetcher:

User-agent: meta-externalfetcher
Disallow: /

To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:

User-agent: meta-externalfetcher
Allow: /

With neither rule, meta-externalfetcher is allowed by default.

Not sure how meta-externalfetcher fits with the others? Build the whole file, or see which AI crawlers to allow.

Is your site blocking meta-externalfetcher?

A CDN or bot-management rule can still block meta-externalfetcher even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Official sources

• Meta’s crawler documentation

How is meta-externalfetcher different from Meta’s other crawlers?

← All AI crawlers