facebookexternalhit: what it does, and what blocking it means

Meta's share-preview crawler, not an AI one. Blocking it is why a link to your site posts as a bare URL with no title, image or description.

What happens if you block facebookexternalhit?

Is facebookexternalhit an AI crawler?

This token has nothing to do with AI training or with Meta AI citations. It fetches a page when somebody shares a link to it, so the post can render a card. Block it and every link to you on Facebook, Instagram, WhatsApp and Messenger degrades to a naked URL, usually discovered weeks later by someone wondering why their posts stopped performing.

It is in this registry for one reason: it is the fifth token on Meta's crawler documentation page, and a "block everything Meta runs" list copied from that page takes it along with the training crawler. The cost of that mistake lands entirely on the site owner's own social referrals.

Meta notes it might bypass robots.txt when performing security or integrity checks, so a rule here is honoured in the ordinary case rather than guaranteed in every case.

How do you spot facebookexternalhit in your logs?

To find facebookexternalhit in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Anyone can send that string, and Meta publishes no address ranges for facebookexternalhit, so there is no way to confirm from outside that a request carrying it is really theirs.

How do you block or allow facebookexternalhit in robots.txt?

To block facebookexternalhit:

User-agent: facebookexternalhit
Disallow: /

To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:

User-agent: facebookexternalhit
Allow: /

With neither rule, facebookexternalhit is allowed by default.

Not sure how facebookexternalhit fits with the others? Build the whole file, or see which AI crawlers to allow.

Is your site blocking facebookexternalhit?

A CDN or bot-management rule can still block facebookexternalhit even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Official sources

• Meta’s crawler documentation

How is facebookexternalhit different from Meta’s other crawlers?

← All AI crawlers