Should you block AI crawlers? It is three decisions, not one

Somebody sends you a snippet for your robots.txt, you paste it in, and you have now made a decision you were never actually shown. The snippet named a company. The decision was about three different programs that company runs, doing three different jobs, only one of which has anything to do with training a model.

The three jobs are worth learning once, because after that the whole subject stops being a matter of opinion. One fetcher reads your text so a future model can learn from it. One reads your page so it can be listed in that assistant’s index. One reads your page because somebody is sitting in front of the assistant right now, asking about you. Blocking the first costs you nothing you can measure. Blocking the third is turning away a person who has already found you.

This post is the decision, not the ideology. If you want the concepts first — what an AI referral is, how it differs from a crawl, and what any analytics tool can honestly see — start with our complete guide to AI referral traffic.

Every row below is generated by the same function our ingest path uses to label a crawler hit, so the table cannot drift from the product. Where a company has fewer than three rows, that is the company’s published set — not an omission here.

Two of those rows are not crawlers at all in the ordinary sense. Google-Extended and Applebot-Extended are opt-out tokens: the fetching is done by the companies’ general search crawlers, and the token governs what may be done with the result afterwards. Google’s documentation is explicit that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal — wording it updated in April 2025, presumably because so many people assumed otherwise.

OpenAI states the same principle for its own three: the settings are independent, so a publisher can disallow GPTBot to stay out of training while leaving OAI-SearchBot allowed and remaining eligible to appear in ChatGPT’s search results. Anthropic’s help centre documents its three robots the same way and says robots.txt is the supported opt-out, because it publishes no IP ranges to filter on.

1. The token list is out of date. Snippets still circulating tell you to disallow Claude-Web and anthropic-ai. Anthropic’s current documentation names three robots and neither of those is among them, so a file built from an old snippet is refusing something nobody sends while allowing the fetchers actually knocking. Our own classifier still matches the retired tokens, on the grounds that a stale crawler or an impersonator can put anything on the wire — but matching them is a courtesy, not a live signal.
2. “Blocking the AI bot protects my Google ranking.” It has nothing to do with it in either direction. Googlebot is what crawls for Search; Google-Extended only governs generative use of what was already crawled. Disallowing it does not make you less visible in Search, and allowing it does not make you more so.
3. “If I disallow it, I disappear from the answers.” Not reliably. Perplexity’s help centre says that when a site disallows PerplexityBot it will not index that site’s text — but that the domain, a headline and a brief factual summary may still appear. Blocking is a control over crawling, not a right to absence.
4. “The Content-Signal line handles it.” Cloudflare launched the Content Signals Policy on 24 September 2025 and applied it across millions of domains, which is why your file may already contain a line you never typed. It expresses a preference in a machine-readable form. It is not a directive anyone has committed to obeying: in October 2025 Google’s John Mueller was reported as saying the new directive has no effect for any crawler or LLM. Worth having. Not worth relying on.

Content delivery networks can prepend a managed block to your robots.txt, and some do it by default on new domains. The rules are perfectly reasonable defaults. They are simply somebody else’s decision, sitting above yours, and crawlers read the whole merged file.

When we audited this site on 28 August 2026 we found exactly that: coruve.com was serving a managed block ahead of the one this application generates, and that block disallowed the major AI fetchers across the board. A product whose main differentiator is telling you which assistants send you visitors was, at that moment, unreadable to those assistants — through a default nobody had chosen. We are recording it here because it is the most likely thing to be true of your site too, and it costs one request to find out. We turned the managed layer off the same week; the file coruve.com serves today is the one in this repository.

So before you edit anything, open your own site’s robots.txt in a browser and read what is served rather than what is committed. If there is a block you did not write, that is where the real policy lives, and editing your repository will change nothing.

The middle position is the one most people mean when they say they want to “block AI”, and it is the one the snippets do worst, because a snippet organised by company cannot express it. Organised by job, it is four lines and a comment:

# Refuse training. Stay citable.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: cohere-ai
Disallow: /

# Left allowed on purpose: OAI-SearchBot, Claude-SearchBot,
# PerplexityBot (index) and ChatGPT-User, Claude-User,
# Perplexity-User (fetched when a person asks about you).

Whatever you choose, write the reason in a comment beside it. The next person to edit that file will be you, in eighteen months, with no memory of why PerplexityBot is allowed and CCBot is not.

The trade is: your text feeding a model and an index, in exchange for being citable and therefore clickable. The second half is measurable. The first half mostly is not, and no analytics tool will change that.

Here is where the line sits, precisely:

• Visits from assistants are measurable. A person clicking a link in ChatGPT or Perplexity arrives in a real browser and can be counted like any other channel — what they read, whether they come back, whether they buy.
• Crawler activity is only a floor. Analytics that runs as a script in the page cannot see a fetcher that requests your HTML and never runs JavaScript, which is what most of them do. Any tool that shows you a confident share of “AI crawler traffic” from a page script is dividing a fraction it cannot size by a number it can. Server or CDN logs are the right instrument for that question.
• Training use is not measurable at all. Whether your text ended up in a model is not a thing your website can observe. Anyone selling you a number for it is selling you a guess.

This is also why you will not find an industry statistic in this post. The pages currently ranking for this question recycle a conversion multiplier for AI traffic and a share of bot volume attributed to training crawlers, and neither figure traces back to a published method. Our test suite refuses that shape of claim on principle, which is a constraint we would rather have than the sentence.

What you can do instead is measure your own site, which is the only number that was ever going to inform your decision. Give it a few weeks either side of a change and compare assistant referrals against your other channels. The free AI traffic checker reads seven days of traffic without an account and prints counts rather than shares for the crawler half, for exactly the reason above. The methodology, including where it goes blind, is set out in AI crawlers vs AI referrals.

Everything above will move. Operators add fetchers, retire tokens and rename products; two of the four corrections in this post exist because advice that was accurate in 2024 was repeated into 2026 without being re-read. A robots.txt with no review date is a file that quietly becomes wrong.

Three lines in a calendar, twice a year:

1. Fetch your own robots.txt and read what is actually served, including anything your CDN prepends.
2. Re-read the operators’ own documentation for the fetchers you name. Company documentation is the only source that is authoritative about that company, and it is usually one page.
3. Look at what assistant referrals did since last time, and decide whether the position you took is still the one you want.

That is the whole discipline. It is more than most sites do, and it is the difference between a policy you hold and a default you inherited.