Anthropic · AI crawler
What is Claude-SearchBot?
Anthropic's search crawler — it reads your page so it can be listed in Claude's index.
On this page
Every AI crawler
What Claude-SearchBot is
Anthropic's search crawler — it reads your page so it can be listed in Claude's index.
Claude-SearchBot reads your page so it can be listed in Claude's index. It is the fetcher that governs whether Claude can find and cite you at all, which makes it the expensive one to refuse.
Until Coruve learned this token in August 2026 it matched only the generic bot rule and landed in "automated, but we cannot say whose". That is the sort of gap this reference exists to make visible: a fetcher nobody has named is a fetcher nobody can count.
Who runs Claude-SearchBot
Coruve attributes this signature to Anthropic.
Coruve files every hit matching claude-searchbot under Claude, Anthropic’s entry in the classifier. That attribution is a reading of the user agent the request sent, which is a claim the client made about itself rather than a proof — Coruve labels it as such everywhere it is reported.
Anthropic publishes all three jobs — a training crawler, a search crawler and an on-demand fetcher — and Coruve recognises each separately. That is what makes it possible to refuse one without refusing the others.
Anthropic's help centre documents Claude-SearchBot as the robot behind Claude's search index, separately from the training and on-demand robots, and names robots.txt as the supported opt-out. (Anthropic’s own documentation, read 31 August 2026.)
How Coruve reads Claude-SearchBot
Generated by the same functions that label a hit as it arrives, so this table cannot drift from the product.
| What Coruve records | Value |
|---|---|
| User agent contains | claude-searchbot |
| Traffic type | crawler |
| Attributed to | Claude (Anthropic) |
| Job | Search crawler — Reads your page so it can be listed in that assistant's index. |
The row that matters most is the second one. Claude-SearchBot is recorded as a crawler, never as a visitor, and Coruve does not add the two together anywhere. A program reading your page is not a customer, and a tool that folds the two into one number is quietly inflating every figure built on top of it.
The other side of that line is an assistant referral: a real person, in a real browser, who clicked a link inside an AI assistant and landed on your site. Those are counted as human visits and reported as their own channel, with the assistant named. Both readings are useful and they answer different questions — how often a machine reads you, and how many people an assistant sends you.
What a page script can and cannot see
Analytics that runs as a script in your page cannot see a fetcher that requests your HTML and never runs JavaScript, which is what most crawlers do. Coruve prints counts rather than shares for the crawler half for exactly that reason. Your server or CDN logs are the complete record; AI crawlers vs AI referrals sets out where every method goes blind, including ours.
Naming Claude-SearchBot in robots.txt
The exact token, in both directions. Copy the one that matches the decision you actually want.
robots.txt matches a fetcher by its product token, so the spelling below is the part that has to be right. Put the reason in a comment beside it: the next person to edit that file will be you, in eighteen months, with no memory of why this line is here.
# Reads your page so it can be listed in that assistant's index.
User-agent: Claude-SearchBot
Disallow: /To leave it allowed explicitly — worth doing when the rest of your file is restrictive and you want the exception on the record — an empty Disallow is the standard way to say “nothing is disallowed”.
# Nothing disallowed for this fetcher.
User-agent: Claude-SearchBot
Disallow:Read what you actually serve first
Content delivery networks can prepend a managed block to your robots.txt, and some do it by default. Crawlers read the whole merged file, so open your own robots.txt in a browser before you edit anything — if there is a block you did not write, that is where your real policy lives. We found exactly that on this site in August 2026.
Should you block Claude-SearchBot?
It depends on the job this particular fetcher does — which is why the answer is different for each of an operator's three.
Claude-SearchBot is a search crawler. Blocking a search crawler is the expensive one. This is the fetcher that decides whether you can be cited at all, so disallowing it is a decision to be absent from that assistant rather than a decision about training. Most people who say they want to block AI do not mean this.
Three positions are coherent: allow everything, refuse training while staying citable, or refuse everything. The middle one is what most people mean when they say they want to block AI, and it is the one snippets organised by company cannot express. The full argument, with the trade written out for each position, is in Should you block AI crawlers? It is three decisions, not one.
One thing that is true whichever you choose: robots.txt is a request. It became a standard in 2022, as RFC 9309, and compliance with it is voluntary. The large operators document their tokens and by their own account honour them; anything determined to ignore you will ignore you, and the place that stops it is your server or your CDN, not a text file.
Seeing it on your own site
You are about to make a trade. The only number that was ever going to inform it is your own.
Before and after are worth more than any opinion about this token. Coruve classifies every event as it arrives — human, assistant referral, or crawler — and reports the three separately, so you can watch what a robots.txt change actually did rather than guessing.
The checker reads seven days of traffic with no account. If you would rather have the concepts first, the complete guide to AI referral traffic explains what an assistant referral is, how it differs from a crawl, and what any analytics tool can honestly see.