Reference
Every AI crawler, explained
Every AI crawler and assistant fetcher Coruve recognises, what each one is for, and the exact robots.txt token that names it.
There is no such thing as an AI crawler in the singular
Each operator runs up to three programs, doing three different jobs. Only one of them has anything to do with training a model.
Somebody sends you a snippet for your robots.txt, you paste it in, and you have made a decision you were never shown. The snippet named a company. The decision was about several different programs that company runs, and blocking the wrong one costs you visits while blocking the right one costs you nothing you can measure.
Learn the three jobs once and the subject stops being a matter of opinion:
| Job | What that fetcher does |
|---|---|
| Training crawler | Reads your text so a future model can learn from it. |
| Search crawler | Reads your page so it can be listed in that assistant's index. |
| On-demand fetcher | Reads your page because a person just asked the assistant about it. |
Every page below tells you which of the three a given token is, using the same function Coruve’s ingest path uses to label a real hit. Nothing here is a list somebody typed and stopped maintaining.
Every fetcher Coruve recognises
Grouped by the company that runs it, because that is how you arrive. Choose by the job, because that is how the decision works.
OpenAI
| Fetcher | Job | What it is for |
|---|---|---|
| GPTBot | Training crawler | OpenAI's training crawler — it reads your text so future OpenAI models can learn from it. |
| OAI-SearchBot | Search crawler | OpenAI's search crawler — it reads your page so it can be listed in ChatGPT's search results. |
| ChatGPT-User | On-demand fetcher | The fetch OpenAI makes because a person's ChatGPT session needs your page right now. |
Anthropic
| Fetcher | Job | What it is for |
|---|---|---|
| ClaudeBot | Training crawler | Anthropic's training crawler — it reads your text so future Claude models can learn from it. |
| Claude-SearchBot | Search crawler | Anthropic's search crawler — it reads your page so it can be listed in Claude's index. |
| Claude-User | On-demand fetcher | The fetch Anthropic makes because a person's Claude session needs your page right now. |
| Claude-Web | Training crawler | Retired token. Still matched, not sent by anything today. |
| anthropic-ai | Training crawler | Retired token. Still matched, not sent by anything today. |
Perplexity
| Fetcher | Job | What it is for |
|---|---|---|
| PerplexityBot | Search crawler | Perplexity's index crawler — it reads your page so Perplexity can list and cite it. |
| Perplexity-User | On-demand fetcher | The fetch Perplexity makes because a person's search needs your page right now. |
| Fetcher | Job | What it is for |
|---|---|---|
| Google-Extended | Training crawler | Google's training opt-out token. Googlebot does the crawling; this governs generative use. |
ByteDance
| Fetcher | Job | What it is for |
|---|---|---|
| Bytespider | Training crawler | ByteDance's published fetcher. Coruve records it as a training crawler. |
Common Crawl
| Fetcher | Job | What it is for |
|---|---|---|
| CCBot | Training crawler | Common Crawl's archive crawler. Its archive is used as training input by many others. |
Cohere
| Fetcher | Job | What it is for |
|---|---|---|
| cohere-ai | Training crawler | Cohere's published fetcher. Coruve records it as a training crawler. |
Meta
| Fetcher | Job | What it is for |
|---|---|---|
| meta-externalagent | Training crawler | Meta's published fetcher. Coruve records it as a training crawler. |
Amazon
| Fetcher | Job | What it is for |
|---|---|---|
| Amazonbot | Training crawler | Amazon's published fetcher. Coruve records it as a training crawler. |
Apple
| Fetcher | Job | What it is for |
|---|---|---|
| Applebot-Extended | Training crawler | Apple's training opt-out token. Applebot itself is a general search crawler. |
A crawler is not a visitor
The distinction this whole reference rests on, and the one most analytics tools quietly blur.
Every fetcher on this page is a program. It requests your HTML, usually never runs your JavaScript, and there is no person on the other end — except in the on-demand case, where somebody triggered it and still did not visit your site.
An assistant referral is the opposite: a real person, in a real browser, who clicked a link inside ChatGPT, Perplexity, Claude or Gemini and landed on your page. That is a visit, and it can be counted like any other channel — what they read, whether they came back, whether they bought.
Coruve never adds the two together. They answer different questions, and a tool that merges them is inflating every number built on top of the total.
What a page script cannot see
Analytics that runs inside your page cannot observe a fetcher that takes your HTML and never executes JavaScript. Coruve therefore prints counts rather than shares for the crawler half and says so; your server or CDN logs are the complete record. Where every method goes blind, including ours, is set out in AI crawlers vs AI referrals.
How this list is kept current
It is not a list. It is the classifier, rendered.
These pages are generated from the same signature table that labels traffic as it arrives in Coruve. A fetcher cannot appear here without the product recognising it, and the product cannot learn one without a page appearing. That is the entire mechanism, and it is why this reference cannot drift the way a hand-typed list does.
The classifier currently recognises 12 assistant products a person can arrive from, and 17 crawler signatures — the ones catalogued above. A human last reviewed the signature lists on 10 August 2026, and when they change, every affected chart in the product is annotated to say so, because a classification change puts a step in a chart that no customer could otherwise explain.
Operators add fetchers, retire tokens and rename products. If your robots.txt has no review date, it is a file that quietly becomes wrong — twice a year is enough.