AI crawlers vs AI referrals: what your analytics can and cannot see

Two things reach your website with an AI company’s name attached, and they have almost nothing in common.

• A referral is a human who clicked a link in an assistant’s answer. They have a browser, opinions and a credit card.
• A crawler is a program fetching your HTML. It has none of those things and never will.

Put them in the same row and you have a number that describes nothing. Worse, you have a visitor count with programs in it — and visitors are the denominator of nearly every other figure you look at. Conversion rate, bounce rate, pages per session: contaminate the visitor count and all of them move, quietly, in the flattering direction.

That is the crawl → cite → click sequence, and it is genuinely useful: a rise in training fetches means something different from a rise in on-demand fetches, and only the second one is downstream of a human being interested in you. A product that reports a single “AI bots” total has thrown that distinction away before you ever saw it.

ChatGPT-User fetches your page because a person asked about it. There is a human at the other end. It is enormously tempting to call that a visit — it is the closest thing to demand that a crawler ever produces.

It is still a program, and Coruve records it as one. The reason is not pedantry: “is this a person?” is the question the traffic-type field answers, and it is the field that dozens of queries throughout the product use to separate humans from machines. Inventing a third, in-between category would have quietly moved these fetches into the human side of every one of those queries — every visitor count in the product changing as a side effect of an honesty feature.

So the job the fetcher was doing is recorded in its own right, and the person-or-not answer is left alone. You get the crawl-cite-click detail without the visitor number moving underneath you.

Coruve labels every AI classification with how it knows. These are the three buckets, in the product’s own words:

The three are never added together. Not in a report, not in a CSV column, not in a headline figure. The moment you sum them, the strongest number inherits the weakest number’s trustworthiness, and the sentence worth being believed about — “this part is proven” — stops being true.

This is the question to put to any vendor selling you AI-traffic reporting: not “how much AI traffic do I have” but “how do you know, and does the screen tell me?”

Coruve measures with a small script in your page. So does almost everything else. The consequence is short and rarely printed: a crawler that fetches your HTML without running JavaScript leaves no trace whatsoever. No script, no event, nothing to count.

And most crawlers behave exactly that way. GPTBot, ClaudeBot and the rest request the page and disconnect. So the crawler activity a script-based tool can see is a small and unknowable fraction of the real crawl.

Which forces an uncomfortable conclusion, and we would rather state it than have you work it out later:

• A crawler count is honest. “We saw at least this many” is a floor, and a floor is a real fact.
• A crawler percentage is not. The numerator is a fraction of the crawl; the denominator is a fairly complete count of humans. Dividing one by the other produces a confident-looking number that no script-based collection method can support.

This is why our free checker prints a count rather than a share for crawler activity, on the page where a big percentage would be most persuasive. If you need the true crawl picture, the place to get it is your server or CDN logs, which see every request whether or not a script ran. That is a genuinely different discipline, and no analytics tag can substitute for it.

The referral half is much better off: those are real browsers running real scripts, so measuring assistant referrals is as complete as measuring any other traffic you have.

Anyone selling you a tag-based tool as a complete answer to “how much is AI crawling my site” is either confused or hoping you are. The two instruments answer different halves, and the honest position is to say which half you cover.

1. Decide about training and retrieval separately. Your robots.txt can allow or refuse each operator’s crawlers by name, and the three jobs are three different decisions. Refusing training while allowing retrieval is a coherent position: it keeps your text out of the next model while leaving you citable today. Refusing everything is also coherent — but it is a decision to be less discoverable, and it should be made on purpose rather than inherited from a default somebody else set.
2. Judge referrals as a channel, not as a novelty. Once assistant visitors are counted apart from the machines, they can be compared against your other channels on the things that matter — what they read, whether they come back, whether they buy. That comparison is the whole point, and it is impossible while programs are in the same number.
3. Watch the on-demand fetches as an early signal. They rise when people start asking assistants about you, which can happen before the clicks do. It is the nearest thing to a leading indicator in this area — as long as you remember it is a floor.

If you want the concepts from the beginning rather than the methodology, start with the complete guide to AI referral traffic. If you want to see the split for your own site, the free AI traffic checker runs on seven days of traffic and prints these caveats next to the numbers rather than underneath them.