What is Applebot-Extended?

Applebot-Extended is an opt-out token rather than a fetcher, in the same way Google-Extended is: the fetching is done by Apple's general search crawler, and the token governs what may be done with the result afterwards.

Coruve records it as a training signature for that reason. Disallowing it is a statement about training, not about being crawled or being found.

Coruve files every hit matching applebot-extended under Apple, Apple’s entry in the classifier. That attribution is a reading of the user agent the request sent, which is a claim the client made about itself rather than a proof — Coruve labels it as such everywhere it is reported.

Why Apple has fewer than three fetchers here: Applebot-Extended is a training opt-out token; Applebot itself is a generic search crawler.

We make no claim here about how Apple behaves beyond what Coruve records, because this repository holds no dated reading of its documentation. A company’s own documentation is the only source that is authoritative about that company, and it is usually one page — read it before you write a rule that names this token.

The row that matters most is the second one. Applebot-Extended is recorded as a crawler, never as a visitor, and Coruve does not add the two together anywhere. A program reading your page is not a customer, and a tool that folds the two into one number is quietly inflating every figure built on top of it.

The other side of that line is an assistant referral: a real person, in a real browser, who clicked a link inside an AI assistant and landed on your site. Those are counted as human visits and reported as their own channel, with the assistant named. Both readings are useful and they answer different questions — how often a machine reads you, and how many people an assistant sends you.

robots.txt matches a fetcher by its product token, so the spelling below is the part that has to be right. Put the reason in a comment beside it: the next person to edit that file will be you, in eighteen months, with no memory of why this line is here.

# Reads your text so a future model can learn from it.
User-agent: Applebot-Extended
Disallow: /

To leave it allowed explicitly — worth doing when the rest of your file is restrictive and you want the exception on the record — an empty Disallow is the standard way to say “nothing is disallowed”.

# Nothing disallowed for this fetcher.
User-agent: Applebot-Extended
Disallow:

Applebot-Extended is a training crawler. Blocking a training crawler costs you nothing you can measure. It does not remove you from any assistant's answers, because a different fetcher does that job, and it does not touch your search ranking. If you are uneasy about your writing feeding a model, this is the line to write, and it is the cheapest decision on the page.

Three positions are coherent: allow everything, refuse training while staying citable, or refuse everything. The middle one is what most people mean when they say they want to block AI, and it is the one snippets organised by company cannot express. The full argument, with the trade written out for each position, is in Should you block AI crawlers? It is three decisions, not one.

One thing that is true whichever you choose: robots.txt is a request. It became a standard in 2022, as RFC 9309, and compliance with it is voluntary. The large operators document their tokens and by their own account honour them; anything determined to ignore you will ignore you, and the place that stops it is your server or your CDN, not a text file.

Before and after are worth more than any opinion about this token. Coruve classifies every event as it arrives — human, assistant referral, or crawler — and reports the three separately, so you can watch what a robots.txt change actually did rather than guessing.

The checker reads seven days of traffic with no account. If you would rather have the concepts first, the complete guide to AI referral traffic explains what an assistant referral is, how it differs from a crawl, and what any analytics tool can honestly see.