AI Crawlers
Who the AI bots visiting your site are, what they want, and how blocking one actually works.
What This Page Covers
Your site gets visited by more than search engines. A growing set of AI-specific bots fetch
your pages to train models, build the indexes that answer engines cite from, or read a
single page live because a user just asked an assistant about it. AIO Radar → Crawlers
shows who's been visiting, when, and — when one is blocked — exactly where the block is
coming from.
Why This Matters
AI crawlers are how your pages reach answer engines. Blocking one removes you from what it
can cite.
Three Kinds of Bot, Three Different Stakes
Every AI bot falls into one of three categories, and the consequence of blocking each one
is different:
| Category | What it does | If you block it |
|---|---|---|
| AI training | Feeds the model's training data. | Keeps future models from learning from you; it does not affect answers today. |
| AI search index | Builds the index answer engines cite from. | Block this and you disappear from AI answers. |
| On-demand fetch | Fetches a page when a user asks about it. | Makes assistants unable to read your page in a live conversation. |
A training bot and a search bot can share a vendor — Anthropic runs both ClaudeBot
(training) and Claude-SearchBot (search) — and blocking one says nothing about the other.
Read the category, not just the name.
Known Bots by Vendor
The registry AIO Radar checks against (kept in sync with the backend's scoring set):
| Bot | Vendor | Category |
|---|---|---|
| GPTBot | OpenAI | AI training |
| OAI-SearchBot | OpenAI | AI search index |
| ChatGPT-User | OpenAI | On-demand fetch |
| ClaudeBot | Anthropic | AI training |
| Claude-SearchBot | Anthropic | AI search index |
| Claude-User | Anthropic | On-demand fetch |
| anthropic-ai | Anthropic | AI training |
| PerplexityBot | Perplexity | AI search index |
| Perplexity-User | Perplexity | On-demand fetch |
| Google-Extended | AI training | |
| Applebot-Extended | Apple | AI training |
| CCBot | Common Crawl | AI training |
| Meta-ExternalAgent | Meta | AI training |
| Amazonbot | Amazon | AI training |
| Bytespider | ByteDance | AI training |
| Diffbot | Diffbot | AI training |
| cohere-ai | Cohere | AI training |
| ImagesiftBot | Hive | AI training |
| YouBot | You.com | AI search index |
New bots get added to this registry as vendors publish them; the Crawlers tab always shows
the current list with a link to each vendor's own bot documentation.
Last Seen, and Never Seen
Each bot in the hits table shows the last time it actually fetched your site, as "3 days
ago" — or never seen when the registry knows the bot but it has no recorded hits. Never
seen is not a dash and not a zero: it means exactly what it says, and it changes the moment
the bot shows up. A bot you've deliberately blocked will typically stay at never seen or go
stale — that's expected, not an error.
Where a Block Actually Comes From
A bot can be blocked at four different layers, and only one of them is something AIO
Radar's own toggle controls:
| Layer | What it means | Who controls it |
|---|---|---|
| CDN | Your CDN (Cloudflare, and others) is returning an error page to the bot before it reaches your server. | Your CDN's dashboard |
| Host | Your hosting provider's own firewall or bot-management layer is blocking it. | Your host's dashboard |
| robots.txt | Your site's robots.txt disallows the bot. | Your theme, an SEO plugin, or a manual edit |
| This plugin | AIO Radar's own crawler-access setting is set to block it. | AIO Radar → Crawlers |
The Crawlers tab labels every bot's access as one of Allowed, **Blocked by
Cloudflare, Blocked by your host, Blocked by robots.txt, or Blocked by this
plugin**. When a bot is blocked upstream (CDN or host), the plugin's own access toggle is
disabled and shows why — flipping it would have no effect, because the request never
reaches WordPress in the first place.
Checking Right Now
Click Check crawlers now on the Crawlers tab to have AIO Radar fetch your home page as
each of the five scoring bots and report back what actually happened — status code, which
layer blocked it (if any), and whether robots.txt disallows it. This is the same check
that runs automatically after every sync and once an hour.
View as AI
View as AI, available on any page (in the editor and on the Markdown table), shows
exactly what an assistant and a crawler each receive when they fetch that specific page:
llms.txtwould read.
it actually got back (allowed, blocked, and which layer), plus a text sample of what
GPTBot received.
This is the fastest way to confirm a fix worked without waiting for the next scheduled
check.
Troubleshooting
A Bot I Want Isn't Being Blocked, But Should Be
Set its access to blocked on the Crawlers tab. If it's already set to blocked and still
getting through, check whether your CDN or host is passing it through despite your
robots.txt — the plugin's toggle only affects requests that reach WordPress.
A Bot I Need Shows as Blocked, and I Didn't Block It
Check the Detected column — it names the layer. A CDN or host block has to be lifted
where it's set; AIO Radar can tell you where the block is, but per
ADR 0005 it never changes anything upstream on your behalf.