CiteRev
CiteRev · Articles

Training, search, user-triggered: the three kinds of AI crawlers

Published November 21, 2026

Many business owners hear that "AI will use my content for training" and block every crawler with "AI" in its name in robots.txt. The result: training is blocked, but AI search can no longer see the website either, so naturally the business doesn't get recommended when customers ask AI. The problem is not telling the crawlers apart by what they do.

Three jobs

TypeWhat it doesWhat happens if you block it
Training crawlerCollects web pages to train future models.Your content stays out of training data. Usually no effect on being cited in AI search.
Search crawlerBuilds the index for AI search. When answering, the AI finds pages in this index and links to them.Your pages will rarely appear in AI search answers.
User-triggered fetchingA user pastes your URL into the chat, or the AI opens your page on the spot to answer the question.The AI can't read your page and has to answer from information found elsewhere.

Each company's crawler names

The table below shows the names in each company's public documentation at the time of writing. Names get added and removed, so check the official documentation before you write any rules.

CompanyTrainingSearchUser-triggered
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
Perplexity—PerplexityBotPerplexity-User
GoogleGoogle-ExtendedGooglebot—

Two things people often get wrong:

A common trade-off

If your goal is "I don't want my content used for training, but I want AI search to recommend me," you can block only the training crawlers and allow the search crawlers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

If you don't mind training and just want to be seen as much as possible, you don't need any AI-specific Disallow rules. Just make sure the wildcard rule User-agent: * isn't blocking your whole site.

What to check beyond robots.txt

Allowing crawlers in robots.txt doesn't mean they can actually get in. Two other places can also keep them out:

To check, look in your server access logs for requests from these crawlers, and see whether they got a 200 or a 403 status code.

Allowing search crawlers only lets AI read your pages. It doesn't guarantee that AI will cite or recommend you. To find out whether you get recommended, ask real customer questions, several rounds each, and then look at the results.

Related articles

Want to know whether your robots.txt currently allows or blocks AI crawlers? The free check lists the result for each of four crawlers: GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot.

Apply for a free check