Training, search, user-triggered: the three kinds of AI crawlers
Many business owners hear that "AI will use my content for training" and block every crawler with "AI" in its name in robots.txt. The result: training is blocked, but AI search can no longer see the website either, so naturally the business doesn't get recommended when customers ask AI. The problem is not telling the crawlers apart by what they do.
Three jobs
| Type | What it does | What happens if you block it |
|---|---|---|
| Training crawler | Collects web pages to train future models. | Your content stays out of training data. Usually no effect on being cited in AI search. |
| Search crawler | Builds the index for AI search. When answering, the AI finds pages in this index and links to them. | Your pages will rarely appear in AI search answers. |
| User-triggered fetching | A user pastes your URL into the chat, or the AI opens your page on the spot to answer the question. | The AI can't read your page and has to answer from information found elsewhere. |
Each company's crawler names
The table below shows the names in each company's public documentation at the time of writing. Names get added and removed, so check the official documentation before you write any rules.
| Company | Training | Search | User-triggered |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | — | PerplexityBot | Perplexity-User |
Google-Extended | Googlebot | — |
Two things people often get wrong:
Google-Extendedisn't a separate crawler that visits your site. It's a switch in robots.txt that decides whether content Google has crawled can be used for Gemini-related purposes. It doesn't affect whether you're indexed in Google Search. The AI summaries in Google Search results use what the regularGooglebotcrawls.- User-triggered fetching opens a page on behalf of a specific user, and the companies don't all say the same thing about whether it follows robots.txt. Don't assume robots.txt will always control this kind.
A common trade-off
If your goal is "I don't want my content used for training, but I want AI search to recommend me," you can block only the training crawlers and allow the search crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
If you don't mind training and just want to be seen as much as possible, you don't need any AI-specific Disallow rules. Just make sure the wildcard rule User-agent: * isn't blocking your whole site.
What to check beyond robots.txt
Allowing crawlers in robots.txt doesn't mean they can actually get in. Two other places can also keep them out:
- A "block AI bots" setting in your CDN or firewall. Some services turn it on by default, or switch it on automatically after an update.
- Blocking by your web host or a security plugin based on User-Agent or IP range.
To check, look in your server access logs for requests from these crawlers, and see whether they got a 200 or a 403 status code.
Related articles
Want to know whether your robots.txt currently allows or blocks AI crawlers? The free check lists the result for each of four crawlers: GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot.
Apply for a free check