How robots.txt blocks AI crawlers, and how to check yours
Before AI search answers a question, it usually has to crawl the pages it draws on. If your robots.txt shuts out the right crawler, AI may never see your page, no matter how good it is. This is easy to miss, because many sites have a robots.txt that was written years ago, before these crawlers existed.
Common AI crawlers
Here are the crawler names used by the major AI services. Each company's official documentation is the final word on the exact names, so check them before you write any rules.
GPTBot: OpenAI's crawler for training data.OAI-SearchBot: OpenAI's crawler for ChatGPT search results.PerplexityBot: Perplexity's crawler for its search index.ClaudeBot: Anthropic's crawler.
Training and search are usually handled by two different crawlers. If you want AI to cite your pages, the search crawler matters more; blocking the training crawler won't necessarily affect citations.
One wildcard rule blocks everyone
User-agent: * Disallow: /
These two lines block every crawler that doesn't have its own rule, including all of the ones above. Many sites added them to keep out junk traffic and ended up locking out AI search too.
Named rules come first
Most crawlers follow the group of rules that names them, and only fall back to * when there isn't one. So the rules below allow ClaudeBot and block everyone else:
User-agent: * Disallow: / User-agent: ClaudeBot Allow: /
If you want to allow every AI search crawler, give each name its own group with Allow: / instead of relying on the wildcard rule.
How to check
- Open
yourdomain/robots.txtand confirm the file exists and returns 200. - Find the rule group for each AI crawler and see whether its
Disallowincludes/. - Watch the paths:
Disallow: /privateonly blocks pages starting with/private, not the root. - If the file returns 404, all crawlers are allowed by default. If it can't be reached at all, treat it as "unreachable" when you check.
CiteRev's free check runs through these steps automatically and lists the result for each crawler.
Related articles
Want to know whether your robots.txt currently lets AI crawlers in or keeps them out? The free check lists the result for each crawler.
Apply for a free check