CiteRev
CiteRev · Articles

How robots.txt blocks AI crawlers, and how to check yours

Published November 20, 2026

Before AI search answers a question, it usually has to crawl the pages it draws on. If your robots.txt shuts out the right crawler, AI may never see your page, no matter how good it is. This is easy to miss, because many sites have a robots.txt that was written years ago, before these crawlers existed.

Common AI crawlers

Here are the crawler names used by the major AI services. Each company's official documentation is the final word on the exact names, so check them before you write any rules.

Training and search are usually handled by two different crawlers. If you want AI to cite your pages, the search crawler matters more; blocking the training crawler won't necessarily affect citations.

One wildcard rule blocks everyone

User-agent: *
Disallow: /

These two lines block every crawler that doesn't have its own rule, including all of the ones above. Many sites added them to keep out junk traffic and ended up locking out AI search too.

Named rules come first

Most crawlers follow the group of rules that names them, and only fall back to * when there isn't one. So the rules below allow ClaudeBot and block everyone else:

User-agent: *
Disallow: /

User-agent: ClaudeBot
Allow: /

If you want to allow every AI search crawler, give each name its own group with Allow: / instead of relying on the wildcard rule.

How to check

  1. Open yourdomain/robots.txt and confirm the file exists and returns 200.
  2. Find the rule group for each AI crawler and see whether its Disallow includes /.
  3. Watch the paths: Disallow: /private only blocks pages starting with /private, not the root.
  4. If the file returns 404, all crawlers are allowed by default. If it can't be reached at all, treat it as "unreachable" when you check.

CiteRev's free check runs through these steps automatically and lists the result for each crawler.

This article only explains how the rules work and how to check them. It doesn't promise that AI will cite you once robots.txt lets crawlers in. Citations depend on many factors, and the only way to know is to look at what AI actually answers.

Related articles

Want to know whether your robots.txt currently lets AI crawlers in or keeps them out? The free check lists the result for each crawler.

Apply for a free check