ClaudeBot: robots token, verification and how to allow or disallow it

All 21 documented crawlers

Identity

Vendor text: 'ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.' The vendor began publishing an IP range file on 2026-08-18 and links it from the help article, so a hit carrying this token is now checked against that file and recorded as verified or spoofed rather than unverifiable (same rule as app/botverify.py). No reverse DNS convention is documented. Read again on 2026-09-17.

Allow or disallow it

User-agent: ClaudeBot
Allow: /

User-agent: ClaudeBot
Disallow: /

A robots.txt rule is a request that a well behaved crawler honours. It is not an access control, and this page does not tell anyone what to choose.

Build a robots.txt with ClaudeBot and every other documented token

Questions

Does ClaudeBot obey robots.txt?

The vendor documentation at https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler states that ClaudeBot respects robots.txt. We fetched that page and it answered our checker on 2026-09-17. This records what the vendor documents, not what any individual request did.

How do I allow ClaudeBot in robots.txt?

Add this group to the robots.txt at the root of the host: User-agent: ClaudeBot Allow: /. The token is matched case insensitively as a substring of the user agent by RFC 9309, and the longest matching rule wins.

How do I disallow ClaudeBot in robots.txt?

Add this group to the robots.txt at the root of the host: User-agent: ClaudeBot Disallow: /. A robots.txt rule is a request that a well behaved crawler honours; it is not an access control.

How do I verify a request really came from ClaudeBot?

The vendor publishes the IP ranges this crawler fetches from at https://claude.com/crawling/bots.json. Check the address of the request against that file. A user agent string on its own proves nothing, because anyone can send one.

What we measure

Of the 12976 seeded domains whose robots.txt we have read, 1648 disallow ClaudeBot for / and 11328 allow it, which is 12.7% disallowed, week 2026-W43. Counts only: no domain is named.

The weekly AI crawler access index

Sources

Free data

• The AEO Watch dataset
• Tranco top 1,000 AI crawler policy (CSV, CC BY 4.0)
• The AEO Watch API
• The checker as a command line tool (one file, no install)

Check a domain against every token

Check your own site

The same checks, run on your domain now: what robots.txt tells each AI crawler, whether the edge answers them, and whether the text is readable without JavaScript. Free, no signup.

Limits: 8 checks and 5 different domains per address per hour; a repeat of the same domain inside 7 days is answered from what we already hold.

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.