# AI crawler robots.txt builder: allow or disallow each of the 21 documented tokens and copy the file

## In short

This builder writes a robots.txt group for every one of the 21 AI crawler tokens in the AEO Watch registry, 0 of them set to Disallow: / and 21 to Allow: /, using the token spelling and the snippet each vendor's own documentation supports, last checked 2026-09-17. Your choices ride in the address of this page, so the file can be linked and rebuilt.

| Fact | Value |
| --- | --- |
| Documented tokens | 21 |
| Tokens set to Disallow: / | 0 |
| Tokens set to Allow: / | 21 |
| Paths disallowed for everyone | 0 |
| Newest documentation check | 2026-09-17 |
| Last verified | 2026-09-17 |

## Your robots.txt

```
# generated at https://bikoosh.com/aeo/robots-builder on 2026-10-09; token facts from the vendor documentation dated 2026-09-17
# 21 documented tokens, 0 disallowed for /, 21 allowed.
# A robots.txt rule is a request, not an access control.

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: Googlebot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Amzn-User
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Google-Extended
Allow: /
```

The file is assembled from the registry: the User-agent line of every group is the spelling that token's own guide page prints, and the Allow and Disallow lines are the two snippets the vendor documentation supports. A robots.txt rule is a request, it is not an access control, it does not stop a request from arriving, and this page does not tell anyone what to choose. Every token starts on Allow, and a token you leave on Allow is written out as an allow group so the file states the whole registry rather than only the part you changed.

- Plain text edition: https://bikoosh.com/aeo/robots-builder.txt
- This configuration: https://bikoosh.com/aeo/robots-builder
- Every documented token, with its vendor and purpose: https://bikoosh.com/aeo/crawlers

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.