AI crawler robots.txt builder: allow or disallow each of the 21 documented tokens and copy the file
This builder writes a robots.txt group for every one of the 21 AI crawler tokens in the AEO Watch registry, 0 of them set to Disallow: / and 21 to Allow: /, using the token spelling and the snippet each vendor's own documentation supports, last checked 2026-09-17. Your choices ride in the address of this page, so the file can be linked and rebuilt.
| Documented tokens | 21 |
|---|---|
| Tokens set to Disallow: / | 0 |
| Tokens set to Allow: / | 21 |
| Paths disallowed for everyone | 0 |
| Newest documentation check | 2026-09-17 |
| Last verified |
Your robots.txt
# generated at https://bikoosh.com/aeo/robots-builder on 2026-10-09; token facts from the vendor documentation dated 2026-09-17 # 21 documented tokens, 0 disallowed for /, 21 allowed. # A robots.txt rule is a request, not an access control. User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Allow: / User-agent: Googlebot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Perplexity-User Allow: / User-agent: anthropic-ai Allow: / User-agent: Bingbot Allow: / User-agent: Applebot Allow: / User-agent: Amazonbot Allow: / User-agent: meta-externalagent Allow: / User-agent: DuckAssistBot Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Amzn-User Allow: / User-agent: Applebot-Extended Allow: / User-agent: Google-Extended Allow: /
The file is assembled from the registry: the User-agent line of every group is the spelling that token's own guide page prints, and the Allow and Disallow lines are the two snippets the vendor documentation supports. A robots.txt rule is a request, it is not an access control, it does not stop a request from arriving, and this page does not tell anyone what to choose. Every token starts on Allow, and a token you leave on Allow is written out as an allow group so the file states the whole registry rather than only the part you changed.
Your choices are in the address of this page, so the configuration can be linked, bookmarked and rebuilt. The page that carries no parameters is the canonical one; a result URL asks search engines not to index it, so the same tool cannot appear as hundreds of near identical pages. This configuration: https://bikoosh.com/aeo/robots-builder
Where these lines come from
- Every documented token on one page: the vendor, the purpose its documentation states, the documented way to verify a request, and the share of read domains disallowing it for /.
- One guide page per token, linked from the table above: the vendor document we fetched, its HTTP status and the date we last read it.
- The AEO Watch dataset
- Tranco top 1,000 AI crawler policy (CSV, CC BY 4.0)
- The AEO Watch API
- The checker as a command line tool (one file, no install)
Check your own site
The same checks, run on your domain now: what robots.txt tells each AI crawler, whether the edge answers them, and whether the text is readable without JavaScript. Free, no signup.
Limits: 8 checks and 5 different domains per address per hour; a repeat of the same domain inside 7 days is answered from what we already hold.
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.