# AEO Watch

## In short

AEO Watch reads the robots.txt of 19967 seeded domains and resolves 19 documented AI crawler tokens against it, then records what it saw with the evidence and the date.

| Fact | Value |
| --- | --- |
| Domains seeded | 19967 |
| Domains with a recorded observation | 155 |
| Observations held | 2953 |
| AI crawler tokens in the registry | 19 |
| Recorded changes in the last 7 days | 12 |
| Newest observation | 2026-09-07 |
| Last verified | 2026-09-07 |

## The crawler guides

- GPTBot (OpenAI): https://bikoosh.com/aeo/crawler/gptbot
- ClaudeBot (Anthropic): https://bikoosh.com/aeo/crawler/claudebot
- PerplexityBot (Perplexity): https://bikoosh.com/aeo/crawler/perplexitybot
- OAI-SearchBot (OpenAI): https://bikoosh.com/aeo/crawler/oai-searchbot
- CCBot (Common Crawl Foundation): https://bikoosh.com/aeo/crawler/ccbot
- Bytespider (ByteDance): https://bikoosh.com/aeo/crawler/bytespider
- Googlebot (Google): https://bikoosh.com/aeo/crawler/googlebot
- ChatGPT-User (OpenAI): https://bikoosh.com/aeo/crawler/chatgpt-user
- Claude-User (Anthropic): https://bikoosh.com/aeo/crawler/claude-user
- Claude-SearchBot (Anthropic): https://bikoosh.com/aeo/crawler/claude-searchbot
- Perplexity-User (Perplexity): https://bikoosh.com/aeo/crawler/perplexity-user
- anthropic-ai (Anthropic): https://bikoosh.com/aeo/crawler/anthropic-ai
- Bingbot (Microsoft): https://bikoosh.com/aeo/crawler/bingbot
- Applebot (Apple): https://bikoosh.com/aeo/crawler/applebot
- Amazonbot (Amazon): https://bikoosh.com/aeo/crawler/amazonbot
- meta-externalagent (Meta): https://bikoosh.com/aeo/crawler/meta-externalagent
- DuckAssistBot (DuckDuckGo): https://bikoosh.com/aeo/crawler/duckassistbot
- Applebot-Extended (Apple): https://bikoosh.com/aeo/crawler/applebot-extended
- Google-Extended (Google): https://bikoosh.com/aeo/crawler/google-extended

## Aggregate pages

- Weekly changes: https://bikoosh.com/aeo/changes
- AI crawler access index: https://bikoosh.com/aeo/index
- Industry reports: https://bikoosh.com/aeo/industry

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.