# AEO Watch AEO Watch reads the robots.txt of 21443 seeded domains and resolves 21 documented AI crawler tokens against it, then records what it saw with the evidence and the date. ## What this is One observation per domain and per documented AI crawler token: what that domain's robots.txt says to the token, with the evidence line and the date it was read. Facts with evidence and dates only, no scores, no grades and no verdicts. A site opts out at any time and the opt out is honoured automatically and permanently. ## How to cite Attribute an observation to bikoosh.com, link the page, and carry the date it was observed. ## AI crawlers in the registry (21) Every token on one page, with the vendor, the purpose, the documented verification and one robots.txt block naming all of them: https://bikoosh.com/aeo/crawlers Allow or disallow each token and get the whole robots.txt group set, one URL per configuration, plain text edition at https://bikoosh.com/aeo/robots-builder.txt: https://bikoosh.com/aeo/robots-builder - GPTBot (OpenAI): purpose per the vendor's documentation, training corpus; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/gptbot - ClaudeBot (Anthropic): purpose per the vendor's documentation, training corpus; documentation verified 2026-09-17. https://bikoosh.com/aeo/crawler/claudebot - PerplexityBot (Perplexity): purpose per the vendor's documentation, search index; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/perplexitybot - OAI-SearchBot (OpenAI): purpose per the vendor's documentation, search index; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/oai-searchbot - CCBot (Common Crawl Foundation): purpose per the vendor's documentation, training corpus, public web corpus; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/ccbot - Bytespider (ByteDance): purpose per the vendor's documentation, training corpus; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/bytespider - Googlebot (Google): purpose per the vendor's documentation, search index; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/googlebot - ChatGPT-User (OpenAI): purpose per the vendor's documentation, user-initiated fetch; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/chatgpt-user - Claude-User (Anthropic): purpose per the vendor's documentation, user-initiated fetch; documentation verified 2026-09-17. https://bikoosh.com/aeo/crawler/claude-user - Claude-SearchBot (Anthropic): purpose per the vendor's documentation, search index; documentation verified 2026-09-17. https://bikoosh.com/aeo/crawler/claude-searchbot - Perplexity-User (Perplexity): purpose per the vendor's documentation, user-initiated fetch; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/perplexity-user - anthropic-ai (Anthropic): purpose per the vendor's documentation, undocumented; documentation verified 2026-09-17. https://bikoosh.com/aeo/crawler/anthropic-ai - Bingbot (Microsoft): purpose per the vendor's documentation, search index; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/bingbot - Applebot (Apple): purpose per the vendor's documentation, search index, user-initiated fetch; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/applebot - Amazonbot (Amazon): purpose per the vendor's documentation, search index, user-initiated fetch; documentation verified 2026-09-17. https://bikoosh.com/aeo/crawler/amazonbot - meta-externalagent (Meta): purpose per the vendor's documentation, training corpus, search index; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/meta-externalagent - DuckAssistBot (DuckDuckGo): purpose per the vendor's documentation, user-initiated fetch, answer generation; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/duckassistbot - Amzn-SearchBot (Amazon): purpose per the vendor's documentation, search index; documentation verified 2026-09-17. https://bikoosh.com/aeo/crawler/amzn-searchbot - Amzn-User (Amazon): purpose per the vendor's documentation, user-initiated fetch; documentation verified 2026-09-17. https://bikoosh.com/aeo/crawler/amzn-user - Applebot-Extended (Apple): purpose per the vendor's documentation, training corpus; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/applebot-extended - Google-Extended (Google): purpose per the vendor's documentation, training corpus, grounding; documentation verified 2026-09-06. https://bikoosh.com/aeo/crawler/google-extended ## AI crawler access index, week 2026-W41 Of the 12585 seeded domains whose robots.txt we have read, this page reports the share that disallow each documented AI crawler token for /. Counts and shares only, no domain is named. - CCBot (Common Crawl Foundation): 1721 of 12585 domains read disallow it for /, 13.7%. https://bikoosh.com/aeo/crawler/ccbot - Bytespider (ByteDance): 1667 of 12585 domains read disallow it for /, 13.2%. https://bikoosh.com/aeo/crawler/bytespider - GPTBot (OpenAI): 1663 of 12585 domains read disallow it for /, 13.2%. https://bikoosh.com/aeo/crawler/gptbot - ClaudeBot (Anthropic): 1519 of 12585 domains read disallow it for /, 12.1%. https://bikoosh.com/aeo/crawler/claudebot - meta-externalagent (Meta): 1343 of 12585 domains read disallow it for /, 10.7%. https://bikoosh.com/aeo/crawler/meta-externalagent - Google-Extended (Google): 1325 of 12585 domains read disallow it for /, 10.5%. https://bikoosh.com/aeo/crawler/google-extended - Amazonbot (Amazon): 1323 of 12585 domains read disallow it for /, 10.5%. https://bikoosh.com/aeo/crawler/amazonbot - Applebot-Extended (Apple): 1305 of 12585 domains read disallow it for /, 10.4%. https://bikoosh.com/aeo/crawler/applebot-extended - anthropic-ai (Anthropic): 1150 of 12585 domains read disallow it for /, 9.1%. https://bikoosh.com/aeo/crawler/anthropic-ai - PerplexityBot (Perplexity): 1016 of 12585 domains read disallow it for /, 8.1%. https://bikoosh.com/aeo/crawler/perplexitybot - ChatGPT-User (OpenAI): 957 of 12585 domains read disallow it for /, 7.6%. https://bikoosh.com/aeo/crawler/chatgpt-user - DuckAssistBot (DuckDuckGo): 744 of 12585 domains read disallow it for /, 5.9%. https://bikoosh.com/aeo/crawler/duckassistbot - OAI-SearchBot (OpenAI): 744 of 12585 domains read disallow it for /, 5.9%. https://bikoosh.com/aeo/crawler/oai-searchbot - Perplexity-User (Perplexity): 730 of 12585 domains read disallow it for /, 5.8%. https://bikoosh.com/aeo/crawler/perplexity-user - Claude-User (Anthropic): 702 of 12585 domains read disallow it for /, 5.6%. https://bikoosh.com/aeo/crawler/claude-user - Claude-SearchBot (Anthropic): 692 of 12585 domains read disallow it for /, 5.5%. https://bikoosh.com/aeo/crawler/claude-searchbot - Amzn-SearchBot (Amazon): 534 of 12373 domains read disallow it for /, 4.3%. https://bikoosh.com/aeo/crawler/amzn-searchbot - Amzn-User (Amazon): 518 of 12373 domains read disallow it for /, 4.2%. https://bikoosh.com/aeo/crawler/amzn-user - Applebot (Apple): 505 of 12585 domains read disallow it for /, 4.0%. https://bikoosh.com/aeo/crawler/applebot - Bingbot (Microsoft): 313 of 12585 domains read disallow it for /, 2.5%. https://bikoosh.com/aeo/crawler/bingbot - Googlebot (Google): 259 of 12585 domains read disallow it for /, 2.1%. https://bikoosh.com/aeo/crawler/googlebot ## What you get - Free monitor: free, one domain, a weekly recheck, and an email only when something changed. - Pro monitoring: 9 USD a month, up to 10 domains, a daily recheck, an alert on any recorded change, and a weekly digest. Checkout: https://gumroad.com/l/bikooshwatch?wanted=true&watch=aeo - Agency monitoring: 79 USD a month, up to 100 domains, a daily recheck, an alert on any recorded change, and a weekly digest. Checkout: https://gumroad.com/l/bikooshwatchgroup?wanted=true&watch=aeo The checkout is titled Bikoosh Watch: one membership shared across Bikoosh products. Activate with the licence key on the receipt or the email it was paid with, and add domains, at https://bikoosh.com/aeo/activate. The free check on https://bikoosh.com/aeo needs no payment and no address. ## Machine editions - Every page answers ?format=md with the same capsule. - The dataset (CC BY 4.0): https://bikoosh.com/aeo/dataset, CSV at https://bikoosh.com/aeo/top-1000.csv, JSON at https://bikoosh.com/api/v1/aeo/top-1000.json - The API: https://bikoosh.com/aeo/api - Weekly access index (RSS): https://bikoosh.com/aeo/feed/index.xml, one changelog feed per crawler at https://bikoosh.com/aeo/feed/crawler/.xml - Aggregate pages: https://bikoosh.com/aeo/changes, https://bikoosh.com/aeo/index, https://bikoosh.com/aeo/industry - MCP tool check_ai_access(domain): stored facts only, never a live check. - Command line tool: https://bikoosh.com/aeo/aeowatch.py, one file, standard library only, no install and no account; pip install https://bikoosh.com/aeo/aeowatch-1.0.0.tar.gz Catalog metadata: Croissant https://bikoosh.com/datasets/croissant/aeo.json, Data Package https://bikoosh.com/datasets/datapackage/aeo.json, DCAT https://bikoosh.com/datasets/catalog.jsonld. AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.