Bikoosh
Check a listing Pricing Guides Fraud report Sign in

AI crawler access index, week 2026-W37

Of the 128 seeded domains whose robots.txt we have read, this page reports the share that disallow each documented AI crawler token for /. Counts and shares only, no domain is named.

Week2026-W37
Domains seeded19967
Domains with a robots.txt reading128
Crawler tokens measured19
Newest reading in this edition2026-09-07
Highest share disallowed for /GPTBot, 20.3%
Last verified

Share of the seeded domains we have read a robots.txt for that disallow each documented AI crawler token for /. The denominator is the domains actually read, never the whole seeded list, because a domain we have not read yet says nothing either way.

Share of the 128 domains read that disallow each AI crawler token for /0%25%50%75%100%GPTBotGPTBot: 20.3% of 128 domains read20.3%CCBotCCBot: 18.8% of 128 domains read18.8%BytespiderBytespider: 18% of 128 domains read18%ClaudeBotClaudeBot: 17.2% of 128 domains read17.2%Google-ExtendedGoogle-Extended: 17.2% of 128 domains read17.2%AmazonbotAmazonbot: 16.4% of 128 domains read16.4%Applebot-ExtendedApplebot-Extended: 16.4% of 128 domains read16.4%meta-externalagentmeta-externalagent: 16.4% of 128 domains read16.4%anthropic-aianthropic-ai: 7.8% of 128 domains read7.8%ChatGPT-UserChatGPT-User: 7.8% of 128 domains read7.8%PerplexityBotPerplexityBot: 7.8% of 128 domains read7.8%OAI-SearchBotOAI-SearchBot: 5.5% of 128 domains read5.5%ApplebotApplebot: 3.9% of 128 domains read3.9%Claude-SearchBotClaude-SearchBot: 3.9% of 128 domains read3.9%Perplexity-UserPerplexity-User: 3.9% of 128 domains read3.9%Claude-UserClaude-User: 3.1% of 128 domains read3.1%DuckAssistBotDuckAssistBot: 3.1% of 128 domains read3.1%BingbotBingbot: 0.8% of 128 domains read0.8%GooglebotGooglebot: 0.8% of 128 domains read0.8%
Share disallowed for /, by token, week 2026-W37. The table below carries the same numbers.
CrawlerVendorDomains readAllowed for /Disallowed for /Share disallowed
GPTBotOpenAI 12810226 20.3%
CCBotCommon Crawl Foundation 12810424 18.8%
BytespiderByteDance 12810523 18.0%
ClaudeBotAnthropic 12810622 17.2%
Google-ExtendedGoogle 12810622 17.2%
AmazonbotAmazon 12810721 16.4%
Applebot-ExtendedApple 12810721 16.4%
meta-externalagentMeta 12810721 16.4%
anthropic-aiAnthropic 12811810 7.8%
ChatGPT-UserOpenAI 12811810 7.8%
PerplexityBotPerplexity 12811810 7.8%
OAI-SearchBotOpenAI 1281217 5.5%
ApplebotApple 1281235 3.9%
Claude-SearchBotAnthropic 1281235 3.9%
Perplexity-UserPerplexity 1281235 3.9%
Claude-UserAnthropic 1281244 3.1%
DuckAssistBotDuckDuckGo 1281244 3.1%
BingbotMicrosoft 1281271 0.8%
GooglebotGoogle 1281271 0.8%

History

WeekDomains readSeeded
2026-W37519967
2026-W367919967

Each week is frozen when it is first published, so a citation of this page keeps meaning what it meant.

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.

Cite this
AI crawler access index, week 2026-W37 - https://bikoosh.com/aeo/index
<a href="https://bikoosh.com/aeo/index">AI crawler access index, week 2026-W37</a>
[AI crawler access index, week 2026-W37](https://bikoosh.com/aeo/index)
AI crawler access index, week 2026-W37. Bikoosh. Retrieved 2026-09-07, from https://bikoosh.com/aeo/index