# AI crawler access index, week 2026-W37

## In short

Of the 128 seeded domains whose robots.txt we have read, this page reports the share that disallow each documented AI crawler token for /. Counts and shares only, no domain is named.

| Fact | Value |
| --- | --- |
| Week | 2026-W37 |
| Domains seeded | 19967 |
| Domains with a robots.txt reading | 128 |
| Crawler tokens measured | 19 |
| Newest reading in this edition | 2026-09-07 |
| Highest share disallowed for / | GPTBot, 20.3% |
| Last verified | 2026-09-07 |

## Share disallowing each token for /

| Crawler | Vendor | Domains read | Disallowed | Share |
|---|---|---|---|---|
| GPTBot | OpenAI | 128 | 26 | 20.3% |
| CCBot | Common Crawl Foundation | 128 | 24 | 18.8% |
| Bytespider | ByteDance | 128 | 23 | 18.0% |
| ClaudeBot | Anthropic | 128 | 22 | 17.2% |
| Google-Extended | Google | 128 | 22 | 17.2% |
| Amazonbot | Amazon | 128 | 21 | 16.4% |
| Applebot-Extended | Apple | 128 | 21 | 16.4% |
| meta-externalagent | Meta | 128 | 21 | 16.4% |
| anthropic-ai | Anthropic | 128 | 10 | 7.8% |
| ChatGPT-User | OpenAI | 128 | 10 | 7.8% |
| PerplexityBot | Perplexity | 128 | 10 | 7.8% |
| OAI-SearchBot | OpenAI | 128 | 7 | 5.5% |
| Applebot | Apple | 128 | 5 | 3.9% |
| Claude-SearchBot | Anthropic | 128 | 5 | 3.9% |
| Perplexity-User | Perplexity | 128 | 5 | 3.9% |
| Claude-User | Anthropic | 128 | 4 | 3.1% |
| DuckAssistBot | DuckDuckGo | 128 | 4 | 3.1% |
| Bingbot | Microsoft | 128 | 1 | 0.8% |
| Googlebot | Google | 128 | 1 | 0.8% |

Aggregates only. No domain is named on this page.

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.