Does example.com allow GPTBot? robots.txt has not been read yet
We hold no robots.txt reading for example.com yet.
| Domain | example.com |
|---|---|
| AI crawler tokens resolved | 0 |
| Allowed for / | 0 |
| Disallowed for / | 0 |
| Checks that ran | none yet |
| robots.txt read (tier A) | 2026-09-06 |
| Full check (tier B) | 2026-09-06 |
| Last verified |
On this domain the robots.txt only check (tier A) last ran on 2026-09-06 and the full check (tier B) last ran on 2026-09-06. The full check adds the home page, the labelled edge probes, llms.txt, the headers and the structured data; the robots.txt check is the one that runs weekly for every seeded domain.
One question per crawler
- GPTBot
- ClaudeBot
- PerplexityBot
- OAI-SearchBot
- CCBot
- Bytespider
- Googlebot
- ChatGPT-User
- Google-Extended
Does example.com allow GPTBot? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow GPTBot
User-agent: GPTBot Allow: /
User-agent: GPTBot Disallow: /
The GPTBot guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow ClaudeBot? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow ClaudeBot
User-agent: ClaudeBot Allow: /
User-agent: ClaudeBot Disallow: /
The ClaudeBot guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow PerplexityBot? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow PerplexityBot
User-agent: PerplexityBot Allow: /
User-agent: PerplexityBot Disallow: /
The PerplexityBot guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow OAI-SearchBot? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow OAI-SearchBot
User-agent: OAI-SearchBot Allow: /
User-agent: OAI-SearchBot Disallow: /
The OAI-SearchBot guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow CCBot? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow CCBot
User-agent: CCBot Allow: /
User-agent: CCBot Disallow: /
The CCBot guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow Bytespider? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow Bytespider
User-agent: Bytespider Allow: /
User-agent: Bytespider Disallow: /
The Bytespider guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow Googlebot? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow Googlebot
User-agent: Googlebot Allow: /
User-agent: Googlebot Disallow: /
The Googlebot guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow ChatGPT-User? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow ChatGPT-User
User-agent: ChatGPT-User Allow: /
User-agent: ChatGPT-User Disallow: /
The ChatGPT-User guide: identity, purpose, documented verification and the vendor sources.
Does example.com allow Google-Extended? robots.txt has not been read yet
No observation recorded for this token yet.
Allow or disallow Google-Extended
User-agent: Google-Extended Allow: /
User-agent: Google-Extended Disallow: /
The Google-Extended guide: identity, purpose, documented verification and the vendor sources.
Every crawler token in the registry
| Crawler | Vendor | Observation | Evidence | Observed |
|---|---|---|---|---|
| GPTBot | OpenAI | no observation recorded | ||
| ClaudeBot | Anthropic | no observation recorded | ||
| PerplexityBot | Perplexity | no observation recorded | ||
| OAI-SearchBot | OpenAI | no observation recorded | ||
| CCBot | Common Crawl Foundation | no observation recorded | ||
| Bytespider | ByteDance | no observation recorded | ||
| Googlebot | no observation recorded | |||
| ChatGPT-User | OpenAI | no observation recorded | ||
| Claude-User | Anthropic | no observation recorded | ||
| Claude-SearchBot | Anthropic | no observation recorded | ||
| Perplexity-User | Perplexity | no observation recorded | ||
| anthropic-ai | Anthropic | no observation recorded | ||
| Bingbot | Microsoft | no observation recorded | ||
| Applebot | Apple | no observation recorded | ||
| Amazonbot | Amazon | no observation recorded | ||
| meta-externalagent | Meta | no observation recorded | ||
| DuckAssistBot | DuckDuckGo | no observation recorded | ||
| Applebot-Extended | Apple | no observation recorded | ||
| Google-Extended | no observation recorded |
History
Every robots.txt version we have observed for example.com.
Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out.
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.