Does xozilla.com allow ClaudeBot? robots.txt as of 2026-09-11
On 2026-09-11 the robots.txt of xozilla.com resolved 19 AI crawler tokens: 11 allowed for / and 8 disallowed for /.
| Domain | xozilla.com |
|---|---|
| AI crawler tokens resolved | 19 |
| Allowed for / | 11 |
| Disallowed for / | 8 |
| robots.txt | robots.txt present, 3215 bytes |
| Checks that ran | robots |
| robots.txt read (tier A) | 2026-09-11 |
| Last verified |
On this domain the robots.txt only check (tier A) last ran on 2026-09-11. The full check adds the home page, the labelled edge probes, llms.txt, the headers and the structured data; the robots.txt check is the one that runs weekly for every seeded domain.
One question per crawler
- GPTBot
- ClaudeBot
- PerplexityBot
- OAI-SearchBot
- CCBot
- Bytespider
- Googlebot
- ChatGPT-User
- Google-Extended
Does xozilla.com allow GPTBot? robots.txt as of 2026-09-11
disallowed for /
robots.txt line 55 in the User-agent: GPTBot group: "Disallow: /" (longest match wins)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow GPTBot
User-agent: GPTBot Allow: /
User-agent: GPTBot Disallow: /
The GPTBot guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow ClaudeBot? robots.txt as of 2026-09-11
disallowed for /
robots.txt line 46 in the User-agent: ClaudeBot group: "Disallow: /" (longest match wins)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow ClaudeBot
User-agent: ClaudeBot Allow: /
User-agent: ClaudeBot Disallow: /
The ClaudeBot guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow PerplexityBot? robots.txt as of 2026-09-11
allowed for /
robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow PerplexityBot
User-agent: PerplexityBot Allow: /
User-agent: PerplexityBot Disallow: /
The PerplexityBot guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow OAI-SearchBot? robots.txt as of 2026-09-11
allowed for /
robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow OAI-SearchBot
User-agent: OAI-SearchBot Allow: /
User-agent: OAI-SearchBot Disallow: /
The OAI-SearchBot guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow CCBot? robots.txt as of 2026-09-11
disallowed for /
robots.txt line 43 in the User-agent: CCBot group: "Disallow: /" (longest match wins)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow CCBot
User-agent: CCBot Allow: /
User-agent: CCBot Disallow: /
The CCBot guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow Bytespider? robots.txt as of 2026-09-11
disallowed for /
robots.txt line 40 in the User-agent: Bytespider group: "Disallow: /" (longest match wins)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow Bytespider
User-agent: Bytespider Allow: /
User-agent: Bytespider Disallow: /
The Bytespider guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow Googlebot? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: Googlebot group with no rule matching / , so / is allowed
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
crawl delay 1 declared: robots.txt line 131: "Crawl-delay: 1"
Allow or disallow Googlebot
User-agent: Googlebot Allow: /
User-agent: Googlebot Disallow: /
The Googlebot guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow ChatGPT-User? robots.txt as of 2026-09-11
allowed for /
robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow ChatGPT-User
User-agent: ChatGPT-User Allow: /
User-agent: ChatGPT-User Disallow: /
The ChatGPT-User guide: identity, purpose, documented verification and the vendor sources.
Does xozilla.com allow Google-Extended? robots.txt as of 2026-09-11
disallowed for /
robots.txt line 52 in the User-agent: Google-Extended group: "Disallow: /" (longest match wins)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow Google-Extended
User-agent: Google-Extended Allow: /
User-agent: Google-Extended Disallow: /
The Google-Extended guide: identity, purpose, documented verification and the vendor sources.
Every crawler token in the registry
| Crawler | Vendor | Observation | Evidence | Observed |
|---|---|---|---|---|
| GPTBot | OpenAI | disallowed for / | robots.txt line 55 in the User-agent: GPTBot group: "Disallow: /" (longest match wins) | 2026-09-11 |
| ClaudeBot | Anthropic | disallowed for / | robots.txt line 46 in the User-agent: ClaudeBot group: "Disallow: /" (longest match wins) | 2026-09-11 |
| PerplexityBot | Perplexity | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| OAI-SearchBot | OpenAI | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| CCBot | Common Crawl Foundation | disallowed for / | robots.txt line 43 in the User-agent: CCBot group: "Disallow: /" (longest match wins) | 2026-09-11 |
| Bytespider | ByteDance | disallowed for / | robots.txt line 40 in the User-agent: Bytespider group: "Disallow: /" (longest match wins) | 2026-09-11 |
| Googlebot | allowed for / | robots.txt has a User-agent: Googlebot group with no rule matching / , so / is allowed | 2026-09-11 | |
| ChatGPT-User | OpenAI | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| Claude-User | Anthropic | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| Claude-SearchBot | Anthropic | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| Perplexity-User | Perplexity | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| anthropic-ai | Anthropic | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| Bingbot | Microsoft | allowed for / | robots.txt has a User-agent: Bingbot group with no rule matching / , so / is allowed | 2026-09-11 |
| Applebot | Apple | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| Amazonbot | Amazon | disallowed for / | robots.txt line 34 in the User-agent: Amazonbot group: "Disallow: /" (longest match wins) | 2026-09-11 |
| meta-externalagent | Meta | disallowed for / | robots.txt line 58 in the User-agent: meta-externalagent group: "Disallow: /" (longest match wins) | 2026-09-11 |
| DuckAssistBot | DuckDuckGo | allowed for / | robots.txt line 31 in the User-agent: * group: "Allow: /" (longest match wins). 2 records name this user agent and RFC 9309 merges them | 2026-09-11 |
| Applebot-Extended | Apple | disallowed for / | robots.txt line 37 in the User-agent: Applebot-Extended group: "Disallow: /" (longest match wins) | 2026-09-11 |
| Google-Extended | disallowed for / | robots.txt line 52 in the User-agent: Google-Extended group: "Disallow: /" (longest match wins) | 2026-09-11 |
What else the robots.txt says
| agents-summary | AI crawler rules: 11 allowed, 8 disallowed 19 crawler tokens resolved against https://xozilla.com/robots.txt (HTTP 200, 3215 bytes) at 2026-09-11T09:17:49Z |
|---|---|
| content-signal | 1 Content-Signal or tdm-reservation line(s) in robots.txt line 30: "Content-Signal: search=yes,ai-train=no,use=reference" |
| crawl-delay:bingbot | crawl delay 3 declared robots.txt line 125: "Crawl-delay: 3" |
| crawl-delay:googlebot | crawl delay 1 declared robots.txt line 131: "Crawl-delay: 1" |
| file | robots.txt present, 3215 bytes GET https://xozilla.com/robots.txt at 2026-09-11T09:17:49Z returned HTTP 200, 3215 bytes, content type text/plain, 27 user agent group(s) |
| sitemap-declared | 1 sitemap(s) declared in robots.txt line 97: "Sitemap: https://www.xozilla.com/sitemap.xml" |
History
Every robots.txt version we have observed for xozilla.com.
Check your own site
The same checks, run on your domain now: what robots.txt tells each AI crawler, whether the edge answers them, and whether the text is readable without JavaScript. Free, no signup.
Limits: 8 checks and 5 different domains per address per hour; a repeat of the same domain inside 7 days is answered from what we already hold.
Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out.
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.