# Does yaml.org allow GPTBot? robots.txt as of 2026-09-10

## In short

On 2026-09-10 the robots.txt of yaml.org resolved 19 AI crawler tokens: 19 allowed for / and 0 disallowed for /.

| Fact | Value |
| --- | --- |
| Domain | yaml.org |
| AI crawler tokens resolved | 19 |
| Allowed for / | 19 |
| Disallowed for / | 0 |
| robots.txt | no robots.txt (HTTP 404) |
| Checks that ran | robots |
| robots.txt read (tier A) | 2026-09-10 |
| Last verified | 2026-09-10 |

## Every AI crawler token in the registry

| Crawler | Vendor | Observation | Evidence | Observed |
|---|---|---|---|---|
| GPTBot | OpenAI | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| ClaudeBot | Anthropic | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| PerplexityBot | Perplexity | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| OAI-SearchBot | OpenAI | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| CCBot | Common Crawl Foundation | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Bytespider | ByteDance | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Googlebot | Google | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| ChatGPT-User | OpenAI | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Claude-User | Anthropic | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Claude-SearchBot | Anthropic | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Perplexity-User | Perplexity | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| anthropic-ai | Anthropic | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Bingbot | Microsoft | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Applebot | Apple | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Amazonbot | Amazon | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| meta-externalagent | Meta | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| DuckAssistBot | DuckDuckGo | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Applebot-Extended | Apple | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |
| Google-Extended | Google | allowed for / | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z answered HTTP 404 and there was no file to read; RFC 9309 section 2.3.1.3 treats an unavailable robots.txt as allow all | 2026-09-10 |

## The robots.txt reading itself

| Observation | Value | Evidence | Observed |
|---|---|---|---|
| file | no robots.txt (HTTP 404) | GET https://yaml.org/robots.txt at 2026-09-10T13:16:52Z returned HTTP 404, 21358 bytes, content type text/html, 0 user agent group(s) | 2026-09-10 |

Robots.txt history: https://bikoosh.com/aeo/site/yaml.org/history

Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out.

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.