# Does yelp.com allow GPTBot? robots.txt as of 2026-09-10

## In short

On 2026-09-10 the robots.txt of yelp.com resolved 19 AI crawler tokens: 2 allowed for / and 17 disallowed for /.

| Fact | Value |
| --- | --- |
| Domain | yelp.com |
| AI crawler tokens resolved | 19 |
| Allowed for / | 2 |
| Disallowed for / | 17 |
| robots.txt | robots.txt present, 11825 bytes |
| Checks that ran | robots |
| robots.txt read (tier A) | 2026-09-10 |
| Full check (tier B) | 2026-09-10 |
| Last verified | 2026-09-10 |

## Every AI crawler token in the registry

| Crawler | Vendor | Observation | Evidence | Observed |
|---|---|---|---|---|
| GPTBot | OpenAI | disallowed for / | robots.txt line 333 in the User-agent: GPTBot group: "Disallow: /" (longest match wins) | 2026-09-10 |
| ClaudeBot | Anthropic | disallowed for / | robots.txt line 333 in the User-agent: ClaudeBot group: "Disallow: /" (longest match wins) | 2026-09-10 |
| PerplexityBot | Perplexity | disallowed for / | robots.txt line 322 in the User-agent: PerplexityBot group: "Disallow: /" (longest match wins) | 2026-09-10 |
| OAI-SearchBot | OpenAI | disallowed for / | robots.txt line 312 in the User-agent: OAI-SearchBot group: "Disallow: /" (longest match wins) | 2026-09-10 |
| CCBot | Common Crawl Foundation | disallowed for / | robots.txt line 333 in the User-agent: CCBot group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Bytespider | ByteDance | disallowed for / | robots.txt line 336 in the User-agent: * group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Googlebot | Google | allowed for / | robots.txt has a User-agent: Googlebot group with no rule matching / , so / is allowed. 3 records name this user agent and RFC 9309 merges them | 2026-09-10 |
| ChatGPT-User | OpenAI | disallowed for / | robots.txt line 322 in the User-agent: ChatGPT-User group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Claude-User | Anthropic | disallowed for / | robots.txt line 322 in the User-agent: Claude-User group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Claude-SearchBot | Anthropic | disallowed for / | robots.txt line 322 in the User-agent: Claude-SearchBot group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Perplexity-User | Perplexity | disallowed for / | robots.txt line 322 in the User-agent: Perplexity-User group: "Disallow: /" (longest match wins) | 2026-09-10 |
| anthropic-ai | Anthropic | disallowed for / | robots.txt line 333 in the User-agent: anthropic-ai group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Bingbot | Microsoft | allowed for / | robots.txt has a User-agent: Bingbot group with no rule matching / , so / is allowed. 2 records name this user agent and RFC 9309 merges them | 2026-09-10 |
| Applebot | Apple | disallowed for / | robots.txt line 336 in the User-agent: * group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Amazonbot | Amazon | disallowed for / | robots.txt line 336 in the User-agent: * group: "Disallow: /" (longest match wins) | 2026-09-10 |
| meta-externalagent | Meta | disallowed for / | robots.txt line 333 in the User-agent: meta-externalagent group: "Disallow: /" (longest match wins) | 2026-09-10 |
| DuckAssistBot | DuckDuckGo | disallowed for / | robots.txt line 336 in the User-agent: * group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Applebot-Extended | Apple | disallowed for / | robots.txt line 336 in the User-agent: * group: "Disallow: /" (longest match wins) | 2026-09-10 |
| Google-Extended | Google | disallowed for / | robots.txt line 333 in the User-agent: Google-Extended group: "Disallow: /" (longest match wins) | 2026-09-10 |

## The robots.txt reading itself

| Observation | Value | Evidence | Observed |
|---|---|---|---|
| agents-summary | AI crawler rules: 2 allowed, 17 disallowed | 19 crawler tokens resolved against https://yelp.com/robots.txt (HTTP 200, 11825 bytes) at 2026-09-10T23:46:59Z | 2026-09-10 |
| file | robots.txt present, 11825 bytes | GET https://yelp.com/robots.txt at 2026-09-10T23:46:59Z returned HTTP 200, 11825 bytes, content type text/plain, 9 user agent group(s) | 2026-09-10 |
| our-checker | robots.txt disallows our checker | robots.txt line 336: "Disallow: /" applies to BikooshBot/1.0 (+https://bikoosh.com/bot); nothing further was fetched | 2026-09-10 |

Robots.txt history: https://bikoosh.com/aeo/site/yelp.com/history

Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out.

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.