Does ya.ru allow GPTBot? robots.txt as of 2026-09-11
On 2026-09-11 the robots.txt of ya.ru resolved 19 AI crawler tokens: 19 allowed for / and 0 disallowed for /.
| Domain | ya.ru |
|---|---|
| AI crawler tokens resolved | 19 |
| Allowed for / | 19 |
| Disallowed for / | 0 |
| robots.txt | robots.txt present, 29928 bytes |
| Checks that ran | edge, llms, optout, render, robots, signals, structured, timing |
| robots.txt read (tier A) | 2026-09-11 |
| Full check (tier B) | 2026-09-11 |
| Last verified |
On this domain the robots.txt only check (tier A) last ran on 2026-09-11 and the full check (tier B) last ran on 2026-09-11. The full check adds the home page, the labelled edge probes, llms.txt, the headers and the structured data; the robots.txt check is the one that runs weekly for every seeded domain.
One question per crawler
- GPTBot
- ClaudeBot
- PerplexityBot
- OAI-SearchBot
- CCBot
- Bytespider
- Googlebot
- ChatGPT-User
- Google-Extended
Does ya.ru allow GPTBot? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow GPTBot
User-agent: GPTBot Allow: /
User-agent: GPTBot Disallow: /
The GPTBot guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow ClaudeBot? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow ClaudeBot
User-agent: ClaudeBot Allow: /
User-agent: ClaudeBot Disallow: /
The ClaudeBot guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow PerplexityBot? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow PerplexityBot
User-agent: PerplexityBot Allow: /
User-agent: PerplexityBot Disallow: /
The PerplexityBot guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow OAI-SearchBot? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow OAI-SearchBot
User-agent: OAI-SearchBot Allow: /
User-agent: OAI-SearchBot Disallow: /
The OAI-SearchBot guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow CCBot? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow CCBot
User-agent: CCBot Allow: /
User-agent: CCBot Disallow: /
The CCBot guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow Bytespider? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow Bytespider
User-agent: Bytespider Allow: /
User-agent: Bytespider Disallow: /
The Bytespider guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow Googlebot? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: Googlebot group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow Googlebot
User-agent: Googlebot Allow: /
User-agent: Googlebot Disallow: /
The Googlebot guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow ChatGPT-User? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow ChatGPT-User
User-agent: ChatGPT-User Allow: /
User-agent: ChatGPT-User Disallow: /
The ChatGPT-User guide: identity, purpose, documented verification and the vendor sources.
Does ya.ru allow Google-Extended? robots.txt as of 2026-09-11
allowed for /
robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)
Observed 2026-09-11, first observed 2026-09-11, last change recorded 2026-09-11.
Allow or disallow Google-Extended
User-agent: Google-Extended Allow: /
User-agent: Google-Extended Disallow: /
The Google-Extended guide: identity, purpose, documented verification and the vendor sources.
Every crawler token in the registry
| Crawler | Vendor | Observation | Evidence | Observed |
|---|---|---|---|---|
| GPTBot | OpenAI | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| ClaudeBot | Anthropic | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| PerplexityBot | Perplexity | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| OAI-SearchBot | OpenAI | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| CCBot | Common Crawl Foundation | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Bytespider | ByteDance | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Googlebot | allowed for / | robots.txt has a User-agent: Googlebot group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 | |
| ChatGPT-User | OpenAI | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Claude-User | Anthropic | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Claude-SearchBot | Anthropic | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Perplexity-User | Perplexity | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| anthropic-ai | Anthropic | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Bingbot | Microsoft | allowed for / | robots.txt has a User-agent: Bingbot group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Applebot | Apple | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Amazonbot | Amazon | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| meta-externalagent | Meta | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| DuckAssistBot | DuckDuckGo | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Applebot-Extended | Apple | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
| Google-Extended | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie) | 2026-09-11 |
What else the robots.txt says
| agents-summary | AI crawler rules: 19 allowed, 0 disallowed 19 crawler tokens resolved against https://ya.ru/robots.txt (HTTP 200, 29928 bytes) at 2026-09-11T11:42:01Z |
|---|---|
| file | robots.txt present, 29928 bytes GET https://ya.ru/robots.txt at 2026-09-11T11:42:01Z returned HTTP 200, 29928 bytes, content type text/plain, 5 user agent group(s) |
| sitemap-declared | 3 sitemap(s) declared in robots.txt line 1100: "Sitemap: https://ya.ru/medicine/sitemaps/sitemap.xml"; line 1101: "Sitemap: https://ya.ru/finance/sitemaps/sitemap.xml"; line 1102: "Sitemap: https://ya.ru/neurum/sitemaps/sitemap.xml.gz" |
The other checks
edge
| agent:bytespider | answers the same as the baseline for the token Bytespider baseline GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/Bytespider" returned HTTP 200, 48000 bytes, observed 2026-09-11T11:42:01Z probe User Agent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/Bytespider Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. |
|---|---|
| agent:ccbot | answers the same as the baseline for the token CCBot baseline GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/CCBot" returned HTTP 200, 48000 bytes, observed 2026-09-11T11:42:01Z probe User Agent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/CCBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. |
| agent:claudebot | answers the same as the baseline for the token ClaudeBot baseline GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/ClaudeBot" returned HTTP 200, 48000 bytes, observed 2026-09-11T11:42:01Z probe User Agent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/ClaudeBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. |
| agent:gptbot | answers the same as the baseline for the token GPTBot baseline GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/GPTBot" returned HTTP 200, 48000 bytes, observed 2026-09-11T11:42:01Z probe User Agent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/GPTBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. |
| agent:oai-searchbot | answers the same as the baseline for the token OAI-SearchBot baseline GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/OAI-SearchBot" returned HTTP 200, 48000 bytes, observed 2026-09-11T11:42:01Z probe User Agent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/OAI-SearchBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. |
| agent:perplexitybot | answers the same as the baseline for the token PerplexityBot baseline GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/PerplexityBot" returned HTTP 200, 48000 bytes, observed 2026-09-11T11:42:01Z probe User Agent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/PerplexityBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. |
| baseline | home page answered HTTP 200 to our own User Agent GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) at 2026-09-11T11:42:01Z: HTTP 200, 496144 bytes probe User Agent: BikooshBot/1.0 (+https://bikoosh.com/bot) Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. |
llms
| llms-full.txt | llms-full.txt not present (HTTP 404) GET https://ya.ru/llms-full.txt at 2026-09-11T11:42:01Z returned HTTP 404 |
|---|---|
| llms.txt | llms.txt not present (HTTP 404) GET https://ya.ru/llms.txt at 2026-09-11T11:42:01Z returned HTTP 404 |
optout
| signal | no opt out signal published TXT _bikoosh-aeo.ya.ru carried no opt out and GET https://ya.ru/.well-known/bikoosh-aeo-optout published none either (it answered HTTP 404) at 2026-09-11T11:42:02Z |
|---|
render
| main-content | main content is present in the raw HTML visible text of the raw HTML 206 characters versus 233 characters after rendering https://ya.ru/ in headless Chromium at 2026-09-11T11:42:01Z: 88.4 percent, threshold 20 percent |
|---|
signals
| canonical | canonical link: https://ya.ru link rel=canonical in the HTML of https://ya.ru/ at 2026-09-11T11:42:01Z: "https://ya.ru" |
|---|---|
| noai | no noai declaration neither the meta tags nor the X-Robots-Tag header of https://ya.ru/ carried noai at 2026-09-11T11:42:01Z |
| noimageai | no noimageai declaration neither the meta tags nor the X-Robots-Tag header of https://ya.ru/ carried noimageai at 2026-09-11T11:42:01Z |
| robots-meta | no robots meta tag no meta name=robots tag in the HTML of https://ya.ru/ at 2026-09-11T11:42:01Z (22 meta tags read) |
| sitemap | declared sitemap answered HTTP 200 HEAD https://ya.ru/medicine/sitemaps/sitemap.xml at 2026-09-11T11:42:01Z returned HTTP 200, content type application/xml |
| tdm-reservation | no tdm-reservation header no tdm-reservation header on https://ya.ru/ at 2026-09-11T11:42:01Z |
| x-robots-tag | no X-Robots-Tag header response headers of https://ya.ru/ at 2026-09-11T11:42:01Z: no X-Robots-Tag header present |
structured
| json-ld | no JSON-LD in the home page HTML no script type=application/ld+json in https://ya.ru/ at 2026-09-11T11:42:01Z |
|---|
timing
| page-bytes | home page HTML 250 to 500 KB GET https://ya.ru/ at 2026-09-11T11:42:01Z transferred 496144 bytes of HTML, content type text/html; charset=UTF-8 (the value is banded, the exact byte count is here) |
|---|---|
| response-ms | home page answered in 200 to 500 ms GET https://ya.ru/ as BikooshBot/1.0 (+https://bikoosh.com/bot) at 2026-09-11T11:42:01Z: HTTP 200 in 455 ms (the value is banded so that ordinary network jitter is not reported as a change) |
History
Every robots.txt version we have observed for ya.ru.
Check your own site
The same checks, run on your domain now: what robots.txt tells each AI crawler, whether the edge answers them, and whether the text is readable without JavaScript. Free, no signup.
Limits: 8 checks and 5 different domains per address per hour; a repeat of the same domain inside 7 days is answered from what we already hold.
Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out.
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.