# Does msn.com allow GPTBot? robots.txt as of 2026-09-06 ## In short On 2026-09-06 the robots.txt of msn.com resolved 19 AI crawler tokens: 2 allowed for / and 17 disallowed for /. | Fact | Value | | --- | --- | | Domain | msn.com | | AI crawler tokens resolved | 19 | | Allowed for / | 2 | | Disallowed for / | 17 | | robots.txt | robots.txt present, 5245 bytes | | Checks that ran | edge, llms, optout, render, robots, signals, structured, timing | | robots.txt read (tier A) | 2026-09-06 | | Full check (tier B) | 2026-09-06 | | Last verified | 2026-09-06 | ## Every AI crawler token in the registry | Crawler | Vendor | Observation | Evidence | Observed | |---|---|---|---|---| | GPTBot | OpenAI | disallowed for / | robots.txt line 201 in the User-agent: GPTBot group: "Disallow: /" (longest match wins) | 2026-09-06 | | ClaudeBot | Anthropic | disallowed for / | robots.txt line 201 in the User-agent: ClaudeBot group: "Disallow: /" (longest match wins) | 2026-09-06 | | PerplexityBot | Perplexity | disallowed for / | robots.txt line 201 in the User-agent: PerplexityBot group: "Disallow: /" (longest match wins) | 2026-09-06 | | OAI-SearchBot | OpenAI | disallowed for / | robots.txt line 201 in the User-agent: OAI-SearchBot group: "Disallow: /" (longest match wins) | 2026-09-06 | | CCBot | Common Crawl Foundation | disallowed for / | robots.txt line 201 in the User-agent: CCBot group: "Disallow: /" (longest match wins) | 2026-09-06 | | Bytespider | ByteDance | disallowed for / | robots.txt line 201 in the User-agent: Bytespider group: "Disallow: /" (longest match wins) | 2026-09-06 | | Googlebot | Google | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed | 2026-09-06 | | ChatGPT-User | OpenAI | disallowed for / | robots.txt line 201 in the User-agent: ChatGPT-User group: "Disallow: /" (longest match wins) | 2026-09-06 | | Claude-User | Anthropic | disallowed for / | robots.txt line 201 in the User-agent: Claude-User group: "Disallow: /" (longest match wins) | 2026-09-06 | | Claude-SearchBot | Anthropic | disallowed for / | robots.txt line 201 in the User-agent: Claude-SearchBot group: "Disallow: /" (longest match wins) | 2026-09-06 | | Perplexity-User | Perplexity | disallowed for / | robots.txt line 201 in the User-agent: Perplexity-User group: "Disallow: /" (longest match wins) | 2026-09-06 | | anthropic-ai | Anthropic | disallowed for / | robots.txt line 201 in the User-agent: anthropic-ai group: "Disallow: /" (longest match wins) | 2026-09-06 | | Bingbot | Microsoft | allowed for / | robots.txt has a User-agent: * group with no rule matching / , so / is allowed | 2026-09-06 | | Applebot | Apple | disallowed for / | robots.txt line 201 in the User-agent: Applebot group: "Disallow: /" (longest match wins) | 2026-09-06 | | Amazonbot | Amazon | disallowed for / | robots.txt line 201 in the User-agent: Amazonbot group: "Disallow: /" (longest match wins) | 2026-09-06 | | meta-externalagent | Meta | disallowed for / | robots.txt line 201 in the User-agent: meta-externalagent group: "Disallow: /" (longest match wins) | 2026-09-06 | | DuckAssistBot | DuckDuckGo | disallowed for / | robots.txt line 201 in the User-agent: DuckAssistBot group: "Disallow: /" (longest match wins) | 2026-09-06 | | Applebot-Extended | Apple | disallowed for / | robots.txt line 201 in the User-agent: Applebot-Extended group: "Disallow: /" (longest match wins) | 2026-09-06 | | Google-Extended | Google | disallowed for / | robots.txt line 201 in the User-agent: Google-Extended group: "Disallow: /" (longest match wins) | 2026-09-06 | ## The robots.txt reading itself | Observation | Value | Evidence | Observed | |---|---|---|---| | agents-summary | AI crawler rules: 2 allowed, 17 disallowed | 19 crawler tokens resolved against https://msn.com/robots.txt (HTTP 200, 5245 bytes) at 2026-09-06T23:42:02Z | 2026-09-06 | | file | robots.txt present, 5245 bytes | GET https://msn.com/robots.txt at 2026-09-06T23:42:02Z returned HTTP 200, 5245 bytes, content type text/plain, 11 user agent group(s) | 2026-09-06 | | sitemap-declared | 6 sitemap(s) declared in robots.txt | line 203: "Sitemap: https://www.msn.com/sitemaps/casualgames/sitemap-index.xml"; line 204: "Sitemap: https://www.msn.com/sitemaps/consumption/index.xml"; line 205: "Sitemap: https://www.msn.com/sitemaps/health/health-sitemap-index.xml" | 2026-09-06 | ## edge - agent:bytespider: answers the same as the baseline for the token Bytespider. Evidence: baseline GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/Bytespider" returned HTTP 200, 48000 bytes, observed 2026-09-06T23:42:02Z. Observed 2026-09-06. User Agent sent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/Bytespider Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. - agent:ccbot: answers the same as the baseline for the token CCBot. Evidence: baseline GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/CCBot" returned HTTP 200, 48000 bytes, observed 2026-09-06T23:42:02Z. Observed 2026-09-06. User Agent sent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/CCBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. - agent:claudebot: answers the same as the baseline for the token ClaudeBot. Evidence: baseline GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/ClaudeBot" returned HTTP 200, 48000 bytes, observed 2026-09-06T23:42:02Z. Observed 2026-09-06. User Agent sent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/ClaudeBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. - agent:gptbot: answers the same as the baseline for the token GPTBot. Evidence: baseline GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/GPTBot" returned HTTP 200, 43932 bytes, observed 2026-09-06T23:42:02Z. Observed 2026-09-06. User Agent sent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/GPTBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. - agent:oai-searchbot: answers the same as the baseline for the token OAI-SearchBot. Evidence: baseline GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/OAI-SearchBot" returned HTTP 200, 48000 bytes, observed 2026-09-06T23:42:02Z. Observed 2026-09-06. User Agent sent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/OAI-SearchBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. - agent:perplexitybot: answers the same as the baseline for the token PerplexityBot. Evidence: baseline GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) returned HTTP 200; the same request with the User Agent "BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/PerplexityBot" returned HTTP 200, 48000 bytes, observed 2026-09-06T23:42:02Z. Observed 2026-09-06. User Agent sent: BikooshBot/1.0 (+https://bikoosh.com/bot) ua-probe/PerplexityBot Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. - baseline: home page answered HTTP 200 to our own User Agent. Evidence: GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) at 2026-09-06T23:42:02Z: HTTP 200, 43932 bytes. Observed 2026-09-06. User Agent sent: BikooshBot/1.0 (+https://bikoosh.com/bot) Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. ## llms - llms-full.txt: llms-full.txt not present (HTTP 404). Evidence: GET https://msn.com/llms-full.txt at 2026-09-06T23:42:02Z returned HTTP 404. Observed 2026-09-06. - llms.txt: llms.txt not present (HTTP 404). Evidence: GET https://msn.com/llms.txt at 2026-09-06T23:42:02Z returned HTTP 404. Observed 2026-09-06. ## optout - signal: no opt out signal published. Evidence: TXT _bikoosh-aeo.msn.com carried no opt out and GET https://msn.com/.well-known/bikoosh-aeo-optout published none either (it answered HTTP 404) at 2026-09-06T23:42:02Z. Observed 2026-09-06. ## render - main-content: main content requires JavaScript. Evidence: visible text of the raw HTML 0 characters versus 1303 characters after rendering https://www.msn.com/de-de in headless Chromium at 2026-09-06T23:42:02Z: 0.0 percent, threshold 20 percent. Observed 2026-09-06. ## signals - canonical: no canonical link element. Evidence: no link rel=canonical in the HTML of https://www.msn.com/de-de at 2026-09-06T23:42:02Z. Observed 2026-09-06. - noai: no noai declaration. Evidence: neither the meta tags nor the X-Robots-Tag header of https://www.msn.com/de-de carried noai at 2026-09-06T23:42:02Z. Observed 2026-09-06. - noimageai: no noimageai declaration. Evidence: neither the meta tags nor the X-Robots-Tag header of https://www.msn.com/de-de carried noimageai at 2026-09-06T23:42:02Z. Observed 2026-09-06. - robots-meta: no robots meta tag. Evidence: no meta name=robots tag in the HTML of https://www.msn.com/de-de at 2026-09-06T23:42:02Z (2 meta tags read). Observed 2026-09-06. - sitemap: declared sitemap answered HTTP 200. Evidence: HEAD https://www.msn.com/sitemaps/casualgames/sitemap-index.xml at 2026-09-06T23:42:02Z returned HTTP 200, content type text/xml. Observed 2026-09-06. - tdm-reservation: no tdm-reservation header. Evidence: no tdm-reservation header on https://www.msn.com/de-de at 2026-09-06T23:42:02Z. Observed 2026-09-06. - x-robots-tag: no X-Robots-Tag header. Evidence: response headers of https://www.msn.com/de-de at 2026-09-06T23:42:02Z: no X-Robots-Tag header present. Observed 2026-09-06. ## structured - json-ld: no JSON-LD in the home page HTML. Evidence: no script type=application/ld+json in https://www.msn.com/de-de at 2026-09-06T23:42:02Z. Observed 2026-09-06. ## timing - page-bytes: home page HTML 25 to 50 KB. Evidence: GET https://www.msn.com/de-de at 2026-09-06T23:42:02Z transferred 43932 bytes of HTML, content type text/html; charset=utf-8 (the value is banded, the exact byte count is here). Observed 2026-09-06. - response-ms: home page answered in 200 to 500 ms. Evidence: GET https://www.msn.com/de-de as BikooshBot/1.0 (+https://bikoosh.com/bot) at 2026-09-06T23:42:02Z: HTTP 200 in 283 ms (the value is banded so that ordinary network jitter is not reported as a change). Observed 2026-09-06. Robots.txt history: https://bikoosh.com/aeo/site/msn.com/history Limitation: this test appends a labelled marker to our own honest User Agent, it never impersonates another agent. Rules keyed to an exact User Agent string, or to a vendor's IP ranges, are invisible to this method and are neither confirmed nor ruled out. AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.