{"name": "AI crawler policy of the Tranco top 1,000", "rows": 95, "columns": ["domain", "tranco_rank", "crawler_token", "rule", "evidence", "checked_at", "source"], "csv": "https://bikoosh.com/aeo/top-1000.csv", "page": "https://bikoosh.com/aeo/dataset", "built_at": "2026-09-07T00:44:50Z", "modified": "2026-09-07", "ranking_source": "Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/", "items": [{"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Amazonbot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Amazonbot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "anthropic-ai", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: anthropic-ai group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Applebot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Applebot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Applebot-Extended", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Applebot-Extended group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Bingbot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Bytespider", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Bytespider group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "CCBot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: CCBot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "ChatGPT-User", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: ChatGPT-User group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Claude-SearchBot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Claude-SearchBot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Claude-User", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Claude-User group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "ClaudeBot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: ClaudeBot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "DuckAssistBot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: DuckAssistBot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Google-Extended", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Google-Extended group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Googlebot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "GPTBot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: GPTBot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "meta-externalagent", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: meta-externalagent group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "OAI-SearchBot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: OAI-SearchBot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "Perplexity-User", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: Perplexity-User group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "msn.com", "tranco_rank": "66", "crawler_token": "PerplexityBot", "rule": "disallow", "evidence": "robots.txt line 201 in the User-agent: PerplexityBot group: \"Disallow: /\" (longest match wins)", "checked_at": "2026-09-06T23:44:04Z", "source": "https://msn.com/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Amazonbot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "anthropic-ai", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Applebot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Applebot-Extended", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Bingbot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Bytespider", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "CCBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "ChatGPT-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Claude-SearchBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Claude-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "ClaudeBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "DuckAssistBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Google-Extended", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Googlebot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "GPTBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "meta-externalagent", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "OAI-SearchBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "Perplexity-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "ntp.org", "tranco_rank": "67", "crawler_token": "PerplexityBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:46:30Z", "source": "https://ntp.org/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Amazonbot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "anthropic-ai", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Applebot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Applebot-Extended", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Bingbot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Bytespider", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "CCBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "ChatGPT-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Claude-SearchBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Claude-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "ClaudeBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "DuckAssistBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Google-Extended", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Googlebot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: Googlebot group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "GPTBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "meta-externalagent", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "OAI-SearchBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "Perplexity-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "adobe.com", "tranco_rank": "69", "crawler_token": "PerplexityBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-06T23:48:52Z", "source": "https://adobe.com/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Amazonbot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "anthropic-ai", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Applebot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Applebot-Extended", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Bingbot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Bytespider", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "CCBot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "ChatGPT-User", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Claude-SearchBot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Claude-User", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "ClaudeBot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "DuckAssistBot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Google-Extended", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Googlebot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "GPTBot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "meta-externalagent", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "OAI-SearchBot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "Perplexity-User", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "myfritz.net", "tranco_rank": "70", "crawler_token": "PerplexityBot", "rule": "allow", "evidence": "robots.txt line 6 in the User-agent: * group: \"Allow: /$\" (longest match wins). Line 5 \"Disallow: /\" also matched and lost the longest match test. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)", "checked_at": "2026-09-06T23:48:52Z", "source": "https://myfritz.net/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Amazonbot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "anthropic-ai", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Applebot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Applebot-Extended", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Bingbot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Bytespider", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "CCBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "ChatGPT-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Claude-SearchBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Claude-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "ClaudeBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "DuckAssistBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Google-Extended", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Googlebot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "GPTBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "meta-externalagent", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "OAI-SearchBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "Perplexity-User", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}, {"domain": "wa.me", "tranco_rank": "71", "crawler_token": "PerplexityBot", "rule": "not mentioned", "evidence": "robots.txt has a User-agent: * group with no rule matching / , so / is allowed", "checked_at": "2026-09-07T00:43:06Z", "source": "https://wa.me/robots.txt"}], "source": "https://bikoosh.com/aeo/dataset?utm_source=catalog", "licence": "CC BY 4.0, https://creativecommons.org/licenses/by/4.0/", "attribution": "Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo", "citation": "AEO Watch Tranco top 1,000 AI crawler policy dataset (Bikoosh, https://bikoosh.com/aeo), retrieved on the access date, CC BY 4.0. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/"}