Tranco top 1,000 AI crawler policy (CSV)
The download, its columns and its terms.
The file
95 rows, 22241 bytes, rebuilt daily. Last build: 2026-09-07T00:44:50Z. sha256 8eb9a451da3b6fa4e825c3170fc9abb69e992207c190f8ba0f5f4170d1141834.
The first lines of the file are comments carrying the licence, the attribution requirement and the ranking citation. Every CSV reader skips them, and a human who opens the file sees the terms first.
# AEO Watch: AI crawler policy of the Tranco top 1,000
# Publisher: Bikoosh, https://bikoosh.com/aeo
# Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
# Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo
# Ranking source: Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/
# Citation: AEO Watch Tranco top 1,000 AI crawler policy dataset (Bikoosh, https://bikoosh.com/aeo), retrieved on the access date, CC BY 4.0. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/
# Built at: 2026-09-07T00:44:50Z
# Source signature: rows=2894;last=2026-09-07T00:43:06Z;optout=0;rank=1000
# Rows: 95 over 5 domains, one row per domain and crawler token
# Each row states what that domain's robots.txt said about that token for / on the date in checked_at, with the evidence line. There are no scores, grades or verdicts in this file.
# Domains that opted out of AEO Watch are excluded permanently.
domain,tranco_rank,crawler_token,rule,evidence,checked_at,source
msn.com,66,Amazonbot,disallow,"robots.txt line 201 in the User-agent: Amazonbot group: ""Disallow: /"" (longest match wins)",2026-09-06T23:44:04Z,https://msn.com/robots.txt
msn.com,66,anthropic-ai,disallow,"robots.txt line 201 in the User-agent: anthropic-ai group: ""Disallow: /"" (longest match wins)",2026-09-06T23:44:04Z,https://msn.com/robots.txtColumns
| Column | Meaning |
|---|---|
| domain | the site the robots.txt was read from |
| tranco_rank | its rank in the Tranco list on the day we seeded it |
| crawler_token | the crawler's robots.txt token as its vendor documents it |
| rule | allow, disallow or not mentioned for / , decided from the stored reading: a quoted robots.txt line means allow or disallow, no line naming the token means not mentioned |
| evidence | the exact line and line number the decision came from |
| checked_at | when this observation was made, UTC |
| source | the robots.txt URL the observation was read from |
Licence and attribution
This dataset is published under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).
Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo. Use it anywhere, including commercially, as long as that credit and a link to https://bikoosh.com/aeo travel with it.
The ranking the domain list is drawn from is not ours. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/. Cite it alongside this dataset.
The same two lines are inside the CSV as header comments, so a copy of the file that leaves this page keeps its terms.
Cite this dataset:
AEO Watch Tranco top 1,000 AI crawler policy dataset (Bikoosh, https://bikoosh.com/aeo), retrieved on the access date, CC BY 4.0. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.