Tranco top 1,000 AI crawler policy (CSV)

The download, its columns and its terms.

The file

11445 rows, 2264867 bytes, rebuilt daily. Last build: 2026-10-22T11:44:39Z. sha256 c633863fba0318b3a108b2df4b9937d31a253b0911f0ce137bee0af001448308.

The first lines of the file are comments carrying the licence, the attribution requirement and the ranking citation. Every CSV reader skips them, and a human who opens the file sees the terms first.

# AEO Watch: AI crawler policy of the Tranco top 1,000
# Publisher: Bikoosh, https://bikoosh.com/aeo
# Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
# Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo
# Ranking source: Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/
# Citation: AEO Watch Tranco top 1,000 AI crawler policy dataset (Bikoosh, https://bikoosh.com/aeo), retrieved on the access date, CC BY 4.0. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/
# Built at: 2026-10-22T11:44:39Z
# Source signature: rows=328951;last=2026-10-22T04:49:17Z;optout=0;rank=1000
# Rows: 11445 over 545 domains, one row per domain and crawler token
# Each row states what that domain's robots.txt said about that token for / on the date in checked_at, with the evidence line. There are no scores, grades or verdicts in this file.
# Domains that opted out of AEO Watch are excluded permanently.
domain,tranco_rank,crawler_token,rule,evidence,checked_at,source
google.com,1,Amazonbot,not mentioned,"robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)",2026-10-19T00:18:56Z,https://google.com/robots.txt
google.com,1,Amzn-SearchBot,not mentioned,"robots.txt has a User-agent: * group with no rule matching / , so / is allowed. Note: Python urllib.robotparser reads this file as disallowed for this token; this checker applies RFC 9309 (records with the same user agent merged, longest match wins, Allow wins a tie)",2026-10-19T00:18:56Z,https://google.com/robots.txt

• Download the CSV

Columns

Licence and attribution

This dataset is published under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).

Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo. Use it anywhere, including commercially, as long as that credit and a link to https://bikoosh.com/aeo travel with it.

The ranking the domain list is drawn from is not ours. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/. Cite it alongside this dataset.

The same two lines are inside the CSV as header comments, so a copy of the file that leaves this page keeps its terms.

Cite this dataset:
AEO Watch Tranco top 1,000 AI crawler policy dataset (Bikoosh, https://bikoosh.com/aeo), retrieved on the access date, CC BY 4.0. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.