AI crawler policy of the Tranco top 1,000
A free, citable CSV of what the most visited sites tell AI crawlers in robots.txt, with the evidence line under every row.
What is in it
One row per domain and per AI crawler token: 76 rows built at 2026-09-07T00:20:04Z. Each row states what that domain's robots.txt said about that token for / on the date in checked_at, and quotes the line it came from.
Coverage right now: we hold a robots.txt reading for 4 of the 664 domains of the top 1000 that are in our seed list. The weekly crawl fills the rest, and a domain arrives here the week it is read, not before.
There is no score column, no grade and no verdict. A row is an observation with its evidence and its date.
Domains that opted out of AEO Watch are excluded permanently, and a domain we hold no robots.txt reading for has no rows.
| Column | Meaning |
|---|---|
| domain | the site the robots.txt was read from |
| tranco_rank | its rank in the Tranco list on the day we seeded it |
| crawler_token | the crawler's robots.txt token as its vendor documents it |
| rule | allow, disallow or not mentioned for / , decided from the stored reading: a quoted robots.txt line means allow or disallow, no line naming the token means not mentioned |
| evidence | the exact line and line number the decision came from |
| checked_at | when this observation was made, UTC |
| source | the robots.txt URL the observation was read from |
Download
- Tranco top 1,000 AI crawler policy (CSV) - 19154 bytes, CC BY 4.0
- The CSV's own page (with the field dictionary)
- The same rows as JSON
- Croissant metadata
- Frictionless Data Package
- DCAT catalog
Licence and attribution
This dataset is published under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).
Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo. Use it anywhere, including commercially, as long as that credit and a link to https://bikoosh.com/aeo travel with it.
The ranking the domain list is drawn from is not ours. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/. Cite it alongside this dataset.
The same two lines are inside the CSV as header comments, so a copy of the file that leaves this page keeps its terms.
Cite this dataset:
AEO Watch Tranco top 1,000 AI crawler policy dataset (Bikoosh, https://bikoosh.com/aeo), retrieved on the access date, CC BY 4.0. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/The rest of the machine surface
- The API, free tier and member tier
- The weekly AI crawler access index
- Weekly access index feed (RSS)
- The embeddable badge for site owners
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.