AI crawler access by industry
738 of the 19967 seeded domains sit under a top level domain whose registry publishes an industry purpose, so those are the domains these reports cover. Everything else is unclassified and appears on no industry page.
| Industries published | 16 |
|---|---|
| Classified domains | 738 |
| Seeded domains | 19967 |
| Refresh | monthly |
| Month | 2026-09 |
| Last verified |
How a domain is classified
The rule, in full: a seeded domain is placed in an industry only when its top level domain is one whose registry publishes that purpose. A top level domain marked verified is one where the registry checks that the registrant qualifies; one marked stated purpose is open to register and the registry states an intended use. Country code top level domains are never read as industries here, because a country code says where a registrant is, not what it does: .ai, .io, .co, .tv and .me are country codes and are excluded on purpose. Every other domain is unclassified and appears on no industry page. Nothing is inferred from a name, a word list or a model.
This is deliberately a smaller set than the seeded list. There is no public list of what every website in the world does that we can cite, so the classification stops where the sourcing stops.
| Industry | Top level domains | Eligibility | Seeded domains | With a robots.txt reading |
|---|---|---|---|---|
| Education | .edu | verified | 232 | 2 |
| Government and public sector | .gov, .mil | verified | 212 | 1 |
| Banking and insurance | .bank, .insurance | verified | 0 | 0 |
| Pharmacy | .pharmacy | verified | 0 | 0 |
| Legal | .law | verified | 0 | 0 |
| Travel | .travel | verified | 1 | 0 |
| Cooperatives | .coop | verified | 0 | 0 |
| Museums | .museum | verified | 0 | 0 |
| Aviation | .aero | verified | 1 | 0 |
| Software and developer tools | .app, .dev, .software | stated purpose | 107 | 0 |
| Cloud and hosting | .cloud, .hosting, .host | stated purpose | 74 | 0 |
| News and publishing | .news, .press, .media, .blog | stated purpose | 39 | 0 |
| Retail | .shop, .store, .shopping | stated purpose | 20 | 0 |
| Technology | .tech | stated purpose | 47 | 0 |
| Health | .health, .healthcare | stated purpose | 2 | 1 |
| Finance | .finance, .financial | stated purpose | 3 | 0 |
Refreshed monthly. Current month 2026-09.
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.