AI crawler access in education

How a domain is classified

The rule, in full: a seeded domain is placed in an industry only when its top level domain is one whose registry publishes that purpose. A top level domain marked verified is one where the registry checks that the registrant qualifies; one marked stated purpose is open to register and the registry states an intended use. Country code top level domains are never read as industries here, because a country code says where a registrant is, not what it does: .ai, .io, .co, .tv and .me are country codes and are excluded on purpose. Every other domain is unclassified and appears on no industry page. Nothing is inferred from a name, a word list or a model.

This is deliberately a smaller set than the seeded list. There is no public list of what every website in the world does that we can cite, so the classification stops where the sourcing stops.

The registry behind this classification

EDUCAUSE, under a cooperative agreement with the United States Department of Commerce. registration is limited to accredited United States postsecondary institutions. Source: https://net.educause.edu/edu-domain.

Share disallowing each token for /

Aggregates only. No domain in this set is named on this page.

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.