AI crawler access in software and developer tools

How a domain is classified

The rule, in full: a seeded domain is placed in an industry only when its top level domain is one whose registry publishes that purpose. A top level domain marked verified is one where the registry checks that the registrant qualifies; one marked stated purpose is open to register and the registry states an intended use. Country code top level domains are never read as industries here, because a country code says where a registrant is, not what it does: .ai, .io, .co, .tv and .me are country codes and are excluded on purpose. Every other domain is unclassified and appears on no industry page. Nothing is inferred from a name, a word list or a model.

This is deliberately a smaller set than the seeded list. There is no public list of what every website in the world does that we can cite, so the classification stops where the sourcing stops.

The registry behind this classification

Google Registry (.app, .dev) and the .software registry. registration is open; the registry states the intended purpose, and .app and .dev are served with HTTPS required by preload. Source: https://get.app/.

Share disallowing each token for /

Aggregates only. No domain in this set is named on this page.

Check your own site

The same checks, run on your domain now: what robots.txt tells each AI crawler, whether the edge answers them, and whether the text is readable without JavaScript. Free, no signup.

Limits: 8 checks and 5 different domains per address per hour; a repeat of the same domain inside 7 days is answered from what we already hold.

AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.