AI crawler access in software and developer tools
107 seeded domains sit under .app, .dev, .software, and we have read the robots.txt of 1 of them. This page reports counts and shares for that set only.
| Industry | Software and developer tools |
|---|---|
| Top level domains | .app, .dev, .software |
| Eligibility | stated purpose |
| Registry | Google Registry (.app, .dev) and the .software registry |
| Domains in the set | 107 |
| Domains with a robots.txt reading | 1 |
| Month | 2026-09 |
| Last verified |
How a domain is classified
The rule, in full: a seeded domain is placed in an industry only when its top level domain is one whose registry publishes that purpose. A top level domain marked verified is one where the registry checks that the registrant qualifies; one marked stated purpose is open to register and the registry states an intended use. Country code top level domains are never read as industries here, because a country code says where a registrant is, not what it does: .ai, .io, .co, .tv and .me are country codes and are excluded on purpose. Every other domain is unclassified and appears on no industry page. Nothing is inferred from a name, a word list or a model.
This is deliberately a smaller set than the seeded list. There is no public list of what every website in the world does that we can cite, so the classification stops where the sourcing stops.
The registry behind this classification
Google Registry (.app, .dev) and the .software registry. registration is open; the registry states the intended purpose, and .app and .dev are served with HTTPS required by preload. Source: https://get.app/.
Share disallowing each token for /
| Crawler | Domains read | Allowed for / | Disallowed for / | Share disallowed |
|---|---|---|---|---|
| Amazonbot | 1 | 1 | 0 | 0.0% |
| anthropic-ai | 1 | 1 | 0 | 0.0% |
| Applebot | 1 | 1 | 0 | 0.0% |
| Applebot-Extended | 1 | 1 | 0 | 0.0% |
| Bingbot | 1 | 1 | 0 | 0.0% |
| Bytespider | 1 | 1 | 0 | 0.0% |
| CCBot | 1 | 1 | 0 | 0.0% |
| ChatGPT-User | 1 | 1 | 0 | 0.0% |
| Claude-SearchBot | 1 | 1 | 0 | 0.0% |
| Claude-User | 1 | 1 | 0 | 0.0% |
| ClaudeBot | 1 | 1 | 0 | 0.0% |
| DuckAssistBot | 1 | 1 | 0 | 0.0% |
| Google-Extended | 1 | 1 | 0 | 0.0% |
| Googlebot | 1 | 1 | 0 | 0.0% |
| GPTBot | 1 | 1 | 0 | 0.0% |
| meta-externalagent | 1 | 1 | 0 | 0.0% |
| OAI-SearchBot | 1 | 1 | 0 | 0.0% |
| Perplexity-User | 1 | 1 | 0 | 0.0% |
| PerplexityBot | 1 | 1 | 0 | 0.0% |
Aggregates only. No domain in this set is named on this page.
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.