Google-Extended: robots token, verification and how to allow or disallow it
Google-Extended is an AI crawler operated by Google. Its robots.txt token is Google-Extended, and the vendor documentation we fetched on 2026-09-06 is the source for every statement on this page.
| Robots token | Google-Extended |
|---|---|
| Vendor | |
| Purpose | training corpus, grounding |
| Documented verification | published IP ranges |
| Vendor documentation last verified | 2026-09-06 |
| Share of read domains disallowing it for / | 17.2% of 128 |
| Last verified |
Identity
| Robots token | Google-Extended |
|---|---|
| Matched as | google-extended |
| Vendor | |
| Purpose | training corpus, grounding |
| Documentation | https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers |
| Documented verification | published IP ranges: https://developers.google.com/static/search/apis/ipranges/special-crawlers.json |
probe_supported is false on the vendor's own evidence: 'Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.' It controls use of content for Gemini training and for grounding, and the vendor states it does not affect inclusion or ranking in Google Search. An edge probe carrying this token would therefore prove nothing and we do not run one.
Allow or disallow it
User-agent: Google-Extended Allow: /
User-agent: Google-Extended Disallow: /
A robots.txt rule is a request that a well behaved crawler honours. It is not an access control, and this page does not tell anyone what to choose.
Questions
Does Google-Extended obey robots.txt?
The vendor documentation at https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers states that Google-Extended respects robots.txt. We fetched that page and it answered our checker on 2026-09-06. This records what the vendor documents, not what any individual request did.
How do I allow Google-Extended in robots.txt?
Add this group to the robots.txt at the root of the host: User-agent: Google-Extended Allow: /. The token is matched case insensitively as a substring of the user agent by RFC 9309, and the longest matching rule wins.
How do I disallow Google-Extended in robots.txt?
Add this group to the robots.txt at the root of the host: User-agent: Google-Extended Disallow: /. A robots.txt rule is a request that a well behaved crawler honours; it is not an access control.
How do I verify a request really came from Google-Extended?
The vendor publishes the IP ranges this crawler fetches from at https://developers.google.com/static/search/apis/ipranges/special-crawlers.json. Check the address of the request against that file. A user agent string on its own proves nothing, because anyone can send one.
What we measure
Of the 128 seeded domains whose robots.txt we have read, 22 disallow Google-Extended for / and 106 allow it, which is 17.2% disallowed, week 2026-W37. Counts only: no domain is named.
The weekly AI crawler access index
Sources
| Source | Type | HTTP | Verified | Note |
|---|---|---|---|---|
| https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers | vendor | 200 | 2026-09-06 | 6 occurrences of Google-Extended in the fetched body |
| https://developers.google.com/static/search/apis/ipranges/special-crawlers.json | vendor | 200 | 2026-09-06 | published IP ranges; fetched 200 by scripts/aeowatch_crawlers_refresh.py on 2026-09-06 and also refreshed weekly into data/bot_ranges.json (270 prefixes on disk) |
Free data
Check a domain against every token
AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.