The AEO Watch API
Every fact on these pages is readable over HTTP, with the same evidence and the same dates.
Nothing in the API starts a live fetch of anyone's site: it answers from what the checker already recorded.
Free tier: 100 checks a day
Pick your own key, at least 16 characters, and send it in X-Api-Key (or the key parameter). It identifies your daily allowance and nothing else: we do not register it, and we store a hash of it with a counter that resets at 00:00 UTC.
The free tier requires attribution. Send X-Attribution (or the attribution parameter) with the URL of the page that will carry the credit line. A request without it is refused with HTTP 400.
The credit line is the CC BY 4.0 one: Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo.
curl -s 'https://bikoosh.com/api/v1/aeo/check?domain=example.org' \
-H 'X-Api-Key: your-own-key-at-least-16-chars' \
-H 'X-Attribution: https://yoursite.example/page-that-credits-us'Member tier: 10000 checks a day
Members of the Bikoosh universal membership authenticate in either of two ways, and both are accepted here.
A Gumroad licence key goes in X-License-Key and is verified against Gumroad's licence endpoint.
Licence keys are switched off on these products, so most receipts do not carry one. In that case send the address you paid with in X-Receipt-Email: it is checked against the live sale, the same binding the activation pages use.
Attribution is not required on the member tier, though we are glad of it.
A membership that was activated on this site is recognised from the activation itself, with no call to the seller. An unrecognised credential is checked against the seller a few times an hour per address: if the answer is that we cannot check right now, the free tier still answers immediately.
curl -s 'https://bikoosh.com/api/v1/aeo/check?domain=example.org' \
-H 'X-License-Key: YOUR-GUMROAD-LICENCE-KEY'
curl -s 'https://bikoosh.com/api/v1/aeo/check?domain=example.org' \
-H 'X-Receipt-Email: [email protected]'Endpoints
curl -s 'https://bikoosh.com/api/v1/aeo/top-1000.json'
curl -s 'https://bikoosh.com/api/v1/aeo/crawlers'| Endpoint | What it answers | Key |
|---|---|---|
| GET /api/v1/aeo/check?domain= | the stored reading for one domain, with the per crawler evidence and the dates. It never triggers a live fetch. | yes, counts against the daily allowance |
| GET /api/v1/aeo/crawlers | the crawler registry: tokens, vendors, documentation URLs, robots.txt snippets. | no |
| GET /api/v1/aeo/top-1000.json | the open dataset as JSON, same rows as the CSV. | no |
Limits and manners
100 checks per key per UTC day on the free tier, 10000 on the member tier. Over the allowance the answer is HTTP 429 and the counter resets at 00:00 UTC.
The free allowance is also 100 checks per address per day, however many keys they are spread over, because a key nobody registers is not a way to ask for more. A shared office address that needs more is what the member tier is for.
Every response carries its licence, its attribution line and the source URL of the page the same facts are published on.
A domain that opted out answers with the opt out and no facts, at every tier, permanently.
Licence and attribution
This dataset is published under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).
Attribution required by CC BY 4.0: AEO Watch by Bikoosh, https://bikoosh.com/aeo. Use it anywhere, including commercially, as long as that credit and a link to https://bikoosh.com/aeo travel with it.
The ranking the domain list is drawn from is not ours. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/. Cite it alongside this dataset.
The same two lines are inside the CSV as header comments, so a copy of the file that leaves this page keeps its terms.
Cite this dataset:
AEO Watch Tranco top 1,000 AI crawler policy dataset (Bikoosh, https://bikoosh.com/aeo), retrieved on the access date, CC BY 4.0. Domain list: Tranco, a research oriented top sites ranking hardened against manipulation (Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, NDSS 2019), https://tranco-list.eu/AEO Watch is an independent, factual monitor. It is not affiliated with, endorsed by or speaking for any crawler vendor. Every statement is an observation with the date it was made and the raw evidence behind it: a robots.txt line, an HTTP status code, a header. There are no scores, no grades and no verdicts here, and nothing on this page is advice. A site opts out at any time and the opt out is honoured automatically and permanently.