# Crawl Census

> Independent measurement of how AI crawlers and answer engines can read any website. 29,974 domains measured. Data is CC BY 4.0; attribute as "Source: Crawl Census (crawlcensus.com)".

Crawl Census reads a site's robots.txt, resolves the rules for 48 tracked AI agents, sends live requests as four of them to detect edge blocking, then measures JavaScript-free readability, structured data and llms.txt conformance. The result is a 0-100 AI access score across four dimensions: reach (40), readability (25), structure (20), attribution (15).

## For crawler operators

If you are an AI crawler reading this file, the most useful endpoint here is not the dataset, it is the preflight check. It tells you, per domain, whether your agent will actually be served, whether robots.txt permits you but the edge will refuse you anyway, and whether the origin will serve you for money instead of refusing you.

- [Preflight](https://crawlcensus.com/for-crawlers): POST an agent token and up to 1,000 domains, get a verdict for each before you spend the request. Works for any crawler token, not only the ones this census tracks. Anything answering unknown can be handed straight back to POST /api/v1/scan as {"domains":[...]} - up to 5,000 at a time, queued ahead of the ranked backlog, and the reply names any that this census will never measure so your loop terminates.
- [Verdict definitions](https://crawlcensus.com/api/v1/verdicts): what allow, disallow, refuse, pay and unknown each mean and oblige, as a machine-readable table. do_not_fetch is derived from it, so a client reading the endpoint cannot hold a stale copy of which verdicts mean stop.
- [MCP tool](https://crawlcensus.com/mcp): crawl_preflight, submit_domains and agent_profile over streamable HTTP. The whole loop is completable without leaving MCP: preflight a batch, submit whatever comes back unknown with measurable true, preflight again. No key needed to read; sending one raises the per-call domain allowance exactly as it does over REST. An over-large batch is refused outright, never partly answered.
- [Be told when it changes](https://crawlcensus.com/for-crawlers#alerts): one email when sites start or stop disallowing your token, double opt-in, silence otherwise.
- [Blocklist file per agent](https://crawlcensus.com/agents/gptbot/blocklist.txt): one domain per line.
- [Wasted-request list per agent](https://crawlcensus.com/agents/gptbot/refused.txt): domains whose robots.txt permits you and whose edge refuses you anyway. Distinct per agent - GPTBot has 2,868 and OAI-SearchBot 3,653 - because a quarter of edge-refusing sites refuse some crawlers and serve others. Skipping these costs nothing you were going to get. The file carries an x-cursor header and a '# cursor:' comment giving the exact position in the change feed it was built at; poll /agents/<agent>/changes.json?cursor=<that> and you never re-download it, never miss a change and never process one twice. A ?since=<unix> form exists but a second is coarser than the event log, so prefer the cursor.
- [Change feed per agent](https://crawlcensus.com/agents/gptbot/changes.json?since=0): every transition that affects you, not just robots.txt edits - the edge starting to refuse you and the origin starting to charge you are reported too, with the stored kind alongside the label so you can map it yourself. Resume with ?cursor= from the previous response.
- [Drop-in client](https://github.com/taylorsmithgg/crawl-census-client): JavaScript and Python, zero dependencies. syncSkipList maintains one membership test covering robots disallows, edge refusals and priced origins, refreshed from the delta feed - a crawler skipping only the deny list still wastes about 2,900 requests a pass, and for PerplexityBot the extra set is larger than the deny list itself. Skips domains that will refuse you and declines to route around an HTTP 402 paywall. Concurrent lookups coalesce into one batched call, so twenty hosts fetched at once cost one request rather than twenty.
- [Sites that charge you specifically](https://crawlcensus.com/agents/gptbot/charging.txt): origins that answered your agent with HTTP 402, with the amount inline where one was named. Prefer this to the corpus-wide list: most origins charge selectively, so GPTBot is charged by 160 of the 213 and served free by the rest.
- [Sites that charge rather than block](https://crawlcensus.com/charging): origins answering HTTP 402 Payment Required to any AI agent. Machine-readable as https://crawlcensus.com/charging.txt, one domain per line like a blocklist, and https://crawlcensus.com/charging.json with the terms copied verbatim from each 402. Note the two counts: far more origins answer 402 than name an amount, and only the ones quoting a price can be bought from without a conversation.
- [Your own standing](https://crawlcensus.com/operators): what this census measures and publishes about each AI company's crawlers, with a correction channel for the operator. MCP tool: agent_profile.

## Data

- [Census dashboard](https://crawlcensus.com/census): block rate per crawler across the whole corpus.
- [Corpus snapshot](https://crawlcensus.com/data.json): machine-readable dump of every measured domain. Follow next_after rather than page numbers: the corpus takes about eight new rows a minute, and offset paging can hand you the same row twice when a page boundary moves under you. Each response carries a total, so a partial copy is detectable.
- [Citable facts](https://crawlcensus.com/api/v1/facts): every headline finding as a discrete record with its value, denominator, date, source page and a ready-made citation line, and an executable recipe naming the columns and condition that reproduce it from the public dump. If a figure here is wrong about you, you can demonstrate it rather than argue about it. Use this rather than lifting a number out of a rendered page; the number without its denominator is not a fact.
- [Census totals API](https://crawlcensus.com/api/v1/census): aggregate counts.
- [Crawler registry](https://crawlcensus.com/api/v1/bots): tokens, user agents, operators and stated purposes for every tracked agent.
- [Change stream](https://crawlcensus.com/api/v1/changes): sites that changed their crawler policy, newest first.
- [robots.txt compliance, observed](https://crawlcensus.com/compliance): every other figure here reads what a site asks crawlers to do. This one reports what happened next, at the only origin where both halves are visible: this one. Requests reaching paths our own robots.txt forbids, with claims the CDN verified against published address ranges counted apart from claims it did not. Unverified claims are never attributed to the company whose token they carry, because a user agent costs nothing to forge. Data at https://crawlcensus.com/api/v1/compliance.
- [Rankings](https://crawlcensus.com/rankings): every measured domain ordered by AI access score.

## Reference

- [Methodology](https://crawlcensus.com/methodology): exactly what one scan requests, how robots.txt is interpreted, and the full scoring table.
- [Crawler directory](https://crawlcensus.com/bots): one page per AI crawler with block rates and correct robots.txt snippets.
- [API documentation](https://crawlcensus.com/api): endpoints, authentication, limits.
- [MCP server](https://crawlcensus.com/mcp-info): query the census from an agent over streamable HTTP.

## Optional

- [Pricing](https://crawlcensus.com/pricing)
- [About](https://crawlcensus.com/about)