The AI Crawler Access Index: What the Top 1,000 Sites Actually Tell AI Crawlers

Two questions come up in almost every conversation about AI visibility, and neither one had a published answer. How many major websites actually block the crawlers behind AI assistants? And how many have adopted llms.txt, the proposed convention for giving language models a clean summary of a site?

Plenty of people have opinions. We could not find a measurement. So on 19 August 2026 we ran one, and we are publishing the numbers, the method, and the raw per-domain data.

What we measured

We took the Tranco top 1,000 domains, a research-standard ranking that averages several traffic lists to resist manipulation, and requested two files from each: /robots.txt and /llms.txt. For robots.txt we parsed the file properly rather than string-matching, grouping consecutive User-agent lines with the rule block that follows them, and letting a named agent group override the wildcard group the way a compliant crawler would. A crawler counts as blocked only when the group governing it disallows the site root.

Of the 1,000 domains, 597 served a parseable robots.txt. The other 403 are mostly CDN, DNS and API endpoints that serve no robots.txt at all, so every blocking percentage below uses 597 as its denominator, not 1,000.

How many sites block AI crawlers

Among the 597 domains that publish a robots.txt:

• CCBot, the Common Crawl agent, is blocked at the root by 146 sites, or 24.5 percent
• ClaudeBot is blocked by 129 sites, or 21.6 percent
• GPTBot is blocked by 128 sites, or 21.4 percent
• Google-Extended is blocked by 119 sites, or 19.9 percent
• PerplexityBot is blocked by 104 sites, or 17.4 percent

Taken together, 160 sites, or 26.8 percent, block at least one of those five at the root. Only 75 sites, or 12.6 percent, block all five. That gap is the interesting part: most sites making blocking decisions are making them unevenly, agent by agent, rather than closing the door on AI crawlers as a class.

The wider pattern is that 175 sites, or 29.3 percent, name at least one AI agent explicitly in robots.txt. A little under a third of the most-visited sites on the web have now written a rule that did not exist as a category three years ago.

How many sites publish an llms.txt

103 of the 1,000 domains serve a genuine llms.txt file. That is 10.3 percent of the sample, or 16.9 percent of the sites that publish a robots.txt.

That number needs a caveat about how it was produced, because this is exactly where a measurement like this goes wrong. A meaningful share of servers return HTTP 200 with a plain-text body for any path you request, which makes a naive check report an llms.txt that does not exist. So every hit was re-tested against a control path that cannot exist. If the control returned the same 200 body, the hit was discarded. Three domains failed that control and were removed. The 103 that remain each returned a real file for /llms.txt and something different for the control.

Adoption skews heavily toward developer-facing companies. Stripe, Cloudflare, GitHub, Atlassian, Shopify, Slack, Zendesk, Datadog, Sentry and Netlify all publish one. So do Adobe, Salesforce, NVIDIA, PayPal, Samsung, Cisco and Red Hat. Almost no publishers or media sites in the sample do.

What the two numbers say together

The overlap is the finding we did not expect. Of the 597 sites with a robots.txt, only 5 both publish an llms.txt and block one of the five major AI crawlers.

In other words, these are two nearly disjoint populations. One set of companies is deciding how much of itself to withhold from AI systems. A different, smaller set is deciding how to be read accurately by them. Very few are doing both, which suggests most organizations still treat AI access as a single yes-or-no switch rather than as two separate decisions: what a model may train on, and what a model should understand about you when it answers a question.

The second decision is the one that shows up in an answer engine's output, and it is currently being made by about one site in ten.

Reproduce it

The full per-domain dataset is published as JSON, and the measurement script that produced it is published alongside. Both are linked below. The sample frame is public, the parser logic is in the script, and the control probe that filters false positives is described above. Anyone can rerun it against a fresh Tranco list and get a comparable number.

We intend to repeat this measurement and publish the drift. If you cite the figures, cite the date with them: crawler policy is a moving target, and a number like 10.3 percent is only true of the week it was taken.