We Checked 100,000 Websites for llms.txt and AI-Crawler Blocking. Here's What We Found.
A real 100,000-domain census of llms.txt adoption and AI-crawler blocking rates — full raw data and script public on GitHub. 1 in 5 sites have llms.txt; Anthropic's own three crawler identities get blocked at wildly different rates from each other.
Most of what gets said about "AI crawlers blocking websites" is guesswork. We ran the actual numbers instead — a real census of 100,000 real domains (from Tranco, a research-grade domain list), checking two things: whether each site has published llms.txt, and how each site's robots.txt treats six real AI crawlers (GPTBot, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Google-Extended).
Full methodology, the raw per-domain data, and the actual script are public and free to verify at https://github.com/freemind2023/vimarsh-llms-txt-census.
Only 1 in 5 sites have even made a decision here
20.8% of the 100,000 sites checked have published an llms.txt file — the newer, AI-facing equivalent of robots.txt. Four out of five businesses haven't published anything telling AI engines who they are or what they do. That's not necessarily a rejection of AI visibility — llms.txt is new enough that silence is still the default, not a decision.
We cross-validated this: a smaller 10,000-domain pilot run (same method, same day) returned 20.1% adoption — within 0.7 points of the full 100k run. That's a real stability signal worth stating, not assuming from one run.
The more interesting finding: AI crawler blocking is wildly inconsistent
Blocking rates by robots.txt ranged from 34.2% (PerplexityBot, least blocked) to 78.3% (Claude-Web, most blocked).
The most non-obvious finding: Anthropic runs three distinct crawler identities — ClaudeBot, Claude-Web, and anthropic-ai — and they get blocked at meaningfully different rates from each other: 60.8%, 78.3%, and 72.8%. Same company, same underlying intent, three very different outcomes. The likely explanation: site owners are copy-pasting "block AI" robots.txt snippets without knowing which bot name maps to which crawler.
What this doesn't prove
This measures declared policy, not observed behavior — a blocked bot might still reach content another way, and robots.txt compliance isn't provable from outside. It also doesn't prove any AI engine actually reads llms.txt when it exists. We're stating these limits plainly because a claim like this is only worth something if you can check our work — and the full dataset is public specifically so you can.
Why this matters for your own site
If you don't know whether your own robots.txt is accidentally blocking the AI crawlers that could be citing you, you're flying blind on exactly the kind of gap that's this common across the open web. Run a free AI visibility check with Vimarsh to see where your own site stands.
Full technical writeup, methodology, and a reproducibility note (including one bug caught and fixed before this run) live on /research — the permanent home for this kind of study going forward.
See where your own site stands — run a free AI visibility check with Vimarsh.
Get your free AI reportRelated reading



