LLM Crawlability

TL;DR. LLM crawlability tests whether an AI system's bot, GPTBot, ClaudeBot, PerplexityBot, and the rest, can actually reach a page, fetch it, and read its content for training or live retrieval. Otterly's AI Citations Report 2026 found 73% of sites carry technical barriers blocking AI crawler access, and Vercel and MERJ's research found 69% of AI crawlers cannot execute JavaScript at all, so a page rendered client-side can look completely empty to nearly every major AI bot while looking perfectly normal to a human visitor.

What is LLM crawlability?

LLM crawlability is the technical measure of whether an AI language model's bot can reach a page, download its content, and parse it into usable text, the same fundamental checkpoint that has governed traditional search crawling for decades, applied now to a newer set of user agents. A page can rank well in Google and still be functionally invisible to an AI assistant if the specific bot behind that assistant gets blocked by robots.txt, denied by a CDN rule, or handed a page that needs JavaScript execution the bot simply cannot perform.

Key highlights

• Otterly's AI Citations Report 2026 found 73% of sites carry technical barriers blocking AI crawler access, before content quality even enters the picture.
• Vercel and MERJ's joint research found 69% of AI crawlers cannot execute JavaScript, which makes client-side-rendered content effectively invisible to them.
• Vercel's own study found GPTBot fetches JavaScript files in only about 11.5% of requests and never executes them, while ClaudeBot downloads JavaScript in roughly 23.84% of requests and also never runs it.
• In Q1 2026, GPTBot ranked as the most blocked AI crawler in robots.txt DISALLOW rules of any AI bot, with Common Crawl's CCBot now blocked even more often across the top 1,000 websites, per an analysis of robots.txt across Cloudflare's network.
• Hypotenuse's ecommerce crawlability guide found GPTBot and ClaudeBot typically re-crawl a site within 7 to 14 days of access being restored, with full LLM citation visibility usually catching up over 2 to 6 weeks.

Training crawlers vs retrieval crawlers

Training crawlers and retrieval crawlers serve different jobs entirely, and deserve different robots.txt treatment, per links-stream's breakdown of how LLM crawlers work. Training crawlers, GPTBot and ClaudeBot among them, feed a model's underlying training dataset, crawl in bulk without urgency, and strictly respect robots.txt. Retrieval crawlers, OAI-SearchBot and PerplexityBot among them, fetch pages in real time to answer a live user query, with far less tolerance for a slow page or a long redirect chain, since any delay there stalls the answer a real person is sitting and waiting on.

Why JavaScript breaks LLM crawlability

Most AI bots fetch a page's raw HTML and stop there, without running anything resembling a browser's rendering step. A page built on heavy client-side rendering can look completely blank to a bot that only reads what arrives in that first server response, even though a human visitor with an actual browser sees the fully rendered version a moment later. Server-side rendering, static generation, or pre-rendering keeps the real content inside that first HTML response, which is the only version of the page most LLM crawlers will ever see.

Checking and fixing LLM crawlability

1. Check robots.txt for DISALLOW rules against GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot, the bots most commonly blocked, whether intentionally or by an old default.
2. Fetch key pages with JavaScript disabled, or through a raw HTTP request, to see exactly what an AI crawler receives, since that is the same view most bots actually get.

1. Confirm CDN and firewall rules are not silently denying AI user agents, a common cause of blocking that never shows up anywhere in robots.txt.
2. Switch any client-rendered page meant to be crawled and cited over to server-side rendering or static generation.
3. Monitor server logs directly for GPTBot, ClaudeBot, and PerplexityBot activity, to confirm the bots are actually reaching the pages built for them.

Frequently asked questions

What is LLM crawlability?

LLM crawlability is whether an AI system's bot can successfully access, fetch, and read a page's content well enough to use it for model training or live answer retrieval. A page with poor LLM crawlability can rank normally in Google and still stay entirely invisible to AI assistants.

Can GPTBot actually execute JavaScript?

No. Vercel's research found GPTBot fetches JavaScript files in only about 11.5% of requests and never runs them, and ClaudeBot behaves the same way, downloading JavaScript in roughly 23.84% of requests without ever executing it.

What separates a training crawler from a retrieval crawler?

A training crawler, GPTBot for instance, feeds a model's underlying training dataset and crawls in bulk without urgency. A retrieval crawler, PerplexityBot or OAI-SearchBot for instance, fetches pages in real time to answer a live user query, with far lower tolerance for slow pages or redirect chains.

How many sites accidentally block AI crawlers without realizing it?

Otterly's AI Citations Report 2026 found 73% of sites carry some technical barrier blocking AI crawler access, whether through robots.txt rules, CDN configuration, or unrendered JavaScript, often with the site owner having no idea it is happening.

Once access is restored, how long before an AI crawler notices?

GPTBot and ClaudeBot typically re-crawl a site within 7 to 14 days of access being restored, according to Hypotenuse's crawlability guide, with full LLM citation visibility usually taking 2 to 6 weeks to fully catch up.