TL;DR. Crawl budget sets a ceiling on how many URLs Googlebot will actually crawl on a site, shaped by server capacity on one side and how badly Google wants to recheck the content on the other. A catalog of just 1,000 products with ten filterable attributes can spin off close to 9.7 million possible URL combinations, though the whole concept rarely matters until a site clears roughly 10,000 pages.
What is crawl budget?
Crawl budget is the number of URLs Googlebot can and is willing to crawl on a given site inside a stretch of time. Nothing about it is a fixed allowance handed out in advance. It's the product of two separate factors, how much crawling the server can absorb and how much Google actually wants to recheck.
Small sites can mostly ignore crawl budget. It becomes a real lever once a site's page count, or the sheer number of URL variations it spins off, outpaces what Googlebot can reasonably visit on any useful schedule.
Key highlights
- A catalog as small as 1,000 products, with ten filterable attributes at five options apiece, can theoretically generate close to 9.7 million distinct URL combinations.
- A Shopify style store carrying the same 1,000 products can produce over 50,000 filtered URLs through faceted navigation alone, the overwhelming majority offering zero unique value.
- Once more than 20 percent of a site's observed crawl activity lands on low value pages, that ratio itself counts as a significant optimization opportunity worth acting on.
- Google's own guidance frames crawl budget as a concern mainly past roughly 10,000 pages, or on any site producing large volumes of filtering and pagination URLs regardless of raw page count.
How crawl budget is determined
Two separate inputs combine to set it: crawl capacity and crawl demand.
- Crawl capacity: how many simultaneous connections Googlebot can open before it risks overloading the server. Fast, stable response times raise the ceiling, while errors and slow responses push it down.
- Crawl demand: how badly Google wants to check the content in the first place, driven by a page's popularity and how often it genuinely changes.
What actually wastes crawl budget
Faceted navigation is the single biggest offender on ecommerce sites. Filters for size, color, price, and material combine into a near infinite set of URLs, the vast majority duplicate or near duplicate content offering nothing unique.
Session IDs and tracking parameters tacked onto URLs, infinite calendar pages, redirect chains, and soft 404s still returning a 200 status all pull crawl activity away from the pages that actually deserve it.
How to check how Google is actually crawling a site
- Review the Crawl Stats report in Search Console, broken out by response code and file type.
- Run log file analysis to see exactly which URLs Googlebot is visiting in reality, not just what it might visit.
- Block low value parameter combinations in robots.txt or through the URL Parameters tool.
- Canonicalize filtered and sorted variants back to the clean category URL.
- Fix redirect chains and server errors, since a slower, error prone server directly shrinks crawl capacity.
Crawl budget and AI crawlers
GPTBot and PerplexityBot run their own crawl budgets, entirely separate from Googlebot's. The same waste, faceted URLs, redirect chains, thin duplicate pages, dilutes their ability to find and cite the best content on a site just as much as it dilutes Google's.
Frequently asked questions
Does my site actually need to think about crawl budget?
Most small sites don't. It becomes a genuine concern once a site clears roughly 10,000 pages, or once faceted navigation, session parameters, or infinite calendar pages start generating URL volumes wildly out of proportion to the actual content, the kind of runaway growth where a 1,000 product catalog can theoretically spin off millions of filtered combinations.
What's the biggest single source of crawl budget waste?
Faceted navigation, by a wide margin, on any ecommerce site. Filters combining across size, color, price, and material generate enormous volumes of near duplicate URLs, and a store with just 1,000 products can produce over 50,000 of them through filtering alone.
How do I find out where crawl budget is actually going right now?
Log file analysis is the most reliable method, since it shows exactly which URLs Googlebot visited rather than which ones it theoretically could. Cross reference that against the Crawl Stats report in Search Console for a fuller picture broken down by response code and file type.
What's the difference between crawl capacity and crawl demand?
Crawl capacity is how many connections Googlebot can open without straining the server, driven by response speed and stability. Crawl demand is how much Google wants to check the content in the first place, driven by popularity and how often it changes. Both have to be healthy for crawl budget to work in a site's favor.