How to Optimize Crawl Budget for Large Sites

By Guru Editorial | August 17, 2026

Googlebot does not have infinite time for your site, and on a large catalog that scarcity shows up as new SKUs that sit unindexed for weeks, price updates that never register, and entire sections Google barely visits. Google's own crawling team has said faceted navigation alone accounts for roughly 50% of the crawling problems reported to it, with action parameters like add-to-cart, sort, and print links adding another 25%. That means three out of four crawl budget fires on the open web trace back to URLs nobody meant to create in the first place.

For a site with a few thousand pages, none of this matters much, Google will eventually get to everything. Past roughly a million pages, or any site adding 10,000 or more pages a day, the math changes: crawl capacity and crawl demand become the actual bottleneck on how fast new and updated content reaches the index. Fixing that requires reading your raw server logs, not just staring at Search Console graphs and hoping.

Why Crawl Budget Becomes a Real Constraint at Scale

Crawl budget is Google's own term for the set of URLs Googlebot can and wants to crawl on your site in a given window. It is not a fixed daily allowance, and it is not something you request more of through a form. It is the practical output of two separate systems working together, and on a small site that output is so far above what you need that nobody notices it exists.

Google's official guidance on managing crawl budget for large sites specifically calls out two thresholds: sites with more than a million unique pages that update at least weekly, and sites with more than 10,000 pages that update daily. Below those thresholds, Google's default crawl capacity almost always outpaces your actual page count, so crawl budget optimization delivers little return. Above them, every URL Googlebot wastes on a filter combination or a redirect chain is a URL it did not spend on a page that could rank.

The symptom is rarely a dramatic crawl error. It is a slow accumulation of pages sitting in the "Crawled, currently not indexed" or "Discovered, currently not indexed" buckets in Search Console, new product pages that take three weeks instead of three days to appear, and category pages that show stale inventory counts because Googlebot has not refetched them since last month. None of that trips an alert. It just quietly caps how much of your catalog can compete in search at all, which is exactly the kind of gap a technical SEO audit is built to surface before it costs you a quarter of lost revenue.

Crawl Capacity and Crawl Demand: The Two Levers Google Actually Pulls

Crawl capacity, sometimes called the crawl rate limit, is a ceiling Google sets to avoid overloading your servers. It is expressed as the number of simultaneous parallel connections Googlebot will open and the delay it leaves between fetches. Google's Search Central documentation, rewritten in July 2026, now states plainly that every site starts at the same conservative crawl capacity limit by default, and that limit is shared across all of Google's crawlers, not assigned per bot. If your site responds quickly and consistently, that ceiling rises over time. If it slows down, returns server errors, or sends HTTP 429 rate-limit signals, the ceiling drops, sometimes within hours.

Crawl demand is the other half, and it is about desire rather than capacity. Google weighs how popular your URLs are relative to the rest of the web, how stale your existing pages have become since the last crawl, and how much of your site is what Google calls "perceived inventory," meaning the pile of duplicate or low-value URLs it has to wade through to find anything new. A site can have generous crawl capacity headroom and still get crawled poorly, because Google has decided, based on past visits, that most of what it finds is not worth fetching again.

Crawl budget is not a single number Google assigns you, it is the overlap between what your server can handle and how much Google actually wants to crawl.

This is why speeding up your server alone rarely fixes a crawl budget problem on a large site. You can raise the capacity ceiling all day, but if 70% of what Google finds when it crawls is a duplicate facet combination or a parameter variant of a page it already has, demand stays suppressed and the extra capacity goes unused. The two levers have to move together.

What Is Actually Eating Your Crawl Budget

The categories of waste are consistent across almost every large site, and they show up in a predictable order of severity. Faceted navigation is usually the biggest offender by a wide margin. Botify's analysis of one e-commerce catalog with fewer than 200,000 product pages found more than 500 million bot-accessible URLs once every filter, sort, and pagination combination was counted, all stemming from a navigation system that never constrained which facet combinations could generate a unique, crawlable URL. On a 20-million-page site Botify separately audited, more than 19 million pages, over 80%, carried three or more stacked facets and drove almost no organic visits.

Parameters that trigger an action rather than change content, things like ?add-to-cart=, ?sort=price-asc, or tracking tags such as ?utm_source=, behave the same way from Googlebot's perspective: each unique combination is a new URL to evaluate, even though the content underneath is identical or irrelevant. Redirect chains compound the problem because Googlebot has to follow each hop before it can credit the destination, burning multiple fetches for a single useful page and, if the chain runs long enough, sometimes abandoning it before reaching the final URL. Soft 404s are the quietest of the four: a page that returns HTTP 200 but is functionally empty still gets treated as real content, so Google keeps recrawling it on a normal schedule instead of dropping it the way it would a clean 404 or 410.

None of these problems are visible from the outside. You can browse a category page, see one clean URL, and have no idea that clicking through its filter UI generates thousands of crawlable variants behind it. That gap between what a human sees and what a crawler encounters is exactly why the fix starts in the logs, not in the browser.

Log File Analysis: Finding Out What Googlebot Actually Does

Server logs are the only ground truth for crawl behavior, because every other signal is either sampled, delayed, or aggregated. Search Console's Crawl Stats report is useful for trends, but it rounds and buckets data in ways that hide exactly which URL patterns are consuming requests. Raw access logs record every single hit, with the exact URL, timestamp, response code, response time, and user agent, which is the level of detail you need to actually find where the waste lives.

A Screaming Frog analysis of roughly 7 million real log file events from one mid-sized site illustrates why this matters: pages that had already been redirected kept receiving Googlebot hits for more than nine months after the redirect went live, 404 pages were recrawled without any apparent limit, and the site's robots.txt file alone was fetched more than 24,000 times over that same 10-month window, ranging from 6 to 60 times in a single day. None of that was visible without pulling the raw logs, and none of it would have shown up as an obvious problem in a standard crawl report.

Run a log file analysis project with this sequence:

1. Pull at least 30 days of raw access logs, ideally from the CDN or edge layer as well as the origin server, since some bot traffic never reaches origin.
2. Filter to verified Googlebot and other target crawlers by resolving IP addresses against Google's published ranges, not by trusting the user-agent string alone, since it is trivially spoofed.
3. Group requests by URL pattern, not individual URL, so that thousands of faceted variants collapse into one line item you can actually act on.
4. Cross-reference against your XML sitemaps and top-converting pages to see what share of crawl activity is landing on money pages versus everything else.
5. Check response codes and response times per pattern, since slow or error-prone URL groups are a signal that capacity is being spent inefficiently.
6. Compare crawl frequency to update frequency, flagging pages that change daily but get crawled monthly, and pages that never change but get crawled daily.
7. Export a prioritized fix list ranked by request volume, so engineering effort goes against the patterns eating the most budget first, not the ones that are easiest to talk about.

This is also where crawl stats and log file work overlaps directly with indexing: a page that never gets crawled cannot get indexed, and a page that gets crawled constantly for no reason is a page starving something else of attention. Tools like Screaming Frog's Log File Analyser, JetOctopus, and Botify all handle log ingestion at scale, but the analysis discipline, grouping by pattern and prioritizing by volume, matters more than which tool you use.

Controlling Crawl With Robots.txt, Noindex, and Canonicals

Once you know what is generating the waste, you have three mechanisms available, and they are not interchangeable. Robots.txt is the only one that actually prevents a crawl. Noindex and canonical tags still require Googlebot to fetch the page before either directive can take effect, which means they stop a URL from appearing in search results but do nothing to protect crawl budget on their own.

• Use robots.txt to block URL patterns you never want crawled at all: internal search results, most faceted combinations beyond the first one or two filters, action parameters like add-to-cart, session identifiers, and staging or admin paths that leaked into the crawlable graph.
• Use canonical tags for near-duplicate pages you want consolidated into one indexable version, such as a product page reachable through several category paths, understanding that Google still has to crawl the duplicate to read the canonical signal.
• Use noindex for pages that should exist and be crawlable for users and internal navigation, but that have no business ranking on their own, such as thin tag pages or paginated pages beyond the first.
• Use true 404 or 410 responses for anything permanently gone, and check that those pages are not soft 404s serving a 200 status with an empty-state template.
• Use parameter handling and URL structure fixes at the source, meaning update internal links and navigation code so they stop generating the wasteful pattern in the first place, rather than only patching it after the fact.

The decision usually comes down to whether the goal is protecting crawl budget or protecting link equity. If a URL pattern has real search demand and internal links pointing at it, canonical or noindex preserves that equity while cleaning up the index. If the pattern is pure waste with no search value, robots.txt is the more efficient fix because it stops Googlebot from spending time on it at all. For faceted navigation specifically, most large e-commerce sites end up using a blend: the first layer of high-demand filters (a single color or size facet, for instance) stays crawlable and canonical, while multi-facet combinations get blocked outright.

Sitemaps, Internal Links, and Prioritizing Crawl to Money Pages

XML sitemaps are a direct signal of what you consider important, and Google treats them as a strong hint about crawl priority, not a guarantee. Keep sitemaps limited to canonical, indexable, 200-status URLs, split them logically by section so you can monitor indexation rates per segment, and keep <lastmod> timestamps accurate, since a sitemap full of stale or inflated lastmod dates trains Google to trust the file less over time.

Internal linking does more work here than most teams assume. Crawl demand responds to popularity signals, and internal links are the strongest popularity signal you directly control. A product page linked from your homepage, three category pages, and a handful of related-product modules will get crawled far more often than an identical page sitting four clicks deep with a single inbound link from a paginated archive. This is the mechanism behind why optimizing your internal anchor text and link structure is one of the highest-leverage crawl budget fixes available, since it costs nothing in server capacity and directly reallocates the demand signal toward pages that convert.

Faceted navigation and action parameters together account for roughly three-quarters of the crawling problems Google encounters, which is why they are the first place to look before touching anything else.

Practical prioritization means walking your top revenue-driving templates, product pages, core category pages, and high-intent landing pages, and confirming each one sits within two or three clicks of the homepage, appears in an accurate sitemap, and shows up in your logs at a crawl frequency that roughly matches how often its content actually changes. Anything failing that check is a page competing for attention it is not getting, and it belongs at the top of the fix list ahead of cosmetic cleanup elsewhere on the site.

Monitoring Crawl Health Over Time

Crawl budget work is not a one-time project, it is a maintenance discipline, because new templates, new filters, and new integrations reintroduce waste continuously on a site with a large, actively developed catalog. Search Console's Crawl Stats report, found under Settings, breaks total requests down by response code, file type, crawl purpose (Discover versus Refresh), and Googlebot type, and it is the fastest place to spot a regression: a rising share of 4xx or 5xx responses, a sudden shift toward smartphone-only crawling, or a drop in total requests that suggests Google has lowered your capacity ceiling in response to server strain.

Set a recurring cadence rather than checking reactively. A monthly pull of the Crawl Stats trend alongside a quarterly deep log file analysis catches most regressions before they compound into an indexing problem. Watch specifically for the average response time trendline, since Google's documentation is explicit that slower, less stable response times reduce the capacity ceiling, and for the ratio of Discover to Refresh crawls, since a healthy large site should show a steady stream of Discover activity on new URLs, not just repeated refreshing of pages Google already knows about.

It is also worth extending this same discipline to non-Google crawlers now that AI answer engines route real referral traffic. GPTBot, ClaudeBot, and PerplexityBot all generate their own log entries, and site owners increasingly have to make a deliberate call, by user agent, in robots.txt about which ones to allow for retrieval versus which to block from bulk training crawls. The same log analysis workflow that protects Googlebot's crawl budget will show you whether those AI crawlers can even reach your money pages in the first place, which matters if the goal extends beyond classic rankings into being citable by AI answer engines at all. Connecting a Search Console integration to your monitoring stack keeps this trend data in one place instead of scattered across exports.

Frequently Asked Questions

What counts as a large site for crawl budget purposes?

Google's own guidance points to two thresholds: sites with more than roughly a million unique pages that update at least weekly, or sites with more than 10,000 pages that update daily. Below that scale, Google's default crawl capacity typically exceeds what a site needs, so crawl budget optimization delivers limited return relative to other technical fixes.

Does crawl budget directly affect rankings?

Not directly, but it gates the prerequisite for ranking, which is indexing. A page Googlebot never crawls, or crawls too infrequently to catch an update, cannot be evaluated or ranked on current content, so crawl budget problems show up as indexing delays and stale rankings rather than a ranking penalty itself.

How do I get access to raw server logs?

Most hosting providers, CDNs, and platforms like Cloudflare, Akamai, or AWS expose raw or near-raw access logs through their dashboard or a log export feature. If your stack sits behind a CDN, request logs from both the edge and the origin, since some bot traffic is served from cache and never reaches the origin server logs.

Should I use robots.txt or noindex to control faceted URLs?

Use robots.txt for facet combinations with no search demand that you want Googlebot to skip entirely, since that is the only mechanism that actually prevents the crawl. Use noindex or canonical tags for facet URLs that still carry internal link equity or occasional search value, since those methods require a crawl to take effect but let you control what enters the index.

How often should I check GSC Crawl Stats?

A monthly review catches most regressions, since the report shows rolling trends in response codes, crawl purpose, and average response time. Pair it with a deeper quarterly log file analysis, since Crawl Stats aggregates and rounds data in ways that hide the specific URL patterns actually driving a trend.

Do AI crawlers like GPTBot count against my Google crawl budget?

No, each crawler operates its own separate capacity and demand relationship with your server, so GPTBot, ClaudeBot, and PerplexityBot traffic does not draw down Googlebot's allocation. It can still strain shared server resources, though, which is worth watching in the same log analysis workflow since server slowdowns affect every crawler's capacity ceiling, including Google's.

How long does it take to see results after fixing crawl budget issues?

Server-level signals like response time and error rate can shift Google's crawl capacity ceiling within days to a couple of weeks. Demand-side improvements, like a rising crawl rate on previously neglected pages, typically take four to eight weeks to show clearly, since Google needs to observe the improved signal consistently before adjusting its behavior.

What's the difference between crawl budget and index budget?

Crawl budget governs how many URLs Googlebot fetches, while indexing is a separate downstream decision about which of those fetched pages Google stores and serves in results. A page can be crawled repeatedly and still not indexed if Google judges it low-quality or duplicative, which is why fixing crawl waste has to be paired with genuine content and quality fixes, not treated as a substitute for them.

Sources

• Crawl Budget Management For Large Sites, Google Search Central
• Google: 75% of crawling issues come from two common URL mistakes, Search Engine Land
• How soft 404s and indexing issues caused a 90% traffic collapse, Search Engine Land
• What I Learnt From Analysing 7 Million Log File Events, Screaming Frog
• What Is Faceted Navigation and How Does It Impact SEO, Botify
• New and improved Crawl Stats report, Google Search Central Blog