Crawl Budget Optimization: Ensuring Google Discovers Your Important Content
Crawl budget is how much of a site Google’s crawlers can and want to crawl. Optimizing it means making sure that, on a site large or fast-changing enough for crawling to be a real limit, Google spends its requests on the pages that matter instead of on duplicates, endless filter combinations, and dead ends. The first step is deciding whether your site is in that group at all; if it is not, the problem behind slow indexing lies elsewhere.
How Google Defines It
Google’s crawl budget guide says: “A site’s crawl budget is determined by two main elements: crawl capacity limit and crawl demand,” and sums them up as “the set of URLs that Google can and wants to crawl.”
| Element | What it is | What moves it |
|---|---|---|
| Crawl capacity limit | How much crawling your server can take, which Google also calls hostload: the number of parallel connections and how long they stay open | Stable or improving response times raise it; slower responses, 5xx errors, or 429 rate-limiting signals lower it |
| Crawl demand | How much Google wants to crawl | For Googlebot, "a site's size, update frequency, page quality, and relevance, compared to other sites"; the factors you can influence are perceived inventory, popularity, and staleness |
Two details in the same guide are easy to miss. “Every site starts with the same default, conservative crawl capacity limit,” which Google raises over time if there is demand and the site stays healthy. And: “While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers,” so heavy demand from one Google crawler, such as one used for ads or shopping, can leave less capacity for others.
Does Your Site Need to Manage It?
Google writes that its guide is “intended primarily for the following types of sites”:
| Site type | Google's description |
|---|---|
| Large sites | 1 million or more unique pages, with content that changes moderately often (once a week) |
| Medium or larger sites | 10,000 or more unique pages, with very rapidly changing content (daily) |
| Sites with an indexing backlog | A large portion of their URLs classified in Search Console as <!–INLINECODE0–> |
Google adds: “The numbers given here are a rough estimate to help you classify your site.” Page count alone does not put a site in the group; a 10,000-page site that changes slowly does not match the description. The Crawl Stats report’s own help page says: “If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail.”
If your site matches none of the rows, keep basic crawl hygiene (a clean robots.txt, a sitemap of indexable URLs, no soft 404s) and look for slow indexing causes in content quality, duplication, or technical blocks instead.
Confirming There Is a Problem
- The Page indexing report. A large and growing share of URLs under
Discovered - currently not indexedis the pattern Google names.Crawled - currently not indexedpoints to a different question: Google fetched the page and decided not to index it. - The Crawl Stats report. It shows crawl requests, average response time, download size, and response codes over time. Rising response times or server errors there explain a falling crawl rate.
- Server logs. They record every URL Googlebot requested and the status returned, which shows when the crawler spends its visits on filter combinations or session URLs instead of content. A log analysis tool such as Screaming Frog’s Log File Analyser parses the standard formats.
Removing Waste
Google’s guide calls perceived inventory “the factor that you can positively control the most”: without guidance, “Google tries to crawl all or most of the URLs that it knows about on your site,” and duplicates or unimportant URLs waste that time. The usual sources:
- Parameters that multiply one page into many URLs: sorting, filtering, session IDs, and tracking tags.
- Faceted navigation, where every combination of filters becomes a crawlable URL.
- Internal search result pages, which can be generated without limit.
- Soft 404s: pages that say “not found” or are empty but return a 200 status. Search Console flags these when “the content suggests an error for Google Search, an empty page or an error message,” as Google’s HTTP status code documentation describes; a real 404 or 410 tells Google the page is gone.
- Redirect chains. The crawl budget guide advises: “Avoid long redirect chains, which have a negative effect on crawling.”
Consolidate duplicate URLs with canonical tags, block crawl traps such as internal search with robots.txt, and keep filter combinations without search value from generating new URLs.
Helping the Right Pages Get Crawled
Sitemaps help discovery but do not control crawling. Google’s sitemap documentation states that “submitting a sitemap is merely a hint: it doesn’t guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site,” and that “Google ignores <priority> and <changefreq> values.” Keep sitemaps to canonical, indexable URLs and keep <lastmod> accurate.
Internal links are the other lever. Important pages should be linked from relevant pages across the site, not only from a sitemap or a footer. The crawl budget guide sets no click-depth rule, so judge depth by whether a page is reachable from the pages that already get crawled regularly. A page with no internal links at all can be found only through external links or the sitemap.
Protecting Crawl Capacity
Server health sets the ceiling. Fast, stable responses let Google raise the limit; slow responses and server errors make it back off. Standard performance work, such as caching and efficient database queries, serves both visitors and crawlers.
Two robots.txt facts matter here. Google’s robots.txt specification lists the fields it supports and states that “other fields such as crawl-delay aren’t supported,” so a crawl-delay line does not slow Googlebot. And a disallowed URL can still be indexed from links elsewhere, because robots.txt controls crawling, not indexing. If you need Google to crawl less for a short time, Google’s guidance on reducing crawl rate is to return 500, 503, or 429 status codes temporarily. The same page limits this to emergencies and warns against doing it “longer than 1-2 days” because “it may have a negative effect on how your site appears in Google products”; a Retry-After header can tell crawlers when to come back.
Two separate checks follow from Google indexing the mobile version of pages. The mobile site needs the same content as the desktop one, and the server needs to answer quickly and reliably for every crawler, since crawl capacity depends on how the server responds.
Large and Fast-Changing Sites
- International sites multiply the URL count with every locale version, and each version is a URL Google may crawl. Hreflang tells Google which version to show to which users; it does not reduce crawling, so the saving comes from not generating locale URLs that have no audience.
- JavaScript-heavy sites depend on a rendering step after crawling; server-side rendering for high-priority templates means their main content no longer waits for client-side JavaScript.
- Calendars, infinite scroll, and other generated URL spaces need explicit limits, or they can produce URLs without end.
After a migration, a large removal, or a new navigation system, expect crawling to change and compare Crawl Stats and logs with the period before.
Questions About Crawl Budget
Can I ask Google for more crawl budget?
No. You can raise the capacity limit by keeping the server fast and stable, and you can make demand count by removing waste and keeping important pages fresh and well linked.
Does blocking URLs in robots.txt save crawl budget?
For URLs Google would otherwise fetch repeatedly with no value, such as internal search results, yes. For pages you want removed from the index, use noindex on a crawlable page instead, because a blocked page’s noindex tag cannot be seen.