What Is Crawling? How Search Engines Discover and Fetch Web Content
Crawling is the stage in which a search engine finds URLs and downloads what is on them. Automated programs called crawlers (Google’s is Googlebot) discover pages through links, sitemaps, and URLs they already know, then fetch the text, images, and other files those pages contain. A page’s content is indexed only after it is crawled, and being crawled does not guarantee that it will be. The exception runs the other way: a URL blocked from crawling by robots.txt can still be indexed without its content if other pages link to it.
Crawling, Rendering, and Indexing
Google’s overview of how Search works separates the stages. Crawling discovers and downloads pages. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit.” Indexing comes after: Google tries to understand what the page is about and decides whether to store it. The same overview states plainly that Google “doesn’t guarantee that it will crawl, index, or serve your page.”
Keeping the stages apart helps with diagnosis. A page that was never fetched has a crawling problem; a page that was fetched and left out has an indexing question.
How Googlebot Finds URLs
Google calls this step “URL discovery,” and its documentation names three routes:
- “Some pages are known because Google has already visited them.”
- “Other pages are discovered when Google extracts a link from a known page to a new page,” such as a category page linking to a new post.
- Other pages are found when a site submits a list of pages in a sitemap.
Google’s sitemaps overview adds that a sitemap “can improve the crawling of larger or more complex sites,” and that it does not guarantee every item listed in it will be crawled and indexed. Of the three routes, two are in the site owner’s hands: internal links from pages Google already crawls, and an accurate sitemap.
From there, “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Google’s Googlebot documentation explains that most Googlebot requests come from the mobile crawler, and that both crawler types obey the same robots.txt token, so you cannot target Googlebot Smartphone and Googlebot Desktop separately.
IndexNow: Push Instead of Wait
IndexNow is a protocol that lets a site notify participating search engines when a URL is added, updated, or deleted, instead of waiting for them to find the change. Its site lists support from Microsoft Bing, Naver, Seznam.cz, Yandex, and Yep. Google is not on that list, so IndexNow does nothing for Google discovery; for the engines that support it, it can shorten the gap between publishing and crawling.
AI Crawlers Are a Separate Category
Several AI companies run crawlers whose jobs differ from search discovery, and each bot has its own robots.txt token. OpenAI’s documentation of its crawlers separates GPTBot, used for training, from OAI-SearchBot, which “is used to surface websites in search results in ChatGPT’s search features,” and ChatGPT-User, which fetches pages for user actions. A site can allow one crawler and block another, with one limit: OpenAI says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.” Anthropic splits its crawlers the same way: ClaudeBot for training data, Claude-SearchBot for search, and Claude-User for user-initiated requests.
Google keeps a separate token too. Google-Extended is a product token, not a separate crawler with its own user agent. It controls whether content Google crawls may be used for training future Gemini models and for grounding answers in Gemini Apps and Vertex AI, and Google’s list of common crawlers states that Google-Extended affects neither a site’s inclusion in Google Search nor its ranking there. Whether a site’s content can appear in AI Overviews and AI Mode is a different setting, the Search generative AI control in Search Console, where inclusion is the default.
What Shapes How a Site Gets Crawled
Server health. Google’s crawlers adjust to how a server responds. In its 2023 post on the crawl rate limiter, Google wrote: “For example, if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling.” A fast, stable server supports more crawling; a slow or failing one gets less.
Internal linking. Pages linked from pages Google already crawls are found through those links; a page that nothing links to depends on the sitemap or on links from other sites.
URL bloat. Google’s guidance on faceted navigation warns that its “most common implementation, which is based on URL parameters, can generate infinite URL spaces.” The result is overcrawling of filter and sort combinations such as ?color=red&size=small&sort=price, and slower discovery: “if crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs.” Endless calendar pages and internal search results produce the same pattern.
Redirect chains. Google’s crawl budget guide advises avoiding long redirect chains because they hurt crawling. Point redirects straight to the final URL.
robots.txt limits. Google’s robots.txt specification caps the file at 500 KiB and ignores anything beyond that limit. Google also does not support the crawl-delay rule; its 2019 note on unsupported rules names it among them.
Slowing Googlebot Down
Search Console no longer has a crawl rate setting. Google announced in November 2023: “The crawl rate limiter tool in Search Console is being deprecated on Jan 8th, 2024.” It pointed site owners to server responses instead: “the fastest way to reduce crawl rate is to instruct Googlebot through server responses as detailed in our documentation.” Google kept a Googlebot report form “for reporting unusual Googlebot activities and for emergency cases.”
Crawl Budget in One Paragraph
Crawl budget, the amount of crawling Google is able and willing to spend on a site, matters mainly for large or fast-changing sites. Google’s guide is written for sites with “1 million+ unique pages” changing about weekly, or “10,000+ unique pages” changing daily, and says that if “your pages seem to be crawled the same day that they are published, you don’t need to read this guide.”
Monitoring Crawl Activity
| What you want to know | Where to look |
|---|---|
| Whether a page is reachable through links | A site crawler run over your own site |
| Which URLs you submitted for discovery | Search Console Sitemaps report |
| How much Googlebot requested, with response codes and response time | Search Console Crawl Stats report |
| How many URLs are indexed or not, why, with example URLs | Search Console Page indexing report |
| The crawl and index status of one URL | Search Console URL Inspection tool |
| Every request a crawler made | Your server's access logs |
The Crawl Stats report “shows you statistics about Google’s crawling history on your website,” grouped by response, file type, purpose, and Googlebot type, with example URLs for each group. Server logs show every request, including those from crawlers other than Google’s, which makes them the place to confirm what the report summarizes.
Where to Start When Pages Are Not Being Crawled
- Check that the pages are not blocked in robots.txt and return a 200 status.
- Check server response time and errors in the Crawl Stats report.
- Check that the pages are linked from pages Google already crawls and listed in the sitemap.
- Look for parameter, filter, and calendar URLs consuming crawling elsewhere on the site.
- Use URL Inspection on a sample of the affected URLs to see what Google last fetched.