What Is Crawling? How Search Engines Discover and Fetch Web Content

Before a page can show up in search results, a search engine has to find it, fetch it, and decide it’s worth keeping. The first of those steps is crawling. It’s easy to lump crawling, rendering, and indexing together as one fuzzy process, but Google treats them as distinct stages, and understanding where one ends and the next begins makes it much easier to diagnose why a page isn’t showing up the way you expect.

Crawling vs. rendering vs. indexing

Crawling is the discovery and fetching stage. Automated programs, generically called crawlers (Google’s is Googlebot), find URLs and download the text, images, and other files associated with them. Rendering is the step where Googlebot executes JavaScript using a recent version of Chrome so it can see the page roughly the way a browser would, and Google has folded this into the crawling and processing pipeline rather than treating it as a fully separate phase. Indexing is what happens after that: Google analyzes the content, including elements like title tags and alt attributes, and decides whether the page qualifies to be stored in the index it serves results from. A page has to be crawled before it can be rendered and indexed, but being crawled is no guarantee it will be indexed. Google lays this whole pipeline out in its overview of how Search works.

How Googlebot finds URLs in the first place

Crawlers don’t browse the web randomly. According to Google’s own documentation, new URLs are mostly discovered in one of three ways:

Links from already-crawled pages. If a category page links to a new blog post, Google can follow that link to find the new URL.
Sitemaps. An XML sitemap lists the pages you want crawled and can include metadata like last-modified dates, which helps a crawler prioritize what to revisit. Google’s sitemaps overview is explicit that submitting a sitemap improves discovery on larger or more complex sites, but it doesn’t guarantee that every listed URL will actually be crawled or indexed.
URLs Google already knows about from previous crawls.

For most sites, internal linking and a clean sitemap cover the bulk of discovery. Details on the crawler itself, including its user agent strings and general behavior, are documented on Google’s Googlebot page.

IndexNow: a push-based alternative, with a real caveat

IndexNow is a real protocol that lets a site notify participating search engines the moment a page is published, updated, or removed, instead of waiting to be discovered through links or a sitemap crawl. It’s worth knowing the exact participant list, because it gets misrepresented constantly: Microsoft Bing, Yandex, Naver, Seznam.cz, and Yep support IndexNow. Google does not participate in IndexNow. Google has acknowledged testing the protocol but has not adopted it, and continues to rely on its own crawling infrastructure instead. If your priority is fast Google discovery, IndexNow won’t help; for Bing or Yandex visibility, it’s a genuinely useful tool. Details on the protocol and its current adopters are at IndexNow.org and on Bing’s IndexNow page.

AI crawlers are a separate category, not a search-discovery mechanism

GPTBot, ClaudeBot, and similar bots get folded into “crawling” conversations constantly, but they don’t feed the search discovery process described above. OpenAI runs three distinct crawlers with documented, separate jobs: GPTBot collects pages to train its models, OAI-SearchBot indexes pages for ChatGPT’s search answers, and ChatGPT-User fetches a single page only when a live user triggers it mid-conversation. Anthropic splits the same way: ClaudeBot for training, Claude-SearchBot for search-style retrieval, Claude-User for on-demand fetches. Blocking one of a vendor’s bots in robots.txt has no effect on its siblings, and none of them touch Google or Bing rankings. Google draws an identical line internally: Google-Extended is a separate token that only governs whether crawled content can train Gemini models, and Google states plainly that using it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” If the goal is opting out of AI training while staying fully crawlable for search, each of these tokens has to be addressed by name. A blanket User-agent: * block or a generic Googlebot disallow won’t reach them.

Crawl budget: a real concept, but only relevant past a certain scale

“Crawl budget” gets thrown around as a generic SEO scare term, but Google defines it narrowly, and most sites never need to think about it. Per Google’s own guidance, crawl budget is shaped by two things:

Crawl capacity limit, the maximum number of simultaneous connections Googlebot will use against your site and the delay between fetches. If your server responds quickly and reliably, this limit can rise; if the site slows down or starts returning server errors, Google pulls back and crawls less.
Crawl demand, which is Google’s own appetite to crawl based on a URL’s perceived value, its popularity, and how stale the content appears.

Crucially, Google states this guide is intended for large sites (roughly a million or more unique pages with content that changes moderately often) or medium-to-large sites (tens of thousands of pages or more) with content that changes very rapidly. If your pages tend to get crawled the same day they’re published, crawl budget isn’t something you need to manage. The full breakdown is in Google’s crawl budget management documentation.

What actually affects crawl efficiency

A few concrete, documented factors influence how effectively a site gets crawled:

Server health and response time. Google’s crawl capacity logic is directly tied to how your server performs. If a site responds quickly and consistently, Google can afford to use more simultaneous connections; if response times degrade or the server starts throwing errors, Google’s crawlers scale back to avoid adding load to a struggling server. Google doesn’t publish a table of exact percentage reductions tied to specific millisecond thresholds, so treat any chart claiming “200ms = X% less crawling” as someone’s guess rather than Google’s policy. The honest, sourced version is directional: a healthy, fast, consistently-responding server supports more crawling; a slow or error-prone one gets throttled.

Internal linking. Pages that are easy to reach through a logical link structure get discovered and revisited more reliably than pages buried many clicks deep or only reachable through orphaned URLs.

Duplicate and low-value URL bloat. This is one of the most common, well-documented sources of wasted crawl activity. Google has called out faceted navigation specifically as the single most common cause of what it calls “overcrawling,” where filter and sort combinations generate enormous numbers of near-duplicate URLs (think ?color=red&size=small&sort=price). According to Google’s guidance on managing faceted navigation crawling, this overcrawling both wastes server resources on URLs nobody needs indexed and slows down the discovery of genuinely new content. The same pattern shows up with infinite date-based calendar pages and runaway parameter combinations on internal search or sort features.

Robots.txt directives. Robots.txt tells crawlers what they’re allowed to request, but it has limits worth knowing. Google enforces a hard 500 kibibyte size cap on robots.txt files; anything past that point is simply ignored, so an oversized file can silently drop rules you assumed were active (see Google’s robots.txt specification). Also worth knowing: Google does not support the Crawl-delay directive at all and ignores it outright, something Google confirmed directly on its Search Central blog. robots.txt won’t slow Google down, and neither will Search Console anymore: Google retired the old Crawl Rate Limiter Tool there in January 2024. The current options are narrower. Returning 500, 503, or 429 status codes will get Googlebot to pull back within a day or two, though Google warns against leaving that in place longer since it hurts visibility, or you can report an unusually high crawl rate directly to Google for a more sustained fix. Other crawlers, including Bing’s, do generally respect Crawl-delay.

Redirect chains. Google’s own crawl budget documentation advises avoiding long redirect chains because they have a negative effect on crawling efficiency. Google hasn’t published an exact maximum chain length it will follow, so don’t repeat a specific number as official; the safe, sourced takeaway is simply to keep redirect hops to a minimum and fix chains when you find them.

A word on the unofficial “5 second / 20 second” JavaScript figure

If you’ve read other SEO articles about crawling, you’ve probably seen a claim that Googlebot allows roughly 5 seconds for initial page load and around 20 seconds total for JavaScript execution before giving up. That figure circulates widely in the SEO industry, but it is not a number Google currently publishes. Google’s own documentation on JavaScript SEO basics says only that a page “may stay on this queue for a few seconds, but it can take longer than that,” and stops short of giving a fixed numeric ceiling. Treat any specific timeout number you see elsewhere as an unofficial, unverified industry estimate, not a documented Google spec. What Google does say plainly: server-side rendering or pre-rendering is preferable because it’s faster for both users and crawlers, and not every bot can execute JavaScript at all.

How to actually monitor crawl activity

You don’t need to guess at how a search engine is treating your site. Google Search Console’s Crawl Stats report shows real, account-specific data: total crawl requests over time, broken down by response code, file type, crawl purpose, and Googlebot type, plus average response time and total download size, all sourced from Google’s own logs of your property. That Googlebot-type breakdown is worth checking specifically: Google’s own Googlebot documentation confirms most sites are crawled primarily by Googlebot Smartphone rather than the desktop crawler, a side effect of mobile-first indexing, and the two can’t be targeted separately in robots.txt since they share the same product token. It’s documented in Google’s Crawl Stats report help article. If you want to know what’s actually happening on a specific URL, including indexing status, the Page indexing report (the current name for what longtime SEOs still sometimes call the old Index Coverage report) is the right place to look, and the URL Inspection tool inside Search Console gives a live, single-URL view.

Discovery or monitoring method	What it tells you	Where to check it
Internal links	Whether a page is reachable from crawled content	Site architecture / crawler tools like Screaming Frog
XML sitemap	Which URLs you've explicitly submitted for discovery	Search Console > Sitemaps
IndexNow	Near-real-time discovery, Bing/Yandex/Naver/Seznam/Yep only	indexnow.org, Bing Webmaster Tools
Crawl Stats report	Actual Googlebot request volume, response codes, response time	Search Console > Settings > Crawl Stats
Page indexing report	Which URLs are indexed vs. excluded, and why	Search Console > Indexing > Pages
URL Inspection tool	Live, per-URL crawl and index status	Search Console

For larger sites, comparing server log data (raw Apache or Nginx logs showing actual Googlebot hits) against the Crawl Stats report is the most reliable way to confirm what’s really being requested, since log files capture every request rather than a sampled summary.

The practical takeaway

Crawling problems are almost always one of two things: a server that’s too slow or too unreliable for a crawler to want to spend time on, or a site architecture that buries good content under mountains of duplicate, parameter-driven, or low-value URLs. Fix server response health, keep internal linking logical and shallow for the pages that matter, prune or block the URL patterns that generate near-infinite duplication (faceted navigation being the single most common offender), and use the Crawl Stats and Page indexing reports to verify what’s actually happening rather than relying on generic benchmarks. Crawl budget itself is a real but narrow concern that mostly applies to very large or fast-changing sites; for most smaller sites, getting these fundamentals right matters far more than chasing crawl-budget optimization tactics that don’t apply to them in the first place.