What Is Indexing? How Search Engines Store and Organize Content

Crawling and indexing are not the same thing, and confusing them is the single most common reason site owners panic over “missing” pages. Crawling is Google fetching a URL. Indexing is what happens after: Google’s systems analyze that page’s content, decide whether it’s worth keeping, and if so, store it in a searchable database so it can be retrieved and ranked for relevant queries. A page can be crawled dozens of times and still never make it into the index. Understanding why is most of what technical SEO is about.

What Indexing Actually Means

When Googlebot crawls a URL, it hands the raw HTML, and in many cases a fully rendered version of the page, to Google’s indexing systems. Those systems parse the text, extract structured signals (titles, headings, links, structured data), and decide whether the page adds something the index doesn’t already have. If it passes that bar, the page’s content gets stored and associated with the words, phrases, and concepts someone might search for, so it can be surfaced later as a search result.

Google has never published an exact, current index size. Its own “How Search Works” materials have at various points cited figures in the tens of trillions of known pages, but Google has been explicit that nobody, including Google, knows precisely how many pages exist or are indexed at any given moment. The closest thing to a hard number on record came out of courtroom testimony rather than an official disclosure: during the US v. Google antitrust trial, Google VP of Search Pandu Nayak testified that the company’s index held roughly 400 billion documents as of 2020, and that he wasn’t certain whether that figure had changed materially since (reported coverage of the testimony). Treat that as a rough historical data point, not a current, official statistic. If you see a precise “Google has X billion pages indexed” claim anywhere, including in older versions of this article, it’s an unsourced estimate.

Crawled, But Not Indexed: Reading the Page Indexing Report

Search Console’s Page indexing report (the report that replaced what used to be called the Index Coverage report) is the authoritative place to see how Google classifies every URL it knows about on your site. The status categories matter because they point to different root causes:

Status (Page indexing report)	What it generally means
Indexed	The page is in Google's index and eligible to appear in results
Crawled, currently not indexed	Google fetched the page but decided not to index it, often a quality or duplication judgment
Discovered, currently not indexed	Google knows the URL exists but hasn't crawled it yet, often a crawl-budget or prioritization issue
Excluded by 'noindex' tag	You told Google not to index the page, and it complied
Blocked by robots.txt	Crawling, not indexing, was blocked; Google may still index the URL using other signals if it's linked to elsewhere
Duplicate, Google chose different canonical than user	Google indexed a different URL than the one you marked as canonical
Page with redirect	The URL forwards elsewhere, so the destination is what gets indexed

“Crawled, currently not indexed” and “Discovered, currently not indexed” are the two statuses that cause the most confusion, because both mean “not indexed” without an obvious error to fix. The practical difference is where the page sits in Google’s pipeline: discovered-but-uncrawled pages are still waiting in line, while crawled-but-unindexed pages were already evaluated and didn’t clear the bar. You can read the full current definitions in Google’s Page indexing report documentation.

Why Pages Get Excluded From the Index

A handful of causes account for most exclusions:

Duplicate or near-duplicate content. If multiple URLs serve the same or substantially similar content, Google consolidates them around one canonical version and indexes that one, not all of them.
A noindex directive. Either a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header tells Google explicitly not to index the page. For this to work, the page must still be crawlable; if robots.txt blocks the page first, Google may never see the noindex tag at all. Applying noindex doesn’t stop crawling, either: Google keeps revisiting the page periodically, on a schedule tied to how important the page otherwise looks, partly so it can notice if you ever remove the tag. Google’s own guidance on this sequencing is worth reading directly: Block Search indexing with noindex.
robots.txt blocking. This is a crawling control, not an indexing control. A disallowed URL can still get indexed (typically with no snippet) if Google finds it referenced elsewhere, such as through external links. Google’s documentation is direct about this distinction: see Robots.txt introduction and guide.
Low-value or low-differentiation content. Google has been explicit that there is no minimum word count and no fixed “thinness” threshold. A short page that directly answers a specific query is not penalized for being short, and a long page padded with repetitive, low-value text isn’t protected by its length. The relevant standard is usefulness to the reader, not a word count. Google’s own framing is in Creating helpful, reliable, people-first content.
Rendering and resource-blocking issues. If critical content or links only appear after JavaScript executes, and Google’s rendering step can’t access the scripts, styles, or APIs needed to build the page, the indexed version of the page may be missing content that’s visible to a human visitor.

It’s worth resisting the urge to attach hard percentages to any of this. You’ll find blog posts claiming Google excludes a fixed share of crawled pages for quality reasons, or that JavaScript content has to render within an exact number of seconds to count. Google has not published those figures, and reciting them as fact (rather than as a general tendency) misleads more than it helps.

Canonicalization’s Role in Indexing Decisions

When Google finds multiple URLs it considers duplicates or near-duplicates, it picks one to actually index and serve in results: the canonical. You can express a strong preference using a rel="canonical" link element, a redirect, or consistent internal linking, but Google ultimately makes the final call by weighing several signals together, and it doesn’t always agree with the canonical you specified. That’s what produces the “Duplicate, Google chose different canonical than user” status in the Page indexing report.

Google ranks redirects and rel="canonical" annotations as strong signals, and sitemap inclusion as a comparatively weak one. The full signal hierarchy and implementation guidance are in How to specify a canonical with rel=”canonical” and other methods and the companion page on canonicalization. If you’re consistently seeing Google override your canonical choice, that’s usually a sign your preferred URL is weaker on those signals (fewer internal links pointing to it, inconsistent self-referencing tags, or a thinner content footprint) than the one Google picked instead.

Mobile-First Indexing

Google primarily crawls and indexes the mobile version of a site’s content rather than the desktop version; this is what “mobile-first indexing” means. The practical implication is that if your mobile page hides or omits content that’s present on desktop, or carries a different robots meta tag, that gap is what Google actually uses to build its understanding of the page. Google’s current best-practices documentation covers what to check, including making sure mobile and desktop carry matching robots directives and that primary content isn’t hidden behind interactions Google won’t trigger; see Mobile-first indexing best practices.

JavaScript and the “Two Waves” of Indexing

Sites built on JavaScript frameworks face an extra wrinkle: Google’s crawling and rendering aren’t a single instantaneous step, and the result is often described as two waves of indexing. In the first wave, Googlebot fetches and parses the raw HTML response, extracting whatever links and content are immediately available. In the second wave, content and links that only exist after JavaScript executes are picked up in a later rendering pass, once Google’s rendering service (a headless, Chromium-based system) has resources available to execute the scripts and build the final DOM. That gap between the first wave and the second can range from short to substantial, and it’s part of why JavaScript-heavy pages sometimes show delayed or incomplete indexing compared to static HTML. Google’s own explanation of this process is in Understand JavaScript SEO basics.

How to Check Indexing Status and Request Indexing

For any individual URL, the URL Inspection tool in Search Console is the most direct source of truth: it shows whether the URL is indexed, which canonical Google selected, when it was last crawled, and whether mobile usability or other issues are blocking it. From that same tool you can request that Google recrawl the URL, which adds it to a priority queue rather than guaranteeing immediate indexing. Documentation: URL Inspection tool and Ask Google to recrawl your URLs.

Realistic expectations matter here. Google does not commit to a fixed turnaround time for crawling or indexing a requested URL; it can happen quickly or take considerably longer depending on the site’s overall crawl demand, the page’s perceived value, and Google’s current capacity. Be skeptical of any source that states an exact number of days as a guarantee, that’s not something Google has committed to publicly.

Indexing in the Age of AI Overviews

It’s also worth understanding why indexing still matters even as AI-generated summaries reshape the search results page. AI Overviews and AI Mode are not built from a separate index or a separate crawl; they draw on the same indexed content Google’s standard search systems already use, retrieving and synthesizing from pages that are indexed and otherwise eligible to appear in regular results. A page that isn’t indexed has no path into an AI Overview citation either. Google’s current guidance on this is in AI features and your website.

A Practical Checklist

When a page isn’t showing up where you expect, work through these in order:

Check the Page indexing report and the URL Inspection tool to see the page’s actual current status, not an assumption.
Confirm the page isn’t blocked by robots.txt and doesn’t carry an unintended noindex tag (check both the HTML and HTTP headers).
Confirm Google isn’t choosing a different canonical than the one you intend, and that internal links consistently point to your preferred URL.
Look at whether the page’s content is genuinely differentiated and useful relative to similar pages on your site and across the web, rather than assuming a word-count fix will help.
If the site relies heavily on client-side JavaScript, verify that the content and key links are present in the rendered HTML, not just the pre-render source.
Use the URL Inspection tool’s “Request indexing” option for individual priority URLs, but treat it as a nudge, not a guarantee.

None of this requires guesswork. Search Console and Google’s own developer documentation describe, in detail, exactly how these systems are supposed to behave. When in doubt, check the primary source rather than a secondhand statistic.