What Is Indexing? How Search Engines Store and Organize Content

Indexing is the stage in which a search engine analyzes a page it has crawled and decides whether to store it in its index, the database it searches when someone runs a query. Google’s overview of how Search works describes it as the step in which Google analyzes a page’s text, images, and video and stores what it learns in the Google index, a large database. Crawling fetches a page; indexing decides whether that page is kept and how it is organized. The two are separate steps, and the same overview states: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.”

What Happens During Indexing

Google reads the page’s content and markup, such as its title element, headings, alt attributes, links, and structured data, and works out what the page is about. For pages built with JavaScript, it uses the rendered version, so content that appears only after scripts run is available to indexing once rendering succeeds.

A large part of indexing is sorting out duplicates. Google explains: “During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical.” To choose, “we first group together (also known as clustering) the pages that we found on the internet that have similar content, and then we select the one that’s most representative of the group.” The canonical is the version that may appear in results; the others in the group may be shown in other contexts. Google also “collects signals about the canonical page and its contents,” which are used when serving results.

Why Pages Are Left Out

Google’s overview names three common indexing issues:

“The quality of the content on page is low”
“Robots meta rules disallow indexing”
“The design of the website might make indexing difficult”

In practice those cover the main causes:

Duplicates. When several URLs carry the same or very similar content, Google indexes the canonical and treats the rest as duplicates.
A noindex rule, in a robots meta tag or an X-Robots-Tag header. Google’s guidance on blocking indexing adds a condition: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.”
Content that does not add enough. Google publishes no word count for indexing; a short page that answers its question is not excluded for being short, and a long page is not protected by its length.
Content Google cannot see. If key text or links appear only after JavaScript runs and the scripts or resources they need are blocked or fail, the indexed version lacks that content.

Crawling Controls Are Not Indexing Controls

robots.txt decides what Google may fetch, not what it may index. The blocking guidance spells out the consequence: “If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.” To keep a page out of the index, leave it crawlable and use noindex. Google adds: “Specifying the noindex rule in the robots.txt file is not supported by Google.”

The Mobile Version Is the Indexed Version

Google’s mobile-first indexing guidance states: “Google uses the mobile version of a site’s content, crawled with the smartphone agent, for indexing and ranking.” Content missing from the mobile version is missing from what Google indexes. The same guidance says: “Use the same robots meta tags on the mobile site and the desktop site.” A noindex on the mobile version applies to the page Google indexes.

Checking and Requesting Indexing

The URL Inspection tool shows whether a URL is indexed, which canonical Google selected, and when it was last crawled, and it lets you request indexing for that URL. The Page indexing report shows the picture across the site: counts of indexed and not-indexed URLs, the reason for each group, and example URLs limited to 1,000 per list.

A request is not a promise. Google’s guide to asking for a recrawl says crawling can take from a few days to a few weeks, and that a request does not guarantee the page will appear in results quickly, or at all.

Indexing and AI Features

Google’s documentation on AI features sets the same baseline for AI Overviews and AI Mode: a page can be a supporting link there only if it is indexed and eligible to appear in Google Search with a snippet. A page that is not indexed cannot be a supporting link there either.

When a Page Is Not Indexed

Check its status in URL Inspection instead of assuming.
Confirm it is not blocked by robots.txt and carries no unintended noindex, in both the HTML and the HTTP headers.
Confirm which canonical Google chose, and that internal links point to the URL you prefer.
Compare the page with similar pages on the site and elsewhere: does it add something they do not?
On JavaScript sites, confirm that the content and links appear in the rendered HTML.
Request indexing for priority URLs, and allow time.