What Is a Sitemap? How Search Engines Discover Your Content

A sitemap is a file that lists the URLs on your website so that search engines can find them more efficiently. It can also carry extra context, such as when a page was last updated, or details about images, video, and news articles attached to a URL. Google’s own documentation describes it plainly: a sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them, and search engines read that file to crawl your site more efficiently, according to Google Search Central’s sitemap overview.

What a sitemap is not is a guarantee. Listing a URL in a sitemap does not force Google to crawl it, index it, or rank it. Google states this directly: a sitemap helps search engines discover URLs on a site, but it doesn’t guarantee that all the items in the sitemap will be crawled and indexed, per the same overview page. Treat a sitemap as a hint you hand to a crawler, not a lever that pulls pages into the index.

The XML Sitemap Protocol: How It’s Structured

The most common sitemap format is XML, built around a small set of tags defined by the Sitemap Protocol. A minimal sitemap wraps a list of URLs inside a root <urlset> element, with each page described in its own <url> block, according to the Sitemap Protocol specification at sitemaps.org.

Tag	Required?	Purpose
<!–INLINECODE2–>	Yes	Root element that wraps the file and declares the sitemap namespace
<!–INLINECODE3–>	Yes	Container for one page entry
<!–INLINECODE4–>	Yes	The full, absolute URL of the page
<!–INLINECODE5–>	Optional	Date the page content last changed significantly
<!–INLINECODE6–>	Optional	How often the page is expected to change
<!–INLINECODE7–>	Optional	A 0.0 to 1.0 value suggesting relative importance

Two of those optional tags are worth flagging specifically: Google ignores the <priority> and <changefreq> values entirely, a point Google states directly in its own build-and-submit documentation, per Google Search Central’s guide to building a sitemap. Other search engines may handle them differently, but for Google there is no benefit to fine-tuning those numbers. The <lastmod> date is treated more seriously, since it gives Google a real signal about when a page actually changed, but it only helps if it’s accurate. Setting it to today’s date on every crawl regardless of whether anything changed just trains Google to trust your timestamps less.

Every sitemap file is also bound by the protocol’s own size limits: a single sitemap file cannot exceed 50,000 URLs and must stay under 50MB uncompressed, a constraint confirmed both by sitemaps.org and by Google’s own documentation. Gzip compression is allowed for the file you upload, but the limits apply to the uncompressed content.

When Sitemaps Matter Most (and When They Don’t)

A sitemap is not mandatory for every site. Google’s own guidance names three scenarios where sitemaps are especially valuable: large sites, sites that are new and don’t yet have many external links pointing in, and sites with rich media or news content that might otherwise be hard for crawlers to find, according to the sitemap overview documentation. A fourth case worth adding from practical experience, even though it’s not one of Google’s listed criteria, is sites that publish or update content frequently, since a sitemap gives crawlers a faster path to new or changed URLs than waiting on internal links and recrawls alone.

Situation	Sitemap value
Large site (roughly 500+ pages)	High, helps crawlers find pages that internal links might miss
New site with few inbound links	High, gives crawlers an entry point before backlinks accumulate
Site with frequent content changes	High, surfaces new or updated URLs faster
Site with images, video, or news content	High, specialized sitemap formats expose metadata crawlers wouldn't otherwise see
Small site with thorough internal linking	Low, crawlers can typically reach every important page through links alone

In other words, a sitemap does the most work when a site’s own link structure can’t be trusted to surface every important page on its own, whether that’s because the site is large, the content changes quickly, or the site is too new to have earned much of a link graph yet.

Sitemap Index Files for Large Sites

Once a site has more URLs than a single sitemap file can hold, the fix is to split the URLs across multiple sitemap files and then list those files in a sitemap index file. A sitemap index uses its own small set of tags (a root <sitemapindex> element containing one <sitemap> entry per file, each with a <loc> pointing to that file) and a sitemap index file can itself reference up to 50,000 individual sitemaps, per Google’s documentation on managing large sitemaps. Google also notes that sitemaps referenced inside an index file must live in the same directory as the index file or lower in the site’s folder hierarchy, and that you can submit up to 500 sitemap index files per property in Search Console.

For a site with, say, two million product URLs, this means breaking the catalog into dozens of sitemap files of up to 50,000 URLs each, then pointing Search Console at one index file that lists all of them, rather than submitting each file by hand.

Specialized Sitemap Types: Images, Video, and News

Beyond the standard URL list, the sitemap format supports extensions for content that crawlers might otherwise miss entirely:

Image sitemaps add an <image:image> block inside a <url> entry, letting you list up to 1,000 images per page along with their location, which helps Google discover images that are loaded dynamically or aren’t otherwise linked directly, per Google’s image sitemap documentation.
Video sitemaps describe video content hosted on a page, which Google recommends particularly for video that was recently added or that its usual crawling methods might not find on their own, according to Google’s video sitemap guidance.
News sitemaps are built for publishers who want articles considered for Google News. They carry a <news:publication_date> tag for the original publish time, and they run on a strict freshness window: Google’s instructions are to only include URLs for articles published in the last two days, and to remove a URL (or strip its news metadata) once it ages past that window, per Google’s news sitemap documentation. A news sitemap is meant to be a rolling, frequently updated file, not a static archive.

How to Submit a Sitemap

There are a few accepted ways to tell Google a sitemap exists:

Submit the sitemap URL through the Sitemaps report in Search Console, described in Google’s Sitemaps report help page.
Reference the sitemap’s full URL in your robots.txt file with a line such as Sitemap: https://example.com/sitemap.xml. Google states it will find the sitemap the next time it crawls robots.txt, per the build-and-submit documentation.
Submit it programmatically through the Search Console API, which is useful if a site regenerates its sitemap on a schedule and wants submission automated.

Referencing the sitemap in robots.txt is worth doing even if you’ve also submitted it manually in Search Console, since it means any crawler (not just Google’s) that checks robots.txt first will find the sitemap without needing a separate submission step.

Checking Whether a Sitemap Is Actually Working

Search Console’s Page indexing report (this is the current name; it replaced what was previously called the Index Coverage report) is where you check whether the URLs in a submitted sitemap are actually being indexed, as described in Google’s Page indexing report help documentation. The report lets you filter by sitemap, so you can isolate just the URLs from one submitted file and see their status: indexed, or excluded for a specific reason.

A few statuses come up often when reviewing sitemap-submitted URLs against this report:

Indexed, not submitted in sitemap: Google indexed the page through some other discovery path (internal links, external links) even though it wasn’t in the sitemap. This usually just means the sitemap is incomplete, not that anything is broken.
Discovered, currently not indexed: Google knows the URL exists but hasn’t crawled it yet.
Crawled, currently not indexed: Google crawled the page but chose not to index it, often a quality or duplication signal rather than a technical error.

To illustrate with a hypothetical, not a reported statistic: a site’s blog posts often index at a higher rate than its filtered or parameter-based product listing pages, simply because the listing pages tend to be thinner and more repetitive in content. The real numbers on any given property depend entirely on that site’s content and crawl history.

Keeping a Sitemap Clean

A sitemap full of broken links, redirects, or pages you’ve told Google not to index sends a mixed message and can waste the crawl attention you’re trying to direct. The practical rule is to include only canonical URLs that return a 200 status and that you actually want indexed, leaving out:

URLs that redirect elsewhere
URLs that return 404 or other error statuses
Non-canonical duplicate URLs (parameter variants, session IDs, and similar)
Pages marked noindex

Periodically auditing a sitemap against your site’s actual crawl and indexing data (rather than generating it once and forgetting about it) is the difference between a sitemap that functions as a clean, trustworthy map of your site and one that just adds noise for crawlers to sort through.

The Bottom Line

A sitemap is a discovery aid, not an indexing guarantee and not a ranking factor. It earns its keep most clearly on large sites, new sites, sites that publish frequently, and sites with image, video, or news content that crawlers might otherwise miss. Built correctly, kept within the protocol’s size limits, restricted to clean canonical URLs, and checked regularly against the Page indexing report, a sitemap does exactly what it’s meant to do: make sure search engines know what’s there, so the rest of your SEO work has a fair chance to be seen.