What Is a Sitemap? How Search Engines Discover Your Content

A sitemap is a file that lists the URLs on a site and can add details such as when each page last changed and which images, videos, or news articles it contains. Search engines read it to discover pages they might not reach through links. Google’s sitemaps overview states the limit in the same breath as the benefit: “A sitemap helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed.” A sitemap is a list of suggestions, not an instruction.

Who Needs One

Google’s overview names the cases:

You might need a sitemap if	You might not need one if
Your site is large	Your site is small, "about 500 pages or fewer"
Your site is new and has few external links	Every important page is reachable through internal links
Your site has a lot of video or images, or appears in Google News	You have few media files or news pages to show in search

On large sites, Google explains, “it’s more difficult to make sure that every page is linked by at least one other page on the site,” and “it’s more likely Googlebot might not discover some of your new pages.” Sites that publish often fall under the same reasoning: a sitemap lists new URLs before internal links reach them.

How an XML Sitemap Is Built

The XML format is defined by the Sitemap protocol. A minimal file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/foo.html</loc>
    <lastmod>2022-06-04</lastmod>
  </url>
</urlset>

Tag	Required	How Google treats it
<!–INLINECODE0–>	Yes	The root element
<!–INLINECODE1–>	Yes	One entry per page
<!–INLINECODE2–>	Yes	The page's full URL
<!–INLINECODE3–>	No	Used if accurate
<!–INLINECODE4–>	No	Ignored
<!–INLINECODE5–>	No	Ignored

Google’s guide to building a sitemap is direct about the optional tags: Google ignores the priority and changefreq values. It uses lastmod only “if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate,” and the value “should reflect the date and time of the last significant update to the page.” A lastmod that changes on every regeneration without the page changing is not accurate.

Size limits: “All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs.” Location matters too: unless you submit the sitemap through Search Console, “a sitemap affects only descendants of the parent directory.”

Sitemap Index Files

A site with more URLs than one file allows splits them across several sitemaps and lists those in a sitemap index. Google’s documentation on large sitemaps sets the rules:

“A sitemap index file may have up to 50,000 loc tags.”
“Sitemaps that are referenced in the sitemap index file must be in the same directory as the sitemap index file, or lower in the site hierarchy.”
“You can submit up to 500 sitemap index files for each site in your Search Console account.”

A catalog of two million product URLs needs at least 40 sitemap files of 50,000 URLs each, listed in one index file that you submit once.

Image, Video, and News Sitemaps

Image sitemaps add image entries to a page’s url block, up to 1,000 image:image tags per url tag.
Video sitemaps help Google find video, “especially content that was recently added or that we might not otherwise discover with our usual crawling mechanisms.”
News sitemaps are rolling files: “Only include recent URLs for articles that were created in the last two days.” After that, remove the URLs or their news metadata. A news sitemap holds up to 1,000 news:news entries.

Submitting a Sitemap

Google’s build guide lists the ways:

The Sitemaps report in Search Console, which also shows when Googlebot read the file and any processing errors.
The Search Console API, for sites that regenerate sitemaps automatically.
A line in robots.txt, such as Sitemap: https://example.com/my_sitemap.xml. Google says: “We will find it the next time we crawl your robots.txt file.” Other crawlers that read robots.txt find it the same way.

Checking Whether It Works

The Page indexing report in Search Console has a sitemap filter that shows “index results by whether or not they are included in a sitemap.” Filter to one submitted sitemap and look at how many of its URLs are indexed and why the others are not. A sitemap URL that is not indexed is a page to examine, not a sitemap error.

Keeping It Clean

List only URLs you want indexed: canonical pages that return a 200 status. Leave out:

URLs that redirect
URLs that return 404 or other errors
Duplicate variants, including parameter URLs that repeat another page (a parameter URL that is itself the canonical version of distinct content can stay)
Pages marked noindex

Generate the sitemap from the same source as the site’s pages, so that new pages enter it and removed pages leave it without manual edits, and compare it against the Page indexing report after major changes.