XML sitemaps are a machine-readable list of the pages you want search engines to crawl, helping new or updated URLs get discovered when internal links or site size make them easy to miss. The quickest win is making the file editorially clean: include only canonical, indexable URLs that return a 200 status, and keep redirects, parameter duplicates, and noindex pages out. For larger sites, split by section or freshness, maintain an accurate lastmod value, and use a sitemap index so crawlers can fetch changes without re-downloading everything. After submitting in Google Search Console and referencing it in robots.txt, watch for a recurring mistake that quietly stalls coverage: a sitemap that contradicts your canonical tags.
XML sitemap basics and what it can improve for indexing
XML sitemap vs HTML sitemap
An XML sitemap is built for crawlers. It lists URLs you want search engines to discover and recrawl, plus a few optional hints (like lastmod). It is not a ranking boost by itself. It is a discovery and maintenance tool. Google’s own documentation frames it as a way to help Google learn about pages, including URLs that might be hard to discover through normal crawling, such as very new pages or pages that are not well linked internally, which is why it is so useful on large or frequently updated sites. You can validate your approach against Google’s guidance in the Build and submit a sitemap documentation.
An HTML sitemap is built for people. It is a navigational page (often linked in the footer) that helps users and crawlers understand site structure. Unlike XML sitemaps, an HTML sitemap can pass internal link signals naturally because it is part of your visible site architecture. In practice, XML sitemaps and strong internal linking work best together. The sitemap helps with discovery and coverage checks. Internal links help search engines understand importance and relationships.
When sitemaps help and when they do not
Sitemaps help most when crawling and discovery are the bottleneck. Common cases include large ecommerce catalogs, sites with faceted navigation, new content hubs, programmatic landing pages, and sites that publish frequently. They also help you spot issues fast, because you can compare “submitted” vs “indexed” in webmaster tools and isolate patterns (like templates that are accidentally noindexed). Bing explicitly positions sitemaps as a way to tell crawlers about URLs that might otherwise be hard to discover in its Sitemaps guidance.
Sitemaps do not help when indexing is blocked by quality or technical rules. If a URL is redirected, canonicalized to a different page, blocked by robots.txt, returns a non-200 status, or is marked noindex, putting it in a sitemap will not fix the underlying problem. And even for clean URLs, a sitemap is only an invitation to crawl. Search engines still decide what to crawl, when to crawl it, and whether to index it.
Sitemap formats that matter: sitemap index, image, video, news
When to use a sitemap index file
Use a sitemap index when you need multiple sitemap files, either because you are hitting protocol limits or because you want cleaner organization. Per the Sitemaps protocol, each sitemap is capped at 50,000 URLs and 50 MB uncompressed, and a sitemap index can list up to 50,000 sitemap files.
In practice, the index is the URL you submit. Search engines fetch the index, then discover the child sitemaps. This setup is also more resilient for large sites because you can update one child sitemap (like “recent products”) without regenerating a massive single file.
Separate sitemaps by content type
Beyond a standard URL sitemap, Google supports sitemap extensions for images, video, and news. You can either keep separate files by type or mix extensions in one sitemap, as long as the XML stays valid.
Separating by content type helps you troubleshoot coverage and performance. If image discovery is weak, an image-focused sitemap makes it obvious whether Google is seeing the image URLs. It also helps in an AI-heavy search landscape, where results are increasingly visual and media-rich. Clear image and video signals make it easier for search engines to connect the right assets to the right pages.
A few practical reminders:
- Image sitemaps are useful for images that may not be easily discovered (for example, loaded via JavaScript), and Google has deprecated several older image tags, so keep implementations current.
- Video sitemaps work best when the video is a main part of the page and all referenced files are accessible to Googlebot.
- News sitemaps have extra rules, including freshness limits.
International and hreflang considerations
For multilingual or multi-region sites, you can include hreflang annotations in your XML sitemap using xhtml:link. Each localized URL needs to list every alternate version, including itself, and your hreflang set should match your canonical strategy (most sites canonicalize each language version to itself).
If you publish news, follow Google’s news sitemaps requirements closely. Most notably, only include articles from the last two days, and remove older URLs (or just the news metadata) as content ages.
Creating an XML sitemap that stays current
Auto-generation vs manual creation
For most sites, auto-generated XML sitemaps are the right default. They stay in sync with your CMS, publishing workflow, and product inventory. That matters because a sitemap that drifts out of date quickly turns into noise, and search engines learn to ignore noisy signals.
Manual sitemaps still make sense in a few cases: a small brochure site with a stable set of pages, a temporary microsite, or a controlled “priority list” sitemap for a migration where you only want to surface a curated set of canonical URLs. Even then, treat manual files as code. Version them, document rules, and schedule reviews so they do not go stale.
A practical 2026 mindset: if your content changes often (pricing, availability, AI-generated refreshes, or frequent editorial updates), automation is usually the only way to keep lastmod honest and your URL set clean.
lastmod rules that search engines trust
Google can use lastmod for crawl scheduling, but only when it is consistently accurate and reflects a significant change. A meaningful update can include changes to main content, structured data, or important links. A tiny footer tweak or rolling the copyright year is not a real update and should not trigger lastmod. Google also notes that it ignores priority and changefreq, so focus your effort where it counts: canonical URLs and trustworthy modification dates. This is spelled out in Google’s Build and submit a sitemap documentation.
URL limits, encoding, and file delivery requirements
Sitemaps have hard limits: 50,000 URLs per sitemap and 50 MB uncompressed. If you exceed that, split into multiple files and use a sitemap index. The sitemap must be UTF-8 encoded, use fully-qualified absolute URLs, and properly escape XML entities. Hosting at the site root is a common best practice because it keeps scope simple, especially if you also declare the sitemap in robots.txt.
URL selection rules: what belongs in a sitemap
Include only canonical, indexable 200 URLs
A sitemap should represent your preferred, indexable URL set, not every URL your server can generate. The clean rule is simple: only include pages that are meant to rank and that return a 200 OK response for Googlebot.
That means every URL you list should:
- Be the canonical version of the page (the one you want indexed).
- Be indexable (no
noindex, not blocked). - Load reliably with a 200 status on both desktop and mobile user agents.
- Offer unique, useful main content (especially important in 2026 as search engines get better at filtering duplicated or low-value pages created at scale).
If a URL is not your canonical choice, do not “test” it by putting it in the sitemap. Make the canonical decision first, then reflect it consistently across internal links, canonicals, and the sitemap. Google’s guidance on canonicalization is the standard reference for aligning these signals.
Exclude redirects, blocked, and parameter URLs
Do not include URLs that redirect (3xx), error (4xx/5xx), or are blocked by robots.txt or noindex. Submitting them wastes crawl attention and creates misleading coverage signals in Search Console.
Be especially strict with parameter URLs. Most parameter variants should be excluded, including tracking parameters (UTM), session IDs, sort and filter parameters, internal search URLs, and infinite faceted combinations. If a parameterized page is truly meant to be indexed (rare, but sometimes valid for core category filters), it should have a stable, canonical URL strategy and consistent internal linking before it earns a place in the sitemap.
Common non-indexable signals to filter out
Filter out URLs with these signals before they reach your sitemap:
- Non-200 status codes (3xx, 4xx, 5xx).
noindex(meta robots or X-Robots-Tag).- Disallowed by robots.txt.
- Canonical tag pointing to a different URL.
- Soft 404 or “thin content” templates (including low-quality at-scale AI pages).
- Duplicate URLs caused by parameters, mixed case, trailing slash variants, or HTTP vs HTTPS.
Auditing a sitemap against crawl and indexability signals
Compare crawled URLs vs sitemap URLs
A sitemap audit starts with a simple comparison: what your site exposes to crawlers versus what your sitemap claims is important. Export your sitemap URL list and compare it to a recent site crawl (from Screpy or any crawler) and, if you have it, server log samples.
Look for two problem buckets:
- In sitemap, not crawled (or rarely crawled): often caused by weak internal linking, accidental blocking, slow pages, or URLs that are technically valid but not compelling enough to earn crawl attention.
- Crawled, not in sitemap: sometimes fine (filters, internal search, tracking parameters), but it can also reveal “hidden” sections that should be indexed and maintained.
This comparison is also useful for AI-driven discovery. If key pages are only findable through on-page JavaScript paths or non-standard navigation, you can end up with inconsistent rendering and weaker coverage.
Find orphan pages and missing important URLs
Orphan pages are URLs that exist (sometimes even in the sitemap) but have little or no internal linking. They can be indexed, but they tend to perform poorly because search engines have fewer context signals.
During your audit, confirm that your most important pages are:
- Included in the sitemap.
- Linked from relevant hubs (categories, guides, breadcrumbs).
- Supported by internal anchor text that matches real user intent.
If you discover “important but missing” URLs, add them to the sitemap only after you have confirmed they are canonical, indexable, and genuinely valuable.
Catch canonical mismatches and soft 404s
A clean sitemap can still fail if it conflicts with page-level signals. The most common mismatch is a sitemap URL that returns 200 but canonicalizes to a different URL. That usually indicates duplication, parameter issues, or inconsistent trailing-slash and HTTP/HTTPS rules. Fix the canonical decision first, then align internal links and the sitemap to that choice.
Soft 404s are another frequent culprit, especially on templated pages and at-scale content. Google flags a soft 404 when a page returns 200 but behaves like “not found” or has effectively no main content. Google’s explanation and fixes are worth reviewing in its troubleshoot crawling errors documentation.
Submitting sitemaps to Google and verifying they were processed
Google Search Console sitemap submission
In Google Search Console, submit your sitemap (or sitemap index) inside the property that represents the site version you want monitored. Open the Sitemaps report, enter the sitemap URL (usually something like /sitemap.xml or /sitemap_index.xml), and submit it. Search Console will then show whether Google could fetch and parse it, plus any detected errors.
Verification is not just “submitted successfully.” In the Sitemaps report, confirm:
- Status is successful (or at least free of blocking errors).
- Last read updates over time, showing Google is re-fetching it.
- The discovered URLs count looks realistic for that sitemap’s scope.
- Errors (fetch issues, invalid URLs, unsupported formats) are addressed quickly, because persistent errors reduce trust in the file.
robots.txt sitemap declaration
Also declare your sitemap location in robots.txt using the standard Sitemap: field (you can list more than one). This is a low-effort safety net: even if a sitemap is not submitted in Search Console yet, major crawlers can still find it.
Multi-domain and subdomain submission basics
For properties, Google distinguishes between Domain properties (cover all subdomains and protocols) and URL-prefix properties (only a specific protocol and prefix). Domain properties are often the cleanest choice when you want unified reporting across http/https and www/non-www, but they require DNS verification.
If you manage multiple domains (for example, separate country sites on different TLDs), create and verify a separate Search Console property for each domain, then submit the appropriate sitemap(s) under each one so coverage and indexing signals stay correctly segmented.
Monitoring indexing outcomes and fixing low indexed coverage
Submitted vs indexed trends in Search Console
In Search Console, watch whether submitted URLs are turning into indexed URLs over time. Use the Page indexing report and focus on the “All submitted pages” view to isolate sitemap-submitted URLs from everything Google has discovered.
Healthy patterns look like this: submitted stays stable, indexed grows steadily, and excluded reasons are explainable (intentional duplicates, alternate canonicals, or retired pages). Unhealthy patterns are usually obvious too: submitted climbs but indexed stays flat, or indexed drops after a release. When that happens, do not guess. Segment by template or directory, then identify which page type is creating most “not indexed” reasons.
Common sitemap errors and how to resolve them
Most sitemap problems are mechanical and fixable:
- Couldn’t fetch / couldn’t read: the sitemap URL returns 403/404/5xx, is blocked, times out, or sits behind a WAF rule that treats Googlebot differently. Ensure the sitemap URL is the final HTTPS, non-redirecting version and returns 200.
- Invalid format: broken XML, wrong encoding, invalid namespaces, or unescaped characters. Validate the XML before shipping.
- URLs not accepted: sitemap lists non-canonical, redirected, noindexed, or blocked URLs. Align the sitemap with your canonical and indexability rules.
- Overstuffed sitemaps: too many low-value URLs. This is common with large faceted sites and at-scale AI content. Split sitemaps and keep the submitted set intentionally “rank-worthy.”
What to do when sitemap URLs are not indexed
When sitemap URLs are not indexed, work in this order:
- Confirm the URL is truly indexable: 200 status, not blocked, no
noindex, correct canonical, and not a soft 404. - Improve crawl paths: add internal links from relevant hubs, fix orphan pages, and avoid relying on JavaScript-only discovery.
- Check duplication and consolidation: if Google is choosing a different canonical, your sitemap is out of sync with reality.
- Address quality at the source: thin pages, near-duplicates, and mass-produced AI pages often get crawled but not indexed. Prune, merge, or upgrade them before pushing more URLs into the sitemap.
- Use URL Inspection selectively: request indexing for a small set of high-priority pages, and use the results to guide broader fixes.