An e-commerce site with several thousand references adds product sheets every week, but Google only crawls a fraction of the new URLs. The problem does not stem from the content or the internal linking. The XML sitemap submitted via Search Console contains redirected pages, noindex URLs, and parameterized duplicates. The engine receives a mixed signal, and it slows down its crawling.
Optimizing a site’s sitemap page is not just about generating a file and forgetting it. It is a management tool that, if poorly maintained, wastes crawl budget instead of focusing it on the pages that matter.
Consistency between XML sitemap, canonical tag, and indexing directive
The first mistake encountered in the field is the inconsistency between what the sitemap declares and what the page says about itself. A URL listed in the sitemap should always return an HTTP 200 code, have its own canonical tag, and not contain a noindex directive.
When a redirected URL (301 or 302) is submitted, the engine must follow the redirection before even deciding whether to index the target. This detour consumes crawl budget without benefit. The same logic applies to deleted pages that return a 404, or URLs with parameters that duplicate content already indexed in another form.
This principle can be illustrated with Bargento’s sitemap page, which structures its URLs to only list accessible pages that are genuinely intended for indexing. This type of rigor prevents sending noise to the bots.
- Each URL in the sitemap must be the canonical version of the page, the one you want to see appear in search results.
- Pages blocked by robots.txt, in noindex, or redirected have no place in the file.
- Parameterized URLs (filters, sorts, sessions) should be excluded unless they target unique and indexable content.
In practice, a clean sitemap contains only indexable URLs with HTTP 200. If your file contains several hundred more than your number of indexed pages in Search Console, it is a warning signal.
Sitemap lastmod tag: useful signal or technical noise
Many CMSs update the lastmod date with each regeneration of the sitemap, even when the page content has not changed. Google has clarified this point: the lastmod value is only relevant if it reflects a real editorial change.
If you republish an article after correcting a typo in the footer, that is not a significant change. However, if you update a guide with new data or restructure a product sheet, the lastmod date should correspond to this update.
Synchronizing lastmod with structured data
Consistency goes beyond the sitemap. The lastmod date must match the update date displayed on the page and the dateModified field in your structured data (schema.org). When these three signals align, the engine has a reliable indicator of freshness. When they diverge, the signal loses all credibility.
On this point, feedback varies: some SEOs observe a faster recrawl after correcting lastmod, while others see no notable difference. What is certain is that a misleading or automatic lastmod fools no one.
Sitemap index and URL segmentation for large sites
Beyond a few hundred pages, a single sitemap file becomes difficult to manage, for you as well as for the engines. Segmenting into several child files, organized under a sitemap index, allows for much more precise diagnostics.
The idea is simple: group URLs by content type. One file for product sheets, one for blog articles, one for category pages. This organization allows for immediate identification in Search Console coverage reports of which segment is problematic.
Transforming the sitemap into a diagnostic tool
On Bing Webmaster Tools, coverage reports allow examination of the URLs known from the sitemap and the reasons for non-indexation. This feature transforms the sitemap into a operational control tool for detecting blocked or unintentionally excluded pages.
Google Search Console offers a similar report. Comparing the number of submitted URLs to the number of indexed URLs by sitemap file reveals discrepancies. A child sitemap file with a low indexing rate signals a structural problem in that segment (duplicate content, insufficient linking, perceived low quality).
Combining XML sitemap and IndexNow to accelerate indexing
The XML sitemap functions as a permanent inventory. Engines consult it periodically, at their own pace. For sites where content changes frequently (editorial, marketplace, catalog), this pace may be insufficient.
IndexNow complements the sitemap by signaling changes in real-time to engines that support the protocol (Bing, Yandex, and others). Instead of waiting for the bot to pass, we send a ping for each URL created, modified, or deleted.
The two tools do not replace each other. The sitemap remains the reference for the overall inventory. IndexNow serves as instant notification. On an e-commerce site that adds dozens of sheets per day, this combination reduces the delay between publication and acknowledgment by the engine.
Concrete implementation
- Generate an IndexNow API key and place it at the root of the site in a text file named after the key.
- Configure the CMS or deployment pipeline to send an IndexNow call for each page creation or modification.
- Keep the XML sitemap updated in parallel, with reliable lastmod dates, so that engines that do not support IndexNow have a freshness signal.
The sitemap and IndexNow cover two different timeframes: the permanent inventory and the one-time notification. Using both increases indexing coverage without solely relying on the natural crawl frequency.
A well-segmented sitemap, purged of unnecessary URLs and synchronized with the technical signals of the page is nothing spectacular. It is a regular maintenance task, not a one-time adjustment. Each submitted URL must earn its place in the file, otherwise, we dilute the bot’s attention on pages that will never serve SEO.
