Sitemap audit and fix

In short. A sitemap tells search engines which pages to crawl and index. When it drifts — pages removed from the nav but still listed, new pages missing, staging URLs that leaked into production — search engines receive conflicting signals: wasted crawl budget, duplicate-content flags, and rankings that do not match the pages you want ranked. This recipe runs the structured fix: find what should not be there, find what is missing, resolve canonical conflicts, and resubmit a clean sitemap.

On this page: When to use this · Before you start · Steps · Issue reference table · Submission checklist · FAQ

When to use this

Before you start

Three data sources are needed before the audit begins:

1. Current sitemap snapshot. Navigate to your-site.com/sitemap.xml in a browser and save the file (File → Save). This is the pre-audit baseline — keep it throughout so you can compare before and after.
2. Search Console coverage export. In Google Search Console → Coverage, export the full indexed URL list. This is what Google currently has — it may differ from your sitemap in both directions.
3. Intended index list. Write down or export the pages that should be indexed. This is not "every page in the admin" — utility pages (login, admin-redirect, staging preview, draft pages) typically should not be indexed. Having this list before the audit gives you something concrete to compare against.

Confirm sitemap submission. In Search Console → Sitemaps, confirm your sitemap URL is submitted and shows a recent discovery date. If it is not yet submitted, submit it at the end of this recipe — after the sitemap is clean, not before.

Where to go

Open SG-Admin → SEO → Sitemap. The panel shows the current generated sitemap URL list, the last generation timestamp, and controls for noindex settings and sitemap submission. The Sitemap health tab auto-flags common drift categories.

Steps — Audit and fix sitemap drift

Steps

1. Run the built-in sitemap health check

In SG-Admin → SEO → Sitemap → Sitemap health, click Run audit. SGEN crawls every URL in the current sitemap and reports:

• 404 URLs — in the sitemap but returning not-found.
• Noindexed URLs — in the sitemap but carrying a noindex meta tag (a contradiction: submitting a URL to be indexed while telling crawlers not to index it).
• Canonical mismatches — in the sitemap but carrying a canonical tag that points to a different URL.
• Redirected URLs — in the sitemap but responding with a redirect (the sitemap should point to the final destination, not an intermediate redirect).
• Duplicate slugs — two sitemap entries resolving to the same content.

Export the health report. This becomes your working list for the rest of the recipe.

2. Resolve 404 URLs

For each 404 URL in the sitemap, choose one:

• Option A: The page was intentionally removed. Remove the URL from the sitemap (set the page to noindex or delete it from the admin) and create a redirect from the old URL to the most relevant live page. Use the bulk redirect creation recipe if you have more than ten to handle.
• Option B: The page should exist but is missing. The page was accidentally deleted or the import skipped it. Restore or recreate the page, then regenerate the sitemap.

Do not leave 404 URLs in the sitemap. Crawl budget spent on 404s is wasted — and if those URLs still appear in the index, visitors clicking from search results hit an error page.

3. Resolve noindex conflicts

A URL that is both in the sitemap and tagged noindex sends contradictory signals. Resolve each one:

• This page should be indexed. Remove the noindex tag in the page's SEO settings, then regenerate the sitemap.
• This page should not be indexed. Confirm noindex is on in the page's SEO settings. SGEN will exclude noindexed pages from the generated sitemap on the next regeneration — confirm the URL drops from the list.

Common source: pages set to noindex during a staging phase and never cleared at launch, or pages on a noindex template that were later promoted to a different template but still carry the old meta tag.

4. Fix canonical mismatches

A canonical mismatch means the sitemap lists URL A, but URL A says its canonical is URL B. Search engines follow the canonical, not the sitemap entry — so the sitemap entry for A contributes nothing except crawl confusion.

Fix: exclude the non-canonical URL from the sitemap (set noindex on URL A, or use the sitemap exclusion list in SEO settings). Confirm URL B is in the sitemap and carries a self-referencing canonical.

For paginated content (page 2, page 3 of a blog archive), it is normal for the canonical on later pages to point to page 1. Exclude pages 2 and beyond from the sitemap and rely on pagination links for discovery.

5. Update redirected URLs in the sitemap

For each URL in the sitemap that returns a redirect, find the final destination URL and update the sitemap to point to it directly. In SGEN, this means updating the page's canonical URL field to match the final destination, or deleting the old page entry and confirming the destination page is in the sitemap.

Sitemaps with redirect-chain entries are not technically broken — search engines follow the redirect — but it signals that the sitemap is not being maintained. Clean sitemaps with direct final-destination URLs are best practice.

6. Regenerate and resubmit the sitemap

After resolving all categories in the health report, click Regenerate sitemap in the sitemap panel. SGEN rebuilds the sitemap from current page state, applying all noindex, canonical, and exclusion rules you have set.

Download the regenerated sitemap and compare it against your pre-audit snapshot. The URL count should be lower (drift removed) and every remaining URL should be intentional.

Submit the regenerated sitemap to Search Console: in the Sitemap panel's Submission tab, click Submit to Google. SGEN calls the Search Console API with your sitemap URL. For Bing, use the Sitemap Submission tab — SGEN supports both.

What success looks like

• Sitemap health audit returns zero issues across all five categories.
• URL count in the regenerated sitemap matches your intended index list.
• Search Console shows the sitemap was discovered with no errors after submission.
• Within 48-72 hours, Search Console coverage no longer shows "submitted URL not found" warnings for the paths you resolved.
• Your team has the pre-audit and post-audit sitemap snapshots saved side by side.

What to do if it does not work

Sitemap health audit does not run or times out. For large sites (1,000+ URLs), the audit may time out. Use the paginated audit option — auditing one section at a time via the category filter — rather than running the full audit at once.

Pages set to noindex still appear in the sitemap. Confirm you clicked Regenerate sitemap after setting noindex. If the URL still appears after regeneration, check whether the page is assigned to a template that overrides noindex settings — template-level SEO settings can conflict with page-level settings.

Search Console still shows old sitemap errors after resubmission. Search Console processes submitted sitemaps asynchronously. Allow 24-48 hours. If errors persist after 72 hours, open the sitemap in Search Console's Sitemaps tool and confirm it shows the new URL count. If it shows the old count, resubmit manually from within Search Console.

Canonical mismatches keep reappearing after fixing. A template or integration may be overriding the canonical tag. Check the page's rendered HTML source and confirm the canonical tag in the <head> matches what you set in the admin. If it does not, a template or integration is overwriting it — investigate which one.

URL count after audit is lower than expected. Some pages you expected to be indexed are missing from the sitemap. Most common causes: page set to noindex, page has a password or access restriction, or the page template has sitemap exclusion on. Check each missing URL individually in the page admin.

Reference — Sitemap issue categories and resolutions

Reference — Ranking impact by issue type

Reference — Sitemap submission checklist

Run through this checklist before submitting to any search engine:

• [ ] Sitemap health audit shows zero issues across all five categories.
• [ ] Every URL in the sitemap returns 200 on the live site.
• [ ] Every URL carries a self-referencing canonical or a canonical pointing to a URL that is also in the sitemap.
• [ ] No staging, development, or non-production URLs appear in the sitemap.
• [ ] Sitemap URL count matches the intended index list (within a small margin for auto-generated archive pages).
• [ ] Last-modified dates are accurate — not all set to the same date, which signals the dates are being generated incorrectly.
• [ ] Sitemap is accessible at /sitemap.xml on the live domain without a login prompt.

Quarterly sitemap maintenance

Sitemaps drift silently. Set a recurring calendar reminder — 30 minutes per quarter is enough for a site under 500 pages.

Between audits, watch Search Console's Coverage report. A sudden spike in "submitted URL not found" errors is a signal to run an ad-hoc audit rather than wait.

Reference — Search Console vs SGEN sitemap

The SGEN sitemap panel and Search Console show two different views of the same truth. They will never be perfectly in sync — expect 24-72 hours for most URLs, and up to 2-4 weeks for low-authority or infrequently crawled pages.

URL in SGEN sitemap but not in Search Console: Google has not yet crawled it (wait, or use URL Inspection to request), the page failed a quality check (thin content, duplicate content, or a previous noindex that Google found before you removed it), or the page has low internal link authority.

URL in Search Console but not in SGEN sitemap: The page is indexed but not submitted. Acceptable for legitimate pages not in the sitemap; a problem if it is a staging URL, draft, or removed page that should not be indexed.

Checklist — sitemap audit complete

• [ ] Sitemap health audit run — zero issues across all five categories.
• [ ] 404 URLs resolved (pages restored or redirects created).
• [ ] Noindex conflicts resolved (removed from sitemap or noindex cleared).
• [ ] Canonical mismatches resolved (sitemap points to canonical URL).
• [ ] Redirected URLs updated to final destination.
• [ ] Sitemap regenerated after all fixes applied.
• [ ] Pre-audit and post-audit snapshots saved side by side.
• [ ] Sitemap submitted to Google and Bing.
• [ ] Next quarterly audit scheduled in calendar.

Frequently asked questions

How often does SGEN regenerate the sitemap automatically? SGEN regenerates the sitemap whenever a page is published, a page's SEO settings change, or you trigger manual regeneration from the SEO panel. Generation does not run on a timed schedule — it runs on content events. After a bulk import or bulk status change, trigger a manual regeneration to catch all changes in one pass.

Does the sitemap include draft pages? No. Draft and scheduled pages are excluded from the generated sitemap. Only published pages with public visibility and without a noindex tag are included.

How many URLs can a single sitemap file contain? The standard sitemap protocol allows up to 50,000 URLs per file and 50 MB uncompressed. SGEN automatically generates a sitemap index file and splits into multiple sitemap files if your site exceeds this. The sitemap index URL is still /sitemap.xml; it references the child files.

Can I exclude an entire page type from the sitemap? Yes. In SEO settings, you can exclude specific templates or post types from sitemap inclusion. For example, if your search-results page or user-profile pages should not be indexed, exclude the template rather than setting noindex on each page individually.

Should I include tag pages and category archive pages? Usually not. Tag and category archive pages are often thin or duplicate content. Set them to noindex and exclude from the sitemap unless they have substantial unique content or strong external backlinks.

Why does my sitemap show a last-modified date of today for all pages? This usually means sitemap generation is using the generation date rather than the per-page last-modified date. Check SEO settings → Sitemap → Last modified source — it should be set to "page last-modified date," not "sitemap generated date."

Can I have multiple sitemaps for different sections? SGEN generates one sitemap (or a sitemap index with child files for large sites). If you need section-specific sitemaps — for example, a separate sitemap submitted to a country-specific Search Console property — use the sitemap filter to export section-scoped URL lists and submit them as separate sitemap files via the Submission tab.

Do I need to resubmit the sitemap every time I publish a new page? No. SGEN updates the sitemap automatically when pages are published. Resubmit only if: you ran an audit and made structural changes, the sitemap URL itself changed, or Search Console is not discovering new pages.

Should I include the sitemap URL in my robots.txt file? Yes. Adding Sitemap: your-site.sgen.com/sitemap.xml to robots.txt helps crawlers that read robots.txt for the sitemap reference. SGEN adds this automatically — confirm it is present under Settings → SEO → Robots.txt.

Related reading

• SEO settings — the panel where sitemap generation settings, noindex controls, and canonical tag configuration live.
• SEO architecture overview — how SGEN generates the sitemap, what triggers regeneration, and the order in which canonical rules resolve.
• SEO health — the automated health monitor that flags sitemap drift between scheduled audits.
• Bulk redirect creation — when the audit finds 404 pages that were in the sitemap, creating redirects is often the resolution.

This recipe is maintained as part of the SGEN operator documentation. If you find a step that does not match your SGEN admin, the platform may have been updated. Post in the community to flag the gap.