Sitemap and Indexing Audit: A Step-by-Step Method
Published by The Branded Agency · · 13 min read

A sitemap and indexing audit answers one question: do the URLs you tell search engines about match the URLs they can fetch, understand and keep in their index? You answer it by validating the sitemap file, checking every URL inside it, and reconciling three lists: what the sitemap declares, what a crawler finds, and what Google has indexed. Our Brand Health Audit runs this kind of check automatically as part of its SEO category, so you get the findings as scored, evidence-linked recommendations instead of a spreadsheet to build yourself.
TL;DR
- A sitemap is a hint, not an instruction. Google does not guarantee that it will download the file or crawl every URL in it.
- The sitemap should list only canonical, fetchable URLs that return a 200 status. Redirected, noindex, blocked and 404 URLs should come out.
- The real diagnosis comes from comparing three lists: sitemap URLs, crawled URLs and indexed URLs. Each gap points to a different cause.
- Respect the limits: 50,000 URLs or 50MB uncompressed per file, with a sitemap index above that.
- Repeat the audit after migrations, redesigns and large content changes, not only once a year.
Table of Contents
- What a sitemap does, and what it does not
- Before you start: gather the inputs
- Step 1: Validate the sitemap file itself
- Step 2: Audit every URL inside the sitemap
- Step 3: Check the metadata, especially lastmod
- Step 4: Find what is missing from the sitemap
- Step 5: Reconcile sitemap, crawl and index
- Step 6: Prioritize and fix
- Sitemap best practices in one place
- Where the audit stops being about the sitemap
- Doing the audit with the Brand Health Audit
- Start with a free scan
- Our perspective
- FAQ
- Sources
What a sitemap does, and what it does not
An XML sitemap is a file that lists the URLs you want search engines to know about. According to Google Search Central, it can also carry extra information about images, video, news and localized versions of a page. Submitting it is a hint. It helps discovery, which matters most for new sites, large sites, sites with few internal links and sites that publish often.
Two consequences follow for an audit:
- A sitemap does not force indexing. A URL can sit in a perfect sitemap and still not be indexed because Google judged it duplicate, thin or blocked.
- A sitemap is a statement of intent. Every URL you list says "this page matters and is the preferred version." If a third of those URLs redirect or return errors, the file contradicts itself, and that is exactly what an audit exists to catch.
That is why the audit has two halves. The first checks the file. The second checks what search engines did with it.
Before you start: gather the inputs
You need four things:
- The sitemap location. Usually
/sitemap.xmlor/sitemap_index.xmlat the root. Google recommends the root because a sitemap there can cover all files on the site. Also check therobots.txtfile, which should contain aSitemap:line pointing to it. - Search Console access for the property, which gives you the Sitemaps report, the Page indexing report and URL Inspection.
- A crawl of the live site. Any crawler that exports status codes, canonicals and meta robots values works.
- A list of the pages that earn money or leads. You will use it to decide which gaps are urgent.
The last item is easy to skip and most often regretted. A sitemap with 4,000 URLs and 600 problems sounds alarming, but if the 600 are tag archives, the fix can wait. If 12 of them are your service pages, it cannot.
Step 1: Validate the sitemap file itself
Start with the container before you examine the contents.
What to check
- The file loads at its stated URL and returns a 200 status.
- The XML is well formed, with no broken tags or stray characters.
- URLs are fully qualified and absolute, and the file uses UTF-8 encoding, as Google's sitemap documentation specifies.
- The file stays under 50MB uncompressed and 50,000 URLs. Above that it must be split into several files and listed in a sitemap index.
- The protocol and host match the live site. A sitemap that lists
http://URLs on an HTTPS site, orwwwURLs on a non-wwwsite, is one of the common sitemap errors that quietly breaks processing.
Where it goes wrong
Plugin-generated sitemaps fail in predictable ways: a stylesheet (XSLT) file that is unavailable, a rewrite rule that returns the homepage instead of the XML, or a cache that serves an old version for days. The same Yoast guide lists these causes, and the symptom is usually a sitemap that "works" in the browser but shows an error in Search Console.
If you use a sitemap index, open every child file. A clean index that points to a broken child sitemap still loses all the URLs in that child.
Step 2: Audit every URL inside the sitemap
This is the step where most of the findings come from. Google's best-practice guidance is short: include only URLs Googlebot can fetch and only canonical URLs, and avoid duplicates, URLs blocked by robots.txt and URLs of pages that do not exist.
Turned into a checklist, every URL in the file should pass six tests:
| Test | Pass condition | Typical failure |
|---|---|---|
| Status code | Returns 200 | 301/302 redirect, 404, 410, 5xx |
| Canonical | Self-referencing canonical | Canonical points to a different URL |
| Indexability | No noindex in meta robots or X-Robots-Tag |
Staging or thank-you pages marked noindex |
| Robots access | Not disallowed in robots.txt | Section blocked after a site update |
| Uniqueness | Appears once | Same URL with and without a trailing slash |
| Parameters | Clean, preferred form only | Tracking or filter parameters listed |
Export the sitemap URLs, feed them into your crawler in list mode and filter by each column. Redirects deserve special attention: a sitemap that lists the old URL after a migration sends crawlers through a hop on every visit. We cover the cleanup in our guide to redirect chains and 4xx errors.
A worked example
Say a services site lists 850 URLs in its sitemap. The crawl returns:
- 790 URLs with status 200 and a self-referencing canonical.
- 35 URLs that redirect to the HTTPS version (the sitemap was generated before an HTTPS move).
- 15 URLs that return 404 because old service pages were deleted.
- 10 URLs with a
noindextag, left from a staging template.
The fix is to regenerate the sitemap from the current URL set, so 60 entries drop out and the file lists 790. Nothing was "wrong" with indexing in this case. The sitemap simply was not an honest list of the site, which makes it a weaker signal for the pages that matter.
Step 3: Check the metadata, especially lastmod
The lastmod tag is optional, but when you include it, it should reflect the time of the last meaningful content change, in W3C Datetime format. Two patterns to look for:
- Every URL carries today's date. The generator stamps the build time on all pages, so the signal means nothing and crawlers learn to ignore it.
- Dates never change. A page was rewritten but the sitemap still shows the original publish date.
A correct value changes only when the page content does. Changing a footer link or a sidebar widget is not a meaningful content change.
Step 4: Find what is missing from the sitemap
Validation shows what should not be in the file. Completeness shows what should be there and is not. Compare your crawl with the sitemap and list URLs that:
- Are indexable and return 200 but are absent from the sitemap.
- Appear in your list of money pages but are not linked from anywhere else on the site (orphan pages, which crawlers reach only through the sitemap or external links).
Not every indexable URL needs to be in the sitemap, and a small, well-linked site often does not depend on it at all. But every page you want to rank should be, and the sitemap is the cheapest way to make the intent explicit.
Step 5: Reconcile sitemap, crawl and index
This is the part of the audit that most checklists skip, and it is where you learn why pages are missing from Google. Build a table with one row per URL and three yes/no columns:
- In sitemap: what you declared.
- In crawl: what a crawler reaches by following links.
- In index: what Google reports as indexed.
For the index column, use Search Console. The Sitemaps report shows the processing status and any errors for each submitted file, and it links to index coverage data. URL Inspection tells you whether a specific URL is available to Google and what Google chose as its canonical.
Then read the combinations:
| In sitemap | In crawl | In index | What it usually means |
|---|---|---|---|
| Yes | Yes | Yes | Healthy. No action. |
| Yes | Yes | No | Google saw the URL and chose not to keep it. Check quality, duplication, canonical and noindex. |
| Yes | No | No | An orphan page. Add internal links, or remove it if it should not exist. |
| No | Yes | Yes | Indexed but not declared. Add it if it matters. Google found it through links. |
| No | Yes | No | Not declared and not kept. Decide whether it deserves to be indexed at all. |
| Yes | Yes, redirects | No | The sitemap lists an obsolete URL. Replace it with the destination. |
The second row, "in sitemap and crawl but not in the index," is the group that deserves the most analysis. Look at those URLs as a set. If they share a template, such as thin tag pages or near-duplicate product variants, the cause is probably a content or canonical rule rather than a one-off problem. If they are your best service pages, check for a canonical pointing elsewhere or an accidental noindex before anything else.
A large share of URLs that are indexed but not in the sitemap is not an error. It means Google is finding pages through links, which is normal. It is worth a line in your report only if important pages are on that list.
Step 6: Prioritize and fix
Sort findings by what they cost, not by how many URLs they touch:
- Blocking issues on money pages. Noindex, robots.txt disallow, canonicals pointing away, 4xx or 5xx status.
- Sitemap-wide faults. Wrong protocol or host, a broken child sitemap, a sitemap that fails to process.
- Hygiene. Redirects, duplicates, parameters, inaccurate
lastmod. - Coverage. Valuable pages missing from the file.
After the fixes, resubmit the sitemap in Search Console and check the report again in a few days to a couple of weeks. A sitemap is processed on Google's schedule, so patience is part of the method. If the status does not change after the file is clean, look at caching, which commonly prevents updates from being seen.
For deeper explanations of individual sitemap findings and their fixes, our article on XML sitemap issues in an audit walks through each one.
Sitemap best practices in one place
The rules that come up in every audit:
- List only canonical URLs that return 200 and can be indexed.
- Use absolute URLs and UTF-8 encoding.
- Stay within 50,000 URLs and 50MB uncompressed per file, and split larger sites with a sitemap index.
- Place the file at the site root and reference it in robots.txt.
- Keep
lastmodhonest: change it only when content changes in a meaningful way. - Generate the file automatically from your CMS or database so it updates when the site does, and never maintain it by hand on a growing site.
- Re-audit after every migration, URL structure change or platform switch.
Where the audit stops being about the sitemap
A clean sitemap and a clean index report tell you that search engines can find and keep your pages. They do not tell you whether those pages are discoverable for the right language, or whether AI assistants and answer engines describe your brand accurately. Those questions sit in adjacent audit categories: SEO visibility, answer-engine readiness and how AI assistants describe your brand. If you are scoping a wider technical review, technical SEO auditing software can help you plan what a full audit should include. For a view of what a full checklist covers in order of priority, see our SEO audit checklist.
Doing the audit with the Brand Health Audit
Doing the six steps by hand means exporting, cross-referencing and interpreting several data sources, then repeating the work after every release. The Brand Health Audit automates the evidence gathering. It scores twelve categories, including SEO, AEO, GEO and website experience, against a fixed checklist, and it ties each recommendation to observable public data. It also separates score, data coverage and confidence, so you can see how much each finding rests on. The website experience audit covers accessibility, structure and technical health, and our write-up of how we audit SEO fundamentals explains the checks and scoring.
Start with a free scan
The initial scan uses public data only and needs no sales call or credit card. If you want the full picture, the full twelve-category audit is $99 one-off, and subscription plans start at $79 per month on the pricing page.
To see what the output looks like before you decide, open the sample brand audit report or start with the free brand audit.
Our perspective
A sitemap audit is a test of whether a site tells the truth about itself. Every URL in the file claims to be a page worth indexing, and every mismatch between that claim and the live site is evidence you can observe.
That is how we build the Brand Health Audit: repeatable findings scored against a fixed checklist, not opinion. We keep score, data coverage and confidence apart, because a finding based on thin data should not carry the same weight as one based on a full crawl.
Technical discoverability is also no longer only about Google. The same clean, canonical, consistent signals feed answer engines and AI assistants, which is why our SEO, AEO and GEO categories sit in one audit. A brand that is easy to crawl, and described consistently everywhere, is easier to find and to cite.
Quincy
FAQ
How often should I audit my sitemap and indexing?
Run a full audit after any migration, redesign, platform change or large content release. Between those events, check the Sitemaps report in Search Console regularly, since it surfaces processing errors as they appear. Sites that publish or remove many pages every week benefit from automated, recurring checks.
Does every page need to be in the sitemap?
No. List the canonical, indexable pages you want to rank. Leave out redirects, noindex pages, duplicates, parameter URLs and anything blocked by robots.txt. A shorter, accurate file is a stronger signal than a long, noisy one.
Why does Search Console show a sitemap as processed but my pages are not indexed?
Processing means Google read the file. It does not mean Google kept every URL. Submitting a sitemap is only a hint, so check each non-indexed URL for duplicate content, a canonical pointing elsewhere, a noindex tag or thin content, using URL Inspection.
What is the size limit for a sitemap?
Each sitemap file can hold up to 50,000 URLs or 50MB uncompressed. If you exceed either, split the URLs across several files and submit a sitemap index that lists them, as described in Google's documentation.
Sources
- Build and Submit a Sitemap, Google Search Central
- Best practices for XML sitemaps and RSS/Atom feeds, Google Search Central Blog
- Sitemaps report, Search Console Help
- Common XML sitemap errors, Yoast
Recommended
Where this shows up in your audit
These scored categories cover what this article talks about.
Industry brand audits
Dental practices brand audit · Mental health & therapy practices brand audit · Veterinary practices brand audit
Want this handled for you?
Content and search work that builds durable organic visibility.
Content Marketing at The Branded AgencyGoing deeper on the strategy behind it: The strategy thinking behind the audit, written for brand and marketing leads. Insights from The Branded Agency.
Measured against real data
Every figure we publish comes from completed audits, reported as anonymised averages.
Related articles
- Copilot SEO: How to Get Your Brand Cited by Microsoft Copilot
Copilot SEO: How to Get Your Brand Cited by Microsoft Copilot! Geometric content path ending at citation card Copilot SEO means making your brand easy for Microsoft Copilot to find, understand and cit
- Review Schema Markup: How to Add It and Earn Stars
Review Schema Markup: How to Add It and Earn Stars! Geometric cards represent review markup and ratings Review schema markup is structured data, usually JSON-LD, that tells search engines a page conta
- Schema Markup Deficits in Your Audit: Meaning and Fixes
Understand schema markup deficits in your audit report. Learn how syntax errors, incomplete entities, and missing nestings impact AI search visibility and SEO scores.
Stay sharp
Get the next brand breakdown in your inbox
Practical brand strategy, messaging and AI-search insights. No fluff, no daily sends — just the work that moves brands.
Written by
Quincy Samycia
Founder & Brand Strategist, The Branded Agency
Quincy leads brand strategy at The Branded Agency, where he has spent over a decade helping founders and B2B teams sharpen their positioning, messaging and creative systems so growth stops depending on guesswork.
More from Quincy Samycia →See where your brand actually stands
Run the Brand Health Audit and get a scored diagnostic of your messaging, positioning and visibility.
