How to Find All Pages on a Website

You think your website has 40 pages. It has 400. The gap between those two numbers is where SEO problems live: orphaned pages, forgotten drafts, duplicate URLs, and thin content Google indexed years ago and never told you about.

Finding every page on a site is the first move in any audit, migration, or content cleanup. It sounds trivial. It is not. No single tool sees everything, so the reliable approach is to combine sources and reconcile them. Here are the methods that actually work, from fastest to most thorough.

Start with the XML sitemap

The sitemap is the site’s own declaration of what exists. Check yourdomain.com/sitemap.xml first. If it redirects or 404s, look in robots.txt, which usually lists the sitemap location at the bottom.

Large sites use a sitemap index that points to multiple child sitemaps, often split by content type: posts, pages, products, categories. Open each one. Modern CMS platforms generate these automatically, so the sitemap is a strong starting inventory.

One warning: the sitemap shows what the site wants indexed, not what actually exists. Old pages, staging URLs, and pages excluded from the sitemap will be missing. Treat it as a floor, not a ceiling.

Use Google Search Console

If you own the site, Search Console is the highest-signal source you have. Open the Pages report under Indexing. It shows every URL Google knows about, split into indexed and not-indexed buckets with reasons: crawled but not indexed, discovered but not crawled, duplicate, blocked by robots, and more.

This is where the surprises show up. Pages you forgot. Parameter URLs multiplying out of control. Old blog posts with almost no traffic. Export the full list. The not-indexed section is often more useful than the indexed one, because it tells you what Google decided to ignore and why.

Crawl the site yourself

A crawler starts at your homepage and follows every internal link, building a map the same way a search engine does. This is the closest thing to ground truth for pages that are actually reachable through navigation.

Point a crawler at your domain and let it run. You will get a full URL list plus status codes, titles, meta descriptions, canonical tags, and redirect chains. This method catches things sitemaps miss and vice versa, which is exactly why you run both.

Two things to watch. First, a crawler only finds pages that are linked from somewhere. Truly orphaned pages, ones with zero internal links, stay invisible. Second, JavaScript-heavy sites need a crawler that renders JS, or you will miss content loaded after the initial HTML.

Query the search index directly

For a fast, no-tools estimate, type site:yourdomain.com into Google. The result count is a rough tally of indexed pages, and you can scroll through actual URLs. Narrow it with operators: site:yourdomain.com/blog for a section, or add a keyword to find pages on a topic.

This number is approximate and Google is upfront about that. But it is a useful sanity check. If site: shows 5,000 results and your sitemap lists 300, you have a serious indexation problem worth investigating: thin pages, parameter bloat, or an old subdomain still live.

Mine your server logs and analytics

Server access logs record every URL that was ever requested, including pages no crawler will find because nothing links to them. This is the best way to surface true orphans and legacy URLs from a previous site version.

Analytics helps too. Pull the full list of landing pages that received any traffic over the last year. If a URL got a visit, it exists, even if it is absent from your sitemap and your crawl. Between logs and analytics you capture the long tail that structured methods leave behind.

Reconcile the lists into one master inventory

Now the real work. Each method gives you a partial view, so pull every URL list into one spreadsheet and deduplicate. The interesting findings are in the mismatches:

• In sitemap, not in crawl: likely orphaned. No internal links point to it. Fix the linking or remove the page.
• In crawl, not in sitemap: either intentionally excluded or an oversight. Decide which.
• In Search Console as not-indexed: weak, duplicate, or blocked pages that need a decision.
• Only in logs or analytics: ghosts from a past site that may still rank or leak link equity.

Normalize everything before comparing: strip trailing slashes, unify http and https, and collapse www and non-www so the same page does not appear as four different rows. A clean master list is the foundation for pruning, redirects, and internal linking decisions. If you are heading into a rebuild, our guide on website migration and SEO covers how to carry that inventory across safely.

Getting started

Do it in this order. Open your sitemap for the baseline. Export the Search Console Pages report for what Google actually sees. Run one full crawl for reachable pages. Then patch the gaps with logs and analytics. Reconcile in a spreadsheet and you will know your site better than most owners ever do.

Once you have the full picture, the next questions are about quality, not quantity: which pages deserve to stay, which need merging, and how internal links should flow. Our piece on technical SEO audit basics walks through what to do with the inventory you just built.

If you would rather have this done for you, or you are staring at thousands of URLs and unsure what to prune, our team runs full crawls and content audits as part of our AI-driven SEO work. Come talk it through in our Telegram community. We are happy to point you in the right direction.