Two methods take under a minute. Open yourdomain.com/sitemap.xml for the list the site itself publishes, and search site:yourdomain.com in Google for the pages Google kept. Between them you have a rough answer immediately. The rest of this page is about why those two disagree with each other, and with reality.
No single method returns the complete list. Sitemaps show what a site wants indexed. Google shows what it chose to keep. A crawler shows what is reachable by link. Each misses a different set, and the pages that fall through every gap are usually the ones causing the problem.
First decide: your site, or someone else’s
Four of the sources below need verified ownership or server access: Search Console, Bing Webmaster Tools, analytics and raw server logs. They are also the only sources that reveal pages nobody links to. For a competitor, or a client site you have not been given access to yet, you are limited to public signals: sitemap, site: operator, crawler, Wayback Machine, Common Crawl. That list will never be complete. Do not report it as if it were.
Method 1: read the XML sitemap
Open yourdomain.com/sitemap.xml. On most sites this is the fastest complete-ish list you will get, and it takes ten seconds.
The sitemaps.org protocol caps a single sitemap file at 50,000 URLs and 50MB uncompressed (52,428,800 bytes). Anything larger is split, which is why big sites serve a sitemap index: a file listing other sitemap files. The index has the same ceiling, 50,000 child sitemaps, putting the theoretical maximum at 2.5 billion URLs.
Google accepts three formats: XML, RSS/mRSS/Atom 1.0, and plain text with one URL per line. Its documentation says it “doesn’t have a preference,” so a text sitemap such as /sitemap.txt is worth checking too.
WordPress has generated a core sitemap since version 5.5, at /wp-sitemap.xml, splitting at 2,000 entries per sitemap page by default. A WordPress site with 9,000 posts therefore has five child sitemaps for posts alone, plus separate ones for pages, taxonomies and authors. Read the index, not the first file you land on.
To pull every URL out of a sitemap index, run curl -s https://example.com/sitemap.xml | grep -oE ‘<loc>[^<]+’ | sed ‘s/<loc>//’, then feed each child sitemap back through the same command. Pipe through gunzip if the file ends in .gz.
Method 2: when /sitemap.xml returns a 404
A 404 at the default path does not mean there is no sitemap. Check /robots.txt first: it always sits at the root of the domain. Google, Bing and the other major engines support the sitemap field defined by sitemaps.org, and most CMS platforms write the real path there.
Google enforces a robots.txt file size limit of 500 kibibytes and ignores everything after it, so on very large sites the sitemap line can sit past the cutoff and never be read.
If robots.txt is silent, try /sitemap_index.xml, /sitemap1.xml, /sitemap.xml.gz, /wp-sitemap.xml and /sitemap.txt. Yoast and Rank Math on WordPress both default to /sitemap_index.xml.
Method 3: the site: operator, and why its number is wrong
Typing site:example.com into Google is useful for spot checks and misleading for counting. Google’s own documentation states that the operator “doesn’t necessarily return all the URLs that are indexed under the prefix specified in the query,” that the list “is not always exhaustive,” and that “bigger sites shouldn’t expect to see all their URLs in the results.” A bare site: query also “doesn’t rank the results.” Google says it generally puts the shortest URL for the prefix at the top, and nothing after that reflects importance, so do not read priority off the ordering. The result count above the listings is an estimate that fluctuates between refreshes; do not put it in a report. Where the operator earns its place is narrowing:
- site:example.com/blog/ counts one section instead of the whole domain.
- site:example.com inurl:? surfaces parameter URLs that may be duplicating content.
- site:example.com intitle:”page not found” finds soft 404s that are still indexed.
- site:example.com filetype:pdf lists indexed documents, which crawlers and sitemaps routinely skip.
Method 4: the Search Console page indexing report
For a site you own, this is the authoritative view of what Google knows. Go to Indexing > Pages and check the sitemap filter above the chart. All known pages is the default and the one you want; if it has been switched to “All submitted pages,” switch it back, because submitted-only hides everything Google found outside your sitemap, which is exactly the set you are hunting. The not-indexed half is split across 15 documented reasons, including “Crawled – currently not indexed,” “Discovered – currently not indexed,” “Duplicate without user-selected canonical,” “Soft 404,” “URL blocked by robots.txt,” and “URL marked ‘noindex’.”
The catch: Google states the example list for each issue “does not necessarily show all URLs with that issue, and is limited to 1,000 rows.” On a site with 40,000 duplicate parameter URLs you see 1,000. That is a sample, not an inventory, and it is where most people stop and wrongly conclude they have the full list.
Method 5: pull the real list through the Search Console API
The API is how you get past the 1,000-row wall. Two endpoints matter.
The Search Analytics API returns the URLs that earned impressions. Its rowLimit accepts a valid range of 1 to 25,000, default 1,000. For larger sets Google’s documentation says to “request data in batches of 25,000 rows at a time by sending multiple queries and incrementing the startRow value each time.” Group by the page dimension over a 16-month range. Per-site quota is 1,200 queries per minute.
The URL Inspection API gives the indexing verdict for a specific URL. Its per-site quota is 2,000 queries per day and 600 per minute. That daily ceiling is the binding constraint: checking 50,000 URLs takes 25 days. Use it on a shortlist.
Method 6: crawl the site yourself
A crawler starts at the homepage and follows internal links, building the same picture a search engine builds. It finds pages that are linked but missing from the sitemap.
Screaming Frog SEO Spider is the standard tool. The free version is capped at 500 URLs per crawl, cannot save crawls, and blocks JavaScript rendering, custom extraction and the Analytics, Search Console and PageSpeed integrations. A licence is £199 per year (€245 on the euro pricing page) and removes the cap, with real limits then set by your machine’s memory and disk.
Sitebulb is the main alternative, with 10,000 URLs per audit on its Lite desktop plan and 500,000 URLs per audit on Pro; check the vendor’s current pricing on the Sitebulb pricing page.
Two settings decide whether the crawl is honest. Turn on JavaScript rendering for a React, Vue or Next.js build, or a client-rendered navigation returns almost nothing. And add the XML sitemap as an extra URL source, so the crawl merges linked and submitted URLs instead of giving you one or the other. Both live in Crawl Configuration, which the free version does not expose, so this step needs the licence.
Method 7: find the orphan pages no crawl will ever reach
An orphan page has no internal link pointing at it, so a crawler starting from the homepage cannot reach it by definition. Old landing pages, retired campaign URLs and pages left behind by a migration live here. Screaming Frog finds them by cross-referencing external sources against the crawl:
- Connect both APIs under Configuration > API Access.
- Enable Crawl New URLs Discovered In Google Analytics on the General tab of the Google Analytics window, and Crawl New URLs Discovered In Google Search Console on the Search Analytics tab of the Search Console window. Without these, the URLs reach the Orphan Pages report but never enter the crawl.
- Add the XML sitemap as a URL source.
- Run the crawl, then run Crawl Analysis. The three Orphan URLs filters, under the Sitemaps, Analytics and Search Console tabs, stay empty until you do.
The Orphan Pages report then lists URLs that came from the Analytics API, the Search Analytics API or the sitemap and were never matched to anything in the crawl. A Source column marks each one GA, GSC or Sitemaps, so you can tell an unlinked page that still gets traffic from one that only a stale sitemap remembers. As a cross-check, filter the Internal tab for a blank crawl depth: any URL with no depth was never reached by a link.
Method 8: server logs, analytics and Bing
Raw access logs are the only source that records every URL ever requested, including pages with zero internal links, zero impressions and zero sitemap entries. On shared hosting they sit in cPanel under raw log files; on a VPS, usually /var/log/nginx/access.log. Deduplicate the request path column and filter out static assets.
Analytics fills a different gap: pages that received human traffic. In GA4, use the Pages and screens report or an exploration on page path. One caveat matters. Google states that “any dimension with more than 500 values should be considered a high-cardinality dimension,” and high-cardinality data risks being rolled into an (other) row instead of listed individually. Google frames 500 as “not a limit, but a guidance.” The Pages and screens report has a high row limit, so small sites are fine, but add a secondary dimension, a filter or a comparison and the (other) row appears much sooner. If you see it, export through the Data API or BigQuery instead of reading the interface.
Bing Webmaster Tools is worth ten minutes for a second index. Its Site Explorer shows the site as a folder tree and sorts URLs into four statuses: Indexed, Error, Warning and Excluded. Bing often knows about URLs Google has dropped, and the reverse.
Method 9: web archives, for pages that no longer exist
For historical URLs, including pages removed years ago, the two web archives are the only source, and neither requires site access.
The Wayback Machine CDX server lives at https://web.archive.org/cdx/search/cdx. Append ?url=example.com&matchType=domain&collapse=urlkey&fl=original&output=json for a deduplicated list of every URL it has captured.
- matchType=domain returns the host and all subhosts; matchType=prefix restricts to a path.
- collapse=urlkey filters out duplicate adjacent captures so you get unique URLs, not one row per snapshot.
- limit=N returns the first N results; a negative value returns the last N.
Common Crawl at index.commoncrawl.org is the second archive. The cdx-index-client tool queries it with wildcards such as *.example.com, and –show-num-pages reports the result set size before you download it.
Archive data is noisy: URLs that 404 today, tracking parameter variants, pages from previous owners of the domain. Treat it as a discovery list to validate, not an inventory.
Reconcile the lists, then act on the gaps
Running the methods is the easy half. The value is in the differences between them. Before comparing anything, normalise: strip trailing slashes consistently, force one protocol, collapse www and non-www, lowercase the host, and decide once whether query strings count as separate URLs.
| Where the URL appears | Where it is missing | What it means |
|---|---|---|
| Sitemap | Crawl | Orphan page. Nothing on the site links to it. |
| Crawl | Sitemap | Either excluded on purpose, or the sitemap generator is broken. |
| Logs or analytics | Everywhere else | Ghost from an earlier version of the site, still being requested. |
| Wayback or Common Crawl | Everywhere else | Removed page. Check whether it should redirect somewhere. |
| Sitemap and crawl | Search Console indexed | Google saw it and declined. Check the not-indexed reason. |
The last row matters most for traffic. A page that exists, is linked, is in the sitemap and still sits under “Crawled – currently not indexed” is telling you about quality or duplication, not crawling. Group those URLs by template before you touch them: when a whole page type drops out at once, the fix is at the template level, not page by page.
Need the full inventory without building it by hand
We run this reconciliation as part of a technical SEO audit: sitemap, crawl, Search Console API export, log sample and archive discovery, merged into one deduplicated list with the reason each URL is missing from the others.
Run it in this order
- Fetch the sitemap index and every child sitemap. Deduplicate. This is your baseline.
- Export Search Console, Pages report, All known pages, both halves.
- Crawl with the sitemap as an extra URL source, JavaScript rendering on if needed.
- Connect the GA and GSC APIs in the crawler and run Crawl Analysis for orphans.
- Pull 90 days of access logs and extract unique paths.
- Run a Wayback CDX query for anything predating your log retention.
- Normalise all six lists and diff them in a spreadsheet or with comm.
For a site you do not own, you have steps 1, 3 and 6 plus the site: operator. Say so when you report the number: the gap between “all pages” and “all pages visible from outside” is usually large on a content-heavy domain.
Frequently asked questions
How do I see all the pages on a website I do not own?
Read its /sitemap.xml, crawl it with Screaming Frog (free up to 500 URLs), and query the Wayback CDX API with matchType=domain&collapse=urlkey. Search Console and server logs need verified ownership.
Does site:example.com show every indexed page?
No. Google states the operator “doesn’t necessarily return all the URLs that are indexed” and that “bigger sites shouldn’t expect to see all their URLs.” Use the Search Console Pages report for your own site.
Why does my sitemap show fewer pages than my crawler finds?
Most likely the sitemap is split. A single file is capped at 50,000 URLs and WordPress core splits at 2,000 entries per page, so you may be reading one child file instead of the index. Otherwise the generator is excluding page types, or the extras are parameter variants the sitemap deliberately omits.
How do I export more than 1,000 URLs from Search Console?
Use the Search Analytics API. Its rowLimit goes up to 25,000 per request, and you page through larger sets by incrementing startRow. The UI report’s per-issue example list stays capped at 1,000 rows.
What is the fastest free way to list a small site’s pages?
Screaming Frog’s free version. It crawls up to 500 URLs, which covers most brochure and small business sites in one pass, and exports to CSV. Above 500 you need the £199 per year licence.
Once you have the list, the question changes from quantity to quality: which pages stay, which merge, how internal links flow between them. Start by checking the pages you keep are marked up properly, with the HTML tags that matter for SEO.
