Find All URLs on a Domain: Why the Sitemap Route Usually Stops

Copy this line to your agent to inspect a domain's sitemap structure.

set up https://monid.ai/SKILL.md and use api.strale.io /x402/sitemap-parse on a domain

"Parse the sitemap" is the standard answer to this question, and on 2026-09-02 it produced a usable URL count for one of the four domains tested. The other three returned a sitemap index and no count at all. That is not a broken endpoint; it is what sitemaps are actually like once a site is bigger than a brochure, and it changes the plan. This guide runs through Monid, the OpenRouter for agent tools.

Can you get every URL on a domain?

No, and starting from that makes the rest of the work sane.

There is no authoritative list

A website is not a directory you can list. It is whatever a server returns for whatever paths exist, and only the operator knows the full set. Everything else is inference from three sources: what the site declares, what is linked, and what someone else already indexed.

The three routes, and what each one misses

A sitemap is a declaration. It contains what the site chose to publish, which is often a subset and occasionally a superset, since sitemaps go stale and list pages that now 404.

A crawl follows links from a starting point. It finds anything reachable by link and misses anything that is not: orphan pages, paginated tails behind a script, anything gated.

A search index returns what a search engine chose to keep. It is the most curated and the least complete, and what a SERP API actually returns covers how much of an index you can reach that way.

What "all URLs" usually means in practice

Nearly always one of: every page that could rank, every page in a section, or every page that changed recently. Those are all answerable. "Every URL that exists" is not, and pursuing it is how a two-hour job becomes a week.

📖 See also SERP History: Track the Shape, Not the Number

Why did three of four sitemaps return no URLs?

Because they are sitemap indexes, and an index does not contain URLs. It contains other sitemaps.

The measurement

api.strale.io/x402/sitemap-parse against four domains on 2026-09-02:

One flat sitemap, three indexes. For the indexes the response carries child_sitemaps and child_count instead: apify.com listed 13 children including pages.xml and actors1.xml through actors5.xml; monid.ai listed 2.

Why indexes are the normal case

The sitemap protocol caps a single file at 50,000 URLs, so any site past that must split, and most sites split long before it for their own convenience: one sitemap per content type, regenerated independently. Five numbered files of actor pages is a site telling you where its bulk is.

The part that surprised us

Passing a specific child sitemap URL does not work. Given https://monid.ai/pages.xml, which a plain fetch confirms is a real <urlset>, the response came back describing https://monid.ai/sitemap.xml instead. The endpoint resolves the domain's root sitemap and ignores the path you hand it.

So this endpoint is a reconnaissance tool, not an enumerator. It tells you how a site organises itself and roughly where the volume sits. It will not walk the index for you, and on a site with an index it returns no URLs at all.

One more thing worth reading

For scrapingbee.com, newest_lastmod and oldest_lastmod were both 2026-08-31. Every one of 1,113 URLs carried the same date. That is a build timestamp written at deploy, not a record of when anything changed, and it means lastmod on that site carries no information about content freshness. If you were planning to use lastmod to detect changes, check that the values actually differ before building on them. This is the same class of check as asserting on values rather than status codes, argued in the Amazon ASIN guide.

How do you actually enumerate a site?

Three steps, and the order matters because the cheap one tells you whether the expensive one is needed.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Step 1. Reconnoitre with the sitemap

What it does. Tells you the shape of the site in one call: flat or indexed, how many sections, and whether lastmod is real.

The endpoints. api.strale.io/x402/sitemap-parse, billed per call. Takes url.

The call.

monid run -p api.strale.io -e /x402/sitemap-parse --query '{"url": "https://example.com"}'

What comes back. Either type: "urlset" with total_urls, has_lastmod, has_changefreq, has_priority, newest_lastmod, oldest_lastmod, top_path_segments and a sample; or type: "sitemap_index" with child_sitemaps and child_count.

Read top_path_segments when you get it. It tells you how the site's URL space is divided, which is usually what someone actually wanted when they asked for every URL.

What it costs. Per call. Current figures at monid.ai/tools.

Step 2. Fetch the child sitemaps yourself

What it does. Fills the gap the endpoint leaves.

The call. No endpoint needed. Child sitemaps are plain XML over HTTP:

import re, urllib.request

def urls_in(sitemap_url):
    xml = urllib.request.urlopen(sitemap_url).read().decode()
    return re.findall(r"<loc>(.*?)</loc>", xml)

all_urls = [u for child in child_sitemaps for u in urls_in(child)]

Regex on XML is normally a bad idea and <loc> is the exception that survives, because the element is flat and the protocol is rigid. Use a real parser if the sitemap carries image or video extensions you care about. The same durability argument applies to any selector you write, per the XPath guide.

What it costs. Nothing but your own bandwidth.

Step 3. Crawl only what the sitemap does not cover

What it does. Finds linked pages the site never declared.

The endpoints. context.dev/web/crawl returns crawled pages as Markdown; mrscraper/scrape/map discovers URL structure and takes url, maxDepth, maxPages, limit, includePatterns and excludePatterns.

An honest note. Both are asynchronous, and on 2026-09-02 the Monid CLI's polling path returned an HTML redirect rather than JSON on both 0.1.6 and 0.1.7, so neither is reported here as measured. The schemas are as described; the responses are not something this article verified, and saying so seemed better than describing output we did not see.

The point of excludePatterns. Faceted navigation generates effectively infinite URLs, and a crawl without exclusions on an ecommerce site will happily enumerate every colour-and-size combination until you stop paying. Set them before the first run, not after the bill.

What does a sitemap tell you that a crawl does not?

Four things, and they are the reason to start here even though it does not enumerate.

Intent

A sitemap is the site's own statement about which pages matter. A crawl cannot distinguish a flagship page from a tag archive; a sitemap that omits the tag archives has told you.

Structure, cheaply

child_sitemaps and top_path_segments describe how a site is organised for the price of one call. Deriving the same picture from a crawl costs thousands of requests.

Orphans, by subtraction

Pages listed in a sitemap but not reachable by link are orphans, and they are usually a bug worth knowing about. You can only find them by comparing the two sources, which is a good reason to collect both rather than picking one.

Freshness claims, and whether to believe them

lastmod is the only cheap change signal available, when it is real. Today it was not on the one site where we could check it. That is worth knowing before designing an incremental pipeline around it.

The metadata that tells you which is which

Both endpoints from this vendor return a _meta.provenance.source. On the sitemap parser it reads http-fetch. On the pricing extractor it reads claude-haiku, meaning a model produced that output. Same vendor, same envelope, and one field separating a deterministic parse from a generated one. That is the field to check before deciding how hard to verify a response.

Which endpoint should I use for which job?

Every row was verified with monid inspect on 2026-09-02. The table gives billing shape rather than figures; shape drives design and current numbers live on monid.ai/tools.

The last row is a genuinely different discovery source and it is the one people forget. Backlink data surfaces URLs that outsiders link to, which will include old pages a redesign orphaned and pages the current sitemap has dropped. If your goal is "find everything that might still get traffic", that set is not reachable from either a sitemap or a crawl. We used exactly that source to audit our own link graph, described in the crawler build-versus-buy piece.

When should you not enumerate a site?

Three cases.

You need a count, not a list. If the question is "how big is this site", the sitemap total or a child_count answers it for one call. A crawl to establish the same number is an expensive way to get one integer.

The site is generated combinatorially. Faceted search, calendars and parameterised filters produce unbounded URL spaces where "all URLs" is not a finite set. Decide which parameters matter and exclude the rest, or the crawl never ends.

Someone else's site, at volume. A thorough crawl is a load event for the operator. Rate-limit, respect robots.txt, and prefer the sitemap for anything you can get from it. The politeness arithmetic is the same one in the rotating proxy guide. The 502 we hit on a different endpoint yesterday was a reminder that origins fall over, and being the cause is avoidable.

And the disclosure: this is Monid's blog and we sell per-call access to all of the above. The honest summary of the paid sitemap endpoint is that it is reconnaissance rather than enumeration, it stops at the index on most real sites, and the follow-up step in this article is free HTTP you can do yourself.

Conclusion

The sitemap route is the right first move and it is not the answer. On 2026-09-02, three of four domains returned a sitemap index rather than a URL list, and the endpoint reports the index without walking it. On the fourth, 1,113 URLs carried an identical lastmod, which means the freshness field there is a deploy timestamp rather than a change signal.

So the working sequence is: one call to learn the shape, free HTTP to pull the child sitemaps, and a crawl only for what the site never declared, with exclusion patterns set before the first run. Reach for backlink data when the goal includes pages the site has forgotten about.

And check _meta.provenance.source when a response offers it. http-fetch and claude-haiku are very different promises about the same-looking JSON.

Free next step: curl https://yourtarget.com/robots.txt and look for the Sitemap: lines. It costs nothing, it often names sitemaps the root file omits, and it takes about five seconds. Start at monid.ai.

FAQ

Last updated September 2026.