Find All URLs on a Domain: List Every Page in One Call

Get every URL on a domain, grouped by section, in one prompt. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to list every URL on scrapingbee.com, drop anything that is not on the www host, group the rest by first path segment, and give me a table with the section, the page count and one example title.

One prompt maps a whole site. 1 min setup · 4-8 hrs saved.

You have a domain, yours or a competitor's, and you want the full list of its pages in a sheet: for an SEO audit, a content inventory, a migration checklist or a crawl seed list. This guide shows the quickest way to get that list into your agent, what it contained when we ran it live on 2026-10-16 against two real sites, and the four ways the list misleads you if you take it at face value. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to find all URLs on a domain?

Monid. Install it once, add a key, and your agent maps a domain with one call: no crawler to host, no sitemap parser to write and no vendor account to open.

• Pay per call. A map call is a flat $0.001 however many URLs come back (Verified 2026-10-16 on the web extraction tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and balance that run the map also run a sitemap reader, a depth-limited crawler and a page-to-Markdown call, so the step after "list the URLs" needs no second account.
• Lives inside your agent. Install the CLI, add your key, run the prompt above, and the tools are there whenever your agent needs them, on demand, with nothing to keep running. Your agent reads SKILL.md and learns the rest, so setup takes about a minute.

On 2026-10-16 one such call returned 1,149 URLs for scrapingbee.com in about two seconds.

How do you list every page on a domain in one call?

Use a map endpoint. It reads the sitemap and follows the site's links in the same request, then hands back the URLs without fetching any page content. Install and authenticate first.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Map the domain

monid run -p firecrawl -e /map \
  -i '{"url":"https://scrapingbee.com","limit":5000}' -j -o map.json

limit goes up to 5,000 URLs per call. The response is an object with a links array, and on our run each row had a url and, for some rows, a title and a description. The rest of this vendor's family, and why a map comes before a crawl, is in the Firecrawl guide.

Group it by section

jq -r '.links[].url' map.json \
  | awk -F/ '$3=="www.scrapingbee.com"{print "/"$4}' \
  | sort | uniq -c | sort -rn | head

On 2026-10-16 that printed the structure of the site in one screen:

431 /scrapers
389 /blog
163 /webscraping-questions
 47 /tutorials
 20 /documentation
 16 /curl-converter
 15 /features

That is usually what someone means by "all the URLs": which parts of the site exist, and how big each one is. To learn what a competitor's site runs on as well as what it contains, technographic data by domain is the next call.

Or hand the whole job to an agent

Can you find all pages on a website yourself?

Yes, with three sources, and none of them is complete alone. The first is the sitemap, a file the site publishes to say which pages it wants found. The second is a crawl, which follows links from the home page. The third is a search engine's index, which is what site: shows and is the least complete of the three.

The sitemap is the free one. Look for the Sitemap: line in robots.txt (scrapingbee.com has one, pointing at https://www.scrapingbee.com/sitemap.xml), fetch the file and read the loc entries. If the file is an index, as the sitemap of our own site is, it points at child sitemaps instead of pages, and you fetch each one:

import re, urllib.request

def urls_in(sitemap_url):
    xml = urllib.request.urlopen(sitemap_url).read().decode()
    return re.findall(r"<loc>(.*?)</loc>", xml)

all_urls = [u for child in child_sitemaps for u in urls_in(child)]

Regex on XML is normally a bad idea and loc is the exception that survives, because the element is flat and the protocol is rigid. That fetch on 2026-10-16 gave 1,132 URLs for scrapingbee.com and 604 for our own site, which is two files of 554 and 50 entries.

A crawl is the part that costs real time. Following links needs a queue, deduplication, politeness delays, a rule for query strings and calendar-style traps, and a browser when the links are built by scripts. Setting that up takes hours before the first useful row, and a site that changes its markup or starts blocking breaks it.

Pick Monid when you will do this more than once or for more than one domain. The sitemap read and the link discovery are one call, the output is JSON instead of XML and HTML you parse, and the collection is maintained for you. The free sitemap fetch above remains the right tool for a single site you already know has a clean sitemap.

What does the list look like, and what can you do with it?

Each row is a URL, and some rows add a title and a description. We ran the map on two sites on 2026-10-16 and the shapes differed in a way you should plan for.

The rows

{ "url": "https://www.scrapingbee.com/documentation/youtube",
  "title": "YouTube API",
  "description": "Our YouTube API allows you to scrape YouTube search results, video metadata, and subtitles in realtime..." }

The url is always present. The title and description appear only on some rows: 135 of the 1,149 on scrapingbee.com and 172 of the 626 on our site. Treat the URL as the key and the title as a bonus. If you need a title for every page, fetch the pages you care about afterwards.

What you can build with it

• A content inventory. Group by first path segment, count, and you have the table above. Add the titles you got and you can see which sections have no titled pages at all.
• A crawl seed list and a Markdown corpus. Map first, then send only the sections you need to a page-to-Markdown call or a content crawler. That keeps the call that bills per page away from the sections you do not want, and turning any URL into LLM-ready Markdown is the next step when the list feeds a RAG store: fetch each URL as Markdown, chunk it and embed it. A mapped site becomes a corpus for an LLM.
• A change detector. Run the same map every week, save each result as map-YYYY-MM-DD.json, and diff the URL lists with comm. New and removed pages are the two lines you read.

Pages that no link and no sitemap mentions will not appear. Backlink data is the one source that can surface them, because outsiders still link to old URLs a redesign orphaned.

What goes wrong when you list a domain's URLs?

Four things hid in our two runs. Each one changes what you should trust in the list.

Why does the map return URLs that are not pages?

Because it collects every URL it can see, and a site links to more than its pages. We compared the 1,149 URLs the map returned for scrapingbee.com with the 1,132 in its sitemap on 2026-10-16. Every sitemap URL was in the map. The map had 17 more, and none of them is a page you would audit:

12  other hosts: dashboard., status., trust., help., benchmark., httpbin., mcp.
 2  files: llms.txt and sitemap.xml
 3  other: /blog, /llms, /#pricing

On our own site the default mode returned 626 URLs against 603 in sitemap-only mode, and the 23 extras were the same kind: five other hosts (api., docs., feedback., hacks., yappers.), SKILL.md, llms.txt, three sitemap files, four .md copies of doc pages and a root URL with a trailing slash. A map finds more than a sitemap, but "more" is subdomains, files and aliases, not hidden pages. Filter on host before you count:

jq -r '.links[].url' map.json | awk -F/ '$3=="www.scrapingbee.com"' | sort -u | wc -l

That prints 1,137 for scrapingbee.com, and the 1,132 sitemap URLs sit inside it.

Why are most rows missing a title?

Because the title comes from pages the tool happened to read, not from the sitemap. In sitemap-only mode our site returned 603 URLs and 0 titles. In default mode it returned 626 URLs and 172 titles. If you ask for "sitemap":"only" you get the cheapest, cleanest list and no metadata. If you need names, plan a second step that fetches the pages.

Why can lastmod tell you nothing?

Because a sitemap's dates are whatever the site's build wrote. Every one of the 1,132 URLs in the scrapingbee.com sitemap carried the same date, 2026-10-14. A single date on every URL is a deploy timestamp, not a record of when anything changed. If you plan to use lastmod to find pages that changed, count the distinct values first. One value means the field carries no information on that site.

Why do two endpoints return different counts?

Because they read different sources. On our site, on the same day:

firecrawl /map, sitemap only     603 URLs
context.dev sitemap              593 URLs
firecrawl /map, default          626 URLs
mrscraper map, depth 1, 3 pages   40 URLs

The sitemap-only map and the sitemap reader differ by 11 doc pages under /docs/api/, which the sitemap reader did not return, and by one tools page that only it returned. None of these is wrong. Each counts a different thing, so pick one source per job and compare like with like when you diff over time.

Which endpoint should I use for which job?

Prices, from monid inspect on 2026-10-16: $0.001 per firecrawl/map call, $0.002533 per context.dev/web/scrape/sitemap call, $0.001 per mrscraper/scrape/map result, $0.002533 per page crawled by context.dev/web/crawl, and $0.0594 per api.strale.io/x402/sitemap-parse call (Verified 2026-10-16). The latency for the crawl is the endpoint's typical (median) run time; the others are from our runs.

The sitemap summary is called like this:

monid run -p api.strale.io -e /x402/sitemap-parse --query '{"url": "https://example.com"}'

It returns counts and path segments, not the list, so use it when the question is "how big is this site and how is it divided" and use the map when you need the URLs themselves.

When is another route a better fit?

You need the content of the pages. A map returns addresses. The page text needs a crawl or a page-to-Markdown call, and those bill per page, so map first, choose the sections, then crawl only those. An unbounded crawl on a faceted shop is an unbounded bill, so set maxPages and an exclude pattern before the first run.

You need pages no link and no sitemap mentions. Neither a map nor a sitemap finds an orphan page, a page behind a login or a page reachable only through a form. If you own the site, your CMS export or server logs are the only complete source.

You are mapping someone else's site at volume. A big crawl is a load event for the operator. Read their terms, respect robots.txt and prefer the sitemap where it is enough. What you may do depends on where you operate and how you use the data.

And the disclosure: we resell these endpoints. The most useful thing in this post is free and does not need us: fetch the sitemap, filter the host, and count the distinct lastmod values before you trust them.

Conclusion

The quickest way to find all URLs on a domain is one map call through Monid: $0.001 a call (Verified 2026-10-16), one key for the map and everything after it, about a minute to set up and ready inside your agent whenever you need it. On 2026-10-16 it returned 1,149 URLs for scrapingbee.com in about two seconds and 626 for our own site. Doing it yourself works for one site with a clean sitemap, and costs hours as soon as you need links followed, a browser or more than one domain.

Two things matter more than the choice of tool. The extra URLs a map finds over a sitemap are subdomains, files and aliases, not hidden pages: on scrapingbee.com all 17 extras were of that kind, so filter on host before you count. And the titles are partial and the lastmod dates can be a single build stamp, so use the URL as the key and check the dates before you build a change detector on them.

How we estimated the time saved: writing and maintaining your own domain lister is roughly 1 to 2 hours for reading sitemap indexes and deduplicating, 2 to 4 hours for a link crawler with delays, query-string rules and a browser fallback, and 1 to 2 hours of upkeep over the first month. That is 4 to 8 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid inspect -p firecrawl -e /map, make one call on your own domain, and compare the count with your sitemap. The gap shows up immediately. Start at monid.ai.

FAQ

Published September 2026.