Web Scraping Services: Hire One or Pay Per Call?

Get the pages of a website as clean text without hiring a scraping service. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to list the URLs on monid.ai, crawl two pages of it into markdown, and tell me how many pages succeeded and what the run cost.

One prompt lists a site and reads its pages. 1 min setup · 8-20 hrs saved.

You need data from websites, and the search results for "web scraping services" are agencies and platforms that want a call, a scoping document and a contract. Sometimes that is right. Often the job is smaller: read the pages of a few sites, on a schedule, and hand the text to your team or your agent. This guide shows how to do that with a few calls, what came back when we ran them on 2026-10-16, and where a service is the better choice. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to get web data without hiring a scraping service?

Monid. Install it once, add a key, and your agent calls a web extraction endpoint with a URL and gets the page back as markdown, with no scoping call, no contract and no vendor account to open.

• Pay per call. Listing a site's URLs, reading one page and crawling a page each cost $0.002533 (Verified 2026-10-16 on the web extraction tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and the same balance call the web extraction endpoints and every other tool on Monid, so a second data source is another call and not another vendor.
• Lives inside your agent. Install the CLI, add your key, run the prompt above, and the tools are there whenever your agent needs them, with nothing to keep running. Your agent reads SKILL.md and learns the rest, so setup is about a minute.

A scraping service is not an API, so there is no official key to apply for here. The comparison is between handing the whole job to a company and calling the pieces yourself.

How do you scrape a list of websites without hiring anyone?

Call a URL-listing endpoint to find the pages, then a crawl or scrape endpoint to read them, and loop over your sites. Three kinds of offer are sold as web scraping services, and they differ in how much work they take off you.

Monid fits a list of sites because the job is the same few calls repeated, and you pay for the pages you read and nothing between runs.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Inspect the price first

monid discover -q "crawl a site and return clean markdown"
monid inspect -p context.dev -e /web/crawl

List the URLs on a site

monid run -p context.dev -e /web/scrape/sitemap --query '{"domain":"monid.ai","maxLinks":20}' -w

Crawl the pages into markdown

monid run -p context.dev -e /web/crawl -w -i '{
  "url": "https://example.com/catalogue",
  "maxPages": 25,
  "maxDepth": 2,
  "useMainContentOnly": true
}'

Each page returned is one billed unit, so a 25-page crawl is at most $0.002533 times 25, or $0.063325 (Verified 2026-10-16).

Read one known page

monid run -p context.dev -e /web/scrape/markdown --query '{"url":"https://example.com","useMainContentOnly":true}' -w

Or hand the whole job to an agent

If you are choosing between tool types first, which kind of web scraping tool you need sorts the category by who operates it.

Can you do the web scraping yourself instead of hiring a service?

Yes, and for a few friendly sites it is an afternoon. You fetch pages with requests, parse the HTML with BeautifulSoup, follow links and write the text to files. Hiring a freelancer or an agency buys the same result with someone else's time.

The cost is everything after the first run. A fetch layer has to render JavaScript pages, survive blocking and retry. A crawler needs rules for which links to follow and when to stop. Every site that changes its layout breaks a parser, and someone has to notice that it broke. The first working version takes hours and the upkeep does not end, whether the person doing it is on your team or on a contract.

The Monid route moves the fetch, the rendering and the proxies off your plate. The /web/scrape/markdown description states that JavaScript rendering, anti-bot handling and premium residential proxies are included in the price and that failed or blocked requests are not billed. You keep the part that is yours: which sites, how often and what to do with the text. For a walk-through of fetching pages as data from Python, see Python web scraping without a scraper.

What does scraped web data look like, and what can you build with it?

A crawl returns one markdown document per page plus a metadata record for it, so the output is text you can read and fields you can filter. We ran /web/crawl on 2026-10-16 starting at https://monid.ai/tools with maxPages 2 and maxDepth 1, so the fields below are what came back.

The response

{ "results": [
    { "markdown": "[1688 ... 1 endpoint](https://monid.ai/tools/1688) [3D Generation ...",
      "metadata": { "sourceUrl": "https://monid.ai/tools", "title": "Monid | Connect your agent to every tool it needs",
                    "statusCode": 200, "crawlDepth": 1, "success": true } },
    { "markdown": "...",
      "metadata": { "sourceUrl": "https://monid.ai/", "statusCode": 200, "success": true } } ],
  "metadata": { "numUrls": 2, "maxCrawlDepth": 1, "numSucceeded": 2, "numFailed": 0, "numSkipped": 0 },
  "cache_metadata": { "status": "miss", "age_ms": 0 } }

Each result has its markdown and a metadata object. The home page record carried finalUrl, title, description, canonicalUrl, image, an openGraph and a twitter block, a headings list with levels, a jsonLd list, crawlDepth, statusCode and success. The summary at the bottom counts the pages tried, succeeded, failed and skipped, and cache_metadata tells you whether the page came from a cache.

The /web/scrape/markdown endpoint returns the same markdown for one URL. On https://example.com it returned the page text, a contentLength of 1342 and a metadata record with sourceUrl, finalUrl, title, language and additionalMeta.

What you can build with it

• A site knowledge base. Crawl a documentation site or a help centre and index the markdown for search or for an agent. The text arrives without navigation chrome when you set useMainContentOnly.
• A site map you can budget. List the URLs first, count them and decide which sections to read. Finding all URLs on a domain goes through the listing step in detail.
• A lead research feed. Read the pricing, about and careers pages of a list of companies into one folder, then ask an agent to summarise what each one sells.
• A change monitor. Crawl the same pages every week and compare the markdown, so you find out when a competitor changes a page.

What goes wrong when you run your own web scraping?

Five things showed up in the 2026-10-16 runs, and each one changes how you write the loop.

Why does an empty URL list still say success?

Because the endpoint reports that it ran, not that it found anything. The sitemap call for example.com returned success: true with urls: [], and a meta block of 2 sitemaps discovered, 0 fetched and 2 skipped. It billed one call, $0.002533, for an empty list (Verified 2026-10-16). Check urls.length before you pass the list to a crawler, and read sitemapsFetched to tell a site with no sitemap from a failed fetch.

Why can one crawled page be huge?

Because a listing page is long. The /tools page came back as 42,884 characters of markdown, mostly a catalogue of links, and the home page as 2,768. A page count does not predict the volume of text, so cap what you send to a model or a database by characters, not by pages. Running the same crawl settings on your own machine is covered in when to run your own crawler.

Why did the same page cost a call on a cache hit?

A cache hit is still a billed call. The /web/scrape/markdown run for example.com came back with cache_metadata status hit and an age_ms of 741,815, about twelve minutes, and billed one unit. The default maxAgeMs is one day. If you need a page as it is now, set maxAgeMs to 0, which the schema describes as always fetching fresh.

Why does a crawl bill pages and not requests?

The crawl endpoint charges one unit per page actually scraped, so failed and skipped pages are free, and the number to read is numSucceeded. Our two-page crawl returned numSucceeded 2 and billed 5,066 micro-dollars, which is $0.005066 (Verified 2026-10-16). The schema says the admission hold is the requested maxPages, with a default of 100, so a crawl you start without a maxPages reserves 100 pages of balance before it reads one. Set maxPages on every crawl.

What does a list cost, and how long does it take?

Every endpoint here is $0.002533 a unit (Verified 2026-10-16), so 100 pages is $0.2533 and 1,000 pages is $2.533. The two-page crawl took about 13 seconds on the wall clock and the single scrape about a second. The median run times in monid inspect are about 4.2 seconds for /web/scrape/markdown, 5.7 for /web/scrape/sitemap and 14.9 for /web/crawl, so run several calls at once when a list is long.

Which endpoint should I use for which job?

Prices Verified 2026-10-16 with monid inspect. Use sitemap when you want the URL list for one flat charge, markdown when you already have the URLs, and crawl when you want the site followed for you. A sitemap call costs the same as one crawled page, so listing URLs first is the cheap way to size a crawl of many pages.

When is another route a better fit?

Nobody will own the pipeline. A collection that stops quietly and has no owner is a risk whoever built it. If you cannot name the person who gets the alert, a service that is contractually responsible for delivery is the better fit.

The requirement keeps changing. New sites every month and new fields every quarter, driven by people who do not write code, is a job for a company that takes requirements and returns data.

The targets are defended and central to your business. These endpoints include rendering and proxies, but a handful of heavily protected sites you must have is the case a dedicated service exists for.

You need someone accountable. If a customer or a regulator will ask who stands behind the data, an invoice from a vendor is an answer and your own script is not.

The job is tiny. A weekly pull of a few pages does not need a contract, and a browser your agent drives or a one-off script does it without a key.

And the disclosure: we resell these endpoints and we do not sell a managed service, so we have a commercial interest in you deciding you do not need one. Read the cases above with that in mind, because they are the cases where we are the wrong answer.

Conclusion

The easiest way to get web data without hiring a scraping service is a few calls through Monid: list the URLs, crawl or scrape the pages, and pay $0.002533 per unit (Verified 2026-10-16). One key covers this and every other tool, setup takes about a minute and the tools are ready inside your agent whenever you need them. A service is the better answer when nobody can own the pipeline or the requirement changes faster than you can build.

Two things matter more than the choice. Read the summary fields, not the word success: an empty URL list and a cache hit both came back as successful runs. And set maxPages on every crawl, because the default reserves 100 pages of balance.

How we estimated the time saved: building your own fetch and crawl layer with rendering and retries is roughly 6 to 12 hours, and a URL discovery step plus a script that tells you when a site stops returning pages is another 2 to 8. That is 8 to 20 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid inspect -p context.dev -e /web/scrape/sitemap, then make one sitemap call on the site you were about to get quoted for and read how many URLs it has.

FAQ

Published August 2026.