Web Scraping Tools: Which Kind to Use and How to Start

Turn any web page into clean markdown and a table, in one prompt. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to scrape https://news.ycombinator.com into markdown and give me a table of the top 10 stories with title, link, points and number of comments.

One prompt scrapes a page and shapes the result. 1 min setup · 8-20 hrs saved.

Search for web scraping tools and you get ranked lists that put a Chrome extension, a Python library, a hosted scraper and an API side by side as though you were choosing between them. You are not. They differ on who operates them, and the operator decides who gets paged when a site changes its markup. This guide sorts the category by operator, shows how to run the kind your agent can call by itself (we ran it live on 2026-10-16), and lists the traps we hit. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest web scraping tool to start with?

Monid. Install it once, add a key, and your agent finds a scraping endpoint, reads its schema for free and runs it, with no vendor account per site and no scraper to host.

• Pay per call. A page scraped to markdown through Context.dev is billed on its own, at $0.002533 per call (Verified 2026-10-16 on the scrapers tools page). You buy calls as you need them, with no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills, and a failed or blocked request is not billed.
• One install, every tool. The same key and the same balance call the scraper and every other tool on Monid, including a free page fetcher and a site crawler. A second source later is another call on the same balance.
• Runs inside your agent. Your agent reads SKILL.md, learns the discover, inspect, run loop and calls the tool whenever the task needs a page. Nothing stays running between jobs, and setup takes about a minute.

At that price, scraping 100 pages costs about $0.25 (Verified 2026-10-16).

What kinds of web scraping tools are there?

Four, and the difference between them is who operates the tool, not what it can extract. A web scraper tool is anything that turns a web page into structured data, which covers a browser extension a marketer clicks and a crawler an engineering team runs.

The extension, operated by a person

A browser plugin where you point at the fields you want and it builds a table. Nothing runs when your laptop is closed, and it is the quickest way to get a few hundred rows from a site you are looking at right now.

The library, operated by a developer

Scrapy, BeautifulSoup, Playwright, Selenium and their equivalents in other languages. You get full control and no vendor, and the whole running cost lands on your team: fetching, rendering, retries and the morning the selectors break.

The hosted scraper, operated by a vendor

A prebuilt scraper for one specific site, run on someone else's infrastructure and sold per run or per result. Apify's store is the largest example. The vendor maintains the parser, which is what you want for a popular site they cover, and you get nothing for a site they do not.

The endpoint, operated at run time by whatever is calling

A capability behind a URL that anything can call, including an agent that did not know it existed a moment ago. It is a separate kind because of discovery: the caller searches the catalog, reads the schema and decides, instead of a developer wiring one integration in advance.

The last row of "chosen by" is the only one that changes at run time, and that one difference is why the fourth kind matters to anyone building agents. A scraper chosen at build time cannot cover a site nobody listed in advance. If you would rather run the crawler yourself, when to run your own crawler lays out the trade.

How do you use a web scraper tool to turn a list of URLs into LLM-ready markdown?

Find a tool that covers your page, run it once on a small input, then decide what repeats on a schedule. Below is that loop for the endpoint kind.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Step 1. Search by capability, not by vendor

Discovery returns endpoints that do the job, ranked, with the provider, the price and a status attached. It is free.

monid discover -q "scrape any website url to markdown"
monid discover -q "instagram profile posts"

On 2026-10-16 the first search ranked a Strale URL-to-markdown endpoint first and Context.dev's scraper second; the second ranked two Apify Instagram scrapers and an AnyAPI posts endpoint at the top. Two searches like these show which targets have a purpose-built tool.

Step 2. Read the schema before you commit

Inspection shows the exact inputs, the billing shape and the current price, and it is free.

monid inspect -p context.dev -e /web/scrape/markdown

For this endpoint the schema lists one required field, url, and options such as useMainContentOnly, maxAgeMs, waitForMs, country and includeLinks. Its note says JavaScript rendering, anti-bot bypass and premium proxies are included, and failed or blocked requests are not billed.

Step 3. Run one page, then decide

This is the only step that bills.

monid run -p context.dev -e /web/scrape/markdown \
  --query '{"url":"https://example.com","useMainContentOnly":true}' -w

It returned success, markdown, contentLength, url, a metadata object and a cache_metadata object. The call cost $0.002533 (Verified 2026-10-16).

Loop it over your list

Put one URL per line in urls.txt and write one markdown file per page:

mkdir -p out
n=0
while read -r u; do
  n=$((n+1))
  monid run -p context.dev -e /web/scrape/markdown --query "{\"url\":\"$u\",\"useMainContentOnly\":true}" -w > "out/page-$n.json"
done < urls.txt

Each URL is one billed call, and each file holds LLM-ready markdown, ready to feed to an LLM, a file index or a retrieval pipeline. A repeat run on the same URL may return a cached copy unless you set maxAgeMs to 0. Any URL to LLM-ready markdown walks through the same markdown file per URL.

Can you scrape with free Python tools instead?

Yes, and for one site that you will watch yourself it is a fair choice. You fetch the page with requests, parse it with BeautifulSoup, and reach for Playwright or Selenium when the page builds itself in the browser. Scrapy adds a crawler with queues and pipelines. All of these are open source.

What they do not give you is the operations. A fetch layer that survives blocking, a browser for pages that render late, retries, a place to run the job on a schedule, and a repair every time the markup changes. The first working version takes hours. Python web scraping without maintaining a scraper covers that cost in detail.

The Monid route is the choice for anything you will run more than once. The rendering, the proxies and the conversion to markdown are handled by the provider, the output is typed JSON instead of HTML you parse, and a page is $0.002533 (Verified 2026-10-16). Free online scraper sites and extensions are quicker for one page, but they are built for a person, not for a list or an agent.

What does scraped data look like, and what is it used for?

A scraper tool returns the page as text plus a record about the page. We ran two pages through the Context.dev scraper on 2026-10-16, example.com and news.ycombinator.com, and the fields below are what came back.

The response

{ "success": true,
  "markdown": "This domain is for use in documentation examples ...",
  "contentLength": 1342,
  "url": "https://example.com",
  "metadata": { "sourceUrl": "https://example.com", "finalUrl": "https://example.com/",
                "title": "Example Domain", "language": "en" },
  "cache_metadata": { "status": "hit", "age_ms": 8857248 },
  "finalDOMState": "loaded",
  "request_id": "01c7d403-..." }

markdown is the page, metadata describes it, and cache_metadata says whether the copy was fresh. On Hacker News the markdown came back as a table with one row per story: rank, title, link, points, submitter, age and comment count, 30 stories in all, contentLength 14,739, and the RSS address in metadata.alternates. Titles, links and counts are in the text, so an agent can turn them into rows without a parser. Deciding which fields of such a response to keep is the same job as in what you can keep from a search API.

The free TinyFish /fetch endpoint returns a different shape: url, final_url, title, language, text, latency_ms and format, plus description, author and published_date, which came back null for example.com.

What people use scraping for

• Feeding a model. Research agents and retrieval pipelines need page text, not a DOM. Markdown in one call replaces a fetch, a parse and a cleanup.
• Monitoring on a schedule. Prices, rankings, job postings and stock levels. The volume is small and the job is to keep working when nobody is watching.
• One-off lists. Competitor prices for a deck or a directory of suppliers. This is the extension's territory.
• Filling a database. Lead lists and product catalogues. If the record already exists behind a data endpoint, calling that is better than visiting the page at all.

What goes wrong when you scrape with a tool?

Four things showed up in the 2026-10-16 runs.

Why does the cleanup option leave the navigation in?

useMainContentOnly is documented as dropping headers, footers, sidebars and navigation "when detectable". On Hacker News it did not detect them: the markdown began with the site's navigation row (new, past, comments, ask, show, jobs, submit, login) and ended with the footer links. On example.com there was nothing to strip. Treat the flag as a hint and trim the first and last blocks yourself.

Why is the page I get not the page that exists now?

Because results are cached. The first example.com call returned cache_metadata with status hit and an age of 8,857,248 milliseconds, about two and a half hours. The maxAgeMs parameter defaults to one day. The Hacker News call set maxAgeMs to 0 and came back with status miss and an age of 0. For anything that changes by the hour, pass maxAgeMs: 0; live web data for agents shows where freshness matters.

What happens when one URL in a batch fails?

On the free TinyFish /fetch endpoint the batch still succeeds. We sent two URLs, example.com and a page that does not exist, and the call returned HTTP 200 with one entry in results and one in errors: the missing page with the code page_not_found and status 404. Check errors on every batch, because a call that returned 200 can still be missing URLs. The endpoint's note adds that it has no anti-bot bypass, so protected pages fail with bot_blocked.

What does a list cost, and how long does it take?

A paid scrape is one billed unit per page. The two calls we made billed $0.002533 each, $0.005066 in total (Verified 2026-10-16). The endpoint's median run time is about 1.9 seconds and its 95th percentile about 21.4 seconds, per monid inspect; our Hacker News call took about 3.4 seconds. A hundred pages is about $0.25 and, one after another, a few minutes.

Which endpoint should I use for which job?

The crawl endpoint holds the price of maxPages while it runs, 100 by default, so set maxPages to the number you need before you start. We inspected the crawl and search endpoints and ran the scrape and fetch endpoints. Site-specific scrapers, such as the Instagram ones discovery listed, price per result, and monid inspect shows the figure before you spend anything.

When is another route a better fit?

You need a few hundred rows once, and you are looking at the site. A point-and-click extension gets that in twenty minutes with no key and no code review. The route through an agent only pays off once the job repeats or the person asking cannot sit there.

The site has an official API. A documented interface from the source is allowed, stable and shaped for the data. If one site dominates your usage and it has an API, use the API.

You run a very large, steady crawl. At the top of the volume curve, a committed rate with a vendor, or your own pool of browsers, can fit better than billing per call. That needs engineering time to operate.

And the disclosure: we resell these endpoints. The most useful thing in this post is the habit that works whoever you buy from: run a page small, read the whole response, and check the cache and error fields before you scale the list.

Conclusion

There is no single best web scraping tool, because the four kinds are not substitutes and a ranked list that mixes them compares a hammer to a contractor. Decide who operates the tool first: you now, your developers, a vendor's team, or the caller at run time. When the caller is an agent, the quickest start is one scrape call through Monid: $0.002533 a page (Verified 2026-10-16), one key for this and every other tool, and about a minute to set up.

Two things matter more than the tool. Read the whole response, because the cleanup flag left the navigation in on the one real page we tried, and the cache can hand back a copy that is hours old. And check the error list on every batch, because a 200 does not mean every URL came back.

How we estimated the time saved: a first working scraper for one kind of site (fetch, rendering, cleanup, tests) is roughly 4 to 10 hours. For two kinds of site that is 8 to 20 hours. The 1 minute is the install, key and first call once you already have a key.

Free next step: run monid discover -q "scrape any website url to markdown", inspect the top result, then make one call on a page you know well and compare the markdown with what you see.

FAQ

Published August 2026.