Blog/Reading the web, and agents
5 min read

Web Scraping MCP Server: Live Web Data for Your Agent

Connect a web scraping MCP server to Claude Code, Codex or ChatGPT and let your agent search, scrape and read live pages, billed per call.

Web Scraping MCP Server: Live Web Data for Your Agent

Give your agent live web search and page scraping through one MCP connection. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to search the web for web scraping MCP servers, read the top three pages as Markdown, and give me a table with name, what it does and the link.

One prompt gives an agent search and page reading. 1 min setup · 20-40 hrs saved.

You want your agent to look something up on the web instead of guessing, and you have found that "MCP server" is the way to wire a tool into Claude Code, Codex or ChatGPT. This guide covers the quickest connection, the exact commands, what a live search and a live page read return (we ran both on 2026-10-16), and the choice between a server that wraps one vendor and one that lets the agent search a catalogue. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to give an agent live web data through an MCP server?

Monid. It is a remote MCP server at one URL, so there is nothing to install or host, and the agent gets web search, page-to-Markdown scraping and every other tool in the catalogue behind that one connection.

  • Pay per call. A page read is billed at $0.002533 and a web search at $0.000253 per result (Verified 2026-10-16 on the AI tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
  • One connection, every tool. The same URL and the same balance reach search, scraping, company data and social data. When the agent needs a source nobody wired in, it searches the catalogue instead of reporting that it cannot.
  • Lives inside your agent. Connect the URL, or let the agent read SKILL.md, and the tools are there whenever it needs them, with nothing to keep running. Setup is about a minute.

The agent gets three verbs: search the catalogue, inspect an endpoint, run it. The first two cost nothing, so the agent can read what a call will cost before it spends anything.

How do you connect the server to Claude Code, Codex or ChatGPT?

Add one URL. The server speaks streamable HTTP, so there is no package to install:

https://mcp.monid.ai/v1

In Claude.ai that is Settings, Connectors, Add custom connector, then Connect and authorise. The terminal clients are one line each:

claude mcp add --transport http monid https://mcp.monid.ai/v1
codex mcp add monid --url https://mcp.monid.ai/v1

OpenCode takes a block in opencode.json and then opencode mcp auth monid; ChatGPT adds it as a plugin with the same URL. Full steps for each are in the MCP quickstart.

For agents

If your client prefers a skill file to a connector, paste this and hand it your key from app.monid.ai:

set up https://monid.ai/SKILL.md

It learns the discover, inspect, run workflow itself. More in the agent quickstart.

For humans

There is a CLI too:

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Payment is a prepaid balance by default, or per run with USDC over x402 if you would rather not hold one.

The three verbs an agent gets

monid discover -q "instagram profile"    # search the catalogue, free
monid inspect -p tikhub -e /api/v1/instagram/v1/fetch_user_about_info    # schema and price, free
monid run -p tikhub -e /api/v1/instagram/v1/fetch_user_about_info --query '{"user_id":"182988865"}'   # the only paid step

Two of the three cost nothing, which changes how an agent behaves. It can survey what exists and read what a call will cost before spending, so "is this available and what will it run me" stops being a question you answer for it in advance.

Read one page as Markdown

The scrape endpoint is a GET with the URL as a query parameter:

monid run -p context.dev -e /web/scrape/markdown \
  --query '{"url":"https://www.python.org/","useMainContentOnly":true,"maxAgeMs":0}'

useMainContentOnly drops navigation and footers. maxAgeMs of 0 forces a fresh fetch instead of a cached copy.

Can you run a web scraping MCP server yourself?

Yes, and for a single known job it is a reasonable thing to do. Our search for web scraping MCP servers on 2026-10-16 returned a long list of them: the official Firecrawl server on GitHub, ScrapingAnt's hosted endpoint, Bright Data's guide, Import.io, Spider, ZenRows and a directory page titled "632 Web Scraping MCP Servers". Some are free and open source, some wrap a paid vendor.

Running one yourself means choosing it, installing it, giving it a key or a browser, and keeping it alive. The extra weight comes from the pages themselves: JavaScript rendering, blocking and proxies are the parts that break. The scrape endpoint above lists them as handled server-side and included in the price, and says failed or blocked requests are not billed. That is the trade: with your own server you own the repairs, with a managed endpoint the provider does.

The same logic holds if you write the scraper rather than install a server. If you are weighing a vendor server first, what you are installing with the Bright Data MCP lists the trade-offs of that route. Pick Monid when the agent will read pages it did not know about in advance, or when you would otherwise create one account per source.

What does a live web data call return?

Two kinds of record: a search result list and a page. We ran both on 2026-10-16, so the fields below are what came back.

The page

We read python.org with useMainContentOnly on and the cache off. The call returned HTTP 200, billed one unit, took about five seconds end to end, and produced 4,869 characters of Markdown. The news list inside it was dated up to 2026-10-13, three days before the run, which is the point of a live read: the agent sees today's page, not training data.

Beside the Markdown came the page metadata: title, description, language, sourceUrl, finalUrl, siteName, image, openGraph, alternates, a headings array with level and text, jsonLd, and a cache_metadata block reading miss with an age of 0. That block tells you the page was fetched fresh, not served from the one-day default cache.

A web search for "model context protocol server for web scraping" with ten results returned HTTP 200 in about a second and a half. Each row had url, title, description and a relevance label (all ten read high). Two more fields, markdown and highlights, came back empty with the code NOT_REQUESTED: page text is opt-in through markdownOptions, which keeps a plain search cheap.

The same question through Exa returned ten rows in about four seconds, each with id, title and url, and a publishedDate on one of the ten. There was no description or page text. It suits a list of candidates to hand to the scrape endpoint, not an answer to read directly.

What you can build with it

  • A research step that cites live pages. Search, take the top three URLs, read each as Markdown, and answer from them. The pattern of search, then read, is the one in give your agent live web context.
  • A monitor for one page. Read the same URL on a schedule with maxAgeMs of 0 and diff the Markdown.
  • A source list for retrieval. Pull a set of pages into Markdown, then chunk them for your own index.

Should the server be one vendor or a catalogue?

It depends on whether you know in advance what the agent will need. A vendor server wraps one company's API: Firecrawl's server gives an agent Firecrawl, and the tool list is set when you connect. That is the right shape when one vendor covers the job and you already know it.

A catalogue server exposes a search over many providers. The agent does not receive a list of scrapers; it asks what exists, reads a schema, sees a price, and calls. The three runs above went to two different providers, Context.dev and Exa through BlockRun, on one connection and one balance. A third would have been a parameter change, not a new integration.

The practical difference appears the first time an agent needs something outside its wiring. With a fixed toolset it reports that it cannot do that. With a catalogue it looks, finds an endpoint or finds none, and says which. If you want the general case, MCP vs API for AI agents walks through it.

The same shape protects you when a source goes away. A single-vendor server couples your agent to one company's access; when that vendor loses a source, the agent stops. With a catalogue the agent runs discovery again and either finds another endpoint or reports that none exists. That only helps when a second endpoint exists for your source, and a catalogue is itself a dependency. What it buys is that swapping happens one layer up, as a changed provider name rather than a rebuilt integration.

What goes wrong when an agent pulls live web data?

Four things show up once the agent runs more than one call.

Why does the same query return different lists?

Because search engines are different products. Our two searches for the same sentence overlapped on only two results, the Firecrawl GitHub repository and Import.io's MCP page, out of ten each. Use one search endpoint for one purpose and do not merge lists without deduplicating on url.

const seen = new Set();
const rows = [...contextRows, ...exaRows].filter(r => !seen.has(r.url) && seen.add(r.url));

Why is a search billed per result and a scrape per call?

The billing shapes differ, and the agent should know which it is on. The search endpoint bills per result: ten results billed ten units, $0.00253 in total (Verified 2026-10-16). The scrape endpoint bills per page. Exa through BlockRun bills per call, $0.0121 for ten results (Verified 2026-10-16). Run an endpoint once small, read the charge, then loop. The free web search API for agents compares search sources on the same point.

Why did my page look old?

Because the scrape endpoint reuses a cached copy younger than one day by default. For a news page or a price, set maxAgeMs to 0 and read cache_metadata in the response to confirm miss.

Why is the search result empty of text?

Because page text is opt-in. The search returned markdown as null with NOT_REQUESTED. Enabling markdownOptions scrapes each result and bills extra credits, so when you already hold the URL, call the scrape endpoint directly: the endpoint's own hint says it is the cheaper route.

Which endpoint should I use for which job?

JobEndpointWhat it returnsBilling shape
Clean page text for a promptcontext.dev /web/scrape/markdownMarkdown plus page metadataPer call
Find pages worth readingcontext.dev /web/searchRanked rows with url, title, description, relevancePer result
Neural web searchblockrun.ai /api/v1/exa/searchRows with id, title, urlPer call
Every page of one sitecontext.dev /web/crawlOne Markdown document per pageListed on the tool page
List a site's URLs firstcontext.dev /web/scrape/sitemapThe site's URLsListed on the tool page

The scrape, search and Exa rows were inspected and run on 2026-10-16. The crawl and sitemap rows were returned by the scrape endpoint's own hints and were not run; finding all the URLs on a domain uses that pair. Current figures are on monid.ai/tools, and monid inspect prints them for free.

When is another route a better fit?

You need one source and you know which. If your agent only ever reads pages as Markdown, connect that vendor's own MCP server. One vendor, one bill, fewer moving parts. The catalogue argument is about not knowing in advance.

You need a contract and an SLA. Buying direct from a scraping vendor can come with an account manager. A catalogue is built for breadth and for switching, which is a different thing to want.

You need a logged-in or private page. A managed endpoint reads public pages. A browser tool driven from your own session reaches what you can see, with all the account risk that carries.

You only want Firecrawl. Then use its own server; Firecrawl with Claude over MCP, or one key sets the two routes side by side.

And the disclosure: we resell these endpoints. Run one small call before you loop and read the charge yourself, whoever you buy from.

Conclusion

The quickest way to give an agent live web data is one MCP connection to Monid: search at $0.000253 a result and a page read at $0.002533 (Verified 2026-10-16), one URL and one balance for everything else, and a setup measured in about a minute. Running a scraping server yourself works for one known source, and costs you the repairs.

Two things matter more than which server you pick. Whether the agent can see a price before it spends, because an agent that cannot has to be capped by guesswork. And whether it can ask what else exists, because the first time it needs a source nobody wired in is the moment a fixed toolset stops.

How we estimated the time saved: wiring a search tool, a page-to-Markdown tool and a third source into an agent yourself is roughly 7 to 13 hours each for accounts, keys, schemas and testing. That is 20 to 40 hours. The 1 minute is the connection and a first call once you have a Monid key.

Free next step: connect https://mcp.monid.ai/v1, ask your agent what exists for a source you care about, read the price, then make one small run.

FAQ

Does connecting an MCP server cost anything by itself?

No. Connecting is free, and so are searching the catalogue and reading an endpoint's schema and price. Only a run bills, at the price shown beforehand, against a pay-as-you-go balance. An agent that has to spend money to find out what things cost cannot budget.

Is there a free web search MCP server?

Free servers exist, and our 2026-10-16 search for web scraping MCP servers listed several open-source ones. Through Monid a search is not free but small: $0.000253 per result, so ten results cost $0.00253 (Verified 2026-10-16), with no monthly plan and no minimum top-up.

What is the difference between an MCP server and an API?

An API is an interface your code calls; an MCP server wraps tools so an agent can find and call them itself. The server still calls an API underneath. The practical difference is discovery: the agent reads the tool list instead of you writing the glue.

What should I use for the scraping layer in an n8n automation?

One HTTP node pointed at a catalogue, with the provider and endpoint as parameters, rather than a dedicated node per vendor. When a source changes you edit two strings instead of rebuilding a node.

Published August 2026.

web scraping mcp servermcp server live web dataweb search mcpbest mcp serversmcp server for ai agents