Is Web Scraping Legal? Your Options for Getting Page Data

Read a public web page as clean markdown with one prompt. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to read this public page as clean markdown, keep the title, canonical URL and publish date, and quote only the section I need.

One prompt reads a page. 1 min setup · 5-10 hrs saved.

You are about to point a program at someone else's website and want to know whether that is allowed. No verdict holds everywhere, because the rules differ by country, by what you collect and by what you do with it. This guide gives you the practical side: the routes to page data (public pages, official APIs, licensed data, a provider), what robots.txt does and does not say, and a call you can run in a minute. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to read a website through an API?

Monid. Install it once, add a key, and your agent calls a page reader that returns clean markdown, with no proxy pool, no browser to host and no vendor account to set up first.

• Pay per call. One page read is billed at $0.002533 (Verified 2026-10-16 on the web extraction tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and balance call page readers, search, company records and everything else on Monid. When the job changes from "read this page" to "find the pages first", you do not open a new account.
• Lives inside your agent. Your agent reads SKILL.md, learns the discover, inspect and run steps itself, and the tools are there whenever it needs them. Setup is about a minute.

Is web scraping legal?

There is no single answer, because "web scraping" names an act, not a legal category. Whether a given scrape is allowed depends on where you are, where the site and its users are, what the pages contain, what you do with the data, and what the site's owner has said about access. Two people running the same code against the same page can be in different positions.

What that means in practice is a short list of questions that you answer about your own project before you build it:

• What is on the page? Product prices and public documentation are one kind of material. Personal data about identifiable people is another, and many countries have specific rules for it.
• Who is allowed to see it? A page anyone can open is different from one behind a login, which usually comes with terms you agreed to.
• What will you do with it? Counting items, storing a quote with a link back, and republishing full text under your own name are different acts.
• How do you get it? An official API or a licensed feed comes with terms written for this use. A scraper comes with whatever the site's owner and your local rules say.
• Where are you, and where is the site? The law that applies to you may not be the law of the country where the server sits.

Is web scraping legal in my country, and for this site?

The honest answer to both is "check yours". People type this question with the US, UK, Germany, India, Canada, Singapore, South Africa and Europe added to it, and a question that has to be asked separately for each country has different answers in different countries. For your country, the steps are:

1. Name your use. Write one sentence: what you collect, from which kinds of sites, for what product. "Public prices from retail sites for an internal dashboard" and "profiles of people for a sales database" are different projects.
2. Look up the rules that apply to that use where you operate and where the people in the data live. Personal data, contracts, and rights in content and databases are the usual areas. A national regulator's site or a local lawyer is the right source; a blog post, including this one, is not.
3. Keep a record of what you decided and why. A short note with the date, the sites, the fields and the purpose is useful if a site owner ever writes to you.

For a specific site, start with the site's own pages. Many sites publish terms of use, a developer page or an API. Those documents describe what that owner offers and expects, and they are the only current source, so we do not summarise any site's terms here. When the answers touch personal data, a login or a product built on someone else's content, take them to a lawyer in your country. This is a general framing, not legal advice.

Does robots.txt make scraping legal or illegal?

Neither. robots.txt is a convention for telling crawlers which parts of a site the owner would rather they skip, and its own specification says it is not a permission system. We read the specification today (RFC 9309, the Robots Exclusion Protocol, published September 2022) through the page reader this guide is about. Its introduction says the rules are what crawlers "are requested to honor", and then states: "These rules are not a form of access authorization."

The same document is plain about what the file cannot do. Its security section says the protocol is not a substitute for valid content security measures, and that listing paths in robots.txt makes them publicly discoverable. So a site that wants to restrict access has to use real access control, and a site that lists a path in robots.txt has not made it private.

The practical reading for a builder: a disallow rule is a clear statement of what the owner wants, and ignoring it is a choice you should be able to explain to the owner and to your lawyer. A missing file is not a grant either. The specification says a crawler may access resources when the file is unavailable (a 4xx response) and must assume a complete disallow when the server errors (5xx). How much legal weight the file carries differs by country, and neither the specification nor this guide settles that. If you run your own crawler, read the file first and follow it.

How do you get web page data in a way you can defend?

Use the route with the clearest terms for the data you need. There are four, in the order most teams should try them.

Route 1. An official API

If the site publishes an API, use it. The terms are written for programmatic use, the fields are documented, and the owner has chosen what to expose. Check for a developer page before you build anything, and look at the price, because "there is no API" is often "there is an API and it has a price".

Route 2. Licensed or purchased data

For company records, people data, prices or reviews that someone already compiles, buying the record can remove the access question. Dataset vendors often price by the seat or by the year, and per-call endpoints price by the record. B2B data providers: buy the dataset or call the API? walks through when each fits.

Route 3. Public pages, read at a modest pace

For pages anyone can open without an account, a page reader fetches what a visitor would see and turns it into markdown. Read the owner's terms and robots.txt first, keep your request rate low, and store what you need. Here is the call:

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Read one public page

monid run -p context.dev -e /web/scrape/markdown -w \
  --query '{"url": "https://example.com/pricing", "useMainContentOnly": true}'

example.com has no /pricing page, so this exact command returns a not-found error (more on that below). Swap in a real page you are allowed to read and it returns the markdown and the page metadata.

Walk a small public site

monid run -p context.dev -e /web/crawl \
  -i '{"url": "https://example.com/docs", "maxPages": 20, "maxDepth": 2, "useMainContentOnly": true}'

Set maxPages yourself: the endpoint holds funds for the cap you request and bills only pages actually scraped. We inspected this endpoint today and did not run it.

Route 4. A provider that handles collection for you

When you need pages at volume, a provider runs the browsers, rendering and retries so your agent does not. You still decide which pages and what to do with them, and the provider's terms apply on top of the site's. That is the route Monid covers.

📖 See also Web scraping tools: which kind do you actually need?

Can you scrape websites yourself with Python?

Yes, and it is a normal way to start: request the page, parse it, store the fields. It stops being short when pages render in JavaScript, when the site blocks repeated requests, and when the markup changes and your parser silently returns nothing.

Building it yourself means a browser or proxy setup, retries and rate limits tuned per site, and selectors you repair when the page changes. The rules side (reading terms, honouring robots.txt, keeping your notes) is your job whoever fetches the page. What Monid maintains for you is the collection: JavaScript rendering, blocking and proxy handling are included in the per-call price, and the output is markdown plus typed metadata instead of HTML you parse. Pick Python for one stable page on a schedule, and the call for anything across many sites, because the maintenance is what grows. Python web scraping without a scraper shows where the DIY route stops paying.

What does a scraped page contain, and what goes wrong?

You get the page as markdown plus a metadata record, and four things trip people up. We ran two calls on 2026-10-16 against the reader above, so these are observed fields, not a spec.

What comes back

The second call read the Robots Exclusion Protocol specification page. The response held success, the markdown (33,964 characters), the final url, a metadata object, cache_metadata, finalDOMState and a request_id. The metadata carried the source and final URLs, the title, a description, the language (en), the author, alternate links, a headings list with a level and text for each of 31 headings, and extra meta tags.

{ "success": true,
  "contentLength": 33964,
  "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
  "metadata": { "title": "RFC 9309: Robots Exclusion Protocol", "language": "en",
                "headings": [ { "level": 1, "text": "RFC 9309" } ] },
  "cache_metadata": { "status": "miss", "age_ms": 0 },
  "finalDOMState": "loaded" }

That shape suits a citation-first pipeline: keep the URL, the title, the date you fetched it and the passage you need, rather than every page body you downloaded.

A wrong URL returns an error, and it was not billed

The first call used the command from the previous section and returned a NOT_FOUND error: the target page was a 404. The run was marked completed, billedUnits was 0 and nothing was billed. Check providerResponse.httpStatus before you read output, because a completed run can carry an error and a null output.

Cache and crawl caps

maxAgeMs defaults to one day (86,400,000 milliseconds), so set it to 0 when you need the page as it is now. Our second call reported cache_metadata.status as miss, so it was fetched fresh. The crawl endpoint sizes its hold by maxPages, which defaults to 100, so a five-page job can hold a hundred pages' worth of balance while it runs. Set maxPages to what you want.

📖 See also Your scraper is blocked: what actually gets through in 2026

Which endpoint should I use for which job?

Prices, all verified with monid inspect on 2026-10-16: the page reader is $0.002533 a call (Verified 2026-10-16), the crawl is $0.002533 per page scraped (Verified 2026-10-16), and the company record is $0.10 a call (Verified 2026-10-16). Search is billed per result. We ran only the page reader; the other three were inspected. Ten page reads cost $0.02533 (Verified 2026-10-16).

The last row asks whether you need to scrape at all: a good deal of what teams collect from pages is company information that a provider already compiles and licenses, and the record comes with the provider's terms. Current prices for all of them are on monid.ai/tools.

When is another route a better fit?

The site sells the data or offers an API. If a licensed feed or an official API exists and the objection is price, that is a commercial decision, and a scraper does not make it go away. Use the official route.

The pages hold personal data about people. Rules on personal data are specific and differ by country. Talk to a lawyer before you build, and consider a licensed source whose terms already cover the use.

The pages are behind a login. An account usually carries terms you agreed to. A page reader is built for public pages; do not hand it credentials to get past a login.

You need volume across thousands of pages on a tight schedule. Calls add up: ten thousand page reads cost $25.33 at the page-read price (Verified 2026-10-16), so price the job before you start.

One disclosure: we sell these endpoints, which is why this guide gives you questions to answer and a source to read instead of a verdict. Nothing here is legal advice.

Conclusion

The direct answer to "is web scraping legal" is that it depends on your country, your data and your use, so write down what you collect and why, and check the rules that apply to you. For the how-to side, the quickest way to read a public page is one call through Monid: $0.002533 a page (Verified 2026-10-16), one key for the reader and everything else, and ready inside your agent in about a minute.

Two things matter more than the tool. Choose the route with the clearest terms for your data: an official API, then licensed data, then public pages at a modest pace, then a provider for volume. And keep a short record of what you fetched and why, storing citations rather than corpora.

How we estimated the time saved: building and maintaining your own page reader with rendering, blocking handling and markdown conversion is roughly 5 to 10 hours for one reliable version, counting setup and the first month of fixes. The 1 minute is the install, key and first call once you already have a Monid key. Free next step: run monid inspect -p context.dev -e /web/scrape/markdown, then read one public page and print its title and headings. Start at monid.ai.

FAQ

Published August 2026.