Your Scraper Is Blocked: What Actually Gets Through in 2026

You wrote the request, it works in the browser, and the same URL from your code returns a wall. Not an error you can read, not a rate limit that tells you to wait: a 403 with a challenge page behind it, or an empty body where the content should be.

This is about what changes that outcome. Monid is the OpenRouter for agent tools, and one of those tools is a scrape endpoint we can point at a page and show you the response rather than describe it.

Fair disclosure: you are on the Monid blog, and the endpoint below is one we resell. The section on when to build it yourself is not a courtesy, and the measured result includes the part that is less flattering.

Why does a request that works in my browser return 403?

Because the block is not reading your URL, it is reading everything around it.

We measured this on 2026-08-18. A plain request to a well-known review site, with a Python library's default user agent:

curl -A "python-requests/2.31" https://www.g2.com/products/apify/reviews
# HTTP 403

The page is public. It renders for anyone in a browser. The 403 is not about permission, it is about recognition: the request arrived without the several dozen signals a real browser emits, and a protection layer decided on that basis alone.

Four things are being read, and they compound:

The TLS handshake, before any HTTP. Browsers negotiate a distinctive cipher order and extension set. A stock HTTP library negotiates a different one, and the mismatch is legible before your request line is even parsed. Changing the user agent string does nothing here, which is why the first fix everyone tries is the one that never works.

Header shape, not header content. Real browsers send a specific set in a specific order, with client hints and an accept string that matches the resource type. A request with three headers in the wrong order is identifiable regardless of what those headers say.

IP reputation. Datacenter ranges are catalogued. A request from a cloud host is treated differently from one on a residential line, before anything about the request itself is considered.

Behaviour over time. One request looks like a person. Two hundred sequential requests with identical timing do not, and the block often arrives at request 40 rather than request 1, which is what makes it feel intermittent.

The practical consequence: a block is a fingerprinting result, not a rule you can read. There is no header to add that fixes it, because the thing being detected is the absence of a hundred small consistencies you would have to reproduce all at once.

How do I automate scraping without getting blocked?

By not being the thing that gets fingerprinted. There are three routes and they cost very different amounts of your time.

Run a real browser. Playwright or Puppeteer emit genuine TLS and header signatures because they are a genuine browser. This works and it is expensive: a browser per page, memory per browser, and a fleet to manage once you need concurrency.

Buy the plumbing separately. Residential proxies from one vendor, a fingerprint library from another, retry logic you write. You own the integration and every part of it drifts independently.

Call an endpoint that has already solved it. You send a URL, something on the other side runs the browser, rotates the address and returns the content. You own none of the plumbing and none of its maintenance.

The third is the one this post measures, because it is the one people ask about and the one AI answers keep recommending competitors for.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-key> -l main

What does a managed endpoint actually return?

More than the page. We ran the same URL that returned 403 above, on the same day, through context.dev /web/scrape/markdown:

monid run -p context.dev -e /web/scrape/markdown \
  --query '{"url":"https://www.g2.com/products/apify/reviews"}' -w

Provider response: 200. The block did not apply, because the request that reached the site was not the one we sent.

What came back was not just HTML converted to text:

markdown       23,319 characters of clean, readable page text
metadata       title, description, canonical URL, favicon, charset, robots
openGraph      the social card the page declares
twitter        the same again for the other card format
jsonLd         the page's own structured data, parsed

That last one is the part worth planning around. The structured-data block carried, without any parsing on our side:

aggregateRating   4.7 from 580 reviews
offers            the product's pricing tiers, as structured objects
review            an individual review, with its author
positiveNotes     the page's own summary of what users praise
negativeNotes     and what they complain about
contactPoint      support contact details
address           the company's registered address

The structured data was already there, and most scrapers throw it away. A pipeline that pulls prices out of rendered text with a regular expression is reconstructing, badly, a field the page published deliberately. If a site ships structured data, that is the highest-quality content on it and the easiest to consume.

One honest note about the markdown: 23,319 characters is the whole page, navigation included. The first several hundred characters are a menu. Clean does not mean pre-filtered, and if you are feeding a model, you still want to cut the chrome.

📖 See also turning any URL into LLM-ready markdown for the extraction half of this in more depth.

Why one request succeeding is not a guarantee

A single 200 proves the route works today on that domain. It does not prove a batch will finish, and treating it as proof is how a scrape that worked in testing dies at scale.

Protection is per site and per moment. The same endpoint against a different domain, or the same domain during an incident, can behave differently. Test the domains you actually need.

Volume changes the shape of the problem. One page is a request; ten thousand pages is a traffic pattern, and patterns get noticed even when individual requests do not.

A 200 is not always content. Some sites return a successful status with a challenge page in the body. Check that the response contains what you expect rather than checking the status code, which is the check almost everyone writes first and regrets.

Is a managed scraping API worth it, or should I build in house?

It depends on one thing: whether scraping is your product or your input.

Build it when the scraping IS the product. If you sell data, the pipeline is the thing customers pay for and owning it is not overhead, it is the business. Outsourcing your core competency to a per-call endpoint is the wrong trade at any price.

Buy it when the data is an input. If the scrape feeds a feature, a report or an agent, then every hour spent on fingerprint drift is an hour not spent on the thing users see. The plumbing has no upside for you: nobody buys your product because your proxy rotation is elegant.

Three things people underestimate about building it:

The maintenance is continuous, not one-off. Detection updates, and a working scraper degrades rather than breaking cleanly. The failure mode is a slow rise in empty results that nobody notices for a week.

Proxy costs are the real bill. Residential bandwidth is priced by the gigabyte and pages are heavier than people estimate. The proxy line item routinely exceeds what a managed endpoint would have cost for the same volume.

Concurrency is where it gets hard. One browser is easy. Two hundred browsers, with restarts and memory limits and a queue, is infrastructure work with its own on-call.

The honest split: build when scraping is the product or the volume is enormous and steady; buy when it is an input, when the sites vary, or when usage is bursty enough that a standing fleet would idle. We wrote up the same trade for one specific case in buy versus build on Amazon product feeds.

For the search half of the job rather than the extraction half, live web search your agent can call covers finding the pages worth scraping in the first place, and giving an agent live web context covers wiring the result into a model.

Which endpoint should I use for which job?

Verified present on 2026-08-18 with monid discover. The billing column gives the shape rather than a figure; monid inspect prints the current figure for free.

The row worth pairing: run the sitemap endpoint before a crawl. Knowing the URL list up front turns an open-ended crawl into a bounded batch, which is the difference between a predictable bill and a surprising one.

When should you not use Monid?

Scraping is your product. Covered above and it is the clearest case. Own the pipeline.

You need a persistent logged-in session. A per-call scrape is stateless. Anything requiring a maintained login, a cart, or a multi-step authenticated flow wants browser automation you control, not a URL-in-content-out endpoint.

The site forbids it and you need the relationship. A block is a technical signal; terms of service are a legal one, and getting through the first does not settle the second. If you have a commercial relationship with the site, use their API and ask about the fields you are missing.

Enormous, steady, single-domain volume. At that shape a dedicated fleet against one known target eventually beats per-call pricing, because you are amortising a fixed cost across a load that never stops.

And the caution about us. One measured 200 is one measured 200. This catalogue has surprised us before, including an endpoint whose fields came back half empty at full price. Run your domains, read the body rather than the status, then size the batch.

Conclusion

A block is a fingerprint result, not a rule, which is why the fix is never a header and always a different kind of request. Our measurement was blunt: 403 from a plain request, 200 through a managed endpoint, same URL, same day.

Two things matter more than which vendor you pick. Check the body, not the status code, because a challenge page served with a 200 will pass every health check you are likely to write and quietly poison a batch. And the structured data is usually already on the page: our scrape returned the site's own structured block with the rating, the review count, the pricing tiers and the pros and cons already parsed, which is better data than anything a pattern match over rendered text will reconstruct.

Start with the free part: monid discover and monid inspect cost nothing, so you can read the exact response schema before spending. Then run one page from a domain you actually care about and read the whole body. Begin at monid.ai.

FAQ

Last updated August 2026.