Scraper Blocked? Why You Get a 403 and What Gets Through

Get the text of a page your scraper keeps getting blocked on, in one prompt. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to fetch https://www.trustpilot.com/review/apify.com as markdown and tell me the rating, the number of reviews and the page title.

One prompt fetches a protected page. 1 min setup · 8-20 hrs saved.

Your request works in the browser, and the same URL from your code returns a 403 with a challenge page behind it, or an empty body where the content should be. This guide shows which kinds of sites do that to a plain request, what you can do about it, and what a managed scrape endpoint returned when we ran it on 2026-10-16, including the page it could not get. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to get past a blocked scraper?

Monid. Install it once, add a key, and your agent calls a scrape endpoint with a URL and gets the page back as markdown with its metadata, while the browser, the proxies and the retries run on the other side of the call.

• Pay per call. A page costs $0.002533 (Verified 2026-10-16 on the web extraction tools page). Buy calls as you need them: no monthly plan and no minimum top-up. A request the site blocks is not billed.
• One install, every tool. The same key and the same balance call the scraper, the sitemap lister and every other tool on Monid, so search, scrape and enrichment share one account.
• Lives inside your agent. Install the CLI, add your key, run the prompt above, and the tool is there whenever your agent needs a page, with nothing to keep running. Your agent reads SKILL.md and learns the rest, so setup takes about a minute.

At that price, scraping 100 pages costs about $0.25 (Verified 2026-10-16).

Why does a scraper that works in the browser return 403?

Because the site is judging the request, not the URL. We sent a plain request with a Python library's user agent to eleven public pages on 2026-10-16:

curl -s -o /dev/null -w "%{http_code}\n" -A "python-requests/2.31" https://www.trustpilot.com/review/apify.com
# 403

Nine of them answered 403: G2 (two pages), Glassdoor, Indeed, Crunchbase, Zillow, Capterra, Trustpilot and Etsy. Two answered 200 to the same request: nowsecure.nl and the Walmart home page. Every one of those pages opens in a browser, so the 403 is about recognition, not permission.

Changing the user agent did not help where we tried it: a desktop Chrome user agent on the G2 reviews page still returned 403.

The usual reason is that a protection layer reads more than the headers you set:

• The TLS handshake. Browsers negotiate a distinctive set of ciphers and extensions, and a stock HTTP library negotiates a different one before any HTTP is parsed. A user agent string does not change that.
• Header shape. Real browsers send a specific set of headers in a specific order, with client hints that match the resource type.
• IP reputation. Cloud and datacenter address ranges are treated differently from residential lines.
• Behaviour over time. One request looks like a person; hundreds of identical ones do not.

A single header will not fix it, because what is detected is the absence of many small consistencies at once. And a block is a per-site result: the same code that gets a 200 from one page gets a 403 from the next, so test the domains you actually need.

📖 See also datacenter versus residential proxies for what the address itself changes, and what it leaves alone.

How do you scrape without getting blocked?

By sending requests that look like a browser from an address that looks like a person. There are three routes, and they cost very different amounts of your time.

Run a real browser. Playwright or Puppeteer drive a genuine browser, so the handshake and headers are genuine. It works, and it is heavy: memory per browser and a fleet to manage once you need concurrency.

Buy the plumbing separately. Proxies from one vendor, a fingerprint library from another, retry logic you write. Each part drifts on its own.

Call an endpoint that already runs the browser. You send a URL, the other side renders the page, rotates the exit address and returns the content. The scrape endpoint on Monid is this route: its description lists JavaScript rendering, anti-bot bypass and residential proxies as handled server-side and included in the price.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Inspect, then run one

monid inspect -p context.dev -e /web/scrape/markdown
monid run -p context.dev -e /web/scrape/markdown \
  --query '{"url":"https://www.g2.com/products/apify/reviews"}' -w

That second command is the page that failed for us on 2026-10-16. The url must include the scheme; everything else in the query is optional.

Or hand the whole job to an agent

📖 See also Python web scraping without a scraper for the script that calls an endpoint instead of fetching the page itself.

What does a managed scrape endpoint return?

More than the page text. We ran /web/scrape/markdown on three of the pages that returned 403 to the plain request, with maxAgeMs set to 0 so that each was fetched fresh. All three came back with a 200.

Each response carries the markdown and a metadata object with these keys: sourceUrl, finalUrl, title, description, language, canonicalUrl, siteName, image, favicon, publishedTime, robots, openGraph, twitter, alternates, headings and additionalMeta, plus jsonLd. The Trustpilot title read "Apify is rated Excellent with 4.8 / 5 on Trustpilot", and the markdown showed 804 reviews.

The structured data is already on the page, and most scrapers throw it away. The Capterra response carried a SoftwareApplication block with an aggregateRating of 5 from 6 reviews, parsed, next to a FAQPage block with question and answer pairs. A pipeline that pulls a rating out of rendered text with a regular expression is rebuilding a field the page published on purpose. Read jsonLd first and fall back to the text.

Listing the URLs before you scrape them

/web/scrape/sitemap takes a domain and returns the URLs from its sitemaps in one call. We ran it on apify.com with maxLinks set to 20 and got 20 URLs back from six sitemaps with no errors, in about three seconds.

monid run -p context.dev -e /web/scrape/sitemap \
  --query '{"domain":"apify.com","maxLinks":20}' -w

It is billed once per call whatever the number of links, at $0.002533 (Verified 2026-10-16). Running it before a crawl turns an open-ended job into a bounded list. Finding all URLs on a domain goes through the options in more depth.

What goes wrong when the endpoint is blocked too?

A managed endpoint raises your chances, and it does not remove the block. The page we started with, the G2 reviews page, came back from the endpoint on 2026-10-16 with no content at all:

{ "status": "COMPLETED",
  "output": null,
  "providerResponse": { "httpStatus": 400,
    "error": { "error_code": "WEBSITE_BLOCKED",
               "message": "The site served an anti-bot challenge or access wall instead of the page ..." } },
  "billedUnits": 0 }

Five things follow from that run and the four that worked.

Why is a COMPLETED run not a successful page?

Because status: COMPLETED means the call ran, not that you got the page. On the blocked run output was null and the failure sat in providerResponse: HTTP 400 with the error code WEBSITE_BLOCKED. Check output.success and providerResponse.httpStatus, not the run status, before you treat a result as content. The same habit catches a 200 with a challenge page in the body: check that the response holds what you expect, such as a title or a rating.

Does a blocked request cost anything?

No. The blocked run billed 0 units and 0 micro-dollars, which matches the endpoint's notes that failed or blocked requests are not billed. The four successful runs billed $0.002533 each, or $0.010132 for the five calls we made in total (Verified 2026-10-16). The cost of a block is time: the blocked G2 run took about 54 seconds on the wall clock, against a median of about 4 seconds in monid inspect, so put a timeout and a retry budget around a loop.

What should you do with a block?

Retry later, change the country, or give up on that page. The error message says retrying later or from another country sometimes succeeds, and the endpoint takes a two-letter country parameter for the proxy exit. Treat a blocked page as a normal outcome in your code: log the URL, skip it and carry on.

Why did the response name a different product?

Because the request URL and the page you got can differ. We asked for Capterra product id 153495 under the slug Apify; the response had sourceUrl for that URL and finalUrl pointing to the YesInsights page, because the id belongs to that product and the site redirected to it. The data was correct for the final page and wrong for the page we thought we were reading. Compare sourceUrl with finalUrl and canonicalUrl before you store a record under the name you asked for.

Is the markdown clean?

It is complete, which is different from clean. All three pages began with navigation: Trustpilot with a suggested-companies strip and a menu, Etsy with a category menu, and Capterra with a banner and dozens of category links before the product name. The endpoint has a useMainContentOnly switch that drops headers, footers and sidebars when it can detect them, and it is worth setting when the text goes to a model. Results are cached for a day by default, so pass maxAgeMs set to 0 when you need today's page.

📖 See also a free way to extract page content for RAG for cutting the navigation off before the text reaches a model.

Should you build this yourself or buy it?

It depends on one thing: whether scraping is your product or your input.

Build it when the scraping is the product. If you sell the data, the pipeline is what your customers pay for, and owning it is the business. Handing your core skill to a per-call endpoint is the wrong trade at any price.

Buy it when the data feeds something else. If the scrape supports a feature, a report or an agent, every hour spent on fingerprints is an hour not spent on the thing people see.

Three things are easy to underestimate about building:

• The upkeep never ends. Detection changes, and a working scraper degrades rather than breaking cleanly, as a slow rise in empty results.
• Proxies are a separate bill. Many proxy vendors price residential traffic by bandwidth, and pages are heavier than people estimate.
• Concurrency is infrastructure. One browser is easy. Two hundred, with restarts, memory limits and a queue, is a system that needs someone on call.

Build when scraping is the product or the volume is large and steady against a few known targets; buy when it is an input, when the sites vary, or when usage comes in bursts that would leave a standing fleet idle. Crawl4AI and when to run your own crawler covers the build side.

Which endpoint should I use for which job?

We ran the first two rows on 2026-10-16. The other four we inspected and did not run. Crawl and search reserve a balance hold sized by the page cap or result count you request, and crawl bills only the pages it scraped, so set maxPages low on a first run. Parse bills more when OCR runs; a screenshot counts as five credits.

When is another route a better fit?

Scraping is your product. Own the pipeline.

You need a logged-in session. A per-call scrape is stateless. A maintained login, a cart or a multi-step flow wants a browser you control.

The page stays blocked. Our G2 run shows it: some pages still return a wall. If the site matters to you and the endpoint cannot get it, look for a data partnership or a provider that already holds that data, and see whether another Monid provider covers the site you need by running monid discover.

You have a relationship with the site. A block is a technical signal and terms of service are another matter, and getting through the first settles nothing about the second. If you have a commercial relationship with a site, ask for its API and the fields you are missing. Rules differ by place; check yours.

One page, once. A browser and the copy button need no key.

📖 See also browser automation without an API for the logged-in and multi-step cases that a URL-in endpoint does not cover.

And the disclosure: you are on the Monid blog, and Monid sells the endpoint above. The most useful result in this post is the one that makes the sale harder: a page that came back blocked.

Conclusion

The quickest way past a blocked scraper is to stop sending the request yourself and call an endpoint that renders the page: Monid does that for $0.002533 a page (Verified 2026-10-16), with one key for this and every other tool, about a minute to set up and ready inside your agent whenever you need it. It got us three of the four pages we tried that return 403 to a plain request, and it did not get the fourth.

Two things matter more than the endpoint you pick. Check the body, not the status: a run can complete with no content, and a 200 can carry a challenge page. And read the structured data first, because the page's own jsonLd block is usually better than anything you can pattern-match from the text.

How we estimated the time saved: running your own browser with proxies and retries is roughly 6 to 12 hours for the first working version, plus 2 to 8 hours in the first month as sites change their checks. That is 8 to 20 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid inspect -p context.dev -e /web/scrape/markdown, then make one call on a page you actually need and read the whole response, including the failures. Start at monid.ai.

FAQ

Published August 2026.