Web Scraping in Python Without Maintaining a Scraper

Copy this line to your agent to pull a page into clean text without writing a fetcher.

set up https://monid.ai/SKILL.md and use context.dev /web/scrape/markdown to read a page into markdown

Every Python scraping tutorial teaches the same twenty lines, and those twenty lines work on the sites nobody needed help with. The gap between a tutorial that parses a static page and a job that runs every morning against a site that does not want you is not a Python problem, and no amount of better Python closes it. This guide splits the task into the half Python is genuinely best at and the half it is the wrong place to solve, then shows the second half as one call through Monid, the OpenRouter for agent tools.

What is web scraping in Python?

Web scraping in Python is two separate jobs that share a name: getting the bytes, and turning the bytes into records. Python is excellent at the second and has no particular advantage at the first, and almost every difficulty people describe belongs to the first.

The parsing half, which Python owns

Once you hold the HTML, Python is close to unbeatable. BeautifulSoup makes a messy document navigable in a line, lxml is fast enough for anything, pandas turns a table into a dataframe, and the whole downstream stack for cleaning, joining and storing the result already exists in the same language. Nobody writes a blog post complaining about this half, which is why it fills the tutorials.

The fetching half, which is infrastructure

Getting the bytes means owning an exit address the site will accept, a browser or a close enough imitation of one, a retry policy, a proxy pool, and a plan for the week the markup changes. None of that is Python code. It is operations work that happens to be triggered from Python, and it is the part that turns a weekend script into a thing you maintain.

The honest way to see the split is to ask what breaks in production. It is never the selector logic. It is a 403 that appeared overnight, a page that now renders its content after a JavaScript call, or a cookie wall that was not there in April.

What "scraper" hides

The word bundles both halves, so a decision that should be made twice gets made once. Teams evaluate "should we build or buy a scraper" as a single question, decide building is fine because the parsing is easy, and inherit the fetching. Split the two and the answer is usually obvious: keep the parsing in Python, where you want the control, and stop operating the fetch.

The pattern is that the half people build is the half that was cheap, and the half they inherit is the one with the running cost.

📖 See also The Best Web Scraping API for AI Agents in 2026

How do you do web scraping in Python?

Do the fetch as a call and the parse as code. The three steps below are the working shape: find an endpoint that returns the page already rendered, call it from Python, then do everything after that in the language you chose Python for.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Step 1. Find an endpoint that already renders the page

What it does. Returns a URL as clean markdown or fully rendered HTML, with the browser, proxies and anti-bot handling on the far side.

The endpoints. context.dev/web/scrape/markdown for markdown, context.dev/web/scrape/html when you want to keep your selectors, tinyfish/fetch for a free batch of up to ten URLs.

The call.

monid discover -q "scrape any website url to markdown"
monid inspect -p context.dev -e /web/scrape/markdown

What comes back. The schema, the billing shape and the current price, plus the vendor's own note on what is included. For the markdown endpoint that note reads: JavaScript rendering, anti-bot bypass and premium proxies included, and failed or blocked requests are not billed. Verified 21 August 2026. Those last two clauses are the reason most Python scrapers never need a rotating proxy of their own.

What it costs. Nothing for these two commands. Discovery and inspection are free, so you can read the exact price of every candidate before you spend.

Step 2. Call it from Python

What it does. Puts the fetch behind one function, so the rest of your program never learns that scraping was involved.

The endpoints. context.dev/web/scrape/markdown.

The call.

import json
import subprocess

def fetch_markdown(url: str) -> dict:
    proc = subprocess.run(
        [
            "monid", "run",
            "-p", "context.dev",
            "-e", "/web/scrape/markdown",
            "--query", json.dumps({"url": url, "useMainContentOnly": True}),
            "-w", "-j",
        ],
        capture_output=True,
        text=True,
        check=True,
        env={"NO_COLOR": "1"},
    )
    return json.loads(proc.stdout)

page = fetch_markdown("https://example.com")
print(page["metadata"]["title"], page["contentLength"])

What comes back. success, markdown, contentLength, url, and a metadata object carrying sourceUrl, finalUrl, title and language. Verified against a live run on 21 August 2026.

What it costs. A fraction of a cent per page, billed per call, and only on pages that came back. Current figures on monid.ai/tools.

Step 3. Parse in Python, which is the part you kept

What it does. Turns the returned document into records. This is where BeautifulSoup, lxml, pandas and your own judgement belong, and none of it changes because the fetch moved.

The endpoints. None. This step is your code.

The call.

import pandas as pd

rows = []
for url in urls:
    page = fetch_markdown(url)
    if not page.get("success"):
        continue
    rows.append({
        "url": page["metadata"]["finalUrl"],
        "title": page["metadata"]["title"],
        "language": page["metadata"].get("language"),
        "text": page["markdown"],
    })

pd.DataFrame(rows).to_parquet("pages.parquet")

What comes back. A dataframe, and a program with no proxy configuration in it.

What it costs. Your time, which is the resource this whole exercise was trying to protect.

📖 See also Any URL to LLM-Ready Markdown

How do you scrape a page BeautifulSoup cannot read?

You cannot, and that is the correct answer rather than a defeat. BeautifulSoup is a parser: it reads a document you already have. If requests returned a shell with no content in it, there is nothing for BeautifulSoup to find, and every hour spent on selectors is spent on the wrong layer.

Tell the two failures apart first

Print what you actually received before changing anything. If the HTML contains your data and your selector missed it, that is a parsing bug and you are ten minutes from fixing it. If the HTML is a skeleton, a challenge page, or a login form, the fetch failed and the parser is irrelevant. People lose days by skipping this check, because a wrong selector and an empty page produce the same empty list.

Selenium is a fetch fix, and an expensive one

Reaching for Selenium or Playwright is the standard next move and it does work, because it solves a fetch problem with a real browser. What it also does is move a browser into your process: a driver to keep in step with Chrome, memory per session, a stealth patch set that goes stale, and a pool to run more than one page at a time. That is a reasonable thing to own when browser control is your product. It is a lot to own when you wanted the text off a page.

The endpoint version of the same fix is a query parameter. context.dev/web/scrape/html returns the fully rendered HTML, so your existing BeautifulSoup code keeps working with no driver anywhere in your repo, and waitForMs covers the pages that populate late.

When the page needs a session, not a fetch

Logins, carts, multi-step forms and anything behind a wizard need state carried across requests, which no single-shot fetch endpoint provides. That is a genuine browser job, and the way to do it without hosting the browser is a remote session: x402.browserbase.com/browser/session/create returns a connectUrl you attach Playwright to over CDP, so the driver runs somewhere else and your code stays a client.

📖 See also Your Scraper Is Blocked: What Actually Gets Through in 2026

Python vs JavaScript vs Go for scraping: does the language matter?

Less than the search volume on that question suggests, and the reason is the split above. All three languages parse HTML competently and none of them changes whether a site serves you. Pick the one your team already writes and spend the argument budget elsewhere.

Where the languages genuinely differ is concurrency shape. Go gives you cheap parallel fetches with no ceremony, which matters if you are running your own fetch layer at volume. Node sits closest to the browser automation ecosystem, so Playwright and Puppeteer feel native there. Python has the best analysis stack downstream, which is usually why the data was being collected in the first place.

Notice that two of those three advantages evaporate if the fetch is an endpoint. Concurrency becomes the provider's problem, browser automation happens on their side, and what is left is the analysis, which is Python's home ground. The language question is really a proxy for "who runs the fetch", and answering the real question makes the proxy one uninteresting.

📖 See also Apify Alternatives: You Probably Want a Different Way to Buy It

Which endpoint should I use for which job?

Every row verified with monid inspect on 21 August 2026. The table gives the billing shape rather than a figure, because the shape is what changes your code and a number ages badly.

When should you just write the scraper?

When the site is static, public and small. A council page, a docs site, a table of reference data: requests plus BeautifulSoup in fifteen lines is the right answer, it has no running cost, and reaching for a service would be silly. Most of what people scrape is this, and most of it never appears in a blog post because it works.

Write it yourself when the extraction logic is the value. If your product is a parser that understands one site's quirks better than anyone else's, that logic is your moat and it belongs in your repo. Buying the fetch does not conflict with this; it is exactly the split this guide argues for.

And go direct to one vendor when a single site dominates your usage. If ninety percent of your calls hit one platform, its official API or a committed plan with a specialist will beat per-call pricing on a catalog. A catalog is worth it when the list of sites is long, changes, or is not known until run time.

Conclusion

Python web scraping is not one skill, and treating it as one is why the tutorials feel useless in production. Parsing is a Python problem with a good Python answer. Fetching is an infrastructure problem that Python cannot improve, and the cost of pretending otherwise is a browser pool, a proxy bill and a maintenance rota nobody signed up for.

The check that saves the most time is the cheapest one: print the HTML you received before you touch a selector. It tells you which of the two problems you actually have, and half the questions in this category are people debugging the wrong layer.

Free next step: run monid discover -q "scrape any website url to markdown" and monid inspect the top row. Both are free, and the schema will tell you in under a minute whether the fetch you were about to build already exists. Start at monid.ai.

FAQ

Last updated August 2026.