Web Scraping News Articles: Three Jobs, Three Endpoints
Scraping news is three jobs under one name. Find articles, read one article, watch one company. Each needs a different call and a different budget.
Copy this line to your agent to pull news coverage on a topic and read the articles in one pass.
set up https://monid.ai/SKILL.md and use context.dev /web/search with markdownOptions to find and read news articles on a topic
"Scraping news" is three jobs sharing one name, and most pipelines built for it are expensive because they do all three with the tool for one. Finding articles about a topic, reading an article whose URL you already have, and watching one company across every publication are different operations with different inputs, different outputs and different billing. Sort them first and the cost problem mostly disappears. Everything below runs through Monid, the OpenRouter for agent tools.
How do you web scrape news articles?
You start by deciding which of the three jobs you are actually doing, because the answer changes everything downstream.
Job one: you have a topic and no URLs
You want everything published about tariffs on semiconductors this week. You do not have a list of articles; producing that list is the work. This is a search problem, and building it as a crawler over a set of news homepages is the classic mistake: you get whatever those particular outlets published, ranked by where it sat on the page.
Job two: you have a URL and want the text
Somebody handed you a link, or job one produced one. You want the article body without the navigation, the newsletter modal and the related stories rail. This is an extraction problem, it is the cheapest of the three, and it is where most homegrown pipelines waste the most effort writing per publisher rules.
Job three: you have a company and want its coverage
You are watching a customer, a competitor or a portfolio company, and you want to know when anything is written about it, including when the company writes it about itself. This is an entity problem, not a keyword problem, and searching a name gets it wrong constantly because company names collide with ordinary words.
Why the split matters more than the tooling
The three jobs have different cost curves. Search bills for results you asked for whether or not any is useful. Extraction bills per page and you control exactly which pages. Entity monitoring bills per article and runs on a schedule forever, so a poorly scoped one is the only recurring line item of the three.
A pipeline that runs everything through job one, searching a broad keyword daily and reading every result, pays search prices for extraction work and reads the same syndicated article eleven times. Recognising the split is worth more than any tool choice in this guide.
📖 See also A Free API to Extract Page Content for RAG: Read This First
Why do news sites break a generic scraper?
Because news publishers are the most aggressively instrumented pages on the open web, and almost none of that instrumentation is the article.
The article is a minority of the page
A typical news page is a headline, six hundred words of body, and then a recirculation rail, a newsletter interstitial, an embedded video player, a comment widget, three ad slots and a related stories block that is itself full of headlines. A naive text extraction returns all of it, and downstream you cannot tell the article's own words from the eleven other headlines glued to the end. For retrieval this is worse than useless: it puts unrelated claims inside the chunk you will later cite.
Syndication means you read the same story repeatedly
One press release becomes forty articles. A wire story runs verbatim in nine outlets. If your index does not know these are one story, a question about it retrieves nine near identical chunks and your model reads the same sentence nine times. Deduplication by URL does nothing here, because the URLs are genuinely different.
Blocking is normal and quiet
Publishers block. So do law firms, banks and anyone else with a compliance page worth reading. A concrete instance from writing this article: fetching a Farella Braun and Martel publication page with an ordinary HTTP client returned 403 Forbidden and no body. The same URL through a scrape endpoint on 2026-08-25 returned the full article, its JSON-LD, its heading tree and its Open Graph metadata. Nothing about the page had changed; what changed was who was asking and from where.
That is the ordinary condition rather than an edge case, and the fuller treatment is in Your Scraper Is Blocked: What Actually Gets Through.
Roll your own vs endpoint: what actually differs
| Aspect | Per publisher scrapers | A news endpoint |
|---|---|---|
| Coverage | The outlets you wrote rules for | Whatever the index reaches |
| Boilerplate removal | Your selectors, per site | Main content detection, built in |
| Syndication | You dedupe, somehow | Grouped by story identifier |
| Blocked publisher | You buy proxies | Included, and failures are not billed |
| Cost when a site redesigns | An engineer's afternoon | Nothing |
| Best for | A handful of critical outlets | Broad coverage of a topic or entity |
The pattern: per publisher rules buy you precision on a fixed list and cost you everything outside it.
How do you find news about a topic?
Three steps for job one, and the first two are free.
For agents
Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:
set up https://monid.ai/SKILL.md
It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.
For humans
npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main
Step 1. Search with a freshness window
What it does. Returns ranked results for a query, restricted to content published inside a window, so a daily job does not re-read last month.
The endpoints. context.dev/web/search, billed per result.
The call.
monid run -p context.dev -e /web/search -w -i '{
"query": "semiconductor tariffs",
"numResults": 20,
"freshness": "last_24_hours",
"excludeDomains": ["reddit.com", "quora.com"]
}'
What comes back. A results array with url, title, description and a relevance label of high, medium or low, plus the query echoed back. Running a comparable query on 2026-08-25 returned exactly that shape, with the low relevance band correctly catching the forum and Q&A pages that a raw search would have mixed in with the reporting.
The relevance band is the field to filter on, and excludeDomains is the field to use before you need to. The query string also accepts Google style operators, so site:, -site:, quoted phrases and OR all work.
What it costs. A tiny fraction of a cent per result, billed per page of results rather than per individual row. Current figures on monid.ai/tools.
Step 2. Read the ones worth reading
What it does. Converts a chosen URL into clean Markdown with the recirculation and newsletter furniture removed.
The endpoints. context.dev/web/scrape/markdown, billed per call, or the same conversion inline by setting markdownOptions.enabled on the search itself.
The call.
monid run -p context.dev -e /web/scrape/markdown -w \
--query '{"url": "https://example.com/article", "useMainContentOnly": true}'
What comes back. markdown, a contentLength integer, and a metadata object with title, language, canonicalUrl, siteName, a parsed headings tree, openGraph, twitter and, when the publisher ships it, the full jsonLd block. That last one is worth more than it sounds: on the law firm page above it contained the article body, the publication date and the named authors as structured data, which is a cleaner byline than anything you would parse out of the rendered page.
What it costs. A fraction of a cent per successful call, and blocked or failed fetches are not billed. Do this as a second step rather than inline whenever you plan to read fewer results than you retrieve, which is almost always.
The decision rule. Search inline when you will read everything you get back. Search then scrape when you will read a subset. The second is cheaper the moment your read rate drops below about half.
Step 3. Deduplicate before indexing
What it does. Stops nine copies of one wire story becoming nine chunks.
The call. No endpoint. Hash the normalised first two hundred characters of the body and drop collisions, or use the story grouping from the entity endpoint in the next section, which does it for you.
What comes back. A smaller index and better retrieval. This step is skipped constantly and it is the single biggest quality difference between a news RAG store that works and one that returns the same paragraph five times.
📖 See also Any URL to LLM-Ready Markdown: A Copy-Paste Cookbook
Give this to your agent
Set up https://monid.ai/SKILL.md, and then use Monid to find everything published in the last 24 hours about semiconductor tariffs, drop anything below high relevance, read the survivors into markdown, and give me one paragraph per distinct story.How do you monitor news about one company?
Job three is a different endpoint because it is a different question. You are not matching a string, you are matching an entity, and the useful version knows the difference between an article about Apple the company and one about apples.
What it does. Returns coverage of one company identified by name, website domain, stock ticker or ISIN.
The endpoints. context.dev/news/search, billed per result.
The call.
monid run -p context.dev -e /news/search -w -i '{
"searchBy": {"type": "entity", "entity": {"type": "domain", "domain": "example.com"}},
"limit": 10
}'
What comes back. Running this against a real company domain on 2026-08-25 returned articles carrying url, title, description, language, an authors array, image_url, published_at, a source object with the publication name and domain, and three fields that make this endpoint worth using over a keyword search.
story_id groups syndicated copies of one announcement, so the deduplication problem from job one is solved in the response rather than in your code.
type classifies each article as editorial, press release, regulatory filing or advisory. In that run the PRNewswire item came back typed press_release with the company itself listed as the author, while the CNBC and Verge coverage of the same event came back as editorial. That distinction is the difference between "the market noticed" and "the company issued a statement", and it is the field most monitoring setups do not have.
match carries a level of primary or secondary and a numeric confidence. Primary means the company is the subject; secondary means it was mentioned. Filtering to primary is what stops a competitor watch from filling with articles where your competitor is the eleventh company listed.
What it costs. A tiny fraction of a cent per article, billed per page of ten. A daily watch on twenty companies is small change per month. Figures on monid.ai/tools.
The fuller build of this, including the digest that lands in a channel every morning, is in Company News API: Press Releases Without the Newswire Contract and Build a Company News Watcher That Posts a Daily Digest.
Which endpoint should I use for which job?
| Endpoint | What it does | Input | Output | Best for | Billing |
|---|---|---|---|---|---|
context.dev/web/search | Finds articles by query | Query, freshness, domain filters | URL, title, description, relevance band | You have a topic, not URLs | Per result |
context.dev/web/scrape/markdown | Reads one article | A URL | Markdown, JSON-LD, headings, Open Graph | You have the URL | Per call |
context.dev/news/search | Coverage of one company | Name, domain, ticker or ISIN | Articles with story grouping, type, match level | Watching an entity | Per result |
context.dev/web/crawl | Walks a publisher's archive | Start URL, depth, URL regex | Markdown per page | Backfilling one outlet | Per page |
apollo/news_articles/search | Company news tied to a CRM shaped record | Organisation identifier | Articles keyed to the org | Sales triggers | Per call |
Every row verified with monid inspect on 2026-08-25. The table gives billing shape rather than figures, because shape is what changes your design and current numbers live on monid.ai/tools.
Worked through: a daily topic watch pulling twenty search results, reading the six that clear a relevance filter, plus an entity watch on twenty companies at ten articles each. Search bills twenty results, extraction bills six calls, monitoring bills two hundred articles. All three together land in cents per day and low single digit dollars per month, and the largest of the three is the one running on a schedule, which is why scoping the entity list matters more than optimising the search.
Is web scraping news legal?
Short answer, and it is genuinely short: reading publicly accessible pages has been treated very differently by US courts from copying and republishing what you read, and the recent rulings have turned on whether you were logged in and whether the data was public, not on scraping as an activity.
The longer answer needs actual case citations and is a whole article, which is why it is a separate guide rather than three careless paragraphs here. What belongs in this one is the practical part: news content is copyrighted expression, and the distance between indexing an article for retrieval and republishing its text is the distance that matters. Store what you need to cite, cite what you store, and link back. None of that is legal advice and none of it substitutes for asking a lawyer about your specific use.
Conclusion
There is no single way to scrape news, because "news" names three jobs and building all three on one tool is where the cost and the quality problems both come from. Find articles with a search endpoint and a freshness window. Read the ones worth reading with an extraction endpoint, as a second step, so you pay for reads you actually wanted. Watch a company with an entity endpoint that knows syndication from original reporting.
The detail that matters more than any of those choices is the classification. An article typed as a press release and an article typed as editorial are evidence of different things, and a monitoring system that cannot tell them apart will report a company's own announcements as market reaction. That field costs nothing extra and almost nobody filters on it.
Free next step: run monid discover -q "search news articles" and monid inspect whichever result matches your job. Both are free and take a minute, and you will see the exact response fields before spending anything. Start at monid.ai.
FAQ
How do you scrape news articles with Python?
The Python part is the easy half: newspaper3k, trafilatura and readability-lxml all do respectable article extraction, and trafilatura in particular is very good at separating body from furniture. What none of them solves is getting the page in the first place, which is where news pipelines actually fail. Keep the Python parser if you like it and put a fetch endpoint underneath it; the longer version of that argument is in Web Scraping in Python Without Maintaining a Scraper.
Can you scrape Google News?
You can search for news through a search endpoint, which is what most people mean, and it is a better idea than pointing a crawler at the Google News interface. Google News is a ranked aggregation over publishers, so scraping it gets you Google's ordering rather than the underlying coverage, and the interface changes often enough to break selector based scrapers regularly. If what you want is the SERP itself rather than the articles, that is a different product category and we covered it in the SERP API guide.
How do you scrape a specific news site like the BBC?
Two shapes, depending on whether you want the archive or the feed. For a backfill, crawl the section you care about with a URL regex to keep the crawl inside it, and expect the page cap to matter more than the depth. For ongoing coverage, check for an RSS feed first, because a large number of publishers still ship one and it is both free and more reliable than anything you would build. Fall back to search with a site: operator when neither is available.
What is the best web scraper for news headlines?
Headlines specifically are the cheapest thing to get, because a search endpoint returns the title and description without reading the page at all. If headlines are all you need, do not scrape the articles: run the search, keep title and description, and skip extraction entirely. That turns a per page cost into a per result cost and is the single easiest saving available in a news pipeline.
Last updated August 2026.
