Web Scraping News Articles: Get Clean Text, Author and Date

Turn a list of news article links into clean text with title, author and date. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to read each of these news article URLs into Markdown, and give me a table with title, author, publish date and the first 100 words of the body for every one.

One prompt reads a pile of articles. 1 min setup · 30-60 hrs saved.

You have news articles to collect: links from an alert, a spreadsheet of URLs, or a topic you want covered, and you want the text, the author and the date in a form a script or a model can use. This guide covers the quickest way to do that, what one scraped article contained when we ran it on 2026-10-16, how to find articles when you have a topic and no links, and where the output still needs your cleanup. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to scrape news articles?

Monid. Install it once, add a key, and your agent turns any news URL into clean Markdown with the page metadata attached, with no scraper to write and no proxy to buy.

• Pay per call. A scraped page is billed on its own at $0.002533 (Verified 2026-10-16 on the news tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and balance cover article reading, web search, company news and everything else on Monid. Building and wiring those yourself is the 30 to 60 hours this page claims to save.
• Lives inside your agent. Install the CLI, add your key, run the prompt above, and the tools are there whenever your agent needs them. Your agent reads SKILL.md and learns the rest, so setup is about a minute.

At that price 100 articles cost about $0.25 (Verified 2026-10-16).

How do you get the text of a news article from its URL?

Call the scrape endpoint with the URL and ask for the main content only. It returns the article as Markdown plus a metadata block with the title, language, canonical URL, publish date and author.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Read one article

monid run -p context.dev -e /web/scrape/markdown -w \
  --query '{"url": "https://example.com/article", "useMainContentOnly": true}'

The endpoint is a GET, so the URL goes in --query. We ran it on 2026-10-16 against an NPR story. It came back with HTTP 200, one billed unit at $0.002533 (Verified 2026-10-16) and took about 3.5 seconds.

Read a list of articles

Put one URL per line in urls.txt, then run:

mkdir -p out
i=0
while read -r url; do
  i=$((i+1))
  monid run -p context.dev -e /web/scrape/markdown \
    --query "{\"url\":\"$url\",\"useMainContentOnly\":true}" > "out/article-$i.json"
done < urls.txt

Each URL writes one file. Failed or blocked fetches are not billed, so a list with dead links costs nothing for the dead ones.

Or hand the whole job to an agent

How do you find news articles on a topic when you have no URLs?

Run a web search with a freshness window, then read only the results worth reading. The search endpoint takes a query with Google-style operators, a result count from 10 to 100, domain allow and block lists, and a freshness window of the last day, week, month or year.

monid run -p context.dev -e /web/search -w -i '{
  "query": "semiconductor tariffs",
  "numResults": 20,
  "freshness": "last_24_hours",
  "excludeDomains": ["reddit.com", "quora.com"]
}'

We inspected this endpoint but did not run it on 2026-10-16, because its billing is per result with one credit per page of 10, and a request starts at 10 results. The listed price is $0.000253 per result (Verified 2026-10-16), so the 20 results above come to about $0.005. The endpoint's description says each result comes back with a URL, a title and a relevance label, and that setting markdownOptions.enabled scrapes every result in the same call.

The search endpoint here is one of several, and free web search APIs for agents compares the others. Search first, scrape second is the cheaper order whenever you will read fewer pages than you retrieve. At the listed prices a page of 10 results costs about the same as one scrape, so reading only the 6 results that look right costs 6 scrapes on top of the search.

If you want the coverage of one company rather than one topic, the news endpoint takes a name, domain, ticker or ISIN and returns articles with a story id for syndicated copies and an article type. That is a different job with its own guide, company news API and press releases.

Can you scrape news articles yourself with Python?

Yes, and for a handful of fixed sites it works well. Libraries such as trafilatura, newspaper4k and readability-lxml extract the body from HTML you already hold, and they are free. The part they leave to you is getting the HTML: proxies, browser rendering for script-heavy pages, retries, and a repair every time a publisher changes its layout or its blocking.

News pages are the hard case for that part. They are rendered by script, wrapped in consent and newsletter layers, and watched closely for automated traffic, so the fetch is where news pipelines break. Python web scraping without a scraper shows where the DIY route stops paying.

The Monid route is the choice for anything you will run more than once. JavaScript rendering, anti-bot handling and residential proxies are included in the per-call price, the output is typed JSON instead of HTML you parse, and a blocked or failed request is not billed. You can keep your Python parser if you like it: fetch through the endpoint, then post-process the Markdown in your own code. A download of a news dataset is quicker for a one-off look, but it is a snapshot of one date for the outlets its author chose.

What does a scraped news article look like, and what can you build with it?

One call returns the Markdown and a metadata object. We ran the NPR article on 2026-10-16, so the fields below are what came back.

The response

{ "success": true,
  "markdown": "...",
  "contentLength": 9989,
  "url": "https://www.npr.org/nx-s1-5993835",
  "metadata": {
    "finalUrl": "https://www.npr.org/2026/10/16/nx-s1-5993835/gen-z-trump-voting-support",
    "canonicalUrl": "https://www.npr.org/2026/10/16/nx-s1-5993835/gen-z-trump-voting-support",
    "title": "'Everything got worse': Gen Z express frustration, regret after Trump vote",
    "description": "...",
    "language": "en",
    "siteName": "NPR",
    "publishedTime": "2026-10-16",
    "openGraph": { "...": "..." },
    "twitter": { "...": "..." },
    "alternates": [ { "type": "application/rss+xml", "href": "https://feeds.npr.org/139482413/rss.xml" } ],
    "headings": [ { "level": 1, "text": "..." } ],
    "jsonLd": [ { "@type": "NewsArticle", "headline": "...", "datePublished": "...", "author": { "name": "Elena Moore" } } ] } }

Four parts matter. markdown is the body. metadata.title, language, canonicalUrl and publishedTime give you a record without parsing HTML. jsonLd held a NewsArticle block with the headline, publish and modified timestamps, the author and the publisher, which is a cleaner byline than anything you would scrape from the rendered page. alternates listed the publisher's two RSS feeds. The fetch also followed a redirect: the short URL we sent came back as a dated canonical path in finalUrl.

The markdown itself was 9,989 characters for a story whose body is a few paragraphs, which is the subject of the next section.

What you can build with it

• A reading queue for a model. Feed a list of links through the loop above, keep title, publishedTime and the body, and your agent summarises each. For the page-reading half of a retrieval pipeline see a free API to extract page content for RAG.
• A topic archive. Search a topic every day with a freshness window, scrape what is new, and append rows to a CSV or a database with the canonical URL as the key.
• A byline and date index. jsonLd and publishedTime let you sort a pile of articles by date and author without reading them.

What goes wrong when you scrape news articles at scale?

Three things show up as soon as you run more than a few URLs. The first two came out of our 2026-10-16 run.

Why is the Markdown longer than the article?

Because "main content only" is a best effort, not a guarantee. With useMainContentOnly set to true the NPR result still contained the audio player controls and embed code, three "Sponsor Message" lines, three related-story cards in the middle of the body, a membership promo and the words "Close modal" at the end. The headings array also listed teaser titles from the "More Stories" and "Popular" rails.

Trim the furniture before you index. Cut at the first line you recognise as the end of the story, drop lines that equal "Sponsor Message", and take the title and date from metadata rather than from the body. A retrieval index that keeps related-story cards will later retrieve claims that belong to other articles.

const body = md.split("\n").filter(l => l.trim() !== "Sponsor Message").join("\n");

Why does the URL you stored not match the URL you got?

Because publishers redirect. We sent https://www.npr.org/nx-s1-5993835 and the response reported a finalUrl and a canonicalUrl with the date and slug in the path. Key your own store on canonicalUrl, or the same story arrives twice under two links. A hash of the first 200 characters of the cleaned body catches the same wire story running at two outlets, where the URLs differ legitimately.

How many calls is a news archive?

One per article. At the listed price, 1,000 articles is about $2.53 (Verified 2026-10-16). The number to watch is how many you read, not how many you find: a daily search that returns 100 results and reads all of them is 100 scrapes a day, which is why you filter before you scrape.

Which endpoint should I use for which job?

The scrape row was run on 2026-10-16; the other rows were inspected the same day and the current figures are on monid.ai/tools.

When is another route a better fit?

The publisher offers a feed. Many news sites ship RSS, and NPR's page listed two feeds in its metadata. For ongoing headlines from a few outlets, a feed is free and needs no scraper. Use the scrape endpoint for the article body behind each feed item.

You need every article from one outlet for years. That is an archive crawl, and a licensed dataset or the publisher's own archive access will be a better fit than a per-call fetch of each page.

You are republishing the text. News articles are copyrighted. Reading and indexing them for retrieval is one act and reproducing them on your own site is another, and the legal side is covered in is web scraping legal in 2026.

You are paywalled. The NPR page we read carried article:content_tier: free. A subscriber-only article is a different case, and logging in is outside what this endpoint does.

And the disclosure: we resell these endpoints. The most useful habit in this post is to trim the page furniture and key on the canonical URL yourself, which costs nothing whoever you buy the data from.

Conclusion

The quickest way to scrape news articles is one scrape call per URL through Monid: $0.002533 a page (Verified 2026-10-16), one key for article reading and everything else, about a minute to set up, and ready inside your agent whenever you need it. When you have a topic and no links, search first with a freshness window and read only the results worth reading.

Two things matter more than the choice of tool. Clean the Markdown before you index it, because "main content only" still left sponsor lines, related-story cards and a modal string in our run. And key your store on the canonical URL, because the link you send is not always the link you get back.

How we estimated the time saved: building your own news pipeline is roughly 10 to 20 hours each for getting past blocking and rendering, parsing the body, byline and date across outlets, and the first month of upkeep when layouts change. That is 30 to 60 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid inspect -p context.dev -e /web/scrape/markdown, scrape one article you know, and print the length of the Markdown next to the length of the story you expected. The gap is the cleanup you will need. Start at monid.ai.

FAQ

Published August 2026.