Free API to Extract Page Content for RAG: What Each Route Returns
Pull clean page content into a RAG index with one call per URL, or use a source's own free API. Fields and sizes from live runs on one Wikipedia page.

Turn a web page into clean text you can chunk and embed, in one prompt. Paste this into your agent:
Set up https://monid.ai/SKILL.md, and then use Monid to fetch https://en.wikipedia.org/wiki/Retrieval-augmented_generation as markdown with main content only, and split it by heading into chunks that each keep their heading path.
One prompt turns a URL into chunks with headings. 1 min setup · 8-20 hrs saved.
You are building a RAG index and you need the text of web pages in it, without the menus, the footers and the cookie banners. Somebody on a forum asks whether a free API does this, and the answers name scraping vendors. This guide answers the question straight: when a source publishes its own free API, use it, and for the rest of the web it shows the quicker route, one call per URL, with what four different endpoints returned for the same page on 2026-10-16. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.
What is the easiest way to extract page content for RAG?
Monid. Install it once, add a key, and your agent calls an extraction endpoint with a URL and gets the page back as markdown or as structured blocks, with no scraper to host and no vendor account to set up.
- Pay per call. One page is billed on its own, at $0.002533 through the markdown endpoint (Verified 2026-10-16 on the web extraction tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
- One install, every tool. The same key and the same balance call the extraction endpoints and every other tool on Monid. When the index also needs search or a site crawl, it is another call on the same balance, not another vendor.
- Lives inside your agent. Install the CLI, add your key, run the prompt above, and the tool is there whenever your agent needs it, on demand, with nothing to keep running. Your agent reads
SKILL.mdand learns the rest, so setup is about a minute.
At that price, a hundred pages cost about $0.25 (Verified 2026-10-16).
Is there a free API to extract page content for RAG?
For a source that publishes its own API, yes, and it is the best extraction endpoint you will find for that source. For the open web there is no single free endpoint, and the paid ones bill per page.
The thread behind this question was about wiki content, and Wikipedia serves a REST summary endpoint with no key. We called it on 2026-10-16 with a descriptive User-Agent header:
curl https://en.wikipedia.org/api/rest_v1/page/summary/Retrieval-augmented_generation
It returned 16 fields. The ones that matter for an index are extract, which was 689 characters of clean prose, title, pageid, lang, dir, revision (1377227095) and timestamp (2026-09-28T11:20:51Z). The revision and the timestamp are what make a stored chunk reproducible: you can say exactly which version of the page the text came from.
Two limits keep this from being the whole answer. The summary is the lead of the article, not the article. And it only exists for sources that run an API; documentation sites, vendor pages and blogs mostly do not, so there the job is to fetch the page and strip it. Check whether the source offers an API before you pick a scraper.
The rest of this guide is the case with no API. The web extraction tools page lists the endpoints, and the free monid inspect shows the schema and price of each before anything bills.
How do you turn a list of URLs into clean text for a RAG index?
Call the scrape endpoint with each URL and ask for the main content only. It returns the page as markdown plus a metadata block with the title, language, canonical URL and the headings it found.
For agents
Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:
set up https://monid.ai/SKILL.md
It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.
For humans
npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main
Inspect, then run one
monid inspect -p context.dev -e /web/scrape/markdown
monid run -p context.dev -e /web/scrape/markdown \
--query '{"url":"https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}' -w
The endpoint is a GET, so the URL goes in --query. To drop headers, footers and sidebars where the page allows it, add useMainContentOnly:
monid run -p context.dev -e /web/scrape/markdown \
--query '{"url":"https://en.wikipedia.org/wiki/Retrieval-augmented_generation","useMainContentOnly":true}' -w
Both calls came back with HTTP 200 and one billed unit at $0.002533 (Verified 2026-10-16).
Loop it over a list
Put one URL per line in urls.txt, and write one file per page:
mkdir -p pages
i=0
while read -r u; do
i=$((i+1))
monid run -p context.dev -e /web/scrape/markdown \
--query "{\"url\":\"$u\",\"useMainContentOnly\":true}" -w > "pages/$i.json"
done < urls.txt
Each URL bills one call, and a failed or blocked fetch is not billed. If you would rather send a batch, octen /extract takes up to 20 URLs in one POST body:
monid run -p octen -e /extract \
-i '{"urls":["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"]}' -w
It bills per successful URL, $0.001 each (Verified 2026-10-16), and a failed URL comes back with status: "failed" and is free.
Or hand the whole job to an agent
Give this to your agent![]()
Set up https://monid.ai/SKILL.md, and then use Monid to read every URL on this list as markdown with main content only, split each page by heading, and give me one JSON line per chunk with the source URL, the heading path and the text.Can you extract page content yourself with Python?
Yes, and for a few static pages it is an afternoon. You fetch the page with requests, parse it with BeautifulSoup or a readability library, drop the tags you recognise as chrome, and write out text. That is the method behind every extractor, and for your own pages or a fixed handful it is enough.
The cost is everything around the parser. Many pages return an empty shell until JavaScript runs, so you need a browser. Sites block repeated requests from one address, so you need proxies. Every site has its own markup, so the rule that strips one site's sidebar leaves another's in. And each of those rules breaks when a template changes. The first working version takes hours and the upkeep never ends, which is the same trade described in Python web scraping without a scraper.
The Monid route is the choice for anything you will run more than once. The rendering, the exit address and the conversion are maintained for you, the output is markdown plus typed metadata instead of HTML you parse, and a page is $0.002533 (Verified 2026-10-16). If you want to run the crawler on your own machine instead, Crawl4AI and when to run your own crawler covers that side.
What does extracted page content look like, and what can you build with it?
It looks different on each endpoint, and the difference decides how much cleaning you do before chunking. We ran four routes on 2026-10-16 against the same page, the Wikipedia article on retrieval-augmented generation, so the fields below are what came back.
The markdown scrape
With default options the response reported contentLength 77,327 and the markdown opened with [Jump to content] followed by Main menu and the navigation list. The article's first sentence starts about 11,900 characters in, so the first sixth of the file is navigation. With useMainContentOnly set to true, contentLength was 32,971 and the markdown opened with the line From Wikipedia, the free encyclopedia, then the lead paragraph.
Beside the markdown, metadata carried the title, language, canonicalUrl, a headings list with the level and text of each heading, openGraph, alternates and jsonLd. The JSON-LD held a dateModified of 2026-09-28T11:20:51Z, the same timestamp the Wikipedia API gave for the revision.
The batch extractor
octen /extract returned one result per URL with url, status, title and full_content, plus time_published (2023-11-05T13:19:20Z), time_last_crawled (2026-10-17T04:39:03Z), a category, a page_structure with a primary and secondary label (Content Page, Encyclopedia) and a resolved_mode. The response reported a latency of 815 ms. The two dates are useful in an index, because they let a chunk carry both when the page was published and when you last read it.
The structured parser
dataforseo /onpage/content-parsing with markdown_view set to true returned an item with status_code, fetch_time, page_content and page_as_markdown. The useful part is page_content. It splits the page into header (15 entries), footer (8 entries) and main_topic, which held 14 sections, each with an h_title, a level, its primary_content and any table_content. The first section was Retrieval-augmented generation at level 1 and the last was References. The boilerplate is already separated from the body, and the sections are already cut on headings.
What you can build with it
- A heading-aware chunker. Split the markdown on its headings, or read the sections straight from
main_topic, and store the heading path with each chunk so a retrieved fragment knows where it sat. Ingesting mixed documents for LLM embedding shows the same idea with PDFs and Office files. - A source-of-truth index with dates. Keep the canonical URL, the fetch date and the page's own modified date next to every chunk, and you can refresh only what changed.
- A news or docs mirror. Read a list of article URLs on a schedule and write the text to your store, as in web scraping news articles.
- A search-then-read agent. Search for pages on a question, then read the top results in full, the pattern in Octen's search, extract and embed endpoints.
What goes wrong when you extract pages for RAG?
Four things showed up in the 2026-10-16 runs, and none of them raise an error.
Why does the default markdown start with a menu?
Because a faithful conversion keeps everything on the page. The default scrape of the Wikipedia article was a valid markdown file with HTTP 200, and 11,891 of its 76,816 characters came before the first sentence of the article. Nothing in the status tells you that. Read the first thousand characters of a new source before you index a thousand pages of it, and set useMainContentOnly to true when the page has a main region.
Why is main content only still not just the article?
Because the option drops the header, footer and sidebar, not the rest of the article page. The main-only Wikipedia output still carried the [edit] link beside every heading, the whole reference list, the Retrieved from line and the category lists. For an index you usually want those out. Cut at the first heading called References or Notes, and strip lines that are only an edit link. A schema that returns named fields avoids this, which is why the structured parser's main_topic is attractive for sources it handles well.
Why does a cache hit still bill?
Because the price is for the call, not for the work behind it. Our second scrape reported cache_metadata status hit with an age of 3,129 ms and still billed one unit at $0.002533 (Verified 2026-10-16). The maxAgeMs option controls how old a reused result may be, and its default is one day, so a page you fetched yesterday may come from cache. Set maxAgeMs to 0 when freshness matters, and do not expect a cached call to be cheaper.
Why do the routes disagree about what a page is?
Because each one draws the boundary in a different place. The markdown scrape gave one string. octen /extract gave one string with relative links such as #cite_note-IBM-2023-1 and trailing text that read 25 languages [Add topic]. The structured parser gave sections, but its page_as_markdown field opened with the H1, then ## References and a navigation table, so it is not the field to chunk. Pick one route per source and read one page of its output before you automate the loop.
Which endpoint should I use for which job?
| Endpoint | What it does | Input | Billing | Price (Verified 2026-10-16) |
|---|---|---|---|---|
context.dev /web/scrape/markdown | One page to markdown plus metadata, useMainContentOnly available | url | Per call, failures not billed | $0.002533 |
octen /extract | Up to 20 URLs to markdown or text, with dates and page type | urls[], optional query | Per successful URL | $0.001 |
dataforseo /onpage/content-parsing | Page split into header, footer and headed sections | url, markdown_view | Per call, extras cost more | $0.00015 |
context.dev /parse | A file at a URL (PDF, Office, images) to markdown, optional OCR | file_url, ocr | Per call, OCR call bills higher | $0.002533 |
Prices Verified 2026-10-16 with monid inspect. We ran the first three on one page. We inspected /parse and did not run it, because its bill depends on whether OCR ran: the listed price is one credit, and an OCR-executed call is five. The structured parser lists extra cost when you turn on JavaScript or browser rendering, so its price is a floor.
For a first test, use the markdown scrape with useMainContentOnly. For known URLs in batches, octen /extract takes twenty at a time. To get sections without writing a splitter, try the structured parser and read main_topic.
When is another route a better fit?
The source has an API. Use it. The Wikipedia summary above is free, versioned and carries a revision, and no scraper returns that.
You index your own content. Read it from your CMS or repository, where the structure and the identifiers already exist. Scraping your own site to get text you already hold is extra work.
The documents cannot leave your network. A hosted endpoint fetches the page from outside. Anything under a rule that says the bytes stay put belongs in a local pipeline.
You need pages behind a login. A per-call fetch is stateless. A maintained session needs a browser you control.
You want chunking, embeddings and a vector store in one product. These endpoints return text and metadata; the chunking, embeddings and index are yours to build or buy.
And the disclosure: Monid resells these endpoints. The most useful thing in this post is the habit of reading the first thousand characters of any new source, which costs one billed call and is right whoever you buy the extraction from.
Conclusion
The quickest way to extract page content for RAG when the source has no API is one call per URL through Monid: $0.002533 a page (Verified 2026-10-16), one key for this and every other tool, about a minute to set up and ready inside your agent whenever you need it. When the source publishes a free API, use that first, as the Wikipedia summary shows. Writing the extractor yourself works for a few static pages and costs hours of rendering, proxies and per-site rules.
Two things matter more than which endpoint you pick. Read the first thousand characters of a new source, because a valid markdown file can open with eleven thousand characters of menu. And store the metadata with every chunk: the canonical URL, the fetch date and the page's own modified date are what let you refresh the index and show where an answer came from.
How we estimated the time saved: building and running your own page extractor is roughly 6 to 12 hours for a fetch layer with rendering, proxies and the first boilerplate rules, plus 2 to 8 hours for the first month of upkeep as sites change. That is 8 to 20 hours. The 1 minute is the install, key and first call once you already have a Monid key.
Free next step: run monid inspect -p context.dev -e /web/scrape/markdown, then make one call on a page you know well and read the first thousand characters. Start at monid.ai.
FAQ
Is there a free API to extract page content for RAG?
For a source that publishes its own API, yes: Wikipedia's REST summary endpoint returned a 689-character extract, a revision number and a timestamp with no key on 2026-10-16. For the open web there is no free general endpoint in this guide; through Monid a page costs $0.002533 (Verified 2026-10-16), with no plan and no minimum.
How do I extract the main content of a web page for an LLM?
Call a scrape endpoint with the URL and useMainContentOnly set to true, which drops the header, footer and sidebar where the page allows it. Then cut the reference list and edit links, because they stay in the output.
How do I strip navigation from scraped markdown?
Ask the endpoint to do it with useMainContentOnly, or use a parser that returns the header, footer and body as separate fields. Matching the menu text yourself breaks on each site, because each template words and orders its menu differently.
Which is cheaper for a batch, one call per page or a batch endpoint?
The batch endpoint octen /extract bills $0.001 per successful URL (Verified 2026-10-16) and takes up to 20 URLs in one request. The markdown scrape bills one call per page and does not bill failures. Both are small enough that the output you want should decide.
Does a cached result cost less?
No. Our second scrape reported a cache hit and still billed one unit. Set maxAgeMs to 0 to force a fresh fetch when the page changes often.
Published August 2026.


