A Free API to Extract Page Content for RAG: Read This First

Somebody building a local RAG setup wants page content in, embeddings out, and asks the reasonable question: is there a free API for this. The answers they get name scraping vendors, which is a fine answer to a slightly different question.

This guide starts with the answer that is not ours, because for the case that generated the question it is the right one. Monid is the OpenRouter for agent tools, and the extraction endpoints below are in our catalogue, so the comparison is measured rather than asserted.

Fair disclosure: you are on the Monid blog, and the first recommendation in this post is to not use us.

Is there a free API to extract page content for RAG?

For the source most people are asking about, yes, and it is published by the source itself.

The thread behind this question was about wiki content. Wikipedia runs a free REST API, no key, no rate plan. We ran both routes against the same page on 2026-08-19.

The official API:

curl https://en.wikipedia.org/api/rest_v1/page/summary/Retrieval-augmented_generation

extract      689 characters of clean prose
title        the canonical title
pageid       a stable identifier
revision     the exact revision this text came from
timestamp    when that revision was made
lang, dir    language and text direction

The same page, scraped:

monid run -p context.dev -e /web/scrape/markdown \
  --query '{"url":"https://en.wikipedia.org/wiki/Retrieval-augmented_generation"}' -w

Provider Response: 200
contentLength:     74,552
markdown begins:   "[Jump to content] Main menu ... move to sidebar hide
                    Navigation - [Main page] - [Contents] ..."

Six hundred and eighty-nine characters against seventy-four thousand, and the scrape opens with the navigation menu. The official route also carries a revision identifier, which the scrape has no equivalent of and which is the field that makes a RAG index reproducible.

The general rule, and it holds well beyond Wikipedia: if the source publishes an API, that is the extraction endpoint. It returns the content without the chrome, it carries identifiers you can cite, and it is what the publisher intends you to use. Reaching for a scraper against a source with an official feed is choosing the worse data and the weaker legal footing at the same time.

Why is scraped page text worse than it looks?

Because clean markdown is not the same as clean content, and the difference is most of the file.

Our 74,552 characters were genuinely well-converted: readable markdown, working links, correct headings. They were also mostly not the article. A Wikipedia page carries a navigation menu, a sidebar, a language list, an edit toolbar, a references apparatus and a footer, and all of it converts perfectly into markdown that is perfectly useless to a RAG index.

Four consequences, in the order they bite:

Your embeddings get diluted. Chunk that file naively and a meaningful fraction of your vectors encode navigation text. Those chunks are retrievable, they match on generic queries, and they push real content out of the top results.

Boilerplate is identical across pages. Every page from one site shares its chrome, so those chunks are near-duplicates of each other. A similarity search over the index returns the same menu from forty different pages.

Token costs scale with the junk. If chunks go into a context window, you are paying for the sidebar on every retrieval.

The signal-to-noise ratio is invisible in the status code. The request succeeded, the markdown is valid, and nothing in the response tells you that the first eight hundred characters are a menu.

The fix is not a better scraper, it is a boundary-aware extraction step: keep the main content region, drop nav and footer, split on headings rather than on character counts, and carry the heading path into each chunk's metadata so a retrieved fragment knows where it came from. We covered the practical version of that in batching transcripts into a RAG store, where the timestamps do the same job that headings do here.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-key> -l main

When do you actually need a general extraction endpoint?

When there is no official API, and that covers most of the web.

Wikipedia is the easy case precisely because it is unusually well served. The sources that make a RAG index worth building rarely are:

Documentation sites. Most publish no API, and the content is the reason you are indexing at all.

Competitor and vendor pages. Nobody publishes an API for their own marketing site, and pricing and feature pages are exactly what a competitive index wants.

News and blog archives. Some have feeds, most feeds are truncated, and a summary is not the article.

PDFs and documents. Reports, filings, manuals. A different extraction problem and one a general endpoint handles better than a hand-rolled parser.

For those, the choice is between running your own extraction and calling an endpoint, and the honest split is the same as everywhere else in this catalogue: build it when extraction is your product, buy it when the text is an input. Fingerprint drift and boilerplate rules are continuous maintenance with no upside for a team whose product is the thing on top.

One thing worth checking before you commit: a schema-directed extractor is often the better tool for a RAG source. context.dev /web/extract takes a schema and returns JSON matching it, so instead of a page of markdown you get the four fields you actually wanted, already separated. For structured sources that is a materially cleaner input than any amount of post-processing.

📖 See also turning any URL into LLM-ready markdown for the extraction mechanics, and giving your agent live web context for the retrieval side.

What does a RAG pipeline need beyond the text?

Four things, and only one of them is content. The other three are what makes a retrieved chunk trustworthy.

A stable identifier. Something that says which document and which version this text came from. Wikipedia's API gives a revision and a pageid; a scrape gives a URL that may point at different content next month. Without it you cannot tell whether an answer was drawn from current information or from something you indexed in March.

A timestamp. When was this fetched. A RAG index silently ages, and staleness is invisible at query time unless the chunk carries its own date.

Structure, preserved. The heading path a chunk sits under is context a retrieval step cannot reconstruct. Splitting on character count throws it away; splitting on headings and storing the path keeps it, and it improves both retrieval and the answer.

Provenance you can show. A citable URL per chunk. This is the difference between an answer a reader can check and one they have to trust, and it is the single feature that makes an internal RAG tool credible to the people using it.

The mistake this list is meant to prevent: treating extraction as a text problem when it is a metadata problem. Getting readable text out of a page is largely solved. Knowing what that text was, when it was true and where it came from is what separates a RAG index people rely on from one they stop trusting after the first confidently wrong answer.

How to test a source before indexing all of it

Four checks, on one page, before a crawl commits you to a thousand:

Read the first thousand characters. Not a sample from the middle, the opening. That is where the chrome lives, and it tells you immediately how much of the file is not the article. Our Wikipedia scrape failed this check in the first line.

Count the ratio. Rough is fine: how much of the extracted length is the content you wanted. If it is under half, you need a boundary-aware step before indexing, and you have just saved yourself finding that out from bad retrieval results a week later.

Check two pages from the same site against each other. The parts that are identical are boilerplate by definition. This is the cheapest boilerplate detector there is and it needs no rules, no configuration and no third page.

Confirm you can produce a citation. For a chunk in the middle of the document, can you say which URL, which section and which version it came from. If the answer is only the URL, your index cannot support a citation and you should decide that now rather than after somebody asks where an answer came from.

All four run on one or two extraction calls, which on a per-call endpoint is close to nothing, and they answer the questions that otherwise surface as unexplained retrieval quality problems after the index is built.

Which endpoint should I use for which job?

Verified present on 2026-08-19 with monid discover. The billing column gives the shape rather than a figure; monid inspect prints the current figure for free.

The first row is not a joke and it belongs at the top. The last row is the one people forget: a large share of the documents worth indexing are PDFs, and running them through a parser endpoint is far less work than a local extraction stack that has to handle scanned pages.

When should you not use Monid?

The source publishes an API. Use it. Free, cleaner, versioned, and intended for this. That covers Wikipedia, most government data, many documentation platforms and every source with a real developer programme.

You are indexing your own content. Read it from your CMS or your repository, where you already have the structure and the identifiers. Scraping your own site to get text you already own is a strange amount of work.

Extraction is your product. Own the pipeline.

You need a managed vector store too. We return text. The chunking, the embeddings and the index are yours to build or buy, and a platform that does all of it may fit better than assembling the pieces.

And the caution about us. Our scrape of a Wikipedia page returned seventy-four thousand characters beginning with a navigation menu, with a 200 and valid markdown. Nothing in that response flagged that most of it was chrome. Read the first thousand characters of any new source before you index a thousand pages of it.

Conclusion

There is a free API to extract page content for RAG, and for the source that generated this question it is Wikipedia's own. The measurement was not close: 689 clean characters with a revision identifier against 74,552 that open with a navigation menu.

Two things matter more than which extraction vendor you pick. Check for an official API before comparing scrapers, because a publisher's own feed gives cleaner text, stable identifiers and a defensible legal position, and it is usually free. And extraction is a metadata problem, not a text problem: readable markdown is close to solved, while knowing which document a chunk came from, which version, and when it was fetched is what decides whether anyone keeps trusting the answers.

Start with the free part: check the source for an API, and where there is none, monid inspect shows the extraction schema before you spend. Then extract one page and read the first thousand characters before indexing the site. Begin at monid.ai.

FAQ

Last updated August 2026.