Ingesting Mixed Documents for LLM Embedding: PDF to Markdown

Copy this line to your agent to turn a folder of mixed files into clean Markdown.

set up https://monid.ai/SKILL.md and use context.dev /parse to convert these PDFs and Office files to Markdown

Every retrieval pipeline has the same first problem and almost nobody writes it down: the documents are not one format. A knowledge base is PDFs, DOCX, a few spreadsheets, some scanned faxes and a pile of web pages, and the embedding model wants one clean text stream. The conversion step is where quality is won or lost, well before chunking or retrieval, and it is the step that gets three lines of code and no tests. Monid is the OpenRouter for agent tools: one key and one balance across many providers, so the conversion step can be one endpoint instead of six libraries.

Why is ingesting mixed document types harder than it looks?

Because format conversion is not one problem, it is four, and they fail at different points. The r/LocalLLaMA thread that names this best lists them in the title: PDF, Office and HTML conversion, OCR, de-duplication and chunking. Twelve comments in, the consensus is that people underestimate the first and over-engineer the last.

A PDF is a layout format, not a document format

PDF describes where marks go on a page. It does not describe reading order, and a two-column academic paper or a table-heavy report will extract into interleaved nonsense if the extractor reads coordinates naively. The text is all there and it is in the wrong order, which is worse than missing text, because it embeds cleanly and retrieves garbage.

This is the failure that survives all the way to production, because nothing errors. A chunk of interleaved column text is a valid chunk with a valid embedding. It just answers no question correctly.

Scanned documents have no text at all

A scanned contract or a fax is an image inside a PDF wrapper. A text extractor returns an empty string and reports success. If your ingestion pipeline logs a page count rather than a character count, an entire scanned archive can pass through it and produce nothing, silently.

The check that catches this is one line: assert a minimum character count per page, and route anything below it to OCR rather than to the embedder.

HTML brings furniture you did not ask for

Web pages carry navigation, footers, cookie banners and sidebars, and all of it embeds. We measured the extreme version of this in a free API to extract page content for RAG: a Wikipedia page came back at 74,552 characters through a naive scrape, starting with the nav menu, against 689 clean ones through the structured route. Those extra characters are not merely wasted tokens, they dilute the embedding of every chunk they land in.

One library per format versus one conversion endpoint

The pattern is control against surface area. Local libraries are the right answer when documents cannot leave your network, and the wrong answer when the real cost is six half-maintained format handlers.

📖 See also Any URL to LLM-Ready Markdown: A Copy-Paste Cookbook

What are the best practices for ingesting mixed document types for LLM extraction?

Convert first, normalise second, chunk last, and put the assertions between the steps rather than at the end. The order matters because each step can fail quietly, and a check after conversion costs nothing while a check after embedding costs a reindex.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-key> -l main

Step 1. Convert every format through one door

What it does. Takes a file at a URL and returns GitHub Flavored Markdown, across more than sixty formats, so the branch in your code is data rather than control flow.

The endpoints. context.dev/parse handles PDF, DOCX, XLSX, PPTX, RTF, HTML, images, code files and structured data including JSON, CSV, YAML and XML, from any public HTTPS URL up to 25MB.

The call.

monid inspect -p context.dev -e /parse
monid run -p context.dev -e /parse \
  -i '{"file_url":"https://example.com/report.pdf","useMainContentOnly":true}' -w

What comes back. The parsed document as Markdown. Four switches shape it, verified 2026-08-20: includeLinks preserves hyperlinks and defaults on, includeImages adds image references and defaults off, shortenBase64Images truncates inline image payloads and defaults on, and useMainContentOnly drops headers, footers, sidebars and navigation where they can be detected. That last one is the HTML furniture problem solved with a boolean.

There is also an extension hint for when neither the URL nor the Content-Type reveals the format, which is the case more often than you would like with files pulled from object storage.

What it costs. A fraction of a cent per call, billed per call rather than per page, so a two hundred page report costs the same as a one pager. Current figures at monid.ai/tools.

Step 2. Route the scanned files to OCR, and only those

What it does. Detects and reads text from images embedded in PDF pages, which is the only way to get anything at all from a scanned archive.

The endpoints. The same context.dev/parse call with ocr: true. For loose images rather than PDFs, api.strale.io/x402/image-to-text runs OCR through vision and returns text with a confidence score, which is useful when you need to threshold on quality.

The call.

monid run -p context.dev -e /parse \
  -i '{"file_url":"https://example.com/scanned-contract.pdf","ocr":true}' -w

What comes back. The Markdown, plus an ocr_ran field telling you whether OCR actually executed. That field is worth reading rather than ignoring, because it is also how the billing settles.

What it costs. This is the one genuinely clever piece of billing in the ingestion path. Setting ocr: true holds the higher amount, but the OCR line only settles if OCR actually ran, detected from the vendor's own meter. A file with nothing to OCR bills the base rate. So you can set the flag across a mixed batch without paying the OCR price for the text-layer files, which means you do not need to pre-classify the batch yourself.

Step 3. Assert, then chunk

What it does. Catches the silent failures before they become embeddings.

Three assertions cover almost everything that goes wrong. Check a minimum character count per source page and route failures to OCR. Check that the Markdown contains at least one heading if the source had structure, because a document that converted to one undifferentiated block usually lost its reading order. And hash the normalised text before embedding, because de-duplication is far cheaper on strings than on vectors.

Chunk after all of that. Chunking a bad conversion produces bad chunks efficiently.

📖 See also A Free API to Extract Page Content for RAG and The Best OCR API for Messy Images in 2026

How do you convert a PDF to Markdown?

One call, and the interesting parts are what happens when the PDF has no text layer and where the output actually lives.

The straightforward case

A born-digital PDF, one where the text is real text rather than a picture of text, converts directly. Running context.dev/parse against a public research paper on 2026-08-26 returned success: true, type: "pdf", and a document object holding a signed download_link with content_type: "text/markdown".

monid run -p context.dev -e /parse -w -i '{"file_url": "https://example.com/report.pdf"}'

Two fields in that response are worth reading before you design anything around it. The Markdown comes back as a link rather than as a string, and the link carries link_expires_at about an hour out with file_expires_at seven days out. Fetch the bytes in the same job that produced them. A pipeline that stores the URL and reads it next week has stored nothing.

The scanned case, and how you know

The same run returned ocr_ran: false and ocr_units: 0, because that document had a text layer and nothing needed optical recognition.

That pair of fields is the honest answer to the scanned-PDF problem. You pass ocr: true on documents that might be scans, and the response tells you afterwards whether it actually ran. The billing follows the same logic: the endpoint is TIERED, the base call is one credit, an OCR-executed call totals five, and passing ocr: true holds the larger amount but settles the smaller one when the file turned out to have text after all.

So the safe default on a mixed folder is to turn OCR on and let the meter decide, rather than trying to detect scans yourself first. Detecting them costs a read anyway, and getting it wrong costs a silent empty document.

Python, and why the library is not the hard part

The most common version of this question asks how to do it in Python, and the honest answer is that Python has good libraries for it. PyMuPDF, pdfplumber and marker all convert well, and marker in particular produces very clean Markdown.

What none of them ships is the OCR engine for the scanned pages, the layout model for multi-column academic papers, or somebody to keep both current. That is the same split we drew for web pages in Web Scraping in Python Without Maintaining a Scraper: keep the code that is genuinely yours, and buy the part that is infrastructure. Calling the endpoint from Python is four lines and leaves your existing pipeline intact.

What about converting for Claude or another model?

Nothing special is required, which is the point of converting at all. Markdown is what every current model reads most reliably, so the same output serves a chunker, a retrieval index and a prompt you paste by hand. If the document is going straight into a context window rather than into an index, set useMainContentOnly to drop running headers and page furniture, and skip the chunking half of this guide entirely.

Does converting to Markdown actually save tokens?

Yes, and the size of the saving is the part people get wrong in both directions. Two separate builders on r/LocalLLaMA and r/LLMDevs shipped HTML-to-Markdown converters this year with the same headline claim, roughly two thirds fewer tokens than raw HTML. That number is believable for a content-heavy web page and misleading as a general rule.

The saving comes from deleting markup, not from compressing prose. So it is large for HTML, where tags and attributes can outweigh the text, and near zero for a DOCX whose content was already mostly words. If your corpus is web pages, the conversion pays for itself in embedding costs alone. If it is Office documents, convert for consistency rather than for tokens, and do not budget a saving that will not arrive.

The second-order effect is bigger than the token count anyway. Markdown keeps headings, lists and table structure as text, which means a chunker can split on semantic boundaries rather than on character counts. A chunk that starts at a heading retrieves better than a chunk that starts mid-sentence, and that improvement does not show up in a token comparison at all.

Which endpoint should I use for which job?

Every row verified with monid inspect on 2026-08-20. The billing column is the shape rather than a figure, because per call and per result change how you batch and a price does not stay true.

What does an ingestion run actually cost?

Less than the embeddings, in almost every case, which is why the conversion step deserves more attention than its budget line suggests.

A corpus of a few thousand mixed documents converts for single-digit dollars, because the main conversion endpoint bills per call rather than per page and most documents are one call. The number that moves is OCR, and only for the files that genuinely need it, since the higher rate settles only when OCR actually ran.

The comparison worth making is against the embedding bill rather than against doing it yourself. If a naive HTML extraction inflates a page from 689 characters to 74,552, you pay that inflation once in conversion and then again on every embedding and every retrieval that includes the diluted chunk. Cleaning at the door is the cheapest place in the pipeline to fix it.

Discovery and inspection are free, so the whole format list, every switch and the exact billing behaviour are readable before spending anything. That is the property that makes a metered balance suit an ingestion job: access costs nothing until it is used, so a one-off corpus load does not need a plan. Prices at monid.ai/tools.

When should you not use Monid?

If the documents cannot leave your network, run it locally and accept the maintenance. Regulated corpora, client-confidential files and anything under a data residency commitment belong in a local pipeline, and the honest answer is that six format libraries and a native OCR build are the price of that constraint. No hosted endpoint solves a rule that says the bytes stay put.

If you have one format and one shape, use the library. A pipeline that only ever sees clean text-layer PDFs from one generator does not need a general conversion service. A well-chosen local parser will be faster and free.

If your volume is enormous and steady, the arithmetic changes. Per-call conversion is excellent for a corpus load and for a steady trickle of new documents, and it stops being the cheapest option somewhere above a sustained high rate where a self-hosted converter on your own compute wins on unit cost.

And if you need conversion accuracy guarantees for legal or medical documents, no general endpoint provides them. Specialist vendors sell validated extraction with an accuracy commitment attached, and that commitment, not the conversion, is what you would be buying.

Conclusion

The best practice for ingesting mixed document types is to treat conversion as its own step with its own tests, rather than as a preamble to chunking. Convert everything through one door, turn OCR on across the whole batch because it only bills when it runs, assert on character counts and structure before embedding anything, and chunk last.

What matters more than the tool choice: the failures in this pipeline do not raise exceptions. A scanned PDF returns an empty string, a two-column paper returns interleaved text, and an HTML page returns a navigation menu. All three embed successfully and retrieve badly, and none of them appear in a log. Three assertions between conversion and chunking catch all three, and they are the cheapest code in the whole system.

The free next step costs nothing. Run monid discover -q "parse pdf document to text" to see what exists, monid inspect -p context.dev -e /parse to read the full format list and the OCR billing behaviour, then one paid run on the ugliest ten documents in your corpus rather than the cleanest. Start at monid.ai.

FAQ

Last updated August 2026.