Crawl4AI API: Crawl a Site to Markdown Without Hosting

Crawl a documentation site into Markdown, one document per page, without installing a crawler. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to crawl https://docs.crawl4ai.com/ (5 pages, one link deep), give me one markdown document per page, and tell me which pages came back with fewer than 200 characters of markdown.

One prompt crawls a site and flags the empty pages. 1 min setup · 10-22 hrs saved.

You searched for a Crawl4AI API or a Crawl4AI API key, and found a Python library that runs on your own machine. It parses well and costs nothing, but the browser, the exit address and the upkeep are yours. This guide shows the other route: crawl a whole site into Markdown with one call from your agent, what comes back (we ran it live on 2026-10-16 against the Crawl4AI docs), and the traps in the output. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

Is there a Crawl4AI API key, and what is the easiest way to get a crawl API?

The Crawl4AI library takes no API key, because it runs inside your own Python process, and the easiest way to get a crawl behind a key is Monid. Install it once, add a key, and your agent calls a crawl endpoint with a start URL and gets one Markdown document per page back as typed JSON.

• Pay for what you run. A crawl is billed per page actually scraped, at $0.002533 a page (Verified 2026-10-16 on the web extraction tools page). Failed and skipped pages are not billed. Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and the same balance call the crawler and every other endpoint on Monid, so a site that needs one page, a URL list or a search is another call on the same balance.
• Lives inside your agent. Install the CLI, add your key, run the prompt above, and the crawler is there whenever your agent needs it, with no browser pool and nothing to keep running. Your agent reads SKILL.md and learns the rest, so setup is about a minute.

At that price, a 100 page crawl costs about $0.25 (Verified 2026-10-16).

How do you crawl a whole site to markdown without hosting a crawler?

Send the start URL to the crawl endpoint with a page cap and a depth, and read the Markdown out of the response. The endpoint is /web/crawl from the provider context.dev. It follows links from the URL you give it and converts every page it reaches.

Monid fits a one-off or bursty crawl because the work is one call and you pay for the pages that came back, and nothing in between.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Find the endpoint, then crawl

monid discover -q "crawl an entire website to markdown"
monid inspect -p context.dev -e /web/crawl

Both are free. The schema lists url as the one required field, plus maxPages (default 100, hard cap 500), maxDepth, urlRegex, followSubdomains and useMainContentOnly. Then the run, which is the only step that bills:

monid run -p context.dev -e /web/crawl -w -i '{
  "url": "https://docs.example.com/",
  "maxPages": 50,
  "maxDepth": 2,
  "useMainContentOnly": true
}'

We ran the same shape against the Crawl4AI documentation with maxPages 5 and maxDepth 1: five pages came back, numSucceeded was 5, and the call took about 22 seconds. Set maxPages to the number you need before you start, because the endpoint holds the price of maxPages while it runs.

Read the total

The summary sits next to the pages. numUrls, numSucceeded, numFailed and numSkipped tell you what the crawl did, and billing follows numSucceeded. Five pages billed five units, $0.012665 in total (Verified 2026-10-16 with monid inspect).

Check the pages before you index them

Put the check in whatever consumes the output:

for page in result["results"]:
    md = page.get("markdown") or ""
    if len(md) < 200:
        log_and_skip(page["metadata"]["sourceUrl"])
        continue
    index(md, page["metadata"])

The goes-wrong section below says why the length check is not optional.

Can you run Crawl4AI yourself with Python and Docker?

Yes, and for many jobs you should. Crawl4AI is an open-source Python crawler: its own docs page gives the install as pip install -U crawl4ai followed by crawl4ai-setup, which installs the browser once, and shows a few lines of AsyncWebCrawler that print a page as Markdown. There is also a Docker server for people who would rather not manage Python and Chromium on one machine, and the project states that it forces no API keys.

What you take on with it:

• The browser. Rendering JavaScript means running Chromium, and running it at any concurrency means a pool with memory limits and a restart policy.
• The exit address. The crawler leaves from your IP. A cloud VM address is a datacentre address, the category a defended target refuses first, so a self-hosted crawler pointed at such a site means buying and rotating proxies. The longer version is in do AI agents need a rotating proxy.
• The upkeep. Target sites change, the library has frequent releases, and nobody but you is paged when a run comes back empty.
• The rules. Respect robots.txt and each site's terms, whoever runs the crawler.

Monid maintains the rendering, the proxies and the conversion for the crawl endpoint, so what is left to you is the start URL, the cap and the check on the output. The Crawl4AI parser itself is the thing you give up, and for ordinary documentation and article pages the hosted conversion does the same job.

What does a crawl return, and what can you build with it?

A crawl returns a results array with one entry per page, plus a summary. Here is what the Crawl4AI docs crawl returned on 2026-10-16.

The layers of the response

Each result carries markdown, a url, a crawlDepth, a statusCode and a success flag. Next to them sits a metadata object: sourceUrl, finalUrl, title, description, language, canonicalUrl, favicon, additionalMeta and a headings list, each heading with a level and its text. The summary has numUrls, maxCrawlDepth, numSucceeded, numFailed and numSkipped, and a cache_metadata object whose status was miss on our run.

The headings list is the underrated field. It gives you each page outline without parsing the Markdown back, which is what a chunker wants.

What you can build with it

• A retrieval index of a documentation site. Crawl the docs once, chunk each page by its headings, embed the chunks. The same shape for any source is in a free API to extract page content for RAG.
• A batch of known pages. When you already hold the URLs, skip the crawl and convert them one by one; any URL to LLM-ready markdown shows the batch route.
• A change monitor. Crawl on a schedule, keep the canonicalUrl as the key and diff the Markdown to see what moved.
• A news archive. Crawl a publisher's archive section once; for ongoing coverage a search call is the better job than a recurring crawl.

What goes wrong when you crawl a site to markdown?

Pages can succeed without delivering content, and the numbers inside a page can be old. Three things showed up in our run.

Why does a successful page come back with empty markdown?

On the second page of our crawl, /core/ask-ai, the result had success true, statusCode 200, a title and empty markdown. The summary counted five successes and no failures, so a pipeline that watched status codes would have indexed an empty document. The page is an interface, and the crawl read none of it as content.

We then fetched the same URL with the single-page endpoint, with waitForMs set to 5000 and maxAgeMs set to 0. It returned 6,512 characters, and they were the navigation menu and a site banner, not the page's tool. The rule that follows: assert on content length, skip and log what is short, and judge a page by what you can use, not by whether it loaded.

Why do the numbers on a page not match today?

The crawl returned the project's stats page with 60,904 GitHub stars, 914.2K monthly downloads, 9.72M total downloads and 1.41M Docker pulls, and the page says it was updated on February 24, 2026. The page header we fetched on 2026-10-16 showed 85.7k stars. Crawled text is a snapshot of when the site last published it, so store the sourceUrl and the fetch date with every document and never quote a crawled figure without its own date.

Why does main-content-only still leave widget text?

We set useMainContentOnly to true, and the quickstart page still ended with page-widget text such as "Page Copy" and "ESC to close", and the headings list for each page contained entries such as "On this page" at level 4 and "Search" at level 5. The option drops headers, footers and sidebars when it can detect them, which is not the same as clean. Strip lines you recognise before you chunk, and drop headings at level 4 and below if you build an outline from them.

Which endpoint should I use for which job?

Use the crawl endpoint for a section or a site, the single-page endpoint for a URL you already have, and the sitemap endpoint to list URLs before you pay to crawl them.

Prices Verified 2026-10-16 with monid inspect. We ran all three: the sitemap call on docs.crawl4ai.com with maxLinks 50 returned 50 URLs, the cap, from one sitemap with no errors, for the price of one call.

That makes the sitemap endpoint the cheap first step. It turns "crawl this site and hope" into a list you can filter with urlRegex before you spend on pages. For a site with a map of its own, finding all URLs on a domain covers the options.

When is another route a better fit?

Your targets are friendly and your volume is large. If you crawl your own properties or documentation that wants to be read, there is no access problem to buy your way out of. At millions of pages a month, a fleet you run is cheaper than any per-page price, and Crawl4AI is a good crawler to build it on.

You need the parsing to be yours. Custom extraction schemas, per-site rules and chunking tied to a document structure belong in a library you control. A hosted endpoint gives you one conversion; a library gives you the one you wrote.

Nothing may leave your network. Regulated data, air-gapped environments and internal wikis cannot go through a third party, and that is not a cost decision.

The job is a click-through, not a crawl. If the data sits behind a login or a form, a browser script is the tool, and browser automation without an API covers when to check for an endpoint first.

You want the Crawl4AI parser on someone else's browsers. The project now offers its own hosted version, with a free monthly allowance, and that is the direct route if the Crawl4AI output format is the thing you are attached to.

And the disclosure: we resell the crawl endpoint. The habit that is worth having whoever you buy from is the length check on every page.

Conclusion

The quickest way to crawl a whole site into Markdown without running a crawler is one call through Monid: $0.002533 a page returned (Verified 2026-10-16), one key for this and every other tool, about a minute to set up and ready inside your agent whenever you need it. Running Crawl4AI yourself is free software on your own machine; Monid keeps the browsers, the exit address and the conversion maintained.

Two things matter more than the choice. A page that loaded is not a page that delivered: our crawl reported five successes and no failures while one page had no content, so check length, not status. And a crawled number is as old as the page it sits on, so keep the source and the date next to every document.

How we estimated the time saved: standing up Docker, a browser pool and a proxy source is roughly 3 to 6 hours, getting rendering and blocked pages to behave is 4 to 8 hours, and the first month of upkeep is 3 to 8 hours. That is 10 to 22 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid discover -q "crawl a site to markdown" and monid inspect -p context.dev -e /web/crawl, then crawl five pages of a site you know and compare the page count with what you expected. Start at monid.ai.

FAQ

Published August 2026.