Crawl4AI: When Should You Run Your Own Crawler?
Crawl4AI does the parsing for free. The bill is the fleet you run around it. A decision rule for when to self-host a crawler and when to call one.
Copy this line to your agent to crawl a documentation site into Markdown without installing anything.
set up https://monid.ai/SKILL.md and use context.dev /web/crawl to turn a docs site into one Markdown document per page
Crawl4AI is the best free thing in its category and the reason people search for it is not the reason they end up needing something else. Its parsing is excellent, its Markdown output is genuinely LLM ready, and it costs nothing. What it does not ship, and has never claimed to ship, is an exit address: the IP your requests leave from, the browsers they run in, and somebody to care when a target starts refusing them. This guide separates the two halves so you can pay for only the one you actually lack, using Monid, the OpenRouter for agent tools.
What is Crawl4AI, and what is it used for?
Crawl4AI is an open source Python crawler that turns web pages into clean Markdown for language models. You give it a URL, it renders the page in a browser, strips the navigation and boilerplate, and hands back text a model can read without a parsing step in between. That last part is what separates it from the previous generation of crawlers, which returned HTML and left the conversion to you.
What it is genuinely good at
Three things, and they are the three that matter for retrieval pipelines. It produces Markdown rather than a DOM, so the output goes straight into a chunker. It handles JavaScript rendering, session reuse and hooks, so single page applications are not a special case. And it does structured extraction two ways, with a language model when the page is irregular and with plain CSS or XPath selectors when it is not, which is the cheaper path and the one its own tutorial spends the most time on.
The adoption is not marketing. We crawled the project's own statistics page with a crawl endpoint on 2026-08-25 and it reported 60,904 GitHub stars, 914,200 PyPI downloads in the latest month, 9.72 million cumulative, 1.41 million Docker pulls, and 6,217 forks across 57 contributors. That page carries its own freshness stamp of February 24, 2026, so treat the figures as of that date rather than today's.
What people are actually searching for
Look at what autocomplete completes after the library's name and a pattern appears immediately: crawl4ai api key, crawl4ai api token, crawl4ai authentication. Those are queries for a hosted service, typed at a library that does not have one. The project itself has noticed. Its documentation currently carries a banner for a closed beta cloud API, pitched as being more cost effective than existing solutions, which is a maintainer concluding in public that the open source half is not the whole product.
That is the honest frame for this whole comparison. The question is not open source versus paid. It is which half of the job you are short of.
📖 See also A Free API to Extract Page Content for RAG: Read This First
Why does running your own crawler cost more than it looks?
Because the software is the cheap part and it is the only part that is free. A crawler has four running costs and Crawl4AI removes exactly one of them.
The exit address is the actual product
Your crawler runs somewhere, and that somewhere has an IP address that a target can classify. A cloud VM's address is a datacentre address, which is the easiest category in the world to refuse, so the moment you point a self-hosted crawler at a defended site you are shopping for proxies. The official Crawl4AI tutorial makes this concrete without meaning to: the one hour walkthrough by the project's maintainer has a chapter list, and the last technical chapter before the outro is proxy rotation. The library's own introduction ends where the infrastructure bill begins.
We wrote up the full version of this argument in the rotating proxy guide, and the short version is that a proxy is an input priced by the gigabyte and billed on every attempt, including the attempts that failed.
The browser fleet is a service you now operate
Rendering JavaScript means running Chromium, and running Chromium at any concurrency means a pool of them, with memory limits, zombie process cleanup, and a restart policy. This is ordinary work and entirely doable. It is also a service, with an on call rotation, and it exists only so that some other job can read text off a page. Teams consistently underestimate this because the first ten pages work perfectly on a laptop.
Failure is silent, which is the expensive kind
Here is a real result rather than a warning. Crawling three pages of the Crawl4AI documentation on 2026-08-25 returned three successes and zero failures by the crawler's own count, but one of the three, the /core/ask-ai page, came back with HTTP 200, full metadata, a page title, and completely empty Markdown. Nothing errored. A pipeline that checked status codes would have written an empty document into its index and moved on.
That failure mode is not specific to any one tool, and it is the reason crawl output needs a content length assertion rather than a status check. It is also the reason a per page success counter is worth less than it looks: the page succeeded, the content did not.
Old way vs endpoint: what actually differs
| Aspect | Self-hosted Crawl4AI | A crawl endpoint |
|---|---|---|
| Parsing quality | Excellent, and yours to tune | Fixed, and good enough for most ingestion |
| Exit address | You buy and rotate proxies | Included in the call |
| Browser fleet | You run it | Somebody else runs it |
| Blocked requests | You pay for the attempt | Not billed on a failed fetch |
| Cost shape | Servers, proxies, and engineer time | Per page actually returned |
| Best for | High volume on friendly targets | Bursty volume on defended targets |
The pattern is that everything you keep by self hosting is control over parsing, and everything you buy is somebody else's problem with access.
How do you crawl a site without hosting a crawler?
Three steps, and none of them installs a browser. The example below crawls a documentation site into one Markdown document per page, which is the single most common thing people want a crawler for.
For agents
Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:
set up https://monid.ai/SKILL.md
It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.
For humans
npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main
Step 1. Find the crawl endpoint
What it does. Searches the catalog for a capability by description rather than by vendor name, which matters when you do not already know who sells what.
The endpoints. context.dev/web/crawl follows links from a starting URL and returns Markdown for every page it reaches.
The call.
monid discover -q "crawl an entire website to markdown"
monid inspect -p context.dev -e /web/crawl
What comes back. A ranked list with the provider, the price and a verified flag, then the full input schema for whichever one you pick. Both commands are free, so you can read the exact current price before anything bills.
What it costs. Nothing. Discovery and inspection never bill; only run does.
Step 2. Crawl the site
What it does. Walks the site from a starting URL, respecting a page cap and a link depth, and converts each page it reaches.
The endpoints. context.dev/web/crawl, billed per page actually returned.
The call.
monid run -p context.dev -e /web/crawl -w -i '{
"url": "https://docs.example.com/",
"maxPages": 50,
"maxDepth": 2,
"useMainContentOnly": true
}'
What comes back. A results array with one entry per page, each carrying markdown plus a metadata object holding sourceUrl, finalUrl, title, description, language, canonicalUrl, a parsed headings list with levels, crawlDepth and statusCode. Alongside it, a crawl summary with numUrls, numSucceeded, numFailed and numSkipped.
The headings array is the underrated field. It gives you the page outline without parsing the Markdown back, which is exactly what a chunker wants.
What it costs. A small fraction of a cent per page returned, and the billing tracks pages actually scraped rather than pages requested, so failed and skipped pages are free. Current figures are on monid.ai/tools.
Step 3. Assert on content, not on status
What it does. Catches the silent failure described above before it reaches your index.
The call. There is no endpoint for this. It is four lines in whatever consumes the output:
for page in result["results"]:
md = page.get("markdown") or ""
if len(md) < 200:
log_and_skip(page["metadata"]["sourceUrl"])
continue
index(md, page["metadata"])
What comes back. A count of pages you refused, which is the number worth alerting on. A crawl whose empty page rate moves from two percent to thirty is telling you something a success counter never will.
What it costs. Nothing, and it is the highest value four lines in the pipeline.
📖 See also Any URL to LLM-Ready Markdown: A Copy-Paste Cookbook
Give this to your agent
Set up https://monid.ai/SKILL.md, and then use Monid to crawl https://docs.example.com to a depth of 2, keep only main content, and tell me which pages came back with fewer than 200 characters of markdown.Crawl4AI vs Firecrawl vs a crawl endpoint: what actually differs?
The comparison people search for is Crawl4AI versus Firecrawl, and it is a fair fight on output quality. It is the wrong axis anyway. What separates these options is not how good the Markdown is, it is who is responsible when a page will not load.
| Crawl4AI | Firecrawl | A crawl endpoint | |
|---|---|---|---|
| What you install | Python package or Docker | Nothing, or the open source version | Nothing |
| Who runs the browsers | You | Them | Them |
| Who buys the proxies | You | Included | Included |
| Signup required | No | Per vendor | One key for the whole catalog |
| Cost when idle | Your servers | Plan minimum, if any | Nothing |
| Callable by an agent that has never heard of it | No | Only if pre-wired | Yes, via discover |
That last row is the only one that is genuinely new. A library has to be imported by a developer before it can be used, and a hosted vendor has to be signed up for. An endpoint in a catalog can be found, read and called by a program at run time, which is what makes it usable by an agent handling a site nobody anticipated. The wider version of that argument is in Web Scraping Tools: Which Kind Do You Actually Need?.
The comparison against Scrapy and Playwright is a different question and a shorter one. Scrapy is a crawling framework and Playwright is a browser driver; Crawl4AI sits on top of a browser and adds the Markdown conversion. If you are choosing between them you are choosing how much to build, not what to buy, and the same reasoning applied to the Python stack generally is in Web Scraping in Python Without Maintaining a Scraper.
One case deserves calling out because a crawler is the wrong shape for it. Crawling a news publisher's archive works, but ongoing news coverage is a search job rather than a crawl job, and the difference is worth real money at any volume. That split is worked through in Web Scraping News Articles.
Which endpoint should I use for which job?
| Endpoint | What it does | Input | Output | Best for | Billing |
|---|---|---|---|---|---|
context.dev/web/crawl | Follows links, converts every page | Start URL, page cap, depth, URL regex | Markdown plus metadata per page, and a crawl summary | Ingesting a docs site or a whole section | Per page returned |
context.dev/web/scrape/markdown | Converts one known URL | A single URL | Markdown, contentLength, metadata, JSON-LD | You already have the URL list | Per call |
context.dev/web/scrape/sitemap | Lists a site's URLs | Site URL | URL list | Deciding what to crawl before paying to crawl it | Per call |
context.dev/web/extract | Crawls and fills a JSON Schema | Start URL plus a schema | Typed JSON, not Markdown | You want fields, not prose | Per result |
context.dev/web/search | Finds URLs, optionally with content | A query | Ranked results with relevance | You do not have the URLs yet | Per result |
Every row was verified with monid inspect on 2026-08-25. Billing shape is what belongs in a table; the figures move, and current ones are always on monid.ai/tools.
The one people miss is sitemap. Enumerating a site's URLs costs a fraction of crawling it, and it turns "crawl this site and hope" into a list you can filter before spending anything.
Worked through on a real shape: a documentation site of roughly 400 pages, refreshed monthly for a retrieval index. Listing the URLs is one call. Filtering to the 250 pages that are documentation rather than changelog costs nothing. Crawling those 250 bills per page returned, in fractions of a cent, so twelve refreshes a year stay under the cost of one afternoon spent debugging a proxy pool. The self hosted version of that same job is free software plus a proxy allowance, plus an always on instance, plus the engineer who gets paged when Chromium starts leaking, and that last number dominates the other two.
When should you run Crawl4AI yourself?
There are three cases where self hosting is clearly correct, and they are not edge cases.
Your targets are friendly and your volume is large. If you are crawling your own properties, a partner's site, or documentation that wants to be read, there is no access problem to buy your way out of. At millions of pages a month against undefended targets, a fleet you run is genuinely cheaper than per page billing, and Crawl4AI is a very good crawler to build that fleet on.
You need the parsing to be yours. Custom extraction schemas, per site rules, chunking that respects a specific document structure. A hosted endpoint gives you one good conversion; a library gives you the conversion you wrote. If the difference between them is the product you are building, keep the library.
Nothing may leave your network. Regulated data, air gapped environments, internal wikis that cannot be sent to a third party. This is not a cost decision and no amount of per call convenience answers it.
There is also a case where a different vendor wins outright. If you need enormous sustained volume through residential exits with fine grained geographic targeting, Bright Data has infrastructure at a scale nobody in this comparison matches, and paying them for that is the right call. Per call access is optimised for the opposite shape: bursty, unpredictable, many different sites, no commitment.
And for the record, you are reading this on Monid's blog. We sell per call access to tools, so the case we argue best is the one where you have not yet decided to become an infrastructure team. If you already are one, the honest answer is that Crawl4AI is excellent and free and you should use it.
Conclusion
The question is not whether Crawl4AI is good. It is very good, and the parsing is the part of the problem it solves completely. The question is whether you also need the part it does not solve, which is a browser fleet with a defensible exit address and somebody to maintain both. Crawling forty pages of friendly documentation needs none of that; crawling four hundred pages of a site that would rather you did not needs all of it.
The thing worth noticing more than the choice itself is how failure shows up. Blocking rarely announces itself as an error, which is why the most valuable line in any crawl pipeline is the one that checks whether the content came back rather than whether the request did. That holds whichever side of this decision you land on.
Free next step: run monid discover -q "crawl a site to markdown" and monid inspect the top result. Both cost nothing and take a minute, and you will know the current price and the exact schema before you spend anything. Start at monid.ai.
FAQ
How do you install Crawl4AI?
pip install crawl4ai followed by the post install browser setup, or run the official Docker image if you would rather not manage Python and Chromium on the same machine. The Docker route is the one to prefer for anything scheduled, because the browser dependencies are the part that breaks on upgrade. Neither path gives you proxies, which is the piece most people discover they are missing on the first defended site.
Is there a Crawl4AI MCP server?
Yes, the project ships an MCP server so a model context protocol client can drive the crawler directly. It exposes your local crawler, which means the agent inherits your machine's IP address and your proxy configuration, or the absence of one. If the goal is an agent that can reach arbitrary sites, that is the constraint to plan around, and an MCP gateway that fronts many tools behind one credential solves a different half of the same problem.
How does Crawl4AI convert HTML to Markdown?
It renders the page in a real browser, applies content filtering to drop navigation, headers, footers and sidebars, then converts the remaining DOM subtree to Markdown, with options for whether links and images survive the trip. The main content detection is the part that varies most between tools, and it is worth testing on your actual targets rather than on a blog post, because news sites and documentation sites fail in opposite directions.
What are the alternatives to Crawl4AI?
Firecrawl and Scrapling are the closest open source comparisons, Scrapy is the answer if you want a crawling framework rather than a Markdown converter, and a hosted crawl endpoint is the answer if what you are missing is access rather than parsing. Sort them by which half of the job you already have covered, because on output quality the modern options are close enough that the difference will not decide anything.
Last updated August 2026.
