C# Web Scraping: Keep HtmlAgilityPack, Drop the Fetch

Copy this line to your agent to fetch a page from .NET without maintaining a scraper.

set up https://monid.ai/SKILL.md and use context.dev /web/scrape/markdown to fetch pages for a .NET service

Every article ranking for C# web scraping is a library roundup, and they are correct about the libraries. HtmlAgilityPack and AngleSharp are good, .NET is a perfectly reasonable language for this, and none of that addresses why C# scrapers stop working. They stop because a request left your server and was refused, which is not a parsing problem and not a language problem. This guide keeps the .NET half and replaces the half that was never about .NET, running through Monid, the OpenRouter for agent tools.

What do you use for web scraping in C#?

The honest stack, stated plainly, because the pages ranking for this term have it right.

HtmlAgilityPack for parsing

The default and still the pragmatic choice. Forgiving about malformed markup, XPath support, and enough of the ecosystem depends on it that it will not disappear. If you are selecting fields out of HTML in .NET, this is where you start.

AngleSharp for standards correctness

A proper HTML5 parser with a DOM that behaves the way a browser's does, and CSS selector support that feels like the front end. Slower to learn, more correct on complicated documents. The FAQ below has the actual selection rule.

HttpClient for the request

Built in, fine, and the source of the single most common .NET-specific bug in this domain, which the FAQ covers because it looks exactly like being blocked and is not.

Playwright for .NET when the page needs a browser

Officially supported, works well, and is the correct tool when the content only exists after JavaScript runs. It is also the heaviest thing on this list and there is a section below about checking whether you need it.

What none of them do

Provide an address the target will accept. Every package above runs on your machine, from your IP, and the moment the target starts refusing that IP, no amount of correct parsing helps. That is the gap this article is about and it is the one the roundups do not mention because it is not a library. The four kinds of tool and who has to operate each are sorted in Web Scraping Tools: Which Kind Do You Actually Need?.

📖 See also Web Scraping in Python Without Maintaining a Scraper

Why does a C# scraper break, and is it the language's fault?

No, and that is the useful finding. Sort the failures and none of them is about .NET.

It gets refused

A 403, a challenge page, or a 200 containing something that is not the page. This is a classification decision made about your IP address before anything in your code runs. Rewriting the parser changes nothing. The address is a datacentre address if the service is on a cloud instance, which is the easiest category to refuse, and the economics of changing that are in Do You Still Need a Rotating Proxy in 2026?.

The content was never there

An empty selector result because the markup arrives from JavaScript after load. HtmlAgilityPack parsed exactly what it was given, which was a shell. This is a rendering problem and it is the one Playwright genuinely solves.

The markup moved

A selector that matched last month and matches nothing now. This is the maintenance cost of owning a parser and it is real, ongoing and unavoidable if the parser is yours.

It looks like blocking and is a socket bug

Specific to .NET and worth its own line. Creating a new HttpClient per request exhausts sockets under load, and the resulting timeouts and connection failures look exactly like rate limiting. Teams reach for proxies to fix a problem that was a using statement. The FAQ has the fix.

Where the language actually helps

Two places, and they are real. .NET's async model and Parallel.ForEachAsync make concurrent fetching genuinely pleasant compared to several alternatives, and static typing over a response you have modelled catches schema drift at compile time rather than in production. Neither of those advantages is in the fetching layer, which is the point.

Own the stack vs buy the fetch: what actually differs

The last row of code is the honest summary: you keep the part that was your product and drop the part that was infrastructure wearing a NuGet package.

How do you call a fetch endpoint from C#?

Three steps, two of them free, and the third is ordinary .NET.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Step 1. Read the schema before writing a model class

What it does. Gives you the exact response shape so your record type matches on the first try rather than after three runs.

The call.

monid discover -q "fetch a page as clean markdown"
monid inspect -p context.dev -e /web/scrape/markdown

What comes back. The full input and output schema, plus the pricing note. Reading it on 2026-08-31, that note states JavaScript rendering, anti-bot handling and premium proxies are included, and that failed or blocked requests are not billed.

What it costs. Nothing.

Step 2. Call it from a typed client

What it does. The fetch, from .NET, with none of the infrastructure.

The code. An IHttpClientFactory client, which is also the fix for the socket problem above:

public sealed record ScrapeResult(string Markdown, int ContentLength);

public sealed class MonidClient(HttpClient http)
{
    public async Task<ScrapeResult?> ScrapeAsync(string url, CancellationToken ct)
    {
        var body = new { provider = "context.dev", endpoint = "/web/scrape/markdown",
                         input = new { queryParams = new { url, useMainContentOnly = true } } };

        using var res = await http.PostAsJsonAsync("v1/run", body, ct);
        res.EnsureSuccessStatusCode();

        var run = await res.Content.ReadFromJsonAsync<JsonElement>(cancellationToken: ct);
        var output = run.GetProperty("output");
        return new ScrapeResult(
            output.GetProperty("markdown").GetString() ?? "",
            output.GetProperty("contentLength").GetInt32());
    }
}

Register it once and the factory handles the socket lifetime for you:

builder.Services.AddHttpClient<MonidClient>(c => {
    c.BaseAddress = new Uri("https://api.monid.ai/");
    c.DefaultRequestHeaders.Authorization = new("Bearer", config["Monid:ApiKey"]);
});

What comes back. Clean Markdown plus metadata: title, canonical URL, language, a parsed heading tree, Open Graph and the page's JSON-LD when it publishes any.

What it costs. A fraction of a cent per successful call, with blocked fetches unbilled. Current figures at monid.ai/tools.

Step 3. Assert on content, not on status

What it does. Catches the failure that has no exception attached.

The code.

if (result is null || result.ContentLength < 200)
{
    logger.LogWarning("Thin response for {Url}: {Length} chars", url, result?.ContentLength ?? 0);
    return null;   // do not index this
}

The same three-step shape written out end to end, in a different language, is in Any URL to LLM-Ready Markdown.

What comes back. A count worth alerting on. EnsureSuccessStatusCode passes happily on a well-formed response containing a challenge page, which is why the length check rather than the status is what protects the pipeline.

📖 See also Your Scraper Is Blocked: What Actually Gets Through in 2026

Do you need Playwright for .NET?

Sometimes, and less often than the roundups imply, because they answer the question by adding a package rather than by testing.

The test that takes two minutes

Fetch the page without a browser and look for the content you want. If it is in the markup, you never needed a browser and adding one costs you a Chromium process, a container that has to ship browser binaries, and a memory profile that makes your service harder to host.

A large share of pages that look dynamic serve their content server-rendered and hydrate afterwards. The visible interactivity is not evidence that the data arrives late.

When it is genuinely required

Content assembled by client-side calls after load, a flow where step three depends on step two, or anything behind a login. Those are real and no fetch replaces them, and the wider version of that decision is in Browser Automation When the Site Has No API.

The middle option people skip

A fetch endpoint that renders JavaScript on the provider's side. You get the post-render markup back as text, with no browser in your deployment. That covers the "content arrives late" case without covering the "I need to click things" case, and it is most of what people install Playwright for.

The .NET-specific cost

Shipping Playwright means shipping browser binaries into your container image, which changes your base image, your image size and your cold start if you are on anything serverless. That is a deployment decision rather than a code decision, and it is worth making deliberately rather than as a side effect of a NuGet install.

Which endpoint should I use for which job?

Every row was verified with monid inspect on 2026-08-31. The table gives billing shape rather than figures; shape drives design and current numbers live on monid.ai/tools.

The row worth knowing in a typed language is extract. Supplying a JSON Schema and getting typed fields back means your C# record and the response shape are agreed in one place rather than two, which removes the class of bug where a parser silently returns null and a nullable property quietly accepts it.

When should you write the whole thing in C#?

Three cases, and they are legitimate.

The targets are friendly and the volume is large. Scraping your own properties, a partner's site, or public documentation that wants to be read. No access problem exists, so there is nothing to buy, and .NET's concurrency story makes this genuinely pleasant at scale.

The parsing is the product. Custom extraction rules per site, domain-specific normalisation, anything where the transformation is what you are selling. Keep it in your codebase where you can test it.

You are crawling a whole site rather than fetching pages. That is a different shape with a different cost curve, taken apart in Crawl4AI: When Should You Run Your Own Crawler?.

It has to run inside your process. Air-gapped environments, regulated data, or a deployment where an outbound call to a third party is a compliance conversation rather than a config change.

There is also an honest limit on the argument above. If you are already running browser infrastructure for other reasons, adding scraping to it is close to free, and the case for buying a fetch is much weaker. The pitch here is aimed at teams who would be standing that up specifically for this.

And the disclosure: this is Monid's blog and we sell per-call access to fetch endpoints, so we have a commercial interest in the conclusion. On the three cases above we are the wrong answer and the libraries in the first section are the right one.

Conclusion

C# web scraping is two jobs and only one of them is a C# question. The parsing stack is mature, HtmlAgilityPack and AngleSharp both do their job, and .NET's async model is a real advantage once the data is arriving. None of that touches the reason scrapers stop working, which is that a request left your server from an address the target decided to refuse.

The thing worth carrying past this is the diagnostic order. Before reaching for a proxy, check whether you are creating an HttpClient per request, because socket exhaustion in .NET produces timeouts that look exactly like rate limiting. Then check whether the content was in the markup at all, because an empty selector is a rendering problem rather than an access one. Only what survives both is actually a blocking problem, and that is the one you buy your way out of.

Free next step: run monid inspect -p context.dev -e /web/scrape/markdown and read the output schema before you write the record type. It is free, and matching the model to the response on the first attempt is worth more than the two minutes it takes. Start at monid.ai.

FAQ

Last updated August 2026.