C# Web Scraping: Fetch Rendered Pages, Parse with .NET

Get a JavaScript-rendered page into your C# code as data, without running a browser. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to fetch https://quotes.toscrape.com/js/ as rendered HTML and give me a table with each quote and its author.

One prompt replaces the fetch layer. 1 min setup · 10-20 hrs saved.

You searched for C# web scraping and found library roundups. HtmlAgilityPack and AngleSharp are good, and none of that helps when the page is built by JavaScript, or the target refuses your server, or you need a few thousand pages a night from a worker service. This guide splits the job: .NET keeps the parsing and the typed models, and one call does the fetching. We tested that call on 2026-10-16 against a page whose content only exists after a script runs. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to scrape a web page in C#?

Monid. Install it once, add a key, and one call returns any URL as rendered markdown or HTML, with the browser, proxies and anti-bot handling on the far side and your C# reduced to parsing and modelling.

• Pay per call. A page costs $0.002533 through the markdown endpoint (Verified 2026-10-16 on the web extraction tools page), and the endpoint's note says failed or blocked requests are not billed. Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and balance call the page fetchers here and every other tool on Monid, from search to enrichment. Nothing to sign up for when the next job is not scraping.
• Lives inside your agent. Install the CLI, add your key, run the prompt above, and the fetch is available whenever your agent needs a page. Your agent reads SKILL.md and learns the rest, so setup is about a minute.

At that price a thousand pages cost about $2.53 (Verified 2026-10-16), and TinyFish's fetch endpoint, which takes up to ten URLs per call, is free.

How do you call a fetch endpoint from a C# service?

Install the CLI, find the endpoint, and have a small .NET class run it and deserialise the JSON it prints. The routes compare like this.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Find the endpoint and read its schema

monid discover -q "scrape any website url to markdown"
monid inspect -p context.dev -e /web/scrape/markdown

Both are free. Inspect prints the input schema, the billing shape and the current price, so you read the cost before you spend and write your record types from the real field names.

Call it from C#

The CLI prints one JSON document with -j, and -w makes it wait for the result and include the output. This is the command the class below runs:

monid run -p context.dev -e /web/scrape/markdown \
  --query '{"url":"https://example.com","useMainContentOnly":true}' -w -j

And the class, with a record per response so that schema drift fails at the deserialiser:

using System.Diagnostics;
using System.Text.Json;

public sealed record PageMeta(string? Title, string? Language, string? FinalUrl);
public sealed record MarkdownPage(bool Success, string Markdown, int ContentLength, PageMeta Metadata);
public sealed record Run<T>(string Status, int BilledUnits, T Output);

public sealed class MonidCli
{
    static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);

    public async Task<Run<T>> RunAsync<T>(
        string provider, string endpoint, string inputFlag, string inputJson, CancellationToken ct)
    {
        var psi = new ProcessStartInfo("monid")
        {
            RedirectStandardOutput = true,
            RedirectStandardError = true,
        };
        foreach (var arg in new[] { "run", "-p", provider, "-e", endpoint, inputFlag, inputJson, "-w", "-j" })
            psi.ArgumentList.Add(arg);
        psi.Environment["NO_COLOR"] = "1";

        using var proc = Process.Start(psi)!;
        var stdout = proc.StandardOutput.ReadToEndAsync(ct);
        var stderr = proc.StandardError.ReadToEndAsync(ct);
        await proc.WaitForExitAsync(ct);

        if (proc.ExitCode != 0)
            throw new InvalidOperationException(await stderr);
        return JsonSerializer.Deserialize<Run<T>>(await stdout, Json)!;
    }
}

// GET endpoints take their fields with --query, POST endpoints with -i
var monid = new MonidCli();
var run = await monid.RunAsync<MarkdownPage>(
    "context.dev", "/web/scrape/markdown", "--query",
    """{"url":"https://quotes.toscrape.com/js/","useMainContentOnly":true}""", ct);
Console.WriteLine($"{run.Output.Metadata.Title} {run.Output.ContentLength} {run.BilledUnits}");

Our run of that call returned the title Quotes to Scrape, a contentLength of 1566 and one billed unit. The envelope around output also carries status, runId and billing.reportedCost, which for that run was 2533 micro-dollars, so you can log a bill per page. Arguments go through ArgumentList, so the JSON needs no shell quoting. The same parse/fetch split written for Python is in Python web scraping without a scraper.

Can you scrape pages yourself with HtmlAgilityPack?

Yes, and for friendly targets it is the right call: HttpClient gets the HTML and HtmlAgilityPack selects fields out of it with XPath. It works until the page changes or the target stops answering your server.

The libraries split by job. HtmlAgilityPack parses messy markup forgivingly and selects with XPath (the XPath cheat sheet for scraping covers the expressions). AngleSharp is a standards-based HTML5 parser with a browser-like DOM and CSS selectors. Selenium WebDriver and Playwright for .NET drive a real browser, which is what you reach for when the content only exists after a script runs. None of them gives you an address the target accepts: every one runs from your machine, so a datacentre address on a cloud instance is the easiest kind to refuse, and rotating around that costs more than most teams expect.

Doing it all yourself means a retry layer, a proxy or browser setup, parsing code and a repair every time a selector stops matching. That is hours before the first useful row, and it recurs. Calling Monid moves the fetch, the rendering and the blocked-request handling off your side, and what is left in your repository is the part that was your product: selectors, records, validation. Rules about what you may collect differ by site and country, so check yours.

How do you scrape JavaScript-rendered pages in C#?

Fetch the rendered DOM from an endpoint and hand it to HtmlAgilityPack as a string. We tested this on https://quotes.toscrape.com/js/, a page whose quotes are written by a script.

A plain curl of the page found class="quote" zero times: the markup you get without running JavaScript holds a script and no quotes, which is exactly what HttpClient.GetStringAsync would hand your parser. The rendered-HTML endpoint, called with waitForMs of 1000 and maxAgeMs of 0 for a fresh fetch, returned 8,940 characters containing 10 div.quote elements, each with a span.text and a small.author:

monid run -p context.dev -e /web/scrape/html \
  --query '{"url":"https://quotes.toscrape.com/js/","waitForMs":1000,"maxAgeMs":0}' -w -j

using HtmlAgilityPack;

public sealed record HtmlPage(bool Success, string Html);
public sealed record Quote(string Text, string Author);

var run = await monid.RunAsync<HtmlPage>(
    "context.dev", "/web/scrape/html", "--query",
    """{"url":"https://quotes.toscrape.com/js/","waitForMs":1000,"maxAgeMs":0}""", ct);

var doc = new HtmlDocument();
doc.LoadHtml(run.Output.Html);

var quotes = doc.DocumentNode.SelectNodes("//div[@class='quote']")
    .Select(n => new Quote(
        HtmlEntity.DeEntitize(n.SelectSingleNode(".//span[@class='text']").InnerText),
        HtmlEntity.DeEntitize(n.SelectSingleNode(".//small[@class='author']").InnerText)))
    .ToList();

On our run that selection gives 10 quotes. That is a page a static fetch calls empty, and no Chromium in your container image.

Do you need Playwright for .NET?

Only when you have to interact: click through a flow, fill a form, or work behind a login. For reading a page after its script has run, the rendered-HTML endpoint covers it. When you do need a session, the wider decision is in Browser automation when the site has no API. Shipping Playwright also means browser binaries in your container image, which changes your base image, its size and your cold start, so make that choice on purpose rather than as a side effect of a NuGet install.

What does a fetched page look like, and what can you build with it?

Three response shapes cover most work, and all three came back on 2026-10-16.

Markdown

The markdown endpoint returns success, markdown, contentLength, the requested url and metadata with sourceUrl, finalUrl, title, language, a headings list and additionalMeta. It also returns cache_metadata (status and age) and finalDOMState. On the quotes page the markdown held all ten quotes with their authors and tags as text, ready for a prompt or an index.

Rendered HTML

The HTML endpoint returns the same envelope with html in place of markdown. Use it when you want selectors, as above.

A batch of pages

TinyFish's fetch takes up to ten URLs in one POST and returns results[] and errors[]:

monid inspect -p tinyfish -e /fetch
monid run -p tinyfish -e /fetch \
  -i '{"urls":["https://quotes.toscrape.com/js/","https://example.com"],"format":"markdown","ttl":0}' -w -j

Each result had url, final_url, title, language, text, latency_ms and format. For our two URLs the latencies were 3,327 ms and 98 ms, and the run was free. The records for it:

using System.Text.Json.Serialization;

public sealed record FetchResult(
    string Url,
    [property: JsonPropertyName("final_url")] string FinalUrl,
    string? Title, string? Language, string Text,
    [property: JsonPropertyName("latency_ms")] double LatencyMs);
public sealed record FetchError(string Url, string Error, int? Status);
public sealed record FetchBatch(List<FetchResult> Results, List<FetchError> Errors);

What you can build with it

• A document ingester for a worker service. Fetch pages as markdown, store the text and metadata, and feed a search index; the same pattern works for a whole URL list.
• A typed extractor. Rendered HTML plus HtmlAgilityPack plus a record per page type gives you a pipeline where the compiler catches schema drift on your side.

What goes wrong when you fetch pages from .NET at scale?

Four things hide until you run more than a handful of URLs.

Why did I get an old copy of the page?

Because results are cached. The markdown and HTML endpoints reuse a cached result younger than maxAgeMs, which defaults to 86,400,000 (one day) per monid inspect. Our runs set maxAgeMs to 0 and the response said cache_metadata.status was miss with an age of 0. For anything that changes daily, set it explicitly and read the field you get back.

Why did a bad URL not throw?

Because a batch fails per URL. We sent TinyFish two URLs, one of them a 404. The CLI exited 0, the run was COMPLETED, and the 404 appeared in errors[] as {"url": ..., "error": "page_not_found", "status": 404}, with resultCount and billedUnits both 0 on the envelope. A try/catch around the call sees nothing. Check Errors.Count and handle each entry:

foreach (var e in batch.Errors)
    logger.LogWarning("Fetch failed for {Url}: {Error} ({Status})", e.Url, e.Error, e.Status);

Why is a 200 not enough?

Because a response can be well-formed and thin. Check the content, not only the status:

if (!page.Success || page.ContentLength < 200)
{
    logger.LogWarning("Thin response for {Url}", page.Metadata.FinalUrl);
    return null;   // do not index this
}

How do I run many fetches without exhausting the machine?

Chunk the list to the endpoint's limit of ten and cap the parallelism. If you call the target directly with HttpClient, create one instance through IHttpClientFactory rather than one per request, because a new instance per request leaves sockets in TIME_WAIT and produces timeouts that look like rate limiting. With the CLI, each call is a child process, so the cap protects your CPU and memory:

var batches = urls.Chunk(10).ToList();

await Parallel.ForEachAsync(
    batches,
    new ParallelOptions { MaxDegreeOfParallelism = 4, CancellationToken = ct },
    async (batch, token) =>
    {
        var json = JsonSerializer.Serialize(new { urls = batch, format = "markdown", ttl = 0 });
        var run = await monid.RunAsync<FetchBatch>("tinyfish", "/fetch", "-i", json, token);
        foreach (var r in run.Output.Results) await store.SaveAsync(r, token);
        foreach (var e in run.Output.Errors) logger.LogWarning("{Url}: {Error}", e.Url, e.Error);
    });

Which endpoint should I use for which job?

The first three were run on 2026-10-16. The extract row was inspected and not run. Prices: markdown and HTML are $0.002533 a call, and extract is $0.02533 a call whatever the schema size or page count (Verified 2026-10-16). TinyFish's fetch lists no per-call charge and its note says protected pages fail with bot_blocked, so use the context.dev endpoints when the target defends itself. The four kinds of scraping tool and who operates each are sorted in Web scraping tools: which kind do you need?.

The row worth knowing in a typed language is extract: you supply a JSON Schema and get typed fields back, so your C# record and the response shape are agreed in one place.

When is another route a better fit?

The targets are friendly and the volume is large. Your own properties, a partner's site, public documentation that wants to be read. No access problem exists, so HttpClient plus HtmlAgilityPack costs nothing per page and .NET's concurrency suits it.

The parsing is the product. Custom extraction rules per site and domain-specific normalisation belong in your codebase where you can test them. That stays true with a fetch endpoint: only the fetch moves.

You need to click. Logins, forms and multi-step flows need a browser session, and Playwright for .NET is the tool.

It has to run inside your process. Air-gapped environments and regulated data, where an outbound call to a third party is a compliance conversation, rule out any hosted fetch.

And the disclosure: this is Monid's blog and we sell per-call access to these endpoints, so we have a commercial interest in the conclusion. In each of the cases above the libraries from the third section are the better answer.

Conclusion

The easiest way to scrape a page in C# is to let one call fetch it and keep .NET for what it does well: Monid's markdown or HTML endpoint at $0.002533 a call (Verified 2026-10-16), one key and one balance for every tool, about a minute to set up, and a page that only exists after JavaScript runs came back as 10 parseable quotes.

Three things matter more than the choice of library. Set maxAgeMs yourself, because the default can hand you yesterday's page. Read errors[] on batches, because a 404 exits with code 0. And assert on content length, because a success flag does not mean a useful page.

How we estimated the time saved: standing up your own fetch layer for rendered pages, with a browser in your image, retries, and a proxy story, is roughly 5 to 10 hours; keeping it alive through the first months of blocking and markup changes is another 5 to 10. That is 10 to 20 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid inspect -p context.dev -e /web/scrape/markdown, read the output schema, and write your record type from it before the first call. Start at monid.ai.

FAQ

Published August 2026.