XPath Cheat Sheet for Scraping: Expressions That Survive

Get the title, price and stock status from a product page without writing a single selector. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to extract the title, price, availability and UPC from https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html as JSON, and tell me which fields came back empty.

One prompt turns a page into typed fields. 1 min setup · 3-6 hrs saved.

You are scraping a page, a selector broke overnight, and you want the short list of XPath expressions that will still match after the next redesign. This guide is that list, ranked by how long each anchor survives, with every expression run against one live page so the results below are what came back. It also covers matching by text, running XPath in Python, and the case where you can skip selectors entirely. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to get fields off a page without writing XPath?

Monid. Install it once, add a key, and your agent hands a URL and a list of fields to an extraction endpoint and gets typed JSON back, with no selector to write or repair. There is no official API for "any page", so this is the route that replaces the selector, not a wrapper around one.

• Pay per call. A page-to-Markdown call is $0.002533 and a schema extraction is $0.02533 (Verified 2026-10-16 on the web extraction tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and balance call the scraper, the extractor and every other endpoint on Monid.
• Lives inside your agent. Install the CLI, add your key, run the prompt above. The agent reads SKILL.md and learns the rest, so setup is about a minute.

Everything below works without Monid. The expressions are plain XPath 1.0, for the times you write the selector yourself.

📖 See also Browser Automation Without an API

Which XPath expressions do you actually need for scraping?

About a dozen, and the useful way to order them is by how long they survive a redesign, not by syntax category. The rule that does most of the work: anchor on something a person put there on purpose, then search downwards from it.

The durability ranking

Work down the table and stop at the first row the page supports. The ratings are judgement, not measured. The expressions below were all run with lxml against the product page of books.toscrape.com (a demo shop built for scraping practice), and the counts are what that page returned.

Attribute anchors

//*[@data-testid='product-price']     any element with a test id
//*[@itemprop='name']                 schema.org microdata
//a[@rel='next']                      pagination, semantically marked
//*[@role='navigation']//a            links inside a nav landmark
//p[@class='price_color']             exact class value (1 match, £51.77)
//p[contains(@class,'price')]         class fragment (1 match here)

The first four are patterns; the last two ran on the book page. contains(@class,'price') also matches price-was-struck-through and pricing-footer on pages that have them. The exact-class form is longer and correct for a multi-class attribute:

//p[contains(concat(' ',normalize-space(@class),' '),' instock ')]

On the book page the stock paragraph has class="instock availability", and that expression found it (1 match).

Label-value anchors, the ones worth memorising

//th[normalize-space()='UPC']/following-sibling::td[1]        a897fe39b1053632
//th[normalize-space()='Availability']/following-sibling::td   In stock (22 available)
//th[normalize-space()='Tax']/ancestor::tr                     the whole row
//th[normalize-space()='UPC' or normalize-space()='Tax']/following-sibling::td[1]   2 cells

Specification tables and invoice fields are label-value pairs. Anchoring on the label survives rows being added or reordered, because a human maintains the label. The last line shows boolean logic in a predicate, which is one of the four things CSS cannot do well.

Axes: up, across and after

//td[normalize-space()='Books']/preceding-sibling::th         Product Type
//th[normalize-space()='Tax']/parent::tr                      the row, one level up
//h2[normalize-space()='Product Information']/following::table[1]//tr[1]/th   UPC
//ul[@class='breadcrumb']/li[last()]                          A Light in the Attic

ancestor:: walks to the nearest matching ancestor, parent::* goes one level, and following::table[1] finds the first table after a heading. CSS selects descendants only, which is why XPath survives in extraction work.

Predicates and the bracket trap

(//ul[@class='breadcrumb']//a)[1]    the first link on the page: 1 match
//ul[@class='breadcrumb']//a[1]      the first link within EACH parent: 3 matches
//ul[@class='breadcrumb']/li[position()<=2]   the first two items: 2 matches

The second line is the classic mistake. a[1] means the first a child of each parent, so the breadcrumb returned three links. Parenthesise the whole expression when you mean "the first one on the page".

How do you select an element by its text in XPath?

Use normalize-space() as the default, and use contains(., 'text') only with an element constraint. Searches for "xpath contains text" and "xpath like text" are looking for exactly this, and there are three forms that behave differently. We ran all of them on the stock line of the book page, which reads In stock (22 available).

//p[text()='In stock (22 available)']                  0 matches
//p[normalize-space()='In stock (22 available)']       1 match
//p[contains(.,'In stock')]                            1 match
//*[contains(.,'In stock')]                            13 matches

Why does text() fail on text you can see?

Because HTML is full of whitespace you cannot see. The stock paragraph in the source is:

<p class="instock availability">
    [icon element]
    
        In stock (22 available)
    
</p>

The paragraph has two text nodes. The first is whitespace before the icon element, the second is the line break, indentation and the words. text()='In stock (22 available)' compares against a text node that still carries that whitespace, so it matches nothing. normalize-space() collapses runs of whitespace and trims the ends, which is why it should be your default and not your fallback.

Why does contains(text(), ...) miss the paragraph?

Because in XPath 1.0 a node set turned into a string yields its first node only. contains(text(),'In stock') on the paragraph looks at the whitespace before the icon and fails. The same expression on the table cell, which has one text node, matched. If an element mixes tags and text, test the text nodes one by one:

//p[text()[contains(.,'In stock')]]       1 match

Why does contains(., ...) over-match?

. is the whole subtree as a flattened string, so every ancestor that contains the text also matches. //*[contains(.,'In stock')] returned 13 elements, from html and body down to the paragraph. Put an element name before it, or take the deepest match.

Starts with, and ignoring case

//p[starts-with(normalize-space(),'In stock')]    1 match
//th[contains(translate(.,'ABCDEFGHIJKLMNOPQRSTUVWXYZ','abcdefghijklmnopqrstuvwxyz'),'upc')]   1 match

XPath 1.0 has no case-insensitive contains, so the translate() form is the portable answer. ends-with() belongs to XPath 2.0 and is covered in the failures section.

Text anchors survive layout changes and break on copy edits. If the page offers a data- attribute or structured data for the same fact, prefer it and use text only when nothing else marks the value.

How do you run XPath in Python?

Parse the HTML with lxml and call .xpath() on the tree. It is the standard route and it is what the expressions on this page were run with.

from lxml import html
import requests

page = requests.get("https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")
tree = html.fromstring(page.content)

upc = tree.xpath("//th[normalize-space()='UPC']/following-sibling::td[1]/text()")
price = tree.xpath("//p[@class='price_color']/text()")
stock = tree.xpath("//p[normalize-space()='In stock (22 available)']")
print(upc, price, len(stock))

xpath() always returns a list, even for one match, so index it or check its length. An empty list means the expression matched nothing; there is no error. Libraries built on lxml, such as Scrapy selectors and parsel, take the same expressions, and Python web scraping without a scraper covers the wider fork. Selenium also takes XPath, and it evaluates against the rendered DOM, which matters below.

For a page that needs JavaScript before the data exists, a plain requests.get returns the empty shell and every expression returns an empty list. That is a rendering problem, not a selector problem, and it is the point where a scrape endpoint that renders the page is the shorter path.

What does a page look like to a scrape endpoint?

It comes back as Markdown plus page metadata, and the facts you want are usually already labelled. We called /web/scrape/markdown on the book page on 2026-10-16. It returned success: true and a contentLength of 1,754 characters, and the Markdown carried the title as a heading, the price, the stock line, and the product table as plain rows:

# A Light in the Attic

£51.77

In stock (22 available)

Product Information

UPC | a897fe39b1053632
Product Type | Books
Price (excl. tax) | £51.77
Availability | In stock (22 available)
Number of reviews | 0

The metadata object held sourceUrl, finalUrl, title, description, language, favicon, robots, a headings list and additionalMeta. No jsonLd key came back for this page, so the page publishes no structured data and XPath, or an extractor, is the only way to its fields. On a page that does publish JSON-LD, the response carries it, and scraping news articles shows what that metadata looks like when it is there.

The extractor does the selector work

/web/extract takes a URL and a JSON Schema and fills it. We asked for title, price, availability and upc with factCheck: true, and it returned:

{ "title": "A Light in the Attic",
  "price": "51.77",
  "availability": "In stock (22 available)",
  "upc": "a897fe39b1053632" }

Four fields, none empty, no expression written. Note that price came back as 51.77 without the pound sign, which is the page's £51.77. The extractor normalised it, so write the schema with the format you want.

What you can build with it

• A price watcher. Run the extractor on a list of product URLs each week and compare the price field.
• A selector fallback. Keep XPath for pages you control and call the extractor when an expression returns an empty list.

What goes wrong with XPath in a scraper?

Four things, and the costliest ones fail silently, because a selector that matches nothing returns an empty list instead of an error.

Why does the XPath your browser gave you break?

"Copy XPath" in developer tools produces an absolute path. On the book page, the UPC cell's absolute path from lxml is:

/html/body/div/div/div[2]/div[2]/article/table/tr[1]/td

Every index in it is a fact about the current layout. Add a banner above the content and div[2] becomes div[3] and the path matches something else or nothing. The label-based expression for the same cell, //th[normalize-space()='UPC']/following-sibling::td[1], does not care where the table sits. Browsers also insert an implied tbody that parsers such as lxml do not, so a path copied from devtools can fail in Python before any redesign happens.

Why does a copied expression fail with "Unregistered function"?

Because it uses an XPath 2.0 function. We ran ends-with(@href,'.html') and matches(.,'^a8') in lxml and both raised Unregistered function. lxml implements XPath 1.0, so ends-with(), matches() and tokenize() are unavailable. The portable replacements are starts-with(), contains() and substring() with string-length().

Why does an empty list look like success?

.xpath() returns [] and the loop carries on, so a broken selector produces an empty column instead of a crash. Assert on what you extracted, for example that the price field is non-empty on every row, not on whether the request succeeded.

Why can a cached extraction still be billed?

Because the extractor bills per call. Our /web/extract run returned cache_metadata with status: hit and an age of 387,236,653 milliseconds, about 4.5 days, and was billed one unit of $0.02533 all the same (Verified 2026-10-16). Set maxAgeMs to 0 when you need a fresh read, since a cached answer costs the same as a new one.

Which endpoint should I use for which job?

Prices Verified 2026-10-16 with monid inspect; current numbers live on monid.ai/tools. For the same fork seen from the buyer's side, see web scraping services. We ran the first two once each; /web/crawl was inspected and not run. Install and add your key once:

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Then read a page as Markdown, a GET call with --query:

monid inspect -p context.dev -e /web/scrape/markdown
monid run -p context.dev -e /web/scrape/markdown --query '{"url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}' -w

And fill a schema, a POST call with -i:

monid run -p context.dev -e /web/extract -i '{"url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html","schema":{"type":"object","properties":{"title":{"type":"string"},"price":{"type":"string"},"availability":{"type":"string"},"upc":{"type":"string"}}},"maxPages":1,"factCheck":true}' -w

Or hand the job to an agent:

The right first move is the smaller call: scrape the page as Markdown for $0.002533 (Verified 2026-10-16), see whether the fields are labelled, and only then pay for the extractor.

When is another route a better fit?

XPath is the better tool for a site nobody has built for. When you control the markup, or you scrape one site for years, an expression that anchors on a label costs nothing per call and runs as fast as your parser.

Extraction rules that are part of your product. If the selectors are the thing you ship, write them yourself.

Volume on one stable template. At $0.02533 a call (Verified 2026-10-16), a typed extraction of 1,000 pages is about $25.33. If every page shares one template with a good anchor, a single XPath expression handles the same 1,000 pages for the cost of your own compute.

A page that publishes structured data. JSON-LD, microdata or an embedded state blob is a contract the site maintains for its own reasons. Read it first. It needs no selector and no extractor.

And the disclosure: you are reading Monid's blog and we resell these endpoints, so we gain when you skip the selector. The expressions above are the right tool when you are writing them yourself, which is why they are on this page.

Conclusion

The quickest way to get fields off a page is to name them and let an endpoint fill them: through Monid, $0.02533 a call (Verified 2026-10-16), one key for this and every other tool, about a minute to set up and ready inside your agent whenever you need it. When you write the selector yourself, anchor on a label or a deliberate attribute, never on a position.

Two things matter more than any single expression. Use normalize-space() by default, because text() compared against the stock line matched nothing while normalize-space() matched it. And treat an empty list as a failure to investigate, since XPath never throws when it finds nothing.

How we estimated the time saved: writing, testing and later repairing the selectors for one page template takes roughly 1 to 2 hours, so three templates is 3 to 6 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid inspect -p context.dev -e /web/extract, then scrape one page you were about to write selectors for and check whether the fields you need are labelled. Start at monid.ai.

FAQ

Published August 2026.