Browser Automation Without an API: Check for an Endpoint First

Get data from a site that has no API without writing a browser script. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to find an endpoint that already covers builtin.com job listings, and give me 25 data engineer jobs with title, company, location, salary and the job URL.

One prompt checks the catalog before any browser code. 1 min setup · 8-24 hrs saved.

"This site has no API" usually means the site owner does not publish one, which is a different statement from "nobody sells the data". Before you write Playwright code, selectors and a restart policy, you can ask a catalog whether an endpoint already covers the site, read its schema for free, and run one small call. This guide walks that check with live runs from 2026-10-16: a job board, a news front page and a hosted browser session. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to automate a website that has no API?

Monid. Install it once, add a key, and your agent searches a catalog of endpoints by plain description, reads the schema, and calls the one that covers your site, so the first move is a question instead of a script.

• Pay per call. Each run bills on its own. A job-board search is $0.01 and a page to Markdown is $0.002533 (Verified 2026-10-16). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills.
• One install, every tool. The same key and balance reach a purpose-built endpoint for a popular site, a generic page-to-Markdown endpoint for any URL, and a hosted browser session when you need to click. See the tools page for agents for what is listed.
• Lives inside your agent. Your agent reads SKILL.md and learns the discover, inspect, run loop itself, so setup is about a minute and nothing keeps running between calls.

How do you check whether an endpoint already exists for the site?

Describe the site and the data in plain words, run monid discover, then inspect the best match. Four routes cover most "no API" jobs, and they differ in what you maintain; the same sort applied to scraping in general is in web scraping tools: which kind do you need.

Monid fits because the first step is a free search: the answer to "does anything already cover this site" arrives before you spend anything.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Step 1. Ask the catalog before you build

Search by description, not by provider name.

monid discover -q "job listings from a job board"
monid discover -q "amazon product details by asin"

On 2026-10-16 the first query returned job endpoints from Wellfound, Web3 Career, Clay, Apollo and Akta, plus a Strale endpoint for national employment services; the Built In search used below is in the catalog too. The second returned an Apify Amazon product-details scraper, three DataForSEO Amazon endpoints and a MrScraper Amazon product endpoint. A third query, "etsy listings", returned no Etsy endpoint in its top five: eBay endpoints and Amazon seller lists came back instead. For Amazon the catalog lists product detail by ASIN, seller offers per ASIN and ranked keywords per ASIN, so an Amazon listing job starts from a catalog lookup, not a browser: one ASIN in, the Amazon product page fields (price, variants, rating, specs) and the seller list for that ASIN (price, condition, shipping) out. The ASIN workflow, step by step, is in automating Amazon product detail lookups. An empty or off-target result is also an answer. It tells you the site is a generic-extraction job, which is the next route down the table.

Step 2. Read the schema before you spend

monid inspect -p builtin -e /search_jobs

The inspect output shows the input fields (query, work_type, location, page), the billing shape (per call) and the price, all free. If the fields are not the ones you need, you learned that without a billed call.

Step 3. Run one, small

monid run -p builtin -e /search_jobs -w --query '{"query": "data engineer"}'

That call returned HTTP 200 for $0.01 (Verified 2026-10-16), with 25 job records on page 1 of 125. Current prices for every endpoint are on monid.ai/tools.

Can you automate it yourself with Playwright or Python?

Yes, and for a site you will keep automating for years it can be the right call. Playwright, Puppeteer and Selenium are free, open source and well documented, and Playwright and Selenium both have Python packages, so the starting cost is an afternoon.

The cost shows up after the first run. A browser script encodes the page's structure: this selector, then that button, then wait for this element. Each of those is a promise the site never made. A redesign breaks the script, and the break is often silent, because a selector that matches nothing returns an empty list rather than an error. You also own what surrounds the script: a machine to run Chrome on, a restart policy, a proxy so the requests do not all come from one address, and a rule for what to do when the site starts challenging automated visitors.

What Monid takes off that list depends on the route. A purpose-built endpoint puts the parsing and its upkeep with the provider. The page-to-Markdown endpoint states that JavaScript rendering, anti-bot bypass and residential proxies are handled server-side and included in the price, and that failed or blocked requests are not billed. The hosted browser session gives you Chrome on someone else's machine, but the code that drives it is still yours. For the wider question of when to run a crawler yourself, the Crawl4AI guide works through a live example of a request that succeeds and returns nothing.

If you do build it yourself, send the load through a managed exit address instead of one machine, as measured in do AI agents need a rotating proxy.

What does the data look like, and what can you build with it?

It depends on the route, so here is what each of our three runs returned.

A purpose-built endpoint returns typed rows

The Built In search returned 25 job records on page 1, with total_pages of 125. Each row carried id, title, url, company, company_url, company_logo, work_type, location, salary, experience_level and posted. Note work_type ("Hybrid", "In-Office", "Remote or Hybrid") and experience_level ("Mid level", "Senior level", "Entry level"): those are the board's own labels. Rebuilding them from a rendered page means writing your own classifier.

A generic endpoint returns Markdown plus metadata

Pointed at the Hacker News front page, the Markdown endpoint returned HTTP 200 in about a second for $0.002533 (Verified 2026-10-16). The response held markdown (14,739 characters, the 30 front-page stories as a table with title, points, submitter, age and comment count, each with its link), url, a metadata object (sourceUrl, finalUrl, title, language, favicon, alternates) and finalDOMState. Your agent can read that as is. Turning it into typed fields is a second step, for which the same provider lists a schema-driven extract endpoint.

A hosted browser returns a way in, not data

The session endpoint returned sessionId, connectUrl, liveUrl, authToken, paidMinutes, expiresAt and pricing. No page content comes back. You connect your own Playwright or Puppeteer code to the connectUrl and do the work there.

What to build

• A job-alert feed. Loop the search over a handful of queries and cities, and keep new id values. Pair it with the scoring approach in job scraping software, board by board.
• A reading list for an agent. Feed a news or docs page through the Markdown endpoint on a schedule and let the agent summarise what changed.
• A one-off action. Book, upload or confirm something behind a login with a hosted session and your own script.

What goes wrong when you automate a site that has no API?

Four things showed up in our own runs, and each one has a rule that follows from it.

The minimum session is five minutes

Our first session call asked for one minute and got HTTP 400 with the message estimatedMinutes must be an integer of at least 5. That call was billed $0 (zero billed units). The second call, with estimatedMinutes set to 5, returned HTTP 200, was billed $0.011 (Verified 2026-10-16), and came back with paidMinutes 5, a ratePerHour of 0.12 and an expiresAt five minutes after creation. The rule: size the session to the job, connect right away, and treat the connectUrl and authToken as secrets.

The Markdown can be a cached copy

The Hacker News response carried cache_metadata with status hit and an age of 995,657 milliseconds, about 16.6 minutes. The endpoint's default is to reuse a result up to one day old. For a front page that changes by the hour, send maxAgeMs set to 0 to always fetch fresh.

monid run -p context.dev -e /web/scrape/markdown -w --query '{"url": "https://news.ycombinator.com", "maxAgeMs": 0}'

Main-content mode still keeps some chrome

We sent useMainContentOnly as true and the Markdown still began with the site's navigation bar and ended with the footer links. Strip those yourself before you hand the text to a model, or ask for typed fields instead.

Typed rows still have gaps and loose formats

Of the 25 Built In rows, 3 had a null salary and 3 had a null experience_level. salary is a string such as "98K-158K Annually", posted is relative text such as "2 Days Ago" or "Reposted 2 Hours Ago", and location is sometimes "3 Locations" instead of a place. Capital One alone held 9 of the 25 rows. Parse the strings, convert posted to a date at collection time, and de-duplicate by id and company before you count anything.

Pagination is the last cost. 125 pages at 25 rows means one full sweep of that query is 125 calls, $0.01 each (Verified 2026-10-16), so about $1.25. Decide how deep you need before you loop.

Which endpoint should I use for which job?

Work down the table and stop at the first row that covers your case.

Prices as inspected on 2026-10-16: the extract endpoint is $0.02533 per call, the crawl endpoint is $0.002533 per page scraped, and the session endpoint is $0.011 per call (Verified 2026-10-16). Check them against the tools pages when you plan volume.

When is another route a better fit?

The task carries state. A logged-in session, a multi-step form or a cart needs a browser. Extraction endpoints read pages; they do not click, and a hosted session only gives you the Chrome, not the script. The split between consumer and programmable browsers is covered in when your agent needs a browser.

The action is the point. Submitting, uploading and confirming are actions. Use a hosted session or your own browser.

You are testing your own application. Playwright and Puppeteer are testing tools first, and for that job you want them on your machine.

Browser automation is your product. A vendor whose whole business is browser infrastructure will go deeper on session management and concurrency than a catalog of endpoints, and you should read their documentation first.

The site publishes an API after all. Some do, as the Hacker News page we fetched links to its own in the footer. Check the footer and the developer docs before you pay for anything.

And the disclosure: you are reading Monid's blog and we resell per-call access to these tools. The case we argue best is the one where the browser turns out to be unnecessary; when it is necessary, the table above says so.

Conclusion

The quickest way to automate a site with no API is to ask a catalog first: one free monid discover, one free monid inspect, one small run. On Built In that meant 25 typed job rows for $0.01 (Verified 2026-10-16), and on a news front page it meant a page of Markdown for $0.002533, with no selectors written. Monid keeps the one key, the one balance and the per-call billing, and it runs inside your agent whenever the next site turns up.

Two things matter more than the route you choose. Read the rows you get back: null salaries, relative dates and cached copies are where the quiet errors live. And size a hosted session to the job, because the minimum is five minutes.

How we estimated the time saved: building and maintaining a browser script for one site is roughly 6 to 12 hours to build and 2 to 12 hours of upkeep in the first month, which is 8 to 24 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid discover with a plain description of your site and the data you want, then monid inspect on the best match.

FAQ

Published August 2026.