Scraping API Fallback: Stop Depending on One Vendor

Give your scraping pipeline a second route before the first one fails. Paste this into your agent:

Set up https://monid.ai/SKILL.md, and then use Monid to find two endpoints that turn a web page into markdown, run both on https://news.ycombinator.com/, and show me which fields differ.

One prompt maps a second route and compares the output. 1 min setup · 6-14 hrs saved.

Your scraper works until the day it does not: the vendor changes a field, drops an actor, rate-limits you or goes down, and the pipeline stops while you read their docs for a replacement. This guide is about having that replacement ready. It covers how to find a second vendor for the same job, how to put both behind one record shape, and a fallback that fires on bad content as well as on errors, with the two vendors' live output side by side from 2026-10-16. It runs through Monid, the OpenRouter for agent tools: one key, one balance, every tool priced per call.

What is the easiest way to add a fallback to a web scraping API?

Monid. Install it once, add a key, and your agent can search for every endpoint that does the same job, read each schema for free and call whichever one works today, with no second vendor account to open.

• Pay per call. A page to markdown through Context.dev costs $0.002533 and through Keenable $0.004 (Verified 2026-10-16 on the web extraction tools page). Buy calls as you need them: no monthly plan and no minimum top-up. Discovery and inspection are free; only a run bills, and on both of these a failed request is not billed.
• One install, every vendor. The same key and the same balance call both vendors and every other tool on Monid. Adding the second route is a second line in your code, not a second contract.
• Lives inside your agent. Your agent reads SKILL.md, learns the discover, inspect, run loop and can reach for the other route when the first one returns nothing. Setup takes about a minute and nothing runs between jobs.

At those prices, fetching 100 pages through the primary alone is about $0.25, and the secondary is billed only for the pages the primary fails (Verified 2026-10-16).

How do you set up a second scraping route for a list of URLs?

Find an endpoint that does the same job under a different vendor, read its schema, and run it once on a page you already know. The three commands below do that, and the first two cost nothing.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Step 1. Find the second route before you need it

Search by the job, not the vendor. On 2026-10-16 this search returned Amazon product pages from three different providers at the top: an Apify actor, a DataForSEO endpoint and a MrScraper endpoint.

monid discover -q "amazon product detail page by asin"

The same search for the job this guide runs, a page to markdown, ranked Context.dev first, MrScraper second and Keenable sixth, each with a price and a health status attached.

monid discover -q "scrape web page to markdown"

Step 2. Read both schemas

monid inspect -p context.dev -e /web/scrape/markdown
monid inspect -p keenable -e /v1/fetch

Both take one required field, url, and both are GET calls, so the request side is nearly the same. The differences that matter are on the response side, and the next two sections show them.

Step 3. Run each once

monid run -p context.dev -e /web/scrape/markdown --query '{"url":"https://news.ycombinator.com/"}' -w
monid run -p keenable -e /v1/fetch --query '{"url":"https://news.ycombinator.com/"}' -w

That is $0.006533 for the pair on 2026-10-16 (Verified 2026-10-16). The agent-side search for free fetchers shows what else sits on the same key.

Can you build the failover yourself in Python?

Yes, and the core is a short loop. The hard part is not the loop, it is everything that makes the second vendor usable: another account, another key, another billing relationship, another error format and another quota to watch.

A hand-rolled version means two sign-ups, two SDKs or raw HTTP clients, and a test that proves the second route works, which you have to keep running or it rots. Through Monid the second vendor is the same CLI call with a different -p and -e, billed from the same balance. The code below uses that, so the only vendor-specific part is the adapter.

import json, subprocess
from dataclasses import dataclass

@dataclass
class Page:
    url: str
    title: str
    text: str
    source: str          # which vendor answered

def run(provider, endpoint, query):
    p = subprocess.run(
        ["monid", "run", "-p", provider, "-e", endpoint,
         "--query", json.dumps(query), "-w", "-j"],
        capture_output=True, text=True)
    if p.returncode != 0:                 # provider refused: non-2xx
        return None
    return json.loads(p.stdout)["output"]

def from_context(o):
    return Page(o["url"], o["metadata"]["title"], o["markdown"], "context.dev")

def from_keenable(o):
    return Page(o["url"], o["title"], o["content"], "keenable")

ROUTES = [
    ("context.dev", "/web/scrape/markdown", from_context),
    ("keenable",    "/v1/fetch",            from_keenable),
]

def fetch(url):
    for provider, endpoint, adapt in ROUTES:
        out = run(provider, endpoint, {"url": url})
        page = adapt(out) if out else None
        if page and len(page.text.strip()) > 200:     # the content check
            return page
    raise RuntimeError(f"no route returned usable content for {url}")

Two details are measured rather than assumed. The CLI prints a formatted report by default, so the helper passes -j to get the run object as JSON, with the vendor's payload under output. And when a vendor refuses a request the CLI exits non-zero with a one-line message, which is why a failed call returns None above.

If you would rather fetch and parse the pages yourself, which kind of web scraping tool to use lays out who operates what in each option.

What do two scraping vendors return for the same page?

The same facts in two different shapes, which is why the adapter above exists. We called both endpoints on https://news.ycombinator.com/ on 2026-10-16, within a few seconds of each other.

Where the fields live

Neither response is wrong. They model the same page differently, and a pipeline that reads o["markdown"] directly breaks the day it meets the second one.

Links are not the same either

Context.dev returned absolute links, for example https://news.ycombinator.com/item?id=50106747. Keenable returned the page's own relative links, for example item?id=50106747, beside absolute links to the stories. If you store links and follow them later, resolve each one against the page URL in the adapter, once, so the rest of the pipeline never sees the difference.

And the counts differ

In the same minute, Context.dev showed the top story with 41 points and 9 comments, "1 hour ago". Keenable showed 71 points, 18 comments and "2 hours ago". Context.dev reported cache_metadata with status hit and an age of 4,019,315 ms, so the text it returned was a cached copy about 67 minutes old; the default for maxAgeMs is one day, and 0 forces a fresh fetch.

Two vendors can both return 200 with plausible markdown and still disagree on the numbers, because each has its own cache and its own idea of fresh. Treat the vendor as part of the record. Whether to hire a scraping service or pay per call is the single-vendor version of this decision.

What goes wrong when you fail over between vendors?

Four things show up once the second route is live.

Why does a fallback on status codes miss the bad cases?

Because a vendor can answer 200 and give you nothing useful: an empty body, a login wall, a block page, or a cache that is a day stale. A fallback that only fires on errors passes those straight through. The check in fetch above asks whether the text is usable. Make that check specific to your pages: a title that is present, a minimum length, a string you expect to find. The failure that reaches your database is the one that looks like a success.

Why do two vendors fail in different ways?

We sent both a domain that does not resolve. Context.dev answered 400 with error_code WEBSITE_ACCESS_ERROR and credits_consumed 0. Keenable answered 500 with {"error":"Internal server error"}. Both billed nothing, as their endpoint notes say, but a 500 from one vendor means "that site is unreachable" while from another it could mean "the vendor is down". Do not retry on a status code alone. Classify the response first, and send a dead URL to a dead-letter list instead of down the fallback chain. The Bing search API retirement is a case where a whole endpoint disappeared and callers needed another route.

Does a fallback double your cost?

Only for the calls that fail. You pay the second route when the first returns an error or fails your check, and both of the endpoints here bill only successful requests. A request that fails on the primary costs nothing there, and the same URL then costs $0.004 on Keenable (Verified 2026-10-16). Running both routes on every URL and comparing is a different choice, and it doubles the bill.

Is the second route ever tested?

Not unless you test it. A route that has not run in three months is a route that has not been checked. Send a small share of traffic, or one canary URL per day, through the secondary and alert if its output stops passing the content check. The cost is $0.004 a day on Keenable (Verified 2026-10-16).

Which endpoint should I use for which job?

The first two are the pair we ran, and the agents tools page lists what else sits on the same key. The third was inspected and not run: its price is metered per token, so the bill for a page depends on the page, and monid inspect shows the meter. Its median run time is about 24.6 seconds, against about 3.8 for the first, so it suits a last resort rather than a first choice.

Keenable's prompt option makes an LLM extract just the answer instead of returning the whole page, at the same price, and live fetches from the source instead of the index. Pass live when the pages change quickly and you need the current text.

When is another route a better fit?

There is only one source for the data. Some pages or datasets exist in exactly one place, and a second vendor has nothing to return. The right answer is a contract and a contact at that source, not an architecture.

Downtime is cheap. A weekly report that can run again on Monday does not need failover. Build it when an outage costs more than the adapter.

Your volume justifies running it yourself. At a large, steady volume of simple pages, your own fetcher can cost less than any per-call price, and you control the failure modes. That is a staffing decision, not a vendor decision.

You need guarantees in writing. A per-call marketplace is not a negotiated service level. Where a contract with a named uptime figure is a requirement, buy it directly from one vendor.

And the disclosure: Monid resells both endpoints in this post, so "keep a second vendor ready" is in our interest. What does not depend on us is the shape problem. A routing layer removes the second account and the second invoice, and it leaves the field names, the link format and the error codes as different as the table above shows, which is why the adapter is yours to write.

Conclusion

The way to stop depending on one scraping vendor is to put a second one behind the same record shape and fall back on bad content, not only on errors. Through Monid the second vendor is one more call on the same key and balance: $0.002533 on the primary and $0.004 on the secondary (Verified 2026-10-16), about a minute to set up and ready inside your agent when the first route fails.

Two things matter more than which vendors you pick. Define your own record and map each vendor into it, keeping the vendor name on every row, because the same page came back with different fields, different links and different counts. And test the secondary on a schedule, because a fallback nobody runs is a guess.

How we estimated the time saved: signing up for a second vendor, reading its docs and writing and testing the adapter and the content check is about 4 to 8 hours, and wiring a scheduled canary, a dead-letter path and error classification is another 2 to 6. That is 6 to 14 hours. The 1 minute is the install, key and first call once you already have a Monid key.

Free next step: run monid discover for the one source your pipeline would most hate to lose, then monid inspect on the second result. Start at monid.ai.

FAQ

Published September 2026.