How to Stop Depending on One Scraping Vendor

Copy this line to your agent to find a second route before you need one.

set up https://monid.ai/SKILL.md and find two endpoints that answer the same question

Between 2026-09-01 and 2026-09-03 this pipeline hit ten distinct provider failures while researching unrelated articles. Nobody was stress-testing anything. That rate is the argument: single-vendor dependency is not dangerous because a vendor is bad, it is dangerous because at any given moment some fraction of any catalogue is not working, and switching costs an integration project you end up doing while already broken. This guide runs through Monid, the OpenRouter for agent tools.

How often does a data source actually break?

More often than a status page suggests, and the failures do not look alike.

Three days of incidental failures

Ten, across nine providers, in seventy-two hours, none of it deliberate. Every one of these interrupted an article about something else.

What that rate means

It does not mean these vendors are unreliable. It means the base rate of "some route is down right now" is high enough that a pipeline touching several sources will meet one most days. Designing as though a working endpoint stays working is designing against the evidence.

The failure that should worry you most

The first row. A 200 with a full field list and no values is the one that reaches production, because it passes every check a normal client makes. The 502s and 500s are loud and self-announcing; the empty record is silent and gets written to your database. We took that apart properly in the Amazon ASIN guide.

And the one that is permanent

Actors get removed and companies shut down. Proxycurl's closure took an entire LinkedIn data pipeline offline for everyone who depended on it, which we wrote up in the Proxycurl shutdown guide. No retry policy covers that.

Why is a second vendor not the same as a fallback?

Because signing up with two providers is procurement, and failover is engineering. Most teams do the first and believe they have done the second.

What having a second account actually gets you

A second set of credentials, a second SDK, a second response shape and a second set of parameter names. When the first breaks you still have to write the adapter, and you are writing it under pressure, at the worst possible time.

The shape problem is the real cost

Two providers answering the same question return different models of it. We measured this twice this week. Two real-estate providers asked for agents in one city returned 48 flat rows with phone numbers and 15 UI cards with review data, described in the MLS guide. Two Instagram providers asked for one profile returned edge_followed_by: { count } and followersCount, in the engagement accuracy post.

Neither pair is interchangeable at the field level even though both pairs answer the same question. That gap is the integration project, and it does not disappear because you have two contracts.

What a real fallback requires

One internal shape that your code consumes, and an adapter per provider that maps into it. The adapter is the work; the second provider is trivial once it exists. Written before an incident, it is an afternoon. Written during one, it is a bad week.

The test that tells you whether you have one

Turn your primary off and see whether the pipeline produces correct output. If you have never run that test, you have a second vendor rather than a fallback. Run it on a schedule, because an untested failover path is a path that has silently rotted.

📖 See also Why One Vendor Cannot Cover Company News

How do you build a route you can actually switch?

Three steps. The first is free and it is the one that removes most of the pain.

For agents

Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:

set up https://monid.ai/SKILL.md

It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.

For humans

npm install -g @monid-ai/cli
monid keys add -k <your-api-key> -l main

Step 1. Find the second route before you need it

What it does. Establishes that an alternative exists and what it returns, while nothing is on fire.

The call.

monid discover -q "amazon product detail page by asin"

What comes back. On 2026-09-01 that query returned a dedicated ASIN actor, a reviews endpoint and a generic product extractor. The dedicated one was the one returning empty records that day; the generic one worked. Knowing the second existed turned an outage into a one-line change.

What it costs. Nothing. Discovery never bills, and neither does inspect, so mapping alternatives for every source you depend on is free work you can do this afternoon.

Step 2. Define your shape, then adapt into it

What it does. Makes providers substitutable.

The call. No endpoint:

@dataclass
class Product:
    id: str
    title: str
    price: float | None
    source: str          # which provider answered

def from_axesso(r):  return Product(r["asin"], r["title"], r["price"], "axesso")
def from_context(r): return Product(r["url"], r["product"]["name"],
                                    r["product"]["price"], "context.dev")

Keep source on every record. When a number looks wrong three weeks later, the first question is which provider produced it, and a pipeline that discarded that cannot answer.

Step 3. Assert, then fail over on content

What it does. Triggers the switch on the silent failure, not just the loud one.

The call. No endpoint:

def fetch(asin):
    for provider in (primary, secondary):
        rec = provider(asin)
        if rec and rec.title and rec.title.strip():   # the real check
            return rec
        log_degraded(provider.__name__, asin)
    raise NoRouteWorked(asin)

A status-code failover would have sailed straight past the 26-empty-fields response, because it was a 200. Fail over on whether the answer is usable, which is the same discipline as classifying responses before retrying them.

What should you do when an actor disappears?

Removal is different from an outage: there is nothing to wait for.

Recognise it quickly

A 404 that persists across days, on a call whose parameters have not changed, is a removal rather than a fault. The tell is that it never recovers and no status page mentions it. Track a per-endpoint success rate so this is visible as a flat line at zero rather than as a support ticket.

Do not port to a single replacement

The instinct is to find the closest equivalent and swap. That reproduces the exact position you were just burned by. If you are doing the work anyway, do it into an adapter, so the third provider costs an hour instead of a week.

Expect the field set to be different, not just renamed

A replacement rarely returns the same information. The agent and brokerage fields on one property provider had no counterpart on the other; one Instagram provider carried per-post engagement and the other did not. Some of your downstream logic will be about a field the new provider does not have, and finding that out during a migration is normal rather than a surprise.

Keep the old response samples

Store a few complete raw responses per provider. When you migrate you need to know exactly what you were consuming, and reconstructing it from your own normalised records loses precisely the fields you dropped.

Which endpoint should I use for which job?

The answer is not one endpoint, which is the whole point of the article. What matters is that a second route exists and that finding it is free.

Every pairing was verified with monid discover and monid inspect in the first week of September 2026. The table gives billing shape rather than figures; current numbers live on monid.ai/tools.

Note what the table does not claim: that the pairs are drop-in equivalents. They are not, and the rows where we measured both, real estate and Instagram, returned materially different field sets. The value is that the alternative exists and its schema is readable for free before you commit.

When is one vendor the right answer?

Three cases, and they are common.

The data only has one source. Some information exists in exactly one place. A fallback cannot be invented, and the honest response is a contract and a support relationship rather than an architecture.

Downtime is genuinely cheap. For a weekly report, an endpoint being down for an afternoon costs nothing. Building failover for it is engineering spent on a problem you do not have.

You are small enough that the adapter is the risk. One provider called directly is less code than two behind an abstraction, and less code is more reliable. Write the adapter when you have a second provider, not in anticipation of one.

And the disclosure, which this article needs more than most: Monid is a routing layer, so "do not depend on one vendor" is our commercial interest. Two things cut against it and belong here. Three of the ten failures above came through us, including our own discovery call returning HTML on one attempt in four and a CLI polling bug present on two consecutive releases. And a routing layer removes the credential and billing problem while leaving the schema problem entirely intact, which is why step 2 of this article is an adapter you write yourself rather than something we provide.

Conclusion

Ten failures across nine providers in three days, none of them sought. That base rate, rather than any judgement about vendor quality, is the reason single-source pipelines break: the question is not whether a route fails but whether you can reach another one without writing an integration first.

So do the cheap part now. Discovery and schema inspection cost nothing, so map a second route for every source you depend on while nothing is broken. Define one internal shape and adapt into it, keeping the provider name on every record. And fail over on whether the content is usable rather than on the status code, because the failure most likely to reach your database returns HTTP 200 with a full field list and nothing in it.

Free next step: run monid discover for the single source your pipeline would most hate to lose. If a second route exists, you have removed your largest dependency risk for the price of one free call. Start at monid.ai.

FAQ

Last updated September 2026.