The Data Layer for n8n: One Key Instead of a Node Per Vendor
n8n workflows rarely break at the logic. They break at the data source. Build the scraping and enrichment layer so a dead provider is a config change.
Copy this line to your agent to build the data step of an n8n workflow.
set up https://monid.ai/SKILL.md and find the endpoint that enriches a company domain into firmographics
n8n is good at the part people worry about and bad at nothing in particular. The workflows hold up. What does not hold up is the layer underneath them, the place where a scraper, an enrichment vendor or a social endpoint supplies the rows, because that layer is somebody else's product and it changes without asking. Every long r/n8n thread about a broken automation is really a thread about that. Monid is the OpenRouter for agent tools: one key and one balance across many providers, which in an n8n context means the data step is an HTTP Request node whose provider is a variable.
Why does the scraping layer break an n8n workflow more often than the workflow does?
Because the workflow is yours and the data source is not. An n8n canvas changes when you change it. A scraping endpoint changes when a platform ships a layout, when a vendor deprecates an actor, or when an account gets rate limited, and none of those events reach your canvas as an error.
The three failures, and why none of them raise an exception
A restricted account returns a page. Not a 403, a page. Often a login wall or an interstitial, rendered and served with a 200. Whatever parses it gets valid HTML and extracts nothing.
A truncated result is a short list. Pagination that stops early, a rate limit that trims a response, a vendor silently capping results, all produce a shorter array rather than an error. The workflow iterates it happily.
A changed selector returns null. The request succeeds, the shape is right, the field is empty. Downstream nodes write empty strings into a CRM, which is worse than writing nothing, because empty overwrites good data.
The r/n8n thread from someone struggling with the scraping layer for an Instagram and X assignment collects all three across ten comments. The build was never the problem.
Why this hits automation harder than it hits code
A script that breaks gets run by a person who notices. A workflow that breaks runs on a schedule, at night, and reports success to nobody. The gap between failure and discovery is measured in days, and everything downstream keeps consuming the bad rows in the meantime.
That asymmetry is the whole argument for putting assertions in the data step specifically. It is the one node where a wrong answer looks exactly like a right one.
One node per vendor versus one endpoint with a variable
| Aspect | A community node per vendor | One HTTP node, provider in a variable |
|---|---|---|
| Adding a source | Install and configure another node | Change a string |
| Vendor deprecates | Rewire the canvas | Change a string |
| Credentials | One per vendor, each stored separately | One key, one balance |
| Failure surface | Each node's own error shape | One response shape to assert on |
| Best for | One vendor you are committed to | Workflows that touch several sources |
The trade is real: a dedicated node gives you typed parameters and a nicer editing experience. What it costs you is the ability to change your mind cheaply.
📖 See also A Model Is Half an Agent: AIHubMix for Models, Monid for Tools
How do I build a lead enrichment workflow in n8n to find social media profiles?
Four nodes, and only one of them is the interesting part. Trigger, resolve the company, find the people, enrich the profiles. The mistake almost everyone makes is doing these in the wrong order and paying person prices for company facts.
For agents
Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:
set up https://monid.ai/SKILL.md
It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.
For humans
npm install -g @monid-ai/cli
monid keys add -k <your-key> -l main
Step 1. Resolve the company before you touch a person
What it does. Turns a domain, a name or a URL into firmographics, so the expensive person-level calls only run against accounts that qualify.
The endpoints. apollo/organizations/enrich takes a domain, LinkedIn URL, name or website and returns firmographics, funding and technology data in one call.
The call.
monid inspect -p apollo -e /organizations/enrich
What comes back. Enough to disqualify most rows before they cost anything. Headcount, industry and stack are usually what the filter was really about.
What it costs. Per call, so it costs the same whatever comes back. Putting this node first is the single biggest cost lever in the whole workflow, because it removes rows before the per-result nodes see them.
Step 2. Find the people, with a narrow filter and a hard cap
What it does. Turns company criteria into a list of profiles.
The endpoints. apify/harvestapi/linkedin-profile-search filters by job title, company, school, location, industry, seniority and headcount. apify/harvestapi/linkedin-company-employees walks one company's staff directly, which is usually the right shape inside a lead workflow.
What comes back. Search pages, then profile records at the depth you requested.
What it costs. Per result, on more than one tier, which makes a wide filter the classic n8n cost accident. Cap the item count in the node itself, not just in the query, and look at one page before letting the schedule run it.
Step 3. Enrich the profiles that survived
What it does. Fills in the record you are actually going to use.
The endpoints. apify/dev_fusion/linkedin-profile-scraper for LinkedIn, apify/apify/instagram-profile-scraper when the social profile is the target, and tikhub endpoints where per-call billing suits a spot check better than per-result.
What comes back. For LinkedIn, verified 2026-08-20: work history, education, skills, certifications and discovered contacts. For Instagram: follower and following counts, verification date, join date and recent media.
What it costs. Per result, and this is where the money goes, which is why the two nodes above it exist.
Step 4. Assert before you write
What it does. Stops the silent failures from reaching your CRM.
One IF node, checking that a field you actually consume is populated on the record, and a Stop and Error branch when it is not. Not the status code, the field. This is four minutes of work and it is the difference between finding out tonight and finding out next quarter.
Give this to your agent
Set up https://monid.ai/SKILL.md, and then use Monid to build me the data step: resolve this domain to firmographics, then find the heads of engineering, then enrich only the ones at companies over 200 people.📖 See also Turn a LinkedIn Profile URL Into an Enriched Lead and Ship an Instagram Profile Enricher This Afternoon
Apify got barred from scraping Apollo. What should I use to pull fresh leads instead?
That is the question as it was asked on r/n8n, and the premise deserves separating from the answer. What is verifiable is that people building lead workflows keep discovering that a specific source stopped working for reasons outside their build. What that source was on any given month is less useful than what it implies, which is that the sourcing step is the least stable node in a lead workflow and should be designed as replaceable rather than as correct.
The practical answer is to stop thinking of it as one source. Fresh leads come from at least three different shapes of data, and a workflow that depends on one of them is a workflow with a single point of failure:
Directory-shaped sources give you a filtered list from criteria. Profile search endpoints do this, and they are what most people mean by lead sourcing.
Signal-shaped sources give you a reason to reach out now rather than a name. Post and activity endpoints like apify/harvestapi/linkedin-profile-posts return engagement and comments, and a workflow triggered by a signal converts differently from one triggered by a filter.
Company-shaped sources give you the account and let you find people afterwards. This is the cheapest route and the most durable, because company data is available from more places than person data and is far less contested.
If the sourcing step is a variable rather than a node, losing one of the three is a bad week rather than a rebuild. If it is a hard-wired integration, it is the rebuild the thread was about.
Which endpoint should I use for which job?
| Endpoint | What it does | Input | Output | Best for | Billing |
|---|---|---|---|---|---|
apollo/organizations/enrich | Company firmographics | Domain, name or LinkedIn URL | Firmographics, funding, tech stack | The qualifying node, first in the flow | Per call |
apify/harvestapi/linkedin-profile-search | Filtered profile search | Title, company, seniority, location | Search pages plus profiles | Building the list | Per result, tiered |
apify/harvestapi/linkedin-company-employees | One company's staff | Company plus filters | Employee profiles | Account-based workflows | Per result plus a flat fee |
apify/dev_fusion/linkedin-profile-scraper | Full profile enrichment | Profile URLs | History, skills, discovered contacts | The enrichment node | Per result |
apify/apify/instagram-profile-scraper | Public Instagram profile | Usernames | Counts, bio, links, join date | Social profile enrichment | Per result |
apify/harvestapi/linkedin-profile-posts | Posts with engagement | Profile or page | Posts and comments | Signal-triggered outreach | Per result |
tinyfish/search | Live web search | Query plus filters | Ranked results | Filling gaps the enrichers miss | Per call |
context.dev/web/scrape/markdown | A page as clean Markdown | URL | Markdown | Reading a company site in-workflow | Per call |
Every row verified with monid inspect on 2026-08-20. The billing column is the shape rather than a figure, because per call and per result decide where in the flow a node belongs, and a price does not stay true.
What does the data layer actually cost in a running workflow?
The cost is set by node order, not by node choice, and that is the useful thing to know before optimising anything.
A lead workflow that qualifies on company data first and enriches second spends per-call money on every row and per-result money on the survivors. The same workflow with those two steps reversed spends per-result money on every row. Same endpoints, same output, and the second one can cost an order of magnitude more on a list with a low qualification rate.
A realistic sourcing and enrichment run over a few hundred qualified leads lands in single-digit dollars. The scenario that produces a surprising bill is almost never a repeated small workflow, it is one wide filter that matched far more people than intended and ran unattended overnight.
Discovery and inspection stay free, so the billing shape of every node is readable before you wire it. That is what makes a metered balance suit automation: access costs nothing until it is used, so an idle workflow costs nothing at all, which a per-seat tool does not. Where metered loses is steady, high, predictable volume, covered in pay per call versus subscription. Prices at monid.ai/tools.
When should you not use Monid?
If you use exactly one vendor and expect to keep using it, install its node. A dedicated community node gives you typed fields, inline documentation and a better editing experience than a generic HTTP call, and the flexibility you give up is flexibility you were not going to use.
If your workflow needs a native n8n trigger from a specific service, the integration has to be that service's. Nothing generic can replace a webhook that a vendor only fires into its own node, and dressing that up as a limitation of the vendor would be dishonest.
If you are inside a platform that already bundles the data, use what you have. Teams on a sales platform with enrichment credits included should spend those credits before adding a second bill, and the same goes for anyone whose CRM already ships the firmographics they were about to buy.
And if the workflow is a one-off you will run twice and delete, do the simplest thing. Any layer of indirection is a cost paid for future change, and some workflows have no future.
Conclusion
n8n workflows do not usually fail at the logic, they fail at the data, and the data layer fails quietly. Design that layer for replacement rather than for correctness: provider and endpoint in variables, one credential rather than one per vendor, and an assertion on a field you actually consume rather than on a status code.
What matters more than which endpoint you pick: node order sets the bill. Put the per-call qualifying step before the per-result enriching step and the same workflow costs a fraction of what it otherwise would, because the expensive nodes only see rows that survived a cheap one. That reordering takes ten minutes and outperforms any amount of vendor shopping.
The free next step costs nothing. Run monid discover for the step you are about to wire, monid inspect to read its schema and billing shape, then one small paid run against ten rows before the schedule ever touches ten thousand. Start at monid.ai.
FAQ
Do I need a community node, or is the HTTP Request node enough?
The HTTP Request node is enough, and it is the more durable choice when a workflow touches more than one data source. A community node buys you typed parameters and inline docs for one vendor; an HTTP node with the provider and endpoint held in workflow variables buys you the ability to swap that vendor without editing the canvas. Pick the node if you are committed to the vendor, and the variable if you are not.
How do I stop a partial result from passing as a success?
Assert on a field you consume, not on the status code, because every quiet failure in this category returns a 200. Add an IF node after the data step that checks a specific field is present and non-empty on a known-good record, and route the false branch to Stop and Error. A truncated list, a login wall and a changed selector all fail that check and all pass a status check.
Should the agent pick the endpoint, or should the workflow?
The workflow, for anything on a schedule; the agent, for anything exploratory. A scheduled workflow wants a fixed, cheap, predictable call, and letting a model choose the endpoint each run makes the bill and the output shape both variable. An agent doing research benefits from the opposite property, which is why discover exists as a runtime call rather than only as a design-time one.
What breaks first when I scale a workflow from ten rows to ten thousand?
The per-result nodes and the rate limits, in that order, and both show up as cost before they show up as errors. The thread from someone pulling a thousand LinkedIn leads a day collected ninety-three comments and the recurring advice is the same: cap the item count at the node, batch where billing is per call and loop where it is per result, and add the assertion before you add the volume.
Last updated August 2026.
