LLM Gateway vs MCP Gateway: Four Families, One Word
An LLM gateway routes prompts to models. An MCP gateway routes tool calls to vendors. Four gateway families, and which problem each one solves.
Copy this line to your agent to give it the tool half rather than another model.
set up https://monid.ai/SKILL.md and use exa /search to research a company from the live web
Four different products are called a gateway in agent infrastructure, and they are not variants of each other. An API gateway fronts your own services. An LLM gateway fronts model providers. An MCP gateway fronts tools. An agent gateway is whichever of those a given vendor decided to call it that week. This piece sorts them out and then argues that the two that matter for an agent are the model half and the tool half, only one of which most teams have wired.
What is an LLM gateway?
An LLM gateway is a single endpoint that accepts a prompt and routes it to one of many language models, handling keys, fallback, caching and cost accounting on the way. It exists because a team that uses more than one model provider otherwise ends up with a provider's SDK, a provider's key and a provider's failure modes repeated once per provider.
What it does for you
Three things, in descending order of how much people care. It normalises the request format, so one client shape reaches many models. It fails over, so a provider outage or a rate limit does not become your outage. And it accounts, so somebody can answer what the month cost and which feature spent it. Caching and prompt logging usually come attached.
OpenRouter is the best-known hosted example and routes across hundreds of models. LiteLLM is the best-known self-hosted one, which is why "llm gateway vs litellm" is a search people run. Amazon Bedrock and Cloudflare AI Gateway are the platform-native versions.
What it does not do
It does not give the model anything new to work with. A prompt routed perfectly to the best available model still cannot read a page published this morning, look up a company that is not in the training data, pull a comment thread or place a phone call. The gateway made the model reachable and interchangeable. It did not make it informed.
That gap is not a criticism of LLM gateways. It is a category boundary, and the reason a second gateway exists.
How does an LLM gateway work?
An LLM gateway works as a reverse proxy with a translation layer: your client speaks one dialect, usually the OpenAI chat completions shape, and the gateway rewrites each request into whatever the destination provider expects, then rewrites the response back.
Routing
The routing decision is the product, and it is why some vendors call themselves a model router rather than a gateway: router emphasises the per-request choice, gateway emphasises the control plane around it, and most products do both. A simple one routes by the model name you asked for. A better one routes by policy: cheapest model that passes a quality bar, fastest under load, or a fallback chain when the first choice errors. This is the part Stripe reportedly paid billions for when it agreed to buy OpenRouter in August 2026, because deciding per request which model should serve it, and at what price, is a metering problem before it is an AI problem.
Keys and quota
The gateway holds the provider credentials, and your application holds one credential for the gateway. That is the single biggest operational win, and it is the same trick a tool gateway plays one layer over: the thing that needs access does not hold the secrets for every backend.
Accounting
Every request gets attributed and priced. Without this, multi-provider AI spend becomes unattributable within about a month, which is why the observability vendors and the gateway vendors keep converging on each other.
LLM gateway vs MCP gateway: what is the difference?
An LLM gateway routes prompts to models; an MCP gateway routes tool calls to vendors. They sit on different axes of the same agent and neither substitutes for the other.
The two halves, side by side
| Aspect | LLM gateway | MCP gateway |
|---|---|---|
| What is routed | A prompt | A tool call |
| Backends | Model providers | Tool and data vendors |
| Protocol to the client | HTTP, usually OpenAI-shaped | Model Context Protocol |
| Chosen by | Your policy, before the call | The agent, at run time |
| Failure looks like | A worse answer, or none | The agent cannot attempt the task |
| Metering unit | Tokens | Calls or returned records |
The row that matters most is who chooses. You choose the model, usually once, in config. The agent chooses the tool, per task, at run time, which means the tool layer has to be discoverable by something that is not a human reading documentation.
Why the failure modes differ
A missing model degrades quality. A missing tool removes a capability. If the model gateway is down your agent gives a worse answer; if the tool layer is missing your agent tells the user it cannot browse the web, and no amount of model quality fixes that. Teams tend to over-invest in the first and under-invest in the second because the first is the one with a dashboard.
Why the metering differs
Token billing is continuous and roughly predictable from input length. Tool billing is lumpy: some endpoints charge per call regardless of what comes back, others charge per record returned, and a few charge per unit of output such as characters of speech or minutes of audio. That difference is architectural, not cosmetic. An endpoint that bills per result means the row count is the dial you have to control, and an agent that does not know this will happily ask for a thousand rows. The gateway's job is to make the price visible before the call, not after.
LLM gateway vs API gateway: what is the difference?
An API gateway fronts services you own, for clients that already know the contract. Both AI gateways front services you do not own, for a client that has to learn the contract at run time. The operational machinery is shared; the assumptions are not.
What all three have in common
Authentication, rate limiting, retries, timeouts, observability and request shaping. This is why the companies who built API gateways are shipping LLM and MCP gateways now: the hard-won parts transfer directly, and nobody should rebuild them.
Where the AI ones diverge
Two places. First, self-description: an agent cannot be compiled against a schema, so the gateway has to advertise what it offers in a form a model can act on, and cheaply enough that the advertisement does not eat the context the task needs. Second, third-party billing: an API gateway fronts services you already pay for somehow, while an AI gateway is spending money with other companies on your behalf, every call, which makes price transparency a functional requirement rather than a reporting nicety.
An "agent gateway", where the term is used, is generally one of these three with agent-specific policy bolted on. Read the product page rather than the label.
What is the best LLM gateway?
The best LLM gateway is the one whose operational model matches yours, and the choice comes down to one question: do you want to run the proxy. Hosted options remove the operational surface and put a vendor in the request path. Self-hosted options do the reverse. Both are defensible and the comparison rarely turns on features.
The more useful question is the one people are starting to type instead: what is the equivalent for tools. That is what Monid is, the OpenRouter for agent tools. One key and one balance let an agent discover and call over a thousand tools across many providers, billed per call, with no separate signup per vendor. It ships as an MCP server, so the agent reaches it natively, and it routes tool calls only. It is never in the inference path, and there is no relationship with OpenRouter beyond borrowing the shape of their idea to explain ours.
If you want the full wiring walkthrough with both layers connected at once, that is the two-integrations guide. What follows is just the tool half.
For agents
Grab an API key at app.monid.ai, then paste this to your agent and hand it the key:
set up https://monid.ai/SKILL.md
It learns the whole discover, inspect, run workflow itself. More in the agent quickstart.
For humans
npm install -g @monid-ai/cli
monid keys add --label main --key <your-api-key>
More in the CLI quickstart.
Step 1. Let the agent find the tool, then read its price
What it does. Searches the catalog by description and returns the schema and billing shape for whatever looks right, before anything bills.
The endpoints. The catalog spans web search and extraction, company and people enrichment, social and platform data, browser automation, generative media and agent telephony. Browse it at monid.ai/tools.
The call.
monid discover -q "research a company from recent web coverage"
monid inspect -p exa -e /search
What comes back. Ranked endpoints with provider and billing shape, then for the chosen one its full body schema: query, search type from instant through deep-reasoning, a category filter for company, people, news, research paper or financial report, domain filters and inline content extraction.
What it costs. Nothing. Both steps are free, which is the property that makes a large catalog safe to hand to an agent.
Step 2. Run it, and pay for that call only
What it does. Executes the endpoint and draws the shared balance at the price shown in step 1.
The endpoints. exa/search for neural retrieval with optional structured output across sources; context.dev/web/search when you would rather have search results with each page already converted to markdown.
The call.
monid run -p exa -e /search \
-i '{"query": "llm gateway adoption at enterprises", "category": "news", "numResults": 10}' -w 120
What comes back. A result list with urls and optional extracted contents, highlights or summaries, and an outputSchema option that synthesises structured JSON across several sources rather than handing you ten pages to read.
What it costs. About a cent per call on the Exa endpoint, billed per call regardless of result count. The context.dev search endpoint bills per result instead, so there the count is the dial. Prices at monid.ai/tools.
Give this to your agent
Set up https://monid.ai/SKILL.md, and then use Monid to find the last month of news coverage about a company and return a structured summary with sources.📖 See also Is OpenRouter an MCP Server? What It Actually Exposes
Which endpoint should I use for which job?
| Job | Endpoint | Input | Output | Billing |
|---|---|---|---|---|
| Neural search, structured output | exa/search | query, type, category, outputSchema | results with optional contents | per call |
| Search plus page text in one hop | context.dev/web/search | query, numResults, domain filters | ranked results with optional markdown | per result |
| One URL to prompt-ready markdown | context.dev/web/scrape/markdown | url, extraction options | markdown plus page metadata | per call |
| Domain to company firmographics | pdl/v5/company/enrich | name, website, LinkedIn or ticker | firmographics, funding, headcount, tech stack | per call |
Every row verified with monid inspect on 20 August 2026. The table states the billing shape rather than a figure, because the shape decides how you architect and a number goes stale silently.
When is a tool layer the wrong thing to add?
When your problem is genuinely about models. If you are choosing between providers, chasing cost per token or building a fallback chain, that is an LLM gateway and Monid does not compete for it. We are not in the inference path.
It is also wrong when the tool count is one and will stay one. An agent that needs Slack and nothing else should get the Slack MCP server. Routing exists for the case where the vendor list is unknown when you write the code, and it is pure overhead when the list has one entry.
And if you need policy and audit over MCP servers your own team runs, buy the inward-facing kind of gateway. The API-gateway vendors moving into this space are building that properly. Monid is the outward-facing kind: right when the agent needs a capability nobody in the company has bought, wrong when the question is who may query the internal database.
Conclusion
LLM gateway and MCP gateway are not competing answers to one question, they are the two halves of the same one. The first makes a model reachable and interchangeable. The second makes it capable of anything outside its own weights. A team that has wired only the first has an agent that talks well and cannot look anything up, which is the most common shape of a disappointing agent demo.
The thing worth taking away is where the leverage sits. Model choice is a decision you make once and revisit quarterly. Tool access is a decision your agent makes every task, which means the tool layer's real job is to be discoverable and priced in the open, not to be clever. Free discovery plus a visible price per call is what lets an agent carry a thousand tools and only spend on the handful it needs.
Free next step: run monid discover against something your agent cannot do today, then monid inspect the top result to see its schema and price. Neither costs anything. Start at monid.ai.
FAQ
What is an LLM gateway key?
It is the single credential your application presents to the gateway, instead of holding one key per model provider. The gateway keeps the provider keys and your app keeps one, which is the same pattern a tool gateway uses: the caller holds one secret and the router holds the rest. Rotating a provider key becomes the gateway's problem rather than a deploy.
Is an AI gateway the same as an LLM gateway?
Usually yes, and the terms are used interchangeably. Where a vendor distinguishes them, "AI gateway" tends to be the broader label covering models plus embeddings, images and sometimes tools, while "LLM gateway" means the model path specifically. Platform products like Amazon Bedrock or Cloudflare AI Gateway sit in the same slot regardless of which word they use for it.
Should I self-host an LLM gateway or use a hosted one?
Self-host when the request path or the custody of provider keys matters enough to own the operations, which is usually a volume or a compliance answer rather than a cost one. Use hosted when you would rather not run the proxy, and accept a vendor in the path. LiteLLM is the usual self-hosted starting point and Portkey's gateway is also open source, so starting hosted and moving in-house is a real option.
How is tool billing different from token billing?
Token billing is continuous and scales with prompt and completion length. Tool billing comes in three shapes: per call, where the request is the unit; per result, where the returned row count is the unit; and per unit of output, such as characters of speech synthesised or minutes of audio transcribed. The shape decides which parameter controls your bill, which is why it belongs in an architecture conversation and not only in a finance one. Current shapes and prices are on monid.ai/tools.
Last updated August 2026.
