What Is an LLM Gateway? The 2026 Guide to Model Traffic Done Right

Toolradar data: of the 10 dedicated LLM gateways we track, 9 offer a free tier. The price of entry to this category is zero; the cost is operating what you install.

An LLM gateway is one endpoint between your applications and every model provider you use. Your code calls the gateway; the gateway holds the provider keys and handles routing, retries, caching, budgets and logging across OpenAI, Anthropic, Google and whatever ships next quarter. This guide covers what that buys you, what it costs you, how the main options differ, and the four signals that you actually need one.

What is an LLM gateway? The quick answer

A reverse proxy specialised for model traffic. Requests go in speaking one API shape (almost always the OpenAI-compatible one), the gateway translates to whichever provider is configured, and the response comes back with latency, cost and token counts logged. To your applications, every model in the world looks like one endpoint with one key.

That is the whole trick, and its value scales with how many apps, providers and teams sit behind it.

The problem it exists to solve

Without a gateway, every application that calls a model owns its own provider keys, retry logic, spend tracking and fallback behaviour. At one app and one provider this is fine, and a gateway would be overhead. The arithmetic changes fast:

• Keys multiply. Five apps and three providers is fifteen key-to-app relationships. When a key leaks, revoking it becomes an audit across codebases instead of one operation.
• Nobody can answer the spend question. "What did we spend on inference last week, by feature?" is a spreadsheet-and-guesswork exercise when each app logs its own calls, and a single query when everything transits one gateway.
• Provider incidents become your incidents. Model API outages stopped being rare events in 2025. Without central failover, each application handles them separately, which in practice means most do not handle them at all.
• Every app reinvents retry logic, usually with subtle bugs: unbounded retries against a rate-limited endpoint are a self-inflicted denial of service, and at least one team per company learns this on an invoice.

How a gateway works

The request path is short: your app sends an OpenAI-shaped request to the gateway with an app-level key. The gateway authenticates the app, checks its budget, applies routing rules (which model, which provider, which fallback chain), translates the request to the target provider's format, and forwards it with the real provider key. On the way back it records latency, tokens and cost, optionally caches the response, and returns it unchanged.

Overhead is real but small when self-hosted: single-digit milliseconds in the proxy itself. Hosted gateways add a network hop, which puts them in the tens of milliseconds. For chat products neither matters; for latency-critical completions the difference between self-hosted and hosted is worth measuring before you commit.

Gateway against direct integration

The last row is the honest one, and it deserves its own section.

The tradeoff the vendor pages omit

A gateway is a single point of failure that you now operate, sitting on the hottest path you have. When it goes down, every AI feature in the company goes down together, which is precisely the blast radius you built it to prevent for provider outages. The mitigation is boring and non-optional: run it redundant, or accept the coupling knowingly for internal tools where an hour of downtime is a nuisance rather than an incident.

The second cost is organisational. A gateway centralises decisions that used to be each team's own: which models are approved, what budgets apply, what gets logged. That is governance when your compliance team asked for it, and friction when a product team wants to try the model that shipped yesterday. Decide who owns the routing table before you install one, because that argument arrives either way.

The main options, compared

We track 10 dedicated gateways. These are the six worth evaluating first, in rough order of how often they win the deployments we see.

LiteLLM

The open-source default, and the highest-traffic product in the category's search results by a wide margin.

Best for: teams that want a self-hosted proxy with the broadest provider coverage and no per-request fee.
Pros: 100+ providers behind the OpenAI format; runs as a Python proxy you own end to end; enormous community; per-key budgets and rate limits built in.
Cons: you operate it, upgrade it, and scale it; the dashboard is functional rather than polished; enterprise features (SSO, audit) sit behind the paid tier.
Pricing: open source, free self-hosted; enterprise tier priced on contact. Our LiteLLM listing.

Portkey

The most complete commercial package: gateway, observability and guardrails in one product.

Best for: production teams that want governance (PII redaction, content policies, approvals) in the same layer as routing.
Pros: guardrails and compliance features are first-class, not add-ons; strong config-over-code routing; both hosted and self-hosted deployment.
Cons: the free tier's request cap is evaluation-sized; full feature depth means more surface to learn than a plain proxy.
Pricing: free tier, then usage-based paid plans. Our Portkey listing.

Helicone

Observability-first: it started as LLM logging and grew the gateway role.

Best for: teams whose first pain is "we cannot see what our LLM calls do or cost", with routing second.
Pros: the logging, cost dashboards and session traces are the best in the group; one-line integration via base-URL swap; generous free tier.
Cons: routing and failover are younger than the observability core; gateway-mode adds their infrastructure to your request path.
Pricing: free tier, then per-request paid plans. Our Helicone listing.

OpenRouter

A hosted many-models-one-API service with a marketplace angle: one account, unified billing across hundreds of models.

Best for: products that want instant access to many models without holding any provider relationship at all.
Pros: zero infrastructure; new models appear without you doing anything; transparent per-model pricing with a small routing fee.
Cons: it is in your data path and your billing path; per-request fee compounds at volume; less of a governance story than the self-hosted options.
Pricing: pay per use, small percentage over provider list prices. Our OpenRouter listing.

Kong AI Gateway

The API-management incumbent extending into model traffic.

Best for: platform teams already running Kong who want LLM routing under the same control plane as the rest of their APIs.
Pros: mature gateway operations (plugins, rate limiting, auth) inherited from a decade of API management; one vendor for all traffic, not just AI.
Cons: heavyweight if you are not already a Kong shop; AI-specific features trail the specialists.
Pricing: open source core, enterprise licensing on contact. Our Kong listing.

ZenMux

A newer entrant in the hosted unified-API mould.

Best for: smaller teams that want OpenRouter-style simplicity and are willing to trade maturity for it.
Pros: simple setup; free tier to start.
Cons: youngest product on this list, with the track record that implies.
Pricing: free tier, then usage-based. Our ZenMux listing.

The four signals you need one

1. Two or more applications call models, or one application calls two or more providers. Below that, direct calls win on simplicity.
2. The spend question takes more than a minute. If "what did feature X cost in tokens last week" has no quick answer, the logging alone pays for the gateway.
3. A provider incident hurt you once already. Failover is the feature teams adopt gateways for the week after the outage, and it is cheaper to be early.
4. Compliance asked for prompt logs or data controls. Central logging and redaction beat retrofitting five codebases.

Absent all four, call the provider directly and revisit in six months. A gateway installed before the need is capacity planning cosplay, and it adds a failure mode while solving nothing you have.

Implementation practices that survive production

Adopt via base-URL swap, not SDK rewrite. Every serious gateway speaks the OpenAI-compatible API. Changing a base URL and key per app is an afternoon; rewriting against a proprietary SDK is lock-in with extra steps.

Set budgets before the first team onboards. Hard per-key budgets with alerts at 80% turn the runaway-agent scenario from an invoice into a log line.

Route by rule, not by hardcode. The point of the gateway is that "which model" lives in configuration. Teams that hardcode model names in app code through a gateway have bought infrastructure to not use it.

Keep an escape path. Test that each app still works pointed directly at a provider. The day the gateway itself is the incident, you want the failover to be a config change, not an architecture project.

Where this sits in the broader stack

A gateway governs model traffic. The adjacent and different problem of governing agent tool traffic belongs to MCP gateways, and the two compose rather than compete: production agent stacks increasingly run both, one for models, one for tools. For the full picture of what sits around them, see the AI agent infrastructure stack, and for the comparative buying view, our LLM gateways guide.

FAQ

Is an LLM gateway the same thing as an AI gateway or an LLM router?

The products converge; the words came from different vendors. "Router" emphasises picking the model per request, "gateway" emphasises the operational wrapper of keys, budgets and logs. Evaluate the feature list, not the noun.

Does a gateway create lock-in?

Less than the alternative. Because the good ones speak the OpenAI-compatible API, leaving is a base-URL change. The real switching cost is your accumulated config and logs, which is why exportable logging is worth checking before you adopt.

Self-hosted or hosted?

Self-hosted (LiteLLM, Kong, Portkey's self-hosted mode) when latency, data residency or compliance dominate. Hosted (OpenRouter, Helicone cloud, ZenMux) when operating another service is the thing you are trying to avoid. Teams with a platform function tend to self-host; teams without one should not acquire one for this.

How much latency does a gateway add?

Self-hosted proxies add single-digit milliseconds. Hosted services add a network round trip, typically tens of milliseconds depending on geography. Measure with your own traffic before believing anyone's number, including these.

Can a gateway cut my inference bill?

Indirectly and genuinely: caching identical requests, routing to cheaper models where rules allow, and making waste visible. Teams typically report the visibility mattered most, because you cannot cut what you cannot attribute.