Self-Hosting LLMs in Production: The 2026 Decision Guide

Toolradar data: the median entry price for B2B software we track is $19 per month against a $116 average, which is the number to hold against a GPU bill when deciding whether to self-host at all.

Running a model on your laptop is a weekend project; serving one to production traffic is an infrastructure commitment. This guide is about the second thing: when it is justified, the stack that has standardised, and the costs that do not appear in the enthusiastic blog posts. For the desktop version, see running LLMs locally.

The three reasons that hold up

Data cannot leave. Regulated data, client confidentiality, or contracts that forbid third-party processing. This is the cleanest justification, and the only one that is not a cost calculation, so it is the one that survives scrutiny.

Predictable heavy volume. APIs price per token; a GPU prices per hour. Sustained high-volume workloads cross the line where owned capacity is cheaper, but the line moves every time API prices drop, and they keep dropping, so redo the arithmetic quarterly against our measured pricing data rather than last year's assumption.

Latency floors. On-prem serving removes network round trips and queue variance. Real for interactive products with tight latency budgets; irrelevant for batch work, where a few hundred milliseconds never mattered.

The reason that does not hold up: "control". Stated without a workload behind it, control means paying for GPUs to feel better about a dependency you still have on the model weights someone else trained.

The stack that standardised

vLLM is the serving engine the ecosystem settled on: continuous batching and PagedAttention turn a GPU into a high-throughput endpoint, and its OpenAI-compatible API means application code does not care whether it talks to you or to a provider. That compatibility is what makes self-hosting reversible, which is the property you want while the economics keep moving.

LocalAI offers the same OpenAI compatibility for smaller deployments. Llama.cpp owns the CPU and edge niche, running models where no datacenter GPU exists. Ollama increasingly appears in production for internal tools, though it is tuned for developer experience over raw throughput, so it fits internal-tool traffic better than a high-QPS public endpoint.

Open-weight model quality is what made all of this viable. The gap to frontier APIs, for bounded jobs, is now routinely inside the tolerance of a well-evaluated use case. The catalog view is in our local LLM tools guide.

The costs the posts leave out

The GPU is the visible cost. The invisible ones are what sink projects:

• An engineer who owns serving. On call, upgrades, CUDA version archaeology. This is a role, not a task, and it does not disappear after launch.
• Evaluation of every model you swap in. Open-weight releases move fast, and each swap is a regression risk against your use case. Without a test set, every upgrade is a gamble.
• Capacity you pay for at peak while the API bills your average. A GPU sized for your busiest hour sits idle the rest of the day; the API charged you only for what you used.
• Security patching of the serving layer. vLLM and its dependencies are software you now operate and must keep patched, a surface the API abstracted away.

Teams that succeed treat self-hosting as a product with an owner. Teams that fail treat it as a deployment and discover the operating cost after the GPUs arrive.

Open-source is not open-weights, and the difference is legal

The distinction that trips up most first deployments: a model with published weights is not necessarily a model you are licensed to use commercially. "Open-weights" means you can download and run the weights; the licence attached decides whether you can do so in production, at what scale, and whether outputs carry restrictions. Some permissive licences allow anything; others cap commercial use by company size or forbid using outputs to train competing models. This is where most self-hosting legal risk originates, and it is a five-minute check per model that teams skip and regret. Read the licence before the model, not after the GPUs arrive.

The pattern that actually wins

Most production systems land on a hybrid: self-hosted for the high-volume bounded workload that justified the GPUs, API for everything else, and one gateway in front of both so application code never knows the difference. The gateway makes the hybrid reversible, which is exactly what you want while the cost lines keep crossing back and forth.

FAQ

What hardware for a first serious deployment?

One modern datacenter GPU serves a quantised mid-size model for internal-tool traffic comfortably via vLLM. Size from your eval set's quality floor, not from the largest model that fits the card.

Fine-tune or prompt?

Prompt first, always: cheaper to iterate, nothing to redeploy. Fine-tuning pays only when the task is stable, narrow and high-volume, the same profile that justifies self-hosting at all.

How do I know if the economics work?

Take your monthly token volume, price it at current API rates, and compare against a GPU-hour cost plus the engineer-time to operate it. If the API number is not comfortably larger, the API wins, because it has no operating cost.

Related

See what an LLM gateway is for the layer that makes hybrid work, and the infrastructure stack for where models sit.