Quinn Favo
Back to Articles
•8 min read

What Does a Private LLM Actually Cost? GPUs, APIs and the Ops Bill Nobody Quotes

A private LLM is rarely cheaper than an API on tokens alone. It pays off when your data cannot leave, when volume keeps GPUs busy, or when you need to pin a model. Here is how to run the numbers honestly.

Short answer

For most teams, an API is cheaper on raw token cost: one always-on H100 rents for roughly $2,400 to $3,100 a month as of October 2026, which buys hundreds of millions of tokens from a mid-tier API. Self-hosting pays off when data must stay on your infrastructure, when steady volume keeps the GPUs busy, or when you need full control of the model.

Start with the right question

When someone asks me whether they should run their own model, the cost question usually hides a different one. Sometimes it is "our legal team will not let customer data go to a third party". Sometimes it is "our API bill tripled last quarter". Those lead to very different answers.

So I split it in two: what does it cost per month to serve your actual workload each way, and what would you be buying besides tokens? Control, data residency and a pinned model version are real things you can pay for. They are just not the same as saving money.

What GPUs cost to rent

As of October 2026, Lambda's on-demand price for an NVIDIA H100 SXM with 80 GB of memory is $3.99 to $4.29 per GPU per hour depending on the instance size, an H100 PCIe is $3.29, an A100 with 40 GB is $1.99 and an A10 with 24 GB is $1.29, plus tax. Other clouds and reserved contracts differ, so check current pricing before you budget.

A month is about 730 hours. One H100 left running all month is therefore roughly $2,400 to $3,100. An A10 is about $940. Reserved capacity is cheaper, but you are committing to the spend whether you use it or not.

What the model needs in memory

GPU choice follows from model size. A rough rule: at 16-bit precision each parameter takes two bytes, so an 8B model needs about 16 GB just for weights and a 70B model about 140 GB. The H100 has 80 GB, so a 70B model at full precision needs at least two of them, or quantisation to bring it down. On top of the weights you need room for the KV cache, which grows with context length and the number of concurrent requests.

Serving engines matter as much as hardware. vLLM, for example, uses PagedAttention to manage KV cache memory, batches incoming requests continuously, caches shared prefixes, and supports FP8, INT8 and INT4 quantisation. Those features are the difference between one GPU serving a handful of users and serving a whole team.

What the same work costs on an API

As of October 2026, Anthropic lists Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output (for prompts up to 100K tokens), Sonnet 5.5 at $2 and $10, and Opus 5.5 at $4 and $20, with 50% off for batch processing and cheaper rates for cached prompt reads.

Take an example workload of 50 million input tokens and 10 million output tokens a month, which is a busy internal assistant for a small company. On Sonnet 5.5 that is about $200. On Haiku 5.5, about $10. Even on Opus 5.5 it is about $400. All of those are well under the cost of one always-on H100.

Flip it around: at Sonnet 5.5 rates, the monthly cost of one H100 buys roughly 600 million input tokens and 120 million output tokens before any batch or cache discount. Most teams are nowhere near that.

Utilisation is where self-hosting wins or loses

Cost-per-token figures for self-hosting assume the GPU is busy. NVIDIA, citing SemiAnalysis benchmarks from April 2026, puts H100 inference at about $0.09 per million tokens for GPT-OSS-120B on vLLM. That is a throughput benchmark at high load. Your internal tool that is busy from 9 to 5 and idle overnight and at weekends will not get anywhere near it.

The honest calculation is monthly GPU cost divided by the tokens you actually serve. If a GPU sits at 15% utilisation, your effective price per token is several times the benchmark. Steady, high-volume workloads such as batch document processing or a product feature with constant traffic are where self-hosting starts to compete.

The ops bill nobody quotes

GPU rental is the line item people compare. These are the ones they forget:

  • Redundancy. One GPU node is a single point of failure. A second node for availability doubles the compute bill.
  • Engineering time. Upgrading drivers, CUDA, the serving engine and the model, and debugging memory issues, is ongoing work. If it takes one engineer a day a week, that is a fifth of a salary.
  • Evaluation. Every model swap needs a test run against your own tasks, or quality quietly drifts.
  • Observability. Latency, queue depth, GPU memory, error rates and token throughput all need dashboards and alerts.
  • Security and access. Authentication, network isolation, patching and audit logs are now your job, not the provider's.
  • Storage and networking. Model weights, logs and any retrieval index need storage, and some clouds charge for egress.

When a private LLM does pay off

Self-hosting is the right call more often than the token maths suggests, just for different reasons. I recommend it when one or more of these is true:

  • The data cannot leave your network for regulatory, contractual or client reasons, and a provider's data terms are not enough.
  • You have steady, high volume that keeps GPUs busy most of the day.
  • You need an air-gapped or isolated environment.
  • You need to pin an exact model version for months, for reproducibility or validation.
  • A smaller open model, possibly fine-tuned, is good enough for a narrow, high-volume task.

The middle path most teams end up on

In practice the answer is often hybrid: sensitive or high-volume work on a private model, and everything else on an API, with a router deciding per request. That keeps the private deployment small and busy, which is where its economics work, and leaves the long tail to pay-per-token pricing.

If you want the numbers for your own workload, my LLM Private Cloud package starts with exactly this sizing and then builds the deployment: model selection, inference endpoints, observability and retrieval over your internal data, on infrastructure you own. If you are not sure yet whether private makes sense, a conversation about your volume and data rules usually settles it.