Private AI vs Proprietary AI: The Real Token Math in 2026

Self-hosting an open-weight 70B model beats Claude Haiku 4.5 on cost somewhere north of a billion tokens a month, and almost never beats GPT-4o mini on price alone. That is the whole argument. The rest of this piece shows the arithmetic, where the assumptions move, and when the non-price reasons (data residency, fine-tuning, latency control) change the answer anyway. Hosted API prices have fallen fast enough that the old "open source is obviously cheaper" reflex is out of date for most workloads; the honest buyer question in 2026 is at what token volume the capex of running your own GPUs clears the opex of a hyperscaler API.

The Price Floor Has Moved Under Everyone

Start with the hosted side, because that is the moving target. Anthropic prices Claude Haiku 4.5 at $1 per million input tokens and $5 per million output tokens, with cached input at $0.10/M and batch at $0.50/M in and $2.50/M out. OpenAI's GPT-4o mini sits lower still, at $0.15 input and $0.60 output per million tokens across a 128K context. Those are the numbers a self-hosted stack has to beat.

The drift is steeper than most capex models assume. The Stanford AI Index found that inference cost for a GPT-3.5-level model fell from $20 per million tokens in November 2022 to $0.07 per million by October 2024, a 280-fold cut in roughly 18 months. Menlo Ventures reports the same compression on the open side: open-source models' share of enterprise token volume climbed from 11% to 19% year over year while their share of spend fell from 6% to 4%, driven by a 3.3x drop in average open-source token prices. Any break-even you compute today has a half-life measured in months.

What an H100 Actually Produces Per Hour

Throughput is where the self-hosted case has to be earned, not quoted. On a single H100 80GB running Llama 3.3 70B in FP8 under vLLM, disclosed 2026 benchmarks show 120–130 tok/s at batch 1, scaling to roughly 2,400–2,780 tok/s at 100 concurrent requests. That batching curve is the whole economics of self-hosting. At batch 1 your effective cost per token is awful; at steady concurrency it is competitive.

Take the middle of the published range: 2,400 output tokens per second sustained. That is 8.64M tokens per GPU-hour, or 6.3B tokens per month at 100% duty cycle. Nobody runs at 100% duty cycle. A realistic production figure is 30–50% average use once you account for diurnal traffic, warm pools, and bursty load. That cuts usable monthly output to roughly 1.9–3.2B tokens on one GPU.

Qwen2.5-72B behaves similarly on the same hardware. The vendor's own Qwen2.5 speed benchmark shows the 72B-Instruct model running at around 31 tok/s single-stream in BF16 across four GPUs, with higher aggregate throughput once batching is applied. The pattern is consistent: a 70B-class open weight model on one H100 under a batched serving stack lives in the 2,000–3,000 tok/s aggregate range.

Hourly GPU Cost Is the Other Lever

The second number you need is rent. AIMultiple's 2026 Cloud GPU Rental Price Index pegs H100 rentals at roughly $1.49 to $12.29 per GPU-hour, with the median near $3.38, down from above $7 in early 2024. Pick your lane: $1.49 is a spot-market Neocloud with no SLAs, $12.29 is on-demand hyperscaler with enterprise support, $3.38 is where most serious reserved commitments land.

At $3.38/hr × 730 hours, a single H100 costs about $2,467/month before any ops overhead. Add engineer time, monitoring, autoscaling, security review, model updates, evals, and the fully loaded figure for a small production fleet usually clears $4,000–$6,000 per H100-month. On-prem purchase economics shift the ratio further if you have the real estate and power, but the billable figure most teams model against is still rented.

The Break-Even Number Buyers Keep Asking For

Collapse those into one table. Assume a typical chat or RAG mix of 70% input / 30% output tokens. Blended API price per million tokens then works out to about $2.20 for Claude Haiku 4.5 and about $0.285 for GPT-4o mini. Self-hosted cost per million tokens on one rented H100 is the monthly bill divided by usable monthly output.

At 50% average use and $3.38/hr, a 70B model at 2,400 tok/s delivers roughly 3.15B tokens/month at a derived cost close to $0.80 per million output tokens under the engine's own benchmark. Add ops loading and the real figure for a small team sits closer to $1.50–$2.00 per million blended. That gives you three honest break-even points:

• Versus Claude Haiku 4.5 ($2.20/M blended): break-even lands around 1.1–1.4B tokens per month on a single H100 before you are saving money. Above roughly 3B tokens/month the margin starts to matter.
• Versus GPT-4o mini ($0.285/M blended): raw price almost never wins. You would need sustained use above 80%, spot-tier GPU rates near $1.49/hr, and essentially zero ops overhead. In practice: not a cost decision.
• Versus Claude Haiku 4.5 with batch discount ($1.10/M blended): break-even roughly doubles, to 2.2–2.8B tokens per month. Batch APIs push the self-hosted case out further than most spreadsheets admit.

Two things move those numbers hard. First, prompt caching: Haiku 4.5 reads cached input at $0.10/M, which kills the economics of self-hosting a repetitive-prefix workload. Second, your mix: a heavy-output workload (code generation, long synthesis) flips in favor of self-hosted sooner because output tokens are where hosted pricing takes its margin. Run the math on your actual input/output ratio, not a generic one.

What the Token Math Leaves Out

Price per million tokens is the wrong lens for several real buying decisions, and pretending otherwise is how teams end up with a GPU cluster they regret. Open weights still win on four axes the API bill does not show: data residency, fine-tuning freedom, latency floor, and vendor risk. Menlo's 2026 data shows three labs capturing 69% of enterprise AI deployment spending; concentrating a regulated workload on a single API vendor is a governance question before it is a cost one.

There is also a license caveat on the "open" side. Llama is open-weight rather than open source, with the Llama Community License requiring a separate Meta license once a product crosses 700 million monthly active users and banning use of Llama outputs to train competing models. Qwen, DeepSeek, Kimi, and GLM have their own terms, and Chinese-origin weights carry procurement review in some sectors. Menlo reports that Chinese models now account for 89% of enterprise open-source LLM spending, led by Kimi at 61%, DeepSeek 15%, GLM 11%, and Qwen 3%. That share is a procurement conversation, not a performance one.

Operational surface is the other hidden tax. A self-hosted LLM is a service: it needs on-call, eval pipelines, GPU driver updates, model version management, abuse controls, and the same observability any other production dependency gets. Teams that cost it correctly build it as part of a production RAG system or an agent stack, not as a standalone VM. If you are not ready to staff that, the API bill is cheaper than the pager.

A Decision Procedure That Survives Price Cuts

Given how fast hosted prices move, the sensible move is to encode the decision as a procedure, not a one-shot spreadsheet. The sequence below is the one that holds up for most AI engagements we take on, whether the eventual stack is private or proprietary.

Two more guardrails. Build the stack so the inference provider is swappable: an OpenAI-compatible API surface in front of your calls lets you move between Claude, GPT-4o mini, a vLLM cluster, or a managed open-weights endpoint without rewriting the application. And keep the eval harness honest. A 70B open-weight model and Haiku 4.5 do not behave identically on your prompts, so a cost comparison that ignores quality delta on your actual traffic is a comparison of two different products. Our write-up on building an AI team that actually ships covers the harness side; the in-house rollout piece covers staffing the ops side.

The Signal to Watch

The number worth tracking is your monthly token volume on the single most expensive route in your app, expressed as a cost per million blended tokens on your current provider. When that line crosses roughly $1,500–$2,000 per month on a route that looks like a 70B-class workload, self-hosting starts to pencil out; when it stays below $500, API pricing is almost certainly the right answer regardless of philosophy. The value that should worry you is the opposite direction: a self-hosted deployment whose fully loaded cost per million tokens stays above $2 after six months in production. At that figure the GPU is losing to Haiku on price and to GPT-4o mini on everything, and the private-AI case has to be made on governance alone.