Why AI Workloads Are Moving Back to On-Prem Hardware

The pitch for renting GPUs was always that elasticity beats ownership. Spin up when you need to train, spin down when you do not, and let someone else carry the capital cost of silicon that depreciates on an eighteen-month cycle. That math held when a handful of research teams were fine-tuning BERT on weekend bursts. It bends when a mid-sized SaaS company is running continuous inference against a 70B model, pulling terabytes of training data per week, and watching a single AWS line item cross six figures a month.

The repatriation story is not about ideology. It is about unit economics, egress fees, and the uncomfortable fact that a single 8-GPU H100 server exceeds $300,000 upfront but pays itself off faster than most finance teams expect once utilization crosses a reasonable threshold. The question for 2027 planning is not whether to leave the cloud. It is which workloads earn a seat on hardware you own.

The Rental Premium Has a Floor, and You Have Already Hit It

Cloud GPU prices fell hard between 2024 and mid-2026, then stopped falling. The AIMultiple index put median H100 rental at roughly $3.38/hr in mid-2026, down from north of $7/hr in early 2024. The spread across providers tells the real story. Marketplace spot sits near $1.49/hr. Hyperscalers charge around $6.88 per GPU-hour on an AWS p5.48xlarge and up to $12.29/hr on Azure for identical silicon.

Rent an H100 at a mid-tier $3.25/hr and you pay about $28,500 per GPU per year at full utilization. Buy the same card for $30,000 and the hardware is paid for before the second year of rental begins. The break-even math is not subtle: an H100 purchased for $30,000 breaks even against a $2.00/hr rental after roughly 15,000 GPU-hours, which is twenty-one months of continuous use.

That is before you count egress.

Egress Is Where the Cloud Bill Actually Grows

GPU-hours are the headline. Data transfer is the surprise. AWS charges $0.09/GB for the first 10 TB of monthly internet egress after a 100 GB free allowance, with GCP Premium at $0.12/GB and Azure at $0.087/GB. For a RAG system serving tens of millions of inference requests a month, that is a rounding error. For a fine-tuning pipeline that re-pulls checkpoints across regions or hands off model artifacts between clouds, it is not.

Specific example from the pricing tables: a 1 TB checkpoint re-pulled from S3 to a fresh instance costs about $90 in egress every time. Common Crawl pulled cross-region inside AWS costs roughly $1,843 at the $0.02/GB inter-region rate. NAT Gateway processing adds $0.045/GB on top of standard egress for anything leaving a private subnet. None of these numbers appear in the proposal from the hyperscaler account team.

Flexera's 2026 survey of 753 cloud decision-makers found wasted cloud spend rose to 29%, the first year-over-year increase in five years, driven largely by AI complexity. The waste is not sloppiness. It is architectures that were designed when storage and compute were cheap and bandwidth was free, now hosting workloads where none of those assumptions hold.

What the TCO Model Actually Looks Like

A credible on-prem or colocated model has to account for more than silicon. The honest line items for an 8-GPU H100 node, amortized over four years:

• Hardware: $300,000–$400,000 for the server, networking, and NVMe storage
• Power: ~10 kW continuous draw at roughly $0.09/kWh wholesale, before PUE overhead
• Cooling and facility overhead: multiplied by the facility's PUE. Uptime Institute puts the 2026 industry-average PUE at 1.52, with newer purpose-built AI halls at 1.3 or lower
• Colocation space: $800–$1,500 per rack per month in most US metros, higher for liquid-cooled high-density halls
• Remote hands, bandwidth, insurance, spares: another 10–15% on top

Run the numbers and a well-utilized 8-GPU node lands around $2.00–$2.50 per effective GPU-hour over four years. That is 30–60% below mid-tier specialist clouds and well under hyperscaler pricing. The caveat matters: these numbers assume utilization above 60%. A cluster that sits idle most of the week will lose to spot pricing, every time.

The reference case that gets quoted is 37signals. The company behind Basecamp and HEY began moving off the public cloud in late 2022 after spending $3.2 million annually, with on-prem operations later dropping to $1.3 million and projected five-year savings exceeding $10 million. That is not an AI workload, but the shape of the argument is identical: predictable, steady-state consumption at scale is where ownership wins.

Hybrid Is Winning, Full Repatriation Is Not

The honest market signal is not a mass exodus. IDC's October 2024 analysis found only about 8–9% of companies plan full workload repatriation, with most selectively moving specific workloads back while remaining hybrid. A separate IDC survey of 2,250 IT professionals found 81% expected some level of compute repatriation within twelve months, with AI workloads most often cited for full repatriation at 10%.

The pattern is pragmatic. Training runs and steady inference go on owned or colocated hardware. Burst capacity, experimentation, and geographically distributed serving stay in the cloud. Teams that have thought about this seriously end up with a private AI stack for the predictable workload and a thin cloud footprint for everything else. This is the same strangler-fig pattern engineers already use for legacy modernization: strip the expensive, stable pieces off first; leave the volatile edges where elasticity still matters.

When Owning Hardware Actually Wins

The decision is not philosophical. It is a utilization-and-data-gravity question with four honest triggers:

1. Sustained utilization above 50–60%. Anything less and spot or reserved cloud pricing wins. A cluster that trains three days a week and sits dark the other four does not justify the capex.
2. Data gravity on your side of the fence. If your training corpus already lives in your own object store, your warehouse, or a regulated environment, every pull across the cloud boundary is a tax. If the data is already in S3 and stays there, the math inverts.
3. Regulatory or contractual isolation requirements. HIPAA, defense work, EU data residency, and bank-grade controls are easier to defend on hardware you control. This overlaps with the usual arguments for multi-tenant isolation but is more acute for model weights, which are themselves regulated artifacts in some jurisdictions.
4. Model weights you own and intend to keep. Fine-tuned Llama, Mistral, or domain-specific models are intellectual property. Running them on infrastructure you also own closes a loop that proprietary API vendors cannot match.

If three of four apply, run the model. If only one or two do, hybrid is almost certainly correct, and the right question is which specific workload to repatriate first, not whether to abandon the cloud.

The 2027 Planning Picture

Infrastructure demand is not slowing. The IEA projects global data center electricity consumption rising from about 485 TWh in 2025 to roughly 945 TWh by 2030, with AI-focused centers accounting for the largest share of growth. That demand is already pricing capacity. Power is becoming the binding constraint for colocation providers in the usual metros. Hardware lead times for H100 and B200 inventory still stretch months in some channels.

The practical implication for anyone budgeting 2027 AI infrastructure: decisions made in Q1 commit silicon that arrives in Q3. If your workload passes the utilization and data-gravity tests, waiting another year to see what cloud prices do is a bet that mostly loses. The reserved-instance commitments getting signed right now already assume this; they are three-year deals at prices that look an awful lot like amortized ownership, minus the depreciation benefit.

None of this means dismantling the AWS account. It means being honest about which workloads benefit from elasticity and which have been subsidizing a vendor's capex for two years. The RAG system serving a thousand internal users every weekday does not need elastic. The nightly batch job that fine-tunes a model on last week's support tickets does not need elastic. The experimentation sandbox for your research team almost certainly does. Price each one on its own terms. The answer stops being a slogan and starts being a line item.