How to Configure an Air-Gapped AI Server for Sensitive Enterprise Data

A report lands in the inbox showing which external endpoints staff have handed company documents to. Near the top sits a public frontier-model API, with volume climbing. Cisco's 2025 Data Privacy Benchmark found nearly half of respondents admitted feeding personal employee or non-public data into generative AI tools, even as 64% worried about accidentally leaking sensitive information to competitors. For a bank, a hospital, or a defense contractor, the fix is not another policy memo. It is a tenant the model lives inside. An air-gapped AI server is a build problem with specific hardware, specific wiring, and specific paperwork.

Hardware and Network Topology

An air-gapped deployment begins with the rack and ends at a cable that was never run. Teams building for classified, PHI, or ITAR workloads usually target an inference-first node rather than a training cluster. A dual-socket chassis with eight NVIDIA H100 SXM5 accelerators, each carrying 80 GB of HBM3 and a 700 W TDP, lands around 6.5-7.5 kW of draw before CPU, networking, and cooling. That exceeds the 10-12 kW envelope of most legacy colocation cabinets. Plan for 40-80 kW racks with rear-door heat exchangers or direct-to-chip liquid loops.

Network topology uses two physically separate fabrics. A control plane for KVM-over-IP, console, and BMC traffic sits on a VLAN that terminates inside the SCIF or secure room; it never touches the internet. A data plane carries RDMA traffic between GPUs and the vector store. Sovereignty mandates and the broader shift of AI workloads back on-prem are reshaping the floor plan. The outbound edge is firewalled at Layer 7 and defaults to deny all. No DNS, no NTP, no package manager resolves past the boundary.

Offline Ingestion of Weights and Dependencies

Models and libraries enter the enclave through a one-way transfer process. A sheep-dip workstation in a lower enclave pulls weights (Llama 3.1 70B, Mistral Large, a fine-tuned Phi variant) and verifies SHA-256 hashes against the publisher's signed manifest. Container images, pip wheels, and system RPMs stage the same way. A data diode or write-once optical media carries the vetted bundle across the boundary.

Inside the enclave, an internal mirror answers every request a model server would otherwise send to the open web. Harbor handles containers. A local Pulp or Artifactory instance serves packages. A Hugging Face mirror backed by MinIO holds weight shards. The inference stack (vLLM or TGI on CUDA 12, pgvector or Qdrant for retrieval, FastAPI for the gateway) resolves against that mirror alone. Dependency pinning gets strict. Every requirements.txt and go.sum is cryptographically signed, and the build system rejects any artifact without a matching entry in the internal SBOM. Nothing floats in from the public registry on a whim.

Patch and Update Workflows Without Internet

Patching an air-gapped system is a logistics problem dressed as an engineering problem. The usual cadence is a monthly import window, aligned to the CVE publication schedule and the organization's change advisory board. A dedicated staging enclave downloads kernel updates, CUDA drivers, container base images, and model weight revisions, runs them against a representative workload, and produces a signed update bundle.

That bundle travels across the enclave boundary by removable media or a cross-domain solution with a content inspection appliance. Inside the enclave, Ansible or a Flux-style GitOps runner applies updates to a canary node first, then promotes once the test suite passes. Skipping months breaks assumptions in uncomfortable ways; CUDA compatibility matrices, in particular, punish teams who try to catch up two or three driver versions at once. Vendor firmware for Mellanox NICs and NVSwitch backplanes needs the same discipline. Build the cadence into the SLA, not the backlog. A patch slot that slips erodes the compliance posture before the next audit even opens.

Admin Access and Egress Controls

The incident that moved Samsung to ban ChatGPT across its semiconductor division involved engineers pasting source code into a public chat to debug it. Inside an air-gapped enclave there is no public chat, but there is still a laptop, a keyboard, and a human. Access flows through hardened jump hosts with FIDO2 hardware keys, session recording, and privileged access management from CyberArk, Teleport, or BeyondTrust. Admin sessions are short-lived and tied to approved change tickets.

Egress controls assume the worst case. A data-loss prevention agent inspects every byte leaving the model gateway, redacts anything that trips a classifier (SSNs, MRNs, CAGE codes), and logs the full prompt and response chain to a WORM-protected store. Reverse proxies enforce per-tenant quotas. A zero-trust posture inside the enclave treats the model as another untrusted workload: mTLS between services, SPIFFE identities, no flat networks. Secrets live in Vault or a FIPS-validated HSM, never in environment variables.

NIST, CMMC, and FIPS Alignment

The paperwork determines whether the build is usable at all. NIST released AI RMF 1.0 on January 26, 2023 as a voluntary framework for incorporating trustworthiness into AI systems; its Generative AI profile fills in model-specific controls. For CUI workloads, NIST SP 800-53 Revision 5 is the control catalog of record, and FIPS 140-3 validated cryptographic modules are mandatory for any key material that touches the model or its storage.

Defense contractors inherit a sharper profile. CMMC Level 3 has to be assessed by DCMA DIBCAC rather than a third-party C3PAO, and the control count builds on the SP 800-171 baseline. European exposure brings its own meter: the EU AI Act carries fines of up to €15 million or 3% of global annual turnover for violations of its high-risk system rules. Document the system boundary, map every control to an implementation statement, and attach the evidence (audit log excerpts, cryptographic attestations, access reviews) to each one. A system security plan is a living artifact, not a Word document on a share drive. A clear read on private versus proprietary AI economics belongs in the same binder, because the finance partner will ask.

Choose an air-gapped build when three conditions hold. First, the data cannot leave a defined boundary under any contract or statute (CUI, PHI, ITAR). Second, query volume is high enough that amortized GPU cost beats per-token API pricing. Third, the organization already has the operational muscle (change management, patch cadence, audit cadence) to run a fleet it cannot phone home. If any of those is missing, a VPC-isolated managed deployment is usually the better fit until it is not. For enterprise AI builds that clear the bar, the configuration above is the floor, not the ceiling.