Field Notes — Building CI/CD Inside an Air-Gapped Datacenter

The first time we deployed our CI inside an air-gapped customer datacenter, the build hung for 38 minutes before anyone said anything. We were standing in the customer's NOC. They were polite about it.

The hang turned out to be a npm install quietly trying to reach registry.npmjs.org through a firewall it would never get through. Nothing surfaced as an error — just retry, retry, retry, until somebody noticed.

That moment is the entire content of this post.

The assumption that broke

Modern CI tools — even the self-hosted ones — assume the network looks something like this:

"We can probably reach the public package registry. If not, we'll fail with a clear error."

Inside that "probably" is everything. Public registry, base image, OIDC issuer, telemetry endpoint, license server. Each of those is a separate spoke in the assumption, and each one fails differently when the network gateway answers connection timed out for sixty seconds and then closes the socket.

We learned to grep our build logs for every outbound DNS lookup in the first week. That sounds paranoid. It is not paranoid. It is the only way to know what your CI actually depends on.

What you have to rebuild, in order

Not three pillars. Not a maturity model. A list, in the order they bit us.

Dependency mirrors. First. Always first. Every language ecosystem assumes a registry one router hop away — npm, PyPI, Maven, Cargo, Go modules. You need an internal mirror for each one your codebase touches, and you need a one-way sync from a transit zone that has very limited internet. We use Verdaccio for npm, devpi for Python, Nexus for Maven and Docker. Total disk: about 2.4 TB after a year, growing.

Base images. The FROM ubuntu:22.04 line is fine on your laptop and an act of theology in the customer datacenter. We banned unqualified FROM in CI policy two months in. Every base image must come from an internal registry that the security team has signed off on. The cost of this rule is real — engineers complain — but the day you can't reproduce a year-old build because the upstream image has rotated, you will agree.

Workers. This is where most cloud-CI mental models break hardest. Workers in cloud are ephemeral; you spin them up, they do one thing, they die. Inside an air-gapped environment, workers are often long-lived because procurement of more hardware takes a quarter. So you start to see things you never see in cloud: secrets bleeding between builds via cached layers, workers slowly accumulating capabilities they shouldn't have, and slow drift between two workers that should be identical because someone manually applied a patch one Tuesday in February.

We solved this with three rules. Workers boot from a signed immutable image. The build runs inside a container inside the worker — yes, double containerization, yes, it costs you. And every credential is a job-bound token that expires when the build does, with no exceptions, including for "just this one quick re-run" requests.

Secrets. External KMS doesn't exist for us. The pattern that has worked is OIDC-based workload identity backed by an internal Vault. The CI presents an OIDC token, Vault inspects the claims, Vault issues a short-lived credential to the actual target system. The token never sits in an env var, in a logfile, or in a config file. If you've never set this up before, budget a week.

Caches and artifacts. Internal S3-compatible storage (Ceph, MinIO) — fine. The trap is the retention policy. On-prem storage is not infinite. We had a build cache that grew to 11 TB before someone asked who owned it (no one). Decide on retention before your first cache write, not after.

The decision table I wish someone had handed me

Five rows. Each one is a quarter of work the first time, a week of work each subsequent customer.

A small note about CollabOps

This is the work that taught us we couldn't just ship "another CI vendor." Every existing CI tool we evaluated had at least one of these assumptions wired in deeply enough that ripping it out would have left us with a fork we'd maintain forever. We wrote our own workflow expression engine and our own runtime because of decisions in this list — not because we wanted to. (Why we built our own expression engine.)

A list you'll read twice

If you're taking cloud-native software into an air-gapped environment at a regulated bank, a public-sector tenant, a defense contractor, or any large enterprise with a real CISO, the five-row table above is the thing to take from this post. Read it once before you start, and once after your first customer is live. The first read will feel theoretical. The second will feel obvious.

If you are not heading into that environment, this post probably reads as paranoia. It is paranoia. It is also accurate.