How Many GPUs Does an Air-Gapped AI Coding Agent Need? — Sizing Notes on H100s

Meet with an institution evaluating AI agents in an air-gapped network and the questions eventually converge on one point: "So how many GPUs do we need?" What happens past a hundred people, what's the minimum to start, what do we even run if cloud models are off the table. All variations of the same question. Here's the whiteboard sketch I keep redrawing, written down once.

How many GPUs per concurrent user count?

The baseline: with a 32B-class coding model, one 80GB GPU (A100/H100) handles 5–10 concurrent agent sessions. So for a mid-size configuration around ten concurrent users, one H100 80GB is the starting point, and with operational headroom and failure tolerance we recommend two. Large institutions with dozens of concurrent users start at two cards and up, and from there the arithmetic is simple:

required GPU nodes ≈ concurrent agent sessions ÷ (5–10 sessions per 80GB card)

The thing to watch is that the unit is concurrent sessions, not headcount. No organization has all hundred employees running agents at the same instant. But the reverse also holds: unlike a person, one user can run several agent sessions in parallel. So as agent usage matures in a hundred-person organization, concurrent sessions start crossing into the dozens, and that's when "dozens of H100s" stops being an exaggeration. Usage patterns size the cluster, not the org chart.

The VRAM arithmetic, for completeness: a model's memory footprint is roughly parameter count × bytes per parameter (FP16 = 2, INT8 = 1, INT4 = 0.5), plus 20–40% for KV cache. A 32B model at INT8 is 32GB plus cache, fitting comfortably on one 80GB card; drop to INT4 and it runs on a single 48GB-class card (L40S). That's the minimum PoC configuration.

We can't use cloud models — what do we actually run?

What runs in an air-gapped network is an open-weight model, and our current sweet spot is Qwen2.5-Coder-32B (Apache 2.0). Coding ability, tool-calling reliability, license terms, and GPU cost balance out right there. Below it, 7B-class lightweight models are fine as assistants but break structured output too often to be the agent's engine. Above it sit the 70B class and 80B MoE families, where you're at two-plus GPUs. The 100B+ large MoE models need an 8-GPU node, and frankly we talk most institutions out of them in meetings — it's over-investment. For organizations with a domestic-model requirement, the LG EXAONE family is the option, with the caveat that commercial use requires a separate license agreement, worth confirming early.

Here's the part that demands honesty. Air-gapped open models are not on par with cloud frontier models. Claude-class models do not come on-prem. What we see in practice is that a 32B open model, with context injection and an execution layer properly attached, approaches the coding ability of a frontier model from a generation or two back. Be wary of vendors who inflate that. There is one advantage unique to on-prem, though: keep the hardware, swap the model file, and performance goes up. The open-model ecosystem improves on a roughly six-month cadence, so the GPU you buy today is one of the rare assets that appreciates. Operational details are in nine months of Llama/Qwen in an air-gapped network.

Why do agents burn so many tokens?

Agents consuming tens of times more tokens than a chatbot is structural, not an implementation flaw. A chatbot produces one answer for one question and stops. A coding agent reads files, plans, edits, runs the build, reads the failure log, edits again. Every lap of the loop feeds the previous context back in as input. One "task" is internally dozens of model calls, and context accumulates across all of them.

The sizing implication is direct: the bottleneck is token throughput, not user count, and token throughput scales with agent usage intensity. Which is why we enforce per-agent token budgets and session length limits at the platform level. Leave it uncapped and the organizations using it best are the ones whose GPUs run dry first. We logged the cloud version of this problem as a billing story in six patterns for controlling LLM cost; on-prem, the same invoice comes back disguised as a GPU queue.

Does this genuinely work in an air-gapped network?

It does. We designed air-gap as a first-class scenario, not an edge case. The model package ships inside the offline install bundle, license activation is handled offline with signature verification, and all inference runs on internal GPUs. We built it around the requirement that a security officer must be able to verify, personally, that outbound traffic is zero. Why air-gapped agents are actually hard (execution authority and audit, not the model) is a separate post.

The advice we actually give institutions weighing a GPU purchase is unglamorous: estimate concurrent sessions first, start with one 80GB card, and scale on queue metrics. You can always buy more GPUs later. What you can't easily survive is the audit finding for a rack of cards bought up front and idling at 10% utilization.