Running Llama / Qwen Air-Gapped — Nine Months of Field Notes

Since June we've been running Llama 3.2 and Qwen 2.5 self-hosted at one air-gapped customer. Adoption guides cover download model + vLLM + GPU. The next nine months expose the four things guides don't.

1 — GPU allocation differs by model

We initially assumed one A100 per model. Broke at week six.

Model                    | Recommended GPU                  | Throughput (req/s)
─────────────────────────┼──────────────────────────────────┼───────────────────
Llama 3.2 8B            | A100 40GB × 1                     | 4–6
Llama 3.3 70B           | A100 80GB × 2 (tensor parallel)   | 0.8–1.2
Qwen 2.5 7B (Korean)    | A100 40GB × 1                     | 5–7
Qwen 2.5 32B            | A100 80GB × 1                     | 2–3
Mixtral 8x7B             | A100 80GB × 2                     | 2–4

Trap — bigger isn't always better. 70B is 5–6× slower than 8B with only 15% accuracy improvement. By task type, the small model is often the right one.

2 — throughput vs latency trade-off

After GPU allocation comes batch size. Larger batch = higher throughput + higher latency.

batch=1:    latency 200ms,  throughput 1.5 req/s
batch=8:    latency 800ms,  throughput 6 req/s
batch=16:   latency 1.2s,   throughput 8 req/s
batch=32:   latency 2.5s,   throughput 9 req/s  (diminishing returns)

Conversational interfaces → small batch (latency wins). Batch analysis → large batch (throughput wins). If you need both, run separate endpoints.

3 — Fine-tuning's actual cost

"We'll fine-tune for our domain" was the hypothesis. Six months later — for most cases, fine-tuning cost > value.

Why:

• Data curation is the most expensive — thousands of labeled examples needed
• Re-training cycle — every new model release means refit
• RAG is cheaper and fresher — if domain knowledge is documents, RAG wins

Fine-tuning is the answer when style / format unification is required (e.g., always speak in formal Korean honorifics). Otherwise, RAG.

→ RAG vs Fine-tune decision guide

4 — Model refresh is a quarterly project

When a new model ships (Llama 3.4, Qwen 3), you don't just swap. Refresh is a small project.

Model refresh checklist:

1. Download + verify (checksum, license)
2. Confirm compatible vLLM / TGI version
3. Re-measure GPU memory profile
4. Validate compatibility with our prompt templates
5. Compare accuracy on *our domain eval set* (old vs new)
6. Decide whether to re-fine-tune (if previously fine-tuned)
7. Canary deploy (10% traffic → compare)
8. Full cutover or rollback

Steps 7–8 follow the agent rollback strategy pattern.

Nine-month cost ledger

Initial setup (6 weeks):          $48,000  (labor)
GPU lease (monthly avg):           $3,200
Operations (monthly avg):          $4,500  (labor)
Fine-tune attempt + abandon:      $11,000
Model refresh (quarterly):         $6,000
─────────────────────────────────
9-month total:                    ~$132,000

Same workload via cloud LLM API — estimated $280,000. Self-hosted saves 55%. Compliance value is separate — for air-gapped customers, without this, the PoC doesn't even start.

Put these on the schedule from day one

Self-hosted LLM deployment (Llama, Qwen) gets you from model download to vLLM boot quickly, but the four areas above each take 6+ weeks. If you're evaluating or in the early stages, putting that time on the schedule up front is the answer. Our nine-month total of ~$132,000 came out of exactly those four areas.