When Feature Flags Stop Being Toggles and Become a Control Plane

We started with feature flags as on/off toggles. Nine months later they had become *the most frequently changed surface in our infrastructure*. The transition point and what changes there.

John Baek
John Baek
Founder, CollabOps
When Feature Flags Stop Being Toggles and Become a Control Plane

We started with 12 boolean toggles on a SaaS like configcat. Nine months later we had 407 flag instances and 14 changes per hour on average. Feature flags had become the most-frequently-changed surface in our production infrastructure.

If this transition happens unintentionally, you get uncontrollable chaos. Intentionally, you get the most powerful tool in the org.

From toggle to control plane — four stages

Stage 1 — feature-launch toggles (1–50 flags)

The starting point. Purpose: gradual activation after deploy. Usually bool flags. Short lifetime — removed 1–3 sprints after stabilization.

Trap at this stage — not removing them. Once a feature stabilizes, the flag should be cleaned up. It rarely is. Past 50 flags, more than half are stale.

Stage 2 — risk kill-switches (50–150 flags)

Next stage. Flags evolve into a tool to immediately disable problematic behavior. A manual circuit breaker. Permanent lifetime (kept for incident response).

Trap — which flag to flip during which incident is undocumented. At 03:11, knowing which flag to kill depends on that one person's memory.

Stage 3 — segment-targeted rollout (150–300 flags)

Next. Flags express targeting rules, not booleans. Enterprise payment users only, Korean users only, beta program members only.

Trap — targeting expressions diverge. The same segment is defined differently across flags. One user is "beta" in flag A but not in flag B.

Stage 4 — control plane (300+ flags)

The transition. Flags no longer represent features but system behavior itself. Traffic ratios, cache TTLs, retry counts, timeouts, external vendor selection, model versions — every runtime decision expressed as a flag.

At this point, the meaning of "deploy" changes. Code deploys are foundation installs. Actual product behavior is decided by flag changes.

What stage 4 means — deploy itself splits in two

Up through stage 3, deploy = feature launch. At stage 4, deploy = code change, flag = behavior change. The two separate.

Implications:

  • Code review and flag review separate. Code reviewed by engineering, flags reviewed by product, SRE, and security jointly.
  • Code deploy frequency drops, flag change frequency rises. Our data: 5–7 deploys/day → 14 flag changes/day.
  • First incident response action becomes a flag change — faster than code revert.
  • A/B tests aren't a separate system. Every flag is a potential A/B.

The expensive parts of getting to stage 4

1. Flag typing

300+ flags carry more than booleans. Numbers, strings, JSON objects, target rules. Types must match the code — string-expecting code with a number flag breaks at runtime.

Fix: declare a schema per flag at definition time, validate against code.

2. Flag audit

Change history of 300+ flags must be auditable. Who changed which flag when, and what was the system state at that moment.

Fix: flag changes go into the same audit trail as deploys. Author, timestamp, before/after value.

3. Flag dependency graph

Flags depend on each other. Flag B is meaningful only when flag A is on. Manual management of these dependencies collapses within three months.

Fix: declare prerequisites in flag definitions; the system blocks impossible combinations.

4. Flag cleanup automation

Flags must not live forever. At creation time, declare expiry or cleanup trigger. Automate periodic cleanup notifications.

Fix: quarterly auto-report of flags unchanged for 90+ days. Half are usually removable.

Our data — incident response time

After moving from stage 3 to stage 4:

Stage 3 (150 flags):
  incident → code revert → deploy → effect
  average: 27 min

Stage 4 (300+ flags):
  incident → flag change → immediate effect
  average: 4 min

23 minutes saved. That's the actual value of stage 4. Infrastructure cost is higher, operational complexity is higher, but incident response is 7× faster.

Is stage 4 right for everyone?

No. Stage 4 needs a team that owns flag operations. Sub-200-person orgs almost always cap at stage 3.

The threshold is 100 flags

The point where a platform or SRE lead should start preparing for stage 4 is when feature flags pass 100 and control starts slipping. The four costs above (typing, audit, dependency graph, cleanup) must be paid now — paying them later means a full redesign in six months.

Tags#feature-flags#deployment#devops#product#infrastructure