Seven Incidents Canary Deploys Missed — Real Numbers

Canary deploys work for most changes. We run 25–30 canaries a week, and 97–98% go exactly as planned. The remaining 2–3% failures all look similar. Seven shapes.

This post walks the seven, with real cases.

1. Same traffic percentage, different user distribution

Sticky-session routing keys 10% of users into canary based on user ID — but the ID is correlated with time of day, region, or payment method. The 10% looks like 10% of users, but is actually 10% of one segment.

Our case: a new payment validation step. 10% canary said normal. 100% rollout broke for enterprise payment users only. The canary's 10% was individual users only. Three-hour impact.

Defense: validate the segment distribution of the traffic split, not just the percentage match.

2. Validated reads only, missed writes

Canary ran during a 95% read window. New code's read path was validated. Write path's deadlock surfaced 9 minutes after full rollout, at peak.

Defense: schedule canary at write peak deliberately, or only during windows where read/write ratio matches baseline.

3. Connection pool slowly leaking

Canary's pool usage looked normal. Within the 30-minute validation window, it was trending up but didn't cross the threshold. Six hours after full rollout, the whole fleet hit pool exhaustion → cascading failure.

Defense: treat trends as signals. Current values alone miss leak-shaped bugs.

4. External API calls faster than baseline

Canary's external API calls were faster than baseline. Initially read as improvement. Actual cause: retry logic was removed — when the external API hiccuped, instead of retrying, the new code fell back to empty response. The canary window had a healthy external API, so the fallback was never exercised. Three minutes after full rollout the API wobbled — every user got empty responses for those three minutes.

Defense: include external API failure injection in canary. Validating only with healthy dependencies misses fallback bugs.

5. Time-shifted background jobs

Canary ran during the day. The changed code only affected a nightly batch job. Direct traffic validation didn't see it. The batch broke more than 24 hours later.

Defense: map the changed code to its trigger. If the trigger is cron / queue / event, canary alone is insufficient.

6. Upstream contract assumption changed

API response added an optional field. Canary validated that existing clients didn't break. Correct at that point. After full rollout, a downstream system using our API failed enum validation on the unexpected field.

Defense: API changes need canary to include downstream integration tests, not just our traffic. Without a downstream inventory, canary value is small.

7. The canary's own monitoring was lagging

The validation dashboard was cached for 5 minutes. We treated it as real-time and rolled to 100% 3 minutes after canary looked good. The data we read was 5 minutes old. The actual data was already showing trouble signals we couldn't see.

Defense: explicitly know the latency of canary data sources. "Real-time" dashboards are often 5-minute-delayed.

What the seven share

All seven involve canary worked — within the validation window given. Failures occurred outside the window — time distribution, user distribution, external dependencies, downstream integrations, data latency.

Canary doesn't validate change safety. It validates delta from baseline. If the baseline is incomplete, the canary is incomplete.

If you already run canary infrastructure and still hit incidents after canary passes, two or three of these seven almost certainly apply to your team. Defend against them one at a time and, in our measurements, you eliminate 1–2 incidents per quarter.