A Year of Actually Using an Error Budget — Q1 Through Q4

Last January we put a 99.9% monthly availability SLO on payments. That's 43 minutes of allowed monthly downtime — our error budget. How we used it defined the year.

This post is the four-quarter ledger. Each quarter taught a different lesson.

Q1 — Budget was too plentiful

Combined downtime in January, February, and March: 11 minutes. We used 8.5% of the quarter's budget.

We were initially pleased. Then we noticed — unspent budget meant we were operating too conservatively. Feature deploys were slow. Risky changes were being postponed.

Half of an error budget is the right to use volatility to ship faster. Not using it isn't saving. It's opportunity cost.

Q1 retro decision: in Q2, deliberately schedule changes that consume budget. That decision triggered the next quarter.

Q2 — Budget gone in three days

Mid-April we started gradual migration to a new payment gateway. 3% of traffic → 10% → 30% → 70% → 100%. One-week validation window per step.

Going from 10% to 30% on a Tuesday in late April, the new gateway's connection pool misconfiguration surfaced only in production. 47 minutes of downtime. The next week, a secondary incident in the same shape — 17 minutes.

Quarter budget: 130 minutes. We had used 64 minutes in three days. We immediately invoked feature freeze — no new feature deploys for the rest of the quarter.

A fast burn isn't bad luck. It's a signal that the risk profile of our changes doesn't match the SLO's assumption.

The real lesson of Q2 wasn't the 47-minute incident. It was the 17-minute follow-on. Big incidents are often followed by aftermath incidents — operator fatigue, side-effects of urgent fixes, fragmented attention. We named this aftermath effect and adopted a policy: seven days of deliberately reduced change frequency after a major incident.

Q3 — The unexpected discovery of being frozen

July, August, September. While feature deploys were paused, the team:

• Reduced flaky tests from 87 to 14
• Cut alarm noise from 41 per week to 9
• Two SREs finished a three-year-deferred alerting infrastructure rewrite

Quarterly downtime: 4 minutes. 3% of the 130-minute budget.

This was the most reliable quarter of the year. The discovery beyond the tautology of no changes means stable — the freeze created a deliberate window to pay technical debt.

An error budget freeze need not be punishment. It can be a deliberate tool to schedule technical debt quarters.

Q4 — We changed the SLO itself

Going into October we lined up the four quarters:

Quarter     Budget burn
Q1            8.5%
Q2          ~95%   (70% from one incident)
Q3            3%
Q4 (planned) ?

What this distribution told us — 99.9% as a single threshold doesn't match our system's natural reliability. The mean is fine, but the distribution is extreme. One quarter near-100%, the next at 3%.

Late November SRE / engineering / product joint review concluded — replace the single threshold with multi-SLI composition.

Before:
  Payment availability 99.9%

After:
  Payment availability 99.95%        (raise availability itself)
  Payment p99 latency < 800ms        (previously unspecified)
  Payment incident detection in 5s   (detection SLI)
  Payment recovery within 30 min     (recovery SLI)

With four SLIs, an incident burns separate budgets. One outage no longer drains one big budget — it drains partial budget of multiple SLIs. That smoothed the extreme distribution of Q1 vs Q2.

The four quarters in one line each

• Q1 — Not spending the budget is a cost
• Q2 — Burning it fast is a signal — and aftermath follows
• Q3 — Freeze can be a debt-payment quarter
• Q4 — If the distribution is extreme, the SLO itself is wrong

The book's error budget is a one-line guideline. In practice, it carries different meaning each quarter.

So how do you actually operate an error budget?

Operating an error budget, once the SLO is in place, starts with identifying which of the four patterns your current quarter is in. Knowing that makes the next move obvious — for SREs and platform leads unsure how to run their SLO, that identification is the whole job. If you're new to SLOs, this is too early. Start with defining SLIs first.