Why Three Pillars Aren't Enough Anymore — Notes From a 23-Minute Incident

03:11. The pager woke me up. The alarm body — checkout success rate −2.3% over the last 5 minutes. Our SLO threshold is −1%.

First place I looked: the Grafana dashboard with the core metrics. All green. CPU, memory, QPS, error rate, p95 latency — all in normal ranges. Next: Loki logs. Nothing abnormal. Next: Tempo traces. Average latency had actually gone down (strange, but not broken).

All three pillars said normal. Meanwhile, 23 users had been unable to pay for 23 minutes.

This is the exact shape, in 2025, of observability not being enough as three pillars.

What had broken

Twenty-three minutes later (the root cause finding, not the alarm), the truth came out. A new payment validation step we deployed three hours earlier was silently rejecting a specific combination — particular card issuer + transaction amount band + time-of-day — without calling the external gateway.

From the server's perspective: normal response (HTTP 200, valid response within our system). From metrics: not an error. From logs: expected branch. From traces: short, clean trace.

All three pillars were correct. The correctness just had nothing to do with what the user was experiencing.

Why three pillars came up short

What three pillars answer:
  - "What's happening in the system?"     (logs)
  - "How healthy is the system?"          (metrics)
  - "What path did this request take?"    (traces)

What three pillars cannot answer:
  - "What did the *user* experience?"
  - "Which *change* caused this?"
  - "Is this incident *similar* to one we've seen before?"
  - "Did anything go *quietly broken* after this change?"

The 03:11 incident hinged on the fourth question — a deploy three hours earlier had quietly broken one code path, and it was only visible from the user side. There's no place to find that answer inside the three-pillar model.

The four signals we added

We layered four more signals on top of the three pillars. Not replacements — just more nodes on the same graph.

1. Events — something meaningful happened

Single discrete events that change system state: deploys, feature flag toggles, DB migrations, credential rotations. If metrics are a timeline, events are markers on it. We push every deploy, flag toggle, and infra change to a single event stream.

If the deploy three hours earlier had been visible on the same screen as the metrics during the 03:11 page, 22 of those minutes disappear.

2. User journeys — one user's flow as one line

A view that ties a specific user ID's session start → checkout attempt → checkout outcome across all systems they touched, with business intent labeled. One step above raw traces.

Putting the 23 failed users' journeys on one screen made the common pattern immediately obvious — issuer X, time band Y, amount Z. Not visible from traces alone.

3. Deploy correlation — which change caused this

Auto-attribute each deploy to the code paths it touched, user segments it affected, and metric changes it correlates with. Heavy infrastructure to build, but with this in place, deploy three hours ago would have automatically connected to checkout failure pattern now. About 70% of incidents resolve in five minutes once that link is automatic.

4. Similarity matching — have we seen this before

Past postmortems indexed in a vector store, matched against the live signal pattern of a new incident. The pager alarm shows "a similar pattern occurred 6 months ago" alongside the firing alert, and you can review that prior root cause immediately.

This was the last signal we added and the most impactful. Incidents often don't get resolved until the third occurrence. Similarity matching makes a first occurrence feel like a second.

Six months later — incident timing

Three pillars only:
  - Mean Time to Detect (TTD):       4.2 min
  - Mean Time to Investigate (TTI):  37 min
  - Mean Time to Recover (TTR):      12 min
  - Total MTTR:                      53 min

Three pillars + four extra signals:
  - TTD:                              3.8 min
  - TTI:                              9 min   ←── 4× shorter
  - TTR:                             11 min
  - Total MTTR:                      23 min

TTD (detection) and TTR (recovery) barely changed. The whole change collapses into TTI (investigation). Three pillars do detection and recovery well; they fail at investigation. That's the one-line summary of the shift.

We didn't abandon three pillars

Logs, metrics, and traces are the foundation everything else sits on. None of the four additions exists without them. What we did was layer four more signals on top — and shift the team's first screen from a metrics dashboard to events + user journeys.

Why does incident investigation take so long?

Teams with logs, metrics, and traces fully built still spend 30–60 minutes on investigation (TTI) because the three pillars are detection and recovery signals, not investigation signals. If detection is fast but TTI won't move, at least one of events, user journeys, deploy correlation, or similarity matching is missing. Adding all four took our TTI from 37 minutes to 9, and reclaims hours of incident-response time per quarter.