After Three Pillars — Six-Month Field Notes on the Four Added Signals
Six months after Why Three Pillars Aren't Enough. How the four added signals (events / user journeys / deploy correlation / similarity) actually performed in production.
Six months after Why Three Pillars Aren't Enough. The operational evaluation of the four added signals.
Six-month data
Signal | MTTR contribution (before → after) | Op burden
─────────────────────────┼────────────────────────────────────┼────────────
1. Events | 53 → 31 min (−22) | Low
2. User journeys | 31 → 24 min (−7) | Medium
3. Deploy correlation | 24 → 17 min (−7) | Mid-heavy
4. Similarity matching | 17 → 23 min (+6) *regression* | HighSimilarity matching turned negative. Reason follows.
Real value vs operational burden per signal
Events — highest ROI
Event stream took two weeks to ship. MTTR −22 minutes. Highest ROI. Every org should pin this first.
User journeys — good but expensive
Journey tracking carries high data-collection + storage cost. MTTR −7 minutes. Still net positive ROI but one-third of Events.
Deploy correlation — surprisingly complex
Auto-mapping each deploy to affected metrics / segments turned out harder than expected. Required manual labeling to compensate. MTTR −7 minutes.
Similarity matching — regression
Biggest finding. Vector matching past incidents produced too many false positives. Each new alarm came with 5 similar past cases, of which only 1 was actually relevant. Reviewing and rejecting the other 4 added time.
We tried — raise vector similarity threshold 0.7 → 0.95. False positives dropped. But recall fell 50% — we missed genuinely similar cases.
Conclusion — similarity matching is net negative as currently configured. We rolled back to explicit search (human-triggered).
Six-month cumulative effect
Starting MTTR: 53 min (three pillars only)
Ending MTTR: 23 min (three pillars + three signals, similarity removed)
Net improvement: −30 min (−57%)Originally projected 23 minutes with all four. Final 23 minutes with similarity removed. The discovery that similarity is net negative was the biggest lesson of the six months.
Plan for the next six months
Hypothesis for net positive similarity matching:
- Activate vector search only on manual trigger
- Two-stage search — first pass for matches, then human picks which look similar
- Don't auto-show past cases — show them when humans search
Plan: measure in Q3.
For SREs and platform leads layering a fourth signal on top of the three pillars, our six-month MTTR data (53 → 23 minutes) is the starting point. The numbers are workload-specific, but the similarity-matching false positive problem is likely a common pitfall. Before adoption, start with explicit trigger mode.
Related posts
When a Product Manager Ships Code, Who Owns the Outage?
In organizations where product managers write code with AI and several agents work at once, who checks what before a change reaches production? A role design for verification, approval, and recovery that is independent of the author, grounded in NIST SSDF, SLSA provenance, and the Google SRE postmortem culture.
John Baek
An Eclipse Plugin for the Agent Era — Task Context Was Already the Problem 20 Years Ago
The problem Mylyn set out to solve in the mid-2000s is the agent context problem. What changed is that the thing reading that context is no longer only a person.
Yeongsang Kim
A VS Code Extension for the Agent Era — What Developers Look At Now
Once writing code got cheap, a developer's time moved to judging and approving. Here is why those jobs cannot live outside the editor, and the choices behind the CollabOps VS Code extension.
Seungbaek Lee