Quarterly SRE reviews reliably include MTTR on the agenda and rarely include concrete plans to change it. The discussion usually surfaces the same action items: improve runbooks, add more specific alerts, consider a dedicated incident commander role. These interventions are not wrong — they're just attacking the smallest slices of the MTTR pie. The teams that actually move median MTTR from 47 minutes to under 10 are addressing the largest slice: the time spent diagnosing root cause, which is typically 60-70% of total incident duration and responds almost entirely to changes in tooling, not process.
Decomposing MTTR: Where the Time Actually Goes
MTTR is a single number that averages across a process with four meaningfully different phases, each responding to different interventions:
- Time to detect (TTD): From incident start to alert firing. Typically 1–5 minutes for threshold-based alerts, potentially longer for anomalies that ramp slowly. Improved by tighter alert thresholds, shorter evaluation windows, anomaly detection.
- Time to engage (TTE): From alert to on-call engineer acknowledging and beginning investigation. Typically 1–5 minutes during business hours, potentially 3–10 minutes overnight. Improved by on-call process, escalation policies, wake-up reliability.
- Time to diagnose (TTDx): From engagement to root cause identification. The largest phase by far in most incident profiles — commonly 20–40 minutes for cascade failures. Improved by observability tooling: correlation visibility, pre-computed signal relationships, unified investigation interface.
- Time to fix (TTF): From root cause identification to remediation complete. Varies enormously by incident type. A rollback of a recent deploy might take 3 minutes. A database corruption might take hours. Improved by deployment automation, feature flags, runbook completeness for known failure modes.
A typical 47-minute P1 incident at a mid-size platform team breaks down roughly as: TTD 3min + TTE 4min + TTDx 32min + TTF 8min. The runbook investments most teams make improve TTF slightly, and alert tuning investments improve TTD slightly. Neither touches the 32-minute TTDx. That's the number worth measuring independently and optimizing directly.
The Golden Signals Are a Starting Point, Not a Destination
The four golden signals (latency, traffic, errors, saturation) from the Google SRE book are the right initial instrumentation checklist. They cover the baseline system health signals that indicate an incident is occurring. They're less useful as the primary investigation interface once you know an incident is occurring.
The problem: golden signals are service-scoped. Latency for payment-service is one number. Errors for auth-service is another number. Saturation for the database connection pool is a third. None of these individual numbers tells you the causal relationship between them. When payment-service latency is high, auth-service errors are elevated, and the database connection pool is saturated, are those three correlated phenomena from a shared root cause, or three independent issues that happened to occur in the same time window?
The shift in tooling that characterizes teams with faster TTDx is from golden-signals-as-investigation to golden-signals-as-triage. The golden signals tell you the blast radius (which services are affected and how severely). Cross-service correlation — the relationship between signal changes across services — tells you the root cause. These are different questions that require different tooling.
Pre-Computed Correlation vs. Query-Time Joins: The Investigation Speed Gap
There are two architectural approaches to cross-service signal correlation, and they produce dramatically different investigation speeds.
Query-time joining: you receive the alert, open your observability platform, manually query each affected service's metrics and logs, align time windows, copy trace IDs between tabs, and construct the causal picture yourself. The data for the correct answer has been in your system since the first seconds of the incident. Getting to that answer takes 25–40 minutes because each step requires human cognitive effort and tool navigation.
Ingest-time correlation: as signals arrive, the correlation engine links them by shared identifiers (trace IDs, service dependency topology, temporal proximity). By the time you open the incident dashboard, the causal relationships are already assembled. The investigation question changes from "what happened and why?" (requires constructing the picture) to "is this correlation finding correct?" (requires validating a picture that's already there). The cognitive load shifts from detective work to confirmation — fundamentally faster, and less prone to human error at 3am.
Consider a platform engineering team that ran an incident retrospective on a 52-minute cascade failure: payment-service latency spike propagated to auth-service, which propagated to api-gateway. All the data needed to identify payment-service as the root cause — the 3-minute latency precursor in payment-service's metrics before auth-service started erroring — was present in the logging system within 90 seconds of the incident starting. Surfacing it required manually querying payment-service's metrics independently, which happened at minute 38 of the investigation. With pre-computed correlation that identifies payment-service as the first-mover across the three affected services, that finding is available in under 2 minutes.
The Runbook Trap
We're not saying runbooks are harmful — they're valuable for encoding institutional knowledge and giving new on-call engineers a structured starting point. We're saying that runbook-following as a primary investigation strategy has a specific failure mode that causes MTTR to plateau even as on-call experience grows.
Runbooks are backwards-looking documents. They record how a previous incident of a similar type was diagnosed. Production incidents are forward-looking events — they have novel elements, novel combinations of failure modes, service topology changes since the last similar incident, code changes in the relevant services. Most P1 incidents have at least one element that doesn't match the closest runbook cleanly.
The failure mode: an experienced on-call engineer encounters an incident that partially matches a runbook but has an unusual element. Instead of pivoting to first-principles investigation when the runbook steps stop matching, they continue following the runbook — "checking the things we always check" — while the actual root cause sits in an unrelated log stream. This is the pattern that produces 47-minute MTTR at teams with excellent runbook discipline. The runbook following took 30 of those minutes.
Runbooks are most effective when combined with correlation visibility that quickly establishes which services are implicated. If the first 90 seconds of an incident provide a pre-computed causal graph showing "payment-service → auth-service → api-gateway," the on-call engineer immediately knows which runbooks to pull. The correlation narrows the hypothesis space; the runbook confirms or rules out the specific failure mode within that space. That's a more effective use of runbooks than using them as the primary search strategy.
What the Teams With the Lowest TTDx Actually Changed
Across incident post-mortems at early-stage and growing platform teams, the operational changes with the highest observed impact on TTDx share a common pattern: they reduce the number of manual transitions in the investigation workflow.
Every transition — open a new tab, switch tools, re-query with a different time window, copy a trace ID from one interface to paste into another — adds 2–5 minutes to investigation time in the best case and costs significantly more when the engineer loses context or makes an error during the transition. A 30-minute investigation with 8 tool transitions and a 10-minute investigation with 1 tool transition can be examining identical data. The difference is how much of the data assembly is done by the investigator versus the tooling.
The specific changes that correlate with TTDx reduction:
- Alert-to-investigation context: when the alert fires, the investigation interface opens pre-populated with the correlated signals from the affected services' time window — not a blank slate requiring manual queries
- First-mover identification: which service's anomaly started first, surfaced automatically rather than requiring manual cross-service comparison
- Trace-log linkage: clicking on an error log line navigates directly to the associated trace, eliminating the trace ID copy-paste step
- Cascade path visualization: the service-to-service propagation path visible as a single view rather than reconstructed through individual service queries
Measuring TTDx Separately From MTTR
The measurement change that enables improvement is tracking TTDx (time to diagnosis) as an independent metric rather than burying it inside MTTR. Most teams have MTTR from PagerDuty (acknowledge-to-resolve time), which conflates all four phases including stakeholder communication, bridge call management, and verification — none of which are investigation time.
A minimal TTDx measurement: add a post-mortem field for "time root cause was identified" alongside the incident start and end times. The difference between incident start and root cause identification is TTDx. Track this distribution over time. The median is more useful than the mean here — a few unusually long incidents from truly novel failure modes will skew the average without reflecting the typical investigation experience.
Once TTDx is tracked independently, the baseline typically reveals that it's substantially longer than most teams estimate (30+ minutes for cascade failures is common at teams that think of themselves as having good observability), and that it's more variable than the total MTTR. High variance in TTDx is a signal that investigation outcome is heavily dependent on which on-call engineer handles the incident and which specific failure mode it is — both indicators that tooling and process are not yet providing reliable structure to the diagnosis phase. Reducing that variance, not just the mean, is the metric that represents real structural improvement in incident response.