Engineering Notes from the On-Call Queue
Writing about incident response, root-cause analysis, SRE tooling, and distributed systems reliability. From the team building Devtract in Austin, TX.
Most cascade failures leave signals 8 to 15 minutes before the alert fires. Here is how to read them.
Why MTTR Is the Wrong Metric for Modern SRE Teams
Mean time to recovery hides what actually matters: the diagnostic gap between alert and understanding.
Correlating Logs, Metrics, and Deploys for Faster RCA
A deploy that touches 3 services, a metric spike in 2 of them, and a log pattern in 1. How do you find the common thread?
On-Call Burnout Has a Root Cause Too
Engineers do not burn out from being paged. They burn out from being paged and not knowing what is wrong.
Distributed Tracing Without the Sampling Tax
Head-based sampling discards the traces you most need during incidents. Here is a practical alternative.
Anatomy of a 5am Page: What Good Incident Context Looks Like
We analyzed 200 incident timelines to understand what context engineers actually need in the first 90 seconds.
Design Principles for an SRE Copilot That Doesn't Get in the Way
A copilot that asks clarifying questions during an incident is worse than no copilot. Here is how we thought about response design.
Alert Fatigue Is a Signal Problem, Not a Volume Problem
Reducing alert volume is the wrong fix. Teams need better signal-to-noise ratio, which requires understanding what each alert actually represents.
Kubernetes Event Correlation Patterns for SREs
Pod evictions, OOMKilled events, and node pressure do not happen in isolation. Here is how to read them as a cluster.
What Automated Postmortems Get Right (and What They Miss)
Automated postmortem drafts save time but they also flatten the nuance that makes postmortems actually useful.
Tuning Your Deploy Correlation Window: How Far Back Is Too Far?
A deploy from 6 hours ago can still be the root cause. A deploy from 48 hours ago almost certainly is not. Here is how to tune the window.
Why We Built Devtract
We kept getting paged at 3am for problems that took an hour to diagnose and five minutes to fix. That gap is why Devtract exists.