Engineering Notes from the On-Call Queue

Writing about incident response, root-cause analysis, SRE tooling, and distributed systems reliability. From the team building Devtract in Austin, TX.

Abstract time-series visualization representing metrics and measurements in incident response
SRE Practices

Why MTTR Is the Wrong Metric for Modern SRE Teams

Mean time to recovery hides what actually matters: the diagnostic gap between alert and understanding.

Abstract visualization of three data streams converging to reveal a root cause
Root Cause Analysis

Correlating Logs, Metrics, and Deploys for Faster RCA

A deploy that touches 3 services, a metric spike in 2 of them, and a log pattern in 1. How do you find the common thread?

Abstract visualization of signal overload and stress in operational environments
SRE Practices

On-Call Burnout Has a Root Cause Too

Engineers do not burn out from being paged. They burn out from being paged and not knowing what is wrong.

Abstract visualization of distributed request traces flowing through a microservices system
Observability

Distributed Tracing Without the Sampling Tax

Head-based sampling discards the traces you most need during incidents. Here is a practical alternative.

Abstract visualization of an alert notification in the early morning darkness
Incident Response

Anatomy of a 5am Page: What Good Incident Context Looks Like

We analyzed 200 incident timelines to understand what context engineers actually need in the first 90 seconds.

Abstract visualization of human-AI collaboration patterns in operational tooling
Product

Design Principles for an SRE Copilot That Doesn't Get in the Way

A copilot that asks clarifying questions during an incident is worse than no copilot. Here is how we thought about response design.

Abstract visualization of signal versus noise in operational telemetry
Observability

Alert Fatigue Is a Signal Problem, Not a Volume Problem

Reducing alert volume is the wrong fix. Teams need better signal-to-noise ratio, which requires understanding what each alert actually represents.

Abstract visualization of container orchestration and event relationships in Kubernetes
Root Cause Analysis

Kubernetes Event Correlation Patterns for SREs

Pod evictions, OOMKilled events, and node pressure do not happen in isolation. Here is how to read them as a cluster.

Abstract visualization of incident timeline documentation and analysis
SRE Practices

What Automated Postmortems Get Right (and What They Miss)

Automated postmortem drafts save time but they also flatten the nuance that makes postmortems actually useful.

Abstract visualization of time windows and correlation boundaries in deployment analysis
Root Cause Analysis

Tuning Your Deploy Correlation Window: How Far Back Is Too Far?

A deploy from 6 hours ago can still be the root cause. A deploy from 48 hours ago almost certainly is not. Here is how to tune the window.

Abstract visualization representing the origin and founding motivation of Devtract
Company

Why We Built Devtract

We kept getting paged at 3am for problems that took an hour to diagnose and five minutes to fix. That gap is why Devtract exists.