The page that wakes you at 2am is rarely about the failure itself. By the time PagerDuty fires, the cascade is already halfway done. What SRE teams consistently underestimate is how much warning a system gives before a cascade completes. When we analyzed 200 incident timelines we had collected through early Devtract pilots, the pattern was striking: in 73% of multi-service outages, at least one anomalous signal appeared 8 to 15 minutes before the first user-facing alert. In 40% of cases, two or more distinct pre-signals appeared across different telemetry sources.
The problem isn't that systems don't warn you. The problem is that nobody is watching the right combination of signals at the same time.
What a Cascade Looks Like in Telemetry
A cascade failure in a microservices architecture follows a recognizable physical pattern. Some upstream dependency gets slow or starts erroring. Services that depend on it start queuing requests or failing fast. Thread pools fill up. Connection pools saturate. Memory pressure builds. Eventually a downstream service that has nothing obviously wrong starts throwing 503s, and that's what your alerting threshold catches. The alert fires on the leaf, but the root is several nodes upstream.
In telemetry, this sequence leaves a temporal fingerprint. Consider a payment processing service backed by a shared Postgres instance. If Postgres starts accumulating lock waits due to a poorly-timed migration, you'll see these in rough order:
- Postgres
pg_stat_activitywait events start climbing (metric, T-14 min) - Database query latency p99 drifts from 40ms to 180ms (metric, T-12 min)
- Upstream service logs start showing
connection pool exhaustedwarnings (log, T-10 min) - Dependent services begin reporting elevated HTTP 500 rates (metric, T-7 min)
- First PagerDuty alert fires on error rate threshold (T-0)
Each of those early signals, viewed alone, looks like noise. Query latency drifted? Maybe a normal load blip. Pool warnings? Someone probably deployed. Individually they don't clear the threshold for alerting. Together, with the right temporal correlation, they constitute a pre-cascade signature.
Why Threshold-Based Alerting Misses Pre-Signals
Conventional alerting is designed to catch states, not trajectories. You define a threshold: error rate > 5% for 3 minutes fires an alert. This works fine for catching failures after they've fully materialized. It does nothing to tell you a failure is forming.
Trajectory-based detection requires comparing the current shape of a metric over time against historical baselines. A query latency that goes from 40ms to 80ms in four minutes is statistically different from one that has hovered between 35ms and 85ms for the past two weeks. The absolute value of 80ms might not breach any threshold. The slope of the change might.
This is where most observability tooling stops short. Prometheus can tell you a value is high. It cannot, out of the box, tell you that the value is rising in a pattern that historically precedes cascades. You need cross-signal correlation and historical pattern context to make that call.
The Pre-Signal Categories Worth Watching
After looking at enough incident timelines, we've found the most reliable pre-cascade indicators fall into a few categories. These aren't universal laws, but they appear frequently enough to be worth instrumenting explicitly.
Saturation Signals in Shared Resources
Connection pools, thread pools, message queue depths, semaphore counts. These fill linearly when load increases and empty linearly when load drops. What distinguishes a pre-cascade state is when they fill and stay filled during what should be a normal load period. A connection pool at 85% capacity at 2am on a Tuesday is a different signal than the same reading during peak traffic.
The key metric isn't the absolute saturation percentage. It's saturation relative to the expected load at that time. That's a comparison your static threshold won't make.
Latency Distribution Shifts, Not Mean Latency
Mean latency is a terrible early-warning signal because outliers get averaged away. p99 latency jumping from 200ms to 800ms while p50 stays flat is a cascade precursor. It means tail requests are piling up somewhere, probably waiting on something that's saturated. The mean might move 10ms and never breach any alert. The p99 tells the real story.
We've seen this pattern in a payment flow we analyzed: p50 checkout latency stayed at 180ms throughout an incident buildup period, while p99 crept from 400ms to 2200ms over 11 minutes before the alert fired. The mean moved less than 20ms.
Log Pattern Velocity, Not Just Log Error Rate
The total error log count isn't always the signal. The rate at which a new error pattern is appearing is. When a specific log line goes from 0 occurrences per minute to 15 occurrences per minute over a five-minute window, something changed. If that log pattern is a known precursor to a known failure mode, you have a pre-signal. But this requires you to have built a library of log-pattern-to-incident mappings from your post-incident reviews.
The Correlation Window Problem
Identifying pre-signals in isolation is useful. Correlating them across services in real time is what makes pre-cascade detection actionable. The challenge is defining the right correlation window: how far back do you look, and which services do you look at together?
Too narrow a window (30 seconds) and you miss the buildup. Too wide (60 minutes) and you're correlating events that have no causal relationship, generating false positives that train on-call engineers to ignore warnings. The right window depends on the specific failure topology of your service graph.
For a tightly coupled synchronous service chain, a 5 to 10 minute correlation window tends to capture most pre-cascade signals. For asynchronous systems with queue-mediated dependencies, you may need 20 to 30 minutes because queue buildup is slower to manifest in downstream latency. Deploy-correlated failures need a different window entirely, which is why deploy correlation deserves its own tuning.
In Devtract, we model the correlation window per dependency edge in the service graph rather than globally. A service calling Postgres gets a different window calculation than a service consuming from Kafka, because the latency propagation characteristics are different. This is one of the harder engineering problems we've worked through, and we're still refining it.
Building a Pre-Cascade Detection Practice
We want to be honest about something: automated pre-cascade detection is not a solved problem, even with good tooling. What we can say is that the teams we've worked with who have the most success at catching cascades early share a few practices.
First, they run postmortem exercises specifically looking for the pre-signals that were visible but not acted on. After every significant incident, they trace back through their telemetry and mark the earliest anomaly. Over time this builds a per-system catalog of what "about to cascade" looks like in your specific environment.
Second, they instrument saturation metrics explicitly. Error rates are commonly instrumented. Queue depths, connection pool fill percentages, and thread pool utilization are frequently not, because they require writing custom exporters or querying internal application state. The teams that catch cascades early do the work of instrumenting these.
Third, they treat log patterns as structured telemetry rather than as raw text for grep. Each known failure mode has an associated log signature, and those signatures are tracked as metrics: occurrences per minute, rate of change, rolling count. When that metric crosses a threshold, it's treated the same way a Prometheus counter would be.
What Pre-Signal Detection Cannot Do
We're not saying pre-cascade detection eliminates incidents. Novel failure modes have no history to compare against and will always surprise you. Systems under unusual load profiles behave differently than their baseline suggests they should. Pre-signal libraries built from past incidents won't catch the failure you haven't had yet.
What pre-cascade detection gives you is lead time on the failures you've already partially understood. For teams running high-traffic, well-instrumented services, that's a meaningful subset of total incidents. Practically, getting 10 minutes of warning before an alert fires means the on-call can start their investigation before the system is already degraded, which changes the diagnosis quality significantly.
The goal isn't perfect prediction. The goal is shrinking the diagnostic gap from the moment something starts going wrong to the moment a human understands what's happening. Every minute you recover from that gap is a minute you didn't spend waking up your team, escalating to a second on-call, or writing an apology to affected users.
That gap is exactly what we built Devtract to address, and pre-cascade signal correlation is a core part of how we do it.