Every few months, a team decides to fix alert fatigue by turning down the volume. They silence the alerts that haven't led to anything actionable in the past 60 days. They raise thresholds to reduce firing frequency. They consolidate five redundant alerts into one. Six weeks later, the on-call rotation is quieter and everyone feels better about the fix.
Then a real incident happens. The relevant alert was in the group that got silenced.
Reducing alert volume is a legitimate tool for managing noise, but it misdiagnoses the root cause of alert fatigue. Teams don't burn out because they receive too many notifications. They burn out because too many of those notifications don't tell them anything useful, and the ones that do are indistinguishable from the ones that don't until after the fact.
The problem is signal quality, not signal quantity. These are different problems with different solutions.
What alert fatigue actually costs
Alert fatigue has a direct safety cost that gets underreported because it's counterfactual: the thing that didn't get investigated because the on-call engineer was desensitized to the pattern. It's hard to write a postmortem entry for an alert you ignored that turned out to matter, because by the time you're writing the postmortem you know it mattered and the instinct is to say you should have acted on it. That framing hides the systemic problem, which is that the engineer had no way to know that alert was different from the twenty before it.
Consider a pattern we've seen in growing platform teams: a service with aggressive CPU alerting. The alert fires whenever CPU crosses 80% for more than 60 seconds. This happens routinely during batch processing windows, mid-day traffic spikes, and occasionally for no discernible reason. The on-call learns to check the alert context, see that it's probably nothing, and acknowledge without investigating. Then the day comes when the CPU spike is caused by a runaway query, and the service starts dropping requests. The alert fired the same way it always fires. The engineer responded the same way they always respond. The SLO breached.
This is not a volume problem. Removing the alert would make the SLO breach harder to detect. Raising the threshold might help, but it might just delay the alert on the day it matters. The actual problem is that the alert doesn't differentiate between "CPU spike that resolves itself" and "CPU spike with downstream impact."
Signal-to-noise ratio as an engineering metric
Signal-to-noise ratio in an alert system has a clear definition: the fraction of alert firings that result in an investigation that finds something real and actionable. If your alert fires 100 times in a month and 12 of those lead to any intervention at all, your signal rate is 12%. That's not a great number. Anything below about 30% is a sign that the alert's definition does not correlate well with actual incidents.
The useful thing about thinking in SNR terms is that it gives you a diagnostic lens for individual alerts, not just the aggregate. A single alert with a 5% signal rate is a different problem from a system where most alerts are at 25-40% but volume is high. The first case is a bad alert definition. The second might be volume that's genuinely proportional to your incident rate.
Tracking per-alert SNR is operationally harder than it sounds. It requires correlating alert firings with postmortem entries or incident records, which most teams don't do systematically. The alert fires, gets acknowledged, gets auto-resolved by the monitoring system when the metric drops back below threshold, and nobody records what happened or whether it mattered. The data exists somewhere (PagerDuty logs, Slack history, your incident tracker), but it's rarely aggregated into a per-alert SNR view.
This is one of the things Devtract tracks automatically: which alerts are leading to active incident investigations versus which ones are getting acknowledged and auto-resolved without any logged action. Over 30 to 60 days, the patterns become clear.
Why threshold-based alerts have a structural SNR ceiling
Most alerts are threshold-based: if metric X exceeds value Y for duration Z, fire. This is simple to implement and easy to reason about, which is why it dominates. The problem is that threshold-based alerts have a structural ceiling on their achievable SNR, because they can only express one kind of signal: "this metric is high."
What they can't express: "this metric is high AND it's correlated with an error rate increase AND there's been a deploy in the last 45 minutes." That combination is almost certainly a real incident. "This metric is high AND it's within the normal batch processing window AND no other metrics are elevated" is almost certainly noise. A threshold alert can't distinguish between these two states. The engineer has to do it manually on every single firing.
Multi-condition alerting helps, but it introduces its own maintenance burden: every combination of conditions you want to express requires its own alert definition. Teams end up with 200 alert rules, which is its own form of overhead. The alert definition becomes infrastructure that needs to be maintained.
The alternative framing: instead of building better alert rules, build a correlation layer that runs after alerts fire. The alert itself can stay simple (CPU high, error rate elevated, latency spike). The correlation layer evaluates the alert in context: what else is happening right now, what changed recently, what does the combination imply? This moves the signal work from alert definition time to alert evaluation time, which is where you have the most relevant context.
The suppression trap
When teams do tackle SNR by silencing low-quality alerts, they often fall into the suppression trap: they remove the alert, things are quiet, they feel better. What they've actually done is accept a lower detection surface. Some real incidents will now be harder to detect early. The team has traded a noisy alert that sometimes catches real problems for silence that never catches them.
We're not saying alert suppression is wrong. Alerts that never lead to anything, fire at predictable times, and have zero correlation with service health should be removed. But the bar for removal should be "this alert has zero signal value in any context," not "this alert fires a lot and we're tired of it." Those are different criteria.
A better heuristic before removing an alert: look at the last 10 times it fired and find the two or three that, in hindsight, were meaningful. If those cases have something in common (they fired alongside another metric, they fired outside the normal window, they fired longer than usual), that's a refinement opportunity. If the meaningful firings are completely indistinguishable from the noise firings even in hindsight, then the alert definition is genuinely broken and removal is probably right.
What good alert hygiene actually looks like
The teams we've worked with who have the healthiest alert environments share a few practices that don't get discussed often enough.
They tag alerts with intent. Not just what the alert measures, but what it's intended to catch. "This alert should fire when the payment processing queue is backing up due to worker starvation, NOT when it backs up during the expected Monday morning batch window." That intent annotation becomes a test case. Every time the alert fires, you can ask: did it fire for the reason we intended?
They review alert firing history in postmortems, even when the alert is not directly related to the incident. If an alert fired during an incident and wasn't part of the causal chain, that's data about whether the alert adds noise to incident response. Over 10 to 15 postmortems, patterns emerge.
They track "time to first useful action" per alert. Not just whether the alert was acted on, but how long between alert firing and the on-call engineer doing something consequential. Alerts with long median times before first action are either poorly defined, not actionable, or not routed to the right person.
None of this requires new tooling. It requires treating your alert estate as a system that degrades over time and needs active maintenance, rather than a set-and-forget configuration that gets revisited only when someone complains.
The feedback loop that makes it stick
Alert fatigue gets worse over time without an explicit feedback mechanism. Alerts get added when incidents happen (good instinct: detect this class of problem earlier). Alerts don't get removed when the thing they were meant to detect stops occurring or the system evolves past it. Over 12 to 18 months, the alert estate grows in one direction only, toward noise.
The feedback loop that works: every quarter, pull alert firing data. Rank alerts by firing frequency and by the ratio of acknowledged-without-action to acknowledged-with-action. The alerts in the top quartile of firing frequency with the lowest action ratios are candidates for refinement or removal. This doesn't have to be a long process. An hour of review every 90 days is enough to prevent the slow drift toward noise that most teams experience.
Alert fatigue is solvable, but only if you frame it correctly. The goal isn't fewer alerts. The goal is alerts where the information they carry tells the on-call engineer something that changes what they do next. That's a design constraint, not a volume constraint. Treat it accordingly.