Distributed tracing is one of the most powerful observability tools available for understanding what happens inside a complex system during an incident. It's also, for most teams at meaningful traffic volumes, unusable at full fidelity without a sampling strategy that makes the data fit within cost and storage constraints. The result is an uncomfortable reality: you instrument everything, you collect a representative sample, and you hope the traces you discarded aren't the ones you needed when something went wrong.
The sampling tax is the gap between the tracing data you could have and the tracing data you do have, measured in incidents where the relevant trace was discarded before you needed it.
The Head-Based Sampling Problem
Most tracing implementations, including the default behavior of OpenTelemetry when configured with a simple probabilistic sampler, use head-based sampling. The sampling decision is made at the start of the request trace, before you know anything about how the request will behave. A 10% sampling rate means 90% of all traces are discarded immediately, regardless of whether they have errors, high latency, or any other interesting property.
Head-based sampling is computationally cheap because the decision happens at trace initialization. The cost is that sampling decisions are entirely uninformed by outcome. A 500-millisecond-with-error request at the p99.9 of your latency distribution has the same 10% chance of being kept as a 20-millisecond healthy request. The errors and outliers you most need during an incident investigation are statistically no more likely to be retained than normal traffic.
During a production incident, this is the worst possible property. You need to find the traces that show the failure path. Those traces are a small fraction of total traffic. They were discarded at the same rate as everything else.
Tail-Based Sampling: The Right Idea with Operational Costs
Tail-based sampling addresses the head-based problem by deferring the sampling decision until the trace is complete. You buffer all spans for a configurable window (typically 10-30 seconds), then evaluate the complete trace against sampling policies before deciding whether to keep or discard it. Policies can include: keep all traces with errors, keep all traces over a latency threshold, keep a random sample of normal traces for baseline comparison.
This is the right theoretical approach. You keep the traces you care about. The practical problem is that tail-based sampling requires buffering all incoming spans before making keep/discard decisions, which means you need a stateful collector layer that holds the entire in-flight trace window in memory. For high-volume systems, this is operationally expensive. You need to handle backpressure correctly, deal with trace completeness (what if spans from a service arrive late?), and ensure the collector layer doesn't become a reliability liability during an incident, precisely when you most need your tracing to work.
OpenTelemetry Collector's tail sampling processor is a real implementation of this, and teams do run it successfully. The overhead is non-trivial. For a service generating 10,000 requests per second across a 30-second window, you're buffering 300,000 in-flight spans before making sampling decisions. That requires careful capacity planning.
Adaptive Sampling: A Middle Path
Adaptive sampling adjusts sampling rates dynamically based on observed system state, rather than using a fixed rate. The basic idea: during normal operation, sample at a low rate (1-5%) to get baseline performance visibility. When anomalies are detected (elevated error rate, latency spike, circuit breaker tripping), automatically increase the sampling rate for affected services to 100% or near-100%.
This approach captures the high-fidelity trace data you need during incidents while keeping steady-state storage costs manageable. The implementation challenge is the detection latency: you need to detect the anomaly and increase the sampling rate before the incident has progressed so far that the relevant traces have already been discarded at the lower rate. For fast-moving failures, this window can be tight.
Jaeger has support for adaptive sampling through its remote sampling service. Datadog APM implements a similar concept with their Analyzed Spans product. The principle is consistent: let normal traffic be undersampled, but ensure error traffic is oversampled.
Priority Sampling and the Error-First Rule
A simpler approach than full adaptive sampling is a two-tier priority sampling rule: always keep traces with errors or anomalous status codes, sample everything else at a low rate. This is sometimes called error-first sampling or priority sampling.
The implementation is straightforward. In your sampling logic, before applying any probabilistic decision, check whether the root span has an error status. If it does, sampling rate = 100%. If it doesn't, apply your normal probabilistic sampler. In OpenTelemetry terms, this can be implemented as a composite sampler that checks span attributes before delegating to a parent-based probabilistic sampler.
// Pseudocode for composite error-first sampler
function sample(span):
if span.status == ERROR:
return RECORD_AND_SAMPLE // always keep errors
if span.http_status >= 500:
return RECORD_AND_SAMPLE // always keep 5xx
if span.latency_ms > p99_threshold:
return RECORD_AND_SAMPLE // always keep high-latency
return parent_based_probabilistic_sampler(span, rate=0.05)
This approach won't catch errors that originate deep in a trace and haven't propagated to the root span at sampling decision time. For synchronous traces, this is usually fine: by the time the response closes, the error status has propagated. For async traces with fire-and-forget spans, you may need to supplement with a log-based approach for the async components.
What Tracing Gives You That Metrics and Logs Don't
Before discussing more advanced approaches, it's worth being explicit about what tracing uniquely provides for incident response and why it's worth the investment in getting sampling right.
Metrics tell you that latency increased on a service. Logs tell you that a specific request threw an exception. Neither tells you the full request path: which services were traversed, at which point in the chain the latency accumulated, whether the slow span was network time or computation time, whether the exception propagated from a downstream dependency or originated locally.
Distributed tracing shows you the full execution graph for a specific request. During an RCA investigation, being able to say "the p99 slowness in the checkout service is entirely attributable to a database query in the payment service that's taking 800ms, correlated with the migration that ran at 14:30" is only possible if you have trace data that links the checkout service call to the payment service database call.
This is why high-fidelity trace capture during incidents is high value. The trace is often the only data source that can definitively locate where in a call chain the problem originated.
Integrating Tracing with Your Broader Correlation Pipeline
Even if you have good trace data, it's only useful during an incident if the on-call engineer can access it efficiently. "Go look at Jaeger/Tempo/Datadog APM and find traces from the last 5 minutes with errors" is still a manual step that requires knowing where to look and what to search for.
The more effective integration is having your trace data feed into the same correlation layer that handles metrics and logs. When an alert fires, the correlation engine should surface: the relevant metric anomaly, the log patterns associated with it, and the trace IDs for representative error traces from the incident window. The engineer shouldn't need to do a separate trace lookup; the traces should be linked from the incident context.
This is something we've built into Devtract's correlation model. When we correlate a log error pattern with a metric anomaly, we also surface any traces that contain both the anomalous timing window and the error log signature, giving the on-call a direct link to the execution path that explains the failure. It's not a replacement for a full tracing UI, but it eliminates the "find the relevant trace" step from the diagnostic process.
The Honest Assessment: Tracing Is Not a Silver Bullet
We want to be direct about the limits of tracing as an RCA tool. Tracing works well for latency analysis and for following the request path through synchronous service calls. It works less well for infrastructure-level failures (OOMKilled pods, node evictions, network partitions) where the failure isn't visible in the request span but in the cluster events around it. It also requires consistent instrumentation across all services in the call chain; a single uninstrumented service creates a gap in the trace that breaks the causal chain.
The sampling tax is real, but it's not the only reason teams don't get full value from their tracing investment. Incomplete instrumentation, slow trace search in high-volume trace stores, and lack of integration with the rest of the incident context are all contributing factors. Getting sampling right is necessary but not sufficient for tracing to actually accelerate your RCA process. It has to be part of a broader observability practice where all three signal types are correlated and accessible in the right context at the right time.
The teams that get the most RCA value from tracing are the ones who treat it as a third signal alongside metrics and logs, not as a standalone investigation tool. That framing changes both how you instrument and how you use the data during incidents.