The Direct Answer
For a B2B command-center SaaS serving multiple teams, the best OpenTelemetry sampling strategy is usually a layered policy rather than a single percentage. Start with parent-based, trace-ID-ratio sampling at the SDK or collector ingress, keep a small deterministic fraction of traces, and preserve selected traces using attributes such as priority, tenant, service, or outcome. Many teams begin with a 1% to 5% baseline, measure ingestion and storage cost, and then adjust it. High-value operations—such as failed checkout, elevated latency, incident responses, or activity from strategic accounts—deserve separate policies, but outcome-based retention should be handled carefully because a span is not complete until the request finishes. OpenTelemetry’s built-in strategies include AlwaysOn, AlwaysOff, TraceIDRatioBased, and ParentBased; the parent-based composite combines these behaviors. A practical production design uses more than one of them instead of pretending that one ratio answers every business question.
Also worth reading: What are the most effective enterprise AI budget control strategies for 2026 operations? · How Should Enterprises Build AI Governance Frameworks for Multi-Agent Operations in 2026? · How Should Engineering Leaders Architect Multi-Tenant Operations Telemetry Ingestion Pipelines in 2026?
The right target is not the lowest possible sampling rate. It is the lowest rate that still supports reliable operational investigation, agreed service-level objectives, and useful evidence during incidents. A command center that samples 0.1% of traces may save storage charges while making a cross-team incident difficult to reconstruct; sampling 100% may preserve detail but create an unaffordable and slow telemetry pipeline. The final policy should therefore connect trace volume to actual platform limits, monthly cost, investigation requirements, and acceptable confidence in aggregate metrics. Tail-based retention is often more useful than indiscriminate head sampling for this purpose, although it adds processing complexity and does not eliminate the need for volume controls.
How OpenTelemetry Sampling Actually Works
Sampling decides which traces the SDK records and exports. It is not the same as generating spans, calculating metrics, or aggregating logs. A trace can be sampled while its service still emits counters and histograms, so observability strategies can remain statistically useful even when individual traces are sparse. The OpenTelemetry specification defines sampling through the Trace SDK, while distributions such as Java, Go, .NET, JavaScript, and Python may expose configuration differently. An AlwaysOff sampler suppresses all traces, AlwaysOn records all traces, TraceIDRatioBased makes a probabilistic decision from the trace ID, and ParentBased delegates to the parent when one exists. Without a parent, a configurable root sampler handles the decision.
The trace ID ratio approach is deterministic: a given trace ID normally produces the same decision across services, which helps avoid fragmented traces where the gateway records a request but downstream services discard it. However, that consistency depends on propagating the sampling decision and trace context correctly. Instrumentation must also respect the parent decision, or a service may generate complete local traces for requests that were meant to be dropped. Head sampling occurs early, often in the application SDK, and is cheap because discarded traces never reach the collector. Its weakness is visibility: an early sampler cannot know whether a request will later become slow, fail, or involve a high-priority customer.
Tail sampling occurs after spans have been generated, commonly in the OpenTelemetry Collector. The collector can buffer spans by trace ID, wait for more information, and apply attributes such as HTTP status, service name, or a computed priority. That makes it better suited to keeping important traces, but it consumes memory, requires a suitable trace ID-ratio upper bound, and adds latency to the pipeline. A collector configured for 100% tail sampling can become a bottleneck even if only 2% of traces are ultimately retained. Sampling therefore has to be treated as a system design decision involving SDKs, gateways, collectors, storage backends, and investigation workflows—not merely a YAML setting.
Head Sampling, Tail Sampling, and Their Best Uses
Head sampling is usually the first cost control because it prevents most data creation and export. It works well for high-volume, low-risk operations where leadership needs trend evidence rather than complete request histories. A platform might retain 2% of successful reads but keep a larger proportion of writes, administrative actions, and background jobs. Head sampling is less suitable when the definition of an interesting trace depends on something that happens after the request begins. It also offers limited control over customer importance unless requests carry an explicit priority attribute early enough for the SDK to use.
Tail sampling allows the pipeline to retain traces after observing richer information. A 2026 multi-team deployment might send all traces to a short-lived collector buffer, keep traces containing errors, traces above a 2-second threshold, and traces marked for incident response, then let everything else expire. The exact thresholds should come from service-level objectives and measured distributions, not copied from an article. If 99% of healthy API calls finish below 400 milliseconds, a 2-second trigger is defensible for investigating outliers; if normal p95 is already 1.8 seconds, that same trigger may capture too much traffic. Tail sampling is also more expensive operationally because spans still travel to the collector, and a collector failure can delay or lose buffered spans.
| Feature | Head sampling | Tail sampling |
|---|---|---|
| Where the decision occurs | Application SDK or early gateway | Collector, after spans are received |
| Data volume before decision | Minimal; most traces stop at the source | Higher; spans must reach and may wait in the collector |
| Context available | Limited to the request at startup | Can use status, latency, tenant, and custom attributes |
| Cost profile | Lower processing and network cost | Higher memory, CPU, and pipeline complexity |
| Typical use | Broad traffic reduction and low-cost baselines | Keeping incidents, slow traces, or priority tenants |
| Main limitation | Cannot reliably know the final outcome | Requires buffering, careful limits, and failure planning |
A Practical Policy for Multi-Team Operations
Begin by measuring a representative period rather than changing production immediately. For at least 7 days, and preferably 14 to 30 days if traffic is seasonal, compare trace counts, spans per trace, collector throughput, storage growth, and the number of investigations that require individual traces. A sensible starting baseline for many B2B platforms is 1% to 5% of ordinary traces, with 100% retention for a narrowly defined set of operational workflows. These are engineering starting points, not OpenTelemetry mandates. A low-volume product may be able to retain all traces, while a high-volume event or telemetry pipeline may need a ratio below 1%.
Define policies by risk instead of applying one percentage to every service. Authentication, authorization, billing, data export, and tenant administration may warrant higher retention than anonymous read endpoints. Tenant concentration matters too: if one account represents 20% of revenue, losing its traces can create more business blind spots than retaining several hundred low-value development requests. A practical design can use explicit service attributes and trusted server-side priority values, with separate policies for normal, elevated, and incident-linked traffic. The policy owner should be named, because an undocumented sampler becomes effectively unowned when teams disagree about whether a missing trace was expected.
Roll out gradually. Compare a proposed 5% policy with the current 1% policy on the same traffic mix, and check whether incident traces, latency distributions, and tenant coverage improve enough to justify the additional volume. Review the decision after 7 days and again after 30 days, since changes in traffic, SDK versions, and service topology can alter the result. Keep an emergency path for raising retention during an incident, but require it to expire automatically. Without an expiry, a temporary “keep everything” switch can become a permanent cost increase.
How to Configure Parent-Based and Ratio-Based Decisions
Parent-based sampling is the default coordination pattern for distributed traces because downstream services should follow the incoming decision. Configure the root sampler, such as TraceIDRatioBased at 0.02 for 2%, and use ParentBased with an appropriate root behavior. In pseudocode terms, the intended policy is “respect a remote parent; otherwise sample 2%.” This is more reliable than independently sampling every service at 2%, which could produce many partial traces. The exact configuration keys vary by language, so the implementation should follow the OpenTelemetry specification and the relevant SDK documentation rather than assume that Java, Go, and browser environments expose identical options.
Instrumentation should propagate W3C Trace Context and the OpenTelemetry sampling flags consistently. If a gateway samples a request, the application and downstream services should follow it; if the gateway drops it, descendants should not create a new unrelated sampled trace. Check that retries do not multiply stored traces unexpectedly, and confirm that asynchronous jobs, queues, and scheduled tasks receive a sensible root decision. Distributed traces can cross process and organizational boundaries, so a partner service may propagate a decision that is valid for its own policy but conflicts with the receiving system’s retention goals. Document those boundaries and decide whether to respect the parent, resample, or create separate linked traces.
Ratio-based sampling is attractive because it does not require a central list of important requests. It is also predictable over large populations, which makes cost forecasting easier. The ratio applies to trace roots, not necessarily to every span, so a retained trace can still contain hundreds of spans. In practice, storage cost depends on both trace count and average spans per trace; a 2% trace rate is not automatically half the cost of a 1% rate. Track compressed payload size as well as trace count, because attribute-heavy spans can dominate expense. Use the collector’s own metrics and the backend’s billing or quota data to verify the relationship.
Alternatives Beyond Built-In SDK Sampling
The built-in samplers are only part of the design space. Teams can add rules based on service, tenant, route, or custom attributes, although custom decision logic should remain deterministic and inexpensive. A rule might retain all traces from a service handling money movement, or keep a higher ratio for enterprise tenants. This is useful when the attribute is trustworthy and available before sampling. It is risky when the decision depends on client-controlled input or when teams independently add overlapping rules. Centralizing policy in the collector can make behavior more visible, but it cannot recover data that application SDKs never sent.
Another alternative is to separate telemetry by purpose. Keep a broad, low-cost stream for trend analysis and create targeted diagnostic traces for commands, errors, or incident windows. This reduces dependence on a single “representative sample” and aligns data quality with the question being asked. Metrics and logs can supplement missing traces: RED metrics, request counts, error rates, and latency histograms can preserve population-level evidence even when individual traces are absent. Log sampling has its own tradeoffs, and logs are not automatically a replacement for traces because they lack equivalent parent-child structure and timing relationships.
Some teams use adaptive sampling based on traffic volume, but “adaptive” should not mean unpredictable. A collector can temporarily increase retention during an alert, then return to a baseline after a defined period. This helps incident response but can create sudden cost spikes and overload. Set a hard upper bound, a maximum buffer duration, and a rollback procedure. A sampling system that cannot answer “what fraction did we retain for this tenant and service yesterday?” is difficult to trust in a command center, even if it saves money in an average month.
Common Mistakes and Failure Modes
The most common mistake is treating the sampling percentage as a promise of representative evidence. Random sampling supports aggregate estimates, but rare failures may be too sparse to investigate, and small tenants can disappear from a sample. A 99% retention rate sounds excellent until the resulting pipeline misses its delivery objective or exceeds storage quotas. The second mistake is sampling everything at the gateway while allowing downstream services to ignore the propagated decision, creating broken or misleading traces. The third is enabling tail sampling without understanding the collector’s memory and trace-ID limits.
Teams also make the mistake of using outcome attributes without validating them. A client can send an HTTP status-looking attribute, but it should not be able to mark a request as an incident. Conversely, relying only on final error status can miss slow requests that never failed. Verify the attributes at the point where decisions are made, keep decision rules in version control, and record the active policy in telemetry or deployment metadata. Do not confuse span-level suppression with trace-level retention; deleting individual spans can make a trace harder to interpret and may break expected trace structure.
Finally, review sampling after topology changes. Adding a service, switching to asynchronous messaging, or moving workloads to serverless can change spans per trace and volume assumptions. A policy tuned for synchronous APIs may be inappropriate for high-cardinality event streams. Test error paths, retries, background jobs, and cross-tenant traffic separately. Sampling bugs often appear under failure conditions, exactly when the data is most valuable, so failure-path tests belong in the same release process as normal telemetry checks.
When to Act, and What It Costs
Act now if telemetry cost is growing faster than useful investigation value, if collector queues are rising, or if the platform cannot answer basic questions during incidents. Those signals are more actionable than a generic claim that sampling is “outdated.” For a multi-team SaaS, a reasonable trigger is a sustained storage increase of roughly 20% month over month without a corresponding improvement in incident resolution, or a trace ingestion volume that consumes more than the team’s agreed telemetry budget. The numbers are organizational thresholds, not universal standards; define the budget first.
OpenTelemetry itself is open source, so the SDK and Collector do not carry a license fee. The direct software cost can therefore be low, while the real expense is the backend, network transfer, collector compute, engineering time, and investigation labor. Cloud trace backends may price by ingested spans, stored spans, queries, retention duration, or a combination. A 1% sampling rate can still be expensive when each trace has 80 spans and high-cardinality attributes, while a 5% rate may be inexpensive for a low-volume service. Calculate expected monthly ingestion by multiplying eligible trace volume, average spans per trace, average bytes per span, and retention days, then test the estimate against invoices or quota alerts.
The practical sequence is to measure, choose a baseline, validate propagation, deploy a small change, observe for at least one full business cycle, and document the result. Revisit the policy when traffic changes by roughly 50%, when a new service is introduced, or when a major incident exposes missing evidence. For a leadership-focused command center, sampling should make cross-team operations more explainable without pretending that every request deserves indefinite retention. That is the balance to optimize as of September 2026: selective, measurable, and reversible.
The Recommended Operating Decision
Adopt ParentBased with a TraceIDRatioBased root policy as the starting control, commonly between 1% and 5% for ordinary traffic, then add narrowly scoped retention for high-risk services and trusted operational workflows. Use collector tail sampling when final status, latency, or tenant context is required to make the decision, but cap the incoming volume and set a finite buffer window. Keep metrics and logs independent so that sampling does not silently remove population-level visibility. The exact ratio should be derived from measured trace volume, spans per trace, storage prices, delivery limits, and incident needs.
Treat the policy as a production configuration with an owner, version history, dashboards, and an expiry mechanism for emergency overrides. Run a rollback test and confirm that SDKs, gateways, collectors, and downstream services agree on the trace context. Do not claim that a sampled trace is complete unless the policy explicitly designed it that way, and do not claim that aggregate metrics are exact when they are estimates from sampled data. That discipline gives multi-team operations a defensible answer to both “what happened?” and “why are we paying this much?” without sacrificing the evidence leaders need when several teams must coordinate under time pressure.