Blog · · 10 min read
AI agent trace sampling without losing rare failures

A production agent can emit perfect spans and still leave you blind. Uniform or head-only sampling may discard the uncommon tool failure, expensive loop, policy denial, or low-volume customer path that your team needs during an incident. The result is an observability bill that keeps growing while the useful evidence keeps disappearing. This guide gives platform and SRE teams an AI agent trace sampling contract that preserves complete failure traces and representative product slices, caps tail-sampler pressure, and exposes the bias in the retained corpus. You will leave with decision rules, an OpenTelemetry configuration, and a retention drill that proves the policy works before the next incident.
Why instrumentation is not enough
End-to-end tracing answers a structural question: which model calls, tool calls, retries, handoffs, and product actions belong to one run? Sampling answers a different question: which of those complete run histories will still exist when somebody needs them?
The interesting property of an agent run often appears late. At the first span, a routine support request and a runaway planner can look identical. The planner may later repeat a tool, cross a cost budget, hit an approval denial, or finish with an incorrect product outcome. A head sampler has already made its decision by then.
OpenTelemetry distinguishes head sampling from tail sampling. Head sampling decides near the start of a trace. Tail sampling waits for more or all of the trace, then decides using information such as status, duration, or attributes. Waiting creates better selection signals, but it also creates state: the processor must buffer spans until it can decide.
Agent workloads make the tradeoff sharper. One user request can branch through several model and tool calls, so trace sizes vary widely. Rare routes may also carry the highest operational value. A low-volume tenant with a custom product configuration can fail in a way that a dominant tenant never does. A one-percent uniform sample can preserve thousands of ordinary traces while preserving none from that smaller group.
Low-volume products often should keep everything. High-volume products need explicit retention rules whose failure behavior is tested like any other production control.
Separate service metrics from the trace corpus
Do not use a sampled trace store as the only source for request rates, error rates, latency distributions, or budget consumption. A keep policy deliberately overrepresents failures, slow runs, costly runs, and selected customer slices. Those traces are useful for diagnosis, but their proportions no longer describe production traffic.
Run two telemetry paths:
- Emit aggregate counters and histograms for every run through a metrics pipeline.
- Send spans through the trace sampling pipeline for detailed diagnosis and evaluation.
The metrics path should count authoritative product outcomes, not only HTTP responses. Useful dimensions include workflow type, outcome class, release version, tenant tier, tool family, and policy decision. Keep cardinality bounded. Do not put raw user IDs, prompts, or unbounded action IDs into metric labels.
The trace path can carry higher-cardinality correlation fields when privacy policy permits. It still needs a clear data contract. Sampling does not make sensitive trace content safe. Apply minimization and redaction before exporting spans to a collector or vendor.
Define a trace class before writing rules
A sampler cannot preserve a business-relevant failure unless the trace exposes a machine-readable signal for it. Status code alone is not enough. An agent may return HTTP 200 while failing to complete the requested action, or it may correctly refuse an unauthorized action.
Add a small, bounded classification record to the root span when the run reaches a terminal state:
{
"workflow": "support_refund",
"outcome": "denied_by_policy",
"tenant_tier": "enterprise",
"risk_class": "financial_write",
"release": "agent-2026-09-09.2",
"tool_calls": 7,
"retry_count": 2,
"duration_ms": 18420,
"cost_band": "over_budget",
"eval_candidate": true
}
The schema is a product-specific implementation recommendation rather than an OpenTelemetry standard. Choose fields that match your product records and operating controls. The values should come from authoritative runtime state. Do not ask the model to label its own success, risk, or cost band.
Keep the vocabulary small enough to test. If outcome can contain arbitrary text, the sampler cannot enforce stable policy. If tenant_tier is a stand-in for tenant identity, it will not protect a small tenant inside a large tier. For that case, assign tenants to a bounded sampling stratum or use a privacy-safe deterministic bucket.
Use head sampling only for decisions known at start
Head sampling is useful when the decision does not depend on the eventual outcome. Examples include synthetic traffic, explicitly debug-enabled runs, or traffic that policy forbids you to export at all.
A practical head decision has three branches:
- Force keep: retain synthetic probes and narrowly authorized diagnostic sessions.
- Force drop: reject prohibited telemetry classes before sensitive span content leaves the process.
- Defer: let the tail processor decide after the run exposes its outcome, duration, cost, and path.
Do not use an early probabilistic drop if the discarded traces never reach a tail processor. The tail stage cannot recover spans it did not receive. Parent-based behavior also matters across services because a propagated sampling decision can affect every child span. Test the complete product path rather than checking one SDK in isolation.
Build ordered tail retention rules
Treat the policy as a decision table, even if the collector evaluates configured policies as a set. Review higher-value classes first so engineers can explain why a trace was retained.
| Decision class | Recommended action | Reason |
|---|---|---|
| Runtime or tool error | Keep 100% | Required for incident diagnosis |
| Timeout or cancelled after side effects | Keep 100% | Needed to reconcile product state |
| Policy denial on consequential action | Keep 100% | Needed to audit control behavior |
| Cost, token, or tool-call budget breach | Keep 100% | Finds loops and amplification |
| Fixed synthetic probe | Keep 100% | Proves the pipeline still works |
| Underrepresented workflow or tenant stratum | Keep to a minimum quota | Prevents dominant traffic from erasing small populations |
| Explicit eval candidate | Keep 100% within privacy limits | Supplies reviewed production cases |
| Routine success | Keep probabilistically | Controls storage without losing all healthy baselines |
The OpenTelemetry Collector tail sampling processor supports policies based on status, latency, numeric attributes, string attributes, trace state, and probability. A minimal starting point can look like this:
processors:
tail_sampling/agents:
decision_wait: 30s
num_traces: 50000
expected_new_traces_per_sec: 2000
policies:
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
- name: slow-runs
type: latency
latency:
threshold_ms: 15000
- name: eval-candidates
type: string_attribute
string_attribute:
key: agent.eval_candidate
values: ["true"]
- name: routine-success-baseline
type: probabilistic
probabilistic:
sampling_percentage: 2
Use this as a shape, not a production-ready capacity value. decision_wait, num_traces, expected arrival rate, and the thresholds must come from measured trace duration and traffic in your system. Add product outcome and stratum rules only after the corresponding root-span fields are reliable.
The processor is stateful. Its documentation requires all spans for a trace to reach the same collector instance for effective decisions. Route by trace ID when scaling the collector tier. If spans for one trace are split across instances, a retained record may be incomplete even though each individual span was exported successfully.
Protect the tail sampler from its own backlog
Tail sampling shifts cost from long-term storage into collector memory and coordination. Long-running agents make this harder because a trace may remain open through retries, background work, or human approval.
Set four limits explicitly:
- Decision wait: how long the processor waits for spans before deciding.
- Trace capacity: how many pending traces it may buffer.
- Arrival estimate: the expected rate used to size internal structures.
- Late-span behavior: what happens when spans arrive after the decision.
Do not choose decision_wait from average latency. Use a high percentile for the class of runs that should remain one trace, then decide how genuinely long-lived workflows will be represented. A workflow paused for hours should usually close one execution trace and resume under a linked trace, rather than keeping a collector buffer open for the entire approval delay.
Capacity exhaustion needs a declared outcome. The OpenTelemetry adaptive tail-sampling implementation tracker documents work around bounded buffers, oldest-pending-trace eviction, probabilistic load shedding, weighted decisions, and performance. Overload behavior belongs in the policy and should not remain an invisible collector detail.
Monitor pending traces, dropped spans, decision latency, evictions, late spans, export failures, and collector memory. Alert when an eviction or overflow can violate a required keep class. If the collector cannot guarantee that, fail visibly and preserve a cheaper incident record through the metrics or event pipeline.
Preserve rare product slices without storing every success
A rule that keeps all errors still misses silent quality failures and small populations. Add bounded stratification for the product slices where behavior can differ materially:
- workflow or tool family
- release or model route
- tenant configuration class
- language or locale
- authorization or risk class
- experiment cohort
Set a minimum retained count per slice and time window, then probabilistically reduce excess routine successes. Use deterministic sampling within a slice when repeatability matters. Hash stable, privacy-safe inputs so the same run is not kept by one service and dropped by another.
Avoid an unbounded rule per tenant. It creates high-cardinality configuration and can make storage scale with customer count. Group tenants by meaningful runtime differences, then create temporary narrow overrides for an incident or launch. Record who created the override, its expiry, and its expected storage impact.
An implementation guide from TianPan describes head, tail, and stratified decisions for error, cost, tenant, and retention classes. Treat it as a design input. Your answer depends on volume, privacy, incident requirements, and the cost of missing a class of failure.
Disclose sampling bias to every downstream user
A retained trace set is not a random production dataset once you keep all failures and selectively preserve rare slices. That is acceptable if consumers know what happened.
Attach these fields to each retained trace or its index record:
- policy version
- matched keep rule
- sampling probability where applicable
- decision timestamp
- collector or partition identity
- completeness state
- privacy transformation version
Evaluation jobs should filter or weight traces according to their question. An incident corpus should emphasize failures. A prevalence estimate needs unsampled metrics or probability-aware weighting. A regression dataset may intentionally balance rare failure classes, but it must not claim to mirror production frequency.
Never calculate a production success rate by dividing successful sampled traces by all sampled traces. Your keep policy changed both the numerator and denominator.
Verify retention with controlled failures
Configuration review cannot prove that real spans reach one processor, finish before the decision, retain the expected fields, and survive export. Run a retention drill before rollout and after every policy or collector-topology change.
Create fixtures for:
- routine success
- tool error
- timeout after a completed write
- denied consequential action
- tool-call or cost budget breach
- low-volume tenant or workflow stratum
- synthetic eval candidate
- late child span
- collector buffer saturation
- prohibited sensitive payload
For each fixture, record the expected decision, matched rule, trace completeness, and authoritative product outcome. Query the destination by trace ID. A passing error fixture requires the complete trace, not merely the root span. A passing privacy fixture proves the forbidden payload never reached the exporter.
Then compare unsampled metric counts with trace-index counts by decision class. Differences should match the configured probabilities and required-keep rules within declared tolerances. Investigate missing required classes immediately. The drill catches a green pipeline that silently retains the wrong traffic.
One practitioner discussion about agent tracing frames sampling as an immediate response to telemetry cost. Cost pressure is real, but a percentage change is not enough. Change the policy only with an estimated storage effect, a retention matrix, and a drill receipt.
Roll out the policy without losing the incident window
Start in observe-only mode. Compute keep and drop decisions while continuing the current retention behavior. Compare projected volume, class coverage, and collector pressure for at least one representative business cycle.
Next, enable sampling for routine successes while keeping required classes at full retention. Hold back one collector partition or traffic cohort so you can compare the sampled corpus with the previous policy. Watch evictions, incomplete traces, rare-slice coverage, and storage spend.
Version every change. A policy release should contain the decision table, collector configuration, expected volume, privacy review, fixture results, rollback threshold, and owner. Roll back when required traces disappear, collector pressure crosses its limit, or the observed class distribution differs materially from the projection.
Run the retention drill now
Choose one high-volume agent workflow and add authoritative terminal outcome, cost band, risk class, release, and eval-candidate fields to its root span. Write the expected keep matrix for the ten fixtures above. Run them through the real collector topology, saturate the pending-trace buffer, and query every expected trace by ID. Do not lower the routine-success rate until all required classes survive as complete traces and the metrics path still reports unbiased production totals.