Blog · · 10 min read

AI agent trace sampling without losing rare failures

AI agent trace sampling without losing rare failures

A production agent can emit perfect spans and still leave you blind. Uniform or head-only sampling may discard the uncommon tool failure, expensive loop, policy denial, or low-volume customer path that your team needs during an incident. The result is an observability bill that keeps growing while the useful evidence keeps disappearing. This guide gives platform and SRE teams an AI agent trace sampling contract that preserves complete failure traces and representative product slices, caps tail-sampler pressure, and exposes the bias in the retained corpus. You will leave with decision rules, an OpenTelemetry configuration, and a retention drill that proves the policy works before the next incident.

Why instrumentation is not enough

End-to-end tracing answers a structural question: which model calls, tool calls, retries, handoffs, and product actions belong to one run? Sampling answers a different question: which of those complete run histories will still exist when somebody needs them?

The interesting property of an agent run often appears late. At the first span, a routine support request and a runaway planner can look identical. The planner may later repeat a tool, cross a cost budget, hit an approval denial, or finish with an incorrect product outcome. A head sampler has already made its decision by then.

OpenTelemetry distinguishes head sampling from tail sampling. Head sampling decides near the start of a trace. Tail sampling waits for more or all of the trace, then decides using information such as status, duration, or attributes. Waiting creates better selection signals, but it also creates state: the processor must buffer spans until it can decide.

Agent workloads make the tradeoff sharper. One user request can branch through several model and tool calls, so trace sizes vary widely. Rare routes may also carry the highest operational value. A low-volume tenant with a custom product configuration can fail in a way that a dominant tenant never does. A one-percent uniform sample can preserve thousands of ordinary traces while preserving none from that smaller group.

Low-volume products often should keep everything. High-volume products need explicit retention rules whose failure behavior is tested like any other production control.

Separate service metrics from the trace corpus

Do not use a sampled trace store as the only source for request rates, error rates, latency distributions, or budget consumption. A keep policy deliberately overrepresents failures, slow runs, costly runs, and selected customer slices. Those traces are useful for diagnosis, but their proportions no longer describe production traffic.

Run two telemetry paths:

  1. Emit aggregate counters and histograms for every run through a metrics pipeline.
  2. Send spans through the trace sampling pipeline for detailed diagnosis and evaluation.

The metrics path should count authoritative product outcomes, not only HTTP responses. Useful dimensions include workflow type, outcome class, release version, tenant tier, tool family, and policy decision. Keep cardinality bounded. Do not put raw user IDs, prompts, or unbounded action IDs into metric labels.

The trace path can carry higher-cardinality correlation fields when privacy policy permits. It still needs a clear data contract. Sampling does not make sensitive trace content safe. Apply minimization and redaction before exporting spans to a collector or vendor.

Define a trace class before writing rules

A sampler cannot preserve a business-relevant failure unless the trace exposes a machine-readable signal for it. Status code alone is not enough. An agent may return HTTP 200 while failing to complete the requested action, or it may correctly refuse an unauthorized action.

Add a small, bounded classification record to the root span when the run reaches a terminal state:

{
  "workflow": "support_refund",
  "outcome": "denied_by_policy",
  "tenant_tier": "enterprise",
  "risk_class": "financial_write",
  "release": "agent-2026-09-09.2",
  "tool_calls": 7,
  "retry_count": 2,
  "duration_ms": 18420,
  "cost_band": "over_budget",
  "eval_candidate": true
}

The schema is a product-specific implementation recommendation rather than an OpenTelemetry standard. Choose fields that match your product records and operating controls. The values should come from authoritative runtime state. Do not ask the model to label its own success, risk, or cost band.

Keep the vocabulary small enough to test. If outcome can contain arbitrary text, the sampler cannot enforce stable policy. If tenant_tier is a stand-in for tenant identity, it will not protect a small tenant inside a large tier. For that case, assign tenants to a bounded sampling stratum or use a privacy-safe deterministic bucket.

Use head sampling only for decisions known at start

Head sampling is useful when the decision does not depend on the eventual outcome. Examples include synthetic traffic, explicitly debug-enabled runs, or traffic that policy forbids you to export at all.

A practical head decision has three branches:

  • Force keep: retain synthetic probes and narrowly authorized diagnostic sessions.
  • Force drop: reject prohibited telemetry classes before sensitive span content leaves the process.
  • Defer: let the tail processor decide after the run exposes its outcome, duration, cost, and path.

Do not use an early probabilistic drop if the discarded traces never reach a tail processor. The tail stage cannot recover spans it did not receive. Parent-based behavior also matters across services because a propagated sampling decision can affect every child span. Test the complete product path rather than checking one SDK in isolation.

Build ordered tail retention rules

Treat the policy as a decision table, even if the collector evaluates configured policies as a set. Review higher-value classes first so engineers can explain why a trace was retained.

Decision classRecommended actionReason
Runtime or tool errorKeep 100%Required for incident diagnosis
Timeout or cancelled after side effectsKeep 100%Needed to reconcile product state
Policy denial on consequential actionKeep 100%Needed to audit control behavior
Cost, token, or tool-call budget breachKeep 100%Finds loops and amplification
Fixed synthetic probeKeep 100%Proves the pipeline still works
Underrepresented workflow or tenant stratumKeep to a minimum quotaPrevents dominant traffic from erasing small populations
Explicit eval candidateKeep 100% within privacy limitsSupplies reviewed production cases
Routine successKeep probabilisticallyControls storage without losing all healthy baselines

The OpenTelemetry Collector tail sampling processor supports policies based on status, latency, numeric attributes, string attributes, trace state, and probability. A minimal starting point can look like this:

processors:
  tail_sampling/agents:
    decision_wait: 30s
    num_traces: 50000
    expected_new_traces_per_sec: 2000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-runs
        type: latency
        latency:
          threshold_ms: 15000
      - name: eval-candidates
        type: string_attribute
        string_attribute:
          key: agent.eval_candidate
          values: ["true"]
      - name: routine-success-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 2

Use this as a shape, not a production-ready capacity value. decision_wait, num_traces, expected arrival rate, and the thresholds must come from measured trace duration and traffic in your system. Add product outcome and stratum rules only after the corresponding root-span fields are reliable.

The processor is stateful. Its documentation requires all spans for a trace to reach the same collector instance for effective decisions. Route by trace ID when scaling the collector tier. If spans for one trace are split across instances, a retained record may be incomplete even though each individual span was exported successfully.

Protect the tail sampler from its own backlog

Tail sampling shifts cost from long-term storage into collector memory and coordination. Long-running agents make this harder because a trace may remain open through retries, background work, or human approval.

Set four limits explicitly:

  1. Decision wait: how long the processor waits for spans before deciding.
  2. Trace capacity: how many pending traces it may buffer.
  3. Arrival estimate: the expected rate used to size internal structures.
  4. Late-span behavior: what happens when spans arrive after the decision.

Do not choose decision_wait from average latency. Use a high percentile for the class of runs that should remain one trace, then decide how genuinely long-lived workflows will be represented. A workflow paused for hours should usually close one execution trace and resume under a linked trace, rather than keeping a collector buffer open for the entire approval delay.

Capacity exhaustion needs a declared outcome. The OpenTelemetry adaptive tail-sampling implementation tracker documents work around bounded buffers, oldest-pending-trace eviction, probabilistic load shedding, weighted decisions, and performance. Overload behavior belongs in the policy and should not remain an invisible collector detail.

Monitor pending traces, dropped spans, decision latency, evictions, late spans, export failures, and collector memory. Alert when an eviction or overflow can violate a required keep class. If the collector cannot guarantee that, fail visibly and preserve a cheaper incident record through the metrics or event pipeline.

Preserve rare product slices without storing every success

A rule that keeps all errors still misses silent quality failures and small populations. Add bounded stratification for the product slices where behavior can differ materially:

  • workflow or tool family
  • release or model route
  • tenant configuration class
  • language or locale
  • authorization or risk class
  • experiment cohort

Set a minimum retained count per slice and time window, then probabilistically reduce excess routine successes. Use deterministic sampling within a slice when repeatability matters. Hash stable, privacy-safe inputs so the same run is not kept by one service and dropped by another.

Avoid an unbounded rule per tenant. It creates high-cardinality configuration and can make storage scale with customer count. Group tenants by meaningful runtime differences, then create temporary narrow overrides for an incident or launch. Record who created the override, its expiry, and its expected storage impact.

An implementation guide from TianPan describes head, tail, and stratified decisions for error, cost, tenant, and retention classes. Treat it as a design input. Your answer depends on volume, privacy, incident requirements, and the cost of missing a class of failure.

Disclose sampling bias to every downstream user

A retained trace set is not a random production dataset once you keep all failures and selectively preserve rare slices. That is acceptable if consumers know what happened.

Attach these fields to each retained trace or its index record:

  • policy version
  • matched keep rule
  • sampling probability where applicable
  • decision timestamp
  • collector or partition identity
  • completeness state
  • privacy transformation version

Evaluation jobs should filter or weight traces according to their question. An incident corpus should emphasize failures. A prevalence estimate needs unsampled metrics or probability-aware weighting. A regression dataset may intentionally balance rare failure classes, but it must not claim to mirror production frequency.

Never calculate a production success rate by dividing successful sampled traces by all sampled traces. Your keep policy changed both the numerator and denominator.

Verify retention with controlled failures

Configuration review cannot prove that real spans reach one processor, finish before the decision, retain the expected fields, and survive export. Run a retention drill before rollout and after every policy or collector-topology change.

Create fixtures for:

  1. routine success
  2. tool error
  3. timeout after a completed write
  4. denied consequential action
  5. tool-call or cost budget breach
  6. low-volume tenant or workflow stratum
  7. synthetic eval candidate
  8. late child span
  9. collector buffer saturation
  10. prohibited sensitive payload

For each fixture, record the expected decision, matched rule, trace completeness, and authoritative product outcome. Query the destination by trace ID. A passing error fixture requires the complete trace, not merely the root span. A passing privacy fixture proves the forbidden payload never reached the exporter.

Then compare unsampled metric counts with trace-index counts by decision class. Differences should match the configured probabilities and required-keep rules within declared tolerances. Investigate missing required classes immediately. The drill catches a green pipeline that silently retains the wrong traffic.

One practitioner discussion about agent tracing frames sampling as an immediate response to telemetry cost. Cost pressure is real, but a percentage change is not enough. Change the policy only with an estimated storage effect, a retention matrix, and a drill receipt.

Roll out the policy without losing the incident window

Start in observe-only mode. Compute keep and drop decisions while continuing the current retention behavior. Compare projected volume, class coverage, and collector pressure for at least one representative business cycle.

Next, enable sampling for routine successes while keeping required classes at full retention. Hold back one collector partition or traffic cohort so you can compare the sampled corpus with the previous policy. Watch evictions, incomplete traces, rare-slice coverage, and storage spend.

Version every change. A policy release should contain the decision table, collector configuration, expected volume, privacy review, fixture results, rollback threshold, and owner. Roll back when required traces disappear, collector pressure crosses its limit, or the observed class distribution differs materially from the projection.

Run the retention drill now

Choose one high-volume agent workflow and add authoritative terminal outcome, cost band, risk class, release, and eval-candidate fields to its root span. Write the expected keep matrix for the ten fixtures above. Run them through the real collector topology, saturate the pending-trace buffer, and query every expected trace by ID. Do not lower the routine-success rate until all required classes survive as complete traces and the metrics path still reports unbiased production totals.

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us