Blog · · 9 min read
Run trustworthy AI agent A/B testing on stateful agents

AI agent A/B testing fails when the experiment randomizes a request but the product runs a durable workflow. One user can receive the control prompt, resume into the treatment model, retry a tool under another policy, and then change shared product state that affects both cohorts. The dashboard may look clean while the treatment is mixed and the outcome is contaminated. A trustworthy experiment assigns the right product unit once, carries that assignment through the complete run, records actual exposure, isolates side effects, and joins verified product outcomes to the same experiment record.
Start with the release decision
An online experiment should answer a release question. Write that question before choosing a feature flag or metric. A useful decision statement has four parts:
- The release manifest being tested, including model, prompt, tools, policies, and runtime version.
- The eligible population and the business unit that receives the treatment.
- One primary outcome and a small set of safety guardrails.
- A decision rule for promotion, rollback, extension, or inconclusive results.
For example: "Promote release support-agent-42 for eligible workspaces if accepted resolutions per eligible ticket improve without increasing reopened tickets, unauthorized tool attempts, or p95 completion time beyond their declared limits."
That wording separates three kinds of evidence. Model preference measures preference, product records prove whether an effect persisted, and eligible-unit outcomes support the release comparison. Tool-call completion and total usage cannot stand in for either of the last two.
Anthropic's evaluation guidance distinguishes offline evals from production A/B tests. Offline suites give fast, reproducible feedback on known tasks. Live experiments measure behavior and business outcomes under real conditions, but need enough traffic and do not explain why a metric moved. Use both: qualify the candidate offline, then use the online test only for the release question that requires real users and product state.
Choose the unit from the business effect
The randomization unit is the entity assigned to a variation. The analysis unit is the entity represented by one observation in the result. They may differ. LaunchDarkly's randomization documentation warns that a randomization unit receiving both variations invalidates the experiment and explains why the analysis unit must match the metric's causal structure.
Choose the unit that contains the treatment's durable effects:
| Agent workflow | Likely randomization unit | Likely analysis unit | Reason |
|---|---|---|---|
| Draft-only reply suggestions | User or ticket | Ticket | The suggestion affects one case and can be evaluated at acceptance |
| Workspace-wide support automation | Workspace | Workspace or ticket clustered by workspace | Shared policies, memory, and routing affect many tickets |
| Procurement agent changing supplier records | Organization | Organization or approved transaction | Writes alter shared records and future decisions |
| Personal scheduling assistant | User | User or appointment | Preferences, calendars, and correction history are user-scoped |
Do not default to a message, HTTP request, model call, or tool call. Those execution events are too short-lived to serve as stable causal units. If a workspace-level policy or shared memory lets treatment behavior influence control users, randomize by workspace or isolate the shared state. If neither is practical, the design cannot support a clean user-level A/B claim. Workspace-level assignment leaves fewer units and less statistical power than a user-level split, but a user-level split that leaks through shared memory gives a tight estimate of the wrong quantity.
Create one durable experiment envelope
Assign the variation after eligibility and consent checks, but before any variant-dependent model or tool behavior. Persist it with the run. Every retry, resume, callback, subagent, worker, and tool adapter should receive the same immutable experiment envelope.
A minimal record can look like this:
{
"experiment_id": "support-resolution-2026-09",
"layer_id": "support-agent-release",
"assignment_unit": {"type": "workspace", "id_hash": "ws_7f3"},
"analysis_unit": "ticket",
"variant": "treatment",
"release_id": "support-agent-42",
"policy_version": "agent-exp-policy-8",
"assigned_at": "2026-09-02T09:14:00Z",
"exposed_at": null,
"run_id": "run_01J...",
"status": "assigned"
}
Keep the raw user or tenant identifier in the authoritative product store. Analytics events should carry the approved pseudonymous key, experiment, layer, variant, release, and run identity needed for a controlled join.
The flag evaluation must also be inspectable. The OpenFeature flag-evaluation specification defines evaluation context, targeting keys, typed resolution, reasons, errors, metadata, and hooks. Whatever flag provider you use, store more than true or false. Record the evaluated flag key, resolved variant, reason, provider or configuration version, and failure path. A default caused by provider failure is not equivalent to an intentional control assignment.
Treat assignment as immutable for the run. A later configuration change may affect new eligible runs, but it must not silently migrate in-flight work. If a release must be withdrawn, terminate or degrade the run under an explicit recovery policy rather than swapping its model and calling it the same treatment.
Separate assignment from exposure
Assignment says which variant the unit should receive. Exposure says the unit actually crossed the treatment boundary. Emit exposure only when the intended release manifest is loaded and the first variant-dependent step begins.
Assignment and exposure can diverge when:
- The user leaves before the agent starts.
- Eligibility changes after assignment.
- The feature-flag provider times out and the runtime falls back.
- A worker cannot load the treatment model or policy.
- A circuit breaker routes the run to a degraded release.
- The run is cancelled while queued.
Each exposure event should include the experiment and layer IDs, assignment and analysis keys, run ID, variant, release manifest, timestamp, evaluation reason, and current product eligibility version. Reject or quarantine duplicate exposures with conflicting variants for the same assignment unit and experiment.
The outcome pipeline should analyze actual exposure according to the declared method. Keep assignment events as well, because they reveal failures between allocation and execution. Do not silently discard those failures. A treatment that cannot load reliably is part of the release result.
Qualify variants before allowing real side effects
An online experiment should not be the first place a candidate agent touches customer state. Use a staged qualification sequence:
- Run deterministic contract, safety, and regression tests for the complete release manifest.
- Replay representative production traces against isolated product fixtures.
- Shadow live inputs without executing external writes.
- Compare proposed actions, latency, cost, and policy interventions.
- Enable reversible or approval-gated effects for a small eligible cohort.
- Expand only after exposure integrity and guardrail telemetry pass.
Gravity's agent A/B testing guide recommends offline evaluation and shadow mode before live treatment, sticky cohorts, and limiting irreversible actions while qualifying variants. FeatBit's experimentation guide also treats workflow-level experimentation, safety gates, rollback, and low-traffic before-and-after evidence as separate controls rather than one flag toggle.
For a shadow phase, run the treatment far enough to evaluate reasoning, tool selection, and proposed commands, but route writes to a simulator or policy sink. Record what would have happened. Never point both variants at real side-effect endpoints and assume a feature flag makes the comparison safe.
Prevent side-effect contamination
Stateful agents create interference when one unit's treatment changes another unit's future inputs or outcomes. Common channels include shared memory, queues, ranking models, inventory, team records, notification policies, caches, and human workload.
Use four controls.
First, randomize at the smallest unit that contains the shared effect. If an agent rewrites workspace knowledge, user-level assignment is too narrow.
Second, partition variant-owned state where the product can support it. Use experiment-aware memory namespaces, cache keys, simulation accounts, and workflow queues. Do not partition authoritative business records merely to make the experiment convenient. Escalate the randomization unit instead.
Third, make overlapping experiments mutually exclusive when they change the same mechanism or primary outcome. LaunchDarkly's mutually exclusive experiment guidance describes layers that place each context in only one experiment, preventing interactions that make individual effects hard to interpret. Store the layer assignment in the experiment envelope and test it at execution time, not only in an analytics query.
Fourth, bind every consequential action to an execution receipt. The receipt should identify the run, release, variant, command, target, precondition, result, and authoritative postcondition. This lets the outcome join distinguish a model claim from a verified state change and catches treatment actions that escaped a shadow or approval boundary.
If treatment effects cannot be contained and the cohorts influence one another, stop calling the comparison an A/B test. Use a staged canary with operational guardrails, a region or workspace switchback with explicit carryover periods, or an annotated before-and-after release. The evidence may still be useful, but it does not support the same causal claim.
Join exposure to verified product outcomes
Build the metric from product events, not agent prose. The join should connect:
eligible unit
-> durable assignment
-> actual release exposure
-> run and tool receipts
-> verified immediate outcome
-> delayed business outcome
-> experiment decision
For a support-resolution agent, the primary outcome might be accepted resolutions per eligible ticket. Guardrails might include reopen rate, policy interventions, unverified completion claims, human correction time, p95 completion time, and cost per accepted resolution. Each metric needs a numerator, denominator, observation window, event owner, and late-arrival rule.
Avoid conditioning the denominator on treatment behavior. If treatment causes more runs to start, "successes per completed run" can improve while outcomes per eligible ticket decline. Preserve the eligibility population and report the assignment-to-exposure and exposure-to-outcome funnels separately.
Keep diagnostic metrics such as model latency, tool errors, and grader scores. They explain operational behavior, but they do not replace the primary product result. A production implementation request in Pydantic AI issue 6347 asks for asynchronous online evaluators, sampled scoring, persistent results, and monitoring. That is useful diagnostic infrastructure. The release decision should still name which scores are guardrails and which verified product event is primary.
Handle retries, resumes, and fallbacks explicitly
A stateful experiment needs deterministic rules for abnormal paths:
- Retry: Reuse the original envelope and release. Increment an attempt counter, but do not re-randomize.
- Resume: Load the envelope from durable workflow state before evaluating any new flags.
- Subagent: Propagate the parent experiment and release identity unless the protocol explicitly creates an isolated nested experiment.
- Provider failure: Record the failure. If a fallback changes the tested release, classify the run as degraded or noncompliant with treatment rather than pretending exposure stayed intact.
- Rollback: Stop new treatment assignments, preserve existing records, and apply a declared policy to active runs.
- Late callback: Resolve the original run and experiment before mutating state or emitting an outcome.
- Duplicate delivery: Deduplicate by stable action or outcome ID while retaining attempt evidence.
Never infer the variant from the current flag configuration when processing a callback. The configuration may have changed since assignment. Read the immutable envelope tied to the originating run.
Verify the experiment before trusting the lift
Run an experiment-integrity suite before evaluating the business result:
- Assign the same unit repeatedly and prove the variant is stable.
- Pause, resume, retry, and move the run between workers without changing the envelope.
- Force flag-provider failure and prove the recorded reason matches the executed release.
- Attempt two experiments in the same layer and prove the unit enters only one.
- Send treatment through shadow mode and prove no authoritative write occurs.
- Execute an approved write and prove its receipt joins to the correct exposure.
- Deliver a callback after rollback and prove it resolves the original run safely.
- Search for one assignment unit exposed to both variants and require zero results.
- Compare eligible, assigned, exposed, and outcome counts for unexplained loss.
- Recompute the result from immutable events using the documented observation window.
The experiment is invalid if mixed exposure, cross-variant state leakage, missing outcome joins, or denominator drift can explain the apparent effect. Report that directly. Extending a broken test only creates a larger biased dataset.
Decide with evidence the design can support
Promote only when the primary outcome, guardrails, and integrity checks answer the original release question. Roll back when a guardrail breaches its limit or when the treatment cannot execute reliably. Extend when the design is valid but the confidence is insufficient. Mark the result inconclusive when traffic is too low, the treatment changed mid-run, or interference cannot be bounded.
Start with one workflow and write its experiment envelope, exposure event, side-effect receipt, and outcome join before adding the flag. Then run the ten integrity tests above against retries, resumes, and real worker movement. If one durable workflow can receive two variants or one treatment effect can alter control state, fix the boundary before collecting another result.