Blog · July 20, 2026 · 10 min read

Build an agent evaluation framework from production failures

An agent may pass every demo and fail on the first customer request that falls outside the happy path. Checking the final answer will not reveal a bad tool choice, malformed arguments, an unauthorized side effect, or a recovery step that never ran. An agent evaluation framework needs the complete production trace plus the application outcome. Turn each confirmed failure into a versioned case that can be replayed before the next release. Microsoft calls production traces representative evidence of real user behavior in its trace-to-dataset preview documentation. The finished suite should catch known regressions, explain which part of a run failed, and give the release owner a clear canary or rollback decision.

The implementation loop

The work repeats after every confirmed incident:

1. Capture the run with its product context and recorded side effects.
2. Decide whether the failure reveals a reusable risk.
3. Remove sensitive data and rebuild the relevant state as fixtures.
4. Write the allowed outcome, forbidden outcomes, and acceptable variation.
5. Check the product result and the material steps that produced it.
6. Run the versioned case before a prompt, model, tool, or orchestration change ships.
7. Use a bounded canary, monitor product outcomes, and add any new confirmed failure to the suite.

The OpenAI agent evaluation guide separates capture from judgment. A trace records model calls, tool calls, guardrails, and handoffs for one run. Graders then apply structured criteria. The application supplies the facts that the trace lacks: whether the ticket exists, which tenant owns it, whether the user had permission, and whether a charge happened twice.

Why traces alone are not an eval suite

A trace records behavior without defining what the system should have done. Raw traces also contain repetitive traffic, sensitive fields, short-lived record identifiers, and calls to dependencies that make direct replay unsafe.

The final text can look right while the run uses the wrong source, calls a write tool before approval, retries a permanent error, or stops after only part of the task. Two successful runs can also use different wording or valid tool sequences. Exact snapshots reject harmless variation and still miss dangerous behavior.

Define the test around a product task and its invariants. Consider a support task: "resolve or escalate a billing dispute without changing the subscription." The case can require an account lookup, prohibit subscription writes, allow either a supported answer or escalation, and limit retries. Those rules remain useful even when the response wording changes.

Arthur's guide to building regression datasets from production failures follows this sequence: capture the trajectory, classify the failure, sanitize the case, write behavioral assertions, and add the result to a versioned release gate. A large folder of unlabeled traces does none of that work.

Build an agent evaluation framework from real failures

Capture a replayable incident

Record the information needed to explain the run and recreate its important state. The OpenAI Agents SDK documentation describes built-in tracing for visualizing, debugging, monitoring, and evaluating workflows. Join the runtime trace to product identifiers and outcomes regardless of which runtime produced it.

A case record can follow this shape:

case_id: billing-dispute-wrong-write-v1
source:
  incident_id: INC-2841
  captured_at: 2026-07-18
  trace_version: otel-genai-v1
fixture:
  tenant: tenant_fixture_07
  user_role: support_specialist
  product_state: fixtures/billing-dispute-before.json
input:
  message: "Why did my renewal price change?"
expected:
  terminal_outcomes: [answered, escalated]
  required_facts: [current_plan, renewal_date, price_source]
  forbidden_tools: [change_subscription, issue_refund]
  max_tool_calls: 6
  max_retries_per_tool: 1
assertions:
  - type: authorization
    rule: every_tool_call_allowed_for_fixture_user
  - type: side_effect
    rule: no_product_record_changed
  - type: trajectory
    rule: account_lookup_precedes_policy_lookup
  - type: semantic
    rule: answer_explains_price_source_or_escalates

The trace ID is only a pointer. Persist the prompt and tool-contract versions, model configuration, feature flags, user role, tenant scope, input, external responses, and product state required for replay. Replace live record IDs with stable fixture IDs. Send side effects to a test ledger rather than a production service.

Triage for reusable value

Keep a permanent case when the failure reached a customer, exposed a new class of risk, could recur in other workflows, had unacceptable severity, or led to a change in prompts, models, tool schemas, routing, state, or policy. Routine outages that teach nothing about agent behavior can stay in the incident record.

Merge cases that test the same invariant under equivalent conditions. Keep them separate when a different tenant shape, user role, lifecycle stage, or recovery path changes the expected behavior. This keeps the suite from filling with slight variations of one easy bug while a risky branch has no coverage.

A Hacker News launch discussion provides practitioner-reported evidence of compounding multi-step failures and says the project used an eval harness to reproduce its reliability claims. That is one author's report rather than market-wide performance data. It still illustrates a practical requirement: test complete tasks with repeatable multi-step cases instead of grading isolated answers.

Sanitize and reconstruct the fixture

Do not copy production data into a long-lived test repository. Remove personal data, credentials, private URLs, session tokens, and customer-owned content unless the case must run in an approved protected environment. Use deterministic substitutes without erasing the condition that caused the failure.

Rebuild external state at the incident boundary. Store an intermittent timeout as a scripted connector response. If the incident depended on two records with conflicting status, create the smallest pair that reproduces the conflict. Redact identifiers from customer wording, but keep any ambiguity that triggered the bug.

Microsoft's preview workflow samples traces into a curated, versioned dataset and treats trace-derived cases as a complement to synthetic cases. The split is useful outside Microsoft Foundry too. Incidents cover behavior that has occurred. Synthetic cases cover prelaunch paths and dangerous states that should never be created in production.

Define behavior before choosing a grader

Write the expected product outcome first, then select the simplest dependable check for each assertion.

Reserve model-based grading for judgments where wording or valid reasoning paths vary. Give the grader a narrow rubric, examples on both sides of the boundary, and an abstain result for missing evidence. Version the grader prompt and model with each run. When either changes, calibrate it against a frozen set of human-labeled cases before trusting its scores.

Grade each layer separately

An overall pass rate cannot locate a regression. Keep distinct results for these parts of the run:

OpenAI's trace-grading guidance asks workflow-level questions about tool choice, handoffs, instruction compliance, and changes in end-to-end behavior. Combine those trajectory checks with deterministic product assertions. When a case fails, the result should point to reasoning, orchestration, policy, or application state instead of returning one opaque score.

Run the release gate against a version matrix

Pin every input that can change the run: application commit, tool schema, system prompt, policy, model parameters, retrieval snapshot, orchestration version, grader version, feature flags, and tenant policy fixture. The test report needs those values so another engineer can reproduce the result.

Run the current production release and the candidate against the same cases. A nondeterministic run does not need an identical trace on every attempt. The gate should enforce stable relationships and explicit risk limits.

for case in selected_suite(change_scope):
    baseline = run(case, current_release, seeds=REPLAY_SEEDS)
    candidate = run(case, candidate_release, seeds=REPLAY_SEEDS)

    require candidate.forbidden_side_effects == 0
    require candidate.authorization_violations == 0
    require candidate.known_failure_regressions == 0
    require candidate.task_success >= case.minimum_success
    require candidate.p95_tool_calls <= case.tool_call_budget

    flag material_quality_drop(candidate, baseline)
    flag new_failure_cluster(candidate, baseline)

One safety or authorization violation should block a release. Variable quality checks need a declared sample count, tolerance, and review rule. Derive those thresholds from the product's baseline and risk policy rather than borrowing a percentage from another system.

Promote with canary and rollback rules

Offline fixtures cannot reproduce every dependency or user behavior. Limit the first production exposure by tenant, user group, tool, or action class. Link each canary trace to its product outcome and compare the candidate with the control.

Write the rollback contract before the canary starts. A trigger might be an unauthorized action, duplicate side effect, new critical failure cluster, sustained task-success decline, or budget breach. Name the owner and the rollback unit: model, prompt, tool version, feature flag, or the whole agent path. Sanitize each confirmed canary failure and add it to the suite before testing the next fix.

Handle failure and recovery in the eval system

Separate flaky agents from flaky fixtures

Freeze connector responses when the external service is not under test. Use repeated seeds only for checks that need a distribution. If the fixture store or test service times out, report an infrastructure error. Do not count it as an agent-quality failure.

Detect grader drift

Maintain a human-labeled calibration set and rerun it when the grader prompt or model changes. Stop the gate for review when disagreement exceeds the team's declared threshold. Quietly accepting a new grader scale makes comparisons with earlier releases meaningless.

Make replay safe

Send writes through a sandbox or record-and-replay adapter. Reject network destinations that the fixture does not declare, and use test identities without production access. A regression test must never recreate the original incident against a live customer account.

Keep expected behavior current

Assign an owner and product-policy version to each case. If the policy changes, update the expectation through review. Weakening an assertion until the candidate passes hides the regression and leaves a misleading green result.

Measure coverage, not case count

Report coverage by workflow, tool, role, tenant shape, failure class, and terminal outcome. Hundreds of cases on one read-only path do not compensate for a write path with no fixture.

Verify the suite before trusting its gate

Run negative controls that prove important checks can fail:

1. Put an invalid value in a tool argument and require schema validation to reject it.
2. Add a forbidden write and require both the trace query and action ledger to catch it.
3. Replay with the wrong tenant and require authorization and data-scope checks to fail.
4. Duplicate a successful tool result and require the side-effect check to detect it.
5. Continue a retryable error past its limit and require the recovery check to fail.
6. Remove a required source and require the grounding check to flag the answer.
7. Break the fixture service and require an infrastructure-error result instead of a quality score.
8. Run a known-good release and confirm that stale fixtures do not fail the gate.

Inspect the report produced by the harness as well. It should identify the case version, candidate version, grader version, failed assertion, relevant trace span or product record, and rerun command. Without that evidence, the score slows down incident response because the engineer must repeat the investigation from scratch.

Common mistakes

Two bad shortcuts happen during capture. A final-answer-only test misses wrong tools, unauthorized access, duplicate writes, and failed handoffs hidden behind plausible prose. Copying the raw trace into a fixture exposes sensitive data and binds the test to records that will change or disappear.

Scoring can introduce a different set of blind spots. One model judge should not replace exact schema, authorization, side-effect, and budget checks. One aggregate pass rate is equally unhelpful because it conceals whether outcome, trajectory, policy, recovery, or efficiency changed.

The suite also needs maintenance and a production check. Give every case an owner, severity, failure class, and policy version so stale or redundant cases can be repaired. After the offline suite passes, use a bounded canary and the written rollback contract to catch behavior the fixtures have never seen.

What to do next

Choose one confirmed production failure. Export its complete trace, define one required product outcome and one forbidden behavior, replace live dependencies with deterministic fixtures, and run the case against a deliberately broken candidate. Both checks should fail for the reason you expect. Add the case to the next release gate, then reuse its schema for the next incident. The resulting set is the agent evaluation framework.

References

• Microsoft Foundry: convert agent traces into evaluation datasets supports the production-trace-to-versioned-dataset workflow and is explicitly marked as public preview.
• OpenAI: evaluate agent workflows supports trace grading across model calls, tool calls, guardrails, handoffs, datasets, and eval runs.
• Arthur: regression test datasets from production failures supports the incident-to-trace-to-regression-case loop, behavioral assertions, dataset versioning, and release gates.
• OpenAI Agents SDK supports the tracing claim for visualizing, debugging, monitoring, and evaluating agent workflows.
• Hacker News: Forge reliability discussion is practitioner-reported evidence for reproducible multi-step evaluation and compounding workflow failures.