Blog · August 24, 2026 · 11 min read

AI Agent Versioning for Safe, Reproducible Releases

AI agent versioning breaks down when the prompt has a revision number but the model, tools, policies, retrieval data, or runtime can change on their own. The team cannot reproduce an incident from one release record. A paused run can resume with behavior that did not exist when it started, leaving operators to guess which pieces belong in a rollback. AWS groups prompts, model configuration, tools, permissions, guardrails, and decision boundaries into one versioned behavioral configuration in its agent behavior versioning guidance. This guide applies that model to an existing product, including the release manifest, run pinning, compatibility rules, and verification.

Treat the release as one immutable unit

Give every behavior-changing bundle an immutable agent_release_id. The release record includes the deployment image and prompt revision, along with every other component needed to explain how the agent interpreted a request and what it was allowed to do.

Build and operate each release in this order:

1. Build a release manifest from immutable component versions.
2. Validate contracts and run the release against a fixed evaluation set.
3. Assign the release ID when each agent run starts.
4. Carry that ID through checkpoints, traces, approvals, tool calls, and audit records.
5. Route new runs to a new release gradually while existing runs follow an explicit pin or upgrade rule.
6. Roll back by changing routing for new work, not by editing an old release in place.
7. Retain the prior runtime and dependencies until no pinned run can require them.

Use the release ID as the join key across application logs, agent traces, evaluation results, deployment records, and side effects. The team can then compare complete releases instead of searching separate prompt, model, and service histories.

Why independent component changes break reproducibility

Source code is only one input to an agent run. System instructions shape the task. The model interprets them, tool schemas define available actions, policy limits those actions, and retrieved data supplies facts. A change in any layer can alter the execution path.

AWS Prescriptive Guidance warns that model and prompt changes can cause behavior drift, policy violations, degraded performance, and lost traceability when lifecycle controls are missing. It recommends version control for prompts and agent configuration so teams can roll back and preserve an audit history (prompt, agent, and model lifecycle management).

A mutable alias such as current creates the failure. A run starts with prompt 12 and tool schema 7, then pauses for approval. While it waits, a deployment moves current to prompt 13 and tool schema 8. If the resume path resolves that alias again, the second half runs on a configuration that was never evaluated with the first half.

Keep three decisions separate:

• Which release should receive a new run?
• Which release should resume an existing run?
• Is an existing checkpoint compatible with a different release?

A traffic router can answer the first question. The checkpoint and release policy must answer the other two. One routing decision cannot safely cover all three.

Build the AI agent versioning manifest

Create the manifest during the build, before the release receives traffic. Each referenced object should be immutable or addressable by an immutable digest. For example:

{
  "agent_release_id": "support-agent-2026-07-29.1",
  "runtime_build": "support-runtime-184",
  "prompt_revision": "triage-prompt-27",
  "model": {
    "provider": "example-provider",
    "model_id": "model-snapshot-2026-07"
  },
  "tools": {
    "search_customer": "3",
    "draft_reply": "5",
    "issue_refund": "2"
  },
  "policy_bundle": "support-policy-14",
  "retrieval_snapshot": "help-center-2026-07-28",
  "state_schema": "support-run-state-6",
  "eval_suite": "support-regression-31",
  "compatibility": {
    "resume_from": ["support-agent-2026-07-21.2"],
    "tool_contract": "support-tools-v4"
  }
}

The names above are examples. Each value must resolve to one historical artifact.

The manifest needs these component classes:

AWS recommends attaching evaluation criteria and results to the behavioral version record so comparisons remain reproducible. Its guidance also calls for automated, tested rollback instead of an improvised incident procedure (AWS Well-Architected).

Do not overwrite a manifest after release. If a component changes, create another release ID, even when the change looks minor. Mutable manifests make historical traces lie about what ran.

Pin each run before the first model call

Resolve the release once, when the product accepts the request. Store the resolved ID in the durable run record before calling a model or tool.

accept request
  -> resolve release for new traffic
  -> create run(agent_release_id, user_id, tenant_id, request_id)
  -> load exact manifest
  -> execute and checkpoint with the same release ID
  -> authorize each side effect against current product state

Release identity covers behavior, while authorization follows current product state. Check the user, tenant, resource, and approval immediately before a tool executes. A pinned release cannot preserve expired authority.

Carry the release ID on:

• checkpoints and resume tokens;
• traces and model-call spans;
• tool-call requests and typed outcomes;
• approval requests and decisions;
• idempotency records and side-effect receipts;
• evaluation results and incident reports.

If a trace contains only a prompt revision, the team still cannot tell which model, tools, or policies were active. If a tool receipt contains only a runtime build, the team cannot connect it to the behavior that selected the tool.

Choose a rule for in-flight runs

A running or paused workflow needs one of three policies. Make the choice by workflow type instead of leaving it to whichever worker receives the next message.

Temporal exposes a similar distinction between pinned and auto-upgrade behavior for workflow deployments. Its Worker Versioning documentation also defines how version behavior interacts with retries, child workflows, and continue-as-new operations. An application does not need Temporal to use the decision rule, but it does need an equally explicit contract.

Pinning is the safest default for stateful, action-taking runs. It has an operational cost: old workers, dependencies, prompt artifacts, and tool adapters must remain available. Put a maximum supported run duration in the product contract so old releases do not live forever.

Auto-upgrade should require a compatibility check, not just matching field names. Verify that:

• the state schema can be read without losing meaning;
• pending tool calls still reference the same effect and argument semantics;
• approval binds to the same action version;
• retry and idempotency behavior did not change;
• policy did not expand authority silently;
• the new release can interpret prior model output and intermediate state.

If any check fails, keep the run pinned or stop it with a visible typed outcome such as release_incompatible.

Release new behavior without mutating old behavior

OpenAI recommends covering prompt changes with representative tests and evaluation checks, then using Git history, review, release tags, feature flags, comparison, and rollback in the deployment process (OpenAI prompting guidance). Apply that discipline to the full manifest.

Move a release through this sequence in an existing product:

1. Resolve immutable component versions, then sign or store the manifest in a write-protected registry.
2. Compare tool inputs, outputs, effects, state schemas, and policy decisions with the previous release.
3. Run fixed regression cases and replayed production failures against the complete release.
4. Resume representative old checkpoints, cancel runs, retry tool outcomes, and roll routing back in a test environment.
5. Execute the new release on copied requests without customer-visible side effects where the product allows it.
6. Route a bounded group of new runs to the release while existing runs keep their assigned versions.
7. Increase new-run traffic only after product outcome, failure, latency, and cost checks pass.
8. Remove an old release only when no run is pinned to it and its retention obligations are complete.

Microsoft Foundry documents saved agent versions, comparison between versions, requests addressed to a specific version, and controlled deployment and rollback (agent development lifecycle). Those platform controls are useful, but the product still owns the manifest fields that live outside the platform, including service code, internal policy, retrieval assets, and side-effect contracts.

Work through an in-flight support case

Suppose release 42 can draft a refund and then wait for approval. Its issue_refund tool accepts case_id, amount, and currency. Release 43 replaces amount with a structured line-item allocation and tightens the approval policy.

A customer run starts on release 42 and pauses after an approver sees the old action. Deploying release 43 should not make that checkpoint call the new tool with old arguments. The product has two defensible choices:

• Keep the run on release 42, recheck the approver's current authority, and execute the exact approved action through the supported version-42 adapter.
• Mark release 42 revoked, cancel the pending action with release_revoked, and create a new release-43 proposal that requires approval again.

Silently translating the arguments is not defensible. The approved action and executed action would differ. Silently resuming with release 43 is also unsafe because the new policy was never evaluated against the saved state.

For new requests, route release 43 according to the canary plan. A rollback sends new traffic back to release 42. It does not rewrite release 43 records or pretend its runs belonged to release 42.

Handle failure and recovery

A version registry or artifact store can fail. Cache manifests by immutable ID, verify integrity before use, and fail closed when a requested release cannot be loaded. Do not resolve a missing historical ID to latest.

A model snapshot may be withdrawn by its provider. Treat that as a release availability incident. Stop new traffic, identify active pinned runs, and decide whether each can migrate, restart, or terminate. Record the decision rather than substituting another model under the same release ID.

A security issue may require immediate revocation. Add release state such as active, draining, retired, and revoked. Revocation should block new calls and define what happens to pending side effects. It should not delete the historical manifest needed for audit and incident analysis.

Rollback also has limits. It changes which release receives future work. It cannot undo side effects already committed by the bad release. Keep idempotency, compensation, and user-visible recovery as separate product controls.

Version retention has a cost. Old runtimes and adapters consume storage and operational attention. Set retention from the longest run duration, audit requirements, and incident window. Conductor's workflow documentation explicitly covers multiple workflow versions, restart behavior, and upgrading running workflows, which illustrates why release retirement must account for existing executions (Conductor workflow versioning).

Verify the release contract

Run deterministic checks before promotion:

• Every manifest field resolves to an immutable artifact.
• Two builds from the same inputs produce the same manifest contents.
• A run stores its release ID before any model or tool call.
• Every checkpoint, trace, approval, and tool receipt carries that ID.
• A paused run cannot resolve a mutable release alias on resume.
• An incompatible state or tool contract returns a typed failure.
• A revoked release cannot start a new call.
• Current authorization is checked even for a pinned release.
• Rollback changes new-run routing without changing historical manifests.
• Old release removal is blocked while active runs remain pinned.

Then test the negative cases: delete a manifest from a test registry, withdraw a model alias, change a tool schema, resume an old checkpoint, revoke a policy bundle, and interrupt the router during rollback. Each test should produce an explicit product outcome and an operator-visible event.

For an incident drill, select one historical run and reconstruct its complete release without reading deployment chat or asking which model was probably active. A release ID that cannot answer that question does not provide reproducibility.

Common mistakes

Versioning only the prompt

Prompt versioning helps review text changes, but it cannot explain a model upgrade, changed tool effect, new policy, or refreshed retrieval corpus. Keep prompt history and make it one field in the release manifest.

Using latest inside a persisted run

Aliases are appropriate for selecting a release for new traffic. Persisted runs need resolved IDs. Never store latest, current, or an environment name as the only behavior identity.

Equating rollback with recovery

Routing new work to an old release does not repair completed side effects or pending approvals. Define compensation, cancellation, and reapproval separately.

Retiring workers on deployment completion

Deployment completion says the new code is available. It does not prove that old runs are finished. Retirement needs an active-run query and a documented maximum lifetime.

Replaying with live external systems

Reproduction should not repeat customer-visible writes. Use recorded responses, sandbox tools, or dry-run adapters. A faithful behavior replay and a safe side-effect boundary are both required.

What to do next

Put AI agent versioning into practice on one production workflow. List every component that can change its behavior. Create a release manifest for the version running today, store its ID on one test run, and prove that the same ID appears in the checkpoint, trace, approval, and tool receipt. Then change one tool schema and verify that an old checkpoint stays pinned or fails with release_incompatible instead of resuming against the new contract.

References

• AWS Well-Architected: implement agent behavior versioning and rollback supports versioning the complete behavioral configuration, attaching evaluation records, staging changes, and testing rollback.
• AWS Prescriptive Guidance: prompt, agent, and model lifecycle management supports lifecycle controls for behavior drift, rollback, audit history, and reproducibility.
• OpenAI prompting guidance supports representative prompt tests, Git history, review, release tags, feature flags, comparison, and rollback.
• Temporal Worker Versioning supports explicit pinned and auto-upgrade behavior for running workflows and related inheritance semantics.
• Microsoft Foundry agent development lifecycle supports saved agent versions, comparisons, version-specific requests, controlled deployment, and rollback.
• Conductor workflow versioning supports operational handling for multiple workflow versions, restarts, and running-workflow upgrades.