Blog · · 10 min read

LLM failover for stateful AI agents without replaying side effects

LLM failover for stateful AI agents without replaying side effects

For a stateless chat request, LLM failover can improve availability. The same tactic can make a stateful AI agent repeat a refund, lose a tool result, exceed the run deadline, or send the next model a context it cannot process. Recovery is hardest when the provider fails after the agent has streamed text, called tools, or changed product state. A safe design classifies the failure, proves whether replay is allowed, checks that the substitute model can satisfy the task, and carries one budget and audit trail across every attempt.

This guide defines that contract so a failed route either preserves the run's semantics or returns an explicit degraded outcome.

Why ordinary LLM failover chains break stateful agents

Gateway products already offer useful reliability mechanics. LiteLLM documents retries, cooldowns, model fallbacks, context-window fallbacks, and provider-level failover. Microsoft's multi-backend gateway guidance covers regional and subscription failover, quota handling, circuit breakers, routing, centralized policy, and observability. Use those mechanisms, but do not mistake route selection for safe recovery.

A stateful run crosses boundaries that a model gateway cannot see:

  • A tool may have committed a write even though its response never reached the orchestrator.
  • Partial model output may already be visible to the user.
  • A hosted tool may keep provider-side state that cannot move to another vendor.
  • A fallback may accept the API call while lacking the required context size, tool behavior, schema fidelity, safety controls, region, or quality floor.
  • An SDK, gateway, worker, and workflow engine may each retry the same failure and multiply cost and load.

Current issue reports show these failures in working implementations. One OpenAI Agents SDK issue reports an overload signal on a WebSocket path bypassing the retry policy used for HTTP responses. The author also warns that replaying a long-running agent run can repeat completed tools and subagent work. A LiteLLM context-window issue reports a payload being sent to a smaller fallback that could not accept it. Another LiteLLM issue reports a blocked primary model group preventing a configured healthy fallback from being attempted.

In each case, the run and routing layer disagree about what work can continue.

Classify failures before deciding to retry or fail over

Do not let raw provider exceptions drive policy. Normalize them at one boundary into product-owned failure classes. Provider taxonomies differ: OpenAI's error guide separates request errors, rate limits, server errors, overload, and connection failures, while Anthropic's API error documentation identifies conditions such as rate limiting, overload, request-size errors, and network errors, and includes request identifiers for support and tracing.

Store the category with the facts the recovery policy needs:

failure:
  class: provider_overloaded
  phase: model_stream
  retryable: true
  fallback_eligible: true
  retry_after_ms: 1500
  provider_request_id: req_7f2a
  response_started: true
  provider_state_created: false
  outcome: no_model_result
  raw_error_ref: encrypted://run-errors/err_941

Keep retryable and fallback_eligible separate. A local validation error is not fixed by another provider. A quota error might justify a different account or region but not a smaller model. A malformed tool schema may fail everywhere. An unknown network outcome may be unsafe to replay until product state is reconciled.

Treat unknown errors as terminal by default for side-effecting work. You can loosen that rule after collecting evidence and adding a specific classifier. A current LiteLLM proposal for structured provider errors and route health addresses the problem of downstream systems maintaining divergent regexes for category, retryability, retry delay, fallback target, and cooldown state.

Make replay safety a persisted run decision

The retry decision belongs to the run controller because only that component has the tool ledger and product receipts. Store progress after each externally visible transition. At minimum, keep:

  • The run ID, release ID, user and tenant scope, deadline, and remaining attempt budget.
  • The model attempt ID, provider request ID, input manifest, and output state.
  • Every proposed tool call with a stable action ID.
  • Authorization and approval decisions tied to the action.
  • Tool receipts that distinguish accepted, committed, rejected, and unknown outcomes.
  • Product postcondition reads used to verify a write.
  • Any text or artifact already delivered to the user.

Then apply explicit rules:

  1. A failure before any output or tool proposal is usually replayable if the request itself is valid.
  2. A failure after partial text is replayable only if the partial response can be discarded and was not consumed as product input.
  3. A failure after a read-only tool may be replayable from the persisted result rather than calling the tool again.
  4. A failure after a verified idempotent write may resume from its receipt, never by asking the model to decide the same write again.
  5. A failure after an unverified or non-idempotent write has an unknown outcome. Reconcile against the product system before any retry.
  6. A failure inside provider-managed state that cannot be exported is not portable. End the run or restart an explicitly safe task from a known checkpoint.

Ask which effects are authoritative and which next transition is still permitted. A missing final message does not answer either question.

Match the fallback to the task, not just the API

Before sending work to a substitute model, intersect task requirements with a versioned capability record. Do not assume that function calling in two APIs means identical behavior.

model_capability:
  model_release: provider-b/model-2026-07
  max_input_tokens: 128000
  tool_calling: required_schema
  parallel_tools: false
  structured_output: strict_json_schema
  image_input: true
  hosted_tools: []
  safety_policy: product-policy-v12
  allowed_regions: [eu-west]
  eval_profile: support-refunds-v8
  minimum_pass_rate: 0.97
  p95_latency_ms: 4200
  cost_class: medium

Build task requirements when the run starts. Include the actual input size after rendering, required modalities, tool set, schema mode, parallel-call behavior, policy version, data residency, maximum latency, cost ceiling, and minimum eval profile. Recompute requirements if the task changes after a tool result.

A fallback is eligible only when it satisfies every hard requirement. Latency, cost, and quality can be ordered preferences, but they need declared floors. If the task requires strict structured output and the substitute only offers best-effort JSON, the route is incompatible. If the context does not fit, do not silently truncate it. If a provider lacks a hosted tool used earlier in the run, do not invent a local equivalent during an outage.

Preflight this intersection before production traffic. The routing layer should be able to answer, for each task class, which models remain valid after any one route is removed. A configuration containing five fallback names may still have zero usable fallbacks for the current task.

Give one component ownership of retries and budgets

Retries become dangerous when every layer believes it owns reliability. Pick one component, usually the run controller or gateway, to allocate attempts. Lower layers may perform a narrowly bounded transport retry only when the request has not been accepted and the operation is known to be safe.

Carry one budget across providers:

def recover_model_attempt(run, failure):
    decision = classify_and_reconcile(run, failure)
    if not decision.replay_allowed:
        return finish_unknown_or_failed(run, decision)

    candidate = choose_compatible_model(
        requirements=run.current_requirements,
        excluded_routes=run.failed_routes,
        deadline=run.deadline,
        remaining_attempts=run.remaining_attempts,
        remaining_cost=run.remaining_cost,
    )
    if candidate is None:
        return finish_degraded_or_unavailable(run)

    run.reserve_attempt(candidate)
    return dispatch_from_checkpoint(
        run=run,
        model=candidate,
        tool_receipts=run.verified_tool_receipts,
    )

reserve_attempt must be atomic. Two workers recovering the same checkpoint must not launch two fallback calls. The reservation should record the chosen route, decrement the shared attempt and cost budgets, and bind the dispatch to the current run version.

Circuit breakers still matter. Microsoft's gateway guidance recommends using backend health, throttling information, and circuit-breaking patterns when routing across deployments. The circuit state should inform candidate selection, not erase the run's replay and compatibility checks.

Worked example: a refund agent loses its model stream

Consider a support agent handling an approved refund. It has already called create_refund with action ID refund/order-1842/v1. The payment service committed the refund and returned receipt rf_901, but the model stream disconnected before the agent produced its final response.

The wrong recovery path restarts the whole prompt on a fallback model. That model sees the approval and asks to create the refund again. Whether the second call duplicates the charge reversal now depends on a tool implementation that the router never checked.

Recover it as follows:

  1. Classify the disconnect as a network failure during model_stream.
  2. Read the tool ledger and find a committed receipt for refund/order-1842/v1.
  3. Verify the refund against the payment service using the receipt or stable action ID.
  4. Mark the tool transition complete and prohibit another create_refund dispatch for that action ID.
  5. Build new requirements for the remaining task: summarize the verified result in the user's language using the approved disclosure policy. No write tool is required.
  6. Select a compatible model within the remaining deadline, cost, locale, and policy budget.
  7. Give the fallback the verified receipt summary, not the original opportunity to decide whether to refund.
  8. Record that the final explanation came from a fallback route while the product effect came from the original verified tool call.

If verification cannot determine whether the refund committed, stop. Show the operator or user an outcome_unknown state and start reconciliation. Keep the run there until reconciliation rules out a duplicate financial action.

Handle partial output and graceful degradation explicitly

Once users can see streamed tokens, recovery must account for what they already read. A second model must not continue the sentence as if it generated the first part. Buffer high-risk responses until a safe boundary, or terminate the visible response and label the retry. Never merge text from two models without a product-specific protocol and tests.

Define degraded outcomes by task class. Examples include:

  • read_only_fallback: the agent may answer from already verified context but cannot call tools.
  • manual_completion_required: the run preserves receipts and routes the next decision to a person.
  • result_delayed: the run waits for the primary capability because no substitute meets policy or quality requirements.
  • outcome_unknown: an external effect may have happened and must be reconciled.
  • unavailable: no valid route remains within the deadline and budget.

Do not hide these states behind a generic success response. A practitioner who built a reliability layer reported provider outages, partial degradation, silent retry cost, and hard vendor coupling. That is one author's experience, not a prevalence measurement, but it captures why fallback state needs to be visible in traces, costs, and product outcomes.

Test the contract with boundary failures

A healthy-provider smoke test proves very little. Build a deterministic suite around the transitions where ownership changes:

  • Fail before the first token, after the first token, after a tool proposal, after tool acceptance, and after a verified commit.
  • Return 429, overload, server, request-size, schema, authentication, and unknown network errors through every transport your SDK supports.
  • Block the primary route before routing and confirm that eligible fallbacks are still evaluated.
  • Give the fallback a smaller context window, missing modality, weaker schema mode, wrong region, and failed task eval. Confirm rejection before dispatch.
  • Run two recovery workers against the same checkpoint and prove that only one reserves the fallback attempt.
  • Put retries in the SDK and gateway during a test, then prove the shared budget still caps total attempts.
  • Exhaust every compatible route and verify an explicit degraded or unavailable terminal state.
  • Commit a side effect but drop its response. Confirm reconciliation occurs and the action is not repeated.
  • Inspect the trace and cost ledger to confirm one run ID links every provider attempt and tool receipt.

Record these tests against the agent release. A model, gateway, prompt, tool schema, policy, or SDK upgrade can change compatibility even when the fallback list stays the same.

Common mistakes to remove before rollout

Price ordering and a list of provider names cover route selection only. Remove these failure patterns before calling the configuration resilient:

  • Matching on status code alone while ignoring transport, phase, and outcome certainty.
  • Retrying an entire run because a final model response is missing.
  • Re-executing read tools instead of carrying forward their persisted results.
  • Treating a nominally idempotent write as verified without an authoritative receipt or postcondition read.
  • Using token count as the only compatibility check.
  • Letting each layer reset attempts, deadlines, or cost budgets.
  • Switching policy, region, or quality silently during an outage.
  • Reporting success because the fallback produced fluent text.

Choose one side-effecting production workflow, draw its model and tool transitions, and inject a provider failure after each transition. Do not enable automatic LLM failover for that workflow until every test produces either a verified continuation or a truthful terminal state.

References

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us