Blog · July 20, 2026 · 10 min read

Stateful AI agents on Kubernetes: survive replica changes

Stateful AI agents on Kubernetes can look healthy in single-pod tests, then lose their place when a Service sends the next request to another replica. The second pod cannot see a session, checkpoint, pending approval, or stream cursor stored in the first pod's memory. A Stack Overflow question describes this MCP failure: replica A initializes the session, replica B receives the later tool request, and the request fails. Increasing pod size does not address state ownership. Store durable run state outside the replicas, make resume idempotent, and test each workflow while forcing it to change pods.

The architecture in one view

A pod should execute work without owning the run. Stable identifiers on each request let any healthy replica load the same state and continue.

client, agent, or approval callback
                 |
                 v
       Kubernetes Service / Ingress
                 |
        +--------+--------+
        |                 |
     replica A         replica B
        |                 |
        +--------+--------+
                 |
     durable control-plane state
       - session registry
       - run checkpoints
       - stream cursors
       - leases and versions
       - action/idempotency records
                 |
        existing product services

Split state by purpose instead of dumping every object into one cache:

Components can use different databases. They still need clear ownership and durable identifiers, plus atomic transitions wherever two records must agree.

Why a replica switch breaks stateful agent work

Kubernetes Services distribute traffic across eligible endpoints. According to the Kubernetes session affinity documentation, the default Service session affinity is None. ClientIP affinity can keep connections from one client IP on the same pod for a configurable period. It can improve locality, but it cannot preserve state after that pod disappears.

Several ordinary events can still move a run:

• a rolling deployment terminates the original pod;
• autoscaling removes it;
• a readiness failure takes it out of rotation;
• an ingress or proxy changes the source IP seen by the Service;
• an approval callback arrives through a different path;
• a reconnect opens a new connection after the affinity window;
• two requests for one run arrive concurrently.

Protocol sessions make the failure easy to reproduce. The MCP Streamable HTTP specification allows a server to assign Mcp-Session-Id during initialization and requires the client to send it on later HTTP requests. It also defines resumable SSE streams and event IDs. When the session registry and stream cursor live only in process memory, another pod receives a valid identifier with no state behind it.

Agent frameworks add another state layer. LangGraph persistence separates thread checkpoints from cross-thread stores and uses checkpointers for continuity, interruption recovery, and fault tolerance. A memory-only checkpointer loses those properties when the process restarts. Every replica that may receive a thread ID needs access to the production checkpointer.

Separate routing state from durable task state

Start by naming the identifiers instead of passing one all-purpose token:

1. session_id identifies a protocol session, if the transport needs one.
2. run_id identifies one execution of a product workflow.
3. thread_id identifies resumable agent state when the runtime uses threads.
4. logical_action_id identifies one intended external side effect across retries.
5. event_cursor identifies the last stream event delivered to a client.

Keep these identifiers separate because their lifetimes and security rules differ. The MCP transport's security warning says servers should implement proper authentication for all connections, while Mcp-Session-Id links the interactions in one protocol session. Authenticate every request, then verify that the authenticated actor and tenant may access the referenced session and run.

The A2A specification treats long-running work as asynchronous tasks that may continue after the initial response. Clients can retrieve or cancel a task, subscribe to updates, and group related interactions with a context ID. Across multiple replicas, those operations need a task ID that resolves to durable state rather than one worker's heap.

Implement stateful AI agents on Kubernetes

Reload state inside each request handler

A handler should need only trusted request context and stable IDs. It loads the current record, validates access, claims one transition, performs bounded work, and persists the result before acknowledging success.

Keep the authoritative copy of each pending tool call, approval, retry counter, and cancellation flag in shared storage. A local object can cache that record for the life of a request.

Fence each checkpoint claim

Two replicas can try to resume the same run at once. Give each transition a monotonically increasing version or fencing token so only one replica can advance it.

resume(run_id, expected_version, request_id):
  run = load_for_update(run_id)

  if request_id in run.completed_requests:
      return recorded_result(request_id)

  if run.version != expected_version:
      return conflict_with_current_state(run)

  if run.status not in resumable_states:
      return current_terminal_or_blocked_result(run)

  claim = acquire_lease(run_id, next_fencing_token())
  checkpoint = load_checkpoint(run.thread_id, run.checkpoint_version)

  result = execute_one_bounded_step(checkpoint, claim.token)

  transaction:
      assert_lease_is_current(claim)
      save_checkpoint(result.next_checkpoint)
      append_events(result.events)
      update_run(version = run.version + 1, status = result.status)
      record_request_result(request_id, result.public_result)

  return result.public_result

A lease limits concurrent work. Its fencing token blocks an old worker from writing after the lease expires and a new worker takes over. Keep claims short. A model or tool call may outlive the original lease, so renew only while the worker is healthy and reject writes that carry stale tokens.

Give external actions stable identities

A database transaction around a checkpoint cannot include an external payment, email provider, or ticketing API. Persist a logical action record before dispatch. Attach an idempotency key when the provider supports one. If the call may have completed but its response was lost, record unknown.

On resume, inspect the action record before asking the model what to do. If the action succeeded, return its recorded result. If the outcome is unknown, reconcile with the external system or send the run to a person. Do not blindly repeat a write because a new replica cannot see the old process's response.

Store stream progress outside the connection

An open SSE connection ends with its pod. Recovery therefore needs an event log rather than a way to preserve the socket. Give events stable IDs, persist them before delivery, and let a reconnecting client continue from its last cursor when the protocol supports it. The MCP transport specification describes Last-Event-ID resumption and per-stream event IDs. A shared event store supplies the cross-pod history behind that wire contract.

Cancellation follows the same rule. Store a cancellation request durably and make workers check it between bounded steps and before every side effect. Closing one connection is not proof that all work stopped.

Drain a replica before it terminates

When a pod receives a termination signal, it should stop claiming new work, fail readiness, finish or checkpoint its current bounded step, release leases, and close streams with a recoverable final cursor. Set the termination grace period from measured checkpoint and connector behavior, not from model response averages alone.

A new deployment may also change checkpoint schemas, prompts, or tool contracts. Persist the versions needed to interpret an existing checkpoint. Either keep the new code backward compatible for the recovery window or migrate stored state deliberately before removing the old reader.

Choose sticky sessions or shared state

Sticky routing is a tactic, not the default architecture for durable work.

Use affinity to improve locality after cross-replica correctness works. Never use it as the only protection for approvals, side effects, checkpoints, or task history.

Handle failure and recovery

Decide how each failure will recover before adding replicas:

• If a pod dies before checkpointing, another worker restarts from the last durable checkpoint. The bounded step may repeat, so external writes need action records.
• If it dies after an external write, load the action record and reconcile an unknown outcome instead of assuming failure.
• When two callbacks resume one approval, accept one versioned transition and return its recorded result to the duplicate caller.
• If the shared store is unavailable, stop advancing runs. Falling back to local memory creates split state that is harder to repair than a visible outage.
• After a stream disconnects, reconnect with the last durable cursor and replay only events for that stream and authorized run.
• When a deployment changes checkpoint format, route old state to a compatible reader or tested migration. Unknown state should not be deserialized and continued optimistically.

Expose these outcomes in product and operator views. A customer should see waiting for approval, resuming, temporarily unavailable, or needs reconciliation, not a spinner tied to one pod's lifetime.

Verify replica-safe behavior

Most integration tests keep hitting the same pod. Add tests that force the run to move:

1. Initialize a session through replica A, then send discovery and tool requests directly to replica B.
2. Pause a run for approval, delete its original pod, and submit approval through a new pod.
3. Kill a worker immediately before and after an external write. Prove the logical action executes at most once.
4. Send the same resume request concurrently to two replicas. Prove one transition wins and both callers receive a consistent result.
5. Disconnect a stream after event N, reconnect elsewhere with the cursor, and prove there are no gaps or cross-stream events.
6. Scale from one replica to several during active runs, then scale down the original owners.
7. Block the shared state store and prove pods fail closed instead of creating local sessions.
8. Roll out a checkpoint schema change while old runs remain paused.

Useful production invariants include:

one active fencing token per run
one durable result per logical_action_id
checkpoint version increases monotonically
terminal runs never return to a working state
all state reads enforce actor and tenant scope
replayed stream events preserve order within one stream

Track conflicts, lease takeovers, replay counts, unknown side effects, checkpoint latency, stream resume failures, and session-not-found responses by deployment version. A rising conflict rate may reveal duplicate callbacks or a missing routing key before users report broken work.

Common mistakes

Session affinity is often mistaken for persistence. It reduces some replica switches, but a terminated pod still takes its local checkpoints with it.

A shared cache without transaction rules only moves the race. Replicas can still overwrite each other, repeat a side effect, or advance the same checkpoint. Version checks and leases decide who may write; action identities stop a retry from becoming a second side effect.

The protocol session is one part of the workflow, not its owner. Session negotiation belongs to the transport, execution state to the agent runtime, records to product services, and external actions to their action ledger.

A valid checkpoint can still contain stale intent. Permissions, prices, records, or policies may change while a run waits for approval. Read current product state and check authorization again before a write.

Failover tests that restart a pod after a completed request miss the risky windows. Kill it during a model call, before checkpoint commit, after external dispatch, while approval is pending, and midway through a stream.

What to do next

Pick one resumable workflow and list every piece of state it needs. Mark each item pod-local cache, shared durable state, or external system of record. Then run the eight replica-switch tests above with session affinity disabled. Stateful AI agents on Kubernetes are replica-safe only when those tests no longer depend on the original pod returning.

References

• Stack Overflow: MCP sessions across Kubernetes replicas - Practitioner report of an MCP session initialized on one replica and rejected by another.
• Kubernetes virtual IPs and Service proxies - Official behavior for Service endpoint selection, ClientIP session affinity, and stickiness timeout.
• Model Context Protocol Streamable HTTP transport - Official session ID, resumable stream, event ID, and session security requirements.
• LangGraph persistence - Official distinction between thread checkpoints and cross-thread stores for continuation and fault tolerance.
• A2A Protocol specification - Official task, context, polling, cancellation, streaming subscription, and idempotency semantics for asynchronous agent work.