Blog · · 13 min read

How to build an AI agent incident response control plane

How to build an AI agent incident response control plane

An AI agent incident response plan fails if its kill switch only changes a database flag. New runs may stop while in-flight tool calls, queued jobs, network requests, delegated agents, and reusable credentials keep producing side effects. Operators see a red banner and assume containment succeeded. It did not. This guide defines a control plane that closes every execution path, returns evidence from each enforcement point, preserves the investigation record, and blocks recovery until product state is safe.

A longer runbook will not fix this. Security and platform teams need an executable containment contract they can test before an incident.

Diagnose the real failure: state is not enforcement

A control record can say stopped without changing runtime behavior. An author report in ZeroClaw issue #9390 described exactly that class of defect: emergency-stop state could be persisted and displayed, but runtime code did not read it. A follow-up request in issue #9802 called out missing enforcement for in-flight and network operations. Both reports describe their own project. They still expose a design flaw that every agent runtime should test.

An action-taking agent can enter product workflows through several paths. Containment must cover all of them:

  1. Admission of new user, API, event, schedule, and retry-triggered runs.
  2. Queued work that has not started.
  3. Active model calls and orchestration loops.
  4. Tool dispatch before an effect begins.
  5. Network egress after a tool process has started.
  6. Credentials and grants that remain valid outside the orchestrator.
  7. Child agents and remote delegated tasks.
  8. Memory writes, artifact publication, and delayed callbacks.

The OWASP Agentic Skills incident response playbook separates detection, containment, eradication, recovery, and post-incident work. Its containment guidance includes disabling compromised capabilities, revoking credentials, blocking network access, preserving evidence, and examining memory. A product control plane must issue commands for those actions and collect an acknowledgement from every enforcement point.

Define containment as a set of invariants

Start with properties that must become true, regardless of framework or model provider. For a hard stop at scope tenant/acme, useful invariants are:

  • No new run can enter the affected scope.
  • No queued item can move to running.
  • Every active run receives cancellation and reaches a terminal or quarantined state within a declared deadline.
  • No new side-effecting tool dispatch can pass the policy gate.
  • Existing network sessions cannot open an unapproved request.
  • Credentials for the affected agent, tenant, or integration are revoked or disabled.
  • Delegated children receive the same or narrower stop scope.
  • No model-proposed memory write or artifact becomes trusted product state.
  • Evidence remains readable to the incident team after execution access is removed.
  • Recovery cannot begin until each required enforcement point reports its result.

Use a command, acknowledgement, and recovery-gate contract

The control plane should create an immutable command with a unique incident ID and monotonically increasing epoch. Every enforcement point records whether it applied the command, rejected it, timed out, or found no matching work.

containment_command:
  incident_id: inc-2026-0819-0042
  epoch: 7
  mode: hard_stop
  scope:
    tenant_ids: [tenant-acme]
    agent_release_ids: [refund-agent-v18]
  actions:
    deny_new_runs: true
    cancel_active_runs: true
    drain_queued_runs: false
    deny_tool_dispatch: true
    block_egress: true
    revoke_grants: true
    quarantine_memory_writes: true
  deadline_at: 2026-08-19T12:05:00Z
  issued_by: security-incident-commander
  reason_code: suspected-credential-exfiltration

enforcement_ack:
  incident_id: inc-2026-0819-0042
  epoch: 7
  component: tool-egress-gateway
  scope_version: tenant-acme/refund-agent-v18
  result: applied
  observed_active: 3
  interrupted: 2
  unknown_outcome: 1
  evidence_refs: [receipt-901, receipt-902, receipt-903]
  acknowledged_at: 2026-08-19T12:01:17Z

The unknown_outcome count matters. A network connection can close after a remote service accepted a request but before the agent received a response. Marking that call cancelled would be false. Route unknown outcomes to reconciliation using an idempotency key or authoritative product read before anyone retries them.

Implement the control plane in six steps

1. Inventory every execution path

Build the inventory from production architecture, not from the agent framework's main run() method. Include API handlers, queue consumers, schedulers, webhooks, event subscriptions, retry workers, browser and code sandboxes, MCP servers, remote agent protocols, and callback endpoints. For each path, name its admission gate, cancellation mechanism, credential source, egress boundary, state store, and owner.

The Microsoft Agent Governance Toolkit incident workflow provides severity levels, containment actions, communication steps, evidence preservation, recovery checks, and post-incident review. Use that process around the inventory, but keep the inventory specific to your product. A generic instruction to "disable the agent" is not enough when five services can start work.

Record enough detail to test each path:

path: scheduled-renewal-agent
admission: scheduler policy gate
active-work control: worker cancellation topic
side-effect gate: billing-tool authorization middleware
egress: integration proxy
credential: tenant grant broker
durable state: agent_runs + tool_receipts
owner: billing-platform
containment test: IR-SCHED-04

Reject a release if it adds an execution path without a containment owner and test.

2. Put the stop check at admission and dispatch

A policy check at run admission stops new work. A second check immediately before every side-effecting tool dispatch catches a run that started before the incident. Both checks must compare the current containment epoch, not a value cached when the run began.

def dispatch_tool(run, tool_call):
    policy = containment_policy.read_current(
        tenant_id=run.tenant_id,
        agent_release_id=run.agent_release_id,
    )

    if policy.epoch > run.last_seen_containment_epoch:
        run.last_seen_containment_epoch = policy.epoch

    if policy.denies(run=run, tool=tool_call.tool_name):
        receipts.record_blocked(run, tool_call, policy)
        raise ContainmentActive(policy.incident_id, policy.epoch)

    return authorized_tool_gateway.execute(run, tool_call)

Do not rely on a prompt telling the model to stop. The OWASP AI Agent Security Cheat Sheet treats tool access, least privilege, validation, monitoring, memory safety, and emergency controls as system boundaries. The model may be the compromised component. Enforcement therefore belongs outside model context.

3. Separate cooperative cancellation from hard containment

Cooperative cancellation asks a run to stop at a safe checkpoint. It is appropriate for ordinary user cancellation. Incident containment needs a deadline and an escalation path.

For each runtime, define:

  • How cancellation reaches active workers.
  • Which operations are interruptible.
  • Which operations must finish to avoid corrupting state.
  • How long cooperative shutdown may take.
  • What hard action follows the deadline.
  • How the runtime records work that ended with an unknown outcome.

A Pydantic AI cancellation proposal distinguishes durable, level-triggered cancellation from a transient event. This matters in a distributed runtime: a worker that reconnects or resumes after the stop command must still observe the stopped state. A one-time message is insufficient because offline consumers can miss it.

Hard containment might terminate a sandbox, remove a worker lease, deny every tool at the gateway, or block the scope at an egress proxy. Choose the action by effect risk. Killing a process cannot retract an already accepted payment, so pair interruption with product reconciliation.

4. Revoke authority and egress independently

Stopping the orchestrator does not invalidate a credential copied into another process. Revoke external-service grants, workload tokens, browser sessions, signed tool capabilities, and remote-agent credentials at their issuing boundary. Apply network policy as a separate control so a leaked credential cannot leave through the affected runtime.

The COMPEL containment guide distinguishes soft pause, hard stop, and containment actions such as revoking credentials, disabling tool access, blocking egress, isolating memory, and preserving evidence. Separate controls give the response team another path when one component fails. Egress denial can limit damage while a credential provider is unavailable, and tool authorization can reject dispatch when the network control fails.

5. Propagate the incident through queues and delegation

A stop command must follow the same identity dimensions as work: tenant, user, agent release, run, tool, and delegated child. Publish the current epoch to workers, but also store it in a strongly consistent or appropriately monotonic source that reconnecting workers read before claiming work.

For queued items, choose one explicit policy:

  • hold: keep the item but deny execution until incident resolution.
  • quarantine: move it to an incident queue for manual inspection.
  • expire: terminate it and require a new user action after recovery.
  • drain_read_only: allow only declared evidence-collection steps.

Do not silently replay quarantined work after recovery. The original authorization, data, and user intent may no longer be valid.

For delegated agents, send a signed cancellation or containment message, revoke the child's grant, and mark the parent incomplete until the child acknowledges. If a remote child cannot prove cancellation, record that scope as unresolved and escalate. The product must not report fleet containment while an unconfirmed delegate remains authorized.

6. Quarantine mutable state and preserve evidence

Containment can destroy the evidence needed to understand the incident. Preserve run events, tool inputs and outputs after redaction, execution receipts, policy decisions, credential identifiers, network destinations, model and agent release identifiers, memory reads and writes, and operator actions in an access-controlled incident store.

Do not continue normal memory learning during an incident. Route proposed writes to quarantine. Keep the original source and provenance so investigators can distinguish malicious input from a later model summary. Do not delete suspicious memory before capturing its affected scope and retrieval history.

A TrustRail proposal for an immediate agent kill switch calls for active-run tracking, stop signals, state transition, and operator-visible termination evidence. Operators need the same answers in any runtime: which runs stopped, which never started, and which remain uncertain.

Run containment as a two-phase transaction

Do not wait for every subsystem serially. Issue commands in parallel where independence is safe, then evaluate the complete acknowledgement set against the incident policy.

A practical sequence is:

  1. Freeze new admissions and side-effect dispatch first.
  2. Publish durable cancellation and mark affected runs containment_pending.
  3. Block scoped egress and revoke grants.
  4. Quarantine queued work, memory writes, artifacts, and callbacks.
  5. Collect acknowledgements with a fixed deadline.
  6. Escalate missing or failed acknowledgements to stronger controls.
  7. Reconcile unknown side effects against authoritative systems.
  8. Declare containment only when all required invariants pass.

The status page should distinguish commanded, partially_contained, contained, and containment_failed. Avoid a single stopped boolean. Partial containment is actionable information, while a green aggregate can hide one service that never received the command.

Example: a refund agent sends data to an unknown host

Suppose an alert shows the refund agent sending customer data to a destination outside the integration allowlist. The incident commander scopes the first hard stop to tenant-acme and release refund-agent-v18 rather than stopping every customer.

The admission service denies new runs. The tool gateway blocks new refund, email, and file-export calls. The worker service receives cancellation for four active runs. Two stop before their next tool call. One sandbox is terminated after its cooperative deadline. One external refund request has an unknown outcome, so the billing team reads the payment processor by the request's idempotency key instead of retrying it. The egress proxy blocks the unknown host. The credential broker revokes the tenant's integration grant. A delegated research agent fails to acknowledge, so its child credential is revoked and its network scope is denied. Proposed memory writes enter quarantine.

Containment is complete only after the control plane records all of those results. Eradication can then remove the malicious instruction or compromised capability, rotate affected credentials, repair policy, and build a regression case.

Gate recovery with evidence, not elapsed time

Recovery is a new authorization decision. It is not the automatic inverse of the stop command. The OWASP playbook recommends validating clean state, rotating credentials, restoring from known-good sources, and increasing monitoring during recovery. The Microsoft workflow similarly calls for validation, approvals, phased restoration, and heightened observation.

Create a recovery gate with explicit conditions:

  • The suspected cause is removed or isolated.
  • Affected credentials and sessions are rotated or invalidated.
  • Product state and external effects are reconciled.
  • Suspicious memory and artifacts are quarantined or restored from a known-good version.
  • The patched release passes the original incident reproduction.
  • Every enforcement point can still apply the current containment epoch.
  • A canary scope completes read-only work before side effects are enabled.
  • Security and the product owner approve the next autonomy level.
  • Monitoring has alerts for the incident's leading indicators.
  • A rollback command is ready and has a shorter deadline than the first response.

Restore capability in stages. Start with observation, then read-only tools, then reversible low-impact actions, then the affected side effects. Do not restore all tenants and tools because one clean test run succeeded.

Test the kill switch before an incident

At minimum, automate these cases:

  1. Start a long model call, trigger containment, and confirm no later tool dispatch occurs.
  2. Pause a worker before it receives the command, reconnect it, and confirm the durable epoch blocks resumed work.
  3. Queue work from an event, schedule, webhook, and retry path, then confirm none can enter running.
  4. Start a slow external request, cut the connection, and confirm the result becomes unknown until reconciliation.
  5. Make the credential provider unavailable and confirm the egress and tool gates still contain the scope.
  6. Make the egress controller unavailable and confirm authorization denies tool dispatch.
  7. Delegate a child task, stop the parent, and confirm the child loses authority even if its process remains alive.
  8. Propose a memory write during containment and confirm it reaches quarantine, not retrieval.
  9. Restart every worker after the stop and confirm it reads the current epoch before accepting work.
  10. Attempt recovery with one missing acknowledgement and confirm the recovery gate rejects it.

Keep the tests tied to the execution-path inventory. When a team adds a new queue, sandbox, tool adapter, or remote agent, CI should require a corresponding containment test.

Verify negative postconditions

Positive acknowledgements are necessary but not sufficient. Query the systems that would reveal continued work:

  • Active-run count for the affected scope is zero or explicitly quarantined.
  • Queue claims do not advance after the containment timestamp.
  • Tool receipts contain blocked decisions and no later permitted side effect.
  • Egress logs contain no post-stop request outside the incident allowlist.
  • Revoked grants cannot obtain or refresh credentials.
  • Delegation graphs contain no active authorized child.
  • Memory and artifact stores contain no newly trusted write from the affected scope.
  • External systems reconcile every unknown outcome.

These checks look for continued execution through every known path. They do not assume that a component stopped merely because it acknowledged the command.

Handle failure inside the response system

Response controls depend on services that may be degraded during the same incident.

If the policy store is unreachable, high-risk dispatch should fail closed for affected scopes. If the acknowledgement bus is down, components should still read durable policy and write local evidence for later collection. If credential revocation fails, deny egress and tool authorization while security escalates to the provider. If a worker cannot stop safely, isolate its process and network, then reconcile its effects. If evidence storage fails, stop destructive cleanup and preserve local snapshots under incident access control.

Avoid global fail-closed behavior for an unscoped outage unless the business has approved it. A malformed incident command should not accidentally disable every tenant. Require explicit scopes, protected break-glass procedures, signed commands, separation between command issuance and policy evaluation, and an audit record for every operator change.

Avoid four false-success patterns

A dashboard flag without runtime readers

Persisting and displaying emergency state is control-plane bookkeeping. Every admission, dispatch, egress, credential, queue, delegation, and memory path must enforce it.

Process termination without side-effect reconciliation

A dead worker can leave a successful remote action and no response. Record unknown outcomes and read the authoritative system before retrying or compensating.

One cancellation event without durable state

Offline or restarting workers can miss an event. Use a level-triggered stop state with an epoch that every claimant checks.

Recovery by clearing the flag

Clearing containment without eradicating the cause, rotating authority, reconciling effects, and passing the incident test recreates the original risk.

Build the first containment slice now

Choose one high-impact agent workflow. List every way it starts work and every boundary through which it can create an effect. Implement a scoped command, admission check, pre-tool check, durable cancellation state, credential revocation, and evidence acknowledgement for that workflow. Then run a test in which one component is offline and prove that recovery remains blocked.

The first test usually exposes missing owners and execution paths. Expand the contract to queues, delegates, sandboxes, memory, and external integrations. The AI agent incident response control plane is complete when every negative postcondition passes across the full product.

References

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us