Blog · · 11 min read

AI agent graceful degradation in existing products

AI agent graceful degradation in existing products

AI agent graceful degradation fails when a product treats availability as a single yes-or-no flag. One optional MCP server goes down, so the whole agent refuses to start. The opposite response is worse: the agent continues silently, then answers as if it checked evidence or held authority that was unavailable. Both designs waste capabilities that still work and mislead the user about what happened.

The fix is an explicit capability-mode contract. Map dependencies to the capabilities they support, calculate what is valid for this user and run, bind that answer to a durable execution step, and expose the same limits in the API and interface. This preserves safe work during a partial outage without disguising a partial result as full success.

Why generic fallbacks are not enough

A production agent rarely has one dependency. It may call a model, several internal APIs, MCP servers, a retrieval index, a memory store, an authorization service, an idempotency ledger, a result verifier, and a notification channel. Each dependency supports a different part of the user promise.

Reliability guidance covers important mechanisms. The AWS Agentic AI Lens on graceful degradation recommends stage-level telemetry, composite health, resource isolation, configuration versioning, and automated degradation. Its memory and state guidance defines full, session-only, and stateless operating modes. Its tool fallback guidance covers per-tool cutoffs, retries, fallback chains, monitoring, and runbooks.

Those controls answer how a component can fail. A product contract must also answer what the agent may still promise.

Consider a support agent with these operations:

CapabilityDependenciesEffect if one is missing
Explain a public policyModel, public knowledge snapshotAnswer only if the snapshot is current enough for the policy
Summarize this ticketModel, ticket read API, current user authorizationBlock if the current user cannot be authorized
Draft a replyModel, ticket read API, user authorizationAllow a labeled draft; do not send
Send a replyDraft dependencies, write API, idempotency ledger, result verifierBlock until every write dependency is healthy
Recall prior preferencesMemory read service, consent stateOmit memory and disclose the limitation
Escalate to a humanQueue API or an explicit manual handoff pathUse the available path and return a receipt

A model fallback list cannot represent this table. Neither can a global degraded=true flag. Both erase the difference between an optional convenience and a missing safety precondition.

Direct product reports show the cost of that erasure. A Google ADK issue describes an agent invocation failing because one MCP server was unavailable even though the agent still had other tools and built-in knowledge. A Cherry Studio issue asks for critical-versus-optional dependency handling so one MCP failure does not block startup. Treat these as author reports, not universal product behavior. They demonstrate a reproducible architecture question: should one missing integration remove every unrelated capability?

Derive modes from capabilities, not service names

Services and user capabilities are many-to-many. One retrieval service may support policy answers and account guidance. One send action may require authorization, a write API, an idempotency ledger, and verification. Marking a service as optional does not tell you whether a particular action is safe.

Define capabilities first, then attach dependency predicates to them. Calculate an effective manifest at run start from service health, current authority, tenant policy, data freshness, and workflow state. The AWS guidance covers component fallbacks but does not generate this cross-layer capability graph. Treat the graph as a first-principles design recommendation rather than a vendor guarantee.

{
  "manifest_id": "cap_20260914_0042",
  "subject": {
    "user_id": "user_184",
    "tenant_id": "tenant_72",
    "run_id": "run_901"
  },
  "mode": "read_and_draft",
  "healthy_dependencies": [
    "primary_model",
    "ticket_read_api",
    "public_knowledge_snapshot"
  ],
  "unavailable_dependencies": [
    "memory_read",
    "ticket_write_api"
  ],
  "capabilities": {
    "read_ticket": "allowed",
    "draft_reply": "allowed_with_disclosure",
    "send_reply": "blocked",
    "use_saved_preferences": "blocked",
    "human_handoff": "allowed"
  },
  "limitations": [
    "Saved preferences were not consulted",
    "Replies can be drafted but not sent"
  ],
  "policy_version": "degradation-policy-18",
  "expires_at": "2026-09-14T10:02:00Z"
}

The mode name is useful for operators and users, but the capability map is the enforcement source. A mode such as read_and_draft should be a generated summary of allowed operations, not a shortcut that bypasses per-action checks.

Use these dependency classes:

  • Hard execution dependency: Without it, the capability cannot produce a truthful result. A ticket summary needs an authorized ticket read.
  • Safety dependency: Without it, the action might execute but cannot be controlled or verified. A payment write needs current authorization, idempotency, and postcondition verification.
  • Freshness dependency: Without it, an answer may be stale beyond the product's declared tolerance. Price, inventory, and policy answers often belong here.
  • Optional enrichment: Its absence reduces personalization or convenience without invalidating the core result. Saved tone preferences may fit here.
  • Recovery dependency: It is not needed for a new safe action but is needed to reconcile an action whose outcome is unknown.

This classification prevents a common mistake: calling a database or MCP server optional because the runtime can start without it. Optional infrastructure is not the same as optional evidence or authority.

Make one mode decision per durable step

A dependency can recover while a model is planning, while a tool call is in flight, or while a workflow is paused for approval. If each tool checks current health independently, one logical step can begin in degraded mode and finish in full mode. The model may have planned under the assumption that memory was unavailable, then a later tool silently uses recovered memory. The resulting action has no coherent evidence boundary.

The reader should bind manifest_id to each durable step, tool request, and receipt. Health recovery can produce a new manifest for the next step. It must not rewrite the contract of a step already in progress.

def execute_step(run_id, step_id, requested_action):
    manifest = load_or_create_manifest(run_id, step_id)
    decision = manifest.check(requested_action)

    if decision.status == "blocked":
        return blocked_result(
            action=requested_action,
            manifest_id=manifest.id,
            reason=decision.reason,
        )

    result = call_tool(
        action=requested_action,
        manifest_id=manifest.id,
        idempotency_key=f"{run_id}:{step_id}:{requested_action.id}",
    )

    return verify_and_record(
        result=result,
        expected_manifest_id=manifest.id,
        capability=decision.capability,
    )

This is a first-principles recommendation derived from durable workflow consistency, not an AWS product guarantee. The AWS guidance calls for explicit mode transitions and automated recovery, while the manifest binding closes a separate cross-layer gap: every decision and effect can be traced to the same availability assumptions.

If a hard dependency recovers during the step, record recovery_pending and offer a retry or a new step. If a safety dependency fails after an external write begins, do not downgrade the result to a generic failure and retry. Mark the outcome unknown, reconcile it against the authoritative product, and only then decide whether another attempt is safe.

Missing evidence can remove write authority

The model may still be available. The write API may still accept requests. That does not mean the agent retains permission to act. If current authorization cannot be checked, required source data is stale, or the result verifier is down, the product cannot establish the conditions under which the write was approved and completed.

Silent degradation is most dangerous when it preserves output generation while removing the evidence chain around that output. Make write eligibility conjunctive:

write_allowed =
    current_user_authority
    AND fresh_required_inputs
    AND policy_service_available
    AND idempotency_ledger_available
    AND postcondition_verification_available

This is an implementation recommendation. It intentionally differs from a fallback that swaps one model or tool for another. Rank the missing dependencies by the guarantee they remove. A missing style-memory service can remove personalization. A missing authorization or verification service should remove write authority.

Offer a useful alternative when one exists:

  • return an evidence-labeled read result
  • create a local draft that cannot trigger a side effect
  • queue a proposal with an expiry and require revalidation before execution
  • route to a human with the missing dependency named
  • provide a status handle instead of claiming completion

Do not queue an already-authorized write for blind execution after recovery. Authority, source freshness, price, inventory, and product version may change while the queue waits. Queue a proposal, then re-check every precondition before it becomes an executable command.

Keep the API and interface in the same mode

A backend that blocks writes while the interface still shows an active Send button creates avoidable retries and distrust. A banner that says "limited mode" without naming the limitation is not enough either.

Return structured mode information with every run and terminal result:

FieldPurpose
modeStable user-facing summary such as read_and_draft
manifest_idTraceable enforcement decision
available_capabilitiesActions the user can still request
blocked_capabilitiesActions currently unavailable
limitationsEvidence, memory, freshness, or side-effect limits
retry_policyWhether retry is automatic, manual, or prohibited
recovery_statusLast successful probe and pending transition
receiptWhat was actually completed or verified

Render those fields where the user makes a decision. Disable unavailable actions with a specific reason. Label drafts as unsent. If memory was not consulted, say so beside the answer rather than in a remote status page. If a partial result omits records because a data source failed, list that source and do not use a success label that implies completeness.

The Zylos graceful-degradation guide covers layered resilience across providers, tools, context, backpressure, coordination, and partial results. A product-specific implementation must connect those mechanisms to one enforceable and visible user contract, not merely install more retries.

Define mode transitions and recovery

Use a small state machine. More mode names make policy harder to understand and test.

FULL
  -> READ_AND_DRAFT
  -> READ_ONLY
  -> HANDOFF_ONLY
  -> UNAVAILABLE

Do not assume every transition is adjacent. Loss of current authorization can move a write-capable run directly to HANDOFF_ONLY. Loss of the primary model may still permit status lookup or cancellation through deterministic product controls. A regional outage may make the whole workflow unavailable while preserving a receipt endpoint served from another control plane.

For recovery, require:

  1. The failed dependency passes a functional probe, not only a process-health check.
  2. The dependency remains healthy for a configured stabilization window.
  3. Current user and tenant authority are re-evaluated.
  4. Data freshness and product-version preconditions are rechecked.
  5. Unknown external outcomes are reconciled.
  6. A new manifest is issued for the next durable step.
  7. The interface receives the new mode and available actions.
  8. A synthetic probe proves the recovered capability end to end.

Automatic recovery is useful only when it restores guarantees as well as connectivity. A green socket does not prove that the tool schema matches, the user remains authorized, or an earlier write did not complete.

Test the contract with dependency faults

Build a fixture for each dependency class. Fault injection should assert both positive and negative behavior. It is not enough to confirm that the agent returns some text.

Use this minimum matrix:

FaultExpected modeMust remain possibleMust be impossible
Optional memory unavailableFull without personalization or named reduced modeAuthorized read and draftClaim that preferences were used
One optional MCP server unavailableCapability-specific reduced modeUnrelated built-in and tool capabilitiesCalls to the missing server
Authorization service unavailableRead-only, cached-public, or handoffPublic status and safe cancellation path if independently authorizedProtected reads and all new writes
Retrieval snapshot staleDraft or unavailable for freshness-sensitive answersClearly labeled historical contextCurrent-price or current-policy claim
Write API unavailableRead and draftProposal creationClaim that the external effect completed
Idempotency ledger unavailableRead and draftNon-effecting workRetried or queued production write
Result verifier unavailable after callOutcome unknownReconciliation and status inspectionAutomatic replay

For every case, assert that the unavailable dependency disables every capability that depends on it and no independent capability. Assert that the API and interface display the same mode. Assert that tool requests carry the manifest ID. Assert that active steps do not switch manifests. Assert that recovery creates a new manifest and rechecks authority, freshness, version, and unknown outcomes.

Also test conflicting health signals. If the model gateway reports healthy but the canary tool call fails, the functional probe should win. If the interface receives a stale mode event, it should compare manifest versions before re-enabling an action. If an operator manually forces a mode, record the actor, reason, expiry, and policy version in the mode receipt.

Common mistakes

Making the model decide what still works. The model can choose among allowed capabilities. It should not infer authorization, dependency health, or product guarantees from an error message.

Using retries as the mode policy. Retries help with transient failures. They do not say whether a missing dependency is optional, whether a partial result is complete, or whether a delayed write remains authorized.

Treating every partial response as useful. A summary based on three of four data sources may be useful when the omission is visible. A compliance decision based on the same omission may be invalid. Capability policy must reflect the reader's actual workflow.

Restoring full mode on process health alone. Recovery requires functional probes and product preconditions. Otherwise the first user becomes the test request.

Hiding mode state in logs. Operators need logs. Users need actionable limits at the point where they request, approve, or interpret an action.

Implement the smallest useful degraded mode

Start with one workflow and list its user-visible capabilities. Map every capability to hard execution, safety, freshness, optional enrichment, and recovery dependencies. Implement one reduced mode that preserves a real outcome, such as read and draft, while blocking writes. Add a manifest ID to tool requests and receipts. Expose blocked capabilities and limitations in the interface. Then run the fault matrix before expanding the policy.

The immediate next action is concrete: choose the highest-value write action in your agent, list every dependency needed to authorize, execute, deduplicate, and verify it, and prove that removing any one of those dependencies makes the write impossible while independent read or handoff paths still work.

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us