Blog · · 11 min read

AI agent rate limiting: protect product APIs from retry storms

AI agent rate limiting: protect product APIs from retry storms

AI agent rate limiting fails when it counts only model requests. One accepted product task can branch into parallel tool calls, subagent work, and retries against APIs that also serve ordinary customers. A provider quota may remain healthy while your ticket, billing, or account service collapses. Use a product-owned capacity contract that counts fan-out per run, admits work before execution, isolates each dependency, spends retries from one budget, expires stale work, and reserves capacity for interactive traffic. This guide shows how to build that contract and test that agent work sheds first under overload.

Use one product-owned capacity contract

Put a capacity controller between the agent runtime and every shared product capability. Do not leave each model client, tool wrapper, and worker to invent its own limit.

The controller needs six decisions:

  1. Classify the request as interactive, approval-resume, autonomous, or maintenance work.
  2. Attach a run ID, tenant, priority, deadline, and remaining fan-out budget.
  3. Admit, delay, coalesce, reject, or expire the work before a worker starts it.
  4. Acquire capacity for the exact downstream dependency, not a global worker pool.
  5. Charge retries to one run-level token budget at one architectural layer.
  6. Record the decision and return a typed outcome the product can display and operate.

This extends ordinary API throttling. Microsoft's throttling pattern covers capping resource use, prioritizing work, deferring lower-priority requests, smoothing bursts, separating tenants, and returning retry guidance. Agent workloads need the same controls, but the accounting unit should be the logical product run as well as the individual call.

Why agent traffic overloads shared APIs

One task becomes many calls

A user asks a support agent to resolve a billing ticket. The run may read the customer, retrieve invoices, inspect plan rules, call a payment service, update the ticket, and send a notification. A planner can launch some reads in parallel. A failed tool can trigger another model turn and another call. One product request has become a small workload graph.

Current AI agent rate-limiting guidance identifies calls per task, per-tenant fairness, token buckets, and retry amplification as agent-specific concerns. Count both each physical request and its parent run. A per-second API limiter cannot tell whether 50 calls came from 50 customer actions or one runaway plan.

Retries multiply across layers

Suppose the agent runtime retries a tool three times, the tool client retries its HTTP request three times, and the queue retries the job three times. One failed logical action can produce 27 downstream attempts. Jitter changes their timing, but it does not fix duplicate retry ownership.

Google's chapter on addressing cascading failures shows how retries increase offered load during an outage and why servers should fail early and cheaply when overloaded. A direct Hacker News practitioner question describes workflows that can call ten or more services and asks how to prevent 429-driven retries from worsening overload. Treat that thread as problem language, not a measured incident or benchmark.

Queues can hide overload

An unbounded queue converts rejected work into long latency and a difficult recovery. The dashboard may show that workers are busy while users wait for tasks whose deadlines already passed. When the dependency recovers, the backlog can create another burst.

Google SRE's overload guidance recommends graceful overload handling, client-side throttling, criticality-aware admission, and load shedding. Current agent backpressure guidance applies bounded queues, per-dependency concurrency, cooldowns, and queue-wait measurements to agent workflows. The product contract must add a deadline and a user-visible terminal outcome so old autonomous work does not survive merely because queue storage is available.

Build AI agent rate limiting around the run

Classify traffic before it reaches a tool

Start with product meaning, not infrastructure labels. Classify each request as one of these types:

  • interactive: a person is waiting in the product UI.
  • approval_resume: a person approved a specific paused action.
  • autonomous: a trigger, schedule, or background agent started the work.
  • maintenance: evaluation, reindexing, reconciliation, or bulk migration.

Reserve capacity for the classes that protect the core product journey. Do not let autonomous work borrow the final interactive slots. Borrowing unused capacity can be safe if it is revocable as soon as foreground demand returns.

Keep priority separate from tenant identity. A large tenant should not consume every autonomous slot, and a low-volume tenant's interactive action should not wait behind another customer's batch. Enforce both a global class limit and a tenant-level share.

Give each run a workload envelope

Create the envelope when the product accepts the task. Carry it through model calls, tool calls, subagents, queues, and retries.

run_capacity:
  run_id: run_7f42
  tenant_id: tenant_18
  traffic_class: interactive
  accepted_at: 2026-08-03T10:00:00Z
  deadline_at: 2026-08-03T10:01:30Z
  max_tool_attempts: 20
  remaining_tool_attempts: 20
  max_parallel_tools: 3
  retry_tokens: 2
  dependency_budgets:
    customer_api: 6
    billing_api: 4
    notification_api: 2
  coalesce_key: null

Those numbers are an example, not a universal recommendation. Derive them from a known workflow, dependency capacity, and user latency target. A read-heavy support task and a payment mutation need different envelopes.

Count attempts before dispatch. If the planner asks for six calls but only three remain, return a typed budget_exhausted result before any call starts. The agent can produce a partial explanation or hand off to a person, but it cannot silently exceed the product's limit.

Admit work before consuming a worker

Admission control should consider the run deadline, queue length, dependency state, traffic reservation, tenant share, and coalescing key. The decision must happen before expensive context assembly or model reasoning when possible.

function admit(step, run, dependency):
    if now >= run.deadline_at:
        return EXPIRED

    if run.remaining_tool_attempts == 0:
        return BUDGET_EXHAUSTED

    if equivalent_work_is_queued(run.tenant_id, step.coalesce_key):
        return COALESCED

    if dependency.circuit_is_open:
        return DEPENDENCY_UNAVAILABLE

    if reserved_capacity_unavailable(run.traffic_class, run.tenant_id):
        if step.may_wait and estimated_start < run.deadline_at:
            return QUEUED
        return THROTTLED

    decrement(run.remaining_tool_attempts)
    acquire(dependency.bulkhead, run.traffic_class, run.tenant_id)
    return ADMITTED

QUEUED is not success. Persist the estimated start, deadline, and cancellation handle. THROTTLED, EXPIRED, and BUDGET_EXHAUSTED should remain distinct because they demand different user messages and operator actions.

Coalesce work only when the result is safely shareable. Ten events asking to refresh the same read-only account summary may share one run. Ten instructions to send a message may not.

Isolate every downstream dependency

A single global semaphore protects total worker count but lets one slow API occupy every slot. Give each dependency its own concurrency bulkhead, queue, timeout, and circuit state. Add a smaller global ceiling to protect the runtime itself.

The customer-record API may tolerate 40 agent reads while the billing API allows only five concurrent mutations. Keep those controls separate. Also separate read and write pools when their cost and failure consequences differ.

Adjust limits from observed latency and error behavior, but apply bounds. An adaptive controller must not reduce interactive capacity to zero or increase concurrency beyond a tested maximum. Google's overload chapter describes adaptive client throttling as a way to react to rejected requests while avoiding synchronized client behavior.

Put retry ownership in one layer

Choose one component to decide whether a failed logical step may retry. Lower layers can expose transport details, but they should not independently repeat the action.

A retry needs all of these conditions:

  • The failure is explicitly retryable.
  • The operation is safe to repeat or carries an idempotency key.
  • The run deadline leaves enough time for another attempt.
  • The run has a retry token and a tool-attempt token remaining.
  • The dependency circuit is closed or half-open for a probe.
  • The server's Retry-After value, when present, fits the remaining deadline.

Spend the retry token before scheduling the next attempt. Jitter the delay to avoid synchronized retries. If any condition fails, return a typed terminal or human-recovery outcome.

A request in the OpenAI Agents SDK repository asks for local rate-limit support rather than manual waits around agent calls. That report supports the need for a reusable control point. Product teams still need to own policy because the runtime cannot know tenant fairness, product deadlines, or which customer traffic must be protected.

Make overload visible

Record one event for every capacity decision with these fields:

  • run ID, tenant, traffic class, task class, and agent release
  • dependency, tool, attempt number, and retry owner
  • admission outcome and reason code
  • queue wait, service time, and remaining deadline
  • remaining fan-out and retry budgets
  • concurrency limit, in-use count, and circuit state
  • originating product request and user-visible task status

Do not collapse throttled, expired, dependency_unavailable, and budget_exhausted into a generic tool error. The agent should not describe rejected work as completed. The product should show whether it will retry, needs a person, or stopped permanently.

Apply the contract to support automation

Consider a support agent that triages new tickets and can issue a billing adjustment after approval. The product API also serves support staff working in the ordinary interface.

Reserve most billing mutation capacity for interactive staff and approved resumes. Let background triage borrow unused read capacity, but revoke that borrowing when foreground latency rises. Give every ticket run a deadline. Coalesce repeated read-only enrichment for the same ticket, but never coalesce mutations. Limit each run to a small number of billing attempts and one retry owner.

When the billing API starts returning 429 responses, stop admitting new autonomous billing steps. Honor a valid Retry-After only when it fits the run deadline. Keep interactive reads available if their dependency pool is healthy. Expire queued background steps whose result would arrive too late to matter. Send unresolved high-risk mutations to a human queue rather than asking the model to improvise.

Provider-level rate limits do not protect this product journey. The product policy must cover run fan-out, traffic classes, tenants, dependencies, deadlines, and retries.

Handle failure and recovery

Open a circuit for one dependency when recent failures or latency cross its tested threshold. Do not stop unrelated tools. While the circuit is open, reject new work cheaply with dependency_unavailable. After a cooldown, allow a small number of half-open probes. Close the circuit only after those probes meet the recovery rule.

Treat an ambiguous write separately from a clean rejection. A timeout after dispatch may have committed the side effect. Do not spend a retry token until the application checks the idempotency record or reads back the product state. Capacity control prevents overload; it does not replace side-effect safety.

On cancellation, remove queued work and mark in-flight work as no longer eligible for follow-up fan-out. A call that cannot be interrupted may finish, but its result must pass the usual current-state and authorization checks before another action starts.

During recovery, drain queues by priority and deadline, not arrival time alone. Continue shedding stale autonomous work. A recovered dependency can fail again if every queued item starts immediately.

Verify capacity protection with overload tests

Run these tests against a staging environment with realistic dependency delays and errors:

  1. Drive the planner past its fan-out ceiling. Excess calls must never reach a dependency, and the task must end with budget_exhausted.
  2. Saturate autonomous capacity before issuing interactive product requests. Their latency must stay within the tested bound while the controller delays or sheds autonomous work first.
  3. Flood one tenant's background queue. Another tenant must retain its reserved share.
  4. Slow the billing API while customer and notification APIs remain healthy. Billing saturation must not consume their slots.
  5. Inject one retryable failure. Exactly one layer should retry, and total attempts must stay within the run's retry-token budget.
  6. Leave work queued until its deadline passes. No worker should start it, and the product must record expired.
  7. Return Retry-After values both inside and beyond the run deadline. The controller should schedule only the feasible retry.
  8. Force the circuit open, then restore the dependency. Only bounded probes should run before normal admission resumes.
  9. Time out after a simulated commit. The system must read back or check idempotency state instead of repeating the mutation.
  10. Recover from a prolonged outage with a backlog. Stale work must shed while interactive traffic retains reserved capacity.

Track queue wait, rejection reason, expired work, attempts per logical step, retry tokens spent, circuit transitions, and p95 and p99 latency by traffic class. A low overall error rate is not enough if autonomous work still starves foreground requests.

Common mistakes

  • A model-call limit misses product APIs, databases, browser services, and notification providers. Enforce limits at each shared dependency and account for the parent run.
  • One queue lets a slow billing service block a customer lookup. Separate queues and concurrency by dependency and traffic class.
  • Retries at every layer multiply attempts. Assign retry ownership once and expose failures from lower layers.
  • Work kept forever turns durable storage into an unbounded backlog. Give the queue a deadline, bounded depth, cancellation, and stale-work shedding.
  • Generic tool errors let the model mistake throttling for success or keep trying a permanent rejection. Return typed outcomes instead.
  • A throughput test without foreground traffic proves the wrong thing. Test that agent load cannot take the product below its customer-facing capacity target.

What to do next

Choose one agent workflow and list every downstream call it can create, including retries and subagent branches. Assign a run deadline, fan-out budget, retry owner, traffic class, tenant key, and dependency bulkhead. Then run the foreground-reservation and retry-ownership tests before increasing autonomous traffic.

References

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us