Blog · · 12 min read

AI agent load testing without triggering real side effects

AI agent load testing without triggering real side effects

AI agent load testing fails when it treats a run like one predictable API request. A single accepted task can branch into several model turns, tool calls, retries, checkpoints, and callbacks. A test that repeats one prompt may miss the expensive path. A realistic test can be worse if write tools reach customer records, email recipients, or payment systems. This guide builds a trace-derived workload, blocks real side effects, drives open-arrival traffic, identifies the first saturated layer, and turns the result into a capacity gate that includes latency, cost, quality, and verified outcomes.

The capacity test has to measure a complete run

Measure how many representative tasks the release can complete inside its operating limits while preserving correct product outcomes. Endpoint acceptance alone cannot answer that.

Answer it with seven linked steps:

  1. Define pass and fail conditions for latency, queue age, cost, quality, and product outcomes.
  2. Derive task classes and branch distributions from sanitized production traces.
  3. Route every external write through a sandbox or behaviorally faithful tool twin.
  4. Test orchestration capacity with a model substitute, then run a bounded live-model phase.
  5. Generate open-arrival traffic that does not slow down merely because the target slows down.
  6. Attribute saturation to the generator, model quota, worker pool, state store, or downstream tool.
  7. Publish a capacity envelope that sets admission, autoscaling, headroom, and release policy.

The sequence connects provider limits, workers, and load generation in one test. It preserves the full agent path while keeping effects safe and measurements interpretable.

Why ordinary load tests give false confidence

Agent traffic has a distribution, not one request shape

Two users can submit the same intent and create different infrastructure demand. One run may answer from context after a single model call. Another may retrieve several pages, call three tools, pause for approval, retry a throttled provider, resume from a checkpoint, and emit a callback. Prompt size, stored state, tool fan-out, model turns, and retry paths all affect service time.

Exact-fit guidance from LoadView treats reasoning time, context, memory, tool dependencies, token cost, and output quality as part of the load surface. That is a better starting point than endpoint latency, but an existing product still needs its own task and branch mix. Uniform prompts test the prompt fixture, not the production workload.

Concurrency controls also need verification. A LangGraph practitioner report says a configured max_concurrency value did not constrain parallel tool execution in the reported path. A separate CrewAI report attributes stalled concurrent runs to synchronous model calls consuming worker threads. Treat both as author reports about specific paths rather than universal framework behavior. They show why declared settings and architecture diagrams are not capacity evidence.

A closed-loop generator can hide overload

A generator that waits for each task to finish before starting another reduces its own offered rate as latency rises. The target appears stable because the test stops applying the intended pressure. Grafana k6 documents constant-arrival-rate execution as an open model: iteration starts are independent of target response time, provided the generator has enough virtual users.

Use an open-arrival model for the main capacity ramp. Track dropped or delayed starts on the generator separately. If the generator cannot maintain the schedule, the run has measured the generator's capacity, not the agent's.

Throughput can rise while the product gets worse

An overloaded agent may still return HTTP 200 while queue age grows, cheaper fallback behavior activates, tools time out, retries increase, or output quality falls. It can also claim success after a write failed. Completed task count alone misses these failures.

The test therefore needs two result planes. The service plane measures accepted and completed throughput, latency, utilization, queueing, and limits. The product plane checks authoritative records, effect receipts, policy decisions, and task quality. A release has usable capacity only where both planes pass.

Build the AI agent load testing contract

Use one versioned test contract for every run. It should bind the release, workload, dependency modes, traffic schedule, budgets, outcome checks, and stop conditions.

release_id: rel_2026_08_27_12
workload:
  source: sanitized-production-traces-v8
  classes:
    - name: read_only_lookup
      share: 0.45
    - name: ticket_update
      share: 0.35
    - name: approval_then_write
      share: 0.20
traffic:
  model: open_arrival
  stages_rps: [2, 5, 10, 15, 20]
  stage_minutes: 10
dependencies:
  model_phase_one: deterministic-substitute-v4
  model_phase_two: bounded-live-provider
  ticket_tool: isolated-tenant
  email_tool: stateful-twin
  production_egress: deny
limits:
  max_queue_age_ms: 5000
  max_task_p95_ms: 30000
  max_error_rate: 0.01
  min_verified_outcome_rate: 0.99
  min_quality_score: 0.92

Set these example values from your product's SLOs, risk, and cost model.

Derive workload classes from traces

Start with completed and failed production traces from a representative window. Remove user content and secrets, then extract structural fields:

  • task type and tenant class;
  • input and context-size bands;
  • number of model turns;
  • selected model and token bands;
  • tool sequence and fan-out;
  • read, reversible write, and irreversible effect counts;
  • approval, callback, retry, cancellation, and resume paths;
  • final outcome class and total service time.

Cluster runs by structure, not wording. Preserve common paths and costly tails. Include failure classes that consume capacity, such as provider throttling, lost tool responses, and retry exhaustion. Keep the workload artifact versioned so a later release can be compared against the same demand shape.

Do not replay raw prompts because they are available. Structural sampling reduces privacy exposure and makes the test easier to reproduce. If model quality depends on linguistic variety, create scrubbed or synthetic prompts inside each structural class and record that limitation.

Put side effects behind one enforcement point

The load driver should not rely on a system prompt that says “do not send.” Route all tools through a dispatch gateway that understands the test lease. Each call must carry the release, workload run, tenant fixture, tool version, and dependency mode.

Allow only three destinations:

  1. a real service in an isolated account or namespace;
  2. a documented provider sandbox;
  3. a stateful twin that preserves the required schema, latency classes, errors, authorization checks, and state transitions.

Deny everything else. Emit an effect receipt for every attempted write, including denied calls. After the test, search production audit surfaces for the test release and fixture identities. Any match invalidates the run.

Staging defines a safe, faithful execution boundary. The load contract supplies a representative workload, rising offered demand, saturation analysis, and a capacity decision.

Separate infrastructure capacity from live-model behavior

Run two phases against the same workload.

In phase one, replace the model with a deterministic service that replays response-size, delay, tool-choice, and error distributions. Keep the real orchestration, queues, checkpoints, tool gateway, state stores, and safe dependencies. Microsoft Databricks recommends a model mock for infrastructure load testing, then ramps queries per second while monitoring time to first token, latency, errors, and resources. This phase reveals worker, queue, storage, and tool bottlenecks without spending provider quota or mixing model variance into every result.

In phase two, send a bounded subset through the live model provider. Reuse the task classes and safe dependency boundary. Measure token use, model latency, tool choices, outcome quality, refusals, and cost. OpenAI's rate-limit documentation distinguishes request and token limits, exposes remaining and reset headers, and notes that failed requests still consume limits. Record those headers and retry attempts next to each run so provider saturation is not misdiagnosed as worker saturation.

Phase one measures how much traffic your runtime can move. Phase two shows how much of that envelope survives real provider behavior and product-quality requirements.

Drive the ramp and prove the generator kept up

Warm the system at low load, then raise the open-arrival rate in fixed stages. Hold each stage long enough for queues, caches, connection pools, and autoscaling to settle. Use the same workload proportions at each level before testing deliberate skew, such as a burst of approval tasks or a large-tenant path.

Record scheduled starts, actual starts, accepted tasks, completed tasks, and timed-out tasks. Compare scheduled and actual starts by second. If the gap grows, add load generators or reduce client-side work. Never substitute active concurrency for offered arrival rate. Long tasks can increase concurrency even when arrivals stay flat.

Stop automatically when safety canaries fire, production egress appears, an irreversible effect reaches the wrong destination, or a hard provider budget is crossed. Operational thresholds may stop a stage too, but retain enough data to distinguish the first limit from the resulting cascade.

Attribute the first saturated layer

Collect metrics under one trace and release identifier:

  • generator start lag and dropped starts;
  • accepted, started, completed, and verified task rates;
  • queue age, queue depth, and task time in each state;
  • time to first token and total model latency;
  • input, output, cached, and reasoning token counts when exposed;
  • worker slots, pollers, CPU, memory, connections, and event-loop delay;
  • checkpoint and state-store latency;
  • tool concurrency, latency, errors, retries, and rate-limit responses;
  • cost per accepted, completed, and verified task;
  • quality and verified-outcome rate by workload class.

Temporal's worker-performance documentation treats worker slots, pollers, task queues, and resource-based concurrency as distinct controls. Keep them distinct in the dashboard. A full worker-slot pool with idle CPU suggests different action from high CPU with idle slots or a growing queue caused by a throttled tool.

Name the first threshold crossed at each stage. Later failures may be symptoms. For example, provider throttling can increase retries, hold worker slots longer, grow queue age, and then produce endpoint timeouts. Scaling workers alone would amplify the provider problem.

Publish a capacity envelope, not one maximum number

A single “maximum requests per second” hides the conditions that made the result possible. Publish an envelope tied to the tested release and workload.

release: rel_2026_08_27_12
workload: sanitized-production-traces-v8
safe offered rate: 10 tasks/second
burst: 15 tasks/second for 2 minutes
first limit: live-model token quota
required headroom: 30 percent
quality gate: passed for all classes
verified outcomes: passed for reads and ticket writes
excluded path: payment tool, sandbox fidelity unresolved
admission policy: reject above 10 tasks/second per region
retest triggers: model, prompt, graph, tool, quota, or workload-mix change

Google SRE's overload guidance connects overload behavior to load shedding, dependency limits, queue control, retries, and graceful degradation. Use the tested envelope to configure those production controls. Keep headroom explicit. Do not let autoscaling assume an external provider or shared product API will scale with your workers.

Work through a support-agent example

Consider a support agent that reads an account, updates a ticket, may grant a service credit after approval, and sends email.

The trace-derived workload has three classes. Read-only lookups usually take one model turn and two reads. Ticket updates use one read and one versioned write. Approval tasks pause, resume, write a credit, update the ticket, and send email. The test tenant contains synthetic accounts and tickets. Billing uses a provider sandbox. Email uses a stateful twin that stores deliveries but has no public route.

Phase one uses deterministic model responses with sampled delays and tool choices. At rising arrival rates, ticket updates fill the workflow worker slots before CPU is exhausted. Adding workers lowers queue age until the ticket API's isolated tenant begins returning throttles. The first limit has moved, and the safe rate rises only if both queue age and verified writes still pass.

Phase two sends a bounded sample to the live model. One workload class now produces longer prompts and more tool retries, causing the token limit to trigger below the infrastructure envelope. Quality also falls for approval tasks at the fallback configuration. The release gate therefore uses the lower live-model envelope. It does not average the two phases or publish the larger infrastructure number.

The outcome checker confirms that every accepted write changed one fixture record, every credit has one idempotency key, every email stayed in the twin, and no production audit record carries the test identity.

Handle failure and recovery during the test

If a dependency twin diverges from the real safe service, invalidate affected workload classes. Do not count them as passes or agent failures. Repair the twin, record its new version, and rerun those classes.

If the live provider throttles, honor its reset signal and bounded retry policy. Do not let every run retry at once. Record throttled time separately from queue wait so the capacity report names the provider as the limit.

If a tool reports an ambiguous write outcome, move that run to reconciliation. Repeating the write to keep throughput high corrupts the very correctness measure the test is meant to protect.

If the generator falls behind, stop the stage and add generator capacity. Results beyond that point do not prove the target sustained the scheduled rate.

If a safety canary detects production egress, revoke the test lease, stop new arrivals, drain or cancel queued work according to the effect policy, and inspect every receipt. Treat the run as a containment event, not a performance result.

Verify that the capacity claim is usable

Before accepting the capacity envelope, require machine-checkable answers:

  • Did the workload preserve production task, branch, tool, retry, and size distributions?
  • Did every write path use an isolated service, sandbox, or stateful twin?
  • Did production-side negative checks return zero matching effects?
  • Did the generator maintain scheduled open-arrival starts at every accepted stage?
  • Were infrastructure and live-model phases run against the same release and workload?
  • Can the dashboard distinguish provider, worker, queue, state-store, and tool saturation?
  • Did quality and verified outcomes pass for every material workload class?
  • Are retries, cancellations, ambiguous outcomes, and failed tasks included in cost and capacity?
  • Does the envelope state headroom, burst duration, excluded paths, and retest triggers?
  • Do production admission and autoscaling settings fit inside the measured envelope?

A missing answer narrows what the test proves. A failed safety or outcome check blocks the release even if latency and throughput look good.

Avoid four misleading shortcuts

Do not multiply one prompt at one concurrency level and call the result representative. It ignores branch and task distributions.

Do not hit real write dependencies for realism. A stateful twin or isolated tenant can preserve relevant behavior without turning a test into a customer incident.

Do not report active runs as throughput. Concurrency is in-flight work; throughput is completed work over time. Track offered, accepted, completed, and verified rates separately.

Do not raise worker concurrency until graphs look flat. Find the first saturated layer and its dependency limit. More workers can increase retries and deepen an external bottleneck.

Take the next action

Start your AI agent load testing with one consequential workflow and a sanitized structural sample of its latest traces. Define three workload classes, put every write behind a safe dependency mode, and run one low-rate model-mocked open-arrival stage. Do not increase load until the generator-start check, effect receipts, production-egress search, and verified-outcome checker all pass. That first valid stage becomes the baseline for the capacity ramp.

References

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us