Blog · July 22, 2026 · 11 min read

AI agent prompt injection security: threat-model every tool call

AI agent prompt injection security breaks down when a team treats untrusted content as a text-cleaning problem. A webpage, ticket, email, file, or tool description can influence the model. The model may then call a legitimate tool with dangerous arguments. OWASP distinguishes direct injection from indirect injection through external sources and says foolproof prevention remains unclear. Build the system so manipulated model output cannot cross authorization, data, or side effect boundaries unchecked. The threat model below shows where to enforce that rule and how to test it.

Put six boundaries between content and side effects

Treat the model as an untrusted planner. It can propose an action, but it cannot authorize that action. A connected agent request crosses six explicit boundaries:

A user does not need to write the hostile prompt. Microsoft documents a tool poisoning pattern where changed MCP metadata steers an agent through sensitive data retrieval and an allowlisted outbound call. Each action looks legitimate in isolation. The mitigation applies controls to metadata, actions, identities, and telemetry instead of depending on one content check (Microsoft Security's connected-agent attack walkthrough).

Why indirect prompt injection reaches real systems

An ordinary application keeps code and data in separate structures. A model receives both as tokens. System instructions, retrieved records, tool descriptions, and hostile text may all appear in the same context. Calling a block "untrusted" helps the model interpret it, but the label does not create a hard security boundary.

A typical attack crosses several components:

1. The agent reads content controlled by someone outside the trust boundary.
2. The content tells the model to reveal data, change its objective, or call a tool.
3. The model emits a valid tool name and well-formed arguments.
4. The runtime sees syntactically valid output and executes it.
5. The tool uses real credentials and returns real data or performs a real side effect.
6. The agent sends the result to a response, file, message, or remote destination.

OWASP's prompt-injection scenarios include hidden webpage instructions, altered retrieval documents, unauthorized access, and data exfiltration. Its mitigations combine input and output controls with least privilege, human approval, separation of external content, and adversarial testing. If one layer misses the attack, the next boundary still evaluates the proposed action.

Connected protocols bring ordinary application security risks into the same path. The MCP security best-practices documentation covers confused-deputy attacks, token passthrough, server-side request forgery, session hijacking, local server compromise, and overbroad scopes. Prompt injection can trigger an attack, while weak token audiences, broad process permissions, or unrestricted network access determine its blast radius.

An open GitHub issue shows why the tool schema and executor both matter. The author reports that filesystem tool path parameters lacked schema-level traversal constraints and describes how hostile content could induce reads or writes outside the intended directory. The issue proposes canonicalization, allowlist enforcement, bounded schemas, and clearer tool descriptions. This is a practitioner report, not a confirmed universal flaw. It still supplies a good test case: a path chosen by the model must never define its own security scope (MCP filesystem issue #3752).

Build an AI agent prompt injection security model

A whole-platform threat model becomes vague quickly. Choose one product action that reads untrusted content and can access sensitive data or change state. Apply the following controls to that path.

1. Record provenance at ingestion

Attach provenance to every context item before it reaches the model. Record the source type, source identity, tenant, retrieval time, trust class, and allowed use. Do not let the model assign or upgrade these labels.

A retrieved support ticket might be valid evidence for summarization but invalid as an authority to change permissions. A tool description may guide tool selection, but a new description version should not silently gain access to another dataset without review. Microsoft's walkthrough recommends treating production tool metadata changes with the same rigor as system prompt changes and reviewing tool versions rather than trusting a stable display name (Microsoft's mitigation guidance).

2. Separate planning from authorization

Require a structured proposal from the model, then evaluate it outside the model. The policy service receives authenticated product context that the model cannot modify:

proposal = model.plan(context)

request = {
  user_id: authenticated_user.id,
  tenant_id: authenticated_user.tenant_id,
  workflow_id: current_workflow.id,
  tool_id: registry.resolve(proposal.tool_name).immutable_id,
  tool_version: registry.resolve(proposal.tool_name).approved_version,
  arguments: schema.validate(proposal.arguments),
  data_classes: classify_requested_data(proposal.arguments),
  destination: resolve_destination(proposal.arguments)
}

decision = policy.evaluate(request)

if decision == "deny":
  audit.record(request, decision)
  stop()
if decision == "needs_approval":
  approvals.create(hash(request), expires_at, required_approver)
  pause()

executor.run(request, policy_scoped_credentials(request))

The model may suggest user_id, tenant_id, or a destination in its arguments. The gateway must derive identity and scope from the authenticated session and product records. Ignore model-supplied security context when it conflicts.

3. Enforce least privilege and least agency

Give each tool only the credential scopes required for its action. Do not pass a user's broad token through an MCP server or downstream API. MCP's official guidance calls token passthrough an anti-pattern because it can bypass audience checks, rate limits, request validation, and audit controls. It also recommends incremental, minimal scopes instead of requesting every supported permission up front (MCP token and scope guidance).

A narrow read token can still expose every record within its scope or combine those records into an unsafe export. Limit records, tools, turns, destinations, time, and data classes for each run. Microsoft calls this "least agency": constrain autonomy as well as permissions, and require approval for high-impact actions (Microsoft's least-agency guidance).

4. Validate tool arguments at the executor

A JSON schema catches malformed model output. The executor must still resolve file paths, resource IDs, and URLs to canonical values, then compare them with an allowlist. Reject unknown fields, oversized values, traversal, wildcard expansion, cross-tenant identifiers, private network destinations, and redirects to unapproved hosts.

Keep security rules close to the effect. A filesystem server enforces its allowed roots. A billing service enforces account ownership and refund limits. A messaging service enforces recipients and attachment policy. The orchestrator should not duplicate these product invariants, and a prompt can never override them.

The filesystem issue report connects a hostile instruction to a valid-looking path argument. Its proposed defense in depth includes path canonicalization and allowlist checks at runtime, plus schema constraints before execution. That sequence makes a useful negative test (reported path-validation gap and remediation).

5. Control outbound data paths

List every route through which data can leave the trusted system: tool responses, rendered Markdown, image URLs, webhooks, email, uploaded files, logs, traces, and model provider requests. Apply destination allowlists and data classification rules to the final outbound payload, not only to the user's initial prompt.

A fraud-check tool may need an invoice ID but not customer names, complete invoice bodies, or credentials. The gateway should reject extra parameters that appear after a tool metadata update. Microsoft recommends inspecting tool call parameters, blocking sensitive outbound payloads, and alerting when an agent begins using new endpoints (Microsoft's action and telemetry controls).

6. Bind approval to the exact action

Approval works only if the approver sees the action that will execute. Store an immutable hash of the tool ID, tool version, arguments, data classes, destination, user, tenant, and expiration. Any change invalidates the approval. A tool name alone is too broad an approval target.

OWASP recommends human approval for high-risk operations. Use it for external sharing, permission changes, destructive actions, broad data access, and any request that exceeds the workflow's normal envelope. Approval supplements authorization; it does not replace it.

7. Correlate the full chain

Create one trace that links source records, model calls, proposed tools, policy decisions, approvals, tool versions, credential scopes, outbound destinations, side effects, and final outcomes. Redact sensitive prompt and response content, but retain hashes and structured fields needed for investigation.

Relationship-based alerts catch sequences that isolated string rules miss. Watch for an untrusted document followed by a privileged read, a new tool description followed by expanded parameters, a cross-tenant lookup, an approved tool contacting a new host, or repeated denied calls. Microsoft's guidance recommends correlating MCP telemetry, behavior signals, external endpoints, and audit logs because each event in the sequence may look harmless by itself (Microsoft's detection guidance).

Work through a connected support-agent example

Take a support agent that summarizes tickets, reads account data, drafts replies, and can issue small service credits. A customer submits a ticket containing hidden instructions to ignore the task, retrieve other customers' records, and send them to an external diagnostic URL.

The system assumes the hidden instruction may work and stops it at independent boundaries:

1. The ticket enters context with trust_class: external_customer_content and the current tenant ID.
2. The model proposes a broad account search and an outbound diagnostic call.
3. The product policy denies account IDs outside the authenticated tenant.
4. The tool gateway rejects the diagnostic host because it is not in the workflow's destination allowlist.
5. The data policy denies customer records in outbound tool parameters.
6. A credit remains possible only within the authenticated account, configured limit, and approval rule.
7. The trace links the hostile source, denied proposals, and policy reasons without storing secrets in logs.

If an approved tool keeps its name but publishes a new description asking for extra customer fields, the registry detects an unapproved version, blocks execution, and requires review. The model does not need to recognize that its own tool instructions are hostile.

Handle failure and recovery

A control needs an explicit failure mode. Otherwise, a timeout or missing field may become an authorization bypass.

• Missing provenance makes content untrusted by default.
• A policy service outage blocks state changes and sensitive reads. A low-risk read may proceed only when a precomputed policy permits that degraded mode.
• A new tool version stays disabled until it passes review.
• Uncertain destination resolution or data classification stops the outbound call.
• An expired approval or changed action hash requires a new approval.
• A side effect that times out with an unknown outcome must be reconciled through the product's action ledger before retry.
• An incident triggers credential revocation, tool-version shutdown, trace preservation, identification of exposed data classes, and review of every run that used the same source or metadata version.

Do not ask the compromised agent to diagnose or remediate itself with the same tools and context. Recovery belongs on a separate operator path with independent credentials and a reduced tool set.

Verify the threat model with adversarial tests

Boundary violations make better test cases than a catalog of jailbreak phrases. OWASP recommends adversarial testing that treats the model as an untrusted user and exercises trust boundaries and access controls (OWASP testing guidance). Include these cases:

Run these tests against the real policy engine, tool gateway, and product authorization layer. You can mock the model and supply the malicious proposal directly. Even a perfectly formed hostile call must satisfy the deterministic controls.

Mistakes that reopen the attack path

• A prompt filter can reduce attack frequency. The policy engine still has to prove that the user may access a record or call a tool.
• Reads deserve the same scrutiny as writes. Data exfiltration is a side effect, and a broad export can cause more damage than a change to one field.
• Tool names and descriptions are mutable. Pin an immutable tool version, then review changes to its description, schema, credential, and destination.
• The gateway derives credentials, tenants, network destinations, and approval scope from authenticated state and registry entries. The model cannot choose them.
• Raw prompts and tool results stay out of logs by default. Investigation needs correlation and structured evidence, but raw logs can become another sensitive data store.

What to do next

Apply this AI agent prompt injection security model to one action that can read sensitive data or change product state. Draw the six boundaries from source to effect. For each boundary, write one deny test with a valid-looking but unauthorized model proposal. Do not connect the action to production until every test fails closed and the trace names the deterministic control that stopped it.

References

• OWASP LLM01:2025 Prompt Injection supports the direct and indirect injection distinction, the limits of perfect prevention, layered mitigations, high-risk approval, and adversarial testing.
• Model Context Protocol Security Best Practices supports the guidance on token audiences, token passthrough, scope minimization, SSRF, session security, sandboxing, and conventional trust boundaries around MCP systems.
• MCP filesystem server issue #3752 is a practitioner report used for the unbounded-path threat scenario, schema constraints, canonicalization, and allowlist test case.
• Microsoft Security: Securing AI agents when AI tools move from reading to acting supports the tool-poisoning attack chain and controls for tool versions, identities, parameters, outbound data, approval, least agency, and correlated telemetry.