Blog · · 10 min read

Test MCP tool annotations across every integration hop

Test MCP tool annotations across every integration hop

MCP tool annotations can be correct in source code and wrong by the time a client makes an approval decision. A gateway may retain an old descriptor, an SDK converter may drop a field, or a host may interpret a truthful destructiveHint in a way that blocks a normal workflow. The result can be dangerous auto-approval, unnecessary friction, or a misleading tool catalog.

Fix this by treating MCP tool annotations as versioned product data. Test the complete path from implemented behavior to the live client's decision. This guide defines a behavior contract, descriptor manifest, hop-by-hop checks, client-policy probes, failure rules, and release receipt that catch stale or incompatible metadata without pretending hints enforce authorization.

Use one annotation-integrity transaction

A reliable release follows eight linked stages:

  1. Behavior contract: classify what the tool actually reads, changes, repeats, and reaches.
  2. Trust source: name the reviewed code and policy version that owns the classification.
  3. Server descriptor: inspect the annotation values returned by tools/list.
  4. Gateway snapshot: verify refresh and cache behavior, including annotation-only changes.
  5. SDK mapping: compare the descriptor before and after every adapter or schema conversion.
  6. Client policy: declare the approval, retry, and visibility behavior expected from each supported host.
  7. Live probe: invoke representative tools through the real integration path and observe the decision.
  8. Release receipt: record exact versions, descriptor digests, probe results, and rollback state.

Each stage consumes the previous stage's evidence. Source-level unit tests do not clear a stale gateway snapshot, and descriptor fidelity does not clear an unusable client policy. Server-side authorization remains mandatory regardless of what the client displays.

Why correct MCP tool annotations become wrong

The MCP Tools specification defines annotations as properties that describe tool behavior. It also says clients should treat them as untrusted unless they trust the server. An annotation helps a host choose presentation or confirmation behavior, but it does not prove the implementation is honest or safe.

The MCP project's guide to tool annotations as risk vocabulary explains that the four hints answer separate questions: whether a tool is read-only, whether it may destroy data, whether repeating it with the same arguments is safe, and whether it interacts with an open-world target. The guide also notes that clients use these hints unevenly. One host may request confirmation. Another may apply a stricter local policy.

Four failure classes appear between declaration and execution.

The implementation changes without the descriptor

A tool that once listed records may later export them to an external service. If the code changes but openWorldHint does not, the descriptor now understates its behavior. The same problem occurs when an additive update becomes an overwrite but destructiveHint remains false.

The descriptor changes without invalidating caches

IBM MCP Context Forge issue 5697 reports a reproducible annotation-refresh defect. The reporter changed upstream annotations while leaving the name, description, and schemas unchanged. Gateway refresh did not recognize the descriptor as changed, so the old annotations survived. This is an author report about one implementation, not proof that every gateway behaves this way. It demonstrates why annotation-only mutation belongs in your test suite.

An adapter drops or rewrites fields

Products often translate one tool model into another. A server descriptor may pass through a gateway registry, JSON serializer, SDK type, internal catalog, and host-specific converter. Any layer that copies only familiar fields can discard annotations silently. Defaults then replace explicit values, and the final client sees a different risk classification.

The client interprets a truthful hint differently

Long-run Hybrid Coach issue 399 reports that one client auto-denied a tool marked destructive, blocking a normal product-entry flow, while another client allowed it. The annotation may accurately describe a tool that writes state, yet the client policy can still be incompatible with the intended journey. Correct metadata and usable client behavior are separate assertions.

Classify behavior before writing annotations

Start from observable side effects, not the name of the operation. A tool called prepare_report may write a draft, open external URLs, or replace an existing artifact. A tool called delete_preview may only calculate what would be removed.

Build a reviewed matrix for every action-taking tool:

QuestionEvidence to inspectAnnotation consequence
Can any successful path change state?database writes, files, messages, jobs, external API callsreadOnlyHint: false
Can it remove or overwrite data?delete, cancel, replace, truncate, revoke, irreversible transitionsdestructiveHint: true
Can the same arguments repeat safely?idempotency keys, deduplication, overwrite semantics, external API guaranteesset idempotentHint from tested behavior
Can it reach outside the product boundary?public web, third-party APIs, email, payments, external storageopenWorldHint: true

Use conservative values when the implementation cannot prove a safer classification. Keep a reason and reviewer beside each value. The official MCP guidance recommends conservative defaults because optimistic hints can cause clients to skip safeguards.

Do not classify a composite tool by its common path only. If one branch sends an email and another only previews it, either mark the whole tool for the riskier behavior or split preview and commit into separate tools. The split often gives clients a useful read-only planning action without disguising the write.

Store a versioned descriptor manifest

The manifest joins implementation behavior to every emitted descriptor. It should be generated from the resolved release, then signed or hashed before distribution.

{
  "manifest_version": 4,
  "tool": "replace_customer_address",
  "behavior_contract": "customer-address@18",
  "implementation_release": "account-service@391",
  "trust_source": "product-owned-mcp-server",
  "annotations": {
    "readOnlyHint": false,
    "destructiveHint": true,
    "idempotentHint": true,
    "openWorldHint": false
  },
  "reason_codes": [
    "overwrites_existing_address",
    "idempotency_key_required"
  ],
  "reviewed_by_policy": "tool-risk-review@7",
  "descriptor_digest": "sha256:manifest-digest",
  "effective_at": "2026-09-09T08:00:00Z"
}

idempotentHint: true needs an enforceable precondition. If the tool repeats safely only when the caller supplies a stable idempotency key, require that field in the input schema and reject calls that omit it. Do not describe conditional idempotency as an unconditional property.

Keep the manifest next to the release record, not in a detached spreadsheet. A reviewer should be able to move from a live descriptor to the implementation release, behavior test, approval policy, and prior value.

Detect changes at every integration hop

Compare canonical representations rather than relying on a gateway's idea of a meaningful update. Sort keys, preserve explicit booleans, and digest the complete tool descriptor, including annotations. A change to any annotation must invalidate downstream caches and trigger the same catalog-update path as a schema change.

Use a simple hop verifier:

expected = canonicalize(reviewed_manifest.tool_descriptor)

for hop in [server, gateway, sdk_adapter, host_catalog]:
    observed = canonicalize(fetch_descriptor(hop, tool_name))
    assert observed.annotations == expected.annotations
    record(hop, descriptor_digest(observed), observed_at)

assert host_catalog.version >= reviewed_manifest.manifest_version
assert no hop substituted defaults for explicit values

Check both positive and negative changes. Flip one annotation at a time while leaving the name, description, and schemas untouched. Then prove that every hop reports a new digest. This specific fixture catches refresh logic that watches schema or description fields but ignores annotations.

Also test omission. Some SDK types distinguish false from absent; others apply defaults when a property disappears. Your comparator should report missing, explicit false, explicit true, and invalid type as separate outcomes.

The Sunpeak guide on testing MCP tool annotations recommends checking protocol output rather than source files, scanning production endpoints, and running tests in real hosts. Extend that approach across every intermediary your product operates. A passing server inspection does not prove the gateway, SDK, and client consumed the same object.

Verify descriptors, then probe client decisions

A descriptor comparison proves data fidelity. It does not prove the host behaves as intended. Define a policy expectation for every supported client and representative tool class.

Your test set should include:

  • a local, read-only lookup
  • an additive write that can be retried safely
  • a destructive overwrite or delete
  • an open-world action such as sending or publishing
  • a tool with an intentionally contradictory test descriptor
  • an untrusted third-party server with optimistic annotations

For each case, observe whether the host displays the tool, asks for confirmation, denies the call, retries it, or exposes the reason to the user. Record the client name and version because host behavior may change independently of your server.

Do not demand identical user interfaces from every client. Define acceptable outcomes instead. A destructive tool might require confirmation in one host and be disabled by policy in another. Both can be valid if the product detects the capability difference and provides a supported fallback. Silent denial, unexpected auto-approval, or infinite retry should fail the probe.

Keep annotations separate from enforcement

MCP tool annotations are not authorization checks. A direct caller can ignore them. An untrusted server can lie. A stale client can act on old values.

Every tool invocation still needs server-side controls:

  • authenticate the current principal and run
  • authorize the exact tool, resource, and operation
  • validate current tenant and object scope
  • bind approval to the exact action and arguments when approval is required
  • apply idempotency and concurrency controls at the write boundary
  • verify the resulting product state
  • emit an auditable outcome independent of the model's summary

Run a negative test that calls each protected tool directly with an unauthorized principal. The server must deny it even if every hint claims the tool is safe and read-only. This proves that metadata influences client policy without becoming a security boundary.

Release with a descriptor receipt

Do not release from a dashboard that says the latest scan passed. Produce one immutable receipt for the exact candidate release.

The receipt should contain:

  • behavior-contract and implementation versions
  • expected annotation values and reasons
  • descriptor digest from every hop
  • gateway refresh event and cache generation
  • SDK and client versions tested
  • client-policy outcomes for each fixture
  • server-side unauthorized-call results
  • exceptions with owners and expiry dates
  • final pass, fail, or incompatible decision
  • rollback release and catalog version

Treat incompatible as a real terminal result. It means the annotation is truthful but a supported host cannot complete the intended journey under its current policy. The fix might be splitting a tool, changing product flow, documenting a client requirement, or withholding that integration. Changing the hint to obtain a green test is not an acceptable fix when it would misrepresent behavior.

Handle failures and recovery

Stale gateway descriptor: Stop rollout, invalidate the affected catalog generation, and restore the last verified manifest. Do not assume a second refresh fixed the defect. Re-run the annotation-only mutation test and compare every hop.

Dropped adapter field: Fail closed for consequential tools. Patch the mapping, add explicit type coverage, and test absent versus false. Do not let an SDK default silently decide risk.

Client policy blocks a legitimate flow: Keep the truthful annotation. Split preview from commit, request an explicit approval path, or mark the host unsupported for that action. Give the user a visible explanation and recovery route.

Client auto-approves a risky flow: Disable the tool for that client or add a product-owned confirmation boundary before the side effect. Server authorization remains mandatory, but authorization alone does not prove the user intended the exact action.

Behavior changes during a rollout: Revoke the candidate descriptor version and publish a new manifest. Do not edit the old receipt in place. Active sessions should refresh the catalog before another consequential call or stop with a clear version-mismatch outcome.

Verify the complete path

Run these checks before every annotation-affecting release:

  1. Change only readOnlyHint; verify new digests at the server, gateway, adapter, and host.
  2. Repeat for destructiveHint, idempotentHint, and openWorldHint.
  3. Remove one explicit field; prove the comparator reports omission rather than treating it as false.
  4. Publish a contradictory fixture; prove policy validation rejects it.
  5. Exercise representative read, write, destructive, and open-world tools in every supported client.
  6. Confirm each approval, denial, retry, and explanation matches the declared client policy.
  7. Restart the gateway and client; prove persisted caches still resolve to the current manifest.
  8. Attempt an unauthorized direct call; prove the server denies it regardless of annotations.
  9. Roll back the tool release; prove descriptors and client behavior return to the prior receipt.
  10. Change a client or SDK version without changing the server; re-run behavioral probes and require a new receipt.

Keep the fixtures in CI and schedule bounded production descriptor scans. Production scans should compare metadata only and use safe synthetic tools or dry-run paths for behavioral probes. Never trigger a real destructive action merely to prove a hint exists.

Common mistakes

Testing annotation constants in source code checks the easiest layer and misses stale registries, serialization loss, defaults, and host policy.

Watching only names and schemas for tool changes leaves behavior metadata outside cache invalidation. Annotation-only changes need first-class versioning.

Treating every write as destructive creates approval fatigue and can make conservative clients unusable. Classify additive, reversible, and destructive behavior accurately rather than marking everything safe or everything dangerous.

Setting idempotentHint: true because retries usually work ignores conditional requirements and downstream systems. Tie the hint to tested replay behavior and required request identifiers.

Trusting first-party annotations without binding them to a release and reviewer makes trust impossible to audit. Record who or what owns the classification and when it became effective.

Changing a truthful hint to satisfy one client policy trades visible friction for hidden risk. Change the tool boundary, product flow, or support matrix instead.

Next action

Pick one read-only tool, one additive write, one destructive tool, and one open-world action from your existing product. Capture the current tools/list descriptor at every integration hop, then change only one annotation upstream. If any hop keeps the old digest or any client produces an undocumented approval outcome, block the release and add that failure as a permanent fixture.

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us