Blog · · 10 min read

AI agent explainability: build evidence-based action records

AI agent explainability: build evidence-based action records

When an agent refunds the wrong invoice or closes the wrong ticket, a fluent paragraph about its reasoning does not establish what happened. AI agent explainability needs stronger evidence: the request the product resolved, the records it read, the policy it applied, the tool it called, and the result the product verified. Capture those facts during execution, then render an explanation from them. This gives users and support staff a useful account without exposing hidden chain-of-thought or trusting a model to invent a plausible story after the action.

An explanation contract built this way survives retries, partial failures, permission changes, and model upgrades. Your team can also check whether each explanation is complete and faithful.

Why generated reasoning is the wrong source of truth

An explanation has two jobs that are easy to conflate. It must help a particular reader understand the result, and it must remain faithful to the process that produced that result. NIST's four principles for explainable AI separate these concerns: an explanation should provide reasons or evidence, be meaningful to its audience, accurately reflect the system's process, and state relevant knowledge limits.

Asking the model to "explain your decision" after a tool call satisfies none of those properties by itself. The model may not have access to the exact policy version, database record version, approval state, retry history, or postcondition check. Coherent prose can still describe a different path from the one that executed.

Chain-of-thought is not a substitute. The submitted paper Thoughts without Thinking reports early evidence that chain-of-thought alone does not provide useful end-user explainability in a modular agent pipeline. A current Vector Institute survey repository likewise treats agent transparency as a broader problem involving trajectories, provenance, causal interventions, governance, user-centered explanation, and evaluation.

Raw traces do not solve the reader problem either. They are built for operators, contain implementation details, and may expose personal data or another tenant's records. Keep the trace, evidence, and reader-facing explanation related but separate:

  1. The trace records runtime events for engineering diagnosis.
  2. The evidence record stores the exact facts needed to reconstruct the action.
  3. The explanation projects an authorized subset of those facts for one audience.

Define an AI agent explainability contract

Create an ExplanationRecord when the run resolves an actionable intent. Append evidence as the action advances rather than reconstructing it from logs at the end. The following schema is an implementation example, not a standard:

{
  "explanation_id": "exp_01",
  "run_id": "run_01",
  "request": {
    "intent": "refund_invoice",
    "target_ref": "invoice:8432",
    "unresolved_fields": []
  },
  "evidence": [
    {
      "source_ref": "invoice:8432",
      "source_version": "17",
      "purpose": "verify payment and refund eligibility"
    }
  ],
  "policy_decisions": [
    {
      "policy_ref": "refund-policy:9",
      "decision": "approval_required",
      "reason_code": "amount_over_self_service_limit"
    }
  ],
  "approval_ref": "approval:5521",
  "action": {
    "tool": "refund_invoice",
    "arguments_view": {"invoice_id": "8432"}
  },
  "receipt_ref": "receipt:9904",
  "outcome": {
    "state": "completed",
    "postcondition": "refund_id_present"
  },
  "knowledge_limits": [],
  "recovery_actions": ["open_refund", "contact_support"]
}

Match the field names to your product model. The contract joins six pieces that usually live in separate systems:

  1. Resolved request. Record the typed intent, target object, and any unresolved fields. Do not copy the user's message and assume it identifies the action.
  2. Versioned evidence. Store stable references and versions for records that materially affected the decision. Avoid copying entire records into the explanation store.
  3. Policy decisions. Capture the policy identifier, version, outcome, and a stable reason code. Keep generated prose out of this layer.
  4. Approval. Link the exact approval object, approver scope, and approved action version when approval was required.
  5. Action receipt. Link the tool invocation to its durable receipt, including idempotency identity and a sanitized argument view.
  6. Verified outcome. Record the postcondition result, not only the tool's success response.

The W3C Provenance Data Model provides useful vocabulary for these relationships. Product records are entities, tool execution is an activity, the acting user and agent are agents, and usage or generation edges connect the evidence to the result. You do not need to implement the full model to borrow its discipline: every important assertion should point to an entity, activity, or agent that the product can identify.

Capture evidence along the execution path

Do not make the explanation renderer scrape traces. Give the runtime explicit write points and reject explanations whose required evidence is missing.

Before planning

Resolve the request into a typed intent and authoritative product target. Record unresolved or conflicting fields. If the system still does not know which invoice the user meant, the explanation state should be needs_clarification; no action explanation should pretend a choice was made.

Record only the context that materially influenced the candidate action. Each entry needs a source reference, source version, retrieval time, and purpose. This keeps "the agent saw account data" from becoming an unauditable claim.

Before the side effect

Run deterministic authorization and policy checks on the typed action. Persist stable decisions such as allowed, denied, or approval_required, together with policy versions and reason codes. If the user approved a preview, bind the approval to the same action identity and arguments that will execute.

Decide at this boundary what the product can later explain. A rule such as amount_over_self_service_limit can safely support reader-facing copy. A hidden prompt fragment or unrestricted reasoning transcript cannot.

After the tool call

Write a receipt whether the tool succeeds, fails, times out, or returns an ambiguous result. Then perform a product-owned postcondition check. A payment provider returning HTTP 200 is weaker evidence than finding the expected refund identifier and state in the authoritative billing record.

Decision provenance in agentic systems distinguishes ordinary observability from evidence that reconstructs why an action occurred. A user-facing product still needs to turn that provenance into a bounded explanation with recovery controls.

Walk through a refund example

Suppose a customer writes, "Refund the duplicate annual charge." The account has two invoices, and only one has a duplicate payment marker.

The intent resolver produces refund_invoice and resolves invoice:8432 from the current account, not from a model-generated identifier. Context assembly records invoice version 17 and the duplicate-payment marker. Authorization confirms that the requesting user owns the billing account. Refund policy version 9 requires approval because the amount exceeds the self-service limit. The user approves a preview bound to invoice 8432 and the displayed amount.

The refund tool then returns receipt 9904. A postcondition read finds the new refund identifier on invoice version 18. The end-user explanation can say:

Invoice 8432 was refunded because the account record marked its payment as a duplicate. You approved the $240 refund after policy 9 required confirmation. The billing record now shows refund 9904 as submitted.

Each sentence maps to stored evidence. The renderer did not ask the model why it chose the invoice. If the postcondition read timed out, the final sentence would instead say that submission was received but confirmation is still pending, with a link to check status. That is a knowledge limit, not a generic apology.

Render a different view for each audience

The right explanation depends on the reader. NIST notes that one explanation may not suit every user. Build views from the same record rather than exposing one universal blob.

For end users, answer: what did the agent understand, what did it do, why was that allowed, what changed, what remains uncertain, and what can I do next? Use product names and familiar objects instead of tool or policy internals.

Support staff may also need the run ID, evidence versions, reason codes, retry state, and recovery controls. The view must still apply the current support agent's tenant and field permissions. A support role that cannot view a protected medical field must not receive that field merely because it influenced an automated decision.

An audit view can include policy and approval histories, provenance edges, retention state, and integrity metadata. It should not automatically include hidden model reasoning or unrestricted prompt content. Its purpose is to expose accountable evidence without collecting every available detail.

Apply authorization when rendering, not only when writing. Permissions can change between execution and later review. Replace a hidden source with a truthful marker such as "A restricted account record affected this decision" rather than silently omitting the fact or leaking its value.

Explain denials and partial failures

Model failure states directly. Otherwise, the explanation disappears when users need it most:

  • denied: name the blocked action, applicable policy reason, and a permitted next step.
  • needs_clarification: show which field is unresolved and ask one bounded question.
  • approval_expired: identify the action version that was approved and explain why a changed action needs new approval.
  • submitted_unverified: state that a tool accepted the request but the product could not confirm the effect.
  • partially_completed: list verified effects separately from failed or unknown effects.
  • compensated: show the original action and the verified reversal.

The 2026 paper Toward Safe and Responsible AI Agents treats transparency, accountability, and trustworthiness as operational requirements. Typed outcomes let the explanation guide recovery because the runtime can distinguish a rejection, timeout, unknown result, and verified completion.

Test fidelity before testing prose

A readable explanation can still be wrong. Start with deterministic invariants, then test whether people understand the result.

For each fixture, assert that:

  1. The explanation's action ID matches the receipt and approval action ID.
  2. Every material source has an authoritative reference and captured version.
  3. Every displayed reason code exists in the policy decision set.
  4. The displayed outcome matches the postcondition state, including unknown and partial outcomes.
  5. Rendering under each role reveals no field that role cannot currently access.
  6. A retry links to the same logical action without creating two conflicting explanations.
  7. A policy or source-version mutation changes the projected explanation where expected.
  8. Missing required evidence blocks a definitive explanation instead of triggering generated filler.

Then test comprehension with representative users. Ask them what the agent did, why it did it, whether the outcome is confirmed, and what action they can take next. The AI Accelerator Institute panel on autonomous-agent transparency emphasizes that explanation needs vary by domain and user. A support engineer may need provenance detail that would only confuse an end user.

A direct Hacker News practitioner question describes a related operational problem: tool and vendor logs fragment the decision chain, leaving teams unsure whether to retain context, actions, or reasoning. Treat that report as a problem example, not prevalence data. Define the evidence contract at the product boundary so no vendor's trace format becomes the explanation schema.

Avoid the common implementation traps

The model may turn structured reason codes into prose, but it must not author the record. Source fields need to come from runtime components that own the facts.

Raw traces are a poor user interface. They mix irrelevant detail with sensitive data and still require interpretation.

Store references, versions, purposes, and integrity metadata by default. Duplicating complete source records creates another retention surface.

Preserve uncertainty in the schema and the copy. "The tool returned success" and "the product verified the effect" are different states.

Separate views still need the authorization, tenancy, retention, and deletion controls of the underlying product data.

Instrument one action this week

Choose one high-consequence tool action that already has an approval or recovery path. Define its required request, evidence, policy, approval, receipt, postcondition, limit, and recovery fields. Capture them at the runtime boundaries, then write fixtures for success, denial, timeout, partial completion, permission loss, and retry.

Add prose generation only after those fixtures pass. Keep a deterministic renderer as the baseline. If a model rewrites the explanation for readability, require every sentence to map back to allowed fields in the same ExplanationRecord. The product can then prove its AI agent explanation instead of trusting the model's story.

References

Be first in line.

Join the waitlist and we'll email you the moment it's ready. No sales call.

Try the demo →
or talk to us