Blog · · 10 min read
Compile AI agent traces into guarded deterministic product workflows

Your support agent has already closed the same "reset sandbox and notify owner" path a hundred times. Tomorrow it will still spend a full reasoning loop rediscovering the tool order, re-looking up IDs, and occasionally inventing a different mutation order. That is how teams burn tokens and drift product state while believing they already solved the job. The fix is not another prompt. It is a product-owned path to compile AI agent traces into guarded deterministic workflows that replay without a model call, refuse unsafe guesses, and fall back to the agent when a guard fails. TraceCompiler frames the research version of that job. etchplan shows a practical compiler with promotion gates and audit logs.
This guide is for product and platform engineers embedding tool-using agents inside an existing SaaS. You will leave with a lifecycle contract, refusal rules for irreversible effects, a shadow-to-canary promotion checklist, and verification gates you can put in CI and ops.
The product contract in one pass
Treat promotion as an explicit transaction with named gates:
TraceCorpus -> PatternMine -> SideEffectGate -> ExecutionPlanCompile
-> ShadowCanary -> ServeWithAgentFallback -> AuditReceipt
- Keep a durable corpus of successful product-agent runs with tool names, arguments, results, user or tenant scope, and outcome labels.
- Mine recurring producer-consumer structure with evidence, not co-occurrence guesses.
- Gate irreversible side effects before any compile emits an executable plan.
- Compile a versioned ExecutionPlan bound to your live tool registry.
- Promote only after shadow agreement, human approval, and canary traffic.
- Serve deterministic replays in production with agent fallback and an append-only audit receipt.
etchplan states the same non-goals clearly: this is not prompt caching, response caching, observability alone, or a hand-authored workflow builder. Those tools help. They do not turn repeated agent success into a serveable product path.
Why agents keep rediscovering the same procedure
Tool-using agents mix reusable structure with retries, exploration, accidental ordering, and repeated lookups. TraceCompiler opens with that failure mode: agents repeatedly rediscover procedures they have already executed. Every rediscovery costs tokens and latency. Worse, exploratory ordering can reach irreversible APIs in a different sequence than the proven path, so "the agent usually works" is not the same as "the product path is stable."
Browser and MCP agents show the cost in the open. recast notes that agents re-reason through browser tasks at runtime, burning LLM tokens on every step forever, while proprietary recordings stay locked out of ordinary CI review. spawn describes the complementary trap: valuable multi-step patterns live only in session transcripts until someone extracts them into an explicit tool boundary.
Product teams often respond with the wrong substitute:
- Caching the last answer still leaves the next run free to choose different tools.
- Offline evals from traces prove quality, but they do not serve a workflow to customers.
- Hand-porting a Zapier scenario into an agent prompt often makes a brittle script that still pays agent overhead.
You need compilation with refusal, not another place to store text.
Capture a TraceCorpus your product can trust
Before you compile AI agent traces, decide what a retainable run looks like.
Minimum fields per tool step:
run_id,tenant_id,actor_id,intent_id- tool name as registered in the product (not a free-text synonym)
- arguments after your existing auth and schema gates
- structured result or typed error
- timestamps and attempt index
- whether the step was exploratory, retry, or accepted
Minimum fields per run:
- user-visible outcome (success, abstain, escalate, fail)
- product postconditions checked after the run
- model/prompt/tool registry versions
- whether a human edited or approved any step
Drop PII you do not need for replay bindings. Keep enough to prove a producer value actually flowed into a later consumer argument. Without that, mining collapses into "these tools often appear near each other," which is not enough for a hard edge.
Mine patterns with producer-consumer evidence
Require unique producer-consumer attribution for hard edges; frequency adjacency is not enough to order tools safely. Popular workflow miners lean on frequency or directly-follows graphs. TraceCompiler requires a stricter rule: admit an inter-tool dependency only when a consumer argument contains a value attributable uniquely to an earlier producer, and attach an auditable evidence tuple to every hard edge. Ambiguous relations stay "suspected" and impose no ordering constraint.
Bindings should be classified, not flattened:
- constants (fixed enums or product defaults)
- user inputs (must be supplied each serve)
- copied outputs (wired from a prior tool result)
- transforms (deterministic functions your runtime owns)
- residual LLM decisions (must remain agent-handled or force a human gate)
AgentJIT approaches the same habit-forming idea from coding agents: watch recurring tool sequences, emit parameterized deterministic skills, and keep a fallback when the skill fails. For a SaaS product agent, prefer TraceCompiler-style evidence edges over "it happened often" edges, especially when money, access, or customer communications are involved.
Practical rule: if you cannot point to the exact earlier field that produced a later argument, do not hard-order those tools in the compiled plan.
SideEffectGate: refuse under-determined irreversible work
Refuse compilation when an irreversible side effect is under-determined; escalate instead of baking in a guessed branch. That refusal is what separates a compiler from a liability.
TraceCompiler correctly refuses to compile a Spotify/Todoist intent when traces leave an irreversible playlist mutation under-determined. Opposite effects (add versus remove) without a resolved branch condition must not become an executable graph. The paper treats that abstention as a safety property, not a bug, and notes that deployments taking irreversible actions warrant approval gating beyond what the research evaluates.
Product translation:
- Tag every tool with an effect class: read, soft-write (idempotent upsert), hard-write (create charge, send email, delete, grant access).
- For hard-write tools, require either a uniquely attributed binding for every identity-bearing argument or an explicit human node in the plan.
- If traces show both directions of an irreversible change without a resolved predicate, refuse compilation for that intent.
- Prefer escalation to the live agent or a human approval queue over inventing a default branch.
Wire this gate to the approval and authorization systems you already run for agents. A compiled plan that bypasses per-user tool authorization is privilege escalation with a prettier name.
Compile an ExecutionPlan bound to your registry
Only after mining and the side-effect gate pass should you emit a plan.
A minimal IR sketch:
ExecutionPlan
plan_id, intent_id, version, tool_registry_hash
inputs: [user/tenant fields required at serve time]
steps:
- id, tool_name, arg_bindings, effect_class
- guards: schema, authz, entitlement, precondition
residual_llm_nodes: [decision points still delegated]
human_nodes: [irreversible or unresolved gates]
fallback: agent_entrypoint
audit_schema_version
etchplan compiles validated routines into deterministic ExecutionPlans, runs validation over trace-derived cases, binds tool names through a registry, and routes guard failures or unbound tools to an agent fallback. spawn is useful when the recurring pattern deserves a new MCP tool boundary rather than an internal plan: score buildability, generate a server with tests, and stop leaving the procedure trapped in chat history. recast covers the browser-recording case by emitting static Playwright for CI-reviewable replay without an LLM at runtime.
For an embedded product agent, keep the plan behind your existing tool registry so auth, idempotency keys, rate limits, and schema validation still wrap every call. Compilation removes model choice of tools. It must not remove product control planes.
Shadow, approve, canary: never self-deploy
Never self-deploy a compiled plan: shadow agreement, human approval, and canary must precede live serving. Compiler demos stop at the happy path. Production still needs a promotion ladder.
etchplan is explicit: nothing serves on day one. Accumulate agreement in shadow, approve once evidence clears your bar, canary in, and keep a drift kill-switch. nika#1640 asks for the same discipline as a lab contract: traces to eval to a reviewable candidate workflow diff to held-out proof, and never self-deploy.
Recommended promotion gates for an existing SaaS:
- Shadow: for matched intents, run the compiled plan without side effects (or against a dry-run tool shim) while the live agent remains source of truth. Compare tool sequences, arguments after binding, and would-be postconditions.
- Agreement bar: require a declared threshold (for example 95% exact tool+binding agreement on N held-out runs) before a human can approve. Track disagreement classes separately: missing branch, auth denial, schema fail, novel input.
- Human approval: review the plan diff, effect tags, residual LLM nodes, and refusal reasons. Approval is a product action with identity, not a chatbot "looks good."
- Canary: serve a small cohort or percentage with automatic rollback on higher fallback rate, error rate, or postcondition failures.
- Kill switch: one config flag disables the plan and restores full agent handling without a redeploy drama.
If your organization already uses feature flags for agent rollout, bind the plan's live serve bit to that system. Do not invent a second silent switch inside the compiler daemon.
Serve with agent fallback and an AuditReceipt
At serve time:
- Resolve intent and load the approved plan version pinned to a tool registry hash.
- Evaluate guards (schema, authz, entitlements, preconditions).
- Execute deterministic steps with the same idempotency and authorization wrappers as ordinary agent tool calls.
- On any guard failure, unbound tool, novel input shape, or residual LLM node, call the agent fallback and record why.
- Append an audit receipt your compliance and incident tools can read.
Suggested receipt fields:
plan_id,plan_version,registry_hashserve_mode(deterministic|fallback_agent|human)fallback_reason(enum, not free prose alone)- per-step tool, latency, authz decision id, idempotency key
- postcondition check results
etchplan emphasizes append-only audit logs for routine decisions, tool calls, and fallbacks. That log is how you prove a customer-facing automation was the compiled path or the agent path when something goes wrong.
Worked example: sandbox reset in an existing product
Intent: reset_customer_sandbox.
Observed successful agent pattern across many tickets:
lookup_customer(customer_id)list_sandboxes(customer_id)-> pick active sandbox idreset_sandbox(sandbox_id)(hard-write)notify_owner(customer_id, channel="email")(hard-write)
Mining should only hard-order step 3 after step 2 if sandbox_id in the reset call is uniquely attributable to the list result for that customer. If traces sometimes reset a sandbox id typed by the user and sometimes one from list_sandboxes, keep that binding as residual or require a human confirmation node.
SideEffectGate notes two irreversible effects: reset and email. If traces sometimes notify via email and sometimes via Slack without a resolved predicate, refuse to compile the notify step as deterministic. Leave notify as residual LLM or force an explicit channel input.
After compile, shadow the plan on the next 200 matching tickets. Approve only when bindings and effect order match. Canary at 5% of eligible tenants. If fallback spikes because a new sandbox type appears, the kill switch returns those tenants to the agent while you re-mine.
Handle failure and recovery
Compilation does not erase failure modes. It changes which layer owns them.
- Guard failure at serve: fall back to agent; do not retry a hard-write blindly.
- Tool error mid-plan: use your existing idempotency and compensation rules; a compiled plan is not a license to double-charge.
- Drift: if shadow agreement decays after a tool schema change, auto-disable the plan when
tool_registry_hashno longer matches. - Under-determined new branch: refuse to extend the plan automatically; queue a re-mine with held-out proof, following the spirit of nika#1640.
- Human override: record overrides as first-class outcomes so mining does not treat them as ordinary success traces without the edit marker.
Verify the solution
Ship a checklist operators can run:
- Corpus retention includes tool args, results, versions, and postconditions for accepted runs.
- Hard edges in any compiled plan cite unique producer-consumer evidence; suspected edges never force order.
- Irreversible tools without resolved bindings refuse compile or require a human node.
- Held-out validation runs before any approval (TraceCompiler-style leave-one-out thinking, even if your test runner is simpler).
- Shadow agreement meets a written bar; disagreements are triaged by class.
- Live serve requires human approval plus canary; no daemon self-deploy path exists.
- Every serve emits an audit receipt with plan version and fallback reason.
- Product authz, entitlements, schema validation, and idempotency still wrap deterministic tool calls.
- Kill switch restores full agent handling within one config change.
- CI fails if a plan references a tool name absent from the current registry hash.
Common mistakes
- Compiling free-text coding churn where arguments are prose or shell, which etchplan usually refuses for good reason.
- Using frequency adjacency as a hard edge and shipping a wrong order into production.
- Compiling opposite irreversible effects because "the model usually picks correctly."
- Serving on day one without shadow evidence.
- Letting compiled plans bypass the same authorization and idempotency wrappers agents use.
- Treating eval datasets as a substitute for a serveable ExecutionPlan.
- Generating MCP servers or Playwright suites without a promotion review when the workflow can mutate customer state.
What to do next
Pick one high-frequency, low-ambiguity intent in your product agent this week. Export twenty successful traces with tool args and postconditions. Try to draw producer-consumer edges with unique attribution. If an irreversible step is under-determined, write the refusal down instead of forcing a compile. If the graph is clean, implement shadow-only comparison against the live agent before anyone discusses canary traffic.
If you are already assembling support or product agents from your own knowledge base and tools with MakeYourAgent, keep those tool calls behind the same registry, auth, and audit hooks you will need when a compiled plan starts serving. The compiler should sit beside that control plane, not replace it.