Blog · · 9 min read
Tenant-specific AI agent configuration management without runtime forks

AI agent configuration management breaks when a SaaS product treats customization as an editable system-prompt field. One customer override can remove a platform rule. A shared default update can break hundreds of customized agents. A copied configuration can strand a tenant on old tools and policies while the rest of the product moves forward.
The fix is to compile typed configuration layers into one immutable effective manifest before any run starts. That manifest must show where every value came from, enforce platform invariants, identify affected tenants, pass tenant-specific fixtures, support cohort rollout, and make rollback semantics explicit. This article defines that contract without requiring a separate runtime or deployment for every customer.
Why tenant customization turns into runtime forks
A configurable agent has more than one legitimate authority. The platform owns safety rules, supported models, tool boundaries, and global operating limits. A tenant owns business instructions, enabled workflows, approved knowledge sources, and local thresholds. A user may own preferences. A request carries temporary context.
String concatenation hides those boundaries. The runtime receives one prompt, but the product cannot reliably answer which layer supplied an instruction, whether a lower layer replaced a required rule, or which tenants inherited a changed default. TianPan's analysis of the multi-tenant prompt problem describes priority inversion, contradictory layers, and invisible blast radius when a platform prompt changes.
Copying the whole default into each tenant record looks safer because the result is easy to inspect. It creates a different failure. Every customized tenant now owns a frozen snapshot. New platform defaults, corrected tool descriptions, and policy changes no longer propagate. The product has effectively forked even if every tenant still uses the same application binary.
Zylos documents this exact distinction in its analysis of default, override, and append semantics. Append semantics can preserve improvements to a maintained baseline, while full overrides can strand customers on older content. Append is not universally correct, though. Appending two output schemas or two model selections produces nonsense. Composition must be defined per field type.
Use one shared runtime that executes many explicit effective configurations. Avoid both a single shared prompt and thousands of unmanaged copies.
Define authority and merge semantics per field
Start with a schema, not a prompt editor. For every configurable field, declare its owner, allowed scopes, composition operator, validation rule, and whether a tenant may disable it.
| Field | Platform behavior | Tenant behavior | Composition rule |
|---|---|---|---|
| Safety policy | Mandatory baseline | May add stricter rules | Union, never subtract |
| Tool catalog | Defines supported tools | Enables an approved subset | Intersection |
| Knowledge sources | Defines connector types | Selects tenant-owned sources | Scoped union after authorization |
| Behavioral guidance | Supplies maintained defaults | Adds business-specific guidance | Ordered append with length budget |
| Output schema | Supplies supported schemas | Selects one or supplies a valid replacement | Replace after schema validation |
| Model | Defines approved pool | Selects within pool | Constrained selection |
| Cost and turn limits | Defines hard ceilings | May set lower limits | Minimum |
| Feature flags | Defines available capabilities | Enables eligible capabilities | Policy evaluation |
The product must enforce these decisions before prompt assembly. A phrase such as "never disclose private records" remains vulnerable if its only protection is an earlier position in a system prompt. Use deterministic authorization and data filtering for enforceable boundaries. The OpenAI Model Spec instruction hierarchy provides a useful authority model. Prompt precedence still cannot replace runtime policy.
Represent the schema as versioned code so a pull request can review a changed operator. A new tenant field should not default to unrestricted replacement. Unknown fields, unknown enum values, and attempts to broaden a hard limit should fail compilation.
fields:
enabled_tools:
owner: platform
tenant_operator: intersect
validator: approved_tool_ids
behavior_guidance:
owner: shared
tenant_operator: append
validator: prompt_policy_and_token_budget
output_schema:
owner: tenant
tenant_operator: replace
validator: supported_json_schema
max_tool_calls:
owner: platform
tenant_operator: lower_only
hard_max: 12
The schema exposes dangerous combinations before runtime. The tenant may narrow tools but cannot add an unreviewed tool. It may lower a call budget but cannot raise the platform ceiling. It may replace an output schema only if the runtime and downstream product support that schema.
Compile one inspectable effective manifest
Resolve configuration before dispatch, then store the result as an immutable manifest. Do not reconstruct it later from whatever happens to be current in five mutable tables.
The manifest should include:
- platform baseline release and policy bundle;
- tenant configuration revision;
- selected workflow and user-preference revision;
- resolved model snapshot, tools, knowledge sources, budgets, and schemas;
- provenance for every field;
- compiler version and validation results;
- a content hash used as the effective configuration ID.
{
"effective_config_id": "cfg_7f3b",
"platform_release": "agent-platform-42",
"tenant_config_revision": "tenant-acme-17",
"workflow": "renewal-review",
"resolved": {
"model": "approved-model-snapshot",
"enabled_tools": ["read_contract", "draft_renewal"],
"max_tool_calls": 8,
"output_schema": "renewal-draft-v3"
},
"provenance": {
"enabled_tools": ["platform:catalog-19", "tenant:allowlist-17"],
"max_tool_calls": ["platform:12", "tenant:8"]
},
"validation": {
"schema": "pass",
"authorization": "pass",
"fixture_suite": "tenant-acme-renewals-6"
}
}
Expose a redacted version in the tenant admin UI. Operators need a computed-config view equivalent to a deployment dry run: what is inherited, what is customized, what is blocked, and what will change under the proposed baseline. Zylos identifies this resolved view as necessary because layered configuration becomes illegible without inspection tooling.
Never expose secret values in the manifest. Store references to credential policies or brokered connections. The run record should bind to the manifest ID and the current user authorization decision, not copy reusable credentials into configuration.
Reject invalid composition before a model call
Treat compilation as a deterministic gate. A useful sequence is:
layers = load_versioned_layers(platform, tenant, user, workflow, request)
assert_layer_owners(layers)
resolved = compose_by_field_schema(layers)
assert_platform_invariants(resolved)
assert_tenant_resource_ownership(resolved)
assert_tool_and_model_compatibility(resolved)
assert_context_and_cost_budgets(resolved)
manifest = persist_immutable_manifest(resolved, provenance=layers)
run_tenant_fixtures(manifest)
promote_only_if_all_required_checks_pass(manifest)
Fail closed on a missing policy bundle, unknown tool version, deleted knowledge source, unsupported model capability, or invalid output schema. Do not silently fall back to the broad platform default if that would expand authority. A safe fallback may remove a capability, lower autonomy, or route the request to human review.
Tenant identity must also survive deployment and resolution. A Microsoft Agents issue reported incorrect tenant assignment when multiple agent instances were deployed across multiple tenants. That report does not establish prevalence, but it shows why tenant identity belongs in compilation tests and run evidence rather than an assumed deployment parameter.
Add negative tests for each operator. Attempt to enable a tool outside the approved pool, raise a hard limit, select another tenant's knowledge source, remove a mandatory policy, and submit a schema the downstream product cannot parse. Each test should prove that no manifest and no runnable job are created.
Calculate blast radius before changing a default
A platform default is a dependency of every tenant that inherits it. Before promotion, compile the proposed baseline against all active tenant revisions or a risk-based superset that includes every distinct override shape. Produce an affected-tenant report:
- unchanged effective configuration;
- behavior-affecting change;
- newly invalid configuration;
- tenant override now redundant;
- tenant override now contradictory;
- manual migration required.
Run representative tenant fixtures against the old and proposed manifests. Fixtures should verify tool selection, forbidden actions, output schema, knowledge-source scope, budget exhaustion, and final product state. Include tenants with no overrides, additive guidance, full artifact replacements, older revisions, and the maximum allowed configuration size.
Microsoft's Copilot Studio application lifecycle guidance uses separate development, test, and production environments, promotes the same managed artifact, binds environment-specific configuration during deployment, and recommends evaluations before and after import. Apply the same discipline to your shared baseline plus tenant deltas. Testing only the platform default misses the configuration combinations that customers actually run.
Promote by cohort and bind every run
Publish a proposed configuration release as an immutable artifact. Promote it first to internal tenants, then low-risk customers, then broader cohorts. Each cohort should have explicit entry criteria, monitored outcomes, and a rollback pointer.
AWS SaaS guidance recommends one universal mechanism for tenant-specific customization instead of deploying and operating separate versions for each tenant. A shared compiler and manifest model supplies that mechanism while still producing tenant-specific outcomes.
At dispatch, record the effective configuration ID on the durable run. Retries and resumed work must use the same manifest unless a defined migration explicitly changes it. Otherwise a paused workflow can resume under different tools or policy because a tenant administrator edited a setting while it waited.
For every cohort, monitor configuration rejection rate, fixture regressions, verified task outcomes, policy interventions, tool-path changes, cost, and support reports. Compare the proposed release with the previous effective release for the same tenant segment. Aggregate health is insufficient if one override shape fails.
Make rollback and inheritance separate decisions
Rollback has two components:
- Move the platform release pointer to a known-good baseline.
- Recompile each affected tenant delta against that baseline.
Do not restore a stored final prompt and call that rollback. It may reference retired tools, deleted knowledge, stale policies, or an unsupported schema. Recompilation proves that the old baseline and current tenant revision remain compatible.
Define what happens when a new default conflicts with an existing tenant delta. Record one outcome: keep the delta, migrate it, disable the affected feature, pin that tenant temporarily, or require administrator review. Never resolve the conflict silently.
Microsoft's ALM guidance notes that deployment history is not automatically a one-click rollback and that recovery can require deploying a previous managed version. Your control plane should therefore test rollback before release, retain previous manifests, and keep required tool and policy versions available for the stated rollback window.
A temporary tenant pin needs an owner and expiry. Permanent pins recreate the fork you were trying to avoid. Track the reason, required migration, deadline, and blocked platform improvements. Alert when a pin approaches expiry or when the pinned release loses security support.
Verify the full tenant configuration lifecycle
Before production promotion, require evidence for this checklist:
- Every field has an owner and one composition operator.
- Platform invariants cannot be weakened by a lower layer.
- Cross-tenant resource references fail before dispatch.
- The effective manifest is immutable, inspectable, and linked to each run.
- Secrets are represented by references, not values.
- The affected-tenant report covers every active override shape.
- Old and proposed manifests pass required tenant fixtures.
- Cohort assignment is stable through retries and resume.
- Rollback recompiles current tenant deltas against the prior baseline.
- Temporary pins have owners, deadlines, and migration plans.
- Negative tests prove rejected configuration creates no executable work.
Then run a controlled exercise. Change one shared tool description, tighten one hard budget, remove one knowledge connector, and introduce one contradictory tenant instruction. The system should identify affected tenants, reject the invalid combination, preserve unaffected manifests, promote only the tested cohort, and return to the previous baseline without losing tenant-owned deltas.
Pick the configuration field that currently causes the most support work. Write its ownership and merge rule, compile it into an effective manifest, and add one negative fixture. Add the next customization field only after that path can prove what ran, why it was allowed, and how it rolls back.