Blog · · 8 min read
AI agent knowledge base freshness: keep answers fresh when help docs and policies change

Your pricing page changed three weeks ago. The help center article is already updated. The support agent still quotes the old plan. The model is not inventing a price. It is repeating text that used to be true, which is why Twig treats outdated answers as a separate accuracy failure from invented ones. This guide gives product and platform engineers a concrete AI agent knowledge base freshness contract: register sources, detect changes, promote a candidate index only when grounded checks pass, tombstone deletions, bind citations to live fingerprints, and disclose degraded retrieval when coverage drifts.
Build a corpus lifecycle, not a nightly re-embed job
Treat the knowledge corpus as a production dependency with an explicit lifecycle:
- SourceRegistry. Every help-center article, policy, pricing feed, and release note has an owner, class, TTL, and canonical URL.
- ChangeDetection. Webhooks, git merges, and release events enqueue affected sources; scheduled scans catch silent drift.
- CandidateIndex. Rebuild or patch a candidate corpus, never mutate the live index in place.
- FreshnessGate. Score content age, embedding lag, stale retrieval rate, and coverage drift before promotion.
- CitationBinding. Answers may cite only request-scoped keys that resolve to a current content fingerprint.
- DegradedRetrieval. When the gate fails, narrow scope and tell the user the corpus may be incomplete.
- AtomicPromotion. Swap candidate to live with a reversible version, then hard-delete tombstoned vectors off-peak.
Freshness metric guides and sync product pages usually stop at one layer. Product teams still need that full transaction inside an existing SaaS agent.
Why stale docs produce confident wrong answers
Retrieval-augmented generation suppresses uncertainty. When any document returns, the model treats presence as validation. Google Research shows that insufficient context sharply increases incorrect answers, and Tian Pan applies that finding to outdated help-center corpora: stale context raises confident wrong answers rather than honest abstention.
Three mechanisms make the failure silent:
- Pipeline lag. Editors update a refund policy at 14:00. A nightly batch re-embeds at midnight. Every matching query for ten hours retrieves the old vector with a healthy similarity score. Atlan calls this embedding lag, and it is independent of whether a human already fixed the source.
- Semantic preference for older detail. An older policy often has richer wording and historical click weight, so pure similarity can rank it above the shorter current page. Temporal validity is not a default retrieval feature.
- Missing deletions. Soft removal from the CMS without a tombstone leaves vectors that still cite as authority.
Practitioner reports match the mechanism. Authors on Kiro issue 6811 describe a product agent confidently explaining features from stale training rather than current docs. OpenAgents issue 4836 records agents consuming an announced registry version before the live endpoint carried freshness metadata. Treat both as author reports, not universal statistics.
Register sources before you measure freshness
Without metadata, freshness scoring is impossible. Minimum fields per document:
source_id: helpcenter.refund-policy
canonical_url: https://help.example.com/refunds
owner: support-ops
document_class: critical # critical | reference | contextual
updated_at: 2026-09-18T11:02:00Z
last_verified: 2026-09-18T11:30:00Z
valid_until: null # or explicit TTL
superseded_by: null
content_fingerprint: sha256:...
embedding_computed_at: 2026-09-18T11:05:12Z
certified_for_ai: true
Class-specific thresholds follow Atlan's production framing: near-zero lag for compliance and live pricing, roughly 24 hours for policies and procedures, longer windows for evergreen reference material. Put thresholds on the document, not as one corpus-wide cron.
Change detection should combine:
- Help-center publish webhooks (Zendesk, Intercom, Confluence, and similar)
- Git merges for docs-as-code
- Product release events that flag related articles for review
- Hash comparison when timestamps lie
- A weekly full scan to catch deletions and coverage drift
Twig treats automated sync as the foundation. Sync alone is not enough: you still need a promotion gate and citation binding.
Score freshness as four signals, not one age field
Atlan separates four dimensions. Implement all four in the product runtime:
| Dimension | Question | Failure if ignored |
|---|---|---|
| Content age | Has a human verified the source within its class TTL? | Old policies stay certified |
| Embedding lag | Is the indexed vector newer than updated_at? | Edited docs still serve old answers |
| Stale retrieval rate | What share of live queries returned a lagging embedding? | You only see theory, not user harm |
| Coverage drift | What share of the corpus is past threshold even if unused today? | Tomorrow's query detonates today's silent debt |
Faithfulness scores can stay green while the retrieved corpus is already wrong, so freshness needs its own metrics. Faithfulness asks whether the answer stayed inside retrieved context. It does not ask whether the context is still true. If your eval suite only tracks faithfulness, you will ship a confidently wrong help agent after every pricing change. Add freshness scorecards beside RAGAS-style metrics, and alert when a composite score drops below your operating threshold (Atlan cites roughly 85% for alerts and 70% for degraded application warnings as practitioner starting points, not universal law).
Log every retrieval with source_id, embedding_computed_at, and current updated_at. Stale retrieval rate is then a rolling window comparison, not a quarterly audit guess.
Soft-delete with tombstones before you drop vectors
Soft-delete tombstones before hard vector removal, or deleted policies keep getting cited. When a policy is removed from the help center, do not delete its vectors in the same request that removes the CMS row. Mark deprecated: true, clear certified_for_ai, set superseded_by when a replacement exists, and keep the fingerprint for citation denial. Tian Pan calls out the dangerous window where the source is gone but embeddings remain authoritative. Batch hard deletes off-peak after promotion proves no live answer still needs the old locator for audit.
Pre-filter before similarity:
candidates = index.search(query, k=50)
candidates = [d for d in candidates
if d.certified_for_ai
and not d.deprecated
and (d.valid_until is None or d.valid_until > now)
and d.embedding_computed_at >= d.updated_at]
Then apply temporal reranking so recency shares weight with similarity for policy, pricing, and feature docs. Pure semantic ranking is insufficient for time-sensitive product truth.
Promote a candidate index, not a hot patch
Promote a candidate index only after grounded-answer checks pass, not merely after embeddings finish. Build a candidate index from the source registry, run freshness scorecards, run a frozen set of gold support questions for grounded answers and citation resolution, then promote atomically.
Recommended promotion checklist:
- Every changed source maps to an allowed domain and current access policy.
- Required critical sources are inside TTL; removals are tombstoned.
- Stale retrieval rate on a shadow replay stays under threshold.
- Gold questions keep citation coverage and answer accuracy above baseline.
- Rollback points at the previous live index version.
If checks fail, keep serving the previous live index and open a content-debt ticket to the document owner. Do not force a half-built corpus into production traffic.
A related control is pre-action revalidation of mutable dependencies immediately before a write, as in FreshCtx. Keep that TOCTOU guard for tool side effects. Keep help-center corpus promotion as a separate lifecycle. Both can coexist: corpus freshness decides what the agent is allowed to know; FreshCtx-style checks decide whether a specific observation is still true at act time.
Bind citations to live fingerprints
A citation is a request-scoped key your application issued for a retrieved passage, not a title the model invents. Allow the model to choose among those keys. Deny free-form URLs, document IDs, and quotes that were not supplied.
Before rendering an answer:
- Resolve each cited key to
source_idandcontent_fingerprint. - Reject keys absent from this turn's retrieval set.
- Reject fingerprints that no longer match the registry (stale citation).
- Reject deprecated or uncertified sources.
- If any factual claim lacks a surviving citation, abstain or qualify that claim.
AI agent knowledge base freshness becomes user-visible at click time. A customer who follows a citation should land on the same version the model saw, or the product should refuse the citation rather than decorate a wrong answer with a broken link.
Disclose degraded retrieval instead of silent confidence
When coverage drift or embedding lag crosses your degrade threshold:
- Narrow retrieval to
document_class: criticalsources that are still certified - Prefer structured product APIs for pricing and entitlement over prose when available
- Tell the user the knowledge base may not reflect the latest change, and offer a human handoff or canonical help link
- Emit an operator alert with the failing sources and owners
Degraded mode tells the truth when the corpus cannot. Shipping nothing can be safer than shipping a confident wrong refund policy, which is the legal and trust lesson Tian Pan draws from public chatbot misrepresentation cases.
Verify the freshness contract
Automate fixtures that prove the contract:
| Fixture | Expected outcome |
|---|---|
| Same-day help-center update | Embedding lag clears inside the class SLA; old vector is not retrieved |
| Deleted policy | Tombstoned source cannot be cited after promotion |
| Contradictory version pair | Pre-filter or temporal rank prefers current; Franken-answers are blocked |
| Embedding-lag window | Stale retrieval rate rises and alerts fire before customers complain |
| Failed gold-question gate | Candidate is not promoted; live index unchanged |
| Citation to missing fingerprint | Answer withheld or claim qualified |
| Coverage-drift threshold | UI discloses degraded mode; critical-only retrieval engages |
| Promotion rollback | Previous index version restores within RTO |
Also re-run a small production-trace sample weekly: high retrieval plus low satisfaction on a document is a staleness signal, as Twig notes, even when timestamps look fine.
Common mistakes
- Nightly full re-index as the only control. It creates a predictable staleness window and still misses soft deletes.
- Measuring only faithfulness. Green evals on stale context hide the real regression.
- Hard-deleting vectors immediately. Audit and late citation checks need tombstones first.
- One TTL for the whole corpus. Pricing and evergreen concepts do not share a shelf life.
- Treating sync webhooks as promotion. Ingestion is necessary; grounded-answer gates decide safety.
- Letting the model invent citations. Free-form references bypass fingerprint checks.
- Confusing memory poisoning controls with corpus freshness. Adversarial memory writes are a different lifecycle from help-center drift.
What to do next
Pick your highest-traffic support or product agent and inventory every source it can retrieve. Add owners, classes, fingerprints, and embedding_computed_at this week. Wire help-center publish events into a candidate-index pipeline with a gold-question gate before promotion. Until that gate exists, turn on degraded disclosure for pricing and policy intents rather than trusting last month's vectors.
If you are building product agents on Make Your Agent, keep the knowledge base you upload under the same release discipline as the product: treat docs changes as deploy-adjacent work, not a content chore that can wait until the next quarterly cleanup.