RAG's Hardest Bug Is Not Hallucination: It Is Authorization Drift

Stale context produces confident wrong answers , which is an accuracy bug. Over-privileged context produces an unauthorized disclosure, which is a reportable security breach. In enterprise retrieval and agentic search, source permissions mutate continuously across Google Drive, SharePoint, Jira, Slack, and relational stores: an executive removes a director from an acquisition folder, legal places a hold on a customer account, or a contractor’s project grant expires at 5:00 PM. Meanwhile, the vector index, the semantic response cache, and the user’s open conversation session keep serving yesterday’s access graph. Teams spend months tuning chunking and rerankers for retrieval quality while their authorization model relies on an hourly cron job that syncs ACL arrays into vector metadata.

Closing authorization drift requires treating read permissions as a live runtime invariant across four distinct surfaces: candidate retrieval, response caching, active session state, and continuous verification. Syncing document ACLs into vector-index metadata is useful as a coarse pre-filter, but ingestion queues lag, webhook deliveries drop, and nested group expansions change without touching the document timestamp.

On the candidate retrieval surface, if you fetch only the top K chunks (say, K = 8) from the index and filter unauthorized chunks afterward, any user whose top matches happen to sit inside a recently restricted folder gets an empty or starved prompt. The production pattern is over-fetch plus late-binding verification: query the retrieval index for K' candidates (K' = 4K, with a bounded cursor fallback when filter selectivity is low), batch-verify live read permissions on those candidate source_ids against your authoritative policy engine or identity graph (such as a ReBAC service or short-TTL permission store) in a single RPC before prompt assembly, and pass only the top K authorized survivors into the context compiler.

The second surface is the semantic or response cache, where well-meaning performance work routinely creates a privilege escalation path. Many platform teams scope their response cache keys by principal_id + acl_group_hash to stop User A from reading a cached answer generated for User B. That key still fails for the same user whenever a single document’s direct share or folder ACL is revoked without the user’s group memberships changing: the user’s acl_group_hash is identical at 10:05 AM to what it was at 10:00 AM, so the cache happily returns a synthesized answer built from the document they just lost access to. Never trust a principal or group hash alone to authorize a cached completion. Record the exact set of contributing_source_ids on every cached retrieval bundle and synthesized response entry, and run the same batch live-permission check against those source_ids on every cache hit before returning a single byte to the caller, while keeping only user-agnostic system instructions and tool schemas in the model provider’s shared prefix KV cache.

The third surface is the active multi-turn session, which is where most enterprise RAG audits fail on follow-up questions and long-running turns. Suppose an analyst retrieves a restricted board deck at 10:00 AM, asks the agent to summarize its margin tables into a workspace file at 10:02 AM, and loses access to the deck at 10:05 AM. Even if your vector index and response cache block the deck on a fresh query at 10:06 AM, the text of that deck is still sitting inside the analyst’s active conversation history, compaction summary, and sandbox scratch files. This is why separating the append-only session event log from the compiled context window is a security requirement, not just a caching optimization: by propagating contributing_source_ids monotonically across retrieved turns, compaction checkpoints, and per-turn sandbox snapshot layers (L_0, L_1, ...), the policy gate can batch-verify every active source_id in the session before turn t starts, before each mid-turn tool invocation, and before final output delivery, immediately canceling any affected in-flight turn, atomically dropping tainted assistant and tool_use/tool_result pairs, rebuilding tainted compaction checkpoints, and rolling back tainted sandbox layers (L_{k..t}) the moment a source permission is revoked.

The fourth surface is the verification loop that proves the first three actually work under load: a continuous three-probe revocation drill. Once every hour in production, an automated harness grants a synthetic test principal access to a canary document containing a unique token, runs a query that populates both the response cache and a multi-turn session (including a workspace file write), launches a slow multi-step turn that references those workspace notes, and then revokes the permission in the upstream source system. Exactly sixty seconds after the revocation commits, the harness verifies three assertions: a fresh query hitting the semantic cache must not return the canary, the already in-flight turn must be canceled or stripped before tool access or output delivery, and a new follow-up turn inside the open session must have its tainted history and sandbox layers evicted. If any probe surfaces the canary token, or if the late-binding policy check fails open on a timeout, the drill pages the platform owner and trips the deployment gate. Ask your retrieval owner one question before your next enterprise security review: if an executive revokes access to a shared folder right now, how many minutes or hours can an existing chat session, in-flight agent turn, or response cache keep quoting the files inside it?