Garbage Context, Confident Answer

The agent answered, and it answered with confidence. The doc it cited had been edited yesterday; the system that doc described had been rebuilt this morning. The retriever scored its top hit at 0.81 and handed it over as a match. The model filled the gaps the stale text left, produced something fluent, and acted on it. Nobody paused, because a confident answer from garbage context is more dangerous than a refusal: the refusal would have been checked, this never was.

“Garbage in, garbage out” predates the transistor. Every team nods and the failure happens anyway, not because engineers forgot the slogan but because the slogan ships no instrument. The obvious fix, measure your context the way you measure your model, hits a wall on day one: you cannot compute retrieval precision in production without the labeled ground truth a working retriever was supposed to give you. So walk the incident backward and ask what instrument would have caught it before the agent acted. You score the feed with proxies, and three of them survive contact with production.

Groundedness would have flagged the fluent answer. Force every answer to cite the spans it used, then run a cheap entailment pass: what fraction of the answer’s claims are actually supported by the retrieved text? The unsupported sentences are the model patching a hole the retriever left, exactly what happened here. A groundedness rate is computable on live traffic with no labels, and it falls before your users complain.

The 0.81 was the second tell, and it was the wrong number to trust. Absolute similarity lies; an embedding model will happily return 0.81 against junk. The signal is not the score but the margin: the gap between the top result and the rest of the top-k. When that margin collapses, when everything is mediocre and nothing stands out, the retriever found no real match and the model is about to improvise. Alert on the margin.

Both proxies drift, so you anchor them with calibration. Hand-label one or two percent of live queries offline and treat that sample as the truth set that tells you how far the proxies have moved. This is the discipline behind why your benchmarks are lying to you : a proxy you never check against reality becomes a number you trust for the wrong reasons.

But the root cause sat upstream of all three: the feed had no idea the system had changed under it. Freshness is the harder half, and a timestamp does not solve it. A doc edited yesterday can describe a system rebuilt this morning, precisely the failure here, and a doc untouched for two years can be eternally correct. Edit time tells you when someone touched the file, not whether it is still true. Bind each source of truth to a subject fingerprint instead: a hash of the config, version, or topology the document assumes. Recompute it from live state at retrieval time, and a mismatch means the document describes a world that no longer exists, regardless of its date. This morning’s rebuild would have changed the live topology, the recomputed hash would not have matched, and the source would have dropped before the agent ever read it. Where nothing machine-checkable exists, decay confidence from the last verification event, a passing test or a human sign-off, never from the last edit. That is how an agent tells a current source from a stale one: not by reading a date, but by checking the world against the claim before it acts. The safeguard lives around the model, not inside it .

Stop waiting for a model upgrade to fix a feed nobody scores. The model is fungible. The instrument on the context is the part you have to build, and it is the part that decides whether your agent compounds or just gets more articulate about the wrong answer.

So the postmortem closes on the dashboard, not the model. For the board it is one line: the percentage of agent actions taken on unverified context. Treat it like change-failure rate. Set a ceiling, and route anything above it to a human or to a refusal. Underneath that number sits the governance artifact, per source: a named owner plus a machine-checkable verification event with a half-life. Give the doc from this incident that owner and that expiring proof, and “the data is current” stops being a hope and becomes a check that fails loudly before the next agent retrieves it.