Agentic Systems at Scale: The New Reliability Contract

A team I’d call careful kept a reliability document for their billing agent. It enumerated scope, listed prohibited actions, named a kill switch. Thorough. The agent still issued duplicate refunds against a malformed webhook until the bank alerts came in. Every prohibited action was prohibited in the document and permitted by the runtime. The model did not go rogue. It did exactly what its unbounded permissions allowed. The damage was a blast-radius problem, not an intelligence problem.

That is the gap nobody likes to name. A markdown file rejects nothing. You can write “the agent may not modify access controls” in bold and it will modify access controls the instant a confused tool call asks it to, because the sentence has no teeth. A reliability contract is real only where each clause maps to something in the control plane that enforces it without the agent’s cooperation.

So the useful question is not what the contract should say. It is what enforces each clause when the agent is wrong. Three substrates do the work.

Scoped credentials, not scoped intentions. The agent runs under an identity that cannot call what it was not granted. If the path is absent from the IAM policy, “may not escalate its own permissions” is enforced by the absence, not by a promise. The test: revoke one permission and confirm the action now fails with an authorization error rather than a polite refusal.

A deny-by-default tool gateway. Every tool call passes through a broker that is closed unless explicitly opened. The broker, not the prompt, holds the rule that external sends need human review, that a single run touches at most N records, that cross-tenant reads return nothing. Blast-radius limits live here because this is the one place that can count and refuse. A limit the agent keeps on itself is a suggestion.

A sandbox the agent cannot widen. Egress, filesystem, and execution bounded at the edge, so a confused or compromised agent fails inside the box instead of pivoting out of it.

If you cannot point to the line of config that enforces a clause, that clause is aspirational. Mark it as such instead of pretending.

The careful version then fails in the other direction. Once you have been burned, the instinct is to require human acknowledgment before the agent resumes after any threshold breach. At one agent, fine. Across a fleet, that rule is an outage generator: one slow upstream pushes p95 past its line on two hundred agents, all of them drop to restricted mode, and you have manufactured two hundred pages and a stalled fleet out of a latency blip in someone else’s service. Outage by policy.

Separate the two things that rule conflates. Degradation (slow, erroring, escalating too often) is a throughput problem: throttle automatically, recover automatically, log it, page no one. Boundary breach (an out-of-scope rejection, a blast-radius hit, a data-scope violation) is a trust problem: halt, and require a human to sign off before resuming, because the question is no longer “is it healthy” but “did it just try to do something we forbade.” Human attention is the most expensive control you have. Spend it where judgment is the point, never as a tax on every metric wobble.

That reframes the decision an executive actually owns. Not the value of p95; your engineers set that. The decision is structural: do you require, as a funded precondition for any agent reaching production, that its authority live in a control plane the agent cannot edit, and who owns that gateway? “Each team self-attests in a doc” and “nothing ships without enforced scope” are different risk postures with different budgets. One is a slide. The other is an organization with a place to put the rule.

Write the contract anyway. It is the spec. But it commits nothing until every line has an owner who can make it true in the control plane, and a stop authority that revokes the credential instead of asking the agent to please stop.