Reliability Is the Autonomy Ceiling

Autonomy is a grant you price, and the price has two inputs: what a single failure costs, and how often the agent fails. The first you usually know. The second is where teams quietly fool themselves.

Run an agent two hundred times in staging and watch it fail zero times. That number licenses nothing. Zero failures in two hundred trials is statistically consistent with a true failure rate near one in sixty-seven, and if a single failure drains a customer’s account, one in sixty-seven is a catastrophe you simply haven’t reached yet. The demo told you what the agent can do. It said almost nothing about how often it fails on the work you’d actually stop watching, and that gap is the real ceiling on how much you can hand off.

So the operative question is how you measure a failure rate on events too rare to stage. The reliability contract names the bar; the work is clearing it.

The bar comes from one piece of arithmetic, the rule of three: with zero observed failures in N independent trials, the 95% upper bound on the true failure rate is roughly 3/N. Tabulated:

Which row you need is set by failure cost times failure rate, the unit autonomy is actually priced in. A cheap, reversible mistake tolerates a high rate, so a small N clears it. An expensive, irreversible one forces you down to a low rate, and the lower the rate you must rule out, the more clean observations you owe before pulling the human. Put that math on the plan, not in the retro after the first bad week.

Three thousand staged trials of a rare failure is impossible by definition, so you buy the observations two ways.

Shadow mode supplies the volume. Run the agent in parallel on real production traffic, executing nothing. Every live request becomes a trial: log what it would have done, diff against the actual outcome. Two weeks of real volume delivers sample size staging never will, on the true input distribution, including the ugly tail, at zero blast radius.

Shadow mode only samples the failures the world happens to send, though. The ones that stay rare because they’re adversarial, or because a dependency seldom breaks, you manufacture: malformed and out-of-distribution inputs, stale data, truncated context, the partial outage, the injection payload. This isn’t estimating a rate; it’s checking containment, whether a known-bad input produces a bounded failure or an unbounded one. Each class survived raises the ceiling; each one that breaches the blast radius caps it.

With both in hand, promote one rung at a time, like privilege escalation:

1. suggest
2. act-with-approval
3. act-with-audit
4. act-unattended-within-bounds

Each rung is earned by clearing the measured bar at the rung below, on shadow data and survived fault injection, never by how the last demo looked. A cheap, reversible mistake at one-in-fifty can ride the top rung unattended; an expensive, irreversible one at the same one-in-fifty stays low: at that rate, the only question is how many runs until it lands. When a rare failure does fire, incident review tightens the bound or drops a rung, instead of patching the prompt and calling it closed.

Both directions cost. Over-grant and you trade a quiet staging win for a production loss whose expected value was always negative: the row you skipped. Under-grant and you pay people to babysit work the data already cleared, margin sitting idle. The honest version of “we haven’t seen it fail” is “we haven’t sampled enough to know.” Stop treating the grant as a judgment call: it’s the row your failure cost selects, checked against the trial count you have actually banked.