Stop Grading Agents on pass@1: Reliability Is pass^k

Building your own evaluation suite from real production traffic with deterministic outcome assertions is the entry ticket to serious AI engineering, yet teams that clear that bar still get ambushed when they ship multi-step agents. The release dashboard says the agent passed 75% of the golden suite on Tuesday; by Friday, operations is paging the team because the workflow fails unpredictably on routine tickets. The bug is not in the assertions. It is in how the harness aggregates multi-run reliability and how it provisions the containers that run the tests.

Start with the metric itself. Most evaluation harnesses inherited pass@1 (single-trial accuracy) and pass@k (the probability that at least one of k independent attempts succeeds) from code-generation benchmarks. pass@k makes sense when an engineer or a compiler sits behind the model to pick one working patch out of five candidates. It is the exact wrong metric for an agent that mutates customer state, triages claims, or executes runbooks in production, where every failed run is a real incident or an escalation. As Anthropic’s January 2026 engineering guide on agent evaluations emphasized, production autonomy requires grading pass^k: the probability that all k independent trials of a task succeed. If reliability is the ceiling on autonomy , pass^k is how you measure the height of the ceiling.

Here is where even careful teams make a costly mathematical error: they estimate pass^k by taking the suite’s average pass@1 score and raising that single number to the k-th power (mean(p)^k). Because of Jensen’s inequality, raising an average to a power is wrong whenever success probabilities vary across tasks, and it hides the difference between two opposite operational realities. Suppose your golden suite has 100 tasks and your agent posts a 75% pass@1 average. In Case A, 75 of those tasks are in-scope workflows that succeed deterministically every single time (p_i = 1.0), and 25 are unsupported edge cases that always fail (p_i = 0.0). Your pooled pass@1 is 75%, and your true three-run consistency (pass^3) across the suite is also 75%; inside its supported boundary, the agent is rock solid, and your only job is to route the 25 unsupported cases away at intake. In Case B, all 100 tasks succeed with an independent per-trial probability of p_i = 0.75. Your pooled pass@1 is still 75%, your pass@3 looks like a flattering 98.4%, and your true pass^3 collapses to (0.75)^3 = 42.2%. Raising the suite average to the third power treats Case A and Case B as identical, confusing a clean scope boundary with a flaky agent that fails more than half the time across three runs.

The fix is to compute pass^k per task before you ever average across the suite. Run each benchmark task i for n independent trials (n >= k, typically n = 5 or n = 10 on release candidates) under the exact retry and timeout rules configured in production. When task i succeeds in c_i of n trials, compute its unbiased per-task consistency using the combinatorial estimator C(c_i, k) / C(n, k). Instead of gating deployments on a single pooled mean, gate them on the per-task distribution: require that at least 90% of benchmark tasks achieve a per-task pass^3 of 1.0 with zero regressions on previously deterministic workflows, and block any individual workflow with c_i < n (0 <= c_i < n) from task-level autonomy promotion until its failure or flakiness is resolved.

Once you measure per-task consistency across multiple trials, a second culprit appears: your evaluation infrastructure is injecting variance of its own. In February 2026, Anthropic published an empirical study quantifying infrastructure noise on coding-agent benchmarks and showed that changing container CPU, memory, and timeout headroom alone swung Terminal-Bench 2.0 scores by six percentage points (p < 0.01). When an evaluation runner packs twenty parallel agent containers onto a contended CI node, a build tool or test subprocess inside the sandbox hits CPU throttling or an OOM killer, times out, and forces the model down a recovery branch it would never take in production. An evaluation harness that cannot separate environment jitter from model nondeterminism will send your engineers chasing phantom model regressions for weeks.

Lock the container envelope as part of the evaluation spec. Pin every eval sandbox to hard cgroup CPU quotas (cpu.max), dedicated cpuset pinning on non-overcommitted runner nodes, strict memory ceilings, disk I/O limits, and per-command wall-clock timeouts that match your production runtime pool, and record container resource-saturation counters (CPU throttled_usec and memory pressure events) on every trial span, discarding and re-running any trial where the host node breached its resource SLA. Before you approve the next agent release or sign off on a vendor’s accuracy deck in technical due diligence, ask one question: what fraction of your benchmark tasks achieved c_i = n across five consecutive runs in pinned containers, and how many of your “passing” tasks only passed because you graded them once?