Leading Senior Engineers in the AI Era: Autonomy, Standards, and Accountability

A support assistant drafted forty refunds last quarter against a policy clause that had been retired three weeks earlier. The drafts read clean, reviewers approved most of them, and nobody could say which day the drift started. The code had passed review. The tests were green. The model did exactly what it was built to do, on context that had gone stale underneath it.

That is the output a senior engineer now owns: something a model produced that the owner cannot fully explain. The management question is no longer “did you write good code.” It is “how do you know the thing you didn’t write is safe to turn on.” The owner ships when four things are true, in writing, and each carries a number they can defend.

It clears an owned eval on the unhappy paths. Not a vibe check but 200 cases the owner curated by hand: refund requests phrased as threats, tickets in the wrong language, prompt-injection pasted from a customer email, accounts with conflicting prior promises. The bar is 95% on that adversarial set, not 99%. Above 95% on a set this small you are usually overfitting: the cases have leaked into prompts, or the team has tuned to the test. Set it where the residual failures are ones the reviewer reliably catches. And the set rots on purpose: every production escalation becomes a new case, a tenth rotate out monthly, the owner holding back cases never tuned against. A frozen eval is a demo with a number on it; closing that gap is evaluation maturity .

Its top failure mode is named, with a detection signal. Here the mode is “confidently drafts a refund the policy forbids.” The signal is specific: a sampled audit comparing drafted amount against the policy ceiling, plus a flag whenever the draft cites a policy section that does not exist. If the owner cannot say what it most often gets wrong and how a dashboard would surface it, they have not found the failure mode. They have hoped there isn’t one.

Rollback has been exercised, not documented. Someone flipped the flag back to human-only handling in production last week, watched the queue drain, and confirmed nothing leaked. A rollback that has only ever lived in a runbook is a guess, and you discover it is wrong during the incident , which is the worst possible time to learn.

The escalation threshold is a number you can measure. Edit rate is easy: page if the reviewer rewrites more than 50% of drafts, because past half the assistant manufactures review work instead of removing it, and the unit economics have inverted. Stale context, the opening refund, is the hard one, so you instrument it rather than assert it. Every draft logs the version of each source it pulled: the policy revision, the read-time of the account record. At audit you label a draft stale when a source it relied on had already been superseded by the time the human acted: the policy version cited is no longer live. Sample a few hundred a day and you have a rate. Page at 3% over a rolling 1,000, because at a 95% review catch rate, 3% stale drafts put roughly 0.15% in front of a customer: the leak you decided you could afford. Raise the catch rate and you earn the right to raise the threshold.

None of this requires reading code to inspect, which makes it the board’s lever, not the team’s secret. Ask for the written definition-of-done for your single most important AI feature: which pass rate on which cases, which named failure mode, which rollback drill, which paging number, and the one name accountable for it. If what comes back is vague, the vagueness is not a documentation gap. It is the failure mode, still unnamed, and you have located your largest AI risk, which most oversight never does. Leading seniors is now mostly making that document exist, owned by one person, and boring, because every blank still interesting is a place the system can fail with no agreed way to know.