A Backup Model You Never Exercise Is Not a Backup

Every resilient-looking AI architecture deck now has the slide: two or three model vendors, arrows into an abstraction layer, the word “diversified.” Compose the failure from the defects I keep finding in real systems and you get the composite Tuesday: the primary has a capacity event, and the team discovers the backup fails a chunk of the eval set, the fallback credential expired unrotated, the negotiated rate limit is a fraction of production load, and the prompts tuned to one model’s habits produce output the downstream parsers reject. The switch that was an arrow on a slide turns out to be a migration, executed during an incident.

Ops compressed this lesson into a slogan decades ago: nobody cares about backups, everyone cares about restores. I made the strategic half of this argument in July ; a warm fallback and a credible exit are what vendor risk management actually consists of. This post is the missing protocol, because “exercise your fallback” fails the same way “have a backup” fails: without a defined drill and a defined clock, it degrades into a checkbox. Vendor count is inventory. Measured failover time is resilience.

The drill has a ladder, because the objection to drilling is real: a database restore can be rehearsed in isolation, but model output goes to customers, and a day of known-worse answers is not a price to pay for rehearsal. So don’t start with customers. Rung one is continuous and free of risk: a weekly automated path check. Credentials valid, rate limits confirmed against contract, a smoke set through the alternate. That alone would have caught two of the composite Tuesday’s four defects. Rung two is shadow replay, quarterly: mirror a day of production traffic, at load, including the peak, against the backup, and score it offline against the eval set you already own . Coverage matters more than volume. The replay must include every critical eval class, the compliance-sensitive categories and the long-tail formats, not just the median request. Shadow is also where you measure the cold-start cache trajectory : cost per workflow cold, the hit-rate curve, time to plateau. That measurement ends at coverage and plateau, not at a calendar day. Rung three, only for workflows whose outputs are reversible and only after shadow passes: a bounded canary, half a percent of live traffic ramping toward five, with pre-declared stop conditions on quality, safety, latency, and cost, and immediate rollback on breach. Workflows with consequential, hard-to-reverse outputs don’t get canaried onto a worse model; for those, shadow evidence plus a human-gated degraded mode is the ceiling of the drill.

Then the clock, defined so nobody can game it. It starts at failover authorization, the simulated incident declaration, not the quiet prep before it. It stops when the alternate has sustained the target load inside every pass bar for a pre-agreed rolling window, not at first success. Report two segments separately: routing time, how long until traffic moves, and stabilization time, how long until it’s genuinely absorbed. The second number is where the truth lives, and the pair, tracked quarter over quarter, is the resilience metric your board slide has been missing. Assume the first drill exposes defects: the stale credential, the hardcoded model name, the prompt that needs a per-vendor variant. Each is a migration task done on a calm afternoon instead of during the incident, which is the drill’s actual product.

Finally, the bill, because this program isn’t free: duplicate inference for shadow runs, eval maintenance, per-vendor prompt variants, minimum-spend commitments on a contract you rarely use, engineering time. Weigh that annual hedge cost against the outage loss the measured failover window avoids, workflow by workflow, with the same tiering that decides which workflows get drills at all . Some workflows won’t justify it, and some drills will return the finding that no acceptable backup exists at any price. Both results are wins, provided they end as a signed, specific risk acceptance instead of a diversification slide. The slide says you have vendors. The clock says whether you have a way out.