Quick take
Most catastrophic database incidents aren’t novel. They’re compounded failures that nobody practiced for. The node-failure test passes, so the team moves on. Then a network partition hits during a schema migration while the on-call engineer is handling an unrelated alert, and suddenly you’re in territory no runbook covers. Structured red-teaming exposes these compound paths before they become customer-visible outages. It costs a fraction of what a single bad incident costs.
Black Swans vs. Ignored Knowns
The term “black swan” gets overused in infrastructure. Most catastrophic database failures are not genuinely unpredictable. They are known failure modes that compound in ways nobody tested.
Consider the canonical distributed database incident: a network partition isolates a minority of nodes, those nodes continue accepting writes because the partition detection is slow, the partition heals, and now you have conflicting data that the conflict resolution logic wasn’t designed to handle at that volume. Every component in this chain is well-understood. The failure isn’t in any single component. It’s in the interaction between them under specific timing conditions.
The honest term for most “black swan” database incidents is “ignored known.” The team knew partitions could happen. They knew conflict resolution had edge cases. They knew detection wasn’t instant. They just never tested all three at once.
Red-teaming is how you turn ignored knowns into practiced scenarios.
Mission-Style Red-Teaming
Chaos engineering tools that randomly kill processes are useful, but they test a narrow failure class: single-component loss. Distributed database failures rarely look like one node dying cleanly. They look like degraded networks, clock drift, slow disks, operator errors during maintenance windows, and combinations of all of the above.
Mission-style red-teaming borrows from military and security practice. A dedicated team designs multi-step failure scenarios with specific objectives, executes them against production-equivalent infrastructure, and scores the defending team’s response. The key difference from chaos engineering is intentionality: the red team isn’t injecting random faults. They’re pursuing a specific failure hypothesis through a sequence of realistic actions.
A red-team exercise has three roles:
- Red team: designs and executes the failure scenario. Their goal is to cause data loss, unavailability, or corruption without triggering detection within a target time window.
- Blue team: the on-call and operations engineers responding as they would in a real incident. They don’t know the scenario in advance.
- White team: observers who control the exercise, ensure safety boundaries, and document everything for the post-exercise review.
The exercise runs for a fixed window, typically two to four hours. The red team executes their scenario. The blue team detects, diagnoses, and responds. Everyone debriefs afterward.
The Stress Scenarios That Matter
Not all failure modes are worth practicing. Focus on scenarios that are plausible, high-impact, and poorly covered by existing automation.
Network partitions with asymmetric visibility. One side of the partition can see the other; the other side cannot. This breaks assumptions in consensus protocols that expect symmetric failure detection. Many teams test clean partitions but never test asymmetric ones.
Clock skew under load. Distributed databases that use timestamps for ordering (which is most of them) behave unpredictably when clocks drift. NTP usually keeps drift small, but under heavy load, NTP corrections can be delayed. The result is transaction ordering violations that are invisible until a consistency check runs, which might be hours or days later.
Quorum erosion during maintenance. You take one node offline for a rolling upgrade. While it’s down, a second node develops a slow disk. You now have a degraded quorum that’s technically functional but one failure away from data unavailability. This is the most common compound failure pattern and the least practiced.
Operator mistakes during incidents. The most dangerous moment for a distributed database is when a human is manually intervening during an incident. Wrong-node restarts, accidental force-quorum operations, and recovery commands run against the wrong cluster are responsible for a disproportionate share of catastrophic data loss. Red-teaming should include scenarios where the operator is given misleading information and time pressure.
Backup restoration under partial failure. Most backup tests verify that a restore works on a clean target. Real restores happen during incidents, when the target environment is degraded, the team is stressed, and the backup might be from a point in time that’s already inconsistent. Test restoration under these conditions, not just in a clean room.
The OODA Loop for Incident Rehearsal
Effective red-team exercises run on a tight observe-orient-decide-act cadence. This isn’t just a framework. It’s a scoring mechanism.
Observe: How quickly does the blue team notice something is wrong? Detection time is the single most important metric. A failure that’s detected in two minutes has a fundamentally different blast radius than one detected in twenty. Measure time from fault injection to first alert, and time from first alert to accurate diagnosis.
Orient: Does the team correctly identify what’s happening? Misdiagnosis is common in compound failures because the symptoms don’t match any single runbook entry. The blue team might see elevated latency and assume it’s a hot key, when the actual cause is a partial partition affecting replication. Measure time from first alert to correct hypothesis.
Decide: Does the team choose an appropriate response? Under pressure, teams often default to the most familiar action (restart the node) rather than the most appropriate one (isolate the partition). Measure whether the chosen action matches the failure mode.
Act: Does the team execute the response correctly? Even when the right decision is made, execution errors under stress are common. Typos in commands, wrong node targets, and forgotten steps in manual procedures are all frequent. Measure execution accuracy and time to containment.
Each phase gets a score. Over multiple exercises, these scores reveal systemic gaps: maybe detection is fast but diagnosis is slow, or decisions are sound but execution is error-prone. That tells you exactly where to invest in automation, training, or tooling.
Scoring Readiness
After each exercise, score three dimensions:
Readiness (1-5): Could the team handle this scenario if it happened tomorrow in production? A 1 means the team didn’t detect the failure. A 5 means they detected, diagnosed, and contained it within SLA.
Blast radius (1-5): If the team had not responded, how bad would it have gotten? A 1 means minor degradation. A 5 means unrecoverable data loss or extended outage.
Time to containment (minutes): Wall-clock time from fault injection to the point where the failure is contained and no longer spreading. This is the metric that matters most to your customers and your SLA.
Plot these over time. Improving readiness scores and decreasing containment times are the clearest signals that your red-teaming program is working. If scores plateau, your scenarios aren’t challenging enough.
From Findings to Backlog
Red-team exercises are useless if findings sit in a postmortem document that nobody reads. Every exercise should produce a prioritized list of concrete improvements, each with an owner and a deadline.
The conversion process is simple:
- List every gap discovered. Detection gaps, diagnostic confusion, tool limitations, missing runbooks, automation failures.
- Score each gap by blast radius times likelihood. Likelihood is informed by the exercise, not guessed.
- Assign an owner for each gap. Not a team. A person.
- Set a deadline before the next exercise. The next exercise will test whether the gap was closed. This creates accountability.
Common improvements that come out of red-team exercises include automated partition detection that currently requires manual observation, runbook updates for compound failure scenarios, guardrails on dangerous operator commands during incidents, and backup restoration procedures tested under realistic conditions.
The backlog items from red-teaming tend to be high-value, low-glamour work. They rarely make it onto a roadmap through normal prioritization because they address risks that haven’t materialized yet. The exercise provides the evidence needed to justify the investment.
A Quarterly Operating Cadence
Red-teaming works best as a regular practice, not a one-off event. A quarterly cadence balances rigor with operational overhead.
Run quarterly. Dedicate the first few weeks to scenario design based on recent incidents and architectural changes, a half-day to executing the exercise against a production-equivalent environment, and the remainder of the quarter to remediating the gaps you found.
This cadence means every quarter your team practices a realistic failure scenario, identifies concrete gaps, and fixes the most critical ones before the next exercise. Over four quarters, you’ve tested and improved your response to a dozen failure modes. That’s a fundamentally different reliability posture than “we tested node failover once during setup and it worked.”
Key Takeaways
- Most catastrophic database failures are compound scenarios that nobody practiced, not genuinely unpredictable events.
- Chaos engineering tests component failure. Red-teaming tests system failure under realistic operational conditions.
- Score every exercise on detection time, diagnostic accuracy, decision quality, and execution correctness. Track trends.
- Convert findings into owned backlog items with deadlines tied to the next exercise.
- Run quarterly. Consistency matters more than intensity.
Red-teaming distributed databases is not theater and it’s not a luxury. It’s the cheapest way to find out whether your recovery assumptions actually hold before your customers find out for you.