Most catastrophic database incidents aren’t novel. They’re compounded failures that nobody practiced for. The node-failure test passes, so the team moves on. Then a network partition hits during a schema migration while the on-call engineer is handling an unrelated alert, and suddenly you’re in territory no runbook covers. Structured red-teaming exposes these compound paths before they become customer-visible outages. It costs a fraction of what a single bad incident costs.

Black Swans vs. Ignored Knowns

The term “black swan” gets overused in infrastructure. Most catastrophic database failures are not genuinely unpredictable. They are known failure modes that compound in ways nobody tested.

Consider the canonical distributed database incident: a network partition isolates a minority of nodes, those nodes continue accepting writes because the partition detection is slow, the partition heals, and now you have conflicting data that the conflict resolution logic wasn’t designed to handle at that volume. Every component in this chain is well-understood. The failure isn’t in any single component. It’s in the interaction between them under specific timing conditions.

The honest term for most “black swan” database incidents is “ignored known.” The team knew partitions could happen. They knew conflict resolution had edge cases. They knew detection wasn’t instant. They just never tested all three at once.

Red-teaming is how you turn ignored knowns into practiced scenarios.

Mission-Style Red-Teaming

Chaos engineering tools that randomly kill processes are useful, but they test a narrow failure class: single-component loss. Distributed database failures rarely look like one node dying cleanly. They look like degraded networks, clock drift, slow disks, operator errors during maintenance windows, and combinations of all of the above.

Mission-style red-teaming borrows from military and security practice. A dedicated team designs multi-step failure scenarios with specific objectives, executes them against production-equivalent infrastructure, and scores the defending team’s response. The key difference from chaos engineering is intentionality: the red team isn’t injecting random faults. They’re pursuing a specific failure hypothesis through a sequence of realistic actions.

A red-team exercise has three roles:

• Red team: designs and executes the failure scenario. Their goal is to cause data loss, unavailability, or corruption without triggering detection within a target time window.
• Blue team: the on-call and operations engineers responding as they would in a real incident. They don’t know the scenario in advance.
• White team: observers who control the exercise, ensure safety boundaries, and document everything for the post-exercise review.

The exercise runs for a fixed window, typically two to four hours. The red team executes their scenario. The blue team detects, diagnoses, and responds. Everyone debriefs afterward.

The Stress Scenarios That Matter

Not all failure modes are worth practicing. Focus on scenarios that are plausible, high-impact, and poorly covered by existing automation.

Network partitions with asymmetric visibility. One side of the partition can see the other; the other side cannot. This breaks assumptions in consensus protocols that expect symmetric failure detection. Many teams test clean partitions but never test asymmetric ones.

Clock skew under load. Distributed databases that use timestamps for ordering (which is most of them) behave unpredictably when clocks drift. NTP usually keeps drift small, but under heavy load, NTP corrections can be delayed. The result is transaction ordering violations that are invisible until a consistency check runs, which might be hours or days later.

Quorum erosion during maintenance. You take one node offline for a rolling upgrade. While it’s down, a second node develops a slow disk. You now have a degraded quorum that’s technically functional but one failure away from data unavailability. This is the most common compound failure pattern and the least practiced.

Operator mistakes during incidents. The most dangerous moment for a distributed database is when a human is manually intervening during an incident. Wrong-node restarts, accidental force-quorum operations, and recovery commands run against the wrong cluster are responsible for a disproportionate share of catastrophic data loss. Red-teaming should include scenarios where the operator is given misleading information and time pressure.

Backup restoration under partial failure. Most backup tests verify that a restore works on a clean target. Real restores happen during incidents, when the target environment is degraded, the team is stressed, and the backup might be from a point in time that’s already inconsistent. Test restoration under these conditions, not just in a clean room.

The OODA Loop for Incident Rehearsal

Effective red-team exercises run on a tight observe-orient-decide-act cadence. This isn’t just a framework. It’s a scoring mechanism.

Observe: How quickly does the blue team notice something is wrong? Detection time is the single most important metric. A failure that’s detected in two minutes has a fundamentally different blast radius than one detected in twenty. Measure time from fault injection to first alert, and time from first alert to accurate diagnosis.

Orient: Does the team correctly identify what’s happening? Misdiagnosis is common in compound failures because the symptoms don’t match any single runbook entry. The blue team might see elevated latency and assume it’s a hot key, when the actual cause is a partial partition affecting replication. Measure time from first alert to correct hypothesis.

Decide: Does the team choose an appropriate response? Under pressure, teams often default to the most familiar action (restart the node) rather than the most appropriate one (isolate the partition). Measure whether the chosen action matches the failure mode.

Act: Does the team execute the response correctly? Even when the right decision is made, execution errors under stress are common. Typos in commands, wrong node targets, and forgotten steps in manual procedures are all frequent. Measure execution accuracy and time to containment.

Each phase gets a score. Over multiple exercises, these scores reveal systemic gaps: maybe detection is fast but diagnosis is slow, or decisions are sound but execution is error-prone. That tells you exactly where to invest in automation, training, or tooling.

Scoring Readiness

After each exercise, score three dimensions:

Readiness (1-5): Could the team handle this scenario if it happened tomorrow in production? A 1 means the team didn’t detect the failure. A 5 means they detected, diagnosed, and contained it within SLA.

Blast radius (1-5): If the team had not responded, how bad would it have gotten? A 1 means minor degradation. A 5 means unrecoverable data loss or extended outage.

Time to containment (minutes): Wall-clock time from fault injection to the point where the failure is contained and no longer spreading. This is the metric that matters most to your customers and your SLA.

Plot these over time. Improving readiness scores and decreasing containment times are the clearest signals that your red-teaming program is working. If scores plateau, your scenarios aren’t challenging enough.

From Findings to Backlog

Red-team exercises are useless if findings sit in a postmortem document that nobody reads. Every exercise should produce a prioritized list of concrete improvements, each with an owner and a deadline.

The conversion process is simple:

1. List every gap discovered. Detection gaps, diagnostic confusion, tool limitations, missing runbooks, automation failures.
2. Score each gap by blast radius times likelihood. Likelihood is informed by the exercise, not guessed.
3. Assign an owner for each gap. Not a team. A person.
4. Set a deadline before the next exercise. The next exercise will test whether the gap was closed. This creates accountability.

Common improvements that come out of red-team exercises include automated partition detection that currently requires manual observation, runbook updates for compound failure scenarios, guardrails on dangerous operator commands during incidents, and backup restoration procedures tested under realistic conditions.

The backlog items from red-teaming tend to be high-value, low-glamour work. They rarely make it onto a roadmap through normal prioritization because they address risks that haven’t materialized yet. The exercise provides the evidence needed to justify the investment.

A Quarterly Operating Cadence

Red-teaming works best as a regular practice, not a one-off event. A quarterly cadence balances rigor with operational overhead.

Run quarterly. Dedicate the first few weeks to scenario design based on recent incidents and architectural changes, a half-day to executing the exercise against a production-equivalent environment, and the remainder of the quarter to remediating the gaps you found.

This cadence means every quarter your team practices a realistic failure scenario, identifies concrete gaps, and fixes the most critical ones before the next exercise. Over four quarters, you’ve tested and improved your response to a dozen failure modes. That’s a fundamentally different reliability posture than “we tested node failover once during setup and it worked.”

Key Takeaways

• Most catastrophic database failures are compound scenarios that nobody practiced, not genuinely unpredictable events.
• Chaos engineering tests component failure. Red-teaming tests system failure under realistic operational conditions.
• Score every exercise on detection time, diagnostic accuracy, decision quality, and execution correctness. Track trends.
• Convert findings into owned backlog items with deadlines tied to the next exercise.
• Run quarterly. Consistency matters more than intensity.

Red-teaming distributed databases is not theater and it’s not a luxury. It’s the cheapest way to find out whether your recovery assumptions actually hold before your customers find out for you.

We have models that can write poetry, generate code, and analyze images. We don’t have enough GPUs to run them reliably. We don’t have pricing that makes sense at scale. And we definitely don’t have the operational maturity to treat these systems like the production dependencies they have become.

I’ve spent December watching AI features I helped build at a fintech company run into every scaling problem distributed systems teams have been solving for twenty years. Rate limits. Cascading failures. Cost explosions. Latency spikes. The problems aren’t new. The industry is just re-learning them with a fresh coat of hype.

The GPU Situation Is Absurd

You can’t get H100s. You can’t reliably get inference capacity from any major provider unless you sign a months-long commitment or an enterprise contract that costs more than most startups raise in a seed round. The entire industry is building products on top of infrastructure that’s supply-constrained, and nobody wants to talk about what happens when demand doubles next year.

I tried to reserve inference capacity for a production workload last month. The response from one provider was “we can put you on a waitlist.” A waitlist. For compute. In 2023. This isn’t a technology problem. It’s a supply chain problem wearing a technology costume.

Rate Limits Are a Production Constraint

Every AI API has rate limits. At low volume, you don’t notice them. At production scale, they become the hardest ceiling in your architecture.

I hit OpenAI’s rate limit during a load test and watched requests queue up until the entire feature became unusable. Not degraded – unusable. The fix wasn’t clever engineering. It was a priority queue, backpressure, and load shedding. Distributed systems 101. The fact that most AI teams are learning this for the first time worries me.

Your Demo Won’t Survive Real Traffic

Here is what happens when your AI feature goes from 100 requests per day to 10,000:

Latency goes from “acceptable” to “users are closing the tab.” Costs go from “rounding error” to “someone just Slacked asking why the API bill tripled.” A provider outage that used to affect a handful of test users now takes down a production feature that the sales team just promised to a client.

I’ve seen all three of these happen at the same company. In the same month.

What You Actually Need

Queues and backpressure. Treat your AI traffic as a managed stream, not an open pipe. Priority queues for critical requests. Backpressure when the system is saturated. Load shedding for low-priority work. This isn’t optional once you have real users.

Circuit breakers. Your model provider will have bad hours. Mine had a bad day last week. Circuit breakers stop a provider outage from cascading through your entire system. They’re boring. They’re essential. I’ve been building systems with circuit breakers since my telecom days. The pattern hasn’t changed. The dependency has.

Graceful degradation. When GPT-4 is down, what happens? If the answer is “the feature breaks,” you don’t have a production system. You have a demo with users. Fall back to cached responses. Fall back to a smaller, faster model. Fall back to a static message that says “this feature is temporarily unavailable.” Anything is better than a spinning loader.

Cost controls that are actually enforced. Per-tenant budgets. Per-feature budgets. Daily caps. If you don’t enforce them, you’ll get a surprise invoice that triggers an emergency meeting. I’ve seen a single prompt change – adding two paragraphs of context – increase monthly costs by 35%. Token pricing is deceptively simple until you multiply it by production volume.

Caching. Exact-match caching is trivial to implement and saves real money. Same question, same context, same answer – serve it from cache. Semantic caching is fancier and worth exploring, but start with the easy wins.

This Is Distributed Systems Work

None of this is novel. Queues, circuit breakers, graceful degradation, cost controls, caching – these are patterns from every distributed systems textbook ever written. The only thing that’s new is the dependency type.

What frustrates me is that the AI community is treating infrastructure as a solved problem while building on top of infrastructure that’s anything but solved. The models are impressive. The plumbing is held together with optimism and rate limit retries.

Build your AI features like you would build any production system that depends on an unreliable, expensive, supply-constrained external service. Because that’s exactly what it is.

Most distributed systems advice reads like a textbook. This is the shortlist I actually use. Timeouts, retries, circuit breakers, sagas, outbox/inbox, and backpressure – applied with discipline, not ceremony.

I’ve built and operated distributed systems at Verizon, AT&T, Decloud, and most recently at a large consumer platform. The failure modes are remarkably consistent across all of them. Network partitions, cascading timeouts, retry storms, stale caches. The systems that survive aren’t the clever ones. They’re the ones with boring, well-applied patterns.

This isn’t a catalog. It’s the set of patterns I keep reaching for, along with real tradeoffs and code where it helps.

Failure Handling

Timeouts and Deadlines

Every remote call without a timeout is a bug waiting to happen. I learned this the hard way at Decloud – a single downstream service that started responding in 30 seconds instead of 300 milliseconds brought down our entire checkout flow. No timeout, no deadline propagation, no circuit breaker. Just threads piling up until the JVM ran out of memory.

Propagate deadlines end to end. If the user’s request has 2 seconds left, the downstream call should know that. In Go, this is context.WithTimeout and it works beautifully:

func fetchUser(ctx context.Context, id string) (*User, error) {
    ctx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, "GET", userServiceURL+"/"+id, nil)
    if err != nil {
        return nil, err
    }

    resp, err := httpClient.Do(req)
    if err != nil {
        return nil, fmt.Errorf("fetch user %s: %w", id, err)
    }
    defer resp.Body.Close()

    var u User
    return &u, json.NewDecoder(resp.Body).Decode(&u)
}

The caller sets the budget. The callee respects it. Simple.

Retry With Backoff and Jitter

Retries fix transient errors. Immediate retries create storms. I’ve seen a single retry loop without jitter generate enough traffic to keep a recovering service down for an extra twenty minutes.

Exponential backoff with full jitter. Cap the attempts. Only retry on errors that are actually transient – a 400 isn’t transient, a 503 probably is.

func retryWithBackoff(ctx context.Context, maxAttempts int, fn func() error) error {
    var err error
    for attempt := 0; attempt < maxAttempts; attempt++ {
        if err = fn(); err == nil {
            return nil
        }

        if attempt == maxAttempts-1 {
            break
        }

        backoff := time.Duration(1<<uint(attempt)) * 100 * time.Millisecond
        jitter := time.Duration(rand.Int63n(int64(backoff)))
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(jitter):
        }
    }
    return fmt.Errorf("after %d attempts: %w", maxAttempts, err)
}

Circuit Breaker

After a threshold of errors, stop calling the failing dependency. Just stop. Return a degraded response, a cached value, or an honest error. Let the dependency recover without your traffic making things worse.

The mental model: closed (normal), open (failing, short-circuit), half-open (testing recovery). The implementation doesn’t need to be complex. A counter, a timestamp, and a threshold.

Bulkheads

At a large consumer platform, we had a service that talked to six downstream dependencies through the same HTTP client pool. One slow dependency drained the pool and every other call started timing out too. Classic.

Separate connection pools per dependency. Separate goroutine budgets. The Titanic metaphor is overused but accurate – bulkheads keep one leak from sinking the whole ship.

Rate Limiting

A predictable 429 is better than an unpredictable timeout. Always. Apply rate limits at the edge and between services. I’ll talk more about this in a future post.

Consistency and Transactions

Sagas

Distributed transactions across service boundaries don’t work. Full stop. Two-phase commit sounds great in a database textbook and falls apart the moment you have services owned by different teams with different SLAs.

Sagas replace a global transaction with local transactions and compensating actions. Two flavors:

Choreography: services react to events. Decentralized, but the flow gets hard to trace once you have more than four or five steps. Debugging a choreographed saga during an incident is an exercise in grep and prayer.

Orchestration: a coordinator drives the flow. Easier to understand, easier to audit, easier to debug. The coordinator becomes a critical path, so it needs to be durable. I default to orchestration unless the flow is truly simple.

Outbox and Inbox

You can’t publish an event and update a database atomically with two separate calls. The outbox pattern solves this: write the event to a table in the same transaction as your state change. A separate publisher reads the outbox and sends the events.

The flip side is duplicate delivery. The inbox pattern handles that – record processed event IDs at the consumer and skip repeats. Pair them together. This is the backbone of every reliable event pipeline I’ve built.

Idempotency

Make writes idempotent. Use client-provided idempotency keys. Store the result keyed by that token. If the same request shows up twice, return the same result without doing the work again.

This isn’t optional in a system with retries. If you have retries (and you should), you need idempotency.

CQRS and Read Models

Strong consistency across services is expensive and usually unnecessary. Separate your write model from your read model. Accept eventual consistency for queries, search, and reporting. The read model can be denormalized, optimized, and rebuilt without touching the write path.

Messaging

Events

Events decouple services in time and deployment. Two styles worth knowing:

• Event notification: “something happened, look it up if you care.” Lightweight, but consumers need access to the source.
• Event-carried state: “something happened, here are the details.” Heavier payload, but consumers are self-sufficient.

Keep schemas versioned and backward compatible. Breaking an event schema in production is one of those things you only do once.

Work Queues

Queues buffer work and let you process at a steady rate. Multiple consumers on the same queue give you horizontal scaling for free. Set visibility timeouts, handle retries explicitly, and always have a dead-letter queue for messages that can’t be processed.

Coordination

Leader Election

Some work must be done by exactly one node. Scheduling, cleanup, deduplication. Use leader election with lease-based expiration. If the leader dies, the lease expires, and someone else takes over.

I’ve seen teams try to avoid leader election by distributing coordination across all nodes. It always ends with split-brain bugs that take weeks to reproduce.

Distributed Locks

Use sparingly. A distributed lock should protect a small, short-lived critical section. If you find yourself holding a lock for seconds, you probably need a different design – partition by key or use a single-writer pattern.

Scaling

Sharding and Consistent Hashing

Partition by key. When you add or remove nodes, consistent hashing minimizes the data that moves. Keep shard ownership explicit and routing predictable. Implicit sharding is debugging hell.

Backpressure

When overloaded, shed work instead of collapsing. Bounded queues, rate limits, and explicit flow control. The system that says “no” gracefully is more reliable than the system that says “yes” and falls over.

This is a cultural thing too. Engineers need to be comfortable returning errors under load instead of trying to serve every request.

Observability

Correlation IDs through every log and event. Latency percentiles (not averages). Error rates per dependency. Queue depth trends. Alert on symptoms – elevated latency, increasing error rates – not on individual errors.

Without observability, you’re guessing. During an incident, guessing is expensive.

The Default Kit

If I’m starting a new service, this is what goes in on day one:

• Timeouts and deadline propagation on every remote call
• Retries with exponential backoff and jitter for transient errors
• Circuit breakers and bulkheads around dependencies
• Idempotency keys for writes, inbox deduplication for events
• Outbox pattern for reliable event publishing
• Rate limiting and backpressure at boundaries
• Correlation IDs and service-level metrics

None of this is novel. That’s the point. Distributed systems are messy by nature. Patterns make the mess predictable. Discipline over heroics.

That’s a distributed team working. No drama. No blocked work. No meeting about it afterward.

I’ve been running Decloud as a remote-first company since before the pandemic made it fashionable. So when March hit and every company on Earth suddenly became “distributed,” I had a front-row seat to a lot of people reinventing wheels. Badly.

Six months in, the panic phase is over. But most teams are still stuck in this weird middle ground where they’re doing office work over Zoom and calling it remote. It’s not. And that gap is where the pain lives.

Writing is the job now

This is the single biggest adjustment and most teams still haven’t made it. In an office, you can be sloppy with communication because you’ll bump into someone at lunch and clear it up. Distributed? That ambiguity sits there rotting until someone makes the wrong assumption.

At Decloud, we write everything down. Not because we love documentation – honestly, nobody does. Because the alternative is having the same conversation three times across two time zones and a Slack thread that’s 200 messages deep.

When someone proposes a change, they write it up. Context, options, recommendation. It takes maybe 20 minutes. It saves hours of meetings that would’ve happened otherwise. The people who push back on this the hardest are usually the ones who are used to winning arguments by talking louder in a room. That doesn’t work in a Google Doc.

Meetings should hurt to schedule

Hot take: if your remote team is in more meetings than your office team was, you’ve failed. The whole point of going distributed is that async is the default. Meetings are the exception.

We have a few standing syncs at Decloud. Short ones. Everything else requires an agenda shared beforehand and a written outcome after. If you can’t write an agenda, you don’t need a meeting. You need to think more about what you actually want.

The time zone thing makes this easier, honestly. When you only have a four-hour overlap with half your team, you get very precious about those hours. Pairing sessions, hard design problems, the stuff where real-time back-and-forth actually matters. Everything else goes async.

Onboarding is where you find out if your system works

New hires are the stress test. If your distributed setup actually works, a new person should be able to get productive without scheduling fifteen “intro calls.” If they can’t, your documentation is bad. Full stop.

We give every new hire a buddy, a setup guide, and a small ticket that ships in the first week. The shipping part matters. Nothing builds confidence like seeing your code in production on day four. Compare that to spending your first week in orientation decks learning about the company values. Please.

The green dot problem

I talk to other engineering leads and the thing that makes me genuinely angry is the surveillance stuff. Screen monitoring. Activity tracking. Checking who’s online at what time.

Stop it. You’re measuring presence, not output. I don’t care if someone takes a two-hour break at 2pm to go for a run. I care if the work ships. If you can’t evaluate your engineers without watching their screen, that’s a management problem, not an employee problem.

At Decloud we’re explicit about this: here’s what we expect this sprint, here’s how we check in, here’s what “done” looks like. That clarity is the actual work of management. It’s harder than installing monitoring software, which is exactly why most people don’t do it.

What still sucks

I’m not going to pretend it’s all figured out. Some things about distributed work are genuinely worse.

Catching burnout is harder. In an office you can see someone looking tired or disengaged. Over Slack, people just go quiet, and by the time you notice, they’re already job hunting. I still don’t have a great answer for this beyond frequent 1:1s and actually paying attention.

Spontaneous collaboration is basically gone. Those hallway conversations where two people accidentally solve a problem? They don’t happen on Zoom. We’ve tried virtual coffee chats and random pairing. It’s fine. It’s not the same.

And onboarding senior people into leadership roles remotely is rough. Building the trust and political capital that lets you make big calls – that takes longer when you can’t read a room.

The actual secret

None of this is complicated. Write things down. Meet less. Trust your people. Ship the onboarding. That’s it. That’s the whole framework.

The reason most teams struggle with remote work isn’t because they lack the right tools or the right process doc. It’s because distributed work is less forgiving of the dysfunction you were already getting away with in the office. Bad communication, unclear ownership, meetings that should’ve been emails – all of that was survivable when everyone sat in the same building. Remove the building, and it falls apart.

Fix the fundamentals and the rest follows. Or don’t, and keep blaming Zoom fatigue. Up to you.

You don’t need Datadog’s enterprise tier. You need structured logs, one good dashboard per service, alerts that don’t cry wolf, and a request_id on everything. That’s 80% of it.

I’ve been working with distributed teams for most of this year. Some are five people across three time zones. Some are thirty across eight. The pattern I keep seeing: they either have zero observability or they went full enterprise cargo cult and now nobody can find anything.

There’s a middle ground. I want to talk about that.

The actual problem with distributed teams

In an office, someone notices something is slow. They say it out loud. Someone else goes “oh yeah, I deployed ten minutes ago.” Problem found in under a minute.

Remote? That same issue sits in a Slack thread for 45 minutes while people in different time zones wake up, read context, and try to figure out what changed. I’ve watched this happen. Repeatedly.

The fix isn’t more tools. It’s making your systems capable of answering basic questions without requiring a human to be online at the right moment.

Three questions. That’s it:

1. Is this thing broken right now?
2. What changed recently?
3. Where do I look next?

If your setup can answer those, you’re ahead of most teams I’ve worked with.

Enterprise observability isn’t your observability

Google has 10,000 SREs. They built custom everything. When you read their SRE book and try to implement the same stack with your team of eight, you end up with:

• A Prometheus instance nobody configured alerts for
• Grafana dashboards copied from a blog post that don’t match your services
• Jaeger running but with 0.1% sampling so traces are useless when you actually need them
• An ELK stack eating 40% of your infrastructure budget

I’ve seen this exact setup at three different companies this year. Not exaggerating.

What you actually need

Here’s my stack recommendation for a small distributed team. Opinionated, yes. But it works.

Logs: Structured JSON to a managed service. Loki if you’re cheap. Papertrail if you want simple. The key is structured, not the tool. Every log line should look roughly like this:

{
  "level": "error",
  "msg": "payment failed",
  "service": "checkout",
  "request_id": "7f3c2c4d",
  "user_id": "u_123",
  "error": "card_declined",
  "duration_ms": 340
}

Same field names. Every service. No exceptions. The request_id alone will save you hours of debugging per incident. I can’t stress this enough. Propagate it through HTTP headers, queue messages, background jobs. Everywhere.

Metrics: Prometheus + Grafana. Still the best bang for buck. But here’s the thing – don’t build 30 dashboards. Build one per service. Three panels:

1. Request rate and error rate (tells you if something is broken)
2. Latency percentiles (tells you if it’s degrading)
3. Recent deploys and config changes overlaid on the graphs (tells you what caused it)

That’s your dashboard. If a new engineer can’t look at it and understand the health of the service in 30 seconds, strip it down further.

Traces: Jaeger or Zipkin, but only if you have more than two services. If you’re a monolith with a database and a cache, traces are overhead you don’t need yet. Just use request IDs in your logs. Seriously.

When you do need traces, bump sampling to at least 10% on critical paths. 0.1% default sampling means you’ll never have a trace for the request that actually broke.

Alerts: Less is more. Every alert that pages someone at 3am and isn’t actionable is erosion of trust. Once your team stops trusting alerts, you’ve lost. They’ll start muting channels. I’ve seen it happen.

My rule: every alert needs three things.

• A condition that’s actually abnormal (not “CPU above 60%”)
• A link to the relevant dashboard
• A link to a runbook that says what to do first

If you can’t write those three things for an alert, the alert shouldn’t exist.

The request_id sermon

I keep coming back to this because it’s the single highest-leverage thing you can do.

At the fintech startup we had services talking to services talking to queues talking to workers. When something went wrong without correlation IDs, the debugging process was: check this log, then that log, then maybe this other log, hope the timestamps line up, piece it together manually. Took forever.

After we standardized on a single request_id header propagated everywhere? Same investigation. One search. Done.

The implementation is trivial. Middleware that reads X-Request-ID from incoming requests. Generates a UUID if missing. Passes it along. Logs it on every line. Takes an afternoon to implement across your whole stack.

An afternoon of work for months of saved debugging time. That’s the kind of trade I like.

Runbooks: the unsexy high-leverage tool

Nobody wants to write runbooks. I get it. But here’s the scenario: it’s 2am in your time zone. The alert fires. The person on call is in a different country. They’ve been on the team for three weeks.

Without a runbook, they’re messaging people, waiting for responses, guessing. With a runbook, they open it, follow the steps, and either fix it or know exactly who to escalate to.

Keep them short. Keep them next to the code. Update them after every incident. A runbook that says “check the database connection pool, then check Redis, then check the upstream API timeout” is worth more than a 50-page incident response process document nobody has read.

Mistakes I keep seeing

Collecting everything. Storage is cheap. Cardinality explosions aren’t. I watched a team’s Prometheus instance fall over because they added a user_id label to a counter. Millions of time series. Dead monitoring system. During an outage. Ironic.

Dashboard graveyards. Thirty dashboards, twenty-eight of which nobody has looked at in months. Two of which are actually useful but you can’t remember which ones. Delete aggressively.

Happy path instrumentation only. Your error paths need more instrumentation than your happy paths. The happy path works. You know this because nobody is complaining. The error paths are where surprises live.

Separate conventions per team. One team calls it user_id, another calls it userId, a third calls it uid. Now your cross-service queries are a mess. Pick a convention. Enforce it in code review. This is boring work that pays off enormously.

What to measure about your observability itself

One meta-metric I track: time from “something seems wrong” to “I know what changed and where to look.” If that number is going down over time, your observability is working. If it’s flat or going up, you’re adding complexity without adding clarity.

The other one: how many alerts fired this week that didn’t need a human response? If it’s more than 20%, you have a noise problem.

Start here

If you’re starting from scratch with a small distributed team, do this in order:

1. Structured JSON logs with a shared request_id. One week of work, max.
2. One Grafana dashboard per service with the three panels I mentioned. Another week.
3. Three to five alerts that are actually actionable. A few days.
4. Short runbooks for those alerts. A day.

That’s a month of work spread across your team. After that, you have a system that answers the three questions. Everything else – traces, SLOs, error budgets, custom metrics – layer it on when you feel the pain, not before.

Don’t let perfect be the enemy of “I can actually debug production at 2am without waking up three people.”

Then Reuters published a correction to an earnings report. The correction event arrived, got processed, updated the record. Except three other consumers had already read the original event, cached it, and built derivative data on top of it. The correction propagated to the main feed within seconds. The analytics service picked it up four minutes later. The notification service? It had already sent push notifications to eight thousand users with the wrong number.

Nobody’s data was technically “lost.” Everything was eventually consistent. But “eventually” turned out to be forty minutes, and by then a bunch of users had made trading decisions based on stale information. The postmortem wasn’t fun.

That incident taught me more about event-driven architecture than any conference talk. Events are easy to publish. The hard part is everything that happens after.

Quick take

Event-driven architecture decouples your services beautifully and then introduces a whole new category of bugs you’ve never debugged before. It’s worth it – sometimes. The key is idempotent consumers, an outbox pattern for reliable publishing, and accepting that you’re trading request-response simplicity for operational complexity. Go into it with open eyes or don’t go at all.

Events vs. commands

This distinction matters more than people think. Get it wrong and your “loosely coupled” system is actually tightly coupled with extra steps.

Events are facts. Past tense. “OrderPlaced.” “UserRegistered.” “PriceCorrected.” The producer publishes what happened and doesn’t care who’s listening. Could be zero consumers. Could be fifty.

Commands are directives. “SendEmail.” “ChargeCard.” “UpdateCache.” Targeted at a specific service. The sender expects something to happen.

At Decloud, I caught us mixing these constantly in the early days. A service would publish “ProcessPayment” as an “event.” That’s not an event. That’s a command wearing an event’s clothes. And it created an invisible dependency – if the payment service went down, the publisher had to care, which defeated the whole point.

Rule of thumb: if removing all consumers would break the producer’s logic, it’s a command, not an event.

The event contract is your public API

This is where the fintech startup’s correction bug really originated. We treated events like internal implementation details. Field names leaked database columns. Payloads included internal IDs that only made sense inside one service.

Your event schema is a contract. Treat it like a public API. Version it. Document it. Don’t shove your ORM struct into JSON and call it a day.

Here’s the envelope pattern we settled on at Decloud:

{
  "id": "7f2b5a1e-6c0a-4c2c-9a7a-9c2c6d49f9a2",
  "type": "pricing.QuoteUpdated",
  "source": "pricing-service",
  "time": "2020-07-06T10:30:00Z",
  "version": 2,
  "correlationId": "req-8f43",
  "data": {
    "instrumentId": "AAPL",
    "price": 364.11,
    "currency": "USD"
  }
}

The correlationId saved us countless debugging hours. When a user reports something wrong, you grep for that ID across every service’s logs and see the full journey. Without it, debugging event-driven systems is archaeology.

Idempotent consumers or die

Every broker I’ve worked with – Kafka, RabbitMQ, NATS – delivers at-least-once. Duplicates will happen. Network blip, consumer restart, rebalance. Your consumer sees the same event twice. Or five times.

If your handler isn’t idempotent, you charge a credit card twice. Or send the same notification to eight thousand people. Again.

Here’s the pattern we use in Go at Decloud:

type EventHandler struct {
    db    *sql.DB
    topic string
}

func (h *EventHandler) Handle(ctx context.Context, event Event) error {
    tx, err := h.db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    // Check if we've already processed this event
    var exists bool
    err = tx.QueryRowContext(ctx,
        "SELECT EXISTS(SELECT 1 FROM processed_events WHERE event_id = $1)",
        event.ID,
    ).Scan(&exists)
    if err != nil {
        return fmt.Errorf("check idempotency: %w", err)
    }
    if exists {
        return nil // Already processed. Done.
    }

    // Do the actual work
    if err := h.processEvent(ctx, tx, event); err != nil {
        return fmt.Errorf("process event: %w", err)
    }

    // Record that we've processed this event -- same transaction
    _, err = tx.ExecContext(ctx,
        "INSERT INTO processed_events (event_id, topic, processed_at) VALUES ($1, $2, NOW())",
        event.ID, h.topic,
    )
    if err != nil {
        return fmt.Errorf("record processed event: %w", err)
    }

    return tx.Commit()
}

The important bit: the idempotency check and the business logic live in the same database transaction. If the process crashes after doing the work but before recording the event ID, the transaction rolls back and the event gets reprocessed safely. If it crashes after commit, the duplicate delivery gets caught by the EXISTS check.

This pattern handles 99% of our idempotency needs. The remaining 1% involves external side effects (sending emails, calling third-party APIs) which need a different approach.

The outbox pattern

This one took me too long to learn. The naive approach: save to database, then publish event.

// DON'T DO THIS
func (s *OrderService) PlaceOrder(ctx context.Context, order Order) error {
    if err := s.db.SaveOrder(ctx, order); err != nil {
        return err
    }
    // What if we crash right here?
    return s.publisher.Publish(ctx, "orders.OrderPlaced", order)
}

If you crash between the database write and the publish, the order exists but the event never fires. Downstream services never learn about it. Data silently drifts.

The fix is the outbox pattern. Write the event to an outbox table in the same transaction as your business data. A separate process polls the outbox and publishes.

func (s *OrderService) PlaceOrder(ctx context.Context, order Order) error {
    tx, err := s.db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    if err := s.saveOrder(ctx, tx, order); err != nil {
        return fmt.Errorf("save order: %w", err)
    }

    evt := OutboxEvent{
        ID:        uuid.New().String(),
        Type:      "orders.OrderPlaced",
        Payload:   marshal(order),
        CreatedAt: time.Now(),
    }
    if err := s.saveOutboxEvent(ctx, tx, evt); err != nil {
        return fmt.Errorf("save outbox event: %w", err)
    }

    return tx.Commit()
}

And the publisher, running separately:

func (p *OutboxPublisher) Poll(ctx context.Context) error {
    events, err := p.db.FetchUnpublished(ctx, 100)
    if err != nil {
        return fmt.Errorf("fetch unpublished: %w", err)
    }

    for _, evt := range events {
        if err := p.broker.Publish(ctx, evt.Type, evt.Payload); err != nil {
            // Log and continue. We'll retry on next poll.
            log.Printf("publish failed for event %s: %v", evt.ID, err)
            continue
        }
        if err := p.db.MarkPublished(ctx, evt.ID); err != nil {
            // Event published but not marked. Will be re-published.
            // Consumer idempotency handles the duplicate.
            log.Printf("mark published failed for event %s: %v", evt.ID, err)
        }
    }
    return nil
}

Is it more code? Yes. Does it guarantee that every committed business change produces an event? Also yes. The outbox publisher can crash, restart, and pick up where it left off. Consumers handle the duplicates. The system converges.

At Decloud, we run the outbox poller on a 500ms tick. Good enough for our latency requirements. If you need sub-100ms, look into CDC (change data capture) with something like Debezium reading the WAL directly. Same principle, lower latency, more operational overhead.

Picking a broker

I’m not going to do a feature comparison chart. Those are outdated the moment you publish them. Here’s how I think about it instead.

Kafka if you need ordered replay. We use it at Decloud for anything where “play back the last 7 days of events” is a real requirement. The retention model is its killer feature. The operational cost is its tax. Running Kafka well requires dedicated attention.

RabbitMQ if you need flexible routing and work queues. Fan-out, topic exchanges, priority queues. If your pattern is “distribute work across competing consumers,” Rabbit is easier to reason about than Kafka consumer groups.

NATS if you want something lightweight. We use NATS for internal service communication that doesn’t need persistence. Fire-and-forget telemetry, cache invalidation signals, that kind of thing. JetStream adds persistence if you need it, but at that point evaluate whether Kafka serves you better.

Managed services (SQS, EventBridge, Pub/Sub) if operational simplicity matters more than configurability. At Decloud we run on AWS, and for non-critical event flows, SQS with a dead-letter queue is hard to beat for simplicity.

Ordering is a lie (mostly)

Kafka guarantees ordering per partition. RabbitMQ doesn’t guarantee ordering at all with competing consumers. NATS JetStream gives you ordering per stream but not per subject with multiple consumers.

In practice, design for out-of-order delivery. If you can’t, partition by the entity that needs ordering (e.g., all events for order ID “abc” go to the same partition).

// Partition key ensures all events for the same order
// land on the same Kafka partition
func partitionKey(event Event) string {
    switch e := event.Data.(type) {
    case OrderEvent:
        return e.OrderID
    case UserEvent:
        return e.UserID
    default:
        return event.ID // fallback: event-level ordering only
    }
}

At the fintech startup, we partitioned by instrument ID. All events for AAPL went to the same partition. Ordering within an instrument was guaranteed. Ordering across instruments didn’t matter.

Dead letters and retries

When a consumer fails, retry with exponential backoff. After N retries, send the event to a dead-letter queue. Don’t retry forever. I’ve seen a single poison event take down a consumer group for hours because it kept crashing and restarting in a loop.

func (h *RetryHandler) HandleWithRetry(ctx context.Context, event Event) error {
    var lastErr error
    for attempt := 0; attempt <= h.maxRetries; attempt++ {
        if attempt > 0 {
            backoff := time.Duration(attempt*attempt) * 100 * time.Millisecond
            if backoff > 10*time.Second {
                backoff = 10 * time.Second
            }
            time.Sleep(backoff)
        }

        lastErr = h.inner.Handle(ctx, event)
        if lastErr == nil {
            return nil
        }
        log.Printf("attempt %d failed for event %s: %v", attempt+1, event.ID, lastErr)
    }

    // Max retries exceeded. Dead-letter it.
    if err := h.deadLetter.Send(ctx, event, lastErr); err != nil {
        return fmt.Errorf("dead letter failed for event %s: %w", event.ID, err)
    }
    return nil // Event is safely in the DLQ. Consumer can move on.
}

The dead-letter queue isn’t a trash can. Set up alerts. Review it weekly. At Decloud we have a Slack alert for any event hitting the DLQ and a weekly review where we categorize failures: transient (infra blip), poison (bad data), or bug (our code).

When events are the wrong answer

I’ve seen teams go all-in on event-driven architecture for a CRUD app with three services. Don’t.

Events make sense when:

• Multiple consumers need to react to the same change independently
• Temporal decoupling matters – the producer shouldn’t wait for or even know about consumers
• Data pipelines need a replay-friendly stream of changes
• Cross-team boundaries where synchronous coupling would create deployment dependencies

Events are overhead when:

• You have one producer and one consumer. That’s a function call with extra steps.
• You need synchronous request-response. An HTTP call is simpler and easier to debug.
• The system is small enough that a monolith or simple service-to-service calls work fine.

At Decloud, roughly 40% of our inter-service communication is event-driven. The rest is gRPC. That ratio feels right. Event-driven for fan-out and cross-domain integration. Direct calls for everything else.

Observability or it didn’t happen

You can’t debug an event-driven system without distributed tracing. Full stop.

Every event carries a correlation ID. Every consumer logs that ID. When something goes wrong, you search by correlation ID and see the full chain: which service published, who consumed, what happened next.

We also track:

• Consumer lag – how far behind each consumer group is. If it’s growing, you’re either underprovisioned or have a bug.
• Processing time per event type – catches performance regressions early.
• Dead-letter rate – our canary. A spike means something broke.
• End-to-end latency – time from event publication to final consumer processing. This is the number users actually feel.

Event-driven architecture is a trade. You give up the simplicity of “service A calls service B and gets a response.” In return you get decoupling, independent scaling, and resilience to individual service failures. Whether that trade is worth it depends on your system, your team, and how much operational maturity you have.

If you can’t answer “how do I debug a failed event end-to-end?” before you start building, you’re not ready. Get observability in place first. The events can wait.

Every replication pattern is a bet on which failure you can tolerate. Pick wrong and you either lose data or lose availability. There’s no option C.

At the fintech startup we ingested financial news and market data from dozens of sources into PostgreSQL. Millions of rows per day. The kind of dataset where “eventually consistent” means a trader sees yesterday’s price and you get a very angry phone call.

That experience shaped how I think about replication. Not as a feature you toggle on, but as an architectural decision that determines how your system fails. Because it will fail.

Why you replicate

The reasons are fewer than people think:

• Availability. Your primary dies, a replica takes over, users keep working.
• Read scaling. Heavy reporting queries stop competing with writes.
• Latency. Put a copy closer to users so reads don’t cross the Atlantic.
• Isolation. Backups, analytics, migrations – all without touching production writes.

That’s basically it. If your reason isn’t one of these, you probably don’t need replication yet. You need a better backup strategy.

The three modes

Asynchronous

The primary commits, returns success to the client, then ships WAL records to replicas whenever it gets around to it.

-- postgresql.conf on the primary
wal_level = replica
max_wal_senders = 5
-- No synchronous_standby_names. That's the point.

The replica connects with a primary_conninfo and streams changes:

# recovery.conf (or standby.signal in PG12+)
primary_conninfo = 'host=primary-db port=5432 user=replicator'

This is the fastest mode for writes. The primary doesn’t wait for anyone. But there’s a window – could be milliseconds, could be seconds under load – where a replica is behind. If the primary dies in that window, those transactions are gone.

At the fintech startup, async replication was fine for our read replicas serving the news feed. A 200ms lag on a news article? Nobody notices. A 200ms lag on a stock price used for trading signals? Different conversation entirely.

Synchronous

The primary waits for at least one replica to confirm it wrote the WAL to disk before telling the client “committed.”

-- postgresql.conf
synchronous_standby_names = 'replica1'
synchronous_commit = on

Zero data loss on failover. Sounds great. The cost: every single write now includes a network round trip to the replica. In the same datacenter, maybe 1-2ms added latency. Across regions? 50-150ms. Per write.

Worse: if that replica goes down, your primary blocks all writes until the replica comes back or you reconfigure. I’ve seen this take down production systems that were “highly available.” The replication designed for availability became the single point of failure.

Quorum (semi-synchronous)

The pragmatic middle ground. Wait for any N out of M replicas to acknowledge, not all of them.

-- Wait for any 1 of these 3 replicas
synchronous_standby_names = 'ANY 1 (replica1, replica2, replica3)'

One replica can die and writes keep flowing. You still get the durability guarantee because at least one standby has the data. This is what I’d recommend for most production PostgreSQL setups that need strong durability.

The math is simple: if you have 3 standbys and require ANY 1, you can lose 2 replicas before writes stall. With ANY 2, you can lose 1. Pick based on how many failures you want to survive simultaneously.

Topologies

Primary-replica (the default)

One writer, N readers. Dead simple. Works for the vast majority of applications.

  Writes           Reads
    |                |
    v                v
[Primary] ----> [Replica 1]
    |
    +---------> [Replica 2]

If you’re building a new system and someone suggests multi-primary on day one, push back hard. Start here.

Cascading replicas

A replica can feed other replicas downstream instead of every replica pulling from the primary. This matters when you have many replicas or they span continents.

-- On the cascading replica (replica2 feeds from replica1, not primary)
primary_conninfo = 'host=replica1 port=5432 user=replicator'

The tradeoff: more lag for downstream replicas. Each hop adds latency. But the primary’s max_wal_senders slots aren’t exhausted by a fleet of read replicas. We used this at the fintech startup to feed analytics replicas off a primary standby – kept the main failover target clean while analytics could thrash their copy however they wanted.

Multi-primary

Multiple nodes accept writes and sync with each other. Sounds appealing. Is usually painful.

The fundamental problem: two nodes write conflicting data to the same row at the same time. Now what?

PostgreSQL’s logical replication can do this with BDR or similar extensions, but you’re signing up for conflict resolution logic, operational complexity, and debugging sessions that make you question your career choices. MySQL Group Replication and Galera have the same fundamental tradeoffs.

Use multi-primary when you have offices in multiple continents that all need local write latency and the business has accepted the cost. Not because it sounds cool on a whiteboard.

Read-your-writes consistency

This is where async replication bites you in the application layer.

User writes a comment. POST succeeds (hits the primary). Page refreshes. GET goes to a replica. Comment isn’t there. User writes the comment again. Now you have duplicates.

Three approaches, from simple to robust:

1. Sticky routing after writes

func handleRequest(w http.ResponseWriter, r *http.Request) {
    cookie, _ := r.Cookie("last_write")
    if cookie != nil && time.Since(parseCookieTime(cookie)) < 5*time.Second {
        // Recent write -- route to primary
        db = primaryDB
    } else {
        db = replicaDB
    }
    // ...
}

Crude but effective. After a write, force reads to the primary for a few seconds. The window just needs to exceed your typical replication lag.

2. Session affinity to a specific replica

Pin a user session to one replica. They won’t see inconsistency within their own session because they always read from the same copy. Doesn’t help with cross-user consistency but handles the most visible problem.

3. LSN tracking

The proper solution. After a write, capture the WAL position (LSN) from the primary. Before reading from a replica, check if the replica has replayed past that LSN.

-- After write, on the primary:
SELECT pg_current_wal_lsn();
-- Returns something like: 0/16B9188

-- Before read, on the replica:
SELECT pg_last_wal_replay_lsn();
-- If this >= the saved LSN, the replica is caught up

More work to implement. More correct.

Conflict resolution in multi-primary

If you went down the multi-primary path despite my warning, you need a conflict strategy.

Last write wins (LWW). Timestamp comparison. Whichever write has the later timestamp survives. Simple, lossy. A perfectly valid update gets silently dropped because a clock was 1ms behind. If your data can tolerate silent overwrites, fine. Financial data can’t.

Application-level merge. You define the merge logic per table or per field. A counter gets added. A set gets unioned. A text field takes the longer version. This works but every schema change means updating merge rules.

CRDTs. Conflict-free replicated data types. Mathematically guaranteed to converge. Limited to specific data structures: counters, sets, registers. You can’t CRDT your way through an arbitrary relational schema. Good for specific use cases, not a general solution.

Failover

Promoting a replica sounds easy. pg_promote() or SELECT pg_promote() in PG12+. Done. Except no.

The hard parts:

Fencing the old primary. If the old primary isn’t actually dead – just slow, or network-partitioned – you now have two nodes accepting writes. Split brain. The nightmare scenario. You need STONITH (shoot the other node in the head) or at minimum, revoke the old primary’s ability to accept connections.

Client reconnection. Your application’s connection string points to the old primary. Now it needs to point to the new one. Options: DNS update (slow propagation), virtual IP (fast but requires infra), connection proxy like PgBouncer or HAProxy that handles routing, or application-level logic.

# HAProxy config for automatic failover routing
listen postgres
    bind *:5432
    option httpchk GET /primary
    server pg1 10.0.1.1:5432 check port 8008
    server pg2 10.0.1.2:5432 check port 8008

Rebuilding the old primary. After failover, the old primary has diverged. It accepted some writes the new primary doesn’t have (or vice versa). You need to either pg_rewind it or rebuild from scratch with pg_basebackup. Test this process before you need it.

I’ve run failover drills at every company I’ve worked at. The drill always surfaces something the documentation missed. Always.

Monitoring replication

Replication lag is a ticking bomb with a variable fuse. You need to watch it continuously.

-- On the primary: check all connected replicas
SELECT
    client_addr,
    state,
    sent_lsn,
    write_lsn,
    flush_lsn,
    replay_lsn,
    pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

Key things to alert on:

• Replay lag exceeding your tolerance. If you promised “reads within 1 second of writes,” alert before that threshold.
• Replica disconnection. A replica that silently falls off is worse than one that loudly crashes.
• WAL accumulation on the primary. If a replica can’t keep up, the primary retains WAL segments. Disk fills. Primary crashes. You’ve now lost the thing that was supposed to protect you.

-- Check WAL retention on the primary
SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0') AS total_wal_bytes;

-- Or more practically, watch the pg_wal directory size
-- and alert when it grows beyond expected bounds.

Choosing your pattern

Skip the decision matrix. Ask three questions:

Can you lose any committed transactions? If no, synchronous or quorum replication. Full stop. This was non-negotiable for the financial data at the fintech startup. A lost trade record isn’t a bug, it’s a regulatory incident.

Do you need writes in multiple regions? If no – and it’s almost always no – use primary-replica. If yes, accept the operational cost of multi-primary and budget engineering time accordingly. At Decloud, we keep writes centralized and replicate reads out. Simpler. Fewer 3am pages.

What’s your read-to-write ratio? If reads dominate (10:1 or more), async replicas for reads with synchronous replication to a single failover standby gives you the best of both: fast reads, safe failover, and write performance that’s only slightly penalized.

Replication isn’t a feature flag. It’s a contract between your system and your users about what happens when hardware fails, networks partition, and Murphy’s Law does its thing. Pick the pattern that matches the promises you’ve actually made, not the ones you wish you could make.

At a mobility startup we processed GPS telemetry from thousands of bikes across the city. Real IoT. Real device data. Real volume. You’d think that’s a textbook edge case. It wasn’t. A well-tuned message queue and a couple of regional cloud instances handled everything we needed. Sub-second was easy without edge infrastructure.

The edge is a spectrum, not a destination

Device. Network edge. Regional edge. Cloud. Each hop adds latency, sure. But each hop closer to the device also adds operational pain: deployment complexity, consistency headaches, debugging in the dark.

The math only works when you have a genuine constraint:

• Latency below 50ms on a critical path. Not aspirational latency. Measured, user-impacting latency.
• Bandwidth costs that actually hurt. If you’re shipping raw video or sensor streams, processing locally makes sense. If you’re sending JSON, it doesn’t.
• Unreliable connectivity. Factory floors, moving vehicles, rural deployments. The network genuinely can’t be trusted.
• Data residency requirements. Regulation says the data stays put. Fine. That’s a real constraint.

If none of those apply, you’re adding distributed systems complexity for a problem you don’t have.

What edge computing actually costs you

Consistency goes out the window. You’re now designing for eventual convergence, conflict resolution, and partial failures at every node. Your deployment pipeline needs to handle hundreds of locations instead of a handful of regions. Observability becomes sampling and aggregation because you can’t ship raw telemetry from every edge node without defeating the purpose.

Security surface area multiplies. Every edge node is a potential compromise point. You’re trusting code running in locations you don’t physically control.

Now at Decloud, we’re building cloud infrastructure tooling. I see teams adopt edge patterns and then spend months building the operational scaffolding that a centralized architecture gives you for free. The edge didn’t solve their problem. It replaced one set of problems with a harder set.

When it’s actually worth it

CDN edge compute is the one place where the tradeoff is almost always favorable. Auth checks, request routing, simple personalization – these are stateless, short-lived operations running on infrastructure someone else manages. Cloudflare Workers and similar products nailed this by keeping the programming model simple and the blast radius small.

Beyond CDN, edge computing earns its complexity in exactly three scenarios: real-time industrial control, high-volume media processing at the source, and latency-sensitive interactive applications where every millisecond is measurable in revenue.

Everything else? Run it in the cloud. Optimize your queries. Use a CDN. Move on.

Bottom line

Edge computing is a targeted optimization, not an architecture paradigm. Treating it as a default is resume-driven development dressed up as forward thinking. Start with the simplest thing that works, measure where it doesn’t, and push compute to the edge only when the numbers force your hand.

That experience taught me something most architecture blogs skip over: multi-region isn’t a scaling decision. It’s a commitment. You’re signing up for a permanent increase in operational surface area, and most teams underestimate what that means on a Tuesday night when replication lag spikes and nobody remembers which region owns writes.

When It Actually Makes Sense

Three situations. That’s it.

Your users are genuinely global and latency affects the product. Not “we have a few customers in Asia.” I mean latency is measurably hurting conversion, trust, or functionality. At the fintech startup, financial data delayed by hundreds of milliseconds was functionally wrong. That’s a real reason.

Compliance requires data residency. GDPR was forcing this conversation for European users. If you must keep EU citizen data in EU, you need a region there. No architecture cleverness gets around legal requirements.

A regional outage is an existential threat. If your product going down for four hours costs more than running a second region for a year, the math works. For most startups I’ve seen – including what we’re building now at Decloud in my founder-program cohort – it doesn’t.

When It Doesn’t

Here is a test I use: if your team doesn’t have 24/7 on-call coverage today, you aren’t ready for multi-region. Full stop.

A second region doesn’t help if nobody is awake to failover when it matters. I’ve watched startups deploy to three regions and then have a single engineer handle incidents at 3am because “the architecture is redundant.” The architecture isn’t the bottleneck. The people are.

Single-region with multiple availability zones handles most failure scenarios. It’s cheaper, the mental model is simpler, and your deploys don’t need a coordination protocol. If you’re a team under twenty engineers, start here and stay here until the pain is specific and measurable.

The Patterns, Briefly

Active-passive is the safe choice. One region handles writes. The other sits warm, ready for failover. You pay for underutilized infrastructure, but your data model stays sane. This is what I would recommend for most teams dipping a toe in.

Active-active is the ambitious choice. Both regions serve traffic and accept writes. Latency improves for everyone. Failover is faster because traffic already flows everywhere. But now you need conflict resolution, and conflict resolution is where engineers go to suffer.

Follow-the-sun is niche. The primary region rotates with time zones. It works for batch workloads and internal tools. I’ve never seen it work well for user-facing products.

Data Will Ruin Your Week

Every multi-region conversation eventually becomes a data conversation. You can handwave traffic routing and load balancing. You can’t handwave “what happens when two regions write to the same row at the same time.”

Classify your data before you do anything else:

• Global and consistent: User accounts, billing state, permissions. This data must be correct everywhere. Synchronous replication or a single write region.
• Regional and isolated: User-generated content tied to geography, local caches. This can live in one region without drama.
• Derived and disposable: Caches, search indexes, computed feeds. Rebuild it if it breaks. Don’t replicate it.

Synchronous replication gives you consistency but adds cross-region latency to every write. Async replication keeps things fast but introduces eventual consistency, which is a polite way of saying “your users might see stale data and you need a plan for that.”

Conflict resolution deserves its own paragraph because it deserves your fear. Last-write-wins sounds simple until you realize your clock synchronization across regions isn’t as tight as you assumed. Application-level merge logic sounds correct until you realize every new feature needs to account for it. CRDTs sound elegant until you realize they only work for specific data structures. Pick your poison. Test it under failure conditions, not just happy paths.

Traffic Routing

Three options, increasing in complexity:

DNS-based routing is cheap and simple. It’s also slow to failover because DNS caching means some users will hit the wrong region for minutes after a switch. Fine for read-heavy traffic. Dangerous if you need fast failover.

Global load balancers give you health checks and faster failover. They cost more and add operational surface. Worth it if your availability targets demand sub-minute recovery.

Application-level routing lets clients or APIs pick a region based on account or data ownership. Maximum flexibility. Maximum chance of a subtle routing bug sending writes to the wrong region.

The Honest Cost

Multi-region costs aren’t just “two of everything.” The infrastructure doubling is the easy part to budget. The hard costs are:

• Cross-region data transfer fees. These add up fast and nobody notices until the bill arrives.
• Doubled deploy pipelines, doubled monitoring, doubled alerting noise.
• On-call engineers who now need to understand two regions and the interactions between them.
• Every new feature ships slower because someone has to ask “does this work multi-region?”

At a startup accelerator, I watched teams burn runway on infrastructure sophistication that their user base didn’t justify. The startup that nailed single-region reliability shipped faster than the one with a beautiful multi-region setup and a three-person team drowning in operational overhead.

My Actual Advice

If you’re reading this in 2019 and wondering whether to go multi-region: probably don’t. Deploy to a single region with multiple availability zones. Set up proper backups and a disaster recovery plan. Get your deployment pipeline fast enough that you can ship fixes in minutes, not hours.

When the pain becomes specific – real latency complaints from real users in a real geography, a compliance requirement with a real deadline, a post-mortem that shows a regional outage cost real money – then revisit. Start with active-passive. Keep your write path simple. Accept that you’re trading velocity for resilience and make that trade deliberately.

Multi-region isn’t wrong. But it’s almost never urgent, and doing it before you’re ready makes everything else slower.

At the fintech startup, I watched a single slow Elasticsearch query cascade through our entire API layer. One degraded dependency. Total platform outage. The fix took ten minutes. The recovery took four hours. All because nothing in the request path had a timeout.

At a mobility startup, a forgotten WAL retention setting filled a disk at 3 AM and I discovered our “tested” failover was eleven hours behind. Forty minutes of locked bikes across the city. The monitoring said everything was fine. The monitoring was wrong.

These weren’t exotic failures. They were boring, preventable ones. The kind that happen when you assume dependencies work and never verify what happens when they don’t.

Three rules I actually follow

Set a deadline on everything. Every outbound call gets a timeout. Every request gets a budget. If a dependency can’t answer in time, you move on without it. Slow failure is worse than fast failure because it holds resources hostage while it dies.

Isolate the blast radius. A slow search index should never starve your payment flow. Separate connection pools. Separate queues. The goal is simple: one problem stays one problem.

Know your fallback before you need it. A stale cache hit is better than a 500. A default list of popular items is better than a blank page. But the fallback has to be intentional. Accidental fallbacks are just bugs you haven’t noticed yet.

The pattern that keeps saving me

Circuit breakers. Dead simple concept. If a dependency is failing, stop calling it. Serve the fallback. Check back later. It turns a cascading outage into a graceful degradation that most users never notice.

The key insight: a breaker that’s open isn’t a failure state. It’s a success state. It means the system chose fast, predictable behavior over slow, unpredictable death.

What I got wrong early on

I used to think resilience meant more redundancy. Add a replica. Add a region. Add a retry. But redundancy without testing is just a more expensive single point of failure. That mobility startup’s replica was a perfect example. It existed. It was running. It was useless.

Now I test the recovery path, not just the happy path. If you haven’t promoted your replica under realistic conditions in the last quarter, you don’t have a failover. You have a hope.

The uncomfortable truth

Designing for failure isn’t a technical problem. It’s a prioritization problem. Every founder and every CTO knows they should do it. Most don’t because the next feature feels more urgent. It always feels more urgent.

Until 3 AM on a Thursday, when it doesn’t.

That incident rewired how I think about distributed systems. The spectacular failures – network partitions, node crashes, disk corruption – those are almost easy. You plan for them because they’re dramatic. The killers are the subtle ones. The slow dependency that doesn’t quite timeout. The clock drift that makes your event ordering nonsensical. The duplicate message that creates two charges on someone’s credit card.

Partial Failure Is the Default State

At the fintech startup we pulled financial data from dozens of sources, ran NLP pipelines, scored relevance, and served personalized feeds. At any given moment, something in that chain was degraded. Not down. Degraded. A source returning stale data. A scoring service running hot. A queue backing up.

Once I accepted that partial failure is the normal operating condition, everything clicked. You don’t design for “what if something breaks.” You design for “something is always broken, how do we still deliver value?”

That means setting explicit targets. Not vague “five nines” aspirations, but concrete numbers: P99 latency under 400ms for feed requests, error rate below 0.1% for content delivery. Then you design every component to hold those targets even when a dependency is misbehaving.

The Patterns That Actually Saved Us

Timeouts. Everywhere. No Exceptions.

I can’t overstate this. Every network call needs a timeout. Not just HTTP requests – database queries, cache lookups, message publishes. All of them.

response = http.get(url, timeout=2)

Two seconds. That’s it. If your financial data provider can’t answer in two seconds, we move on. We had a service early on with no timeout on a third-party API call. That provider had a bad day, responses went from 200ms to 45 seconds, and our entire request pipeline backed up behind it. Thread pools exhausted. Cascading failure across three services. All because of one missing timeout.

Even better: use end-to-end deadlines. If the user’s request has a 3-second budget, propagate that deadline downstream. Don’t let an inner service burn 2.8 seconds and leave nothing for the rest of the chain.

Retries That Don’t Make Things Worse

Retries are a double-edged sword. Done right, they mask transient blips. Done wrong, they turn a struggling service into a dead one.

for attempt in range(max_retries):
    try:
        return call()
    except RetryableError:
        sleep(backoff(attempt))

Exponential backoff with jitter. That jitter part matters – without it, all your retrying clients slam the recovering service at the exact same intervals. Thundering herd. We learned to cap retry budgets too. Three attempts max, then fall back to cached data or a degraded response. Retrying forever is just a distributed denial-of-service attack against yourself.

Circuit Breakers

After the empty-response incident, we added circuit breakers on every external dependency. If a service fails five times in a row, stop calling it. Try again in 30 seconds with a single probe request. If the probe succeeds, gradually let traffic back through.

This is the pattern that prevented the most cascading failures for us. Simple concept, massive impact.

Bulkheads

We ran our NLP pipeline and our content delivery API on shared infrastructure early on. The NLP jobs were CPU-hungry beasts. During a big news event – earnings season, a market crash – the pipeline would spike and starve the API of resources.

Solution: isolation. Separate thread pools, separate connection pools, separate queues. One misbehaving component can’t eat another component’s lunch. Same principle as watertight compartments on a ship. The Titanic metaphor is overused but accurate.

Fallbacks Over Failures

When our relevance scoring service was slow, we didn’t show users an error page. We showed them a feed sorted by recency instead. Less personalized, still useful. When a data source was down, we served cached content with a “last updated” timestamp.

Define what a degraded-but-useful response looks like for every endpoint. Caches, defaults, partial results. Users almost always prefer stale data over no data.

Idempotency

In a system with retries and at-least-once message delivery, duplicate operations are inevitable. Every write operation needs to be safe to execute twice.

POST /orders
Idempotency-Key: 6f9c2c70-4f0e-4f36-8a6a-3b5a9f4e3c2d

We tagged every content ingestion event with a unique key. If the same article arrived twice from different sources or from a retry, the system recognized the duplicate and skipped it. Without this, our feeds would have been full of repeated headlines during any period of instability.

Consistency: Pick Your Battles

Strong consistency across a distributed system is expensive and fragile. We tried it early on with a two-phase commit pattern for content updates. It was slow, it was brittle, and it failed in weird ways under load.

We switched to eventual consistency with explicit conflict resolution. Version numbers on every content record. Last-writer-wins for most fields, merge logic for aggregated scores. Sagas with compensating actions for multi-step workflows – if step three fails, step two gets rolled back automatically.

The key insight: design your conflict resolution strategy before you need it. Under pressure at 2am isn’t when you want to be inventing reconciliation logic.

Operations Are Half the Battle

The best-designed system falls apart without operational discipline.

Health checks: We exposed both liveness (process is running) and readiness (process can serve traffic and reach its dependencies) endpoints. Kubernetes used these to route around unhealthy instances automatically.

Observability: Metrics for rates and latency. Logs for detail. Distributed tracing for following a request across services. After the empty-response incident, we added content-validity checks to our health signals. Green dashboard means nothing if you’re not checking the right things.

Load shedding: When traffic spikes beyond capacity, reject requests explicitly with a 503 rather than letting everything slow to a crawl. Backpressure propagated through the system keeps critical paths alive.

Safe deployments: Canary releases. Feature flags. Fast rollback. We never did big-bang deployments after the first time it went wrong (and it went wrong immediately).

Test for Failure, Not Just Success

Happy-path tests prove your system works when everything is fine. That’s the easy part.

We injected latency into staging environments. Killed instances randomly. Returned garbage from mocked dependencies. Simulated network partitions between services. Every one of these tests caught bugs that would have hit production eventually.

Load testing matters too, but don’t just test peak throughput. Test sustained load. Test spiky traffic. Test what happens when load ramps up while a dependency is already degraded. That combination is what actually happens in production.

The Real Lesson

Every distributed system I’ve worked on has taught me the same thing: failure isn’t an edge case. It’s an input. The question is never “will something fail?” It’s “when this fails, does the system do something reasonable?”

At the fintech startup, the systems that survived best weren’t the ones with the cleverest algorithms or the most redundancy. They were the ones where we’d thought through every dependency and asked: “What do we do when this is gone?” And then actually built the answer.

I spent forty minutes SSH-ing into boxes, tailing logs, and grepping for exceptions. Nothing. The pipeline just… stopped processing. It took me another hour to discover the root cause: a third-party API we depended on had started returning empty 200 responses instead of actual data. Our monitoring checked for errors. It checked for timeouts. It never checked for “success that contains nothing useful.”

That night changed how I think about production systems.

Monitoring Checks Boxes. Observability Answers Questions.

Monitoring is built around things you already know can go wrong. Threshold crossed, alert fires, runbook engaged. It works great for predictable problems: disk filling up, CPU pegged, error rate spiking. We had all of that at the fintech startup. Grafana dashboards everywhere. PagerDuty wired up. It felt safe.

But distributed systems don’t fail in predictable ways. They fail in weird, combinatorial, never-seen-this-before ways. And that’s where monitoring falls apart. You can’t write an alert for a failure mode you haven’t imagined yet.

Observability flips the model. Instead of predefining what questions the system can answer, you instrument it richly enough that you can ask new questions on the fly. During an incident. At 2 AM. Without deploying anything.

The difference: monitoring tells you something is wrong. Observability helps you figure out why.

Three Signals, One Story

After the empty-200 incident, I started rebuilding our telemetry around three pillars.

Metrics give you the bird’s-eye view. Aggregated numbers over time. At the fintech startup, we track request rates, error rates, and latency distributions for every service using Prometheus.

http_requests_total{method="GET", endpoint="/api/stories", status="200"} 15234
http_request_duration_seconds_bucket{endpoint="/api/stories", le="0.5"} 421

Metrics are cheap to store and great for alerting. But they can’t tell you why a specific request failed or what user it affected. They’re the smoke detector, not the fire investigator.

Logs capture the details. Each event, each error, each decision the code made. We moved early to structured JSON logs because free-form strings are nearly useless at scale.

{"timestamp":"2018-07-09T10:30:45Z","level":"error","service":"ingestion","request_id":"abc123","message":"Empty response from provider","provider":"reuters","duration_ms":340}

That structured format meant we could query logs in Kibana instead of grepping through them. Game changer for incident response.

Traces show you the journey of a single request across services. This was the missing piece for us. The fintech startup’s architecture involves an ingestion service talking to NLP processors talking to a ranking engine talking to the API layer. When something is slow, you need to see the whole chain.

Trace abc123
- POST /ingest/batch (120ms)
  - NLP enrichment (45ms)
  - Relevance scoring (30ms)
  - db.write stories (35ms)

We started with Zipkin. Not perfect, but suddenly I could see that our NLP service was adding 200ms of latency on certain content types. That was invisible before.

The Glue: Correlation IDs

Each signal alone is useful. Together, connected by a shared identifier, they’re powerful.

Here’s the workflow that actually matters: a latency alert fires from metrics. You pull up the trace for a slow request. The trace shows a specific span taking too long. You jump to the logs for that span using the same request ID and find the exact query and error.

X-Request-ID: 5f3c9e86c2c84e1b
X-B3-TraceId: 4d1e00a3b9bd1d42
X-B3-SpanId: 6df3a1c2b93f6b1a

We made it a rule: every log line includes the request ID, and every trace propagates context headers. No exceptions. It took weeks to retrofit across all our services, but the payoff during the next incident was immediate. Instead of an hour of archaeology, I had a clear thread to pull.

Instrument the Critical Path First

When I started adding instrumentation at the fintech startup, the temptation was to instrument everything. Don’t do that. You’ll drown in data and your storage costs will spike.

Start with what matters: the critical path your users depend on.

For us that meant inbound API handlers, the news ingestion pipeline, database calls, and every external API dependency. We used RED (rate, errors, duration) for our services and USE (utilization, saturation, errors) for infrastructure resources.

Tracing has a sampling problem. Capturing every single request is expensive. We settled on keeping 100% of errors and sampling successful requests at about 5%. During incidents, we crank sampling up. Good enough for debugging, affordable enough to run continuously.

Designing Systems That Can Be Debugged

Observability isn’t something you bolt on after the fact. It’s a design choice.

Structured logs over free-form strings. Always. If you can’t query it, it’s useless when you’re under pressure at 2 AM.

Watch your cardinality. Metrics with unbounded label values (like user IDs) will destroy your Prometheus instance. High-cardinality data belongs in logs and traces, not metrics.

Add context that matters. A trace that only shows timings is half the story. Include the request type, the tenant, the result count. When something goes wrong, that context is the difference between a five-minute fix and a two-hour hunt.

Our Stack in 2018

For anyone building this out now, here’s what we were running:

• Metrics: Prometheus with Alertmanager, Grafana for dashboards
• Logs: Fluentd piping into Elasticsearch, Kibana for exploration
• Tracing: Zipkin with OpenTracing instrumentation

OpenTracing is getting solid adoption and keeps us from being locked into one vendor. OpenCensus is emerging as an alternative worth watching. Both are pushing toward a world where trace context propagation just works out of the box.

There are commercial options that bundle everything behind one query layer. We looked at them. For our scale, the open source stack made more sense. That calculus changes depending on team size and how much operational overhead you can absorb.

What That 2 AM Incident Taught Me

That silent pipeline failure was a gift. It exposed a blind spot in how I thought about production readiness. Having dashboards isn’t the same as having understanding. Monitoring answers the questions you thought to ask. Observability gives you the ability to investigate the questions you didn’t.

We still have incidents at the fintech startup. But now when something breaks in a way we’ve never seen before, we have the telemetry to figure it out fast. That’s the whole point.

Event sourcing is powerful but unforgiving. Get the aggregate boundaries wrong and you’ll spend months cleaning up. Get them right and you have auditability, replay, and decoupled integrations almost for free.

I’ve been building event-sourced systems at the fintech startup for a while now. We process financial news and market data from hundreds of sources, score relevance in real time, and deliver personalized feeds to users. That pipeline is a natural fit for event sourcing. Every price tick, every news article ingestion, every user interaction – they’re all events that happened at a specific moment, and we need to know exactly what happened and when.

But “natural fit” doesn’t mean “easy.” I’ve made most of the mistakes on this list. This post is what I wish someone had handed me before I started.

The Shift That Changes Everything

In a traditional system you store current state. A row in a table says “Order #789 has status SHIPPED.” You overwrite the old status. History is gone unless you bolted on an audit log.

Event sourcing flips that. You store the facts: OrderPlaced, PaymentReceived, OrderShipped. Current state is derived by replaying those facts. The event log is the source of truth. Everything else – your read models, your dashboards, your search indexes – is a projection you can tear down and rebuild from scratch.

At the fintech startup, this wasn’t a philosophical choice. Regulators wanted to know the exact sequence of data that produced a specific score for a specific user at a specific time. With event sourcing we could answer that question by replaying the stream. With CRUD we would have been guessing.

The Anatomy of an Event

Events are immutable records of things that already happened. Past tense. Specific. Self-contained.

{
  "event_type": "ArticleScored",
  "event_id": "evt_cf_98231",
  "timestamp": "2018-03-19T10:15:30Z",
  "aggregate_id": "article_44712",
  "data": {
    "source": "reuters",
    "relevance_score": 0.87,
    "matched_topics": ["AAPL", "earnings"],
    "scoring_model_version": "v3.2"
  }
}

A few things I learned the hard way about event design:

Include the model version. We score articles with ML models that change. Without the model version baked into the event, you can’t tell whether a score difference came from new data or a new model. We didn’t do this initially. Debugging was miserable.

Don’t store derived data as the event. The event is the fact. “ArticleScored” with a score of 0.87 is a fact. “ArticleIsHighlyRelevant” is a conclusion you draw from the fact. Store the fact. Let projections draw conclusions.

Carry enough context. A projection should never need to reach back into another stream to understand what an event means. If your ArticleScored event requires a lookup to know which source it came from, your event is too thin.

Aggregates: The Hardest Part

Aggregates define your consistency boundary. They validate commands, enforce invariants, and emit events. Their internal state is rebuilt by replaying the event stream.

Getting aggregate boundaries right is the single most consequential design decision in an event-sourced system. Too big and you get contention, slow replays, and serialization bottlenecks. Too small and you can’t enforce invariants that span related data.

We started with a giant “UserFeed” aggregate that tracked everything – subscribed topics, read articles, relevance preferences, notification settings. It grew to thousands of events per active user within weeks. Replaying it on every command was brutal.

We broke it apart. Subscriptions became their own aggregate. Notification preferences became their own aggregate. The UserFeed aggregate shrank to just the core feed interaction logic. Replay times dropped from seconds to milliseconds. Contention disappeared.

The rule I follow now: an aggregate should be the smallest unit that can enforce its invariants independently.

CQRS: Separating the Write and Read Paths

Event sourcing doesn’t require CQRS but you’ll almost certainly end up there. The write model and read model have fundamentally different jobs.

Command -> Aggregate -> Event Store -> Projection -> Read Model

The write side validates and enforces rules. The read side is shaped for queries. At the fintech startup our write side emits events like ArticleScored and TopicSubscribed. Our read projections build denormalized views for the API: a user’s personalized feed, trending topics, source reliability dashboards.

The catch is eventual consistency. Your read model will lag behind writes. For us that means a user might subscribe to a topic and not see it reflected in their feed for a few hundred milliseconds. We made that explicit in the product. No pretending the system is synchronous when it’s not.

Projections and Snapshots

Projections

Projections consume events and write to read models. The cardinal rule: they must be idempotent.

def handle(event):
    if already_processed(event.event_id):
        return
    update_read_model(event)
    mark_processed(event.event_id)

We’ve been bitten by non-idempotent projections exactly once. A network blip caused a batch of ArticleScored events to be delivered twice. Our read model double-counted relevance scores. Users saw garbage rankings. Took us three hours to figure out what happened and another two to rebuild the projection. Idempotency checks would have made it a non-event.

Snapshots

When an aggregate has a long event history, replaying from event zero on every command gets expensive. Snapshots are a cache – a serialized version of aggregate state at a known position. You replay from the snapshot forward instead of from the beginning.

Key point: snapshots aren’t the source of truth. They’re disposable. If a snapshot is corrupt or stale, delete it and rebuild from events. We snapshot our most active aggregates every 500 events and it keeps command processing fast.

Process Managers

Some workflows span multiple aggregates. A user signs up, we create their profile, set default subscriptions, and kick off an initial scoring run. No single aggregate owns all of that. A process manager listens to events from each aggregate, issues commands to others, and tracks progress. Think of it as a long-running coordinator that reacts to facts rather than orchestrating a transaction.

Schema Evolution: Plan Before You Ship

Events are forever. You can’t go back and change them. So you need a schema evolution strategy before you write your first event.

version: 2
payload:
  source: reuters
  relevance_score: 0.87
  matched_topics: ["AAPL", "earnings"]
  scoring_model_version: "v3.2"

What works for us:

• Additive changes with defaults. New field? Add it. Old events that lack the field get a sensible default when read.
• New event types for semantic shifts. If the meaning of an event changes fundamentally, introduce a new event type. Don’t twist the old one.
• Upcasters on read. When loading events, transform old versions into the current shape. The event store stays untouched. The application code only deals with the latest schema.

We’ve gone through four versions of our scoring events. The upcaster chain is a bit ugly but it works and we’ve never had to touch the event store itself.

The Pitfalls That Actually Hurt

Most event sourcing failures come from modeling mistakes, not tooling problems.

Modeling commands as events. “ScoreArticle” is a command. “ArticleScored” is an event. If you store commands in your event log you will confuse every projection that reads them. I’ve seen this mistake more times than I want to admit, including in our own early prototypes.

Giant aggregates. Already covered this. Keep them small. If replay takes more than a few milliseconds, something is wrong.

Synchronous projections on the write path. The moment you make a write wait for a projection to finish, you’ve coupled your write throughput to your read model’s performance. Don’t do this.

Skipping idempotency. Events will be delivered more than once. Network partitions, retries, rebalances. Your projections must handle duplicates gracefully or your read models will drift into nonsense.

Chatty technical events. Events should represent business facts. “DatabaseRowUpdated” isn’t a business fact. “ArticleScored” is. If your events read like a database changelog, you’ve modeled the wrong thing.

When Not to Bother

Event sourcing is a deliberate tradeoff, not a default architecture.

Skip it when CRUD is genuinely sufficient and audit history isn’t a real requirement. Skip it when the domain is simple and the modeling overhead isn’t justified. Skip it when your team isn’t ready to think in terms of eventual consistency.

At the fintech startup it was the right call because of the regulatory requirements, the temporal query needs, and the natural event-driven shape of financial data. For our marketing site? We use a database and a CMS. Not everything needs to be an event.

The Short Version

Model events as business facts in past tense. Keep aggregates small and focused. Build read models asynchronously and accept eventual consistency. Make projections idempotent. Plan schema evolution before you ship. And decide early whether the complexity is worth it for your specific domain – because once you commit to event sourcing, unwinding it is far harder than adopting it.

Multi-region buys you latency and resilience. It costs you sanity. Know which one you’re trading before you start.

At the fintech startup we serve real-time financial news and data to users spread across the UK, Germany, the Nordics, and increasingly beyond. Our backend runs in a single AWS region right now. It works. But I’ve spent the last few months seriously evaluating what a multi-region setup would look like for us, and I want to lay out what I’ve found – both the architecture patterns and the ugly operational reality nobody warns you about.

Physics doesn’t care about your SLA

A request from London to eu-west-1 is fast. A request from Singapore to eu-west-1 isn’t. We’re talking 200-300ms of raw network latency before your application code even runs. Stack a few API calls on top of that, add a database query, and suddenly the page takes two seconds to load. Users don’t file bug reports for slow pages. They just leave.

I measured this myself with a cheap VPS in each continent. London to Dublin: ~10ms. Tokyo to Dublin: ~240ms. That’s not a rounding error. That’s a completely different product experience.

Three patterns, three sets of problems

Active-passive is the one everyone starts with. Your primary region handles all traffic. A standby region sits there with replicated data, waiting. You get disaster recovery. You don’t get latency improvement for remote users. The tricky part is failover – if you haven’t rehearsed it, your “30-second failover” is actually a 45-minute scramble at 3am with someone SSHing into the wrong box.

Active-active is the dream. Multiple regions serving traffic simultaneously, users routed to the nearest one. Fast everywhere. But you’ve just signed up for distributed consensus problems. Two users updating the same record in two regions at the same time? Now you need conflict resolution. Last-write-wins is easy to implement and will silently eat data. Domain-specific merge logic is correct and will take you months to build. Pick your pain.

Follow-the-sun (or read-local, write-central) is what I keep coming back to for our use case. Reads hit the nearest region. Writes go to a single primary. For a read-heavy system like ours – users consuming financial content far more than creating it – this is the sweet spot. Remote users still eat write latency, but writes are a small fraction of our traffic. The tradeoff is acceptable.

Data is where it gets ugly

Every multi-region discussion eventually becomes a data discussion. The CAP theorem isn’t just academic. It’s the thing that makes your on-call engineer cry.

Async replication is the pragmatic choice. Your writes are fast because they don’t wait for the replica. But a user who just updated their watchlist and immediately reads it back might see stale data. For financial content, that’s bad. For user preferences, it’s annoying but survivable. You have to know your data well enough to make that call per table, sometimes per column.

Sync replication gives you consistency. It also means every write waits for a cross-region round trip. One slow region drags the whole system down. I’ve seen this kill write throughput in practice. Unless your write volume is very low and your tolerance for tail latency is very high, avoid it.

The approach I like best is regional data affinity. A UK user’s data lives in the EU region. Period. We only replicate the things that truly need to be everywhere – shared reference data, global config, that sort of thing. This sidesteps most consistency headaches because you’re not actually doing multi-master for user data. You’re doing single-master per user, distributed across regions. Much simpler failure modes.

Getting users to the right place

DNS geo-routing is the obvious first step. Route53 does it, CloudFlare does it, everyone does it. It works well enough. The catch is DNS caching – TTLs are suggestions, not commands. ISPs will cache your records for longer than you want, which means failover via DNS is slower than you’d hope. Minutes, not seconds.

Anycast is faster for failover but harder to operate. You need BGP-level control. Most startups, us included, don’t have the network engineering chops for this. It’s the right answer at scale. It’s premature complexity at our stage.

CDNs solve the easy part (static assets, edge caching) and punt on the hard part (dynamic requests still need to reach an origin). Worth doing regardless of your multi-region strategy, but don’t confuse a CDN with actual multi-region architecture. They’re complementary, not interchangeable.

The operational tax nobody budgets for

Here’s where most multi-region proposals die, and honestly they should.

Deployments get harder. You can’t just kubectl apply and walk away. You need to decide: do you deploy to all regions simultaneously? Serially? Use one region as a canary? We’d probably do serial deployment with the secondary region first, watch it for 15 minutes, then roll to primary. That means every deploy takes longer. Every rollback is more complex. Your CI/CD pipeline just doubled in scope.

Monitoring has to be per-region AND global. You need to know that latency is high in ap-southeast-1 specifically, not just that “p99 latency is elevated somewhere.” If your monitoring runs in the same region as your primary, congratulations – when that region dies, so does your ability to see that it died.

Failover isn’t a feature you ship once. It’s a muscle you exercise. We do game days at the fintech startup for our current single-region setup, and even those surface surprises. A multi-region failover you’ve never tested is a multi-region failover that doesn’t work. Full stop. The first time you practice it, something will break that you didn’t expect. Better to find that on a Tuesday afternoon than during an actual incident.

Should you actually do this?

Honest answer: most teams shouldn’t. Not yet.

If your users are in one geography and your uptime requirements are met by a single region with good practices (multi-AZ, proper backups, tested restores), multi-region is complexity you’re borrowing against future needs. That debt has interest.

For us at the fintech startup, the calculus is shifting. We’re getting real users in Asia. GDPR already constrains where we store certain data. And our customers are making trading decisions based on our content – downtime costs them money, not just patience. We’ll probably go follow-the-sun within the next year. But I’m going in with my eyes open about what it costs.

Build your system so migration is possible. Abstract your data layer. Don’t hardcode region assumptions. Use infrastructure-as-code so spinning up a new region isn’t a six-week project. Then wait until the numbers actually justify the move.

The worst multi-region architectures I’ve seen were built by teams who wanted the résumé bullet point. The best were built by teams who had no choice.

It falls apart the second you split into services.

I learned this the hard way at the fintech startup. We had decent Grafana dashboards. Reasonable alerts. Then we started breaking things into microservices and deploying multiple times a day. A request would fail somewhere in a chain of five services and our dashboards would just… look fine. Every individual service reported healthy metrics. The problem lived in the gaps between them.

That’s the core issue with monitoring. It answers questions you already thought to ask. Latency on this endpoint? Sure. Error rate on that queue? Got it. But the failure you actually hit in production is the one you never predicted. Your dashboards have no panel for it.

Observability is a different mindset

Observability means you instrument your system so you can ask new questions after something breaks. Not just “is it up” but “why did this specific user’s request take 8 seconds at 3am on Tuesday.”

Three signals make this work: metrics for trends and alerts, logs for detail, and traces for stitching a single request across every service it touches. Separately they are useful. Together they are a debugging superpower.

Structured logging is the foundation. Stop writing free-text log lines. Make every log entry a JSON object with a trace ID, service name, version, and whatever fields you actually need to filter on.

{
  "timestamp": "2017-03-20T10:23:45Z",
  "level": "error",
  "message": "Payment processing failed",
  "trace_id": "abc123",
  "service": "payment-service",
  "version": "2.1.3"
}

Then propagate context. Generate a trace ID at the edge and carry it through every downstream call. This is the single most important thing you can do. Without it you’re just grepping logs and praying.

X-Trace-Id: abc123
X-Span-Id: def456
X-Parent-Span-Id: ghi789

How incidents actually change

Alert fires. Pull the trace. Follow the slow span to the source. Read the logs for that trace. Confirm with metrics whether it’s one user or everyone. Done. Five minutes instead of forty-five minutes of bouncing between dashboards while your Slack channel fills with “any update?”

Same thing with performance. User says it’s slow. Pull their trace. Compare to a healthy one. See the difference – a cache miss, an extra database round-trip, a third-party call timing out. You fix the actual cause instead of guessing.

The tools don’t matter that much

Prometheus, InfluxDB, ELK, Jaeger, Zipkin – pick whatever fits your stack. Commercial platforms that bundle all three signals save time. But the tooling isn’t the hard part. The hard part is disciplined instrumentation. Consistent field names. Trace IDs everywhere. Every team following the same conventions.

What actually matters

Observability isn’t a product you buy. It’s a practice you build. You stop staring at dashboards waiting for red. You start asking questions about behavior you didn’t expect. That shift – from reactive to exploratory – is the entire point. And in a world where every team is shipping services independently, it’s the only way to stay sane.