Most catastrophic database incidents aren’t novel. They’re compounded failures that nobody practiced for. The node-failure test passes, so the team moves on. Then a network partition hits during a schema migration while the on-call engineer is handling an unrelated alert, and suddenly you’re in territory no runbook covers. Structured red-teaming exposes these compound paths before they become customer-visible outages. It costs a fraction of what a single bad incident costs.

Black Swans vs. Ignored Knowns

The term “black swan” gets overused in infrastructure. Most catastrophic database failures are not genuinely unpredictable. They are known failure modes that compound in ways nobody tested.

Consider the canonical distributed database incident: a network partition isolates a minority of nodes, those nodes continue accepting writes because the partition detection is slow, the partition heals, and now you have conflicting data that the conflict resolution logic wasn’t designed to handle at that volume. Every component in this chain is well-understood. The failure isn’t in any single component. It’s in the interaction between them under specific timing conditions.

The honest term for most “black swan” database incidents is “ignored known.” The team knew partitions could happen. They knew conflict resolution had edge cases. They knew detection wasn’t instant. They just never tested all three at once.

Red-teaming is how you turn ignored knowns into practiced scenarios.

Mission-Style Red-Teaming

Chaos engineering tools that randomly kill processes are useful, but they test a narrow failure class: single-component loss. Distributed database failures rarely look like one node dying cleanly. They look like degraded networks, clock drift, slow disks, operator errors during maintenance windows, and combinations of all of the above.

Mission-style red-teaming borrows from military and security practice. A dedicated team designs multi-step failure scenarios with specific objectives, executes them against production-equivalent infrastructure, and scores the defending team’s response. The key difference from chaos engineering is intentionality: the red team isn’t injecting random faults. They’re pursuing a specific failure hypothesis through a sequence of realistic actions.

A red-team exercise has three roles:

• Red team: designs and executes the failure scenario. Their goal is to cause data loss, unavailability, or corruption without triggering detection within a target time window.
• Blue team: the on-call and operations engineers responding as they would in a real incident. They don’t know the scenario in advance.
• White team: observers who control the exercise, ensure safety boundaries, and document everything for the post-exercise review.

The exercise runs for a fixed window, typically two to four hours. The red team executes their scenario. The blue team detects, diagnoses, and responds. Everyone debriefs afterward.

The Stress Scenarios That Matter

Not all failure modes are worth practicing. Focus on scenarios that are plausible, high-impact, and poorly covered by existing automation.

Network partitions with asymmetric visibility. One side of the partition can see the other; the other side cannot. This breaks assumptions in consensus protocols that expect symmetric failure detection. Many teams test clean partitions but never test asymmetric ones.

Clock skew under load. Distributed databases that use timestamps for ordering (which is most of them) behave unpredictably when clocks drift. NTP usually keeps drift small, but under heavy load, NTP corrections can be delayed. The result is transaction ordering violations that are invisible until a consistency check runs, which might be hours or days later.

Quorum erosion during maintenance. You take one node offline for a rolling upgrade. While it’s down, a second node develops a slow disk. You now have a degraded quorum that’s technically functional but one failure away from data unavailability. This is the most common compound failure pattern and the least practiced.

Operator mistakes during incidents. The most dangerous moment for a distributed database is when a human is manually intervening during an incident. Wrong-node restarts, accidental force-quorum operations, and recovery commands run against the wrong cluster are responsible for a disproportionate share of catastrophic data loss. Red-teaming should include scenarios where the operator is given misleading information and time pressure.

Backup restoration under partial failure. Most backup tests verify that a restore works on a clean target. Real restores happen during incidents, when the target environment is degraded, the team is stressed, and the backup might be from a point in time that’s already inconsistent. Test restoration under these conditions, not just in a clean room.

The OODA Loop for Incident Rehearsal

Effective red-team exercises run on a tight observe-orient-decide-act cadence. This isn’t just a framework. It’s a scoring mechanism.

Observe: How quickly does the blue team notice something is wrong? Detection time is the single most important metric. A failure that’s detected in two minutes has a fundamentally different blast radius than one detected in twenty. Measure time from fault injection to first alert, and time from first alert to accurate diagnosis.

Orient: Does the team correctly identify what’s happening? Misdiagnosis is common in compound failures because the symptoms don’t match any single runbook entry. The blue team might see elevated latency and assume it’s a hot key, when the actual cause is a partial partition affecting replication. Measure time from first alert to correct hypothesis.

Decide: Does the team choose an appropriate response? Under pressure, teams often default to the most familiar action (restart the node) rather than the most appropriate one (isolate the partition). Measure whether the chosen action matches the failure mode.

Act: Does the team execute the response correctly? Even when the right decision is made, execution errors under stress are common. Typos in commands, wrong node targets, and forgotten steps in manual procedures are all frequent. Measure execution accuracy and time to containment.

Each phase gets a score. Over multiple exercises, these scores reveal systemic gaps: maybe detection is fast but diagnosis is slow, or decisions are sound but execution is error-prone. That tells you exactly where to invest in automation, training, or tooling.

Scoring Readiness

After each exercise, score three dimensions:

Readiness (1-5): Could the team handle this scenario if it happened tomorrow in production? A 1 means the team didn’t detect the failure. A 5 means they detected, diagnosed, and contained it within SLA.

Blast radius (1-5): If the team had not responded, how bad would it have gotten? A 1 means minor degradation. A 5 means unrecoverable data loss or extended outage.

Time to containment (minutes): Wall-clock time from fault injection to the point where the failure is contained and no longer spreading. This is the metric that matters most to your customers and your SLA.

Plot these over time. Improving readiness scores and decreasing containment times are the clearest signals that your red-teaming program is working. If scores plateau, your scenarios aren’t challenging enough.

From Findings to Backlog

Red-team exercises are useless if findings sit in a postmortem document that nobody reads. Every exercise should produce a prioritized list of concrete improvements, each with an owner and a deadline.

The conversion process is simple:

1. List every gap discovered. Detection gaps, diagnostic confusion, tool limitations, missing runbooks, automation failures.
2. Score each gap by blast radius times likelihood. Likelihood is informed by the exercise, not guessed.
3. Assign an owner for each gap. Not a team. A person.
4. Set a deadline before the next exercise. The next exercise will test whether the gap was closed. This creates accountability.

Common improvements that come out of red-team exercises include automated partition detection that currently requires manual observation, runbook updates for compound failure scenarios, guardrails on dangerous operator commands during incidents, and backup restoration procedures tested under realistic conditions.

The backlog items from red-teaming tend to be high-value, low-glamour work. They rarely make it onto a roadmap through normal prioritization because they address risks that haven’t materialized yet. The exercise provides the evidence needed to justify the investment.

A Quarterly Operating Cadence

Red-teaming works best as a regular practice, not a one-off event. A quarterly cadence balances rigor with operational overhead.

Run quarterly. Dedicate the first few weeks to scenario design based on recent incidents and architectural changes, a half-day to executing the exercise against a production-equivalent environment, and the remainder of the quarter to remediating the gaps you found.

This cadence means every quarter your team practices a realistic failure scenario, identifies concrete gaps, and fixes the most critical ones before the next exercise. Over four quarters, you’ve tested and improved your response to a dozen failure modes. That’s a fundamentally different reliability posture than “we tested node failover once during setup and it worked.”

Key Takeaways

• Most catastrophic database failures are compound scenarios that nobody practiced, not genuinely unpredictable events.
• Chaos engineering tests component failure. Red-teaming tests system failure under realistic operational conditions.
• Score every exercise on detection time, diagnostic accuracy, decision quality, and execution correctness. Track trends.
• Convert findings into owned backlog items with deadlines tied to the next exercise.
• Run quarterly. Consistency matters more than intensity.

Red-teaming distributed databases is not theater and it’s not a luxury. It’s the cheapest way to find out whether your recovery assumptions actually hold before your customers find out for you.

Cache-aside is the sane default. Invalidation is where everyone gets burned. Plan for stampedes and penetration on day one, not after the incident. And please – document what staleness window your product can tolerate before you write a single line of caching code.

Phil Karlton’s line about cache invalidation being one of the two hard things in computer science gets quoted constantly. It gets quoted because it’s true. I’ve added caching to systems at a large consumer platform, at a real-time messaging company, and at Decloud. Every time, the initial implementation was straightforward. Every time, the edge cases in invalidation and consistency were where we spent the real engineering effort.

Caching isn’t hard to implement. It’s hard to operate correctly.

What Is Worth Caching

Not everything. The filter is simple:

• High read-to-write ratio. If you’re reading 100x more than writing, caching pays off immediately.
• Expensive to compute or fetch. Database aggregations, external API calls, anything with meaningful latency.
• Tolerant of staleness. A product listing that’s 30 seconds stale is fine. An account balance that’s 30 seconds stale isn’t.

If data changes constantly and correctness matters, don’t cache it. Add an index, optimize the query, or scale the database.

Core Patterns

Cache-Aside

The application checks the cache, falls back to the source on a miss, and populates the cache for next time. This is the default for good reason: it’s simple, it works with any data store, and the application controls the logic.

In Go, a basic cache-aside with singleflight to prevent stampedes:

import "golang.org/x/sync/singleflight"

type UserCache struct {
    redis  *redis.Client
    db     *sql.DB
    group  singleflight.Group
    ttl    time.Duration
}

func (c *UserCache) Get(ctx context.Context, id string) (*User, error) {
    // Check cache first
    data, err := c.redis.Get(ctx, "user:"+id).Bytes()
    if err == nil {
        var u User
        if err := json.Unmarshal(data, &u); err == nil {
            return &u, nil
        }
    }

    // Singleflight: only one goroutine fetches from DB for a given key
    val, err, _ := c.group.Do("user:"+id, func() (interface{}, error) {
        u, err := c.fetchFromDB(ctx, id)
        if err != nil {
            return nil, err
        }

        // Populate cache
        data, _ := json.Marshal(u)
        c.redis.Set(ctx, "user:"+id, data, c.ttl)
        return u, nil
    })
    if err != nil {
        return nil, err
    }

    u := val.(*User)
    return u, nil
}

The singleflight.Group is the key detail here. Without it, a hot key expiry sends N concurrent requests to the database. With it, one goroutine fetches while the others wait and share the result. I’ve seen this single change reduce database load by 10x during peak traffic.

Write-Through

Writes update the cache and the primary store together. Reads are always fast and consistent. The tradeoff is higher write latency because you’re writing to two places.

func (c *UserCache) Update(ctx context.Context, u *User) error {
    if err := c.updateDB(ctx, u); err != nil {
        return err
    }

    data, err := json.Marshal(u)
    if err != nil {
        return err
    }

    return c.redis.Set(ctx, "user:"+u.ID, data, c.ttl).Err()
}

Good fit when writes are moderate and you can’t tolerate stale reads. At a large consumer platform, we used write-through for restaurant availability – it changed infrequently but needed to be correct immediately.

Write-Behind

Writes go to the cache. A background process persists them asynchronously. This absorbs write spikes but introduces data loss risk if the cache crashes before flushing.

I’m cautious about write-behind. It’s appropriate for data you can rebuild – session state, analytics counters, things where losing a few seconds of writes is acceptable. For anything transactional, avoid it.

Invalidation

Three approaches, and most production systems combine two of them.

TTL-based expiration. The safety net. Every cached item has a time to live. When it expires, the next read triggers a refresh. Simple, reliable, and coarse. A 60-second TTL means your data can be up to 60 seconds stale. That’s the contract.

Event-based invalidation. When the source data changes, publish an event that invalidates or updates the cache. More precise than TTL, but it depends on reliable event delivery. If the invalidation event gets lost, the cache serves stale data until the TTL expires.

func (c *UserCache) HandleUserUpdated(ctx context.Context, event UserUpdatedEvent) error {
    return c.redis.Del(ctx, "user:"+event.UserID).Err()
}

Delete on invalidation rather than update. Let the next read repopulate. This avoids race conditions between the event handler and concurrent reads.

Versioned keys. Instead of user:123, use user:123:v7. When the data changes, increment the version. Old keys age out naturally. No explicit deletes, no race conditions. The cost is more keys in the cache and a version lookup on every read.

My default: TTL as the safety net plus event-based invalidation for known write paths. This covers 90% of use cases.

Failure Modes

Stampedes

Hot key expires. Hundreds of concurrent requests miss the cache and hit the database. The database buckles.

Mitigations:

• Singleflight (shown above) – one fetch per key at a time
• Early refresh – refresh the cache before the TTL expires, while the current value is still valid
• Stale-while-revalidate – serve the slightly stale value while refreshing in the background

Penetration

Requests for keys that will never exist in the source. Every request misses the cache and hits the database. Attackers love this.

Fix: cache negative results with a short TTL. If user:999 doesn’t exist, cache nil for 30 seconds. A Bloom filter in front of the cache can also help for large keyspaces.

Inconsistency

The cache says one thing. The database says another. This happens for all sorts of reasons: failed invalidation events, race conditions between writes and cache updates, network partitions between the application and Redis.

Accept eventual consistency where the product allows it. For the few paths that must be strongly consistent, bypass the cache and read from the source.

Multi-Level Caching

In-process memory for the hottest data. Redis for shared distributed caching. CDN for public static content. Each layer reduces latency but adds a place where stale data can hide.

Keep the hierarchy as shallow as your latency requirements allow. Two levels is common. Three is manageable. Four is a debugging nightmare.

Observability

You need four metrics at minimum:

• Hit rate – below 80% and you should question whether the cache is helping
• Latency – cache should be sub-millisecond; if it isn’t, check network or serialization
• Eviction rate – high evictions mean the cache is too small or the TTLs are too long
• Error rate – a cache error should fall back to the source, not fail the request

Document the expected staleness window for each cached entity. “User profiles: up to 60 seconds stale” is a product decision, not a technical one. Make it explicit.

The Honest Summary

Caching is the fastest way to scale reads. It’s also the fastest way to introduce subtle bugs that only show up under load. Pick cache-aside as your default. Use singleflight. Set TTLs that match your product’s tolerance for staleness. Plan for stampedes and penetration before they happen.

And remember: the best cache is the one you can explain to the on-call engineer at 3 AM.

Install pg_stat_statements. Find your top 10 slowest queries. Fix the indexes. Tune autovacuum for big tables. Use a connection pooler. That covers 80% of Postgres performance work. Everything else is refinement.

I’ve tuned Postgres at startups with a single database and at enterprises with dozens of replicas. The pattern is always the same: someone reports “the database is slow,” and the first instinct is to increase instance size or tweak configuration parameters. Almost every time, the actual problem is a missing index or a query that shouldn’t exist in its current form.

The discipline is simple. Measure, change one thing, measure again. If you skip the measurement step, you’re guessing. Guessing is expensive.

Start with pg_stat_statements

This is the single most valuable tool for Postgres performance work. It tracks every query the database executes and accumulates statistics: call count, total time, mean time, rows returned.

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Top 10 queries by total execution time
SELECT
    query,
    calls,
    round(total_exec_time::numeric, 2) AS total_ms,
    round(mean_exec_time::numeric, 2) AS avg_ms,
    rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

Total time matters more than average time. A query that runs in 5ms but gets called 500,000 times per day consumes more resources than a query that takes 2 seconds but runs once an hour. Fix the high-total-time queries first.

Also check for long-running queries and idle transactions, which block vacuum and hold locks:

SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes'
AND state != 'idle'
ORDER BY duration DESC;

EXPLAIN is your microscope

Once you know which queries are slow, EXPLAIN (ANALYZE, BUFFERS) shows you why. Don’t read the entire plan tree. Focus on four signals:

Sequential scan on a large table: Almost always means a missing or ineffective index. A sequential scan on a 100-row lookup table is fine. A sequential scan on a 50-million-row orders table isn’t.

Actual rows far from estimated rows: Postgres is making bad decisions because its statistics are stale or the data is heavily skewed. Run ANALYZE on the table and check again.

Nested loop with high row count on the inner side: Missing index on the join column. The planner is scanning the inner table for every row of the outer table.

Sort with “Sort Method: external merge Disk”: The sort spilled to disk because work_mem was too small for the operation. Either add an index that eliminates the sort, or increase work_mem (carefully – it’s per-operation, not per-connection).

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT o.id, o.total, c.name
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.status = 'pending'
AND o.created_at > now() - interval '7 days'
ORDER BY o.created_at DESC;

Indexing: the right index, not more indexes

Most performance fixes come down to having the right index. Not more indexes. Every index you add slows down writes and makes vacuum work harder.

Composite indexes: Column order matters. Put the equality columns first, range columns last. For the query above, (status, created_at) lets Postgres seek to status = 'pending' and then scan the date range efficiently.

CREATE INDEX idx_orders_status_created ON orders (status, created_at DESC);

Partial indexes: If you only ever query pending orders, index only those. Smaller index, faster lookups, less write overhead.

CREATE INDEX idx_orders_pending ON orders (created_at DESC)
WHERE status = 'pending';

Covering indexes (Postgres 11+): Include columns the query needs so Postgres can answer from the index alone without touching the table.

CREATE INDEX idx_orders_covering ON orders (customer_id, status)
INCLUDE (total, created_at);

Unused indexes: They cost writes and vacuum time for zero benefit. Find them:

SELECT
    schemaname || '.' || relname AS table,
    indexrelname AS index,
    idx_scan AS scans,
    pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
AND indexrelname NOT LIKE '%_pkey'
ORDER BY pg_relation_size(indexrelid) DESC;

If an index has zero scans and has been running for weeks, it’s a candidate for removal.

Configuration: what actually matters

I see teams spend hours tweaking parameters that move the needle by 1%. Focus on the ones that matter.

shared_buffers: Postgres’s own cache. Start at 25% of RAM. Going higher rarely helps because the OS page cache handles the rest.

effective_cache_size: Not an allocation – a planner hint. Tell it how much cache is available total (Postgres cache + OS cache). Typically 50-75% of RAM. This changes whether the planner favors index scans over sequential scans.

work_mem: Memory per sort/hash operation. The trap: this is per-operation, not per-connection, and a single query can have multiple sorts. If you set it to 256MB and have 100 concurrent connections each running a multi-sort query, you can run out of memory fast. Start at 4-16MB and increase carefully for specific workloads.

maintenance_work_mem: Used for vacuum and index creation. Higher is better here. 512MB to 1GB is reasonable for most production systems.

max_connections: Keep it low. 100-200 is plenty for most workloads. Use PgBouncer for connection pooling. Each connection costs ~10MB of memory and context-switching overhead. I’ve seen teams set this to 1000 and then wonder why their 64GB instance is swapping.

Connection pooling: non-negotiable

PgBouncer in transaction pooling mode is the standard answer. The application thinks it has a dedicated connection. PgBouncer multiplexes to a smaller pool of real database connections.

This matters because most web application connections are idle most of the time. An application with 200 worker processes might only have 20 queries running at any given moment. PgBouncer lets you serve 200 workers with 30-50 actual database connections.

Session pooling is necessary if your application uses prepared statements, temp tables, or session-level settings (like SET search_path). Transaction pooling doesn’t preserve these across requests.

Vacuum: the silent killer

Postgres uses MVCC, which means every update and delete creates dead tuples. Vacuum cleans them up. If vacuum falls behind, tables bloat, indexes get slower, and eventually you hit transaction ID wraparound – which forces a stop-the-world vacuum.

Autovacuum works fine for most tables. But large, busy tables need tuned thresholds:

-- Check dead tuple counts
SELECT relname, n_live_tup, n_dead_tup,
       round(n_dead_tup::numeric / greatest(n_live_tup, 1) * 100, 2) AS dead_pct,
       last_autovacuum
FROM pg_stat_user_tables
WHERE n_dead_tup > 10000
ORDER BY n_dead_tup DESC;

For a table with 100 million rows, the default autovacuum threshold of autovacuum_vacuum_scale_factor = 0.2 means vacuum waits until there are 20 million dead tuples. That’s way too late. Set per-table overrides:

ALTER TABLE large_events SET (
    autovacuum_vacuum_scale_factor = 0.01,
    autovacuum_analyze_scale_factor = 0.005
);

The other vacuum killer is long-running transactions. A transaction that has been open for hours prevents vacuum from cleaning up any tuples visible to that transaction. Find and fix these aggressively.

The tuning loop

Performance tuning isn’t a project. It’s a practice.

1. Check pg_stat_statements weekly. Know your top 10.
2. EXPLAIN any query that shows up as a problem.
3. Add or adjust indexes. Remove unused ones.
4. Monitor vacuum health and dead tuple counts.
5. Repeat.

Small, measured changes. Every time. The teams that keep Postgres fast aren’t the ones with the cleverest configuration. They’re the ones who look at the numbers regularly and fix what the numbers show them.

The total outage was maybe eight minutes. Felt like an hour. And the fix was something we should have known: add the column nullable, backfill in batches, then add the constraint. Three deploys instead of one. No lock, no drama.

That experience permanently changed how I think about database migrations. The schema change itself is rarely the problem. The problem is live traffic hitting the change while it runs.

The expand-and-contract pattern

Every breaking schema change should follow expand-and-contract. It isn’t glamorous. It’s slow. It works.

Expand: Add the new structure alongside the old one. Deploy code that writes to both.

Migrate: Backfill data from old to new. Switch reads to the new path.

Contract: Once the old path is fully retired and you have confirmed stability, drop the old columns or tables.

A simple column rename looks like this:

-- Step 1: Expand
ALTER TABLE users ADD COLUMN full_name text;

-- Step 2: Backfill (in batches, see below)
UPDATE users SET full_name = name
WHERE id BETWEEN 1 AND 10000 AND full_name IS NULL;

-- Step 3: Contract (after all code reads full_name)
ALTER TABLE users DROP COLUMN name;

Three deploys. Three PRs. Each one is small, reversible, and independently verifiable. This is slower than a single ALTER TABLE ... RENAME COLUMN, but it doesn’t lock the table and it doesn’t require a maintenance window.

Application code has to play along

The migration only works if the application tolerates both schemas during the transition. This means:

• Write to both columns during the expand phase. Old code writes to name. New code writes to both name and full_name.
• Read from new, fallback to old. COALESCE(full_name, name) in your queries, or handle it in application code.
• Feature flags help. Switch reads to the new column behind a flag. Flip it when the backfill is done.
• Keep old queries working until you ship the contract phase. If something goes wrong after the backfill, you want the ability to revert without data loss.

The key discipline: never deploy a schema change and a code change that depends on it in the same release. Separate them. Deploy the schema first, then the code. If the schema deploy breaks something, you can roll back without touching the application.

Backfills that don’t melt production

A naive UPDATE users SET full_name = name on a table with tens of millions of rows will generate massive WAL, spike replication lag , and potentially lock out autovacuum. I’ve watched backfills bring down read replicas.

Batch it:

-- Process 5000 rows at a time
UPDATE users SET full_name = name
WHERE id BETWEEN :start AND :end
AND full_name IS NULL;

Run this in a loop with a small delay between batches. Keep transactions short. Commit after each batch. Record progress so the job can resume if it crashes.

Practical tips from doing this too many times:

• Rate limit. 5,000 to 50,000 rows per batch depending on table width and load.
• Short transactions. Long transactions hold locks and block vacuum.
• Idempotent. The WHERE full_name IS NULL clause means you can safely restart.
• Off-peak when possible. Not always an option, but backfilling during low traffic reduces risk.
• Monitor replication lag. If replicas start falling behind, slow down or pause.

The operations PostgreSQL will punish you for

Some DDL operations are deceptively dangerous. They look like simple one-liners but can lock tables, rewrite data, or kill your replication.

Creating an index without CONCURRENTLY: A regular CREATE INDEX takes a SHARE lock on the table, blocking all writes until it finishes. On a large table, that can be minutes.

SET lock_timeout = '5s';
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);

Always use CONCURRENTLY. Always set a lock timeout so it fails fast instead of waiting indefinitely.

Adding NOT NULL with a default (pre-PG 11): Before PostgreSQL 11, ALTER TABLE ADD COLUMN ... DEFAULT ... NOT NULL rewrote the entire table. PG 11+ handles this without a rewrite, but if you’re on an older version, add the column nullable first, backfill, then add the constraint.

Large data copies and foreign key additions: Adding a foreign key validates every existing row. On a big table, that’s a scan with a lock. Use NOT VALID to add the constraint without validation, then VALIDATE CONSTRAINT separately:

ALTER TABLE orders ADD CONSTRAINT fk_customer
  FOREIGN KEY (customer_id) REFERENCES customers(id)
  NOT VALID;

-- Later, without blocking writes:
ALTER TABLE orders VALIDATE CONSTRAINT fk_customer;

The operational workflow

Before you touch production:

• Document every step and its expected impact. Not a novel. A numbered list.
• Rehearse on production-size data. Staging with 1,000 rows won’t catch lock issues on a 50 million row table.
• Verify your backups work. Not “we’ve backups.” Actually restore and check.
• Decide your abort criteria in advance. “If lock waits exceed 5 seconds or replication lag exceeds 30 seconds, we stop.”

During the migration:

• Watch lock waits, query latency, and replication lag in real time.
• Validate each step before moving to the next one.
• Have the rollback commands ready in a terminal. Not in a wiki. In a terminal.

After:

• Verify data integrity. Row counts, null rates, spot checks.
• Keep the old columns around for a few days. You want a rollback window.
• Clean up temporary code paths and old columns only after you’re confident.

Rollback isn’t always possible

Additive changes are easy to roll back – just drop the new column. Destructive changes aren’t. If you dropped a column and need it back, you need a restore from backup.

This is why the contract phase should be the last step, done only after the system has been stable for days or weeks. Keep old columns around longer than feels necessary. Storage is cheap. Outages aren’t.

The uncomfortable truth about zero-downtime migrations : they aren’t about clever SQL or sophisticated tooling. They’re about discipline. Small steps. Verified assumptions. Rollback plans written before you need them. The teams that do this well aren’t smarter. They’re more disciplined. And they get to keep their weekends.

Define your RPO/RTO before you pick a replication strategy. Test your restores monthly – most backup failures are actually restore failures. Use safe migration patterns (nullable columns, concurrent indexes, lock timeouts). Monitor replication lag, slow queries, and disk growth. Practice failovers before you need them. This post includes the Postgres configs and SQL patterns I use.

I’ve been responsible for databases at the fintech startup (financial data platform, Postgres), at Decloud, and at several enterprises. The lesson that took me the longest to internalize: database reliability isn’t a technology problem. It’s a habits problem.

You can run the best hardware, the fanciest replication setup, the most expensive managed service. If you don’t practice restores, don’t test migrations against production data volumes, and don’t have runbooks for failover, you’re going to have a bad day. The technology just determines how bad.

Start with numbers, not architecture

Before touching any config, agree on three numbers with your team and your business stakeholders:

RPO (Recovery Point Objective): How much data can you lose? For the fintech startup’s financial data, the answer was “essentially zero” – we couldn’t afford to lose transactions. For an analytics pipeline, losing 5 minutes of data might be fine.

RTO (Recovery Time Objective): How long can the database be down? A consumer app might tolerate 30 seconds. A payment system might need sub-second failover.

Latency SLOs : What’s the p99 query latency you’re committing to? This drives capacity planning, connection pooling, and index strategy.

These numbers dictate everything downstream. Synchronous replication for near-zero RPO. Automated failover for aggressive RTO. Proper connection pooling and query optimization for latency SLOs. Don’t skip this conversation.

Replication: pick your tradeoff

At the fintech startup we ran synchronous replication for our primary financial data. The write latency penalty was real – every commit waited for at least one standby to acknowledge. But for a financial data platform, losing committed transactions wasn’t an option.

For our analytics and reporting databases, we used asynchronous replication. Faster writes, slight risk of data loss on primary failure. Acceptable for data that could be re-derived.

In Postgres, the config difference is small but the operational impact is significant:

# Synchronous replication - zero data loss, higher write latency
synchronous_commit = on
synchronous_standby_names = 'FIRST 1 (standby1, standby2)'

# Asynchronous replication - lower latency, possible data loss on failover
synchronous_commit = off

The mistake I keep seeing: running async replication without understanding the RPO implication. If your primary dies and the standby is 5 seconds behind, you just lost 5 seconds of data. If that includes a payment confirmation, you have a problem that no incident postmortem can fix.

Failover: automate with guardrails

Automated failover sounds great until it promotes a standby that’s behind the primary and you get split-brain. Or it triggers during a network partition and now you have two primaries accepting writes.

At a large telecom client, I helped set up Patroni for Postgres HA. The key decisions:

• Health checks from multiple locations (not just one monitoring server)
• Minimum replication lag threshold before promotion is allowed
• Fencing of the old primary before the new one accepts writes
• Manual fallback documented and practiced

# Patroni failover constraints
failover:
  maximum_lag_on_failover: 1048576  # ~1MB of WAL
  retry_timeout: 10
  ttl: 30

The maximum_lag_on_failover setting is critical. It prevents promotion of a standby that’s too far behind. I’ve seen teams leave this at the default and then wonder why they lost data during failover.

Test your failover. Quarterly at minimum. The first time you practice it, something will go wrong. Better during a planned exercise than during an actual outage at 3am.

Backups: the restore is the backup

I can’t emphasize this enough. Your backup strategy is only as good as your last tested restore.

At the fintech startup, we ran:

• Daily base backups via pg_basebackup
• Continuous WAL archiving to S3 for point-in-time recovery
• Weekly automated restore tests into a staging environment
• Monthly manual restore drills where an engineer actually performed the full recovery

The weekly automated test caught a corrupted backup exactly once. Once was enough. Without that test, we would have discovered the corruption during an actual emergency.

A solid Postgres backup setup:

# postgresql.conf - enable WAL archiving
archive_mode = on
archive_command = 'aws s3 cp %p s3://backup-bucket/wal/%f'
wal_level = replica

Point-in-time recovery is the real value here. Full backups let you restore to the last backup. WAL archiving lets you restore to any point in time. When someone runs a bad DELETE at 2:47pm, you can recover to 2:46pm. That capability has saved me more than once.

Safe migrations: the number one reliability risk

Schema changes cause more database incidents than hardware failures. I’ve seen it at every company. Someone runs an ALTER TABLE that locks a table for 20 minutes during peak traffic. The connection pool fills up. Upstream services start timing out. Cascade failure.

The patterns that keep you safe:

Add columns as nullable. Never add a NOT NULL column without a default to an existing table in a single step. Add it nullable, backfill, then add the constraint.

-- Step 1: Add nullable column
ALTER TABLE orders ADD COLUMN status_v2 text;

-- Step 2: Backfill in batches
UPDATE orders SET status_v2 = status
WHERE id BETWEEN 1 AND 10000 AND status_v2 IS NULL;
-- repeat for remaining batches

-- Step 3: Add constraint after data is clean
ALTER TABLE orders ALTER COLUMN status_v2 SET NOT NULL;

Build indexes concurrently. A regular CREATE INDEX locks the table for writes. On a large table, that’s an outage.

-- This blocks writes:
CREATE INDEX idx_orders_status ON orders(status);

-- This doesn't:
CREATE INDEX CONCURRENTLY idx_orders_status ON orders(status);

Set lock and statement timeouts. Prevent runaway migrations from holding locks indefinitely.

-- Before running migration
SET lock_timeout = '5s';
SET statement_timeout = '30s';

If a migration can’t acquire its lock within 5 seconds, it fails instead of queuing behind long-running queries and creating a pileup.

Test against production volumes. A migration that runs in 2 seconds on your dev database with 1,000 rows might take 45 minutes on production with 50 million rows. I learned this lesson at the fintech startup when a “quick” index build blocked writes for 12 minutes because we hadn’t tested against actual data volume.

Monitoring: watch the right things

Most database monitoring setups track too many things and miss the ones that matter. Here’s what I actually alert on:

Alert immediately:

• Replication lag above threshold (for us, 10 seconds)
• Disk usage above 80%
• Connection pool saturation above 90%
• Failed backup or missed WAL archive

Alert on sustained trends:

• Slow query count increasing over 15 minutes
• Lock wait time increasing over 5 minutes
• Transaction wraparound approaching (Postgres-specific, but it will bite you)

Track in dashboards, don’t alert:

• Query latency distribution (p50, p95, p99)
• Table and index sizes over time
• Autovacuum activity
• Cache hit ratios

A Postgres-specific query I run weekly to find trouble:

-- Tables approaching transaction wraparound
SELECT relname, age(relfrozenxid) as xid_age,
       pg_size_pretty(pg_total_relation_size(oid)) as total_size
FROM pg_class
WHERE relkind = 'r' AND age(relfrozenxid) > 200000000
ORDER BY age(relfrozenxid) DESC;

Transaction wraparound is the kind of problem that gives you no warning and then takes the database down hard. Monitor it.

Maintenance isn’t optional

Postgres needs vacuuming. It needs statistics updates. It needs index maintenance. These aren’t nice-to-haves.

Autovacuum handles most of the work, but the defaults are conservative. For busy tables:

# Per-table override for high-write tables
ALTER TABLE events SET (
    autovacuum_vacuum_scale_factor = 0.01,
    autovacuum_analyze_scale_factor = 0.005
);

The default autovacuum_vacuum_scale_factor of 0.2 means vacuum runs after 20% of the table has changed. On a 100-million-row table, that’s 20 million dead tuples before vacuum kicks in. Dropping it to 0.01 keeps things cleaner.

Index bloat is the other silent killer. Indexes grow as data is updated and deleted. Periodically check pg_stat_user_indexes for indexes with low usage and high size. Rebuild bloated indexes during maintenance windows using REINDEX CONCURRENTLY (Postgres 12+).

The operational habits that matter

Technology is the easy part. Habits are what prevent outages:

• Monthly restore drills. Someone on the team restores a backup from scratch. Time it. Document the steps. Every time.
• Quarterly failover tests. Actually promote a standby. Verify data integrity. Fail back. Fix whatever surprised you.
• Runbooks for every alert. If an alert fires and the on-call engineer has to figure out what to do from scratch, the runbook is missing.
• Post-incident follow-through. Writing the postmortem is easy. Actually fixing the root cause is what most teams skip.

Database reliability isn’t something you build once. It’s something you practice until the team can handle a 3am failover without panicking. The configs and queries above are tools. The discipline to use them consistently is the actual hard part.

I know. DynamoDB is cool. PlanetScale just launched and the developer experience looks great. Fauna promises global consistency over HTTP. Aurora Serverless scales to zero. It’s a good time to be a database vendor.

But I talk to teams every week, and most of them have the same setup: a web app with moderate traffic, a handful of services, and access patterns that are still changing. For that – and that describes the vast majority of projects – Postgres with PgBouncer in front of it is the right choice. It has been for years.

The serverless database pitch

The promise is simple. No connection management headaches. Scales up and down automatically. Pay for what you use. No patching, no failover to think about.

In practice, every one of these has caveats.

“Scales automatically” means Aurora Serverless v1 scales in steps with noticeable resume latency when it wakes from auto-pause. I watched a demo environment take 25 seconds to respond to the first query after an idle period. That’s not a database. That’s a nap.

“No connection management” means DynamoDB, which works great – until you realize you need to design your data model entirely around access patterns you haven’t fully figured out yet. Good luck refactoring that later.

“Pay for what you use” means unpredictable costs. I’ve seen DynamoDB bills spike 5x in a month because of a new feature that hit a secondary index harder than expected. With Postgres on a fixed instance, your bill is your bill.

When they actually make sense

DynamoDB is genuinely excellent if your access patterns are known and stable. Key-value lookups at scale, session storage, event logs with predictable queries. It’s fast, it’s cheap per-request, and the operational story is unbeatable. But you need to know your query patterns before you start. If your product is still evolving, DynamoDB’s rigidity becomes a liability.

Fauna is interesting if you need multi-region writes with strong consistency and you’re willing to learn a new query language. That’s a narrow use case. Most teams don’t need global writes. They need a database that works.

PlanetScale has a genuinely good developer workflow. Schema branching and non-locking migrations are real improvements over raw MySQL. But it’s in beta, it’s built on Vitess (so not all MySQL features work), and you still have connection limits.

The Postgres argument

Postgres handles relational queries, JSON documents, full-text search, and geospatial data. It has battle-tested replication. The ecosystem is enormous. The tooling is mature. Every cloud provider offers a managed version.

For serverless compute specifically, the connection problem is real. Lambda functions spinning up hundreds of connections will kill a Postgres instance. But PgBouncer or RDS Proxy solve this. It’s one more component, but it’s a well-understood one.

The thing about Postgres is that it’s boring. And boring is underrated. You can find answers to Postgres questions on Stack Overflow from 2009 that are still correct. Try that with Fauna’s FQL.

My advice

If you’re building a new project and you’re not sure about your access patterns: Postgres. If you’re running serverless functions and worried about connections: Postgres with a connection pooler. If you have a specific, well-understood, high-scale workload with stable access patterns: okay, look at DynamoDB.

Everything else is probably premature optimization disguised as architecture.

Every replication pattern is a bet on which failure you can tolerate. Pick wrong and you either lose data or lose availability. There’s no option C.

At the fintech startup we ingested financial news and market data from dozens of sources into PostgreSQL. Millions of rows per day. The kind of dataset where “eventually consistent” means a trader sees yesterday’s price and you get a very angry phone call.

That experience shaped how I think about replication. Not as a feature you toggle on, but as an architectural decision that determines how your system fails. Because it will fail.

Why you replicate

The reasons are fewer than people think:

• Availability. Your primary dies, a replica takes over, users keep working.
• Read scaling. Heavy reporting queries stop competing with writes.
• Latency. Put a copy closer to users so reads don’t cross the Atlantic.
• Isolation. Backups, analytics, migrations – all without touching production writes.

That’s basically it. If your reason isn’t one of these, you probably don’t need replication yet. You need a better backup strategy.

The three modes

Asynchronous

The primary commits, returns success to the client, then ships WAL records to replicas whenever it gets around to it.

-- postgresql.conf on the primary
wal_level = replica
max_wal_senders = 5
-- No synchronous_standby_names. That's the point.

The replica connects with a primary_conninfo and streams changes:

# recovery.conf (or standby.signal in PG12+)
primary_conninfo = 'host=primary-db port=5432 user=replicator'

This is the fastest mode for writes. The primary doesn’t wait for anyone. But there’s a window – could be milliseconds, could be seconds under load – where a replica is behind. If the primary dies in that window, those transactions are gone.

At the fintech startup, async replication was fine for our read replicas serving the news feed. A 200ms lag on a news article? Nobody notices. A 200ms lag on a stock price used for trading signals? Different conversation entirely.

Synchronous

The primary waits for at least one replica to confirm it wrote the WAL to disk before telling the client “committed.”

-- postgresql.conf
synchronous_standby_names = 'replica1'
synchronous_commit = on

Zero data loss on failover. Sounds great. The cost: every single write now includes a network round trip to the replica. In the same datacenter, maybe 1-2ms added latency. Across regions? 50-150ms. Per write.

Worse: if that replica goes down, your primary blocks all writes until the replica comes back or you reconfigure. I’ve seen this take down production systems that were “highly available.” The replication designed for availability became the single point of failure.

Quorum (semi-synchronous)

The pragmatic middle ground. Wait for any N out of M replicas to acknowledge, not all of them.

-- Wait for any 1 of these 3 replicas
synchronous_standby_names = 'ANY 1 (replica1, replica2, replica3)'

One replica can die and writes keep flowing. You still get the durability guarantee because at least one standby has the data. This is what I’d recommend for most production PostgreSQL setups that need strong durability.

The math is simple: if you have 3 standbys and require ANY 1, you can lose 2 replicas before writes stall. With ANY 2, you can lose 1. Pick based on how many failures you want to survive simultaneously.

Topologies

Primary-replica (the default)

One writer, N readers. Dead simple. Works for the vast majority of applications.

  Writes           Reads
    |                |
    v                v
[Primary] ----> [Replica 1]
    |
    +---------> [Replica 2]

If you’re building a new system and someone suggests multi-primary on day one, push back hard. Start here.

Cascading replicas

A replica can feed other replicas downstream instead of every replica pulling from the primary. This matters when you have many replicas or they span continents.

-- On the cascading replica (replica2 feeds from replica1, not primary)
primary_conninfo = 'host=replica1 port=5432 user=replicator'

The tradeoff: more lag for downstream replicas. Each hop adds latency. But the primary’s max_wal_senders slots aren’t exhausted by a fleet of read replicas. We used this at the fintech startup to feed analytics replicas off a primary standby – kept the main failover target clean while analytics could thrash their copy however they wanted.

Multi-primary

Multiple nodes accept writes and sync with each other. Sounds appealing. Is usually painful.

The fundamental problem: two nodes write conflicting data to the same row at the same time. Now what?

PostgreSQL’s logical replication can do this with BDR or similar extensions, but you’re signing up for conflict resolution logic, operational complexity, and debugging sessions that make you question your career choices. MySQL Group Replication and Galera have the same fundamental tradeoffs.

Use multi-primary when you have offices in multiple continents that all need local write latency and the business has accepted the cost. Not because it sounds cool on a whiteboard.

Read-your-writes consistency

This is where async replication bites you in the application layer.

User writes a comment. POST succeeds (hits the primary). Page refreshes. GET goes to a replica. Comment isn’t there. User writes the comment again. Now you have duplicates.

Three approaches, from simple to robust:

1. Sticky routing after writes

func handleRequest(w http.ResponseWriter, r *http.Request) {
    cookie, _ := r.Cookie("last_write")
    if cookie != nil && time.Since(parseCookieTime(cookie)) < 5*time.Second {
        // Recent write -- route to primary
        db = primaryDB
    } else {
        db = replicaDB
    }
    // ...
}

Crude but effective. After a write, force reads to the primary for a few seconds. The window just needs to exceed your typical replication lag.

2. Session affinity to a specific replica

Pin a user session to one replica. They won’t see inconsistency within their own session because they always read from the same copy. Doesn’t help with cross-user consistency but handles the most visible problem.

3. LSN tracking

The proper solution. After a write, capture the WAL position (LSN) from the primary. Before reading from a replica, check if the replica has replayed past that LSN.

-- After write, on the primary:
SELECT pg_current_wal_lsn();
-- Returns something like: 0/16B9188

-- Before read, on the replica:
SELECT pg_last_wal_replay_lsn();
-- If this >= the saved LSN, the replica is caught up

More work to implement. More correct.

Conflict resolution in multi-primary

If you went down the multi-primary path despite my warning, you need a conflict strategy.

Last write wins (LWW). Timestamp comparison. Whichever write has the later timestamp survives. Simple, lossy. A perfectly valid update gets silently dropped because a clock was 1ms behind. If your data can tolerate silent overwrites, fine. Financial data can’t.

Application-level merge. You define the merge logic per table or per field. A counter gets added. A set gets unioned. A text field takes the longer version. This works but every schema change means updating merge rules.

CRDTs. Conflict-free replicated data types. Mathematically guaranteed to converge. Limited to specific data structures: counters, sets, registers. You can’t CRDT your way through an arbitrary relational schema. Good for specific use cases, not a general solution.

Failover

Promoting a replica sounds easy. pg_promote() or SELECT pg_promote() in PG12+. Done. Except no.

The hard parts:

Fencing the old primary. If the old primary isn’t actually dead – just slow, or network-partitioned – you now have two nodes accepting writes. Split brain. The nightmare scenario. You need STONITH (shoot the other node in the head) or at minimum, revoke the old primary’s ability to accept connections.

Client reconnection. Your application’s connection string points to the old primary. Now it needs to point to the new one. Options: DNS update (slow propagation), virtual IP (fast but requires infra), connection proxy like PgBouncer or HAProxy that handles routing, or application-level logic.

# HAProxy config for automatic failover routing
listen postgres
    bind *:5432
    option httpchk GET /primary
    server pg1 10.0.1.1:5432 check port 8008
    server pg2 10.0.1.2:5432 check port 8008

Rebuilding the old primary. After failover, the old primary has diverged. It accepted some writes the new primary doesn’t have (or vice versa). You need to either pg_rewind it or rebuild from scratch with pg_basebackup. Test this process before you need it.

I’ve run failover drills at every company I’ve worked at. The drill always surfaces something the documentation missed. Always.

Monitoring replication

Replication lag is a ticking bomb with a variable fuse. You need to watch it continuously.

-- On the primary: check all connected replicas
SELECT
    client_addr,
    state,
    sent_lsn,
    write_lsn,
    flush_lsn,
    replay_lsn,
    pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;

Key things to alert on:

• Replay lag exceeding your tolerance. If you promised “reads within 1 second of writes,” alert before that threshold.
• Replica disconnection. A replica that silently falls off is worse than one that loudly crashes.
• WAL accumulation on the primary. If a replica can’t keep up, the primary retains WAL segments. Disk fills. Primary crashes. You’ve now lost the thing that was supposed to protect you.

-- Check WAL retention on the primary
SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0') AS total_wal_bytes;

-- Or more practically, watch the pg_wal directory size
-- and alert when it grows beyond expected bounds.

Choosing your pattern

Skip the decision matrix. Ask three questions:

Can you lose any committed transactions? If no, synchronous or quorum replication. Full stop. This was non-negotiable for the financial data at the fintech startup. A lost trade record isn’t a bug, it’s a regulatory incident.

Do you need writes in multiple regions? If no – and it’s almost always no – use primary-replica. If yes, accept the operational cost of multi-primary and budget engineering time accordingly. At Decloud, we keep writes centralized and replicate reads out. Simpler. Fewer 3am pages.

What’s your read-to-write ratio? If reads dominate (10:1 or more), async replicas for reads with synchronous replication to a single failover standby gives you the best of both: fast reads, safe failover, and write performance that’s only slightly penalized.

Replication isn’t a feature flag. It’s a contract between your system and your users about what happens when hardware fails, networks partition, and Murphy’s Law does its thing. Pick the pattern that matches the promises you’ve actually made, not the ones you wish you could make.

Stop copying postgresql.conf snippets from blog posts. Measure your actual workload, change one thing, measure again. Most Postgres performance problems are bad queries, not bad config.

I’ve been running PostgreSQL in production since my fintech startup days, where our workload was 90% reads with occasional write spikes when market data flooded in. Then at a mobility startup we had the opposite problem – constant GPS pings from bikes generating a firehose of writes. Same database engine, completely different tuning.

That experience taught me something that most “Postgres tuning guides” skip: there’s no universal config. The defaults ship conservative on purpose. Your job is to understand your workload and nudge the knobs that matter for your traffic pattern.

Here is the playbook I actually follow.

Step Zero: Know Where Time Goes

You can’t tune what you can’t see. Before touching any config, enable pg_stat_statements. This single extension has saved me more hours than any other Postgres tool.

# postgresql.conf
shared_preload_libraries = 'pg_stat_statements'

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

Once it’s running, this query shows you exactly where your database spends its time:

SELECT
  substring(query, 1, 80) AS short_query,
  calls,
  round(total_time::numeric, 2) AS total_ms,
  round(mean_time::numeric, 2) AS mean_ms,
  rows
FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 20;

The top 5 results from this query drive 80% of your tuning decisions. Not config file changes. Not hardware upgrades. Query fixes and index work.

At the fintech startup, this query revealed that a single unindexed join in our news aggregation pipeline was responsible for 40% of total database time. We added one composite index and response times dropped by half. No config changes needed.

Connection Management: The Silent Killer

PostgreSQL forks a process per connection. That’s the model. It’s reliable but it doesn’t scale to hundreds of idle connections sitting around doing nothing.

I’ve seen this pattern at every startup I’ve worked at: the app pool defaults to 10 connections per instance, you scale to 50 instances, and suddenly Postgres is managing 500 processes. Most of them idle. All of them eating memory.

Put PgBouncer in Front

[pgbouncer]
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20

Transaction mode is what you want for web workloads. The connection returns to the pool after each transaction, not after the client disconnects. This means 1000 application connections share 20 actual Postgres connections.

The catch: transaction mode breaks session-level features like prepared statements and temp tables. If you need those, use session mode for that specific pool and transaction mode for everything else.

Set max_connections Deliberately

With PgBouncer handling the fan-in, your Postgres max_connections can be surprisingly low. I typically run 50-100 even on busy systems.

-- Check how many connections you actually use
SELECT count(*) FROM pg_stat_activity;
SELECT count(*) FROM pg_stat_activity WHERE state = 'idle';

If your idle count is more than half your total, you have a pooling problem. Fix that before tuning anything else.

Memory: Four Settings That Matter

Memory configuration in Postgres is a balancing act between the database buffer cache and the OS page cache. Get it wrong and they fight each other.

shared_buffers

Start at 25% of total RAM. This is Postgres’s own cache.

shared_buffers = 4GB  # on a 16GB server

Going higher than 25-30% rarely helps and can hurt because you’re stealing memory from the OS cache, which Postgres also relies on for reads.

work_mem

This one is tricky. It’s allocated per sort or hash operation, not per connection. A single complex query with multiple joins can allocate work_mem several times over.

work_mem = 64MB

Before you bump this, check if queries are actually spilling to disk:

EXPLAIN (ANALYZE, BUFFERS)
SELECT o.*, u.name
FROM orders o
JOIN users u ON u.id = o.user_id
WHERE o.created_at > '2019-01-01'
ORDER BY o.created_at DESC;

Look for Sort Method: external merge Disk in the output. That means the sort exceeded work_mem and hit disk. But the fix is usually a better query or a better index, not a bigger work_mem. Only raise it when you have confirmed the query is already optimal.

effective_cache_size

A planner hint, not an allocation. Tell the planner how much total cache is available (Postgres buffers + OS cache).

effective_cache_size = 12GB  # on a 16GB server

maintenance_work_mem

Used by VACUUM and CREATE INDEX. These operations run infrequently so you can be generous here without affecting query concurrency.

maintenance_work_mem = 1GB

Index Discipline: Where the Real Wins Are

I’ll say this plainly: indexing strategy matters more than every postgresql.conf setting combined. A missing index on a hot query path will dominate your performance profile no matter how perfectly tuned your memory settings are.

Find Tables That Need Indexes

SELECT
  schemaname,
  relname AS table_name,
  seq_scan,
  idx_scan,
  n_live_tup AS row_count
FROM pg_stat_user_tables
WHERE seq_scan > idx_scan
  AND n_live_tup > 10000
ORDER BY seq_scan - idx_scan DESC;

Tables with high sequential scans and low index scans on more than 10k rows are your targets.

Index Patterns I Use Constantly

-- Simple lookup (most common)
CREATE INDEX idx_users_email ON users(email);

-- Partial index: only index what you query
CREATE INDEX idx_orders_pending
  ON orders(created_at)
  WHERE status = 'pending';

-- Covering index: avoid hitting the heap
CREATE INDEX idx_orders_user_covering
  ON orders(user_id)
  INCLUDE (status, total);

-- Expression index: for case-insensitive lookups
CREATE INDEX idx_users_email_lower
  ON users(lower(email));

Partial indexes are underused. If you have a million-row orders table but you only ever query pending orders, indexing only the pending rows saves space and speeds up both writes and reads.

Read the Execution Plan

Don’t guess. Run EXPLAIN (ANALYZE, BUFFERS) on every query you care about.

EXPLAIN (ANALYZE, BUFFERS)
SELECT *
FROM orders
WHERE user_id = 123
  AND created_at > '2019-01-01';

Red flags to look for:

• Seq Scan on tables with more than a few thousand rows
• Large gaps between estimated rows and actual rows (means stale statistics)
• Buffers: read numbers that are disproportionately high (means cache misses)

Query Habits That Scale

• Keyset pagination over OFFSET. WHERE id > last_seen_id ORDER BY id LIMIT 20 stays fast at any depth. OFFSET 10000 doesn’t.
• Avoid SELECT * on wide tables. If a table has 40 columns and you need 3, select 3. The planner can use covering indexes and you reduce I/O.
• Keep transactions short. Long transactions hold locks and prevent vacuum from cleaning up dead rows. This is the number one cause of table bloat I see in practice.

Vacuum: The Maintenance You Can’t Skip

MVCC means every UPDATE creates a new row version and leaves the old one behind. VACUUM reclaims that space. If autovacuum falls behind your write rate, tables bloat, indexes bloat, and everything slows down.

Check Bloat Right Now

SELECT
  schemaname,
  relname,
  n_dead_tup,
  n_live_tup,
  round(100.0 * n_dead_tup / nullif(n_live_tup, 0), 2) AS dead_pct,
  last_autovacuum
FROM pg_stat_user_tables
WHERE n_dead_tup > 1000
ORDER BY n_dead_tup DESC;

A dead_pct above 10-20% means autovacuum isn’t keeping up.

Tune Autovacuum for Your Write Load

The defaults assume modest write traffic. For busy tables, lower the scale factor and raise the cost limit so vacuum runs more frequently and more aggressively.

# Global settings
autovacuum_vacuum_scale_factor = 0.02
autovacuum_analyze_scale_factor = 0.01
autovacuum_vacuum_cost_limit = 1000

For specific hot tables, set per-table overrides:

ALTER TABLE orders SET (
  autovacuum_vacuum_scale_factor = 0.01,
  autovacuum_vacuum_cost_limit = 2000
);

At the mobility startup, our GPS tracking table was accumulating dead tuples faster than default autovacuum could clean them. We dropped the scale factor to 0.005 for that one table and the problem disappeared.

WAL and Checkpoints

Write-heavy systems often hit checkpoint spikes – moments where the system pauses to flush dirty pages to disk. Smoothing checkpoints reduces latency jitter.

checkpoint_completion_target = 0.9
wal_buffers = 64MB

If you’re tempted to set synchronous_commit = off for speed, understand what you’re trading. You can lose the last few hundred milliseconds of committed transactions after a crash. For some workloads that’s acceptable. For fintech, it isn’t. Know your domain.

The Monitoring Minimum

You don’t need a fancy dashboard to start. These five checks, run daily or hooked into alerts, catch most problems early:

1. Connection count and idle sessions – pg_stat_activity
2. Cache hit ratio – should be above 99% for OLTP workloads (check pg_statio_user_tables)
3. Top queries by total time – pg_stat_statements
4. Dead tuple ratio on hot tables – pg_stat_user_tables
5. Checkpoint frequency and duration – pg_stat_bgwriter

If your cache hit ratio drops below 99%, you either need more RAM or your working set has outgrown your instance. If checkpoint writes spike, your write load has changed and you need to revisit WAL settings.

It never ends

Postgres tuning isn’t a one-time task. Workloads change. Traffic grows. New features add new query patterns. The cycle is always the same: measure, identify the bottleneck, fix the bottleneck, measure again .

From what I’ve seen, the fix is a better query or a missing index about 80% of the time. Config tuning accounts for maybe 15%. Hardware for the remaining 5%. Spend your time accordingly.

Stop treating TypeScript migration as a rewrite project. Flip one compiler flag, rename your boundary files to .ts, and tighten strictness over weeks. If the migration takes longer than shipping features, you’re doing it wrong.

The codebase that convinced me

At the fintech startup we had a Node.js backend serving financial news to tens of thousands of users. JavaScript everywhere. It worked until it didn’t.

The breaking point was a refactor to our news ranking pipeline. Someone renamed a field from relevanceScore to score in one module. The change looked clean, the tests passed (because the tests mocked the data with the new name), and it shipped. Two days later we noticed ranking was silently broken in production. The field name mismatch meant scores came through as undefined, and our sorting function treated undefined as zero. Every article ranked the same. Users saw noise instead of signal.

That kind of bug can’t happen in TypeScript. The compiler catches it instantly. Not a fancy type trick. Just basic structural checking. After that incident I decided we were migrating.

Why I was annoyed about it

I’m primarily a Go developer. Go has had static types from day one. The idea that a language community needed years of debate to arrive at “maybe we should check types before running code” felt absurd to me. TypeScript shouldn’t feel like a revelation. It should feel like the floor.

But here we are. JavaScript codebases exist, they power real products, and rewriting them in Go (tempting as that sounds) isn’t always practical. So you migrate. Incrementally. Without drama.

The approach that actually works

Forget the blog posts showing a pristine greenfield TypeScript project with perfect types. Real migrations happen in messy codebases with deadlines.

Step one: make TypeScript compile your existing JavaScript. Add a tsconfig.json that allows JS files and skips type checking them. This changes nothing about your build. It just proves the tooling works.

{
  "compilerOptions": {
    "target": "ES2017",
    "module": "commonjs",
    "strict": false,
    "allowJs": true,
    "checkJs": false,
    "esModuleInterop": true,
    "skipLibCheck": true
  },
  "include": ["src/**/*"]
}

Step two: rename your boundary files. Not all of them. Start with the files that define the shapes of data crossing trust boundaries. API request handlers. Database query result types. Config loaders. These are where type mismatches actually cause production bugs. Rename them from .js to .ts and add types to the function signatures.

export interface Article {
  id: string;
  title: string;
  relevanceScore: number;
}

export function parseArticle(input: unknown): Article {
  const data = input as Record<string, unknown>;
  return {
    id: String(data.id),
    title: String(data.title),
    relevanceScore: Number(data.relevanceScore),
  };
}

That parseArticle function isn’t fancy. It’s a boundary. Everything downstream of it knows what shape it’s working with. The field rename bug that bit us at the fintech startup becomes a compile error instead of a silent production failure.

Step three: tighten the compiler gradually. Don’t flip strict: true on day one. You will get a thousand errors and your team will revolt. Instead, enable checks one at a time over a few weeks:

1. noImplicitAny first. This catches the worst category of silent failures.
2. strictNullChecks next. This eliminates the undefined surprise class of bugs.
3. Full strict once the team has momentum and the backlog of type errors is manageable.

You can even run a stricter config on the directories you have already cleaned up while leaving the rest permissive:

{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "strict": true
  },
  "include": ["src/models/**/*", "src/utils/**/*"]
}

What kills migrations

I’ve seen three TypeScript migrations fail. Same pattern every time.

Making it a side project. If migration work isn’t part of normal sprints, it will never finish. The approach that works: new files are .ts by default, and any file you touch for a feature gets migrated as part of that feature work. No separate migration tickets that rot in the backlog.

Sprinkling any everywhere and calling it done. Renaming .js to .ts and casting everything to any isn’t a migration. It’s a lie that makes the build green while providing zero safety. If you find yourself typing as any more than once per file, stop and think about what you’re avoiding.

Converting giant files first. A 2000-line controller file isn’t where you start. Break it up first, then type the pieces. Trying to add types to a monolith module is miserable and teaches the team that TypeScript migration is miserable. Start with small utility files. Quick wins build momentum.

Third-party types

Most popular libraries ship types or have them on DefinitelyTyped. When they don’t, declare the module and move on:

declare module "legacy-widget";

This isn’t ideal, but it’s honest. You’re saying “I don’t know the types for this dependency and I’m not going to pretend.” Better than a wrong type definition that gives false confidence.

Keep the pressure on

Run tsc --noEmit in CI from day one. Make it a blocking check. Track your migration with something embarrassingly simple like counting file extensions:

echo "JS: $(find src -name '*.js' | wc -l)"
echo "TS: $(find src -name '*.ts' | wc -l)"

When the numbers cross over, buy the team lunch. Seriously. Celebrate boring infrastructure wins. They are the ones that actually matter.

The honest trade-off

TypeScript adds friction. Build times go up. Editor tooling sometimes chokes on complex types. The type system has genuine holes (any is a backdoor, type assertions bypass checking, and some runtime patterns are hard to express statically).

But the trade-off is overwhelmingly worth it for any codebase with more than one contributor that will exist for more than six months. TypeScript directly improves stability by catching an entire class of defects before code reaches production. That alone justifies the migration cost.

Just do it incrementally. Do it as part of real work. And for the love of shipping, don’t make it a three-month rewrite project.

Rate limiting is the difference between a healthy API and a 3am incident. Pick token bucket, use Redis, return proper headers, and stop pretending IP-based limits are enough.

At the fintech startup we serve financial news and data through APIs that get hammered by everything from institutional trading bots to someone’s weekend scraping project. I learned rate limiting the hard way: our search endpoint went down because a single client was firing 4,000 requests per minute with no backoff. No rate limiter in place. Just raw, unprotected endpoints and a very bad Monday morning.

After that, I spent a solid two weeks building out our rate limiting infrastructure. Here’s what I learned.

Why You Need It Yesterday

Rate limiting isn’t a nice-to-have. It’s the control plane for your API. Without it:

• One buggy client retry loop takes out your entire service
• A scraper with no sleep interval saturates your database connections
• Your expensive endpoints (search, aggregation, report generation) eat all the compute, starving the cheap ones
• Your cloud bill explodes because downstream dependencies charge per call

At the fintech startup, our financial data endpoints had wildly different costs. A simple quote lookup was cheap. A full news aggregation with NLP sentiment scoring? That could take seconds and significant compute. Treating them the same was a mistake we made early on.

The Decisions That Matter

Before you pick an algorithm, answer these four questions.

What’s the unit of work?

Requests per minute is the obvious one. But it’s not always right. If your endpoints vary wildly in cost – like ours did – you want cost-based budgets. A sentiment analysis call might cost 10 tokens. A simple metadata fetch costs 1. Flat request counts hide this.

Who are you actually limiting?

Your options:

• IP address – Easy. Also terrible for anything serious. NAT gateways, corporate proxies, shared WiFi. We had a financial institution where 200 different users appeared as one IP. Useless.
• API key / access token – Much better. This is what you want for authenticated traffic.
• Organization / tenant – Best for B2B SaaS. We used org-level limits at the fintech startup because a single company might have multiple API keys across different services.

Pick the most granular identifier your auth model supports. You can always aggregate up.

How much burst do you allow?

Financial data is bursty by nature. Market opens, everyone hits the API at once. We needed to allow short bursts without letting sustained abuse through. This is where algorithm choice matters most.

Global or per-endpoint?

Start with a global limit. Then add per-endpoint limits where you know the cost is uneven. We had a global cap of 600 requests/minute but dropped the search endpoint down to 60/minute because it was 10x more expensive than everything else.

Algorithms: The Real Tradeoffs

Fixed Window

Count requests in a hard time window. 1,000 per hour, reset on the hour.

Dead simple. Also has a nasty edge case: a client can burn 1,000 requests at 12:59 and another 1,000 at 13:00. You just allowed 2,000 in two minutes. For financial APIs where burst = real money, this matters.

Sliding Window Counter

Uses the current and previous window counts with a weighted average. Good enough for most cases. This is what a lot of distributed systems default to because it’s cheap to compute and reasonably accurate.

Token Bucket

Tokens refill at a steady rate. Each request burns one or more tokens. Bucket has a max capacity that controls burst size.

This is what we ended up using at the fintech startup. Here’s why: it naturally handles burst (the bucket fills up during quiet periods) while enforcing a hard average rate. And it maps perfectly to cost-based limiting – an expensive endpoint just costs more tokens.

A simplified Redis implementation:

-- Token bucket in Redis via Lua script
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])  -- tokens per second
local cost = tonumber(ARGV[3])         -- tokens this request costs
local now = tonumber(ARGV[4])

local bucket = redis.call('hmget', key, 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or capacity
local last_refill = tonumber(bucket[2]) or now

-- Refill tokens based on elapsed time
local elapsed = now - last_refill
local new_tokens = math.min(capacity, tokens + (elapsed * refill_rate))

if new_tokens >= cost then
    new_tokens = new_tokens - cost
    redis.call('hmset', key, 'tokens', new_tokens, 'last_refill', now)
    redis.call('expire', key, math.ceil(capacity / refill_rate) * 2)
    return {1, new_tokens}  -- allowed, remaining tokens
else
    redis.call('hmset', key, 'tokens', new_tokens, 'last_refill', now)
    redis.call('expire', key, math.ceil(capacity / refill_rate) * 2)
    return {0, new_tokens}  -- rejected, remaining tokens
end

The Lua script is atomic in Redis. No race conditions. No distributed locks. It just works.

Leaky Bucket

Processes requests at a constant rate, queues or rejects the rest. Great for smoothing traffic, but clients hate it when they can’t burst at all. We tried it briefly and got complaints from trading firms whose workflows were inherently bursty. Switched to token bucket within a week.

Implementation: Where and How

Enforce at two layers

At the API gateway, put a coarse global limit. This catches the obvious abuse before it hits your application servers. Nginx, Kong, whatever you use – it can do basic rate limiting.

Inside your application, add the smart limits. Per-endpoint costs, per-org budgets, tiered plans. This is where the token bucket with Redis lives.

// Middleware pseudocode
func RateLimitMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        orgID := extractOrgID(r)
        cost := endpointCost(r.URL.Path)

        allowed, remaining := tokenBucket.Allow(orgID, cost)

        w.Header().Set("X-RateLimit-Limit", "600")
        w.Header().Set("X-RateLimit-Remaining", strconv.Itoa(remaining))
        w.Header().Set("X-RateLimit-Reset", strconv.FormatInt(resetTime(), 10))

        if !allowed {
            w.Header().Set("Retry-After", "30")
            http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
            return
        }

        next.ServeHTTP(w, r)
    })
}

Always return headers

Every response, not just 429s. Clients need to know where they stand. Minimum set:

X-RateLimit-Limit: 600
X-RateLimit-Remaining: 42
X-RateLimit-Reset: 1539628800

When you reject, give them something useful:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: application/json

{"error": "rate_limit_exceeded", "retry_after": 30}

We included retry_after in the JSON body too because some HTTP client libraries make it annoyingly hard to read response headers on error responses.

Redis for distributed counters

Single-node in-memory counters are fine for prototyping. In production with multiple app servers, you need shared state. Redis with EVAL (Lua scripts), INCR, and EXPIRE gives you atomic operations with sub-millisecond latency. We ran a single Redis instance dedicated to rate limiting – separate from our caching layer – and it handled the load without breaking a sweat.

Tiered limits map to product tiers

Free tier: 100 requests/minute, no access to premium endpoints. Pro: 600/minute, full access. Enterprise: custom limits, dedicated support.

This is a product decision as much as a technical one. Your rate limits are your pricing tiers. Make them explicit in your docs and in your headers.

Fail Open or Fail Closed?

When your Redis instance goes down (and it will, eventually), what happens? You have two choices:

Fail open: let all traffic through. Preserves availability but you’re flying blind. One bad client can take you out.

Fail closed: reject everything. Safe but brutal. Legitimate users get 429s for no reason.

We chose fail open with aggressive alerting. The reasoning: a brief period without rate limiting was less damaging than blocking all our paying customers. But we had circuit breakers on the expensive downstream calls as a secondary safety net. Pick explicitly. Don’t let this be a surprise during an incident.

The Checklist

If you’re implementing rate limiting from scratch, here’s the short version:

• Use API key or org ID as the limiting key, not IP
• Token bucket for the algorithm unless you have a specific reason not to
• Redis with Lua scripts for distributed counters
• Return X-RateLimit-* headers on every response
• Return Retry-After on 429s, both in header and body
• Per-endpoint limits where cost varies significantly
• Log every throttle event, alert on sudden spikes
• Decide fail-open vs fail-closed before you need to

Build this before your traffic forces you to. Retrofitting rate limiting during an outage is one of the worst experiences in backend engineering.

At the fintech startup we dealt with real volume. Financial news, market data, user watchlists, story relevance scores – all of it flowing into PostgreSQL around the clock. Tens of millions of rows across our core tables, write-heavy ingestion pipelines running 24/7. We had every reason to think about sharding early. We didn’t. And that restraint saved us months of engineering time.

What sharding actually solves

Sharding splits your dataset across multiple database instances. That’s it. It gives you more write throughput and more total storage capacity by distributing the load.

What it doesn’t solve: slow queries, missing indexes, bad schema design, lack of connection pooling, or the fact that your ORM is generating six joins for a page that should be one SELECT. Sharding won’t fix your application. It will make your application’s problems harder to debug.

At the fintech startup, before we ever discussed sharding, we squeezed everything out of a single Postgres instance. I mean everything:

-- Finding our worst offenders
SELECT query, calls, mean_time, total_time
FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 20;

That query alone – run regularly – told us more about our scaling problems than any architecture whiteboard session. Nine times out of ten the fix was an index, a rewritten query, or moving a hot path behind a cache.

When sharding is actually justified

You need sharding when all of these are true simultaneously:

• Your primary is pegged on CPU or disk I/O from writes, not reads
• Read replicas are already handling your read traffic and it’s not enough
• You’ve tuned work_mem, shared_buffers, effective_cache_size – the obvious Postgres knobs
• Vertical scaling has hit a wall or the cost curve has gone exponential
• Your query patterns naturally partition along a single key

If you’re missing even one of those conditions, you’re probably not ready. Go back and fix the basics.

The checklist before you shard

This is the order I’d recommend. We followed it at the fintech startup and it bought us a long time on a single primary.

1. Fix your queries. Use EXPLAIN ANALYZE. Actually read the output. Look for sequential scans on large tables, nested loop joins where hash joins would work, and sorts that spill to disk.

EXPLAIN ANALYZE
SELECT s.id, s.title, r.score
FROM stories s
JOIN relevance_scores r ON r.story_id = s.id
WHERE r.user_id = 42
  AND s.published_at > NOW() - INTERVAL '7 days'
ORDER BY r.score DESC
LIMIT 50;

If that’s doing a seq scan on relevance_scores, you don’t have a sharding problem. You have an indexing problem.

2. Add proper indexes. Composite indexes aligned to your query patterns. Partial indexes for hot subsets. Don’t just throw a B-tree on every column.

CREATE INDEX CONCURRENTLY idx_relevance_user_score
ON relevance_scores (user_id, score DESC)
WHERE score > 0.5;

3. Introduce caching. We cached story metadata, user watchlists, and computed relevance scores in Redis. This alone dropped our read load by 60-70%.

4. Connection pooling. PgBouncer. Set it up in transaction mode. PostgreSQL is terrible at handling thousands of connections natively. This is free performance.

5. Archive cold data. We partitioned our historical data by month and moved anything older than six months to separate tables. Queries on recent data got dramatically faster.

6. Read replicas. Route all reporting, analytics, and non-critical reads to replicas. Your primary should only handle writes and the reads that absolutely need consistency.

Only after exhausting all of that should sharding enter the conversation.

Choosing a shard key

This is the decision that will haunt you. Get it wrong and you’re looking at a painful re-sharding down the road. The shard key determines which instance holds which rows.

Good shard keys share these traits: high cardinality, even distribution, stability over time, and presence in nearly every query you run.

For us at the fintech startup, user_id was the obvious candidate for user-facing data. All watchlist queries, relevance scores, and personalization features were scoped to a user. Everything a user needed lived together.

For the ingestion pipeline – stories, sources, raw market data – that was a different story. Literally. We kept that on a single large instance with partitioning rather than sharding because the access patterns didn’t have a natural partition key. Stories get read by many users. There’s no clean split.

Bad shard keys I’ve seen people pick:

• created_at – all new writes go to one shard. Congratulations, you’ve created a hotspot
• country – massively uneven. One shard gets 40% of traffic
• status – three distinct values. That’s three shards at most, and one of them has 90% of the data

Sharding strategies in practice

Hash-based

-- Application-level routing
shard_id = hash(user_id) % num_shards

Even distribution. No hotspots on write. But range queries become fan-out queries across all shards, which is expensive. Adding a new shard means rehashing and moving data around. Consistent hashing helps but doesn’t eliminate the problem.

Range-based

shard_1: user_id 1 - 1,000,000
shard_2: user_id 1,000,001 - 2,000,000

Simple to reason about. Range queries stay local. But new users always hit the latest shard, creating a write hotspot. Rebalancing means splitting ranges and moving data.

Directory-based

A lookup table maps each key to a shard. Maximum flexibility – you can move individual users between shards. The cost is an extra lookup on every single query and another system (the directory) that has to be highly available. If the directory goes down, everything goes down.

Cross-shard operations are where it hurts

Here’s what nobody tells you until you’re already committed: the moment you shard, joins become your enemy.

-- This worked fine on a single instance
SELECT u.name, COUNT(o.id)
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE u.region = 'EU'
GROUP BY u.name;

On a sharded setup with user_id as the shard key, this might still work if users and orders are co-located on the same shard. But what if you need orders joined with products, and products live on a different shard? Now you’re doing application-level joins. Pulling data from multiple shards into your app server and joining in memory. It’s slow and it’s ugly.

Cross-shard transactions are worse. Two-phase commit exists, but it kills throughput and introduces failure modes that will wake you up at 3am. At the fintech startup we avoided this entirely by designing our data model so that everything a single request needs lives on the same shard. Sometimes that meant denormalization. Sometimes it meant copying reference data to every shard. Both are fine tradeoffs.

Routing: who decides which shard?

Three options, each with costs.

Application-level routing – your code computes the shard and talks to the right database. Full control. But now every service that touches the database needs routing logic. We preferred this at the fintech startup because it was explicit and debuggable.

Proxy routing – something like PgBouncer with custom routing, Vitess, or ProxySQL sits between your app and the databases. Simpler application code but you’re adding another piece of infrastructure to monitor, scale, and debug when things go wrong at 2am.

Database-native sharding – Citus for PostgreSQL is the one I’d look at if I were doing this today. It handles routing and distributed queries within Postgres itself. Less custom code. But it has its own performance characteristics and operational model that you need to understand deeply before committing.

The operational reality

Sharding isn’t a one-time project. It’s a permanent increase in operational complexity.

Every shard needs monitoring. Every shard needs backups. Every shard needs failover testing. If you have 8 shards, you have 8 databases to manage, and the failure of any single one affects a subset of your users.

You need to plan for rebalancing before shards fill up. If shard 3 is at 85% capacity and growing 5% per month, you have roughly three months to act. Automating this is hard. Most teams don’t, and then it becomes a fire drill.

You also need runbooks for partial failures. What happens when shard 5 goes down? Do those users see errors? Degraded functionality? Do other shards pick up the load? These are questions you need answered before going to production, not during an incident.

Bottom line

Sharding is a last resort that works. When you genuinely need it – when a single instance can’t keep up and you’ve already optimized, cached, replicated, and partitioned – it’s the right tool. But it permanently changes your operational model, your data model, and how every engineer on the team thinks about the database.

At the fintech startup we pushed a single PostgreSQL instance far further than most people think is possible. Proper indexing, pg_stat_statements, PgBouncer, aggressive caching, table partitioning. That got us to the scale we needed without the overhead of a sharded architecture.

If you’re reading this and considering sharding, run EXPLAIN ANALYZE on your slowest queries first. The answer is probably in there.

I’ve seen this pattern enough times now that I refuse to participate in it. Here’s how I actually approach database performance problems.

“The database is slow” isn’t a problem

That sentence tells me nothing. Which query? Which endpoint? What’s the acceptable latency? When did it start? What changed?

At the fintech startup, when someone says the API is slow, I make them fill in the blanks: “The /api/prices/{ticker} endpoint exceeds 2 seconds during market hours after we added the sentiment join last Tuesday.” Now I have a target. A timeline. Something I can actually investigate.

Without that specificity you’re debugging vibes.

Measure first. Always.

Before touching anything, turn on slow query logging and get a baseline. I run PostgreSQL for everything I can, so:

ALTER SYSTEM SET log_min_duration_statement = '50ms';
SELECT pg_reload_conf();

I set the threshold low — 50ms — because I want to see the full picture, not just the obvious disasters. You’d be surprised how many “fast” queries add up when they run thousands of times per minute.

The slow query log tells you what’s actually slow. Not what your gut says is slow. Not what the intern thinks is slow. What the database measured as slow.

EXPLAIN ANALYZE is your best friend

I mean this literally. If you’re doing database work and you don’t use EXPLAIN (ANALYZE, BUFFERS) regularly, you’re flying blind.

Here’s what I ran for our price history problem:

EXPLAIN (ANALYZE, BUFFERS)
SELECT p.ticker, p.price_date, p.open, p.close, p.volume
FROM price_history p
WHERE p.ticker = 'AAPL'
  AND p.price_date BETWEEN '2017-01-01' AND '2017-08-01'
ORDER BY p.price_date;

The output told me everything. The planner estimated 200 rows. Reality: 147,000. When estimates are that far off, PostgreSQL picks terrible plans. It chose a nested loop where a hash join would’ve been orders of magnitude faster. The statistics on price_history were stale — we’d bulk-loaded six months of historical data and never ran ANALYZE.

ANALYZE price_history;

One command. Query went from 8 seconds to 90 milliseconds. No new index. No config change. Just accurate statistics.

That’s the thing about database tuning. The fix is often embarrassingly simple once you know what’s wrong.

Finding the actual bottleneck

After you read the plan, you’re looking for one of a few usual suspects:

Sequential scans on large tables. If PostgreSQL is scanning 50 million rows to find 12, you need an index. But only for the columns your queries actually filter on.

Joins that explode row counts. We had a query joining price_history to news_sentiment that produced a cartesian product on certain tickers. The join condition was missing a date constraint. The query went from returning 500 rows to materializing 3 million intermediate rows.

Lock contention. Bulk inserts fighting with read queries. We see this during our nightly data loads. pg_stat_activity tells you who’s waiting on whom.

Memory pressure. If your working set doesn’t fit in shared_buffers, every query hits disk. Check your buffer cache hit ratio:

SELECT
  sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read), 0) AS cache_hit_ratio
FROM pg_statio_user_tables;

Anything below 0.99 on a read-heavy workload means you’re probably leaving performance on the table. Literally.

Pick the biggest contributor. Fix that one thing. Measure again.

Indexing: powerful but not free

I love indexes. I also hate bad indexes. Every index you add slows down writes and consumes disk. “Index everything” is a strategy born from laziness.

At the fintech startup our price_history table gets thousands of inserts per minute during market hours. We can’t afford garbage indexes. So I index what the queries need, and nothing else.

CREATE INDEX idx_price_history_ticker_date
ON price_history (ticker, price_date DESC);

Column order matters. This index is perfect for “give me AAPL prices sorted by date.” Reverse the columns and the planner might not use it at all.

PostgreSQL partial indexes are incredibly useful for skewed data:

CREATE INDEX idx_active_alerts
ON price_alerts (ticker, created_at)
WHERE active = true;

Only 5% of alerts are active at any given time. This index is tiny compared to indexing the full table, and it’s the only query pattern we care about for active alerts.

Query shape matters more than you think

Stop writing SELECT *. I see this constantly. You need three columns but you’re transferring 40 because the ORM defaulted to SELECT *. On a table with a JSONB column holding 2KB of metadata per row, that’s the difference between a 50ms query and a 500ms query.

Rewrite subqueries as joins when it makes sense. PostgreSQL’s planner is good but it’s not magic. I had a correlated subquery that checked whether each ticker had recent sentiment data:

-- Slow: correlated subquery, runs once per row
SELECT ticker, close
FROM price_history
WHERE price_date = '2017-08-01'
  AND EXISTS (
    SELECT 1 FROM news_sentiment
    WHERE news_sentiment.ticker = price_history.ticker
      AND sentiment_date > '2017-07-01'
  );

-- Fast: rewritten as join
SELECT DISTINCT p.ticker, p.close
FROM price_history p
JOIN news_sentiment s ON s.ticker = p.ticker
WHERE p.price_date = '2017-08-01'
  AND s.sentiment_date > '2017-07-01';

The join version let PostgreSQL hash the sentiment table once instead of probing it for every row. 40x improvement.

Connection management

This one bites people who don’t expect it. PostgreSQL forks a process per connection. 200 connections means 200 processes fighting over CPU and memory. I’ve seen a server with 16 cores running 500 connections, where cutting to 50 connections through PgBouncer doubled throughput.

Your connection pool size should match your database’s capacity, not your application’s thread count. For most workloads, (2 * CPU cores) + number of disks is a reasonable starting point. We run 24 connections at the fintech startup for a 12-core database server and it handles our load fine.

Configuration: change one thing at a time

PostgreSQL ships with conservative defaults because it has to run on everything from a Raspberry Pi to a 256GB server. A few settings worth adjusting:

• shared_buffers: 25% of RAM is the standard advice. It’s decent advice.
• effective_cache_size: 75% of RAM. Tells the planner how much cache to expect from the OS.
• work_mem: Be careful here. This is per-operation, not per-query. A complex query with five sorts uses 5 * work_mem. I’ve seen people set this to 1GB and then wonder why they’re OOM.
• maintenance_work_mem: Crank this up for VACUUM and CREATE INDEX operations. 512MB or 1GB is reasonable.

Change one setting. Measure. If it helps, keep it. If it doesn’t, revert. This isn’t the place for creativity.

Don’t ignore the ORM

If you’re using Django, Rails, or any ORM — you need to know what SQL it generates. ORMs are convenient. They’re also capable of producing spectacularly bad queries when you’re not paying attention. N+1 queries are the classic, but I’ve seen ORMs generate queries with unnecessary subselects, missing join conditions, and implicit type casts that defeat indexes.

Log the SQL. Read it. If it’s bad, drop to raw SQL for that query. The ORM won’t be offended.

The loop

That’s really all there’s to it. Define the problem precisely. Measure with real data. Read the execution plan. Fix the biggest bottleneck. Measure again. Repeat.

It’s not glamorous. There’s no single magic setting that makes everything fast. But this process works every time, and it works because you’re making decisions based on data instead of folklore.

Every performance win I’ve had at the fintech startup came from this loop. The boring, repeatable loop.

The short version: state scales cheapest when you get more out of one machine before you distribute. Connection pooling is free headroom, replication is the first major lever, and partitioning buys time before sharding complexity. I’ve been through this progression at the fintech startup and the order matters more than any single technique.

The context

At the fintech startup we ingest financial news from thousands of sources. Earnings calls, filings, analyst reports, social signals – all of it lands in PostgreSQL. When I joined as CTO, the database was a single server doing everything: writes from the ingestion pipeline, reads from the API, analytics queries from internal tools. It worked. Then the data volume doubled in three months and it stopped working.

This post is what I learned fixing it. Not theory. The actual progression we followed, the configs we used, and the tradeoffs we hit.

Find the bottleneck first

Everything looks the same when the database is slow. Queries pile up, latency spikes, the app feels sluggish. But the fix depends entirely on where the pressure is coming from.

Read pressure means CPU is saturated serving SELECT queries. Write pressure means WAL generation and fsync are the bottleneck. Connection pressure means the process-per-connection model is eating memory. Storage pressure means the tables are too big for efficient vacuuming and indexing.

At the fintech startup, we had all four. But they didn’t all matter equally. Connection pressure was killing us first because the ingestion workers each held their own connection. We had maybe 300 workers and the server was spending more time context-switching between backends than doing actual work.

Connection pooling with PgBouncer

This was the first fix and it was nearly free. PgBouncer sits between the application and PostgreSQL, multiplexes hundreds of client connections over a small pool of real database connections, and eliminates the fork-per-connection overhead.

Here is close to what we ran:

[databases]
fintech_db = host=127.0.0.1 port=5433 dbname=fintech_db

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 5432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

pool_mode = transaction
max_client_conn = 2000
default_pool_size = 30
reserve_pool_size = 5
reserve_pool_timeout = 3

server_idle_timeout = 300
server_lifetime = 3600

log_connections = 0
log_disconnections = 0

A few things worth noting. pool_mode = transaction is the right default for almost every workload. It returns the server connection to the pool after each transaction completes, so 2000 application connections share 30 real database connections. Session mode keeps the connection pinned for the entire client session, which defeats the purpose. Statement mode is the most aggressive but breaks anything that uses multi-statement transactions or prepared statements.

reserve_pool_size gives you a small buffer for traffic spikes. If all 30 connections are busy, PgBouncer will open up to 5 more for a short window before rejecting clients. This saved us during batch ingestion runs where write volume would spike for a few minutes.

The result: our 300 ingestion workers plus API servers plus internal tools all shared 30 actual PostgreSQL backends. Memory usage on the database server dropped by 40%. Query latency improved because the server wasn’t spending cycles managing hundreds of idle connections.

Streaming replication

Pooling bought us time. The next bottleneck was read traffic. The API served search results, financial summaries, and news feeds – all read-heavy queries competing with the write path on a single server.

PostgreSQL streaming replication is straightforward. The primary streams WAL (Write-Ahead Log) segments to one or more standbys, which replay them and serve read traffic. Here is the relevant part of the primary’s configuration:

# postgresql.conf on the primary
wal_level = replica
max_wal_senders = 5
wal_keep_segments = 64

# For monitoring replication lag
track_commit_timestamp = on

And the standby’s recovery.conf:

standby_mode = on
primary_conninfo = 'host=primary.internal port=5432 user=replicator password=xxx'
trigger_file = '/tmp/postgresql.trigger'

We set up two read replicas behind a simple connection routing layer in Go. Write queries went to the primary. Read queries went to the replicas using round-robin. Nothing fancy.

The thing people underestimate about replication is lag. A user writes a comment, the API returns success, the user refreshes and the comment isn’t there because the replica hasn’t caught up. At the fintech startup this mattered less because our data was financial news – a few hundred milliseconds of lag on search results is invisible. But for any read-after-write path, you need to either route reads back to the primary for a short window or track replication position and only read from replicas that have caught up.

We monitored lag with a simple query on the replicas:

SELECT
  now() - pg_last_xact_replay_timestamp() AS replication_lag;

In steady state we saw 50-200ms of lag. During batch ingestion spikes it would climb to 1-2 seconds. We set alerts at 5 seconds. If lag hits 10 seconds, something is wrong with the replica’s I/O or the WAL shipping is backing up.

Partitioning for high-volume tables

After pooling and replicas, the next problem was table size. Our main events table held every financial event we had ever ingested. Hundreds of millions of rows. VACUUM took hours. Index rebuilds blocked writes. Queries that should have been fast were scanning enormous B-trees.

In 2016, PostgreSQL doesn’t have native declarative partitioning. That’s coming in version 10. What we’ve is table inheritance with CHECK constraints and manual routing. It’s ugly. It works.

-- Parent table
CREATE TABLE financial_events (
    id          BIGSERIAL,
    event_time  TIMESTAMP NOT NULL,
    source_id   INTEGER NOT NULL,
    event_type  VARCHAR(50) NOT NULL,
    payload     JSONB NOT NULL
);

-- Monthly partitions
CREATE TABLE financial_events_2016_10 (
    CHECK (event_time >= '2016-10-01' AND event_time < '2016-11-01')
) INHERITS (financial_events);

CREATE TABLE financial_events_2016_11 (
    CHECK (event_time >= '2016-11-01' AND event_time < '2016-12-01')
) INHERITS (financial_events);

-- Indexes on each partition, not the parent
CREATE INDEX idx_fe_2016_10_time ON financial_events_2016_10 (event_time);
CREATE INDEX idx_fe_2016_10_source ON financial_events_2016_10 (source_id);

CREATE INDEX idx_fe_2016_11_time ON financial_events_2016_11 (event_time);
CREATE INDEX idx_fe_2016_11_source ON financial_events_2016_11 (source_id);

The insert routing was a trigger function:

CREATE OR REPLACE FUNCTION financial_events_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
    IF NEW.event_time >= '2016-11-01' AND NEW.event_time < '2016-12-01' THEN
        INSERT INTO financial_events_2016_11 VALUES (NEW.*);
    ELSIF NEW.event_time >= '2016-10-01' AND NEW.event_time < '2016-11-01' THEN
        INSERT INTO financial_events_2016_10 VALUES (NEW.*);
    -- ... older months
    ELSE
        RAISE EXCEPTION 'No partition for event_time %', NEW.event_time;
    END IF;
    RETURN NULL;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER insert_financial_events
    BEFORE INSERT ON financial_events
    FOR EACH ROW EXECUTE FUNCTION financial_events_insert_trigger();

Yes, you have to maintain this trigger as you create new partitions. We wrote a cron job in Go that created next month’s partition and updated the trigger on the first of every month. Not glamorous. Reliable.

The wins were immediate. VACUUM on a monthly partition with 20 million rows takes minutes, not hours. Queries with WHERE event_time BETWEEN ... hit only the relevant partitions because constraint exclusion prunes the rest. And when data aged past our retention window, we dropped entire partitions instead of running massive DELETE queries – DROP TABLE financial_events_2015_06 is instant and generates zero dead tuples.

Sharding: the last resort

We didn’t shard at the fintech startup. I want to be honest about that. We got to the edge where it was on the table, but the combination of pooling, two replicas, and monthly partitioning handled our volume. We were ingesting roughly 50,000 events per minute at peak and PostgreSQL on decent hardware with these optimizations kept up.

But I’ve seen sharding done at a previous company, and my strong opinion is: don’t do it unless you have exhausted everything above and a single primary genuinely can’t handle the write throughput.

Sharding means splitting data across independent PostgreSQL instances. You pick a shard key – usually a tenant ID or a hash of some natural key – and route writes and reads to the correct shard. The application or a middleware layer owns the routing.

The hidden costs are brutal. Schema migrations have to be applied to every shard. Cross-shard queries become application-level aggregation. Rebalancing shards when data distribution skews means moving live data between databases. Transactions that span shards are either impossible or require two-phase commit, which is slow and fragile. Every operational runbook gets multiplied by the number of shards.

If you’re at the point where sharding is necessary, you have a big enough team and budget to handle the operational complexity. If you don’t have that team, sharding will hurt more than the performance problem it solves.

The progression matters

The order isn’t arbitrary.

1. Pooling is nearly free and should be in place from day one. There’s no reason to let PostgreSQL manage hundreds of connections directly.
2. Read replicas are the first real scaling lever. They offload the most common pressure (reads) with minimal application changes.
3. Partitioning makes large tables manageable and solves vacuum, indexing, and retention problems that replicas don’t help with.
4. Sharding is the nuclear option. Powerful. Expensive. Irreversible in practice.

Each step is worth a significant amount of effort to delay the next one. We spent a week tuning PgBouncer and it delayed the need for replicas by two months. We spent two weeks setting up replication and it delayed the partitioning work by four months. Partitioning has delayed any sharding conversation indefinitely.

What I would do differently

I would set up PgBouncer before the first production deploy, not after the first connection storm. I would build the partitioning infrastructure from the start for any table expected to grow past 50 million rows. And I would invest more in monitoring replication lag early – we flew blind for the first few weeks after setting up replicas and got lucky that lag never caused a visible bug.

Scaling PostgreSQL is a sequence, not a leap. Each step buys you months. Skip the sequence and you pay for every shortcut at once.

So we set up ELK. Elasticsearch, Logstash, Kibana. The pitch was compelling: open source, flexible, great full-text search, a plugin for everything. We stood up a three-node cluster, pointed Logstash at it, gave the team Kibana dashboards. For the first few weeks, it felt like a superpower.

Then the cluster started misbehaving. And I spent the next several months learning a painful lesson about the gap between “powerful” and “worth the operational cost.”

If you don’t have someone who wants to babysit Elasticsearch full time, don’t run ELK yourself. Hosted Elasticsearch, Graylog, or even Splunk will save you more engineering hours than they cost.

The Operational Tax Nobody Warns You About

Elasticsearch isn’t a database you deploy and forget. It’s a distributed system that demands constant attention. Shard allocation, index lifecycle, JVM heap tuning, split-brain prevention, disk watermarks. Every one of these will bite you, and they will bite you at 3 AM.

At the mobility startup, our log volume was moderate. Maybe a few gigabytes a day. Nothing that should stress a three-node cluster. But Elasticsearch doesn’t care about your expectations. It cares about index design, merge policies, and whether you remembered to set bootstrap.memory_lock. We spent more time keeping the logging infrastructure healthy than we spent on some of our actual product services.

The worst part is that when your logging system goes down, you lose visibility into everything else at the same time. Your safety net disappears exactly when you need it most.

What ELK Actually Gets Right

I’m not going to pretend it’s all bad. Elasticsearch’s search is genuinely excellent. When the cluster is healthy, the ability to run arbitrary queries across millions of log lines with sub-second response times is hard to match. Logstash can parse almost any log format into structured fields. Kibana dashboards gave our product team visibility they never had before.

The ecosystem is real. Beats shippers are lightweight, community plugins cover most integrations, and the documentation is solid. If you have the operational muscle, ELK is the most flexible open source logging stack available.

What I’d Do Differently

If I were starting over, I wouldn’t self-host Elasticsearch for logging. Full stop.

Hosted Elasticsearch removes the worst of the operational burden. You keep the same query model, the same Kibana dashboards, the same integrations. You lose some control over cluster configuration and you pay more per gigabyte, but you gain back the engineering hours you were burning on cluster babysitting. For most teams, that tradeoff is obvious.

Graylog is worth a look if you want Elasticsearch search without the full DIY build. It wraps Elasticsearch in a more opinionated log management experience with built-in alerting and stream routing. Less flexible than raw ELK, but faster to get running and easier to keep running.

Splunk is the enterprise answer. Powerful, mature, battle-tested. Also expensive enough to make your finance team flinch. If budget isn’t the constraint, Splunk is a safe bet. For a startup, it rarely makes sense.

Cloud provider logging is the lowest-effort option. AWS CloudWatch Logs, for example, integrates deeply with everything else in AWS and requires zero operational overhead. The query capabilities are basic compared to Elasticsearch, but basic is often enough. You can always export to something more powerful later.

Structured Logging Is the Real Win

Regardless of which aggregation tool you pick, the single best investment is structured logging. A JSON log line with consistent fields turns debugging from archaeology into search.

{
  "timestamp": "2016-09-05T10:23:45Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123",
  "message": "Payment processing failed",
  "error": "Connection timeout",
  "customer_id": "cust_456"
}

Carry a trace ID through every service. Use consistent field names. Emit JSON instead of free-form text. These decisions pay off no matter what sits behind the ingestion pipeline. If you do nothing else, do this.

Put a Buffer in Front of Ingestion

One more thing I learned the hard way: put a queue between your log shippers and your aggregation layer. Kafka, Redis, even a simple file buffer. Traffic spikes will happen. Deploys will happen. If your pipeline has no buffer, you drop logs during the exact moments you need them most.

Pick your logging battles

ELK is powerful software with brutal operational costs. Most teams underestimate how much work Elasticsearch is to run, and they find out at the worst possible time. If you can afford someone who genuinely enjoys tuning JVM garbage collection and shard allocation, go for it. Otherwise, pay for a hosted solution or pick a simpler tool. Your on-call engineers will thank you.

If your migration strategy is “take the site down, run ALTER TABLE, pray, bring it back up,” you’re doing it wrong. Every schema change can be decomposed into steps that keep old and new code running simultaneously. It takes more discipline but zero heroics.

The problem with “just run the migration”

At the fintech startup we serve financial data. Market data doesn’t pause because your engineering team needs a maintenance window. A feed that goes dark during trading hours is a feed that loses subscribers. When I took over database operations, the existing pattern was to schedule migrations at 2 AM on Sundays, take the API offline, run the DDL, and hope the application came back cleanly. It worked until it didn’t.

The breaking point was a column type change on a table with 40 million rows of historical price data. The migration ran for 47 minutes. During that time the table was locked, the API returned errors, and a downstream consumer silently switched to a stale cache it never recovered from. We spent the next two days cleaning up data consistency issues.

After that I decided every migration would be zero-downtime or it wouldn’t ship.

The expand-and-contract pattern

The core idea is simple. Instead of making a breaking change in one step, you split it into phases where the schema is always compatible with whatever application code is currently running.

Expand: add the new structure alongside the old one. Both coexist.

Migrate: deploy code that writes to both old and new, reads from new. Backfill historical data.

Contract: once every running instance uses the new structure, remove the old one.

This pattern handles nearly every schema change. The details vary, but the rhythm stays the same.

Adding a column safely

The simplest case. A nullable column with no default doesn’t rewrite the table in PostgreSQL. It’s metadata-only and takes a lock for milliseconds.

ALTER TABLE trades ADD COLUMN settlement_date DATE;

Fast. Safe. Old code ignores the column. New code starts writing to it.

If you need a NOT NULL constraint, don’t add it in the same statement. PostgreSQL will scan the entire table to verify the constraint, holding an ACCESS EXCLUSIVE lock the whole time. Instead:

-- Step 1: add nullable column
ALTER TABLE trades ADD COLUMN settlement_date DATE;

-- Step 2: backfill in batches (see below)

-- Step 3: add constraint without full table scan
ALTER TABLE trades ADD CONSTRAINT trades_settlement_not_null
  CHECK (settlement_date IS NOT NULL) NOT VALID;

-- Step 4: validate separately (only takes a SHARE UPDATE EXCLUSIVE lock)
ALTER TABLE trades VALIDATE CONSTRAINT trades_settlement_not_null;

The NOT VALID trick is critical on large tables. It tells PostgreSQL to enforce the constraint on new writes immediately but skip the full table scan. The VALIDATE step runs later with a weaker lock that doesn’t block writes. On our 40-million-row tables, this was the difference between a 200-millisecond migration and a 20-minute outage.

Renaming a column

You don’t rename columns in production. Full stop.

What you actually do is expand-and-contract. Here is the concrete sequence we used when renaming price to unit_price on the trades table:

-- Phase 1: Expand
ALTER TABLE trades ADD COLUMN unit_price NUMERIC(18,8);

Deploy code that writes to both price and unit_price. Reads come from unit_price with a fallback to price.

-- Phase 2: Backfill
UPDATE trades SET unit_price = price
WHERE unit_price IS NULL
AND id BETWEEN 1 AND 100000;

-- Repeat in batches...
UPDATE trades SET unit_price = price
WHERE unit_price IS NULL
AND id BETWEEN 100001 AND 200000;

Once backfill is complete and every application instance reads from unit_price:

-- Phase 3: Contract
ALTER TABLE trades DROP COLUMN price;

Three deployments minimum. That’s the cost. The benefit is zero downtime and a clean rollback at every step.

Changing a column type

Same pattern, different details. We had a case where an instrument identifier was stored as INTEGER but needed to become TEXT to support a new exchange format.

-- Expand
ALTER TABLE instruments ADD COLUMN external_id_new TEXT;

-- Backfill
UPDATE instruments SET external_id_new = external_id::TEXT
WHERE external_id_new IS NULL
AND id BETWEEN 1 AND 50000;

Deploy dual-write code. Cut over reads. Drop the old column.

The temptation is to use ALTER COLUMN ... TYPE TEXT which rewrites the entire table under an exclusive lock. On a table with millions of rows of financial data that’s actively being queried, that isn’t an option.

Batched backfills that don’t kill the database

Large backfills are where most zero-downtime migrations go wrong. A single UPDATE ... WHERE condition on 40 million rows will generate enormous WAL, bloat the table, and hold locks that block concurrent operations.

The pattern I use:

DO $$
DECLARE
  batch_size INT := 5000;
  max_id BIGINT;
  current_id BIGINT := 0;
BEGIN
  SELECT MAX(id) INTO max_id FROM trades;

  WHILE current_id < max_id LOOP
    UPDATE trades
    SET unit_price = price
    WHERE id > current_id
      AND id <= current_id + batch_size
      AND unit_price IS NULL;

    current_id := current_id + batch_size;
    PERFORM pg_sleep(0.1);  -- breathe
    COMMIT;
  END LOOP;
END $$;

The pg_sleep matters. Without it, you saturate I/O and replication lag spikes. At the fintech startup our replicas served read traffic for dashboards and analytics. A backfill that caused 30 seconds of replication lag would show stale prices to every user watching a portfolio. We settled on batches of 5,000 rows with 100ms pauses. The backfill took longer but production stayed healthy.

Indexes without locking

CREATE INDEX on PostgreSQL takes a SHARE lock on the table, which blocks writes for the duration of the build. On a large table, that can be minutes.

CREATE INDEX CONCURRENTLY idx_trades_settlement ON trades(settlement_date);

CONCURRENTLY builds the index without blocking writes. It takes longer and does two table scans instead of one, but it’s the only option for production tables. There are two caveats worth knowing:

First, it can’t run inside a transaction. If your migration tool wraps everything in a transaction, you need to handle this case separately.

Second, if it fails partway through, it leaves an invalid index behind. Check pg_stat_user_indexes and drop the invalid one before retrying.

SELECT indexrelid::regclass, indisvalid
FROM pg_index
WHERE NOT indisvalid;

Foreign keys without blocking

Adding a foreign key constraint with ADD CONSTRAINT ... FOREIGN KEY does a full table scan under an ACCESS EXCLUSIVE lock. Same trick as NOT NULL:

ALTER TABLE trades
  ADD CONSTRAINT fk_trades_instrument
  FOREIGN KEY (instrument_id) REFERENCES instruments(id)
  NOT VALID;

ALTER TABLE trades VALIDATE CONSTRAINT fk_trades_instrument;

The first statement adds the constraint for new writes only, taking a brief lock. The second validates existing rows with a much weaker lock.

The deployment sequence

The order matters. Getting it wrong means you have a window where the application and schema disagree.

1. Run expand migration. New columns, new indexes (concurrently), new constraints (NOT VALID). Schema is now compatible with both old and new code.

2. Deploy new application code. Writes to both old and new columns. Reads from new with fallback to old.

3. Run backfill. In batches, during low traffic if possible. Monitor replication lag and query latency.

4. Verify completeness. SELECT COUNT(*) FROM trades WHERE unit_price IS NULL should be zero.

5. Deploy read-cutover code. Reads exclusively from new columns. Still writes to both.

6. Run contract migration. Drop old columns, drop temporary constraints. Schema is clean.

Each step is independently reversible. If the backfill causes problems, pause it. If the new code has bugs, roll back the deployment. The old schema is still there. This is the entire point.

What this costs

More deployments. More code that handles two schemas simultaneously. Migrations that used to be one PR become three or four. Application code carries temporary dual-write logic that gets cleaned up in the contract phase.

It’s more work. But at the fintech startup, the alternative was telling financial data consumers that we needed a maintenance window during market hours. That conversation never goes well.

The discipline pays off in a different way too. When every migration follows expand-and-contract, deployments become boring . Nobody pages you at 2 AM for a schema change. Nobody holds their breath during a release. The process is mechanical and predictable.

The best migration is the one nobody notices. Boring by design, invisible in production.

Pick Postgres. If your workload is dead-simple CRUD and your team already bleeds MySQL, fine, stay there. For everything else in 2016, PostgreSQL gives you more database and fewer workarounds.

The short answer

I’m building the fintech startup on PostgreSQL. Financial data, JSONB documents, full-text search across thousands of sources, strict schema enforcement for anything that touches money. I evaluated MySQL honestly. Postgres won on every axis that matters to me.

That doesn’t make MySQL bad. It makes the decision obvious for my workload. Here is how I see the tradeoffs.

The comparison

Where Postgres pulls ahead

Correctness is the default. I don’t want my database silently truncating a currency field or accepting a string where an integer belongs. Postgres refuses bad data. MySQL lets it through unless you configure it not to. In financial systems, the database being strict isn’t a feature request. It’s the minimum.

JSONB changes what you can do. At the fintech startup we store semi-structured financial events alongside relational data. Postgres lets me index into JSONB, query nested fields, and join it with relational tables in one query. With MySQL I would be serializing JSON, pulling it into the application, and filtering there. That isn’t a comparison. That’s a generation gap.

Full-text search removes a dependency. We search across news sources, filings, and analyst content. Postgres full-text search with ts_vector, dictionaries, and ranking handles this without bolting on a separate search cluster. One fewer service to operate, monitor, and keep in sync.

Window functions and CTEs aren’t optional. If you do any reporting or analytics, you need them. MySQL not having them in 2016 means your choices are ugly subqueries, dumping data into a separate analytics tool, or doing the math in application code. Postgres just does it.

Where MySQL wins

I’ll give MySQL its due.

Connection scaling. Postgres forks a process per connection. At a few hundred connections, memory adds up fast and you need a pooler. MySQL handles thousands of threads without breaking a sweat. If your architecture has many direct database connections and you don’t want to manage PgBouncer, that matters.

Operational simplicity for upgrades. MySQL major version upgrades tend to be less painful. Postgres upgrades have gotten better, but they still require more planning and occasionally downtime.

Ubiquity. MySQL is everywhere. Every shared host, every tutorial, every legacy system. If you’re inheriting a MySQL codebase and the schema is simple, migrating to Postgres just because you prefer it is a waste of time. Use what is there.

My decision framework

Three questions:

1. Does your data need to be correct, or just present? If correctness matters–financial data, health records, billing–Postgres. Its strictness isn’t friction. It’s protection.

2. Do you need more than basic SELECT/INSERT/UPDATE? If you need JSONB, full-text search, window functions, CTEs, or custom types, Postgres gives you those today. MySQL will make you bolt on external tools or wait for features that aren’t shipping in 2016.

3. Is your team already deep in MySQL? Then stay. Database expertise matters more than database features. A well-tuned MySQL with a team that knows it will outperform a poorly operated Postgres every time.

The honest take

Most of the “Postgres vs MySQL” content online bends over backward to be balanced. I won’t. For my workloads–financial data, mixed relational and document storage, search, reporting–Postgres isn’t a marginal winner. It’s the obvious choice.

MySQL is a fine database for simpler workloads and teams that know it well. But if you’re starting fresh in 2016 and your needs are anything beyond basic CRUD, pick Postgres. You will thank yourself when the first complex query lands and you don’t have to rewrite it as three subqueries and an application-side join.

The best database decision is the pragmatic one, not the tribal one. Pick the tool that does more of the work for you and stop arguing about logos.