The Myth of the Headline Price

Most teams compare API pricing to GPU rental and stop there. That is the wrong first-order model.

Token price is the easiest number to quote and the least useful number to trust. The real bill shows up in the work around the model:

• Telemetry & Evals: If you self-host, you must build the pipeline that captures, scores, and reviews output. Vendor APIs may bundle some of this, but then they also own the metadata.
• Graceful Degradation: When the provider throttles you at peak, do you have local fallback? Hybrid systems buy resilience, but they also add systems-engineering work.
• Data Sovereignty : Sometimes the reason to build is simple: the data cannot legally leave your VPC. Once that is true, the token price stops mattering.

When to Buy (The Commodity Highway)

Buy when the AI capability is a feature, not the product.

If you are building an internal documentation chatbot, a support-ticket summarizer, or a semantic search overlay, buy the API. Do not spend engineering throughput standing up vLLM instances and chasing KV-cache optimizations for a problem that is not your moat.

The catch is lock-in at the integration layer. If your code imports vendor-specific classes directly, you will feel the squeeze when prices change or a model line is deprecated. Keep the provider behind an internal interface .

When to Build (The Crucible of Control)

Build when AI sits inside unit economics or inside a hard trust boundary.

You must build if:

1. Your margins depend on it. Billions of tokens a day can make the API tax the difference between a healthy product and a broken one.
2. You operate under zero-trust or residency constraints. In parts of healthcare, finance, and defense, the data cannot touch a multi-tenant cloud edge.
3. You need hardware-level optimization. Sub-150ms tail latency usually means quantization, attention fusion, and serious control over the runtime.

That is the part teams underestimate. You are no longer building a prompt pipeline. You are operating a distributed, heavily constrained state machine. That takes engineers who understand memory bandwidth, not just prompting.

The Hybrid Default

The mature pattern in 2026 is a barbell.

Buy frontier models for complex reasoning, planning, and high-context zero-shot tasks. Build or host quantized, heavily tuned 8B models for the large volume of routing, formatting, and classification work that sits underneath the product.

The CTO’s job is to make the handoff between buy and build a config change instead of a rewrite. Picking a camp is the easy part.

The Execution Path Nobody Drew on a Whiteboard

Agent tasks fan out across DNS resolution, TLS handshakes, token exchanges, service mesh routing, and backend queries. The multi-hop latency problem is well-understood (I covered the general case in the cloud-heavy architecture post ), but the networking-specific failure modes deserve their own treatment: stale DNS caches that route agents to decommissioned endpoints, TLS handshakes that repeat across 40 tool calls when connections aren’t reused, service mesh sidecars that add milliseconds per hop invisibly (more at the tail), and queue depth limits that silently drop requests during agent-scale bursts. All of them surface as agent unreliability.

The Hidden Cost of 20th-Century Network Assumptions

Most enterprise networks were designed around two assumptions: traffic flows north-south through a perimeter, and anything inside the perimeter is trusted. AI agents violate both assumptions simultaneously.

Agent traffic is east-west by default. A single task might call an internal knowledge base, a code execution sandbox, an external search API, and a database, all in a single reasoning loop. The traffic pattern looks like a mesh, not a pipeline. Networks designed for request-response patterns between a frontend and a backend choke on this.

The trusted-network assumption is worse. When an agent has a service account with broad permissions, every tool call inherits those permissions. If the agent can read from a document store, it can read from all of it. If it can write to a database, the blast radius of a prompt injection extends to every table the service account can touch. This isn’t a theoretical risk. It’s the default configuration in most deployments I’ve seen.

Latency compounds differently for agents than for traditional services. A human user tolerates 200ms of added latency on a page load. An agent making 40 tool calls in a single task turns 200ms of unnecessary overhead per call into 8 seconds of total delay. At scale, this means the difference between an agent that completes tasks in seconds and one that takes minutes. Users notice. They lose trust. They stop using the feature.

Zero-Trust Identity for Autonomous Systems

You don’t need a network redesign. You need an identity redesign at the network layer.

Every agent tool call should carry a scoped identity that specifies what the agent can reach, for how long, and on behalf of which user or task. This is standard zero-trust thinking applied to agent traffic patterns. (For the broader tool permission and output validation side of this, see my earlier post on AI security .)

In practice, the networking-specific concerns are:

Per-task credentials with network scope. Instead of a long-lived service account, mint a short-lived token for each agent task. The token carries the minimum permissions needed for that specific workflow, and critically, it limits which network endpoints the agent can reach. When the task ends, the token expires. If the agent is compromised mid-task, the blast radius is one task’s worth of permissions and one task’s set of reachable services.

Per-call authentication overhead. Every tool call crossing a network boundary needs auth, and that auth has a cost. TLS mutual authentication, token validation, and policy lookup all add latency. The design tradeoff is between granular identity (every call authenticated independently) and performance (connection pooling, session tokens, cached auth decisions). Get this wrong and your zero-trust layer becomes the latency bottleneck it was meant to protect against.

Network segmentation per agent class. Not all agents need the same network access. An agent that summarizes documents has no business reaching your billing API. Segment your network so each agent class can only route to the services it needs. This is basic network segmentation, but most teams skip it because their agents all share one service account with broad network access.

Reliability Engineering for Agent Workflows

Traditional reliability patterns need adjustment for agentic workloads. The standard toolkit, retries, timeouts, circuit breakers, still applies, but the parameters and placement change.

Timeouts need to be per-step, not per-request. An agent task might legitimately run for 30 seconds across 20 tool calls. A global timeout of 30 seconds will kill valid workflows. A per-step timeout of 3 seconds will catch hung dependencies without killing the task.

Retry logic needs backpressure awareness. An agent that retries a failed tool call immediately, while 50 other agent instances are doing the same thing, creates a retry storm that takes down the dependency. Exponential backoff with jitter is the minimum. Better: a circuit breaker that trips after a threshold and fails fast for all agent instances, with a clear error message the model can reason about.

Queue depth matters more than you think. Agent workloads are bursty. A user action that triggers 10 agent tasks, each making 15 tool calls, puts 150 requests into your service mesh in seconds. If the target service has a queue depth of 50, you’re dropping requests before the agent even knows there’s a problem. Size your queues for agent-scale fan-out, not human-scale request rates.

Graceful degradation over hard failure. When a tool call fails, the agent should get a structured error it can reason about, not a 500 or a timeout. “Knowledge base unavailable, try alternative approach” is actionable. A raw HTTP error is not. Design your tool contracts to return machine-readable failure modes.

Observability for Agent Decision Traces

Standard APM tools show you request latency and error rates. For agent workflows, you need something more: a trace that follows the agent’s reasoning across tool calls, captures the decision points, and shows why the agent chose one path over another.

This means correlating model inputs, outputs, and tool calls into a single trace. Each agent task gets a trace ID. Each tool call within that task gets a span. The spans include the tool arguments, the response, the latency, and the policy decision. When you look at a slow or failed agent task, you can see exactly which step took too long, which dependency failed, and whether the agent’s retry behavior made things better or worse.

The teams doing this well treat agent traces like they treat database query plans. They review them regularly, look for patterns, and optimize the hot paths. A tool call that takes 500ms and gets called 20 times per task is a bigger problem than a tool call that takes 2 seconds but only gets called once.

Migration Path

You don’t need to rebuild your infrastructure to start.

1. Instrument first. Add trace IDs to agent tool calls. Log latency, errors, and retry counts per step.
2. Add identity boundaries. Replace long-lived service accounts with per-task tokens, starting with agents that have write access.
3. Circuit-break external calls. Add circuit breakers and per-step timeouts for every external dependency. Size queues for agent-scale fan-out.
4. Migrate to mesh. Deploy a service mesh or policy layer for tool call routing. Start in audit mode, then shift to enforcement.

Each step is small and reversible. Together they compound into a fundamentally more reliable agent platform.

Checklist: Risk Reduction in 90 Days

• Map every tool an agent can call, its permissions, and its failure modes
• Add per-task trace IDs to all agent tool calls
• Replace at least one long-lived service account with scoped, short-lived tokens
• Set per-step timeouts on all agent tool calls
• Add circuit breakers for external API dependencies
• Deploy a policy layer in audit mode for tool call authorization
• Review agent decision traces weekly for latency outliers and retry storms
• Load test agent workflows at 10x expected concurrency
• Document failure modes and give agents structured error responses
• Establish an error budget for agent reliability separate from service reliability

Teams that keep tuning prompts to work around infrastructure problems will keep wondering why their agents are “flaky.” Network and identity design is core agent product work. Budget for it accordingly.

The Cloud-Heavy Anti-Pattern

The standard agentic stack looks like this: application code in one cloud region calls a model API in another, pulls context from a vector database in a third, and writes results back through a gateway that adds its own hop. Every step crosses a network boundary. Every boundary adds latency variance, failure surface, and cost.

For a single inference call, the overhead is tolerable. For an agent that chains ten to fifty calls per task, with tool use, retrieval, and self-correction loops, the overhead compounds. A p50 latency of 200ms per hop becomes 2-10 seconds of pure network time on a moderately complex agent run. At p99, you’re looking at timeouts and retries that double or triple your effective cost.

The measurable symptoms are consistent across teams:

• Latency variance dominates execution time. The model itself is fast. The network between your orchestrator and the model, plus the hops to retrieval and tool services, is where time disappears.
• Cost scales with hops, not intelligence. You pay for every round trip: egress, ingress, token overhead from context reassembly, and retry loops when any hop fails.
• Failure modes are combinatorial. When five services must all be healthy for one agent task to complete, your effective availability is the product of their individual availabilities. Five nines times five is not five nines.

None of this argues against cloud. It argues against cloud-only, cloud-default architecture for workloads that don’t need it.

Consolidating Runtime Layers

The fix is straightforward: move compute closer to the data and the user. Consolidate runtime layers so agent orchestration, context retrieval, and lightweight inference happen in the same process or at least on the same machine.

This is not a new idea. Databases figured this out decades ago. You don’t run your query planner in a different availability zone from your storage engine. Agentic systems are hitting the same lesson: when the workload is latency-sensitive and involves tight feedback loops, co-location wins.

In practice, consolidation means running a local inference server for small models (classification, routing, extraction), keeping your retrieval index on the same node as your orchestrator, and reserving cloud API calls for frontier-model tasks that actually need them. The local layer handles the high-frequency, low-complexity work. The cloud layer handles the hard problems.

The cost difference is significant. Handling routine calls locally with a quantized model on commodity GPU hardware removes most of the per-request metering from a typical agent run. The frontier API cost doesn’t disappear, but it shrinks because you’re only sending it the work that justifies the price.

Cloud-Only vs. Hybrid Cost Envelopes

The math depends on workload shape, but the pattern is consistent.

Cloud-only architectures have variable cost that scales linearly with usage and offers no marginal improvement at volume. You pay the same per-token rate whether you run one task or a million. Egress fees, retry overhead, and context window waste compound on top.

Hybrid local-first architectures have a higher fixed cost (hardware, setup, maintenance) but dramatically lower marginal cost. Once the local inference server is running, the incremental cost of a routing decision or an extraction call is effectively zero. You’re paying for electricity and depreciation, not per-request metering.

The crossover point arrives faster than most teams expect. For workloads above a few thousand agent tasks per day, local-first is cheaper within months, not years. Below that threshold, cloud-only is simpler and the cost premium is manageable.

The latency picture is even more decisive. Small classification and routing models on a mid-range local GPU answer in milliseconds, with no network variance on top. For agent loops that make dozens of calls per task, local inference can cut total wall-clock time by 60-80%.

Where Systems Languages Matter

Agent runtimes written in Python work fine for prototyping and low-throughput production. But as you move inference and orchestration onto local hardware, you start caring about memory predictability, startup time, and per-request overhead in ways that an interpreted runtime like Python doesn’t handle well.

Rust is showing up in this layer for practical reasons. It gives you memory safety without garbage collection pauses, which matters when you’re serving inference requests with tight latency budgets.

You don’t need to rewrite your application in a systems language. The candidates are the runtime layer: the inference server, the orchestration loop, the retrieval engine. These are the hot paths where predictable performance translates directly into cost savings and reliability. The application logic on top can stay in whatever language your team knows.

The practical signal: if your agent runtime’s p99 latency is dominated by GC pauses or memory allocation overhead rather than actual inference time, a systems-language runtime will help. If inference time dominates, the language doesn’t matter.

Adoption Without Full Rewrites

Teams with existing cloud-heavy architectures don’t need to rip and replace. The migration is incremental and each step produces measurable improvement.

Step 1: Instrument and classify. Before moving anything, measure what your agent stack actually does. Break down time and cost by call type: routing decisions, context retrieval, small-model inference, frontier-model inference. Most teams discover that the bulk of calls are routine work that doesn’t need a frontier model or a cloud round trip.

Step 2: Add a local inference tier. Deploy a quantized model locally for the routine calls you identified. Route classification, extraction, and simple generation through it. Keep the cloud API as the escalation path. This is a routing change, not a rewrite.

Step 3: Co-locate retrieval. Move your vector index or retrieval layer onto the same infrastructure as your orchestrator. This eliminates the retrieval round trip, which is often the single largest latency contributor after model inference.

Step 4: Evaluate and tighten. With local tiers in place, measure again. Adjust routing thresholds. Identify the next tier of work that can move local. Each iteration reduces cloud dependency and improves predictability.

The entire migration can happen alongside normal feature work. No flag days, no cutover weekends.

Governance and Data Residency

Local-first architecture has a governance benefit that’s easy to overlook: your data stays on your infrastructure. For teams operating under GDPR, HIPAA, or sector-specific data residency requirements, this simplifies compliance significantly.

When agent tasks process user data through a cloud API, that data traverses networks you don’t control and resides, however briefly, on infrastructure you don’t own. The compliance burden of documenting, auditing, and risk-managing that data flow is real and growing. Local inference eliminates the flow entirely for tasks that don’t require cloud escalation.

This doesn’t mean you avoid cloud APIs altogether. It means you have architectural control over which data leaves your perimeter and which doesn’t. That’s a better conversation to have with your compliance team than “everything goes to a third-party API.”

Decision Rubric

When deciding how to place compute for agentic workloads, ask these questions:

1. Volume: Are you running more than a few thousand agent tasks per day? If yes, the economics of local inference likely favor hybrid.
2. Latency sensitivity: Do your agent loops involve more than ten chained calls? If yes, network overhead is probably your bottleneck.
3. Data sensitivity: Does your agent process PII, health data, or regulated information? If yes, local-first reduces compliance surface.
4. Team capability: Do you have infrastructure engineers who can operate local GPU servers? If no, start with managed options or cloud-based inference with a clear migration path.
5. Workload predictability: Are your traffic patterns bursty or steady? Bursty workloads benefit most from local capacity that handles baseline load with cloud burst for peaks.

Common Traps

• Over-investing in local hardware before measuring workload shape. Instrument first. Buy hardware based on data, not intuition.
• Treating local and cloud as either/or. The right answer is almost always hybrid. The question is where to draw the line.
• Ignoring operational cost of self-hosted infrastructure. Local inference is cheaper per request but requires someone to keep it running. Factor in ops time.
• Optimizing for p50 when p99 is what breaks your SLA. Agentic workloads are chains. One slow hop at p99 delays the entire task.

Hardware placement is a first-order architecture decision. Make it early, measure it continuously, and adjust as your workload evolves.

I’ve been building data pipelines since before anyone called them “data pipelines.” At the fintech startup we were ingesting financial data from hundreds of sources, normalizing it, and serving it for real-time retrieval. That was 2017. The core problems haven’t changed.

What has changed is that your pipeline now has a second consumer: a retrieval system feeding an LLM. If you treat that consumer as an afterthought, your AI product will deliver confidently wrong answers. Ruthless focus on the basics separates pipelines that work from pipelines that demo well.

The Shape of an AI Pipeline

Every AI pipeline I’ve seen in production boils down to six stages. The skeleton:

pipeline:
  stages:
    - name: extract
      # Pull from sources, normalize formats
      # PDF, HTML, API responses -> clean markdown or structured text

    - name: diff
      # Hash-based change detection
      # This is the stage most teams skip. Don't.

    - name: chunk
      # Split by document structure first, token count second
      # Preserve section boundaries and headings

    - name: embed
      # Generate vectors using a pinned model version
      # Log the model version. You will need it later.

    - name: index
      # Upsert with stable IDs and rich metadata
      # source_id + chunk_position = deterministic ID

    - name: verify
      # Check for missing chunks, stale entries, orphans
      # Alert on drift from expected source freshness

Nothing exotic. The magic is in the discipline of each stage, not in clever architecture.

The Diff Step Is Everything

Most teams skip change detection and reprocess everything on every run. At small scale, this is fine. At production scale, it’s expensive, noisy, and makes debugging a nightmare.

A simple content-hash approach works well:

func hasChanged(sourceID string, content []byte, store HashStore) bool {
    newHash := sha256.Sum256(content)
    existing, found := store.Get(sourceID)
    if !found {
        store.Set(sourceID, newHash)
        return true
    }
    if existing != newHash {
        store.Set(sourceID, newHash)
        return true
    }
    return false
}

When I built the ingestion pipeline at the fintech startup, adding a diff layer cut downstream processing costs by roughly 60%. Most sources don’t change on most runs. Detecting that early saves everything downstream.

The diff step also gives you auditability. You can answer “what changed and when” instead of shrugging at a vector store that silently drifted.

Chunking: Structure Before Size

This is where most RAG pipelines go wrong. Teams reach for a token-count splitter because it’s the default in every tutorial, then wonder why retrieval returns fragments of ideas instead of coherent answers.

Split by document structure first. Headings, sections, code blocks, and list items are natural semantic boundaries. Only fall back to token-count splitting when a single section exceeds your context window.

def chunk_by_structure(doc: Document) -> list[Chunk]:
    chunks = []
    for section in doc.sections:
        if section.token_count <= MAX_CHUNK_TOKENS:
            chunks.append(Chunk(
                content=section.text,
                metadata={
                    "source_id": doc.id,
                    "section_heading": section.heading,
                    "position": section.index,
                    "doc_version": doc.version,
                },
                # Deterministic ID: no duplicates on re-ingestion
                id=f"{doc.id}:{section.index}",
            ))
        else:
            # Fall back to sliding window only for oversized sections
            for sub in sliding_window(section, MAX_CHUNK_TOKENS, overlap=100):
                chunks.append(Chunk(
                    content=sub.text,
                    metadata={**section.metadata, "sub_position": sub.index},
                    id=f"{doc.id}:{section.index}:{sub.index}",
                ))
    return chunks

Two things matter here. First, the id is deterministic, derived from source and position, not random. This means re-ingesting the same content produces upserts, not duplicates. Second, metadata travels with every chunk. When retrieval returns a chunk, you know exactly where it came from, which version, and which section.

I can’t overstate how many production RAG systems I’ve reviewed where chunks had no stable ID. Every reindex created duplicates. Users got the same passage three times in their context window, and the model hallucinated a consensus that didn’t exist.

Freshness Is an Operational Problem

Your pipeline isn’t done when it runs once. Sources change, APIs update, and documents get deleted. If your index doesn’t reflect reality, your AI lies with confidence.

Three rules I enforce on every pipeline:

1. Reindex on embedding model changes. If you swap or upgrade your embedding model , every existing vector is stale. This is a full reindex event. No exceptions. Pin your model version and log it.

2. Purge on source deletion. If a document disappears from the source, its chunks must disappear from the index. Orphaned chunks are a retrieval poison pill.

3. Alert on freshness drift. Every source has an expected update cadence. If your financial data feed hasn’t updated in 6 hours, something is wrong. Don’t wait for a user to notice.

freshness_policy:
  sources:
    - name: product_docs
      expected_interval: 24h
      alert_after: 36h
    - name: api_changelog
      expected_interval: 7d
      alert_after: 10d
    - name: support_kb
      expected_interval: 48h
      alert_after: 72h
  on_embedding_change: full_reindex
  on_source_delete: purge_chunks

The Mistakes I Keep Seeing

Across the pipelines I’ve built and reviewed, in telecom and elsewhere, the failure pattern is remarkably consistent:

No stable IDs. Updates create duplicates. Retrieval returns the same content multiple times. The model treats repetition as emphasis and doubles down on whatever it found.

Token-count-only chunking. A paragraph about authentication gets split mid-sentence. The first half lands in one chunk, the second half in another. Retrieval finds the first half. The model confidently gives half an answer.

Ad-hoc reindexing. Someone runs a reindex on a Friday afternoon. Nobody knows what changed. Retrieval quality shifts. The team argues about whether it got better or worse. No one can prove either way because there’s no baseline.

Missing permission metadata. The chunks are indexed without access control data. A user with restricted access asks a question and gets an answer sourced from documents they shouldn’t see. This is a compliance incident waiting to happen.

It doesn’t. It’s the same infrastructure engineering we’ve been doing for decades, applied to a workload that happens to involve model inference. The latency problems are the same. The cost problems are the same. The reliability problems are the same. And the solutions are the same.

And yet every week I review a team’s architecture and find they’ve reinvented service meshes, badly, because they assumed AI needed something different.

The Demo-to-Production Gap Is Infrastructure

Here’s what happens: a team builds a demo. It works great at one request per minute. Then real traffic arrives and everything falls apart. Latency spikes. Costs explode. The system goes down when the provider rate-limits them.

None of these are AI problems. They’re infrastructure problems that we solved years ago in every other context. The teams that scale AI successfully are the ones that apply those solutions without reinventing them.

Put a Gateway in Front. Please.

I’m genuinely baffled by how many production AI systems I see where every service calls the model provider directly. No centralized routing. No rate limiting. No budget enforcement. No observability.

This is like building a web application in 2024 without a load balancer. Nobody would do that. But somehow AI gets a pass.

A gateway (call it a broker, proxy, or control plane, whatever you want) does the boring work:

• Routes requests to the right model based on task type
• Enforces rate limits and budgets per user, per feature, per environment
• Caches deterministic responses
• Provides a single point for observability and tracing
• Handles provider failover when one API goes down

You can build a basic version in a day: a YAML config and a reverse proxy. It doesn’t need to be fancy. It needs to exist.

Separate Your Workloads

Interactive requests and batch processing shouldn’t share the same execution path. I keep saying this, and teams keep ignoring it until interactive latency tanks because a batch job saturated the rate limit.

Interactive work gets tight latency budgets and priority access. Batch work gets queued and retried patiently. The split is trivial to implement and painful to retrofit after the fact.

Cache Every Deterministic Request

If you’re sending the same prompt with the same inputs to the same model and not caching the response, you’re burning money. Literally.

Exact-match caching for deterministic requests is table stakes. Similarity-based caching for near-duplicate requests is a bonus. Even a simple TTL-based cache with invalidation on prompt updates can cut costs significantly.

One team was spending $40k/month on model inference. After adding exact-match caching for their classification pipeline, it dropped to $15k. Same outputs. Same quality. Less waste.

Cost Controls Aren’t Optional

“We’ll optimize costs later” is the AI equivalent of “we’ll add tests later.” You won’t. And when the bill arrives, it becomes an emergency.

Budget enforcement belongs in the gateway. Hard caps with clear error messages. Soft limits that degrade to cheaper models or slower paths. Per-user and per-feature attribution so you know where the money goes.

I’ve seen teams discover that a single feature was responsible for 70% of their AI spend because nobody was tracking attribution. The feature wasn’t even high-value. It was just chatty.

Reliability Isn’t Heroics

Retry with backoff. Circuit breakers. Graceful degradation. Provider failover.

This is baseline production engineering. If your AI system doesn’t have them, it isn’t production-ready. It’s a demo with a billing account.

Graceful degradation is a product decision, not an ops feature. If the full response is unavailable, a simpler response or a cached response or even a “try again in a moment” is better than an error page. Design for this upfront. Don’t bolt it on during an incident.

The gateway, the cache, budget enforcement, workload separation, circuit breakers: none of it is novel. All of it is necessary. Your SRE team already knows how to do this. Let them.

We have models that can write poetry, generate code, and analyze images. We don’t have enough GPUs to run them reliably. We don’t have pricing that makes sense at scale. And we definitely don’t have the operational maturity to treat these systems like the production dependencies they have become.

I’ve spent December watching AI features I helped build run into every scaling problem distributed systems teams have been solving for twenty years. Rate limits. Cascading failures. Cost explosions. Latency spikes. The problems aren’t new. The industry is just re-learning them with a fresh coat of hype.

The GPU Situation Is Absurd

You can’t get H100s. You can’t reliably get inference capacity from any major provider unless you sign a months-long commitment or an enterprise contract that costs more than most startups raise in a seed round. The entire industry is building products on top of infrastructure that’s supply-constrained, and nobody wants to talk about what happens when demand doubles next year.

I tried to reserve inference capacity for a production workload last month. The response from one provider was “we can put you on a waitlist.” A waitlist. For compute. In 2023. It’s a supply chain problem wearing a technology costume.

Rate Limits Are a Production Constraint

Every AI API has rate limits. At low volume, you don’t notice them. At production scale, they become the hardest ceiling in your architecture.

I hit OpenAI’s rate limit during a load test and watched requests queue up until the entire feature became unusable. Not degraded. Unusable. The fix wasn’t clever engineering. It was a priority queue, backpressure, and load shedding. Distributed systems 101. The fact that most AI teams are learning this for the first time worries me.

Your Demo Won’t Survive Real Traffic

When your AI feature goes from 100 requests per day to 10,000:

Latency goes from “acceptable” to “users are closing the tab.” Costs go from “rounding error” to “someone just Slacked asking why the API bill tripled.” A provider outage that used to affect a handful of test users now takes down a production feature that the sales team just promised to a client.

I’ve seen all three of these happen at the same company. In the same month.

The Patterns That Hold Up

Queues and backpressure. Treat your AI traffic as a managed stream, not an open pipe. Priority queues for critical requests. Backpressure when the system is saturated. Load shedding for low-priority work. This isn’t optional once you have real users.

Circuit breakers. Your model provider will have bad hours. Mine had a bad day last week. Circuit breakers stop a provider outage from cascading through your entire system. They’re boring. They’re essential. I’ve been building systems with circuit breakers for years, including at a large telecom company I worked with. The pattern hasn’t changed. The dependency has.

Graceful degradation. When GPT-4 is down, what happens? If the answer is “the feature breaks,” you don’t have a production system. You have a demo with users. Fall back to cached responses. Fall back to a smaller, faster model . Fall back to a static message that says “this feature is temporarily unavailable.” Anything is better than a spinning loader.

Cost controls that are actually enforced. Per-tenant budgets. Per-feature budgets. Daily caps. If you don’t enforce them, you’ll get a surprise invoice that triggers an emergency meeting. I’ve seen a single prompt change (adding two paragraphs of context) increase monthly costs by 35%. Token pricing is deceptively simple until you multiply it by production volume.

Caching. Exact-match caching is trivial to implement and saves real money. Same question, same context, same answer: serve it from cache. Semantic caching is fancier and worth exploring, but start with the easy wins.

This Is Distributed Systems Work

None of this is novel. Queues, circuit breakers, graceful degradation, cost controls, caching: these are patterns from every distributed systems textbook ever written. The only thing that’s new is the dependency type.

What frustrates me is that the AI community is treating infrastructure as a solved problem while building on top of infrastructure that’s anything but solved. The models are impressive. The plumbing is held together with optimism and rate limit retries.

Build your AI features like you would build any production system that depends on an unreliable, expensive, supply-constrained external service. Because that’s exactly what it is.

What Is Actually Stored

An embedding model turns text, images, or structured data into vectors: arrays of floating-point numbers that encode meaning. “Cat” and “kitten” end up close together in vector space. “Cat” and “quarterly earnings report” don’t.

A vector database stores these vectors and lets you find the closest ones to a query. That’s the core primitive: similarity search. Traditional databases give you exact matches, range queries, and transactions. Vector databases give you “find me the things most similar to this.”

type Document struct {
    ID        string            `json:"id"`
    Content   string            `json:"content"`
    Embedding []float64         `json:"embedding"`
    Metadata  map[string]string `json:"metadata"`
}

type SearchResult struct {
    Document Document
    Score    float64
}

That’s the data model. A document, its vector, and some metadata. Everything else is indexing and retrieval strategy .

Similarity Metrics

“Closest” needs a definition. The two common choices:

Cosine similarity measures the angle between vectors, ignoring magnitude. Good when you care about direction (meaning) and not length. Most text embedding models are trained with this in mind.

Dot product considers both direction and magnitude. Useful when the embedding model encodes importance in vector length.

Pick the metric that matches your embedding model’s training objective. The wrong choice can look fine on a small test set and produce garbage results at scale. I’ve seen this firsthand: a team using dot product with a model trained for cosine, wondering why their search quality was inconsistent.

func cosineSimilarity(a, b []float64) float64 {
    var dot, normA, normB float64
    for i := range a {
        dot += a[i] * b[i]
        normA += a[i] * a[i]
        normB += b[i] * b[i]
    }
    if normA == 0 || normB == 0 {
        return 0
    }
    return dot / (math.Sqrt(normA) * math.Sqrt(normB))
}

The Deployment Decision

There’s no universal answer. It depends on your constraints:

Extend your existing database. PostgreSQL with pgvector, for example (it currently offers exact search and IVFFlat indexes). Good when vectors are one part of a larger transactional system. You get joins, transactions, and familiar operations alongside similarity search. The tradeoff is that you’re bolting a new capability onto infrastructure not designed for it. Fine for moderate scale. Starts to hurt when similarity search becomes the dominant workload.

Use a dedicated vector database. Pinecone, Weaviate, Qdrant, Milvus. Good when similarity search is central and you need low latency at scale. These systems are built for this workload. The tradeoff is another piece of infrastructure to operate, another data pipeline to maintain.

Go managed. Pinecone or similar. Good when operational overhead is a bigger risk than vendor lock-in. You pay more per query but you don’t page at 3am because the index rebuild failed.

For most teams I talk to, the answer is: start with pgvector . It’s good enough for most workloads, you already know how to operate Postgres, and you avoid adding a new system to your stack until you actually need to. Move to a dedicated solution when latency or scale forces your hand.

Indexing: Exact vs. Approximate

Exact search compares the query against every vector. Accurate, simple, and slow as your data grows. At 10,000 vectors it’s fine. At 10 million it isn’t.

Approximate nearest neighbor (ANN) indexes trade a small amount of accuracy for much better performance. HNSW is the most common algorithm. It builds a navigable graph that lets you skip most comparisons while still finding results that are close to the true nearest neighbors.

type IndexConfig struct {
    Type       string // "flat", "hnsw", "ivf"
    Dimensions int
    Metric     string // "cosine", "dot", "l2"

    // HNSW-specific
    EfConstruction int // build-time quality (higher = slower build, better recall)
    EfSearch       int // query-time quality (higher = slower query, better recall)
    M              int // connections per node (higher = more memory, better recall)
}

The parameters matter. EfConstruction and M control the build-time quality-speed tradeoff. EfSearch controls the same tradeoff at query time. Treat these as part of your performance budget. Tune them with your actual data and actual queries, not defaults from a blog post.

Chunking Strategy

Most text is too large to embed as a single vector. You need to chunk it into pieces that are small enough to be meaningful but large enough to carry context.

type ChunkConfig struct {
    MaxTokens int
    Overlap   int
    Separator string
}

func chunkDocument(content string, cfg ChunkConfig) []Chunk {
    sentences := splitSentences(content)
    var chunks []Chunk
    var current []string
    currentLen := 0

    for _, s := range sentences {
        sLen := tokenCount(s)
        if currentLen+sLen > cfg.MaxTokens && len(current) > 0 {
            chunks = append(chunks, Chunk{
                Text:  strings.Join(current, " "),
                Index: len(chunks),
            })
            // Keep overlap
            overlapStart := len(current) - cfg.Overlap
            if overlapStart < 0 {
                overlapStart = 0
            }
            current = current[overlapStart:]
            currentLen = tokenCountSlice(current)
        }
        current = append(current, s)
        currentLen += sLen
    }

    if len(current) > 0 {
        chunks = append(chunks, Chunk{
            Text:  strings.Join(current, " "),
            Index: len(chunks),
        })
    }

    return chunks
}

Key decisions: chunk size (I default to 256-512 tokens), overlap (10-20% prevents losing context at boundaries), and whether to chunk by sentences, paragraphs, or fixed token counts. Sentence-based chunking preserves meaning better. Fixed-token chunking is simpler.

Store metadata with every chunk: source document ID, position index, creation timestamp, access control labels. You’ll need all of these for filtering, ordering, and debugging.

Hybrid Search

Vector search is great at semantic matching. It’s terrible at exact matches. If a user searches for “RFC 7231” you want the exact document, not the three most semantically similar documents about HTTP specifications.

Combine vector search with keyword search. The simplest approach: run both, merge results, and let a weighted score decide the final ranking. More sophisticated approaches use reciprocal rank fusion or learn the weights from user behavior.

Don’t overthink the blending logic at the start. A simple weighted combination that you can adjust is better than a clever algorithm you can’t debug.

Reindexing

Embeddings go stale. Your data changes. Your embedding model gets updated. You need a reindexing pipeline that’s repeatable and safe.

Build it as a batch job that can run alongside the live index. Write to a new index, validate it against your eval set, then swap. Blue-green deployments for vectors. Nothing fancy, but it has to work reliably because a corrupted index is a corrupted search experience.

The Eval Loop

Before you ship, build a small eval set: 50-100 representative queries with known good results. Measure recall and latency. Then test again at production data volume. A vector search that works on 1,000 documents and breaks on 100,000 is a demo, not a product.

Track these numbers over time. Embedding model updates, data changes, and index configuration tweaks all affect quality. If you aren’t measuring, you’re guessing.

Most cloud cost problems are visibility problems. You’re paying for things you don’t know about. I made the longer version of this argument in October ; this is the short, practical one.

Step One: Know What You’re Paying For

Before optimizing anything, fix your tagging. I know. Nobody wants to hear that. It isn’t glamorous. But if you can’t break down cost by service and environment, you’re guessing.

In a recent engagement, we found that 30% of one organization’s compute spend was on non-production environments running 24/7. Development and staging instances that nobody used after 6pm, burning money overnight and on weekends. The fix (scheduling those environments to shut down outside business hours) took a week to implement and cut the monthly bill meaningfully.

You can’t fix what you can’t see.

The Easy Wins

These require no architectural changes, minimal risk, and work across every cloud provider:

Kill orphaned resources. Unattached EBS volumes, unused Elastic IPs, old load balancers pointing at nothing, snapshots from services that were decommissioned months ago. Every AWS account I’ve audited has thousands of dollars in monthly orphan costs.

Right-size underutilized compute. Pull two weeks of CPU and memory metrics. If your instances consistently sit at 10-15% utilization, drop a size. This is obvious but most teams never look at the data.

Schedule non-production environments. Dev and staging don’t need to run at 3am. Shut them down evenings and weekends. Automate the start/stop.

Apply storage lifecycle policies. Move cold data to cheaper tiers. Set expiration policies on logs. CloudWatch log groups default to never expiring. If nobody configured retention, you’re paying to store every log line forever.

Commitments: Be Careful

Reserved instances and savings plans can save real money on stable workloads. The trap is overcommitting. I’ve seen teams lock in three-year reservations right before a product pivot made half their infrastructure obsolete.

Commit only to what you’re confident will be steady for the commitment period. In late 2022, flexibility is worth more than a marginal discount.

The Recurring Habit

Cost optimization is a habit, not a project. Monthly review of trends and anomalies, clear ownership per service, and guardrails that catch new waste before it accumulates.

Make cost visible to the teams creating it. When engineers see the bill for their services, they make different decisions, because they have information they didn’t have before.

The goal is discipline, not austerity. Spending money on infrastructure is fine. Spending money on infrastructure you don’t need is waste. And in late 2022, waste is the first thing that gets cut, usually clumsily, by someone who doesn’t understand what the infrastructure does. Better to cut it yourself, surgically, while you still have the context.

By now, the pattern is familiar. You start with a few teams, each running their own pipelines and infrastructure. It works. Then you grow to twenty teams. Each one has reinvented service templates, built slightly different CI pipelines, and solved the same security problems in incompatible ways. Suddenly you have twenty bespoke deployment processes, none of which are documented well enough for anyone outside the team to understand.

That isn’t a failure of DevOps principles. It’s a scale problem, and platform engineering is the pragmatic fix. In 2020 I called platform engineering DevOps with a rebrand . At twenty teams, the rebrand earns its keep.

What Platform Engineering Actually Is

Strip away the conference talks and vendor pitches, and platform engineering is simple: a team that builds the paved road so product teams don’t have to pave their own.

The platform team builds opinionated defaults for the boring stuff: service scaffolding, deployment pipelines, observability, security baselines. Product teams use those defaults and focus on product work. If a team needs to deviate, they can, but they start from a working foundation instead of a blank page.

At a large consumer platform, the infrastructure team had elements of this approach. The teams that used the supported path shipped faster and had fewer incidents. The teams that went custom spent meaningful time on plumbing that had nothing to do with their actual domain.

The Golden Path Is a Contract

I like the “golden path” framing because it makes the deal explicit. Follow this path, and the platform team supports you end to end. Deviate, and you own the consequences.

The point is sustainability. A platform team can’t support infinite variation. Opinionated defaults reduce the support surface and let the platform team actually invest in quality instead of fighting fires across twenty different setups.

A good golden path covers the boring lifecycle: create a service, deploy it, observe it, roll it back. Make each step predictable and fast. If “create a new service” takes a developer two days of config and ticket filing, your golden path isn’t golden. It’s bureaucracy with a nice name.

Treat It Like a Product (Or Don’t Bother)

This is where most platform efforts fail. They build internal tooling with an infrastructure team’s mindset: here is the tool, use it, file a ticket if it breaks.

That doesn’t work. An internal platform needs product thinking. Who are the users? What workflows are painful? Where is the friction? What does adoption look like?

If developers avoid your platform, you have a product problem, and more training won’t fix it. Either the platform doesn’t solve a real pain point, or it creates more friction than it removes.

Adoption is the only metric that matters early on. Everything else (consistency, reliability, cost savings) follows from teams actually choosing to use the thing.

What Goes Into a Platform

Keep it focused. An internal platform typically needs three capabilities:

A developer entry point. Somewhere that surfaces service ownership, docs, and supported workflows. This can be a portal , a CLI, or even a well-maintained repo with templates. The format matters less than the discoverability.

Self-service provisioning. Environments, pipelines, databases, queues: the resources teams need to ship. If provisioning requires a ticket and a three-day wait, you haven’t solved the problem.

Shared guardrails. Observability, security controls, cost defaults. Things that every team needs but nobody wants to build from scratch. Bake them into the golden path so compliance is the default, not an afterthought.

Build vs. Buy

Build what encodes your internal standards and unique workflows. Buy what’s commoditized. Most organizations don’t need a custom CI system or a bespoke container registry. They do need custom service templates, deployment policies, and integration glue that ties everything together in a way that makes sense for their stack.

The real cost is the ongoing maintenance and integration work, more than the license fee. Every tool you adopt becomes a dependency you have to keep running.

The Team Matters

A platform team needs infrastructure depth, software engineering skill, and (this gets overlooked) the ability to write decent documentation. A platform nobody can figure out is a platform nobody uses.

Avoid the ticket-queue trap. If the platform team’s primary interface is a Jira board where developers file requests and wait, you’ve built an ops team with a fancier name. The whole point is self-service.

Platform engineering is DevOps growing up, acknowledging that “everyone does everything” doesn’t scale, and building the shared foundations that let teams move fast without reinventing the wheel every sprint.

I’ve managed cloud spend at startups and at enterprises, and I still treat the bill as a design document . The problems are always the same. The solutions are always boring. The industry has wrapped basic hygiene in jargon and created a cottage industry of dashboards, consultants, and conferences around it.

The Problem Isn’t Complicated

Cloud spend drifts for three reasons:

Nobody owns it. Engineering provisions resources. Finance sees the bill. Neither side can attribute cost to a service, team, or decision. When nobody owns a cost center, the cost center grows. Always.

Defaults are generous. Console presets and copy-pasted templates push you toward larger instances, higher throughput, and more redundancy than you need. Good for reliability. Bad for your invoice. An m5.xlarge picked “to be safe” for a service that uses 200m CPU is waste.

Data transfer is invisible. Most teams focus on compute and storage. Data transfer costs creep in silently. Cross-AZ traffic, cross-region replication, egress to the internet. At one company, data transfer was 30% of the monthly bill and nobody on the engineering team knew it.

Six Boring Fixes

Delete Things

Start here. Always start here.

Unused load balancers. Orphaned EBS volumes. Dev environments that have been running for six months with nobody logged in. Staging environments sized like production. Snapshots from 2020.

At every company I’ve consulted for, the first pass of deleting unused resources has cut the bill by 15-25%. No analysis required. No tooling required. Just look at what’s running and ask “does anyone use this?”

Tag Everything

Enforce tags at creation time. Team, service, environment, cost center. If a resource doesn’t have tags, it’s impossible to attribute costs. If it’s impossible to attribute costs, nobody owns them.

This isn’t a technology problem. It’s a policy problem. Terraform can enforce tags. AWS Service Control Policies can enforce tags. The blocker is usually organizational will, not tooling.

Right-Size With Data

Look at two weeks of CPU and memory utilization (the same approach works for Kubernetes requests ). If a service averages 10% CPU utilization, it’s oversized. Downsize it. If it averages 80%, leave it alone or optimize the code.

Don’t right-size based on a single metric snapshot. Don’t right-size based on a vendor tool’s recommendation without verifying it. I’ve seen recommendations that would have put a production database below its working set size.

Commit to the Floor, Not the Ceiling

Reserved instances and savings plans save money on stable baseline usage. The key word is “stable.” Commit to the minimum you’re confident you’ll use for the next year. Keep the variable portion on demand.

Over-commitment is a hidden risk. Product priorities shift. Services get decommissioned. A three-year RI on infrastructure you no longer need is a sunk cost.

Spot Instances for Batch Work

Batch jobs, CI runners, data pipelines: anything that can tolerate interruption. AWS advertises Spot discounts of up to 90% off on-demand prices. The catch is fault tolerance. If your batch job can’t resume after an interruption, spot isn’t for you.

Watch Data Transfer

Put services close to their data. Cache aggressively. Avoid cross-region traffic when same-region will do. Set up a VPC endpoint instead of routing through the public internet. These are small changes with disproportionate cost impact.

Why FinOps Annoys Me

The FinOps movement takes straightforward engineering practices (tagging, right-sizing, deleting waste, monitoring spend) and wraps them in a maturity model with phases like “Inform, Optimize, Operate.” It creates the illusion that cloud cost management requires a specialized function.

It doesn’t. It requires engineering teams that own their costs and a culture that treats waste as a bug. That’s it.

A dedicated FinOps team is a sign that your engineering organization has externalized a responsibility it should own. It’s the same antipattern as a separate “quality” team that owns testing: it lets everyone else stop caring.

No maturity model required. No certifications. No dedicated team. Just engineers who look at what they run and care about what it costs.

On day three, during a traffic spike, the container hit 4 GB. Kubernetes evicted it. The replacement pod landed on the same node, inherited the same traffic, and the cycle repeated. The node started thrashing. Other pods on that node, including a critical payment service, got evicted too. What started as one leaky container turned into a 45-minute partial outage.

The fix was three lines of YAML. The investigation took most of a day.

The One Thing You Need to Understand

CPU is compressible. When a container exceeds its CPU limit, Kubernetes throttles it. The process slows down but keeps running.

Memory isn’t compressible. When a container exceeds its memory limit, the kernel kills it. OOMKilled. The process restarts from scratch.

That asymmetry drives almost every resource management decision. CPU limits trade throughput for predictability. Memory limits are a hard safety net. Treat them differently.

Requests vs Limits

Requests tell the scheduler how much capacity to reserve. They’re a guarantee. The scheduler won’t place your pod on a node unless that capacity is available.

Limits tell the runtime when to intervene. CPU limits trigger throttling. Memory limits trigger OOMKill.

A cluster can overcommit limits: the sum of all limits can exceed the node capacity. It can’t overcommit requests. This is why requests matter more for scheduling stability.

Setting Them

Start with measurements, not guesses. If your monitoring shows a Go service using 150m CPU on average and 200 MB of memory, set requests near those values:

resources:
  requests:
    cpu: "200m"
    memory: "256Mi"
  limits:
    memory: "512Mi"

Notice: no CPU limit. For latency-sensitive services, CPU limits can cause throttling spikes even when the node has spare capacity. The CFS throttling behavior in Linux is well-documented and surprising: a service using 180m of CPU can still get throttled with a 200m limit because of how the quota is applied per scheduling period.

I default to: set CPU requests, skip CPU limits, always set memory limits. That is a change from what I recommended in 2020 . If you need CPU limits for noisy-neighbor isolation, set them generously: 2x to 3x the request.

Measuring and Iterating

Don’t set values once and forget them. Traffic patterns change. Code changes. Dependencies change.

Look at these metrics over at least two weeks:

• Container CPU usage (the actual usage, not the request)
• Memory working set (not RSS; working set is what Kubernetes uses for eviction decisions)
• Throttling events (container_cpu_cfs_throttled_seconds_total)

Set requests around the P50 usage. Set memory limits around the P99 plus a buffer. If you’re seeing throttling events on a latency-sensitive service, either raise the CPU limit or remove it.

Tools like VPA (Vertical Pod Autoscaler) can generate recommendations. Use them as input, not as autopilot. I’ve seen VPA recommendations that would have cut memory limits below the application’s startup footprint. Trust but verify.

QoS Classes

Kubernetes assigns a quality-of-service class based on how you configure requests and limits:

• Guaranteed: requests equal limits for both CPU and memory. Last to be evicted.
• Burstable: requests set but not equal to limits. Most common. Middle priority for eviction.
• BestEffort: no requests, no limits. First to be evicted. Never appropriate for production.

The team from my opening story ran BestEffort pods on shared nodes. When the node ran out of memory, the BestEffort pods died first, as designed. But the leaky pod kept landing back on the same node, and the pressure spread to the other pods running there.

Guardrails

Two cluster-level mechanisms prevent the “no resources specified” scenario:

LimitRange sets default requests and limits for containers that don’t specify their own. Set this on every namespace. The defaults don’t need to be perfect. They need to prevent BestEffort pods from accidentally running in production.

ResourceQuota caps the total resource usage in a namespace. This prevents one team from consuming the cluster. Set quotas per team namespace and review them quarterly.

At a large consumer platform, we enforced both. LimitRange provided sane defaults, and ResourceQuota kept growth predictable. Teams that needed more resources had to justify it, which had the side effect of surfacing inefficient services early.

The Three Failure Modes

OOMKilled. Memory limit is too low, or the application has a leak. Check the limit against actual usage. If usage grows linearly over time, it’s a leak: fix the code, not the limit.

CPU throttling. Shows up as latency spikes with available CPU on the node. The container is being throttled by CFS even though the node isn’t overloaded. Raise or remove the CPU limit.

Pending pods. Requests are too large for available capacity, or scheduling constraints (node selectors, taints, affinity rules) are too restrictive. Either reduce requests, add capacity, or relax the constraints.

All three are observable. All three are fixable. The usual problem is that nobody looks at the metrics until an incident forces it.

The Minimum

1. Set memory limits on every container. No exceptions.
2. Set CPU requests based on measured usage.
3. Apply LimitRange defaults on every namespace.
4. Apply ResourceQuota on team namespaces.
5. Monitor throttling, OOMKill events, and pending pods.
6. Review resource settings quarterly or after significant traffic changes.

Resource management isn’t glamorous work. It’s the kind of thing that only gets attention after an outage. But three lines of YAML and fifteen minutes of looking at Grafana dashboards can prevent the kind of cascading failure that ruins your weekend.

I’ve looked closely at platform teams in a number of large organizations. The pattern is depressingly consistent. There’s a “platform team” somewhere in the org chart. They have built some Terraform modules. Maybe a Jenkins pipeline or two. Possibly a Backstage instance that three people use.

Meanwhile, developers still open Jira tickets to get a database provisioned. Lead time from “I need an environment” to “I have an environment” is measured in days. Sometimes weeks.

The platform team is busy. They are just busy building the wrong thing.

The Five Levels

I use a maturity model when I assess platform teams. Not because I love frameworks, but because it gives me and the team a shared language for “here’s where you are” and “here’s where the payoff is.”

Level 1: Ad hoc. Everything is manual. Knowledge lives in people’s heads. Getting a new environment means finding the one person who knows how. Deployments happen on Tuesdays because that’s when the ops team has capacity. This is where most startups are and where some surprisingly large companies still live.

Level 2: Standardized. Infrastructure as code exists. CI pipelines exist. But the platform team gates everything. Developers can build, but they can’t deploy or provision without filing a request. The automation is real, but the bottleneck just moved.

Level 3: Self-service. Developers can provision environments, deploy services, and roll back on their own, within guardrails. This is where the platform starts paying for itself. Wait time drops. Consistency improves. The platform team stops being a ticket queue and starts being an enablement function.

Level 4: Product-minded. The platform has a roadmap based on user feedback. There are golden paths for common service types. An internal CLI or portal handles the 80% case. Progressive delivery is standard. The platform team runs NPS surveys and actually reads them.

Level 5: Optimized. Data drives changes to platform defaults. Automated remediation handles known failure modes. Cost, reliability, and security metrics improve continuously. This is rare. I’ve seen maybe two organizations here.

Where Most Enterprises Actually Are

Based on what I’ve seen in large, telecom-scale organizations, most enterprises are at Level 2. Some pockets are at Level 3. Almost nobody is at Level 4.

The gap between Level 2 and Level 3 is the most expensive gap in enterprise software delivery. It’s the difference between “we have automation” and “developers can actually use it without waiting.” Every day a developer spends waiting for provisioning is a day they aren’t shipping features. Multiply that by hundreds of developers and the cost is staggering.

Why Platform Teams Get Stuck

Three patterns I see repeatedly:

Building tools instead of workflows. A team builds a slick Terraform module for spinning up EKS clusters. Great. But developers still need to file a ticket to get the module run for them. The tool exists. The self-service doesn’t. The developer experience didn’t actually change.

Centralizing instead of enabling. The platform team takes on more ownership instead of giving ownership away safely. They become the bottleneck for every infrastructure change. They are drowning in operational work and have no time to build the self-service layer that would free them.

Ignoring their users. Platform teams that never talk to developers build platforms that developers avoid. I’ve walked into organizations where three competing shadow platforms exist because the official one didn’t solve actual problems.

How to Move Up

Level 1 to 2: Document and automate the basics

Write down how to get from code to production. Adopt infrastructure as code for core services. Get CI pipelines running. Define standard environments. This isn’t glamorous work, but it’s the foundation.

Level 2 to 3: Kill the ticket queue

This is the most impactful transition. Give developers templates and modules they can run themselves. Automate deployments with guardrails instead of gates that require human approval for every change. Bake observability in by default so developers don’t have to think about it. Clarify who owns what.

The goal is simple: a developer should be able to go from “I want to deploy” to “it’s deployed” in minutes, not days.

Level 3 to 4: Treat it like a product

Build an internal CLI or portal around the most common workflows. Publish golden paths and reduce unnecessary choices. Measure developer satisfaction. Run feedback sessions. Build a roadmap based on what developers actually need, not what the platform team thinks is interesting.

Level 4 to 5: Let data drive

Automate remediations for known incident types. Tune platform defaults based on real usage patterns. Track cost and performance per service. This is continuous improvement territory; it never ends.

The Assessment

When I assess a platform, I score six dimensions independently:

1. Infrastructure provisioning and environment consistency
2. Delivery workflows and release safety
3. Observability and operational feedback
4. Security controls and policy enforcement
5. Developer experience and self-service
6. Platform team operating model and ownership

Each one gets a 1-5. I don’t average them. I look at the lowest two scores because that’s where the biggest drag on delivery lives. A platform with great CI/CD but terrible provisioning still has developers waiting.

Metrics That Show It’s Working

Forget vanity metrics. These tell you if your platform is working:

• Lead time from commit to production. If this is longer than an hour for most services, something is broken.
• Time to provision a new environment. If this takes more than a day, your self-service isn’t actually self-service.
• Deployment frequency. Teams that trust their platform deploy more often.
• Developer satisfaction. Ask them. Regularly. Act on what they say.

The best platform I’ve seen wasn’t the one with the most tools. It was the one where a new developer could deploy their first change to production on day one. Everything else followed from that principle.

It’s an Organizational Problem

Most platform immaturity is organizational. The platform team doesn’t have a mandate to remove ticket queues. Leadership doesn’t treat developer experience as a first-class investment. The team is understaffed for the scope they’re expected to cover.

If you’re a platform engineer reading this and nodding, show this to your VP of Engineering. The conversation needs to happen at the level where headcount and priorities are decided. No amount of Terraform modules fixes a staffing problem.

Last month I looked at a company spending $47k/month on their Kubernetes clusters. Actual utilization across the board? 23%. They were paying for ghost capacity that nobody asked for and nobody noticed.

This isn’t unusual. I see it at almost every large company I look at. Kubernetes makes scaling trivially easy, which makes over-provisioning trivially easy too. Some developer copy-pasted resource requests from a Stack Overflow answer two years ago. Nobody questioned it. The cluster autoscaler dutifully spun up nodes to satisfy those fictional requests. The bill grew.

Why Your Bill Keeps Growing

Kubernetes schedules based on resource requests, not actual usage. Your cloud bill pays for the nodes backing those requests. If every pod requests 2 CPU cores but uses 0.3, you’re paying for 2 cores of node capacity per pod. Multiply that across hundreds of pods. Yeah.

The usual culprits:

• Copy-pasted resource blocks that become tribal defaults
• Memory requests inflated to “never get OOM-killed” levels, without anyone measuring actual usage
• CPU limits set defensively and never revisited
• Zero cost visibility per team or per service

Measure First. Seriously.

Before you touch a single resource request, get visibility into what you’re actually using.

Build a dashboard (Grafana, Kubecost, whatever) that shows:

• CPU requested vs CPU used at p95
• Memory requested vs peak working set
• Utilization broken down by namespace
• Nodes that can’t scale down because of inflated requests

The first time an engineering lead sees their team requesting 64 cores and using 11, the conversation changes fast.

The Cost Comparison

Here is what I typically see before and after right-sizing:

These are real numbers from a mid-sized company running about 200 microservices. Your mileage will vary, but the pattern holds.

Right-Size Resource Requests

Stop guessing . Use p95 CPU as your request baseline. Use peak memory working set plus a 15-20% buffer for memory requests. Set memory limits to protect against runaway processes. Drop CPU limits entirely for most services. They cause throttling that hurts latency more than it helps anything.

resources:
  requests:
    cpu: 250m      # based on p95 actual usage
    memory: 1Gi    # based on peak working set + buffer
  limits:
    memory: 2Gi    # hard cap for runaway protection
    # no CPU limit -- let it burst

Run VPA in recommendation mode first. It watches actual usage and suggests request values. Don’t let it auto-apply yet. Review the recommendations manually, make sure they make sense, then update your deployments. Build trust before you automate.

Fix the Node Layer

Right-sized pods on wrong-sized nodes still waste money. If you have a bunch of pods requesting 500m CPU and 512Mi memory, shoving them onto m5.4xlarge instances is terrible bin packing.

Mix your instance types. Use smaller instances for smaller workloads. Create separate node pools for workloads with different profiles (CPU-heavy vs memory-heavy vs general). Enable Cluster Autoscaler and let it remove nodes that are actually idle.

Autoscaling only works when requests are honest. Inflated requests mean the autoscaler sees “full” nodes everywhere and keeps adding capacity you don’t need.

Spot Instances: Free Money (Almost)

Spot instances are typically 60-70% cheaper than on-demand. The catch: they can be reclaimed with two minutes notice.

Good candidates for spot:

• Stateless services with 3+ replicas
• Batch jobs and queue processors
• Dev and staging environments (all of it)
• Anything that handles graceful shutdown

Bad candidates: single-replica stateful services, databases, anything where losing a node means losing data.

Use taints and tolerations to pin critical workloads to on-demand nodes. Spread spot across multiple instance types and AZs so reclamation doesn’t take out your entire fleet at once.

Guardrails That Stick

Optimization without guardrails is a one-time win. Things drift back within months.

Set up:

• ResourceQuota per namespace so no team can accidentally claim half the cluster
• LimitRange to enforce minimum and maximum resource requests
• Labels for team, environment, and service. You need these for cost allocation
• Monthly review cadence: look at the top 10 over-provisioned workloads, adjust, repeat

That last one matters most. Cost optimization isn’t a project. It’s a habit. A 30-minute monthly review catches drift before it becomes a $10k/month problem.

Cost Is an Ownership Problem

Most Kubernetes cost problems aren’t technical problems. They’re ownership problems. Nobody owns the bill . Nobody sees the bill broken down by team. Nobody gets asked “why are you requesting 8 cores for a service that peaks at 0.5?”

Fix the visibility and the accountability first. The technical optimization follows naturally.

I’ve watched teams cut their bills in half within two months just by making resource usage visible to the teams that own the workloads. No fancy tooling. No replatforming. Just data, ownership, and a monthly conversation.

I’ve seen zero trust from two very different angles. Taking part in cyber-defense exercises put me in environments where “trust nothing” wasn’t a buzzword. It was the baseline assumption. Everything was compartmentalized. Access was explicit, scoped, and audited. You didn’t get access to a system because you were on the right network. You got access because you had the right role, the right need-to-know, and the right authentication.

Then there was a large telecom company I worked with. It had a corporate VPN and the assumption that anything inside the VPN was trusted. Thousands of employees, contractors, and partners, all on the same flat network. When I asked about lateral movement controls, the answer was basically “we have firewalls at the perimeter.” The perimeter. With half the workforce remote.

That gap between defense-grade security thinking and typical enterprise reality is where zero trust lives.

What zero trust actually means

Strip away the vendor marketing and zero trust is four ideas:

Verify explicitly. Every request is authenticated and authorized. Not just at the front door, but at every service boundary. A valid network connection isn’t a valid credential.

Least privilege. Access is scoped to what you need, when you need it. Privileged access is temporary, auditable, and rare. No permanent admin tokens sitting in CI pipelines.

Assume breach. Design your architecture so that a compromised service or endpoint has limited blast radius. If an attacker gets into one service, they shouldn’t be able to reach everything.

No implicit trust. Internal traffic isn’t automatically trusted. The network is a transport layer, not a security boundary.

NIST SP 800-207 formalized this in 2020. It’s worth reading. The core idea: access decisions are policy-driven and tied to identity, device posture, and context rather than IP addresses.

Identity is the control plane

This is where implementation starts. Everything else depends on knowing who or what is making a request.

For humans: MFA everywhere. Non-negotiable. Ideally FIDO2/WebAuthn for phishing resistance. At minimum, app-based TOTP. SMS is better than nothing but barely. Centralize on one SSO provider so policy changes propagate everywhere.

At one telecom company, we found 14 different authentication systems. Some services used LDAP directly. Some had their own user databases. Two services still accepted basic auth over internal HTTP. Consolidating to a single identity provider took months but it was the foundation for everything else.

For services: Every service gets an identity. Short-lived credentials. Mutual TLS between services where possible. OAuth 2.0 / OIDC for API-to-API access. No long-lived static tokens.

The pattern I’ve been recommending:

Client -> Identity Provider -> Short-lived JWT -> Service A
Service A -> mTLS with SPIFFE identity -> Service B
Service B -> IAM role (scoped) -> Database

Each hop has its own authentication. Each credential is scoped and temporary. If Service A is compromised, the attacker gets Service A’s permissions, not Service B’s database credentials.

Segmentation: default deny

The network architecture piece that makes zero trust tangible. Every service-to-service connection is explicitly allowed. Everything else is denied.

In Kubernetes, this starts with NetworkPolicies :

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orders-service
spec:
  podSelector:
    matchLabels:
      app: orders
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api-gateway
      ports:
        - port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - port: 5432

The orders service can only receive traffic from the API gateway and can only talk to its own database. Nothing else. If an attacker compromises the orders service, they can’t pivot to the user service, the payment service, or anything else on the network.

This is basic stuff but I’ve seen it absent at companies with hundreds of engineers. The excuse is always “we’ll add network policies later.” Later never comes.

If you’re running a service mesh (Istio, Linkerd), you get mTLS and authorization policies on top of network segmentation. The mesh handles the identity and encryption at the transport layer. Your services don’t need to implement TLS themselves.

Device posture

With remote work as the default now, you can’t assume devices are on a managed network. Device trust needs to be part of the access decision.

Signals that matter:

• Is this a managed/enrolled device?
• Is the OS patched?
• Is disk encryption enabled?
• Is EDR running?

An unmanaged device connecting from an unusual location at 3am should get constrained access at best. Not blocked entirely (that breaks usability), but restricted to lower-sensitivity resources until additional verification happens.

At that same company, we implemented tiered access. Managed devices with current patches got full access. Unmanaged devices got read-only access to a subset of tools. Unknown devices got nothing except the enrollment portal.

The implementation path

This is how I recommend rolling it out. Four phases, roughly in order:

Phase 1: Inventory and visibility. You can’t secure what you don’t know about. Map your identities (human and service), your assets, and your data flows. Turn on logging for authentication and authorization decisions. This alone will surface problems you didn’t know you had.

Phase 2: Identity. Enforce MFA for all human access. Standardize on SSO. Establish service identities with short-lived credentials. Kill long-lived static tokens.

Phase 3: Segmentation. Implement default deny between services. Start with your most sensitive systems (payments, PII, auth) and expand outward. Network policies in Kubernetes, security groups in AWS, or service mesh authorization policies.

Phase 4: Continuous verification. Add device posture to access decisions. Implement risk-based re-authentication. Monitor for anomalous behavior. This phase never really ends; it’s ongoing tuning.

What companies get wrong

Buying a product. Zero trust is an architecture, not a SKU. I’ve seen companies spend millions on a “zero trust platform” and still have flat networks, long-lived credentials, and no service-to-service authentication. The product doesn’t help if the architecture doesn’t change.

Ignoring legacy. The oldest, crustiest systems are often the most sensitive. Leaving them outside the zero trust model because they “can’t support it” means your biggest risk is unprotected.

Breaking developer workflows. If the security controls are so painful that developers route around them (hardcoding credentials, disabling MFA on service accounts, punching holes in network policies), you’ve made things worse. Security that people circumvent is security theater.

VPN as a crutch. A VPN puts you on the network. That’s it. It’s not authentication. It’s not authorization. It’s not segmentation. I still see organizations where VPN access grants implicit trust to hundreds of internal services. That’s a perimeter model with extra steps.

Where to start

Zero trust is the right direction. The perimeter model is dead and pretending otherwise is dangerous. But implementation is hard, slow, and requires sustained investment. It’s a multi-year journey for most organizations, not a quarterly initiative.

Start with identity. Get that right. Then segment. Then add continuous verification. Don’t try to do everything at once. And don’t let a vendor tell you their product is the answer. The answer is architecture change, and that’s harder than writing a purchase order.

I’ve been responsible for databases at the fintech startup (financial data platform, Postgres), at my infrastructure startup, and at several enterprises. The lesson that took me the longest to internalize: database reliability is mostly a habits problem.

You can run the best hardware, the fanciest replication setup, the most expensive managed service. If you don’t practice restores, don’t test migrations against production data volumes, and don’t have runbooks for failover, you’re going to have a bad day. The technology just determines how bad.

Start with numbers, not architecture

Before touching any config, agree on three numbers with your team and your business stakeholders:

RPO (Recovery Point Objective): How much data can you lose? For the fintech startup’s financial data, the answer was “essentially zero”: we couldn’t afford to lose transactions. For an analytics pipeline, losing 5 minutes of data might be fine.

RTO (Recovery Time Objective): How long can the database be down? A consumer app might tolerate 30 seconds. A payment system might need sub-second failover.

Latency SLOs : What’s the p99 query latency you’re committing to? This drives capacity planning, connection pooling, and index strategy.

These numbers dictate everything downstream. Synchronous replication for near-zero RPO. Automated failover for aggressive RTO. Proper connection pooling and query optimization for latency SLOs. Don’t skip this conversation.

Replication: pick your tradeoff

At the fintech startup we ran synchronous replication for our primary financial data. The write latency penalty was real: every commit waited for at least one standby to acknowledge. But for a financial data platform, losing committed transactions wasn’t an option.

For our analytics and reporting databases, we used asynchronous replication. Faster writes, slight risk of data loss on primary failure. Acceptable for data that could be re-derived.

In Postgres, the config difference is small but the operational impact is significant:

# Synchronous replication - zero data loss, higher write latency
synchronous_commit = on
synchronous_standby_names = 'FIRST 1 (standby1, standby2)'

# Asynchronous replication - lower latency, possible data loss on failover
synchronous_standby_names = ''   # no synchronous standbys; commits don't wait for replicas

The mistake I keep seeing: running async replication without understanding the RPO implication. If your primary dies and the standby is 5 seconds behind, you just lost 5 seconds of data. If that includes a payment confirmation, you have a problem that no incident postmortem can fix.

Failover: automate with guardrails

Automated failover sounds great until it promotes a standby that’s behind the primary and you get split-brain. Or it triggers during a network partition and now you have two primaries accepting writes.

At a large telecom company I worked with, I helped set up Patroni for Postgres HA. The key decisions:

• Health checks from multiple locations (not just one monitoring server)
• Minimum replication lag threshold before promotion is allowed
• Fencing of the old primary before the new one accepts writes
• Manual fallback documented and practiced

# patroni.yml: failover constraints live in the DCS settings
bootstrap:
  dcs:
    ttl: 30
    retry_timeout: 10
    maximum_lag_on_failover: 1048576  # ~1MB of WAL

The maximum_lag_on_failover setting is critical. It prevents promotion of a standby that’s too far behind. The value above is Patroni’s default (1MB of WAL); decide deliberately whether that much potential loss fits your RPO. I’ve seen teams never look at it and then wonder why they lost data during failover.

Test your failover. Quarterly at minimum. The first time you practice it, something will go wrong. Better during a planned exercise than during an actual outage at 3am.

Backups: the restore is the backup

Your backup strategy is only as good as your last tested restore.

At the fintech startup, we ran:

• Daily base backups via pg_basebackup
• Continuous WAL archiving to S3 for point-in-time recovery
• Weekly automated restore tests into a staging environment
• Monthly manual restore drills where an engineer actually performed the full recovery

The weekly automated test caught a corrupted backup exactly once. Once was enough. Without that test, we would have discovered the corruption during an actual emergency.

A solid Postgres backup setup:

# postgresql.conf - enable WAL archiving
archive_mode = on
archive_command = 'aws s3 cp %p s3://backup-bucket/wal/%f'
wal_level = replica

Point-in-time recovery is the real value here. Full backups let you restore to the last backup. WAL archiving lets you restore to any point in time. When someone runs a bad DELETE at 2:47pm, you can recover to 2:46pm. That capability has saved me more than once.

Safe migrations: the number one reliability risk

Schema changes cause more database incidents than hardware failures. I’ve seen it at every company. Someone runs an ALTER TABLE that locks a table for 20 minutes during peak traffic. The connection pool fills up. Upstream services start timing out. Cascade failure.

The patterns that keep you safe:

Add columns as nullable. Never add a NOT NULL column without a default to an existing table in a single step. Add it nullable, backfill, then add the constraint.

-- Step 1: Add nullable column
ALTER TABLE orders ADD COLUMN status_v2 text;

-- Step 2: Backfill in batches
UPDATE orders SET status_v2 = status
WHERE id BETWEEN 1 AND 10000 AND status_v2 IS NULL;
-- repeat for remaining batches

-- Step 3: Add constraint after data is clean
ALTER TABLE orders ALTER COLUMN status_v2 SET NOT NULL;

On a large table, SET NOT NULL still scans the whole table under an exclusive lock. On Postgres 12+, add a CHECK (status_v2 IS NOT NULL) NOT VALID constraint first, run VALIDATE CONSTRAINT, and SET NOT NULL can then skip the scan.

Build indexes concurrently. A regular CREATE INDEX locks the table for writes. On a large table, that’s an outage.

-- This blocks writes:
CREATE INDEX idx_orders_status ON orders(status);

-- This doesn't:
CREATE INDEX CONCURRENTLY idx_orders_status ON orders(status);

Set lock and statement timeouts. Prevent runaway migrations from holding locks indefinitely.

-- Before running migration
SET lock_timeout = '5s';
SET statement_timeout = '30s';

If a migration can’t acquire its lock within 5 seconds, it fails instead of queuing behind long-running queries and creating a pileup.

Test against production volumes. A migration that runs in 2 seconds on your dev database with 1,000 rows might take 45 minutes on production with 50 million rows. I learned this lesson at the fintech startup when a “quick” index build blocked writes for 12 minutes because we hadn’t tested against actual data volume.

Monitoring: watch the right things

Most database monitoring setups track too many things and miss the ones that matter. What I alert on:

Alert immediately:

• Replication lag above threshold (for us, 10 seconds)
• Disk usage above 80%
• Connection pool saturation above 90%
• Failed backup or missed WAL archive

Alert on sustained trends:

• Slow query count increasing over 15 minutes
• Lock wait time increasing over 5 minutes
• Transaction wraparound approaching (Postgres-specific, but it will bite you)

Track in dashboards, don’t alert:

• Query latency distribution (p50, p95, p99)
• Table and index sizes over time
• Autovacuum activity
• Cache hit ratios

A Postgres-specific query I run weekly to find trouble:

-- Tables approaching transaction wraparound
SELECT relname, age(relfrozenxid) as xid_age,
       pg_size_pretty(pg_total_relation_size(oid)) as total_size
FROM pg_class
WHERE relkind = 'r' AND age(relfrozenxid) > 200000000
ORDER BY age(relfrozenxid) DESC;

Transaction wraparound is the kind of problem that gives you little warning anyone notices and then takes the database down hard. Monitor it.

Maintenance isn’t optional

Postgres needs vacuuming. It needs statistics updates. It needs index maintenance. These aren’t nice-to-haves.

Autovacuum handles most of the work, but the defaults are conservative. For busy tables:

-- Per-table override for high-write tables
ALTER TABLE events SET (
    autovacuum_vacuum_scale_factor = 0.01,
    autovacuum_analyze_scale_factor = 0.005
);

The default autovacuum_vacuum_scale_factor of 0.2 means vacuum runs after 20% of the table has changed. On a 100-million-row table, that’s 20 million dead tuples before vacuum kicks in. Dropping it to 0.01 keeps things cleaner.

Index bloat is the other silent killer. Indexes grow as data is updated and deleted. Periodically check pg_stat_user_indexes for indexes with low usage and high size. Rebuild bloated indexes during maintenance windows using REINDEX CONCURRENTLY (Postgres 12+).

The operational habits that matter

Technology is the easy part. Habits are what prevent outages:

• Monthly restore drills. Someone on the team restores a backup from scratch. Time it. Document the steps. Every time.
• Quarterly failover tests. Actually promote a standby. Verify data integrity. Fail back. Fix whatever surprised you.
• Runbooks for every alert. If an alert fires and the on-call engineer has to figure out what to do from scratch, the runbook is missing.
• Post-incident follow-through. Writing the postmortem is easy. Actually fixing the root cause is what most teams skip.

Database reliability isn’t something you build once. It’s something you practice until the team can handle a 3am failover without panicking. The configs and queries above are tools. The discipline to use them consistently is the actual hard part.

I wrote about WebAssembly in March 2020 when the server-side story was mostly theoretical. Sixteen months later, here is the progress report. Some things moved faster than I expected. Some haven’t moved at all.

What’s actually changed

The biggest shift is that Wasm is showing up in production, not just conference demos. Not everywhere. Not for everything. But in specific niches where the tradeoffs make sense.

Edge computing is the clearest win. Cloudflare Workers has supported Wasm since 2018, and Fastly’s Compute@Edge runs Wasm on Lucet, its ahead-of-time compiler, which Fastly is now merging into Wasmtime. The value proposition is straightforward: Wasm modules start in microseconds, use minimal memory, and run in a sandbox. For edge compute, where you need code running close to users, starting fast, and tearing down quickly, that beats spinning up a container.

I’ve been using Compute@Edge for a few side projects. The developer experience is decent. You write Rust or AssemblyScript, compile to Wasm, deploy. Cold starts are fast. The limitation is the restricted API surface: you can handle HTTP request/response flows, but anything much beyond that gets painful.

Plugin systems are the second real use case. Envoy merged Wasm filter support upstream in 1.17 this January. It’s still marked experimental, but teams can already ship custom proxy logic (rate limiting, header manipulation, auth checks) without rebuilding the Envoy binary. At one company, we evaluated this for custom traffic-routing rules that varied per tenant. The alternative was maintaining a fork of Envoy. Wasm filters were dramatically simpler to deploy and update.

The pattern works for any platform that needs user-extensible logic in a safe sandbox. Think database UDFs, SaaS platform plugins, policy engines. The host controls resources and data access. The plugin runs in a box with no ambient authority. Language choice is flexible because Wasm doesn’t care what you compiled from.

WASI: still early, still important

WASI preview1 is the standard everyone targets. It gives you filesystem access (capability-based, not ambient), clocks, random number generation, and environment variables. That’s enough for CLI tools, data transformation, and controlled plugin workloads.

What’s still missing: networking. No sockets, no HTTP client, no DNS. This is the single biggest limitation for server-side Wasm. If your workload needs outbound calls, you’re either working around it through host-provided functions or you’re waiting. The WASI networking proposal exists, but it’s still in early design.

For Go developers specifically, the story is mixed. TinyGo compiles to Wasm and targets WASI, but you lose chunks of the standard library. The main Go compiler has experimental Wasm support, but it’s browser-focused and the binaries are large. I’ve had reasonable results with TinyGo for small, focused modules. I wouldn’t build a full service with it.

Runtimes are getting solid

Wasmtime (from the Bytecode Alliance) and Wasmer are both maturing. Wasmtime in particular is becoming the runtime others build on. The embedding story is good: you can load a Wasm module into a Go, Rust, or Python host, expose a narrow API, and let the module run in a sandbox.

I’ve been embedding Wasmtime in a Go service for a side project where users can define custom validation rules. The module loads in milliseconds, executes deterministically, and can’t touch anything the host doesn’t explicitly allow. Compared to running user code in a Docker container or subprocess, the operational story is dramatically simpler.

The debugging experience, though, is still rough. Print debugging works (you can wire up stdout through WASI). Proper step-through debugging with source maps is technically possible but clunky. Profiling is even worse. This matters less for small, plugin-style modules, but it would be a problem for anything substantial.

Where it doesn’t make sense

I need to be clear about this because the hype cycle is doing its thing.

Wasm isn’t replacing containers. It’s not a general-purpose server runtime. If your workload needs full OS access, mature networking, established tooling, and rich library ecosystems, containers are still the answer. Docker is boring and it works.

Wasm also struggles with anything that needs high-performance I/O or tight integration with native libraries. The cross-boundary calling overhead is real. Data copying between the host and the Wasm module can become the bottleneck. For compute-heavy, data-light workloads this is fine. For data-heavy workloads, measure before committing.

Where I’d use it today

Three scenarios where Wasm outside the browser makes sense right now:

1. Edge compute where startup time and memory footprint matter. Cloudflare Workers and Fastly Compute@Edge are the obvious platforms.

2. Plugin systems where you need safe, sandboxed execution of user-provided or tenant-specific logic. Envoy filters are the most mature example.

3. CLI tools and data transformers where portability across platforms matters and the workload fits within WASI’s current API surface.

For everything else, I’d wait. Wasm is promising, but the tooling, the system interface, and the cross-language interface work all need another year or two of work before server-side Wasm becomes a general-purpose option.

The trajectory

The direction is right. The Bytecode Alliance is doing real work. WASI is evolving (slowly). Edge platforms are proving the model. The interface types and module linking proposals could eventually solve the cross-language composition problem in a way that nothing else does.

But right now, in July 2021, Wasm beyond the browser is a sharp tool for specific problems. Not a platform. Use it where the constraints fit. Wait where they don’t.

At the fintech startup we processed financial data and market data from hundreds of sources. Some of it needed to be available in seconds: breaking news that moves stock prices. Some of it needed to be correct and complete for daily analytics: aggregated sentiment scores, content performance metrics. And some of it sat in between.

We ended up running all three ingestion patterns because different problems demanded different tradeoffs, not because we planned it that way. After living with all three in production, I’ve got strong opinions about when each one belongs.

The comparison

That table is the result of learning things the hard way.

Batch: the boring default that works

At the fintech startup, our analytics pipeline ran on batch. Every night, we pulled the day’s content, computed sentiment aggregates, calculated engagement metrics, and loaded the results into our warehouse for the analytics team.

Batch ingestion is a scheduled extract. Query the source, pull the data, load it somewhere. The tools are mature: Airflow for orchestration, simple scripts or managed connectors for extraction, a cloud warehouse for storage.

Why batch works so well:

• Idempotency is natural. If a job fails, rerun it. You get the same result. There’s no state to recover, no offsets to manage.
• Backfills are trivial. Need to reprocess last month? Change the date parameter and run the job again.
• Schema changes are manageable. Source adds a column? Your next batch picks it up. Source renames a column? You fix the mapping and reprocess.

The latency tradeoff is real. Batch means your data is hours old at best. For dashboards and reports, that’s fine. For anything user-facing and real-time, it’s not.

My rule: default to batch unless someone can articulate a specific business reason for lower latency. “It would be nice to have real-time data” isn’t a business reason. “Users leave the product if news is more than 30 seconds old” is.

CDC: the middle ground most people skip

Change Data Capture reads the database’s write-ahead log (WAL) and streams row-level changes to a target system. Debezium is the standard open-source tool. Most managed data platforms offer CDC connectors now.

At the fintech startup, we used CDC to replicate our content database to the analytics warehouse. Instead of running nightly full extracts that hammered the production database, we streamed changes continuously. The warehouse stayed within minutes of production, and the load on the source database dropped dramatically.

CDC hits a sweet spot:

• Near-real-time without the streaming overhead. You get minute-level freshness without running a Kafka cluster or designing event schemas.
• Auditable by default. Every change is captured. You get a complete history of what changed and when.
• Lower source load. Reading the WAL is far cheaper than running full table scans.

The downsides are real but manageable. CDC connectors need monitoring. They can fall behind, hit WAL retention limits, or break on DDL changes. Schema evolution requires more care than batch because the connector is continuously running. And initial snapshots for large tables can be slow.

For most teams that want better-than-daily freshness, CDC is the right answer. It’s the pattern I recommend most often because it solves the latency problem without introducing streaming complexity.

Streaming: powerful and expensive

True event streaming ( Kafka , Kinesis, Pulsar) is for when your product needs to react to events in real time. At the fintech startup, our breaking news pipeline was streaming. Content arrived from news APIs, got classified and enriched, and appeared in user feeds within seconds.

Streaming is the right tool when:

• User experience depends on seconds-level latency.
• Events need to trigger immediate downstream actions.
• You need to join multiple event streams in real time.

Streaming is the wrong tool when people want it because it sounds impressive on an architecture diagram.

What streaming costs in practice:

• Cluster management. Kafka isn’t a fire-and-forget service. Broker configuration, partition management, replication, consumer group coordination. Even managed Kafka (Confluent Cloud, MSK) requires operational attention.
• Schema discipline. When producers and consumers are decoupled and running continuously, a schema change can break downstream consumers silently. You need a schema registry. You need compatibility policies. You need someone who owns them.
• Ordering and exactly-once semantics. These sound like solved problems until you hit partition rebalancing, consumer crashes, or cross-partition joins. The guarantees are achievable but require careful design.
• Backfills are painful. Need to reprocess a week of events? You need to replay from stored offsets, hope retention was sufficient, and deal with the downstream impact of reprocessing alongside live traffic.

At the fintech startup we had two engineers who understood the streaming pipeline deeply. When they were unavailable, incidents on the streaming side took significantly longer to resolve than batch or CDC issues. That knowledge concentration is a real operational risk.

The transformation layer

Regardless of how data gets in, it needs to be transformed for consumption. ELT (load raw, transform in the warehouse) has become the standard pattern, and dbt is the tool that made it practical.

-- models/content_daily_metrics.sql
select
  content_id,
  source_name,
  published_date,
  count(*) as view_count,
  avg(sentiment_score) as avg_sentiment
from {{ ref('stg_content_events') }}
group by 1, 2, 3

Version-controlled SQL. Testable. Documented. This is one of the few areas where the tooling has clearly gotten better in the last two years. At the fintech startup we adopted dbt for our analytics models and the improvement in confidence and velocity was immediate.

What I tell teams now

Start with batch. Seriously. Get your data warehouse set up, your orchestration running, your dbt models tested. Build the operational muscle for data quality: freshness checks, row count monitoring, schema validation.

When batch latency becomes a proven problem for a specific use case, add CDC for that source. Don’t migrate everything. Just the tables where minutes matter.

Reserve streaming for the use cases where seconds matter and the business impact justifies the operational cost. Have at least two people who understand the streaming infrastructure deeply. Monitor consumer lag obsessively.

The most common mistake I see: teams adopt streaming first because it’s exciting, then spend months building operational maturity that batch would have given them for free. The boring choice is usually the right choice. Optimize for reliability and then add speed where the business actually needs it.

I’ve seen it up close. Some of the large organizations I’ve worked with run workloads across two or three cloud providers. Every single one wishes it didn’t. The ones that are happy with multi-cloud are happy because a specific regulatory or customer requirement forced their hand, not because they chose it for technical elegance.

The rest got there by accident. Different teams picked different clouds. An acquisition brought in a second platform. Someone in leadership read a Gartner report. Now they have two sets of IAM policies, two networking models, two monitoring stacks, and a platform team that’s permanently underwater.

The pitch vs. the reality

The multi-cloud pitch goes like this: avoid vendor lock-in, leverage best-of-breed services from each provider, improve resilience through provider redundancy.

“Avoid lock-in.” You’re not avoiding lock-in. You’re doubling it. Now you’re locked into two vendors instead of one, and you’re also locked into whatever abstraction layer you built to make them look the same. That abstraction layer is its own maintenance burden. It lags behind both providers’ features. And when it breaks, nobody outside your platform team knows how to fix it.

“Best-of-breed.” This sounds rational until you try to integrate BigQuery with an application running on AWS. Cross-cloud data movement is slow, expensive, and architecturally painful. Data has gravity. Once it lives somewhere, everything that touches it wants to live there too. Picking services from multiple providers means fighting data gravity constantly.

“Provider redundancy.” This is the one that really gets me. How many organizations have actually tested a full provider failover? I’ve asked this question dozens of times. The answer is almost always no. They have multi-cloud on a slide deck. They don’t have a tested failover runbook. If AWS goes down, their “redundant” GCP deployment hasn’t been exercised under real load and will probably fall over for entirely different reasons.

When it actually makes sense

I’m not saying multi-cloud is never justified. There are real reasons.

Regulatory or data residency requirements can force specific workloads into specific regions that only one provider serves well. Mergers and acquisitions bring in existing infrastructure that you can’t migrate overnight. Some customers contractually require their data to run on a specific platform.

Those are constraints. You respond to constraints. That’s different from choosing multi-cloud because it feels like good architecture.

What I actually recommend

Default to one cloud. Pick the one your team knows best. Go deep on it. Use its managed services. Learn its IAM model properly. Build in multiple regions for resilience. That covers most failure scenarios with a fraction of the operational complexity.

Design for portability at clear boundaries. Containers and Kubernetes give you a portable compute layer, and that’s real. But Kubernetes doesn’t abstract away networking, storage, IAM, DNS, load balancing, or any of the managed data services. Treat Kubernetes as a helpful layer, not a magic portal between clouds.

Keep data in one place unless you have a specific, measured reason to replicate it. Cross-cloud data synchronization is a beast. Latency, consistency, cost, and operational complexity all increase dramatically. I’ve seen teams spend more engineering hours managing cross-cloud data replication than they spent building the features that use the data.

Have an exit plan even if you’re single-cloud. Know how to export your data. Understand which services are proprietary and which have open equivalents. Negotiate contracts with exit timelines. This is smart vendor management. It doesn’t require running production workloads on two platforms.

Who benefits from multi-cloud

Multi-cloud is mostly sold by people who benefit from the complexity it creates. Cloud vendors sell multi-cloud management tools. Consulting firms sell multi-cloud strategy projects. Platform vendors sell abstraction layers.

The teams who actually run the infrastructure? They’re drowning in operational overhead and wondering why everything takes twice as long.

If someone proposes multi-cloud at your organization, ask one question: what specific problem does a second cloud provider solve that multi-region on our current provider doesn’t? If the answer is vague, you have your answer.

A year ago I was heads-down building a cloud infrastructure company, convinced that the biggest challenge ahead was product-market fit. Then March happened and the biggest challenge became making sure the company didn’t die. Different skill set entirely.

The Company Made It

I’ll start with the thing I’m most proud of. The company survived 2020. But I want to be straight about what that actually looked like, because the LinkedIn version of startup survival stories is always cleaner than reality.

March and April were ugly. Two potential customers froze their budgets overnight. A partnership I’d been working on for months went silent. I spent a few weeks unsure if the company would make it to summer. The thing that saved it was that the product was a cloud infrastructure tool during the exact moment when every company on earth suddenly needed better cloud infrastructure. Lucky timing dressed up as strategy.

By Q3 I was shipping features faster than I was comfortable with. Some of that code is going to haunt me in 2021 and I know it. But the company is alive, and if you’d told me that in April I would have hugged you.

Leading Teams at Larger Platforms

The other big part of this year happened alongside the company. I spent 2020 as a tech lead at two larger platforms. Both taught me things a small startup can’t.

The biggest lesson: large organizations fail at decision-making far more often than at technology. I watched a team of perfectly competent engineers sit paralyzed for three weeks because nobody could get sign-off on a Kubernetes cluster configuration. The technology was the easy part. Getting six stakeholders to agree on a risk tolerance was the hard part.

I also learned that my founder experience translated directly. The problems startups solve with speed and scrappiness are the same problems enterprises solve with process and budget. Same problems, different constraints. Understanding both sides made me better at each.

Remote Work: What the Industry Got Wrong

I need to rant about this because the discourse around remote work in 2020 was insufferable.

Half the industry spent the year writing thought pieces about how remote work is the future and offices are dead. The other half spent it writing thought pieces about how remote work is destroying culture and we need to get back to the office immediately. Both camps were wrong in the same way: they treated remote work as a binary instead of a design problem.

Remote work is infrastructure: a set of systems and processes that either work or don’t. I’ve been running distributed teams since my days at a mobility startup. It works when you design for it. It fails when you take your office habits and paste them onto Zoom.

The companies that struggled in 2020 had never invested in written communication, async decision-making , or documentation. They replaced hallway conversations with eight hours of video calls and then blamed the format.

My company had been remote since day one, so the lockdown changed little about how I worked there. Not because remote is magic. Because I already documented decisions and wrote things down instead of keeping them in my head.

Office or remote won’t decide which companies thrive in 2021. The ones that design their communication systems will beat the ones that default to whatever they did before.

What I Got Wrong

I try to own my mistakes in these posts, so here goes.

I underestimated how much the pandemic would affect the teams I worked with. I spent Q2 optimizing for output when I should have been optimizing for sustainability. We shipped a lot, but a couple of people burned out, and I should have noticed sooner.

I also overestimated how quickly large organizations would modernize. Compressed timelines don’t mean compressed politics. Some organizations moved their infrastructure to the cloud in weeks but took months to update their security policies to match. The gap between technical capability and organizational readiness is wider than I thought.

The SolarWinds Wake-Up Call

I can’t write a 2020 review without mentioning SolarWinds . In December we learned that a major software supply chain had been compromised for months. Government agencies, Fortune 500 companies, the works.

This is the kind of thing I’ve been warning about for years. Your security is only as good as your weakest vendor. Most organizations treat vendor risk as a compliance checkbox. SolarWinds proved it’s an operational reality. I expect this to dominate security conversations well into 2021.

What I’m Carrying Into 2021

The company has momentum. The work keeps growing. I have a clearer picture of how enterprises and startups can learn from each other. Specifically:

I want to build better. The code I shipped under pressure in 2020 needs attention. Technical debt is a loan and the interest is compounding.

I want to write more. 2020 made everything feel too heavy to write about casually. But I think writing forces clarity and I need more of that.

2020 was the strangest year of my career. It broke every plan I had and replaced them with better ones. I wouldn’t want to do it again. But I’m grateful for what it taught me, and I’m heading into 2021 with more conviction than I’ve had in years.

Onwards.

I run a cloud infrastructure startup. I think about server architectures constantly. And my reaction to the M1 is: great laptop chip, almost irrelevant to my server decisions today.

The hype vs. the reality

Apple designed the M1 for consumer devices. Tight integration between CPU, GPU, unified memory, the whole SoC approach. That’s what makes it fast. It’s also what makes it completely unlike anything you’d rack in a data center.

Server ARM is a different animal. Graviton2 and Ampere Altra exist already. They’re real. They’re decent. But they’re not the M1, and the M1 isn’t them. People conflating Apple’s consumer chip with the state of ARM servers are making a category error.

The real impact is on developer laptops

The M1’s biggest server impact is on developer laptops.

Every Mac developer is about to be running arm64 natively. That means your CI pipeline, your Docker images, your build tooling: all of it needs to handle multi-arch or you’re going to have a bad time. This is already annoying. It’s about to get much more annoying.

At my infrastructure startup I started shipping multi-arch images a few months ago. Not because I love ARM servers. Because I saw this coming and didn’t want the Go binaries breaking the moment developer laptops switched to ARM.

Cross-compile your stuff (Go makes this nearly free ). Publish arm64 and amd64 images. Test on both. This isn’t optional anymore.

ARM servers: be skeptical, not dismissive

I’ve benchmarked Graviton2 instances for several of our workloads. Results are mixed. Some services run 15-20% cheaper at comparable performance, which matters if you read your cloud bill . Others are worse. The “ARM is always more efficient” line is marketing. Reality is workload-specific.

Native extensions are still a pain. Some vendor SDKs don’t have ARM builds. The ecosystem is getting there but it’s not there.

My advice: pick one stateless, non-critical service. Run it on ARM for a month. Measure everything. Then decide if it’s worth the migration overhead for your specific stack. Don’t migrate because a blog post told you ARM is the future.

My prediction

Five years from now, ARM will have meaningful server market share. Not because of Apple. Because of economics: AWS is building its own chips, and when your cloud provider is also your chip vendor, the pricing incentives get very real.

But that’s a five-year story. Not a next-quarter story. And definitely not a “rewrite your deployment pipeline this weekend” story.

Fix your multi-arch builds. Ignore the hype. Check back in 2022.

March 2020. Every company on earth scrambles to get employees working from home. The answer for most? Crank up the VPN capacity . Buy more licenses. Pray.

I watched this unfold from the other side. At my infrastructure startup I was building infrastructure that assumed zero trust from day one, and before that, at the fintech startup, we had moved most services behind identity-aware proxies. Meanwhile, companies that had spent a decade building fat VPN pipes were routing all their cloud traffic through a single gateway in an office nobody was sitting in.

The absurdity was staggering.

The VPN mental model is from 1996

Here’s how a VPN works in practice:

Remote laptop -> encrypted tunnel -> office network -> everything

That arrow into “everything” is the problem. Once you’re through the gate, the network treats you like you’re sitting at a desk in the building. Same access. Same trust. Same flat network where a compromised laptop can reach the database server, the admin panel, the file shares, all of it.

VPNs do get a few things right. Encryption in transit is solid. The connect-then-work model is simple for users. And for weird legacy protocols that can’t be proxied, sometimes a network tunnel is the only option.

But the failure modes are brutal:

• Lateral movement is trivial. One stolen credential and the attacker is “inside.” Game over on a flat network.
• Performance is terrible for cloud workloads. You’re backhauling traffic to an office gateway just to reach AWS. In 2020. While nobody is in the office.
• Visibility stops at the perimeter. You know someone connected. You don’t know what they’re doing once they’re in.
• Scaling is painful. VPN concentrators have hard capacity limits. Ask anyone who hit them in March.

In the cyber-defense exercises I took part in, the model was already shifting. You can’t protect a network perimeter when the network spans many organizations, temporary sites, and several trust levels. The answer there was granular access control tied to identity and role, not network location.

Zero trust is an architecture decision

I need to say this because vendors have completely destroyed the term. You can’t buy zero trust, and a firewall with a new label isn’t it. It’s an architecture decision : no request is trusted by default, regardless of where it comes from.

Every access decision considers:

• Who is making the request (identity, authentication strength, MFA)
• What device they’re on (managed? encrypted? patched?)
• What context surrounds the request (location, time, risk signals)
• What resource they’re trying to reach (and whether they actually need it)

In practice, this looks like per-application access policies instead of network-wide trust:

app: internal-wiki
rules:
  - name: employee-access
    conditions:
      - identity: authenticated
      - mfa: required
      - device: managed
    action: allow
  - name: default
    action: deny

Simple. Explicit. Every app gets its own policy. No more “you’re on the VPN so you can reach everything.”

What changes day to day

Blast radius. VPN compromise gives attackers the network. Zero trust compromise gives them one application session, maybe, if they also have the right device posture and MFA token.

Performance. VPN routes everything through a chokepoint. Zero trust lets users connect directly to the resource. Your team in Madrid doesn’t need to hairpin through a London office to reach a Frankfurt data center.

Auditability. With a VPN you get connection logs. With zero trust you get per-request decisions with full context. When something goes wrong, and it will, you actually know what happened.

User experience. This one surprises people. Zero trust done right is invisible. No “connect to VPN” step. No split-tunnel headaches. You open the app, the policy engine checks everything behind the scenes, and you’re in. Or you’re not.

When to keep the VPN

I’m not dogmatic about this. VPNs still earn their place in specific scenarios:

• Legacy applications that require L3/L4 network access and can’t be proxied
• Air-gapped or highly isolated environments where direct internet connectivity isn’t allowed
• Temporary access for third-party contractors working with ancient systems
• Specific protocols that don’t speak HTTP and can’t be wrapped

The key distinction: VPN as a tactical tool for specific legacy needs, not as your security architecture.

A migration path that works

At my infrastructure startup, I built with zero trust from scratch. Most companies don’t have that luxury. Here’s the migration path I’ve seen work:

Step 1: Get identity right. Centralize authentication. Enforce MFA everywhere. Not “for admins.” Everywhere. This is the foundation and nothing else works without it.

Step 2: Know your devices. Define what a “trusted device” means. Managed. Encrypted. Patched within some reasonable window. Start enrolling endpoints.

Step 3: Move web apps first. SaaS tools and internal web applications are the easiest to put behind an identity-aware proxy. BeyondCorp style. This covers 70-80% of what people actually use daily.

Step 4: Shrink the VPN scope. Every app you move to per-app access is one less reason for the VPN to exist. Segment your internal network. Tighten VPN access to only the systems that need it.

Step 5: Instrument everything. Log every access decision. Not just “allowed/denied” but the full context. You’ll need this to tune policies and prove the model works.

Step 6: Iterate. Review access logs weekly. Find overly broad policies and tighten them. Find unnecessary friction and reduce it. This isn’t a project with an end date. It’s how you operate now.

The traps I’ve seen teams fall into

Buying a “zero trust product” and calling it done. You didn’t adopt zero trust. You bought a proxy. The architecture change is what matters.

Exception creep. Someone important complains about access friction, you create a broad exception, and suddenly you’ve rebuilt the perimeter you were trying to eliminate. Every exception should have an expiration date and an owner.

Ignoring device health. MFA plus a compromised laptop is still a compromised session. Device posture isn’t optional.

No communication plan. People will lose access to things. If you don’t explain why and provide a clear path to regain access, you’ll face a revolt and the project will get rolled back.

Where this is going

The perimeter model worked when your employees, your servers, and your data all lived in the same building. That world is gone and it’s not coming back.

Zero trust isn’t new. Google published the BeyondCorp papers years ago. The military has operated on need-to-know principles forever. What’s new is that 2020 made it impossible to ignore. Every company just ran a forced experiment in remote access at scale, and the VPN-first companies suffered the most.

Start the migration now. Identity and MFA this month. Per-app access for your top ten applications this quarter. Shrink the VPN to legacy-only by end of year.

The network perimeter is dead. Stop defending a border that doesn’t exist.

I’ve been working with companies on their infrastructure stories for a while now. I keep seeing the same pattern. Someone reads a blog post about how Spotify or Netflix built an internal developer platform. They get excited. They rebrand their ops team. Nothing actually changes.

The naming problem

Our industry has a compulsive need to mint new labels. We did it with “DevOps” when we meant “ops engineers who can code.” We did it with “SRE” when we meant “on-call but with better job titles.” Now we’re starting to do it with “platform teams” and “internal developer platforms” when we mean “build internal tools so developers stop filing tickets.”

The underlying idea is fine. Good, even. Build self-service paths. Automate the repetitive stuff. Treat your internal tooling like a product. None of this is new. It’s what good infrastructure teams have done for years, and what I tried to build at the fintech startup . We just keep repackaging it.

What a platform has to do

Strip away the branding and you get a short list. Your developers should be able to spin up a service, deploy it, and observe it without waiting on another team. That’s it. That’s the whole thing.

The hard part was never the concept. It’s the execution. I’ve seen companies build elaborate internal platforms that nobody uses because the team built what they thought was cool instead of what developers actually needed. Classic product mistake, applied to infrastructure.

The teams that get it right have three habits:

They start embarrassingly small. One good deploy pipeline beats a half-finished portal with a service catalog. Ship the boring stuff first.

They talk to their users. Your users are the other engineers. If you’re not sitting with them, watching them struggle with your tooling, you’re guessing. And you’re probably guessing wrong.

They don’t mandate adoption. If your platform is better, people will use it. If you have to force adoption, that’s feedback. Listen to it.

The cynical take

Part of me expects the “platform” rebrand to become a way to hire for the same roles at higher salaries. New title, new comp band. I’d get it. Supply and demand.

But the other part of me worries that every time we rename a discipline, we lose institutional knowledge. Junior engineers will think platform teams were invented last year. They miss the decades of operations wisdom that came before. They reinvent mistakes that were solved in 2012.

Do the boring work

If you’re building an internal platform right now, my advice is simple. Ignore the conference talks. Ignore the vendor pitches about developer portals. Go sit with a developer on your team. Watch them deploy something. Count the manual steps. Automate the worst one. Repeat.

That’s platform engineering. It’s also DevOps. It’s also just… good engineering.

Call it whatever you want. Just do the work.

My startup manages cloud infrastructure. VMs, networks, storage volumes: all of it provisioned and lifecycle-managed through Kubernetes operators . Not because it’s trendy. Because our customers expect kubectl apply and then things just work.

I’ve written six operators over the past year and a half. Some are rock solid. Two of them cost me weekends. The difference was never the complexity of the domain. It was always the same three things: how I structured the reconciliation loop, how I designed the CRD, and whether I handled deletion properly.

This is the post I wish I’d had when I started.

What an operator actually is

Strip away the marketing and an operator is just a control loop. You define a custom resource (your API), you write a controller that watches it, and every time something changes, your controller tries to make reality match the spec.

That’s it. The entire pattern:

User writes CR spec -> Controller sees change -> Controller reconciles -> Updates status -> Waits

The “waits” part matters. Your controller isn’t a script that runs once. It runs forever. It gets called when things change, when things break, when Kubernetes restarts it, when someone fat-fingers a kubectl edit. If your reconciler can’t handle being called twice in a row with the same input and produce the same result, you’re going to have a bad time.

Setting up with controller-runtime

I use controller-runtime directly. Kubebuilder generates scaffolding on top of it, and that’s fine for getting started, but I prefer knowing exactly what’s under me. Less magic, fewer surprises when something breaks.

Here’s the skeleton. I’ll use a simplified version of one of our operators: a CloudVolume that provisions storage on a provider API and attaches it to a node.

First, the types:

package v1alpha1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

type CloudVolumeSpec struct {
    SizeGB   int    `json:"sizeGB"`
    Region   string `json:"region"`
    NodeName string `json:"nodeName"`
}

type CloudVolumeStatus struct {
    VolumeID   string `json:"volumeID,omitempty"`
    Phase      string `json:"phase,omitempty"`
    AttachedTo string `json:"attachedTo,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
type CloudVolume struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec              CloudVolumeSpec   `json:"spec,omitempty"`
    Status            CloudVolumeStatus `json:"status,omitempty"`
}

Small spec. Obvious fields. No map[string]interface{} anywhere. I can’t stress this enough: the CRD is your product interface. If a user has to read your source code to figure out what to put in the spec, you’ve already lost.

The reconciliation loop

Here’s the real thing. Not a toy example. This is close to what runs in our production clusters, simplified for clarity but structurally identical.

func (r *CloudVolumeReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("cloudvolume", req.NamespacedName)

    var vol v1alpha1.CloudVolume
    if err := r.Get(ctx, req.NamespacedName, &vol); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Handle deletion first. Always.
    if !vol.DeletionTimestamp.IsZero() {
        return r.reconcileDelete(ctx, &vol)
    }

    // Ensure finalizer is present
    if !containsFinalizer(vol.Finalizers, finalizerName) {
        vol.Finalizers = append(vol.Finalizers, finalizerName)
        if err := r.Update(ctx, &vol); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil
    }

    // Provision if we don't have a volume ID yet
    if vol.Status.VolumeID == "" {
        return r.reconcileProvision(ctx, log, &vol)
    }

    // Attach if not attached
    if vol.Status.AttachedTo != vol.Spec.NodeName {
        return r.reconcileAttach(ctx, log, &vol)
    }

    // Steady state. Check back in 5 minutes.
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

Notice the structure. Deletion check first, then finalizer setup, then provision, then attach, then idle. Each branch returns early. No nested ifs six levels deep. No giant switch statement. The reconciler reads top to bottom like a decision tree, and every path is idempotent.

This structure isn’t clever. It’s boring. That’s the point.

The provision step

func (r *CloudVolumeReconciler) reconcileProvision(
    ctx context.Context,
    log logr.Logger,
    vol *v1alpha1.CloudVolume,
) (ctrl.Result, error) {

    log.Info("provisioning volume", "sizeGB", vol.Spec.SizeGB, "region", vol.Spec.Region)

    volumeID, err := r.Provider.CreateVolume(ctx, vol.Spec.SizeGB, vol.Spec.Region)
    if err != nil {
        log.Error(err, "failed to provision volume")
        r.setPhase(ctx, vol, "ProvisionFailed")
        // Don't return the error. Requeue with backoff instead.
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }

    vol.Status.VolumeID = volumeID
    r.setPhase(ctx, vol, "Provisioned")

    if err := r.Status().Update(ctx, vol); err != nil {
        return ctrl.Result{}, err
    }

    log.Info("volume provisioned", "volumeID", volumeID)
    return ctrl.Result{Requeue: true}, nil
}

Two things I learned the hard way here.

Don’t return errors from external API calls. If you return ctrl.Result{}, err, controller-runtime uses exponential backoff, which is fine. But for provider API failures, I want explicit control over retry timing. A provider outage shouldn’t hammer their API with exponential retries that max out at 16 minutes. I return RequeueAfter with a fixed delay and handle the error myself.

Update status immediately after the side effect. If I provision the volume but crash before writing the status, the next reconciliation will call CreateVolume again. That’s why our provider client is idempotent: it checks if a volume with the same labels already exists and returns it. But the status update should still happen as fast as possible to minimize the window.

Deletion and finalizers

This is where most operators break. I see people skip finalizers because “it’s just Kubernetes resources, they’ll get garbage collected.” Sure, for in-cluster stuff. But we’re provisioning cloud resources. If someone deletes the CloudVolume CR and we don’t clean up the actual volume, we’re leaking money.

const finalizerName = "cloudvolume.example.com/cleanup"

func (r *CloudVolumeReconciler) reconcileDelete(
    ctx context.Context,
    vol *v1alpha1.CloudVolume,
) (ctrl.Result, error) {
    log := r.Log.WithValues("cloudvolume", vol.Name)

    if !containsFinalizer(vol.Finalizers, finalizerName) {
        return ctrl.Result{}, nil
    }

    if vol.Status.VolumeID != "" {
        log.Info("deleting external volume", "volumeID", vol.Status.VolumeID)

        err := r.Provider.DeleteVolume(ctx, vol.Status.VolumeID)
        if err != nil && !isNotFound(err) {
            log.Error(err, "failed to delete external volume")
            return ctrl.Result{RequeueAfter: 15 * time.Second}, nil
        }
    }

    // Remove finalizer
    vol.Finalizers = removeFinalizer(vol.Finalizers, finalizerName)
    if err := r.Update(ctx, vol); err != nil {
        return ctrl.Result{}, err
    }

    log.Info("cleanup complete")
    return ctrl.Result{}, nil
}

The isNotFound check matters. If the volume was already deleted externally (someone cleaned it up in the cloud console, provider auto-deleted it, whatever), we still need to remove the finalizer. Without that check, the CR gets stuck in Terminating forever. Ask me how I know.

Wiring it up

The main setup is straightforward. This is where you tell controller-runtime what to watch and how to build the manager.

func main() {
    var metricsAddr string
    flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "metrics endpoint")
    flag.Parse()

    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:             scheme,
        MetricsBindAddress: metricsAddr,
        Port:               9443,
        LeaderElection:     true,
        LeaderElectionID:   "cloudvolume-controller",
    })
    if err != nil {
        setupLog.Error(err, "unable to start manager")
        os.Exit(1)
    }

    if err := (&CloudVolumeReconciler{
        Client:   mgr.GetClient(),
        Log:      ctrl.Log.WithName("cloudvolume"),
        Scheme:   mgr.GetScheme(),
        Provider: provider.NewClient(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller")
        os.Exit(1)
    }

    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

LeaderElection: true: don’t skip this. Without it, running two replicas means two controllers fighting over the same resources. We had this bug for exactly one deploy before someone noticed duplicate volumes showing up.

Testing operators

Testing operators is painful. I won’t pretend otherwise. But envtest from controller-runtime makes it tolerable. It spins up a real API server and etcd in-process. No mocks for the Kubernetes API layer.

func TestReconcileProvision(t *testing.T) {
    env := &envtest.Environment{
        CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
    }
    cfg, err := env.Start()
    require.NoError(t, err)
    defer env.Stop()

    k8sClient, err := client.New(cfg, client.Options{Scheme: scheme})
    require.NoError(t, err)

    // Create a CloudVolume
    vol := &v1alpha1.CloudVolume{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "test-vol",
            Namespace: "default",
        },
        Spec: v1alpha1.CloudVolumeSpec{
            SizeGB:   100,
            Region:   "eu-west-1",
            NodeName: "worker-1",
        },
    }
    require.NoError(t, k8sClient.Create(context.TODO(), vol))

    // Run reconciler with a fake provider
    reconciler := &CloudVolumeReconciler{
        Client:   k8sClient,
        Log:      ctrl.Log.WithName("test"),
        Scheme:   scheme,
        Provider: &fakeProvider{volumeID: "vol-abc123"},
    }

    result, err := reconciler.Reconcile(ctrl.Request{
        NamespacedName: types.NamespacedName{Name: "test-vol", Namespace: "default"},
    })
    require.NoError(t, err)
    assert.True(t, result.Requeue)

    // Verify status was updated
    var updated v1alpha1.CloudVolume
    require.NoError(t, k8sClient.Get(context.TODO(), types.NamespacedName{
        Name: "test-vol", Namespace: "default",
    }, &updated))
    assert.Equal(t, "vol-abc123", updated.Status.VolumeID)
}

Mock the external provider, not Kubernetes. The fake provider interface is trivial: CreateVolume, DeleteVolume, AttachVolume. Maybe five methods total. This gives you fast, reliable tests that catch real API interaction bugs.

Unit tests for pure logic (like “given this spec and this status, what phase should we be in?”) are even simpler. Regular table-driven Go tests. No envtest needed.

Mistakes I’ve made (so you don’t have to)

Mutating spec in the reconciler. I did this once to “normalize” a field. Created an infinite reconcile loop because every spec update triggered a new event. Took me an embarrassingly long time to figure out why CPU was pinned at 100%.

Not watching owned resources. Our volume operator creates a PersistentVolume as a downstream resource. If someone manually deletes the PV, the operator needs to know. Forgot to add it to the watch and spent a day debugging why volumes were “attached” but pods couldn’t mount them.

func (r *CloudVolumeReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&v1alpha1.CloudVolume{}).
        // PVs are cluster-scoped, so they can't carry an owner reference to a
        // namespaced CloudVolume. Map them back through labels we set on create.
        Watches(&source.Kind{Type: &corev1.PersistentVolume{}},
            &handler.EnqueueRequestsFromMapFunc{
                ToRequests: handler.ToRequestsFunc(func(obj handler.MapObject) []reconcile.Request {
                    labels := obj.Meta.GetLabels()
                    name, ok := labels["cloudvolume.example.com/name"]
                    if !ok {
                        return nil
                    }
                    return []reconcile.Request{{NamespacedName: types.NamespacedName{
                        Name:      name,
                        Namespace: labels["cloudvolume.example.com/namespace"],
                    }}}
                }),
            }).
        Complete(r)
}

Writing status before the side effect completes. Optimistic status updates sound reasonable until your provider call fails after you already told the world the volume exists. Then you have phantom resources in your status that don’t exist anywhere. Always: do the thing, then update the status.

No rate limiting on requeue. Early versions of our operator requeued immediately on every provider error. During a 30-minute cloud outage, we sent thousands of requests to a failing API. Our provider rate-limited us, which made the recovery take even longer. Fixed it by adding RequeueAfter: 30 * time.Second minimum for any external call failure.

When not to write an operator

Operators are the wrong tool when the thing you’re managing doesn’t have meaningful lifecycle beyond deploy-and-forget. A stateless web app with a Deployment, Service, and Ingress? Helm chart. Done.

I write an operator when:

• The thing has external state that Kubernetes doesn’t know about
• Day-2 ops involve multi-step coordination (backup, failover, resize)
• Humans currently follow a runbook that involves kubectl and another API

If none of those apply, you’re adding complexity for no reason. And complexity in controllers is expensive because it runs 24/7 and fails silently.

Where we are now

Six operators. Three manage cloud resources (volumes, networks, VMs). Two handle internal platform concerns (cert rotation, DNS). One manages database provisioning. The three cloud operators are the most complex because external APIs are unreliable and slow. The internal ones are surprisingly simple once you get the pattern down.

Total lines of Go across all six: roughly 12,000. Most of that’s tests. The actual reconciler logic is small in every case. If your reconciler is more than 200-300 lines, you’re probably doing too much in one controller.

The operator pattern works. But it demands discipline. Idempotent reconciliation. Small CRDs. Defensive deletion. Boring code that a sleepy on-call engineer can read at 3am and understand what went wrong.

That last part is the real test.

Between running infrastructure at my infrastructure startup and working with other teams, there’s one pattern I see in almost every cluster I look at: resource settings are either copy-pasted from a tutorial, set to absurdly high values “just to be safe,” or missing entirely.

The result is always the same. Either you’re paying for 3x the compute you actually need , or your pods are getting evicted during the one traffic spike that matters.

Requests vs limits: two different jobs

People treat these as the same thing. They’re not.

Requests are what the scheduler sees. When you set requests.cpu: 200m, you’re telling Kubernetes “this pod needs at least 200 millicores to run, find me a node that has that available.” The scheduler uses requests to bin-pack pods onto nodes.

Limits are runtime caps. The kernel enforces these. Hit your CPU limit and you get throttled. Hit your memory limit and you get OOM-killed. Very different consequences.

resources:
  requests:
    cpu: "200m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "512Mi"

One thing that trips people up: if you set a limit but skip the request, Kubernetes sets the request equal to the limit. If you set neither, congratulations, you’ve got a BestEffort pod. First to die when the node gets tight.

CPU throttling vs OOM kills

This is the part that bites people.

CPU is compressible. Your container hits its CPU limit? It gets throttled. Slower responses, higher latency, but it stays alive. Annoying but survivable.

Memory isn’t compressible. Your container exceeds its memory limit? The kernel kills it. No negotiation. The pod restarts (if your restart policy allows it), your users see errors, and you get an alert.

I’ve seen teams set memory limits too close to actual usage, then wonder why they’re getting random OOM kills during garbage collection spikes. Leave real headroom on memory. I’d rather waste 50Mi per pod than deal with cascading restarts.

QoS classes matter more than you think

Kubernetes assigns every pod a Quality of Service class based on its resource settings. The class shapes eviction order when a node runs out of resources.

Guaranteed: requests equal limits for both CPU and memory, in every container. These pods get evicted last. Use this for anything you actually care about in production.

Burstable: requests are set but limits are higher. Most pods in real clusters end up here. Fine for most workloads, but know that you’re in the middle of the eviction queue.

BestEffort: no requests, no limits, no guarantees. First to get evicted. I only use this for batch jobs I don’t care about.

The mistake I see constantly: teams running critical APIs as BestEffort because nobody set resource fields in the deployment spec. Then a memory spike on the node evicts those pods before the logging agent that has Guaranteed QoS. Backwards.

How I actually size resources

I don’t guess. I measure, then set values based on real data.

Start with kubectl top to get a quick snapshot:

kubectl top pods -n production
kubectl top pods -n production --containers

But a snapshot isn’t enough. You need percentile data over days or weeks. Pull p50, p75, p95, and p99 from whatever metrics system you’re running: Prometheus, Datadog, whatever.

Then I set values like this:

• CPU request: p75 usage plus a small buffer. This is your “normal operating range.”
• Memory request: p95 usage plus a buffer. Memory spikes are less forgiving.
• CPU limit: 2-3x the request. Let it burst for short periods.
• Memory limit: request plus enough headroom for GC spikes and transient allocations. But not so much that a leak runs unchecked.

resources:
  requests:
    cpu: "200m"      # p75 + headroom
    memory: "300Mi"  # p95 + headroom
  limits:
    cpu: "500m"      # room for bursts
    memory: "450Mi"  # room for GC, not for leaks

Review these after any significant traffic change. Resource settings should evolve with your workload, not stay frozen from the day someone first deployed the service.

Autoscaling helps but isn’t magic

HPA (Horizontal Pod Autoscaler) scales replicas based on CPU utilization relative to your requests. If your requests are wrong, your scaling behavior is wrong. Garbage in, garbage out.

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

VPA (Vertical Pod Autoscaler) is interesting but still beta. I run it in recommendation mode only: let it tell me what it thinks the requests should be, then I decide whether to apply that.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"

I don’t trust VPA to mutate running pods in production yet. Maybe next year.

LimitRanges and ResourceQuotas: the guardrails you need

Every namespace in a shared cluster should have a LimitRange. Full stop. Without one, any developer can deploy a pod with no resource settings, and now you’ve got a BestEffort pod sitting in production waiting to get evicted.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      default:
        cpu: "500m"
        memory: "256Mi"

This gives every container sane defaults even if the developer forgot to set them. It’s a safety net, not a replacement for proper resource specs.

ResourceQuotas cap total consumption per namespace. Essential for multi-tenant clusters where you don’t want one team eating the whole cluster.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"

What to actually monitor

Set it and forget it doesn’t work here. Watch these:

• CPU throttling rate: if it’s consistently high, your CPU limit is too low
• OOM kill count: any non-zero number means your memory limit needs attention
• Request-to-usage ratio: if you’re requesting 4x what you use, you’re wasting cluster capacity
• Allocatable vs requested: tells you how much room the cluster actually has
• Eviction events: if pods are getting evicted, something is wrong with your resource math

I check these weekly for the clusters I manage. It takes ten minutes and prevents most resource-related incidents before they happen. It belongs on the same boring checklist as probes and upgrades.

Nobody writes conference talks about setting CPU requests correctly. Get it wrong anyway and a BestEffort pod gets evicted at 3am and takes your API gateway with it.

We moved my infrastructure startup’s internal APIs to gRPC about eight months ago. Before that, everything was REST with hand-rolled JSON serialization, version mismatches everywhere, and a weekly ritual of “why is the client sending a string where we expect an int.” Moving to gRPC fixed the type safety problem. It also introduced a whole new category of problems I didn’t anticipate.

This is what I wish someone had written down before we started, with the Go code we actually run.

Why we picked gRPC (and where we didn’t)

Short version: internal service-to-service calls where latency matters and both sides are services we control. That’s it. That’s the use case.

We kept REST for:

• Anything browser-facing. gRPC-Web exists but it’s a subset and adds a proxy layer. Not worth it for us.
• Third-party integrations. Nobody wants to learn your proto schema.
• Simple CRUD admin tools. curl is a better debugger than grpcurl and I don’t care what anyone says.

The real win was code generation. We have Go, Python, and a small Rust service. Generating typed clients from one .proto file eliminated an entire class of integration bugs. Before gRPC, every language had its own hand-written client that drifted independently. Now drift is a compilation error.

Proto design: get this wrong and you’ll pay for years

Field numbers are forever. I mean that literally. Once you ship a proto, those field numbers are carved into every binary that’s ever been compiled against it. Get the schema design wrong early and you’re living with it or doing a painful migration.

Here’s what our service definitions actually look like:

syntax = "proto3";

package infra.nodes.v1;

import "google/protobuf/timestamp.proto";

service NodeService {
  rpc GetNode(GetNodeRequest) returns (Node);
  rpc ListNodes(ListNodesRequest) returns (ListNodesResponse);
  rpc WatchNodeStatus(WatchNodeStatusRequest) returns (stream NodeStatusEvent);
}

message GetNodeRequest {
  string node_id = 1;
}

message ListNodesRequest {
  int32 page_size = 1;
  string page_token = 2;
  NodeFilter filter = 3;
}

message ListNodesResponse {
  repeated Node nodes = 1;
  string next_page_token = 2;
}

message NodeFilter {
  repeated string regions = 1;
  NodeStatus status = 2;
}

message Node {
  string id = 1;
  string hostname = 2;
  NodeStatus status = 3;
  google.protobuf.Timestamp created_at = 4;
  google.protobuf.Timestamp last_heartbeat = 5;
  NodeResources resources = 6;

  reserved 7, 8;
  reserved "legacy_provider";
}

message NodeResources {
  int64 cpu_millicores = 1;
  int64 memory_bytes = 2;
  int64 disk_bytes = 3;
}

enum NodeStatus {
  NODE_STATUS_UNSPECIFIED = 0;
  NODE_STATUS_PROVISIONING = 1;
  NODE_STATUS_READY = 2;
  NODE_STATUS_DRAINING = 3;
  NODE_STATUS_OFFLINE = 4;
}

message WatchNodeStatusRequest {
  string node_id = 1;
}

message NodeStatusEvent {
  string node_id = 1;
  NodeStatus previous = 2;
  NodeStatus current = 3;
  google.protobuf.Timestamp occurred_at = 4;
}

A few things worth noting:

Every RPC gets its own request and response messages. Even if GetNodeRequest only has one field today. You will add fields later. If you use a raw string as your request type, you’ve locked yourself out of adding filters, field masks, or anything else without a breaking change.

Enums start with _UNSPECIFIED = 0. Proto3 defaults to zero. If you put a meaningful value at zero, you can’t distinguish “the client explicitly set this” from “the client didn’t set it.” We learned this the hard way with a status enum that defaulted to ACTIVE at zero. Debugging phantom active nodes wasn’t fun.

Reserve removed fields. See that reserved 7, 8 and reserved "legacy_provider"? Those are fields we removed during a redesign. Without the reservation, someone could reuse field number 7 for a completely different type and corrupt data in old clients that still have the old schema cached.

Use Timestamp not int64. I’ve seen people use epoch millis as int64 fields. Looks fine until you’re debugging across time zones and nobody remembers whether it’s seconds or milliseconds or which epoch.

Go server implementation

Here’s a stripped-down but realistic server. This is close to what we actually run:

package main

import (
	"context"
	"log"
	"net"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/health"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/status"

	pb "example.com/infra/api/nodes/v1"
)

type nodeServer struct {
	pb.UnimplementedNodeServiceServer
	store NodeStore
}

func (s *nodeServer) GetNode(ctx context.Context, req *pb.GetNodeRequest) (*pb.Node, error) {
	if req.GetNodeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "node_id is required")
	}

	node, err := s.store.Get(ctx, req.GetNodeId())
	if err != nil {
		return nil, status.Errorf(codes.Internal, "store lookup failed: %v", err)
	}
	if node == nil {
		return nil, status.Errorf(codes.NotFound, "node %q not found", req.GetNodeId())
	}

	return node, nil
}

func (s *nodeServer) WatchNodeStatus(req *pb.WatchNodeStatusRequest, stream pb.NodeService_WatchNodeStatusServer) error {
	if req.GetNodeId() == "" {
		return status.Error(codes.InvalidArgument, "node_id is required")
	}

	events := s.store.Subscribe(req.GetNodeId())
	defer s.store.Unsubscribe(req.GetNodeId(), events)

	for {
		select {
		case <-stream.Context().Done():
			return nil
		case event, ok := <-events:
			if !ok {
				return nil
			}
			if err := stream.Send(event); err != nil {
				return err
			}
		}
	}
}

func main() {
	lis, err := net.Listen("tcp", ":9090")
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}

	srv := grpc.NewServer(
		grpc.ChainUnaryInterceptor(
			loggingInterceptor,
			recoveryInterceptor,
		),
	)

	pb.RegisterNodeServiceServer(srv, &nodeServer{store: NewNodeStore()})

	hsrv := health.NewServer()
	healthpb.RegisterHealthServer(srv, hsrv)
	hsrv.SetServingStatus("infra.nodes.v1.NodeService", healthpb.HealthCheckResponse_SERVING)

	log.Printf("listening on :9090")
	if err := srv.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

A few patterns here that took us a while to settle on:

Embed UnimplementedNodeServiceServer. This is the forward-compatible pattern. When you add a new RPC to the proto, the server still compiles. It returns Unimplemented for the new method until you add the handler. Without this, adding a method to the proto breaks every server binary.

Register the health service. This isn’t optional. Kubernetes liveness and readiness probes need it (through grpc_health_probe, since Kubernetes has no native gRPC probe). Our deploy pipeline rejects services that don’t register the gRPC health check. No exceptions.

Chain interceptors. Logging, recovery, metrics: all cross-cutting concerns go in interceptors, not in every handler. The ChainUnaryInterceptor API landed relatively recently and it’s much cleaner than the old single-interceptor pattern where you’d nest them manually.

Client patterns that don’t break at 3am

The client side is where most people get sloppy. Here’s what we enforce:

func newNodeClient(addr string) (pb.NodeServiceClient, func(), error) {
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	conn, err := grpc.DialContext(ctx, addr,
		grpc.WithTransportCredentials(loadTLSCredentials()),
		grpc.WithDefaultCallOptions(
			grpc.MaxCallRecvMsgSize(4*1024*1024),
		),
		grpc.WithChainUnaryInterceptor(
			retryInterceptor(3, 100*time.Millisecond),
		),
	)
	if err != nil {
		return nil, nil, fmt.Errorf("dial %s: %w", addr, err)
	}

	cleanup := func() { conn.Close() }
	return pb.NewNodeServiceClient(conn), cleanup, nil
}

func getNodeWithDeadline(client pb.NodeServiceClient, nodeID string) (*pb.Node, error) {
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

	node, err := client.GetNode(ctx, &pb.GetNodeRequest{NodeId: nodeID})
	if err != nil {
		st := status.Convert(err)
		switch st.Code() {
		case codes.NotFound:
			return nil, nil // not an error, just missing
		case codes.DeadlineExceeded:
			return nil, fmt.Errorf("node lookup timed out after 3s")
		default:
			return nil, fmt.Errorf("node lookup failed: %s", st.Message())
		}
	}
	return node, nil
}

Always set deadlines. Every single call. No exceptions. A call without a deadline is a goroutine leak waiting to happen. We had an incident where a downstream service hung and our caller accumulated ~40,000 goroutines because nobody set a timeout. The fix was one line. The outage was three hours.

Handle status codes explicitly. Don’t just check err != nil. A NotFound is fundamentally different from Internal. Your retry policy, your alerting, and your user-facing message all depend on the code. status.Convert(err) gives you the code. Use it.

Retry with backoff, but only on the right codes. We retry on Unavailable and DeadlineExceeded. We do not retry on InvalidArgument or NotFound. Retrying a bad request is just a faster way to burn your quota.

Interceptors: the gRPC middleware pattern

This is where gRPC shines versus REST. Interceptors are typed and composable, with separate hooks for unary and streaming RPCs. Here are our unary logging and recovery interceptors:

func loggingInterceptor(
	ctx context.Context,
	req interface{},
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (interface{}, error) {
	start := time.Now()
	resp, err := handler(ctx, req)
	duration := time.Since(start)

	code := codes.OK
	if err != nil {
		code = status.Code(err)
	}

	log.Printf("method=%s code=%s duration=%s",
		info.FullMethod, code, duration)

	return resp, err
}

func recoveryInterceptor(
	ctx context.Context,
	req interface{},
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (resp interface{}, err error) {
	defer func() {
		if r := recover(); r != nil {
			log.Printf("panic in %s: %v", info.FullMethod, r)
			err = status.Errorf(codes.Internal, "internal error")
		}
	}()
	return handler(ctx, req)
}

The recovery interceptor has saved us more than once. A nil pointer dereference in a handler used to crash the entire server. Now it returns Internal and logs the panic. The server stays up. Other RPCs keep working. We still fix the bug, but we don’t page the on-call team at 3am for a panic in a non-critical endpoint.

Error handling: the part everyone gets wrong

I’ve seen codebases where every gRPC error is codes.Internal. That’s like every HTTP response being a 500. Useless for clients. Useless for monitoring.

Our rule is simple: pick the status code that tells the client what to do.

Internal means “the server has a bug.” If you’re returning Internal for a missing resource or bad input, you’re lying to your monitoring system. Your alerts will fire for things that aren’t server bugs. Alert fatigue follows.

Proto evolution: the non-obvious rules

Adding fields is safe. Removing them isn’t. That much is obvious. The non-obvious part:

Changing a field from string to bytes is wire-compatible but semantically different. We did this once with a field that held a UUID. Wire format was identical. But the generated Go code changed from string to []byte and broke every caller at compile time. “Wire compatible” and “API compatible” are different things.

Removing a field without reserved is a time bomb. Six months later, someone reuses that field number for a different type. Old clients that haven’t recompiled send the old type. The new server interprets it as the new type. Data corruption that only manifests in production with old client versions. Good luck debugging that.

Version at the package level. It’s the same idea as URL path versioning for REST . When you need a breaking change:

// Old clients still work
package infra.nodes.v1;

// New clients use v2
package infra.nodes.v2;

Run both versions side by side. Migrate clients one at a time. Kill v1 when the last client is gone. We track this with a Grafana dashboard that shows request counts per proto package version.

Testing: skip the mocks

We don’t mock gRPC clients. We spin up a real grpc.Server in tests with bufconn:

func setupTest(t *testing.T) pb.NodeServiceClient {
	t.Helper()

	lis := bufconn.Listen(1024 * 1024)
	srv := grpc.NewServer()
	pb.RegisterNodeServiceServer(srv, &nodeServer{store: NewMemoryStore()})

	go func() { srv.Serve(lis) }()
	t.Cleanup(func() { srv.GracefulStop() })

	conn, err := grpc.DialContext(
		context.Background(), "",
		grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
			return lis.DialContext(ctx)
		}),
		grpc.WithInsecure(),
	)
	if err != nil {
		t.Fatalf("dial bufconn: %v", err)
	}
	t.Cleanup(func() { conn.Close() })

	return pb.NewNodeServiceClient(conn)
}

func TestGetNode(t *testing.T) {
	client := setupTest(t)

	// Happy path
	node, err := client.GetNode(context.Background(), &pb.GetNodeRequest{NodeId: "node-1"})
	if err != nil {
		t.Fatalf("unexpected error: %v", err)
	}
	if node.GetHostname() != "worker-01.example.com" {
		t.Errorf("hostname = %q, want %q", node.GetHostname(), "worker-01.example.com")
	}

	// Not found
	_, err = client.GetNode(context.Background(), &pb.GetNodeRequest{NodeId: "nonexistent"})
	if status.Code(err) != codes.NotFound {
		t.Errorf("code = %v, want NotFound", status.Code(err))
	}

	// Validation
	_, err = client.GetNode(context.Background(), &pb.GetNodeRequest{})
	if status.Code(err) != codes.InvalidArgument {
		t.Errorf("code = %v, want InvalidArgument", status.Code(err))
	}
}

bufconn gives you an in-memory listener. No ports, no flaky tests from port conflicts, no network overhead. But you still test the full gRPC stack: serialization, interceptors, status codes, deadlines. Mocking the client interface skips all of that. We found real bugs with bufconn that mocks would have hidden: serialization of oneof fields, deadline propagation through interceptors, and metadata handling.

Load balancing: the HTTP/2 gotcha

gRPC runs on HTTP/2, which multiplexes requests over a single TCP connection. This means traditional L4 load balancers don’t work. All requests go to whichever backend received the connection. One hot server, N-1 idle servers.

We use client-side balancing with a service registry. The gRPC resolver API lets you plug in your own discovery:

conn, err := grpc.Dial(
	"dns:///nodes.example.internal:9090",
	grpc.WithTransportCredentials(loadTLSCredentials()),
	grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)

The dns:/// scheme tells the resolver to use DNS. round_robin distributes across all A records. For us this works because we run headless Kubernetes services that return pod IPs.

If you’re behind Envoy or Istio , make sure they’re configured for HTTP/2, not HTTP/1.1 upgrade. I’ve seen service meshes silently downgrade gRPC to HTTP/1.1 and nobody notices until streaming breaks.

What I’d do differently

Eight months in, this is what I’d change if we started over:

Invest in proto linting from day one. We use buf now but adopted it late. Early protos have inconsistent naming, missing field reservations, and enum values that don’t follow the TYPE_NAME_VALUE pattern. Fixing these in a live system is painful.

Put buf next to protoc from the start. The protoc plugin ecosystem is a maze, and we still generate code with it. But buf handles linting and breaking change detection in one tool. Should have started there.

Don’t over-decompose services . Our first instinct was one proto per entity. Node service. Deployment service. Billing service. Network service. That’s fine in theory. In practice, most operations touch three or four services, so every user-facing action became a cascade of RPCs. We’ve since consolidated the ones that always move together.

gRPC is a great tool for internal APIs. Use it where it helps and REST where that’s simpler.

What happened: organizations sized their VPN for maybe 20% concurrent usage. Sales people on the road. The odd remote day. Then overnight, 100% of the workforce went remote and stayed there. Five times the sessions, five times the bandwidth, encryption overhead pegging every CPU on the gateway. Licensing servers falling over because nobody ever imagined needing that many concurrent seats.

I’m not going to pretend this was some unforeseeable event. It wasn’t. We all knew these VPN setups were fragile. We just got away with it because the failure condition never arrived. Until it did.

Everything broke at once

The fun part about VPN failures is how they cascade. It’s never just one thing.

License checks hit a ceiling first. Then the gateway CPU maxes out from encryption overhead. Then someone notices that all SaaS traffic (Teams, Slack, G Suite, everything) is hairpinning through the data center because full-tunnel was the default. So now your WAN link is saturated too. Then RADIUS or your directory service starts timing out because it was never load-tested for this volume. Then MFA starts queuing.

Meanwhile, you have one gateway and one ISP, because “we’ll add redundancy next quarter” has been on the roadmap for three years.

The help desk tickets pile up. Leadership asks why people can’t work. You explain that the VPN was designed for a different world. Nobody wants to hear that.

“Just scale the VPN” is the wrong answer

This is where I get annoyed. The knee-jerk response I keep seeing is: buy more licenses, add more gateways, throw bandwidth at it. And yes, that stops the bleeding. We did it too. But if your takeaway from this crisis is “we need a bigger VPN,” you’ve missed the point entirely.

VPNs grant network-level access. You connect, you’re on the network, you can reach things. The implicit trust model is insane if you think about it for more than thirty seconds. An employee’s compromised laptop gets VPN access and now it’s sitting on your corporate network with lateral movement potential. That was always the problem. COVID just made it impossible to ignore because now every employee’s home network is your attack surface. The same pressure is wrecking cloud security defaults too.

The “just scale it” crowd is essentially saying: let’s take this fundamentally flawed security model and make it bigger. No thanks.

Split tunneling buys time, nothing more

I’ll give split tunneling credit for being the single most impactful quick fix. Stop routing SaaS traffic through your data center. Keep internal resources on the tunnel, let everything else go direct. VPN load drops dramatically.

But look at what you’re doing. You’re acknowledging that the VPN was never needed for half the traffic flowing through it. Which should make you ask: what else doesn’t actually need to be on the VPN?

The answer, for most organizations, is almost everything. Most of your apps are already SaaS or could be accessed through an identity-aware proxy. The VPN is protecting a shrinking set of truly internal resources while adding overhead and fragility to everything else.

Where this actually needs to go

Zero trust . Identity-based access. Whatever you want to call it, it’s the model where you authenticate and authorize at the application layer instead of the network layer. Where a compromised device doesn’t get you lateral movement because there’s no implicit network trust to exploit.

I realize I sound like a vendor pitch right now, and I hate that. But the principle is sound even if the marketing around it’s insufferable. Stop granting network access. Start granting application access. Verify the device, verify the identity, verify the context, grant access to exactly what’s needed.

You don’t have to rip out VPN tomorrow. That’s unrealistic. But you should be moving your most critical applications behind identity-aware access right now, and treating VPN as the legacy fallback for the stuff you haven’t migrated yet. Not the other way around.

What we did on that team

Short term, the same ugly stuff everyone else did. Activated emergency licenses. Deployed additional gateways. Enabled split tunneling with endpoint posture checks. Added a second ISP. Staggered login windows to smooth the peaks.

None of it was elegant. All of it was necessary.

But the real work started after the fires were out. Moving applications behind identity-based access. Treating VPN as a shrinking perimeter, not an expanding one. Setting up monitoring that warns you before users start complaining (session counts trending up, auth failure rates, gateway CPU) instead of post-mortem log analysis.

The goal is simple: every month, fewer things should require VPN. If that number isn’t going down, you’re just waiting for the next crisis to expose the same structural weakness all over again.

I’m watching companies triple their AWS footprint in weeks right now. Entire engineering teams went remote overnight. New services are shipping without review. IAM policies are getting * slapped on them because someone needed to unblock a deploy at 11pm.

I’ve seen this pattern before, in cyber-defense exercises rather than startups. The scenario is always the same: operational tempo increases, shortcuts happen, and adversaries walk through the gaps those shortcuts create. The difference is that in a cyber-defense exercise, you get a debrief. In production, you get a breach notification.

At the cloud infrastructure company I run, I build tooling for companies that are already scaling hard. The ones I’ve seen get through it without security incidents made the secure path the easy path. Security team size has little to do with it. That’s the entire thesis of this post.

Permissive Defaults Are the Problem

Everyone wants to blame the pace. “We’re moving too fast to do security properly.” Nonsense.

The problem is that most cloud environments are configured to be permissive by default. When things are calm, humans manually tighten things up. Reviews happen. Someone catches the overly broad IAM role before it ships. But when you’re scaling fast, those manual catches disappear. And now you’re running on whatever the defaults are.

If your defaults are permissive, you’re exposed. Period.

I dealt with this at the fintech startup when we were scaling our data infrastructure. More process didn’t fix it. Better defaults did.

IAM: Where Most Breaches Actually Start

I’ll say it bluntly: if you’re not using IAM permissions boundaries right now, you’re gambling. Permissions boundaries are the ceiling on what any role can do, regardless of what policies are attached to it. They’re the single most underused security control in AWS.

Here’s a permissions boundary I enforce on every account that creates IAM roles:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSpecificServices",
      "Effect": "Allow",
      "Action": [
        "s3:*",
        "dynamodb:*",
        "sqs:*",
        "sns:*",
        "logs:*",
        "cloudwatch:*"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyIAMEscalation",
      "Effect": "Deny",
      "Action": [
        "iam:CreateUser",
        "iam:CreateRole",
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy",
        "iam:CreateAccessKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyOrganizationChanges",
      "Effect": "Deny",
      "Action": "organizations:*",
      "Resource": "*"
    }
  ]
}

Even if a developer attaches AdministratorAccess to their role (and they will, at 11pm, to unblock a deploy), the permissions boundary prevents privilege escalation. They can’t create new roles, attach policies, or generate access keys. The blast radius is capped.

A common shortcut is giving every Lambda execution role full S3 access because “it’s easier.” Then a single compromised function can read every bucket in the account. Permissions boundaries would contain that to the specific service’s buckets.

Security Groups: The 0.0.0.0/0 Problem

I run a script on every new project that searches for 0.0.0.0/0 in security group ingress rules. It always finds something. Always.

Here’s the Terraform pattern I enforce in shared modules :

resource "aws_security_group" "app" {
  name_prefix = "app-"
  vpc_id      = var.vpc_id

  # No inline ingress rules. Force explicit definition.
  # This is intentional: default deny.

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "HTTPS outbound only"
  }

  lifecycle {
    create_before_destroy = true
  }

  tags = {
    ManagedBy = "terraform"
    Review    = "required"
  }
}

# Separate rule resources: auditable, reviewable, deletable
resource "aws_security_group_rule" "app_from_alb" {
  type                     = "ingress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.alb.id
  security_group_id        = aws_security_group.app.id
  description              = "ALB to app on 8080"
}

Notice: no ingress defined inline. Every ingress rule is a separate aws_security_group_rule resource with a description field. When someone adds a rule, it shows up clearly in the Terraform plan. When you need to audit, you can grep your codebase for aws_security_group_rule and get a complete inventory. Try doing that with inline rules.

This pattern came directly from a cyber-defense exercise where we had to rapidly audit network access during a simulated attack. The team that had structured, auditable firewall rules found the compromised path in minutes. The team with ad-hoc rules was still mapping their network topology when the exercise ended.

Make Encryption Non-Negotiable

Every data store module should enforce encryption. Not suggest it. Enforce it.

resource "aws_db_instance" "main" {
  publicly_accessible    = false
  storage_encrypted      = true
  deletion_protection    = true
  kms_key_id             = var.kms_key_arn
  iam_database_authentication_enabled = true

  # No default security group: force explicit assignment
  vpc_security_group_ids = [var.db_security_group_id]
}

resource "aws_s3_bucket" "data" {
  bucket_prefix = "app-data-"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm     = "aws:kms"
        kms_master_key_id = var.kms_key_arn
      }
    }
  }
}

resource "aws_s3_bucket_public_access_block" "data" {
  bucket = aws_s3_bucket.data.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

The S3 public access block is the one I see missing most often. People enable encryption but forget to block public access at the bucket level. Then someone uploads an object with a public ACL and you’re in the news.

Detection: Turn It On Before You Scale

This is the mistake I see repeatedly. Teams add services, then plan to “add monitoring later.” Later never comes.

Before you provision a single new service, these should be running across every account in your organization:

# AWS Config rules: deploy via StackSets across all accounts
config_rules:
  - s3-bucket-public-read-prohibited
  - s3-bucket-public-write-prohibited
  - encrypted-volumes
  - cloudtrail-enabled
  - root-account-mfa-enabled
  - iam-user-no-policies-check
  - iam-root-access-key-check
  - restricted-ssh
  - vpc-flow-logs-enabled

CloudTrail, Config, and GuardDuty. All three. Non-negotiable. GuardDuty in particular has gotten good enough that it catches IAM anomalies (unusual API calls, credential usage from unexpected IPs) with minimal tuning.

The trick is filtering. GuardDuty will generate findings. Most of them are low severity. Set up an SNS topic that only fires on HIGH and CRITICAL findings. Pipe those to a Slack channel or PagerDuty. Everything else goes to a weekly review.

Treat this as a baseline for every account. The teams that actually review the weekly findings catch configuration drift before it becomes exploitable. The teams that ignore them lose that early warning.

Remote Access: Kill the VPN, Use Session Manager

Everyone’s spinning up VPNs right now. I get it. But VPNs are a liability. They’re a single point of trust. Once you’re on the VPN, you’re “inside.” That model is broken.

For AWS access, use SSM Session Manager instead. It gives you shell access to EC2 instances through the AWS API. No SSH keys to manage. No inbound security group rules. Full session logging to CloudTrail and S3.

For application access, put services behind an identity-aware proxy. Verify identity and device state on every request. The network boundary isn’t your security boundary. That’s the core of zero trust .

I learned this the hard way during a cyber-defense exercise. The red team’s first move was always to compromise VPN credentials. Once inside, they had lateral movement across the entire network. The teams that had segmented access (per-service authentication, no implicit trust from network position) held up. The teams running flat networks behind a VPN got owned.

The Seven-Item Checklist

I keep this short because nobody follows long checklists under pressure.

1. IAM permissions boundaries on every account that creates roles. Deploy today.
2. S3 public access blocks on every bucket. aws s3api put-public-access-block across all accounts. Takes five minutes.
3. Security groups: audit for 0.0.0.0/0 ingress. Remove or justify every one.
4. CloudTrail + Config + GuardDuty: enabled in every account, every region. Use AWS Organizations and StackSets.
5. Encryption by default: KMS keys provisioned, Terraform modules enforce storage_encrypted = true and publicly_accessible = false.
6. SSM Session Manager instead of SSH. Kill inbound port 22.
7. Weekly access review: who has access to what, and why. Remove what’s not justified.

That’s it. Seven items. You can get through most of them in a day if you’re motivated.

Stop Treating Security as a Phase

The framing I keep hearing is “we’ll fix security after we stabilize.” That’s backwards. Security is a property of how you scale, and you don’t get to it afterwards.

If your Terraform modules are secure by default, every new service you deploy is secure. If your IAM boundaries are in place, every new role is constrained. If your detection is running, every new account is monitored.

The work is in the defaults. Get those right and scaling fast stays scaling fast instead of scaling into a breach.

Two weeks ago, someone called me in a panic. Their video platform, built for 10,000 concurrent users, was handling 80,000. SFUs at 94% CPU. TURN servers hemorrhaging bandwidth. The whole thing held together by one engineer who hadn’t slept in three days.

This is March 2020. I’ve had four conversations like this in the last ten days. The pattern is always the same: the architecture was fine for the old world, and the old world ended overnight.

What follows is what I keep telling these teams: the decisions that matter when your video infra is on fire.

Video Isn’t HTTP

I keep meeting backend engineers who treat video like a REST API with bigger payloads. It’s not.

A video call is a continuous bidirectional stream that has to feel natural to a human brain. That brain notices latency above ~150ms. It notices audio glitches before video glitches. It definitely notices when your echo cancellation is broken and everyone hears themselves with a half-second delay.

• Latency under 150ms or the conversation feels wrong. People start talking over each other.
• Audio is sacred. Users tolerate blurry video for minutes. They hang up after five seconds of choppy audio.
• NAT traversal is the norm. Corporate firewalls, hotel wifi, mobile networks: assume the worst.
• Packet loss kills you silently. Bandwidth can look fine while jitter makes the whole thing unwatchable.

P2P, SFU, MCU: Pick the Right One

I see teams agonize over this choice like it’s a religious decision. It shouldn’t be.

P2P works for 2-4 people. Direct connection, lowest latency, almost no server cost. The catch: every participant uploads their stream to every other participant. Past 4-5 people, uplinks die.

SFU (Selective Forwarding Unit) is the right answer for almost everything else. It receives streams and forwards them selectively. No mixing, no transcoding on the server. This is what Zoom, Google Meet, and basically every serious platform runs.

MCU (Multipoint Control Unit) mixes everything into a single composed stream per participant. Sounds nice. In practice, the server-side CPU and GPU cost is brutal. Only for very low-capability clients or massive broadcast rooms.

Most production systems are hybrid. One system I worked on ran P2P for 1:1 calls and SFU for everything larger. Simple rule, huge savings.

The rough guidelines I give every team:

• 2-4 people: P2P. Don’t overthink it.
• 5-20 people: SFU with simulcast.
• 20+ people: SFU with aggressive active-speaker optimization.
• Webinars / broadcasts: SFU for the audience, dedicated pipeline for presenters.

Scaling SFUs Without Losing Your Mind

SFUs scale horizontally in theory. In practice, calls are stateful. Every participant in a room has to land on the same SFU instance. This isn’t a stateless web tier you can throw behind a load balancer.

Session stickiness first. Your load balancer needs to understand rooms, not just connections. Get this wrong and you’ll spend days debugging “ghost participants” and one-way audio.

Room size caps are a safety valve. One team removed their 50-person cap “because sales wanted it” and promptly discovered a single 200-person room could saturate an SFU. Cap it. Enforce it. Explain to sales later.

Cascading SFUs across regions is how you handle distributed calls without routing Sydney’s traffic through Virginia. Each region gets its own SFU. They talk over backbone links. Users connect locally.

Sydney user -> Sydney SFU <--backbone--> London SFU <- London user

This keeps last-mile latency low and contains failure domains. When the Sydney SFU has issues, London keeps running.

Simulcast Is Non-Negotiable

If you’re running an SFU without simulcast, stop reading and go implement it. I’m serious.

Clients send multiple quality layers simultaneously:

simulcast:
  high: 1280x720 @ 30fps, ~1500kbps
  mid:  640x360 @ 30fps, ~700kbps
  low:  320x180 @ 15fps, ~200kbps

The SFU forwards the appropriate layer per receiver based on bandwidth and screen size. Active speaker gets high quality. Thumbnail tiles get low. Someone on hotel wifi gets mid or low for everything.

Without simulcast, you’re either crushing constrained clients with full-resolution streams or wasting bandwidth by downscaling everything. No middle ground.

Pair simulcast with adaptive bitrate control. When the SFU detects rising packet loss or sustained jitter, it signals the sender to drop resolution. Fastest lever you have to stabilize a degrading call.

The TURN Server Problem

TURN is the thing nobody budgets for and everybody needs.

STUN lets peers discover their public IP and try a direct connection. When that fails (corporate firewall, symmetric NAT, hotel wifi blocking UDP), you need TURN. TURN relays all media through your server. Every stream hits your infra twice (in and out). At scale, it becomes your single largest cost.

I worked with a team whose TURN relay ratio was 40%. Their bandwidth bill was obscene. We got it down to 15% by improving ICE candidate gathering, adding more TURN regions, and implementing better network detection on the client side.

Monitor your TURN relay ratio religiously. Anything above 20% means something is wrong with your connectivity story. More TURN regions with latency-aware routing helps. Anycast or geo-DNS for region selection is table stakes.

What to Actually Monitor

I’ve walked into too many video platforms where the dashboard shows CPU, memory, and “active calls.” That tells you nothing about whether calls are good.

The metrics I insist on:

Time to first media. How long from clicking “join” until the user sees/hears someone? This is the single most important UX metric. If it’s over 3 seconds, people think it’s broken.

Join success rate. What percentage of join attempts actually result in a connected call? I’ve seen rates as low as 85%. That’s 15% of users clicking join and getting nothing.

Packet loss and jitter per stream. Not aggregated across the SFU. Per stream. You need to know which participants are suffering.

TURN relay ratio. Track it. Trend it. Freak out if it’s climbing.

SFU CPU and egress bandwidth. Per instance, not averaged. One overloaded SFU ruins everyone on it while the average looks fine.

Operating It Under Pressure

The architecture is the easy part. The hard part is operating it under pressure.

You need admission control. When an SFU is at 80% CPU, stop putting new rooms on it. Almost nobody does this until after their first outage.

You need graceful degradation . When things get bad, drop to audio-only automatically. A stable voice call beats a stuttering video call every time.

You need fast regional failover. When a region goes down, redirect new joins to the nearest healthy region. Don’t wait for someone to notice at 2 AM.

And you need someone who owns this. Not “the platform team” collectively. A specific person who gets paged when quality drops. The team I mentioned at the start had a bigger ownership problem than an architecture problem: nobody owned the system end-to-end until it was already on fire.

Where This Goes

We’re two weeks into what I think is a permanent shift. The companies that figure out SFU scaling, simulcast, and TURN optimization now will have a massive advantage.

The ones still treating video as a feature rather than infrastructure will keep making panicked calls at 11 PM.

What actually makes sense

Wasm outside the browser has a few properties that are hard to get anywhere else. Near-instant cold starts. True sandboxing where the host decides exactly what the guest can touch. Tiny binaries you can ship anywhere. If you’re running short-lived functions at the edge or loading untrusted plugin code, these properties matter more than anything Docker gives you.

Edge computing is the obvious winner here. Cloudflare Workers already proved the model: run user code in tight sandboxes, spin up fast, tear down fast. Wasm formalizes that. Plugin systems are the other sweet spot. I’ve dealt with enough “just run their code in a subprocess” architectures to appreciate a sandbox that doesn’t require a container per tenant.

The gaps: WASI, threads, Go support

WASI networking is basically nonexistent right now. You can read files if the host lets you. That’s about it. Threading support is patchy. Debugging is painful. If your service needs to open a socket or talk to a database, you’re going to have a bad time.

I’ve been following the Go side of this closely. Go can compile to Wasm, but only for the browser, and the binaries are big. TinyGo gets you much smaller binaries, but it has no WASI target yet, and it’s a subset of Go. Not all standard library packages work. Reflection is limited. You hit walls fast. Rust is in a much better position here: it already has a WASI target and the tooling is good. Go’s story will get there, but we’re not there yet.

The hype I’m skeptical of

Blockchain smart contracts in Wasm. Replacing Kubernetes with Wasm runtimes. Running your entire backend as Wasm modules. These are conference talk ideas, not production architectures. The gap between “cool demo” and “thing I’d run at 3am” is enormous.

I watched the same pattern play out with microservices. The concept was sound. The early adopters built real things. Then everyone decided they needed it for their CRUD app and created distributed monoliths. Wasm outside the browser is at the “concept is sound” stage. Let’s not skip ahead.

What I’m actually doing with it

Experimenting. I compiled a small data transformation service to Wasm and ran it in Wasmtime. Cold start was impressive: under a millisecond. Binary was tiny. But I hit the WASI wall the moment I needed HTTP. Ended up passing data through stdin/stdout like it was 1985.

The capability-based security model is the part I keep coming back to. A process that can’t access anything you didn’t hand it beats one you trust not to escape. That’s a better model for multi-tenant anything.

When I’d use it

For edge functions, plugin systems, and sandboxed execution, Wasm outside the browser is usable today. For everything else, it’s a promising prototype. I’d start experimenting now, but I wouldn’t migrate anything critical until WASI networking lands and the tooling matures. That’s not happening next quarter.

I broke production with a Terraform typo last year. Changed a security group rule, fat-fingered a CIDR block, and opened port 22 to the internet for about forty minutes. Nobody caught it in review. Our CI pipeline at the time was terraform plan and vibes.

That was the wake-up call. I spent the next six months building out infrastructure testing at my infrastructure startup, trying every approach I could find. Unit tests for IaC. Policy engines. Full integration runs with real resources. Some of it was worth the effort. Some wasn’t.

Three Approaches, Three Different Jobs

The mistake I see most teams make is treating “infrastructure testing” as one thing. There are three fundamentally different approaches, and they catch fundamentally different problems.

That table is the whole thesis. Each layer fills in what the previous one misses.

Static and Unit Tests: The Fast Floor

These are your terraform fmt, terraform validate, tflint, kubeval checks. They run in seconds. They cost nothing. They should be on every single commit.

terraform fmt -check -recursive
terraform validate
tflint terraform/
kubeval --strict k8s/*.yaml

What surprised me: these catch more than you’d think. At my infrastructure startup, roughly 40% of our infra PRs had at least one issue caught at this layer. Mostly formatting, occasionally an actual invalid reference or deprecated resource argument.

What they won’t catch: anything that’s syntactically valid but semantically wrong. That security group I opened to the world? Perfectly valid HCL. terraform validate had no opinion.

Policy Tests: Where the Real Value Is

This is the layer I underestimated. OPA with Conftest lets you write rules against Terraform plan JSON, Kubernetes manifests, basically any structured config.

The SSH rule that would have saved me forty minutes of panic:

package terraform

deny[msg] {
  r := input.resource_changes[_]
  r.type == "aws_security_group_rule"
  r.change.after.cidr_blocks[_] == "0.0.0.0/0"
  r.change.after.from_port == 22
  msg := "SSH must not be open to the world"
}

Small. Specific. Tied to an actual incident. That’s the pattern.

I tried building comprehensive policy libraries. Dozens of rules covering every possible misconfiguration. Bad idea. Teams bypass large generic policy sets. They add exceptions, disable checks, or just stop running them. The policies that stick are the ones written after something went wrong. Each one represents a scar.

We also run Checkov as a complement. It only came out in December, but its list of built-in checks for common misconfigs is growing fast. Good for catching things you haven’t been burned by yet. But it’s a floor, not a ceiling.

My rule of thumb: if a scanner catches it, great. If it doesn’t, and it bit you, write a policy for it. Your policy library should grow from incidents, not from a wish list.

Integration Tests: Expensive and Worth It (Sometimes)

Integration tests create real AWS resources, run assertions against them, then tear everything down. Terratest is what we use. Go-based, works well with our existing stack.

These are the only tests that catch networking and IAM problems. You can validate a Terraform plan all day and it won’t tell you that your NAT gateway routing is broken or that your RDS instance can’t actually reach the VPC endpoint it needs.

But they’re slow. Our VPC module integration test takes about twelve minutes. Our EKS test takes twenty-five. And they cost money. Not a lot per run, but it adds up if you’re running them on every PR.

We don’t integration test every module. That’s the trap. Test the ones where a static check can’t tell you if it works. Our S3 bucket module? Unit tests and policy checks are enough. Our networking module? Needs the real thing.

The cleanup problem is real. Terratest has defer terraform.Destroy() but it’s not bulletproof. We’ve had orphaned resources from failed tests sit around for weeks. I wrote a nightly cleanup job that tags test resources and nukes anything older than 24 hours. Unsexy but necessary.

How We Wire It Together

Our CI pipeline at my infrastructure startup:

1. Pre-commit hooks: terraform fmt, basic linting. Runs locally in seconds.
2. PR checks: terraform validate, tflint, Conftest policies, Checkov. Blocks merge on failure. Takes about 90 seconds.
3. Nightly on main: Integration tests for critical modules. Results posted to Slack. Takes about 45 minutes total.
4. Post-deploy: Smoke tests hitting real endpoints. Health checks, latency thresholds, basic connectivity. Alerts on failure.

Fast checks on every commit, expensive checks on a schedule. That split is what makes the pipeline scale with the codebase . I’ve seen teams try to run Terratest on every PR and give up within a month because the feedback loop is too slow.

Practical Stuff That Matters

A few things I wish someone had told me six months ago:

• Tag everything with a test identifier. When tests fail mid-run and leave resources behind, you need to find them.
• Set hard cost limits. We capped our test AWS account at $200/month. Hit it once. Fixed the test that was spinning up m5.4xlarge instances.
• Never put real credentials in test fixtures. Sounds obvious. I’ve reviewed code where people did it anyway.
• Store plan outputs. When something breaks in production, being able to diff against the last tested plan is invaluable.

Where We Stand After Six Months

Static checks are table stakes. There’s no excuse not to have them. Policy tests are the highest ROI layer and the one most teams skip. Integration tests are worth it for complex modules but not for everything.

If I were starting from scratch tomorrow, I’d add static checks on day one, write my first policy rule on day two after reading our incident history, and add integration tests only for the modules that scare me.

Perfect coverage was never the goal. The goal is knowing that the categories of mistakes that have hurt you before won’t hurt you again.

Leaving the fintech startup

I spent a good stretch at a fintech startup. Good team. Interesting problems. We built systems that handled enormous volumes of financial data with tight latency requirements. I learned a lot about what it means to run infrastructure that can’t go down. When money is involved, a 99.95% uptime target stops being an academic exercise and starts being your weekend.

But by early 2019, I knew I wanted to build something of my own. Not because the fintech startup was the wrong place. It was the right place for the stage I was in. The itch was something else: I wanted to own the problem, not just the solution.

So I left.

Joining a founder program

I joined a founder program’s cohort this year. The expectation is that you find a co-founder and build a company. In weeks.

It’s as intense as it sounds.

The previous year I’d gone through a startup accelerator , which was a great experience but structured differently. The accelerator gives you resources and mentorship around an existing idea. The program strips you back to zero. You walk in with credentials and walk out with either a company or a very clear understanding of why you didn’t start one.

What the program teaches you fast: your technical skills are table stakes. The hard part is finding someone whose strengths complement yours, whose risk tolerance matches, and who you can argue with productively at 11pm on a Tuesday. The co-founder search is brutal. You’re essentially speed-dating for the most important professional relationship of your life.

Founding a company

Out of the program, I founded a cloud infrastructure company on my own. The thesis was straightforward: cloud infrastructure is absurdly complex, and most teams are paying for complexity they don’t need. I wanted to build tooling that made cloud operations simpler without dumbing them down.

This wasn’t an abstract idea for me. I’d spent years in Go, contributing to open-source projects and building cloud-native systems. I’d watched teams at the startups where I’d worked struggle with the same problems. Kubernetes was maturing, sure. But “maturing” in practice meant the operational tax was becoming clearer, not smaller. Managed control planes helped, but day-2 operations (upgrades, policy, incident response) were still eating engineering time that should have gone to product work.

The company was born from that frustration. I wasn’t trying to replace Kubernetes or any specific tool. I was trying to reduce the cognitive load of operating in the cloud.

Starting a company from scratch is humbling, even the second time. At the fintech startup, I inherited systems, processes, a team. At the new company, I had a laptop and a pitch deck that needed rewriting every week. The feedback loops are different. The loneliness is real. And the decisions hit different when it’s your name on the incorporation documents.

What I Learned This Year

Building from zero versus building on top of something are fundamentally different skills. I had founded a startup before, but the years since had gone into building technology inside other companies. Building the company itself is not the same thing, and I had to relearn it.

The industry’s complexity problem is real and getting worse. I watched the cloud-native ecosystem grow more powerful and more impenetrable simultaneously. TypeScript was becoming the default for frontend at scale. Go was dominating infrastructure tooling. Rust was expanding beyond systems niches. Good trends, all of them. But the surface area of “things you need to know to ship a product” kept expanding.

Simplicity is a harder sell than it should be. The “monolith first” argument went mainstream in 2019. People started publicly defending boring technology choices. But in practice, teams still reached for distributed architectures before they needed them. Resume-driven development is a powerful force.

The best founders I met in the program weren’t the smartest people in the room. They were the most honest about what they didn’t know. That’s a different skill than intelligence, and it’s harder to fake.

What I Noticed in the Industry

A few things I noticed from the outside looking in, which shaped how I thought about the company:

Service meshes gained real production usage but proved heavy. The value was clear for large distributed systems: mutual TLS, canary deployments, consistent observability. For everyone else, it was another layer to debug.

Edge computing found its feet. CDN providers turned their points of presence into compute platforms, and real workloads started running there: personalization, auth at the perimeter, feature flagging close to the user. For most teams it’s still premature optimization , but the stateless CDN cases are real.

GitOps moved from blog posts to actual deployment models. Git as the single source of truth for environments. Pull-based deployments. Declarative infrastructure. This was the kind of operational maturity the industry needed.

Security conversations got more concrete. Zero trust shifted from strategy decks to implementation work. High-profile cloud incidents, above all the Capital One breach disclosed in July, drove home that configuration risk is as real as application risk. IAM policies are code, and they deserve the same rigor.

Remote work kept becoming more normal, at least for engineering teams.

What I’m Carrying Into 2020

I’m ending 2019 as a founder again. A year ago I was at a fintech company. Now I’m trying to build something from nothing, with the conviction that cloud infrastructure can be simpler and the awareness that conviction alone doesn’t ship products.

The things that stayed hard: operating Kubernetes well, managing dependency sprawl, treating security as engineering rather than compliance theater, and tying cloud costs back to architecture decisions. These are the problems I’m now trying to solve professionally, which means I get to be frustrated by them in a whole new way.

2019 wasn’t a comfortable year. It was a good one. There’s a difference.

At a mobility startup we processed GPS telemetry from thousands of vehicles across the city. Real IoT. Real device data. Real volume. You’d think that’s a textbook edge case. It wasn’t. A well-tuned message queue and a couple of regional cloud instances handled everything we needed. The same logic applies to multi-region : add geography when measurements demand it. Sub-second was easy without edge infrastructure.

The edge is a spectrum

Device. Network edge. Regional edge. Cloud. Each hop adds latency, sure. But each hop closer to the device also adds operational pain: deployment complexity, consistency headaches, debugging in the dark.

The math only works when you have a genuine constraint:

• Latency below 50ms on a critical path. Not aspirational latency. Measured, user-impacting latency.
• Bandwidth costs that actually hurt. If you’re shipping raw video or sensor streams, processing locally makes sense. If you’re sending JSON, it doesn’t.
• Unreliable connectivity. Factory floors, moving vehicles, rural deployments. The network genuinely can’t be trusted.
• Data residency requirements. Regulation says the data stays put. Fine. That’s a real constraint.

If none of those apply, you’re adding distributed systems complexity for a problem you don’t have.

What edge computing costs you

Consistency goes out the window. You’re now designing for eventual convergence, conflict resolution, and partial failures at every node. Your deployment pipeline needs to handle hundreds of locations instead of a handful of regions. Observability becomes sampling and aggregation because you can’t ship raw telemetry from every edge node without defeating the purpose.

Security surface area multiplies. Every edge node is a potential compromise point. You’re trusting code running in locations you don’t physically control.

Now I’m building cloud infrastructure tooling at my own startup. I see teams adopt edge patterns and then spend months building the operational scaffolding that a centralized architecture gives you for free. The edge didn’t solve their problem. It replaced one set of problems with a harder set.

When it’s worth it

CDN edge compute is the one place where the tradeoff is almost always favorable. Auth checks, request routing, and simple personalization are stateless, short-lived operations running on infrastructure someone else manages. Cloudflare Workers, Lambda@Edge, and similar products nailed this by keeping the programming model simple and the blast radius small.

Beyond CDN, edge computing earns its complexity in exactly three scenarios: real-time industrial control, high-volume media processing at the source, and latency-sensitive interactive applications where every millisecond is measurable in revenue.

Everything else? Run it in the cloud. Optimize your queries. Use a CDN. Treating edge as a default is resume-driven development dressed up as forward thinking. Start with the simplest thing that works, measure where it doesn’t, and push compute to the edge only when the numbers force your hand.

By the time I started building my own infrastructure startup in a founder program this year, I’d already learned this lesson the hard way. The problem is never Terraform itself. The problem is that infrastructure grows quietly. You add a few resources here, a new environment there, and one morning you run terraform plan and it takes eleven minutes and touches things you forgot existed. That’s the moment you realize your Terraform setup doesn’t scale. And by then, untangling it is painful.

Stop dumping everything into one state file. Split by domain, pin everything, automate plan-and-apply, and treat -target like a fire extinguisher. The rest of this post is what I wish I’d known before the first time a plan returned 200+ changes because someone refactored a module without understanding what depended on it.

The breaking point

Small Terraform setups are a joy. You describe what you want, you apply it, it exists. Beautiful.

At scale, the problems are predictable:

• Plans take forever. API rate limits turn a 30-second plan into a 10-minute gamble.
• State becomes a bottleneck. Two engineers run apply at the same time. One wins, one gets a lock error and has to wait.
• Blast radius grows silently. A change to a security group touches the same state as your database config. One bad merge and you’re having a very bad afternoon.
• Drift accumulates. Resources nobody has looked at in months quietly diverge from what Terraform thinks they are.

None of these kill you immediately. They just make everything slower and scarier until one day someone is afraid to run apply at all.

Split state by domain, not by convenience

The single most impactful thing you can do is break your state apart. Not by team, not by project. By domain.

terraform/
|-- network/
|-- security/
|-- data/
|-- kubernetes/
`-- services/
    |-- api/
    `-- web/

Each directory gets its own state backend. Each one can be planned and applied independently. A change to your VPC config doesn’t require Terraform to also reconcile your Kubernetes cluster and all your application services.

For environments, use separate directories and backends. I know workspaces exist. I’ve used them. They work fine when it’s just you. They get confusing fast when three engineers are working across staging and production and nobody remembers which workspace they’re in. Separate directories are boring. Boring is good.

Remote state with locking. Always.

If you’re running Terraform at scale and your state is local, stop reading this and go fix that first.

terraform {
  backend "s3" {
    bucket         = "terraform-state"
    key            = "network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

DynamoDB for locking. Encryption on. Versioning on the S3 bucket so you can recover when (not if) someone corrupts state. This is infrastructure for your infrastructure. Treat it seriously.

Cross-state references

When stacks need to talk to each other, use remote state data sources. Keep the outputs stable: changing an output name in your network stack shouldn’t break your services stack.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "terraform-state"
    key    = "network/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_subnet" "app" {
  vpc_id     = data.terraform_remote_state.network.outputs.vpc_id
  cidr_block = "10.0.20.0/24"
}

Think of outputs as your stack’s API. You wouldn’t change an API contract without versioning. Same principle.

Modules: small, versioned, boring

I’ve seen Terraform modules that try to do everything. A “vpc” module that also creates subnets, NAT gateways, route tables, security groups, and somehow a Lambda function. Don’t do this.

Keep modules focused on one thing.

module "vpc" {
  source = "./modules/vpc"
  cidr   = "10.0.0.0/16"
}

module "public_subnet" {
  source = "./modules/subnet"
  vpc_id = module.vpc.vpc_id
  cidr   = "10.0.1.0/24"
}

Pin versions. Pin everything. Terraform version, provider versions, module versions.

terraform {
  required_version = "~> 0.12"
}

provider "aws" {
  version = "~> 2.28"
  region  = "us-east-1"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.15.0"
}

I learned this one when an unpinned provider upgrade changed the default behavior of an AWS resource and our plan suddenly wanted to destroy and recreate a production database. Nobody was hurt. My heart rate was.

The review workflow

At the fintech startup I watched us go from “just apply it” to a proper review flow, and the number of infrastructure incidents dropped to near zero. The process isn’t exciting:

1. Branch for the change.
2. terraform fmt and terraform validate locally.
3. CI runs plan and posts the output to the PR.
4. The team that owns that stack reviews the plan.
5. apply runs only after approval, then the PR merges.

We used Atlantis for this and it worked well:

# atlantis.yaml
version: 3
projects:
- name: network
  dir: terraform/network
  workflow: default

workflows:
  default:
    plan:
      steps:
      - init
      - plan
    apply:
      steps:
      - apply

The plan output is the thing being reviewed, not just the code diff. A four-line HCL change can produce a 200-line plan. The plan is what matters.

The stuff that will save you at 2 AM

Plan files. Always save the plan and apply from the saved file. What was reviewed should be exactly what gets applied.

terraform plan -out=plan.tfplan
terraform apply plan.tfplan

-target is for emergencies only. It’s tempting to use -target to apply just one resource when you’re in a hurry. Don’t make it a habit. It skips dependency resolution and creates drift. Use it when something is on fire, then follow up with a full plan to clean up.

Drift detection. Schedule terraform plan -detailed-exitcode to run regularly. Surface drift before it surprises you during an incident. Exit code 2 means there’s drift. Pipe that into an alert.

terraform plan -detailed-exitcode
# Exit 0: no changes
# Exit 1: error
# Exit 2: changes detected (drift)

Parallelism tuning. Default is 10. You can bump it to 20 for faster applies, but watch for API rate limit errors. More parallelism isn’t always faster.

Guard the gates

Treat Terraform like application code. Lint it. Format it. Run policy checks before it reaches production.

# .pre-commit-config.yaml
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.19.0
  hooks:
  - id: terraform_fmt
  - id: terraform_validate
  - id: terraform_tflint

Pre-commit hooks catch the easy mistakes. For the hard ones, like someone trying to open port 22 to the world or creating an unencrypted S3 bucket, use Sentinel or OPA. Policy as code is one of those things that feels like overhead until the first time it stops a bad change from reaching production.

Own your stacks

Define CODEOWNERS by directory. The network team reviews network changes. The platform team reviews Kubernetes changes. Nobody should be able to merge infrastructure changes without review from someone who understands the blast radius.

Keep documentation minimal and close to the code. A README in each stack directory with the inputs, outputs, and any non-obvious decisions. Not a wiki that nobody updates.

Terraform scales fine. The part that doesn’t scale is humans making ad hoc decisions about where things go and how changes flow. The IaC patterns I wrote about last year still hold; the state layout is where most teams need to start.

I ran into this the hard way at a fintech startup. We spun up our first production cluster in a hurry: tight deadlines, the usual. Anonymous auth on the API server? Enabled. etcd encrypted at rest? No. NetworkPolicy? We didn’t even know it existed yet. We were one exposed port away from giving our financial data to anyone with kubectl and an internet connection.

That experience made me paranoid in a productive way. By the time I was setting up infrastructure at my infrastructure startup during the founder-program cohort, I had a hardening checklist I ran through before any cluster touched real traffic. This is that checklist, updated for early 2019. The short version: lock down the API server, encrypt etcd, enforce RBAC and PodSecurityPolicy, add NetworkPolicy, and stop trusting your container images.

Lock Down the Control Plane

API Server

Three flags you should set immediately:

--anonymous-auth=false
--authorization-mode=Node,RBAC
--audit-log-path=/var/log/kubernetes/audit.log

Disable anonymous access. Require RBAC. Turn on audit logging so you can actually answer “who did what” during an incident. Keep the API server on a private network. If it’s reachable from the public internet, you’ve already lost.

Enable NodeRestriction and PodSecurityPolicy admission controllers. These are your baseline. Without them, any pod can request privileges it shouldn’t have.

etcd

etcd holds your cluster state and your Secrets in plaintext by default. Treat it like a credentials database, because that’s what it is.

Run it on control plane nodes only. Firewall it from pod networks. Require TLS for both peer and client connections. And encrypt Secrets at rest:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-key>
  - identity: {}

The identity provider at the end is a fallback for reading old unencrypted secrets. Remove it once everything is re-encrypted.

Controller Manager and Scheduler

Run these only on control plane nodes. Enable leader election. Give each controller its own service account with tight permissions. The principle of least privilege applies to cluster internals too.

Harden the Nodes

Host OS

Minimal OS. Patched. SSH access restricted and ideally behind MFA. Only required ports open. Disable swap. Don’t relax kernel defaults for legacy workloads. If something needs a relaxed seccomp profile, make it an exception, not a default.

Kubelet

The kubelet is the most powerful thing on each node, and it ships with the door unlocked:

--anonymous-auth=false
--authorization-mode=Webhook
--read-only-port=0

Anonymous kubelet access is how you get remote code execution on every node in your cluster. Disable it. Use webhook authorization. Kill the read-only port.

RBAC: Less Is More

Use namespace-scoped Role and RoleBinding by default. ClusterRole and ClusterRoleBinding should be rare and audited.

Never grant cluster-admin to human users. Never grant it to CI. I’ve seen CI pipelines with cluster-admin running untrusted code from pull requests. That’s not a deployment pipeline, that’s a breach pipeline.

One service account per workload. Disable auto-mounting on the default service account. That token sitting in every pod by default is a lateral movement gift.

Workload Security

PodSecurityPolicy

In 2019, PSP is the built-in way to prevent pods from running as root, mounting host paths, or using host networking. Use it. Block privileged containers by default. Grant exceptions only with documented justification.

Security Contexts

Every workload should run with:

• Non-root user
• All capabilities dropped
• allowPrivilegeEscalation: false
• Read-only root filesystem when possible

These four lines stop a surprising number of container escape techniques.

Resource Limits

Set CPU and memory requests and limits on everything. This isn’t just a reliability thing. Without limits, a compromised pod can starve the node and take down every other workload on it. Crypto miners love clusters without resource limits.

Network Isolation

Default-deny NetworkPolicy for both ingress and egress. Then add explicit allow rules. If you haven’t done this, every pod can talk to every other pod. I covered the default-deny policy in my Kubernetes production checklist .

Keep control plane components on separate subnets from application pods. The API server shouldn’t be reachable from your web frontend pod.

Secrets Management

Encrypt at rest. Scope access through RBAC. Never put secrets in images, ConfigMaps, or environment variables you can see with kubectl describe pod.

Mount secrets as read-only files. Rotate them. For anything high-value, use Vault or your cloud provider’s KMS and inject at runtime. Kubernetes Secrets are base64-encoded, which is not encryption.

Image Supply Chain

At the fintech startup we once deployed a container image pulled from Docker Hub that hadn’t been updated in 18 months. It had three known CVEs, one critical. We found out during a penetration test, not before.

Limit deployments to trusted registries. Pin images by digest, not tag; latest isn’t a version. Scan images in CI with Clair or Anchore. Block deploys with high-severity findings. Enable Docker Content Trust for workloads where provenance matters. More on this in container security beyond the basics .

Audit and Monitoring

Collect audit logs and ship them somewhere the cluster can’t delete them. Alert on:

• clusterrolebinding creation
• exec into pods
• Changes to PodSecurityPolicy
• Any access to Secrets

Falco adds runtime visibility at the container level. It’s worth the setup time.

Do It This Week

If you’re running Kubernetes in production and haven’t gone through a hardening pass, do it this week. Not next sprint. The defaults are designed for getting started fast, and they succeed at that. They aren’t designed to keep your data safe.

I keep seeing the same pattern. Someone gets a scary AWS bill. Leadership asks engineering to “optimize cloud costs.” Engineering buys reserved instances, saves 20%, and calls it done. Meanwhile, half the staging environments are running 24/7, there are unattached EBS volumes left over from instances terminated months ago, and nobody knows which team owns the NAT gateway that’s costing $400/month.

Reserved instances are a pricing mechanism. Most teams burn money on idle resources they can’t even find, then negotiate a discount on the waste. Fix the waste first. Below are the strategies that work, ranked by effort against impact.

Strategies Compared

That table tells a story most FinOps consultants won’t: the highest-ROI work is the boring stuff at the top.

Start With Visibility, Not Discounts

At the fintech startup, I inherited an AWS account with zero tagging discipline. Costs were allocated to “engineering” as a single line item. Completely useless. My first week, I enforced four tags on every resource:

Environment: production | staging | dev
Team: platform | data | product
Service: api | worker | web | ml-pipeline
Owner: <person who provisioned it>

Within two weeks, we found three full staging environments nobody was using. One had been running for five months. That single discovery saved more than the next quarter’s reserved instance commitment would have.

Tags are free. Use them.

Right-Sizing: The Unsexy Middle Ground

Most instances are oversized. This isn’t controversial. Every cloud provider’s own tooling will tell you this. The question is what to do about it.

My approach, applied at every place I’ve worked:

1. Collect two weeks of utilization data. Not one day. Not peak hour. Two weeks across normal traffic patterns.
2. Start with non-production. Dev and staging environments are almost always 2-4x oversized because someone copied the production config.
3. Target p95 CPU below 60% after resizing. You want headroom. This isn’t about running hot.

That last column is the one that matters most. Sometimes the right size is zero.

Spot Instances: High Reward, High Effort

I’m a fan of spot capacity for the right workloads. At my infrastructure startup, I run all CI builds on spot instances. That saves roughly 70% on build infrastructure. But I wouldn’t touch spot for anything user-facing or stateful.

How I decide what goes on spot:

Data Transfer Costs

Cross-AZ traffic on AWS is $0.01/GB each way, one of the hidden cloud costs that never shows up in the pricing calculator. Doesn’t sound like much until you’re pushing 10TB/month between services in different availability zones. That’s $200/month just for services talking to each other inside your own VPC.

Three fixes that cost nothing but attention:

• Keep chatty services in the same AZ. Schedule pods to co-locate when latency and bandwidth matter.
• Use VPC endpoints for AWS services. S3 traffic through a NAT gateway is money on fire.
• Cache aggressively at the edge. A CDN in front of your API responses (where appropriate) cuts both latency and egress.

At the mobility startup, data transfer was our third-highest line item after compute and RDS. Moving our real-time location services into a single AZ and adding a VPC endpoint for S3 cut that bill by 40% in one sprint.

My Cost Routine

I don’t have a FinOps team. At startup scale, cost discipline is an engineering habit, not a department. Here’s my actual routine:

Weekly: Glance at the cost dashboard. Look for anything that jumped more than 20% week-over-week. If something spiked, find it before the month closes.

Monthly: Review top 10 services by cost. Check for idle resources: unattached volumes, orphaned load balancers, forgotten test clusters. Five minutes of cleanup saves hundreds.

Before any commitment: Right-size first. Always. Buying a reserved instance for an oversized instance just locks in the waste at a discount.

Cloud cost optimization has no finish line. Like writing tests or reviewing pull requests, you either build the habit or pay the tax forever. The teams that manage spend well are the ones where every engineer can see what their service costs and cares enough to keep it reasonable.

At the fintech startup, our “deployment pipeline” for the first few months was me running kubectl apply from my MacBook at 11pm. I had a checklist in Notes.app. It worked until I forgot step 4 one night and took down the price feed for 45 minutes.

The outage wasn’t catastrophic, but it made me realize that if I got hit by a bus, nobody else could deploy. The entire process lived in my head and my shell history.

GitOps fixed that. It took a while, and it stuck.

What GitOps is

GitOps is a principle: Git is the single source of truth for what your infrastructure should look like, and a controller continuously reconciles reality to match.

That’s it. Everything else is implementation detail.

The word gets thrown around a lot in 2019, mostly by vendors trying to sell dashboards. Strip away the marketing and you get three rules:

1. Declare your desired state in files checked into Git
2. Run a controller that watches those files and applies changes automatically
3. Never touch the cluster directly

If a change isn’t in Git, it didn’t happen. Full stop.

Why pull-based beats push-based

Most CI/CD pipelines are push-based. Jenkins builds your image, then runs kubectl apply against the cluster. The problem: Jenkins now needs cluster credentials, network access, and the ability to mutate production state. Your CI system becomes a god-mode attack surface.

Pull-based GitOps flips this. The controller runs inside the cluster. It watches a Git repo and pulls changes when it sees a new commit. The cluster reaches out to Git, not the other way around.

Push model:  CI --> kubectl apply --> Cluster
Pull model:  Cluster <-- controller watches <-- Git repo

The pull model wins on security first. No cluster credentials in CI. No network path from build servers to production. And because the controller keeps comparing the cluster with Git, drift shows up instead of accumulating quietly.

Setting it up: a real example

Here is the repo layout I settled on after trying three different approaches. It’s for Go microservices on Kubernetes , so it’s Kustomize-flavored.

infra/
  apps/
    api-gateway/
      base/
        deployment.yaml
        service.yaml
        kustomization.yaml
      overlays/
        staging/
          kustomization.yaml
          replicas.yaml
        production/
          kustomization.yaml
          replicas.yaml
          hpa.yaml
    auth-service/
      base/
        ...
      overlays/
        ...
  platform/
    namespaces/
    network-policies/
    rbac/

The apps/ directory is owned by service teams. The platform/ directory is owned by me (or whoever is on infra that week). This separation matters more than any specific tool choice.

The base deployment

Nothing exotic here. A standard Kubernetes deployment with explicit resource requests, health checks, and a pinned image tag. No latest. Ever.

# apps/api-gateway/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
        - name: api-gateway
          image: registry.example.com/api-gateway:v0.8.3
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5

The production overlay

Production gets more replicas, an HPA, and a specific image tag. The overlay touches only what differs.

# apps/api-gateway/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base
  - hpa.yaml
patchesStrategicMerge:
  - replicas.yaml
images:
  - name: registry.example.com/api-gateway
    newTag: v0.8.3

This is the file that changes on every deploy. A version bump is a one-line diff in a pull request. Reviewable. Revertable. Boring.

Choosing a controller: Flux vs Argo CD

In early 2019, the two serious options are Flux and Argo CD. I’ve run both.

Flux is simple. It watches a Git repo, syncs to a cluster, done. Installing it is one Helm chart that runs a Deployment pointed at your repo:

helm repo add weaveworks https://weaveworks.github.io/flux
helm install --name flux \
  --set git.url=git@github.com:example/infra.git \
  --set git.branch=master \
  --set git.path=apps/ \
  --namespace flux \
  weaveworks/flux

Flux feels like a Unix tool. It does one thing. If you want a UI or multi-cluster visibility, look elsewhere.

Argo CD is heavier but more featureful. You define Applications as custom resources:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-gateway
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/infra
    targetRevision: master
    path: apps/api-gateway/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true

Automated sync with prune: true makes a merged commit the deploy, deletions included. And when someone runs a manual kubectl edit in production, the UI marks the application OutOfSync, so drift is visible to the whole team. That alone is worth the setup cost.

My recommendation: start with Flux if you have one cluster and want simplicity. Move to Argo CD when you need multi-cluster, a UI for the team, or you’re tired of explaining to people what state the cluster is in.

We started with Flux. Moved to Argo CD three months later. No regrets on either decision.

The deployment workflow

Here is how a deploy actually works once this is set up:

# Developer bumps the image tag after CI builds a new version
git checkout -b deploy/api-gateway-v0.8.4
vim apps/api-gateway/overlays/production/kustomization.yaml
# Change newTag: v0.8.3 -> v0.8.4
git add -A && git commit -m "deploy: api-gateway v0.8.4"
git push origin deploy/api-gateway-v0.8.4
# Open PR, get review, merge

After merge, the controller picks up the change on its next poll (every few minutes by default; the interval is configurable). The rollout happens. Monitoring confirms. Done.

Compare that to the old world:

# The old way -- do not do this
kubectl set image deployment/api-gateway \
  api-gateway=registry.example.com/api-gateway:v0.8.4 \
  -n production
# Hope you remember what you just did
# Hope the next person knows too

The first approach gives you an author, a reviewer, a timestamp, a diff, and a one-command rollback (git revert). The second gives you nothing.

Secrets: the hard part

Secrets are where GitOps gets uncomfortable. You want everything in Git, but you can’t commit plaintext credentials. In 2019, the options are:

Mozilla SOPS encrypts values in-place using KMS or PGP. The file structure stays readable, only the values are encrypted. This is what I use.

# secrets.enc.yaml (encrypted with SOPS)
apiVersion: v1
kind: Secret
metadata:
  name: api-gateway-secrets
data:
  DATABASE_URL: ENC[AES256_GCM,data:abc123...,type:str]
  API_KEY: ENC[AES256_GCM,data:def456...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:eu-west-1:123:key/abc-def

Bitnami Sealed Secrets encrypts secrets with a cluster-side key. Safe to commit the encrypted form. Decryption only happens inside the cluster.

External secret managers such as Vault or AWS Secrets Manager. The secret reference lives in Git, the actual value lives elsewhere. More moving parts, but better for large orgs.

Pick one. Don’t commit plaintext. I’ve seen production database passwords in public GitHub repos more times than I want to admit.

Handling emergencies

Sometimes you need to bypass the process. A P1 incident at 3am isn’t the time to open a pull request.

The rule: fix it now, commit it immediately after.

# 1. Fix the fire
kubectl rollout undo deployment/api-gateway -n production

# 2. Immediately capture in Git (before you forget or go back to sleep)
git checkout -b hotfix/api-gateway-rollback
# Update manifests to match the rolled-back state
git commit -m "hotfix: rollback api-gateway to v0.8.3 (incident #47)"
git push && # open PR

# 3. Write a note in the incident channel explaining why you bypassed GitOps

If you skip step 2, the controller will eventually reconcile and re-deploy the broken version. I learned this the hard way: we had a demo the next morning and I rolled back manually without committing. Woke up to the broken version redeployed by Flux at 6am. Fun times.

What drift detection tells you

When you first enable GitOps, the controller will fight you. It will revert manual changes, flag resources you forgot about, and generally make noise.

Good. That noise is telling you things:

• A developer ran kubectl edit because the PR process felt too slow. Fix the process.
• A resource exists in the cluster that isn’t in Git. Either add it to Git or delete it.
• Someone created a secret manually during an incident. Capture it properly.

Drift detection is your audit system running in real time. Treat every drift event as a process fix to make, and don’t suppress it.

Repo structure: mono vs split

I tried three approaches before settling:

Mono-repo (app + infra together). Simple for tiny teams. Falls apart when you have more than ~5 services because every deploy touches the same repo and branch protection gets weird.

Split repos (one for app, one for infra). Clean ownership boundaries. The infra repo becomes the system of record. Cross-cutting changes require two PRs, which is annoying but forces you to think about coupling. This is what I recommend for most teams.

Per-app manifests in app repos. Each service owns its deployment config. Great for autonomy, terrible for platform-wide changes like “update the sidecar version on every service.” You will end up writing a bot to open 30 PRs.

For a team of 3-10 engineers, split repos with Kustomize overlays is the sweet spot. You can always migrate later; your manifests are just files in Git.

What I got wrong

A few mistakes worth sharing:

I over-automated image updates. I set up Flux’s auto-image-update feature, which watches a registry and commits new tags automatically. Sounds great. In practice, it deployed a broken build at 2am because CI had a flaky test that passed on a bad commit. Explicit tag bumps via PRs are slower but safer. I turned off auto-updates within a week.

I didn’t enforce branch protection early enough. For the first month, anyone could push directly to master on the infra repo. One accidental force-push later, I added required reviews and status checks. Should have been day one.

I skipped the README. New engineers had no idea how to deploy. The process was obvious to me because I built it. It wasn’t obvious to anyone else. A one-page doc explaining “how to deploy a new version” saved hours of Slack questions.

Getting started

If you’re reading this and still deploying with kubectl from a laptop, here is what I would do tomorrow:

1. Create an infra repo. Move your manifests there.
2. Install Flux and point it at the repo.
3. Enable branch protection on master. Require one review.
4. Deploy something small through a PR. Watch it sync.
5. Delete kubectl access for your CI system. If CI can’t touch the cluster, it won’t.

You can do steps 1-4 in an afternoon. Step 5 takes courage, but it’s the one that makes the whole thing stick.

This is the checklist I use, not a getting-started guide. Everything here exists because I’ve been paged at 3am for the opposite.

Upgrades Should Be Boring

If your Kubernetes upgrade process feels heroic, something is wrong. Two years of running clusters taught me that the boring parts are the hard parts. Stay on a supported minor version. Have a rollback plan. Run it on a schedule, not as a fire drill.

I keep a dead-simple rule: if we’re more than one minor version behind, it’s a P1. Not because something is broken, but because something will break and we’ll be debugging with stale docs and missing patches.

Resource Requests and Limits. On Everything.

This is the single most impactful thing you can do. No exceptions.

resources:
  requests:
    cpu: "250m"
    memory: "512Mi"
  limits:
    cpu: "1000m"
    memory: "1Gi"

Requests are your scheduling guarantee. Limits are your safety net. Without both, you’re one misbehaving pod away from cascading node pressure. I’ve seen a single Go service with no memory limit take down a shared node at the fintech startup because someone forgot to close a channel and goroutines piled up. That was a fun Friday evening.

Size these from actual metrics, not guesswork. The metrics server and Prometheus give you enough signal. Start conservative, then adjust.

Namespaces With Actual Guardrails

Namespaces without resource quotas are just labels. Enforce boundaries at the namespace level so one team’s experiment can’t starve another team’s production workload.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi

Probes Are Mandatory

Not optional. Not “we’ll add them later.” Mandatory.

Liveness probes restart broken processes. Readiness probes keep unready pods out of the Service’s endpoints, so they get no traffic. Without them, you’re routing traffic to containers that are technically running but functionally dead.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

One thing I always add: make /healthz and /ready do different things. Your liveness check should be “is the process alive.” Your readiness check should be “can this instance serve traffic right now”: database connected, caches warm, dependencies reachable.

Handle Shutdown Properly

Your container will get SIGTERM’d. Handle it. Stop accepting new work, drain in-flight requests, exit clean. Set terminationGracePeriodSeconds to match your actual drain time, not the default 30 seconds.

In Go (which is what I write most services in), this means catching the signal and giving your HTTP server a context-bounded shutdown. It’s ten lines of code. There’s no excuse.

Secrets Deserve Respect

Two rules:

1. Mount secrets as files, not environment variables. Env vars leak into crash dumps, child processes, and logging. Files don’t.
2. Enable encryption at rest in etcd. If you’re on a managed provider, verify this is actually on. Don’t assume.

For anything beyond toy projects, use an external secret manager. Kubernetes native secrets are base64-encoded, not encrypted. Big difference.

Default-Deny Networking

Without network policies, every pod can talk to every other pod. That’s not a cluster, it’s a flat network with extra steps.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Start with deny-all, then explicitly allow what’s needed, the same default-deny idea behind the zero trust setup we built . Yes, it’s more work upfront. But “every service can reach every other service plus the internet” isn’t a security posture I’m comfortable with, and you shouldn’t be either.

My order of priorities is always the same: security, stability, performance. In that order. Network policies fall squarely in bucket one.

RBAC and Pod Security

Run as non-root. Drop capabilities. Use read-only root filesystems where you can. Enforce these with PodSecurityPolicy so it’s not just a suggestion.

For RBAC: if any service account has cluster-admin, fix that first. Service accounts get the verbs and resources they need. Nothing more.

Roll Out Safely

Use rolling updates with sensible surge and unavailable settings. Add Pod Disruption Budgets so node drains and maintenance don’t accidentally kill your service. Spread replicas across nodes and zones with anti-affinity rules.

None of this is glamorous. All of it prevents 2am pages.

Observability That Helps

Three things:

Structured logs. JSON, consistent fields, correlation IDs. If your logs are unstructured strings, your incident response is grep and prayer.

Application metrics tied to SLOs. Service-level objectives tell you whether users are happy; vanity dashboards don’t. Expose the metrics with Prometheus and build dashboards that answer questions.

Consistent labels. Every resource gets app, team, version, environment. Those labels are what make cost allocation, debugging, and incident response possible at any scale beyond one person.

Back Up etcd. Test the Restore.

A backup you’ve never restored is a hope, not a backup. Schedule restores. Verify them. The five minutes this takes will save you from the worst day of your career.

The Pattern

The teams I’ve seen run Kubernetes well aren’t doing anything clever. They do the basics consistently and spend their creativity on the product instead of firefighting infrastructure.

Istio is impressive technology. It’s also a complexity bomb that most teams have no business adopting.

What it does

The pitch is simple. You have microservices, they all need retries, timeouts, mTLS, and observability. Instead of implementing that in every service, you push it to a sidecar proxy layer. Istio manages those proxies. Fine.

In practice you get three things:

Traffic control. Route by version, headers, percentage. Canary deployments become a YAML change instead of a code change.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
  - api
  http:
  - route:
    - destination:
        host: api
        subset: v1
      weight: 90
    - destination:
        host: api
        subset: v2
      weight: 10