It works well for a few months. Then the agent approves a six-figure payment to a vendor who submits a duplicate invoice with slightly different formatting. The model treats it as new. The validation layer doesn’t exist because “the AI handles it.”

A six-figure loss, because someone treated a probabilistic system like a deterministic one.

Seeing it more than once crystallized a principle I come back to often: AI decides, deterministic code acts. Never the other way around.

The Architecture That Survives

The separation is simple in concept and surprisingly rare in practice. The AI component receives structured context, produces a structured decision with a rationale, and stops there. Everything after that (validation, side effects, and the actual work) is deterministic code with explicit rules.

The flow:

1. Trigger arrives with metadata (a ticket, a document, an event)
2. AI decision produces structured output : classification, extraction, routing recommendation, confidence score, and a short explanation of why
3. Deterministic validation checks the decision against hard policy rules, allowlists, deny lists, and threshold constraints
4. Action or escalation. If validation passes and confidence is high, execute. If not, route to human review with the full context attached
5. Audit trail stores the input, the decision, the rationale, the validation result, and the final action

Every step is logged. Every decision is replayable. If something goes wrong, you can trace exactly where and why.

Confidence Tiers Aren’t Optional

Not every AI decision deserves the same treatment. A classification the model is 95% sure about is different from one it’s 60% sure about. Your automation should know the difference.

I use three tiers everywhere:

• High confidence: auto-approve, execute the action, log for periodic review
• Medium confidence: queue for human review with the AI’s recommendation and rationale attached
• Low confidence: escalate immediately, flag for manual handling, don’t proceed

Thresholds depend on your domain. For invoice processing, I set the bar high because the cost of a wrong action is real money. For ticket triage, I set it lower because a misrouted ticket is annoying but recoverable.

The point is that uncertainty is a normal operating state. It isn’t a bug. Your system should be designed to handle it gracefully instead of pretending every decision is confident.

Context Discipline

Feed the AI the minimum context needed to make a good decision. Not a raw database dump. Not the entire ticket history. Use a structured package: the specific document or event, the relevant policy excerpt, and a few representative examples of how similar cases were decided.

When teams dump everything into the context window , two things happen: token costs explode, and the model starts hallucinating connections between unrelated data points. More context isn’t better context. Be deliberate about what matters for a specific decision.

Where AI Automation Fits

Good fits: request triage, document classification, data extraction from messy formats, policy-based routing where ambiguity is expected and escalation is normal.

Bad fits: anything safety-critical, anything requiring hard real-time guarantees, anything where a wrong decision is irreversible and expensive. If you can’t tolerate occasional uncertainty, don’t automate with a probabilistic system.

From what I’ve seen, the most successful automation projects started with a single workflow that already had a manual review path. They ran the AI in shadow mode first, compared its decisions to the human decisions, measured agreement rates, and only then moved to live execution, with review still in place for the first few weeks.

Why the Duplicate Payment Went Through

That duplicate payment isn’t a model failure. The model does exactly what it was designed to do: it classifies the invoice and approves it. The failure is architectural. Nobody built the validation layer that should have caught a duplicate vendor-amount-date combination. Nobody defined the hard boundaries.

AI automation works when you respect what it is: a probabilistic decision engine. Wrap it with deterministic guardrails , log everything, and keep humans in the loop for anything your business can’t afford to get wrong.

Guardrails beat talent. Always.

There’s a moment in every agentic AI demo that makes the audience gasp. The agent reads a database, reasons about the results, drafts an email, and sends it. Autonomously. It feels like magic.

Then someone asks: “What happens if it sends the wrong email?” And the room gets quiet.

I’ve been building agentic systems for several months now. The demo-to-production gap here is wider than almost anywhere else in AI engineering. A chatbot that hallucinates is annoying. An agent that hallucinates and then acts on the hallucination is a liability.

The difference between teams that ship agents successfully and teams that revert after a week comes down to three things: boundaries, structure, and boring reliability work.

Boundaries first, capabilities second

Almost every team starts with capabilities. “What tools should the agent have? What actions can it take?” Wrong starting point.

Start with constraints. What is the agent not allowed to do? What’s the maximum blast radius of a single run? What happens when it goes wrong?

A policy config is the simplest way to make these constraints explicit and auditable:

agent_policy:
  allowed_tools: [read_db, write_ticket, send_email_draft]
  max_steps: 8
  max_runtime_seconds: 120
  max_cost_usd: 0.50
  approval_required: [send_email, issue_refund, modify_production]

Treat this as the foundation. The allowed tools list is an allowlist: the agent can only use what’s explicitly permitted. Step and time limits prevent runaway loops. Cost caps prevent a single request from draining your budget. The approval list separates actions that are safe to automate from actions that need a human in the loop.

At one company I worked with, a team skipped the approval step for “low-risk” actions. One of those low-risk actions turned out to be updating customer records. An agent misinterpreted a support request and bulk-updated addresses for a batch of orders. The fix took two days. The approval gate would have taken two seconds.

If the policy feels too restrictive, relax it intentionally and document why. If you can’t explain why a tool is on the allowed list, it shouldn’t be there.

Structured workflows beat free-form loops

The temptation with agents is to give them a goal and let them figure out the steps. This works beautifully in demos. In production, it creates systems that are impossible to debug, test, or audit.

I prefer structured workflows with a small number of decision points. The model chooses among defined paths. Deterministic logic handles state transitions. The result is a system you can trace, test, and explain.

Think of it as a state machine where the model influences transitions but doesn’t control them entirely. The model might decide whether a customer inquiry needs escalation or can be handled automatically. But the escalation path itself (what happens, in what order, and with what approvals) is defined in code, not improvised by the model.

When a task genuinely doesn’t fit a clean workflow, isolate it. Put the free-form reasoning in a narrow, heavily instrumented sandbox with tight constraints. Don’t make it the default path for everything.

The boring reliability checklist

I know this section won’t go viral. That’s fine. It’s the section that keeps your agent from becoming an incident.

Idempotent steps. If a step fails and retries, it shouldn’t duplicate work. The agent shouldn’t send two emails because the first one timed out after actually sending. Design every action to be safe to retry.

Checkpointing. Long-running workflows should save their state at each step. If the process crashes or the model call times out, the workflow should resume from the last checkpoint, not start over.

Time and step caps. Hard limits. Non-negotiable. An agent stuck in a reasoning loop should hit a wall after N steps or M seconds, return whatever partial results it has, and report the failure. I set these conservatively and loosen them only after seeing production data.

Retry discipline. Retry on clearly transient failures such as rate limits and network timeouts. Don’t retry on semantic failures, where the model misunderstood the task or the tool returned an error because the input was wrong. Retrying bad logic just wastes money and time.

Observability isn’t optional

If you can’t trace what an agent did (every tool call, every model response, every decision point), you can’t debug it. And you will need to debug it.

Structured logging for every step:

• What tool was called and with what inputs
• What the model returned and what confidence signal it provided
• Whether an approval was required and who approved it
• How long each step took and how many tokens it consumed
• The final outcome and whether it matched the intent

This log is also your feedback loop. It tells you which prompts need refinement, which tools are unreliable, which workflows cost too much, and where the model consistently makes bad decisions.

One caution: be disciplined about what you log. Inputs and outputs may contain sensitive data. Define retention policies and access controls before you ship, not after an auditor asks.

Rolling out without regret

The teams that succeed with agentic workflows share a rollout pattern:

1. Shadow mode first. The agent runs alongside the existing process but doesn’t take any actions. Log what it would have done. Compare to what the human actually did. This gives you real quality data without any risk.
2. Low-risk tasks with clear success criteria. Start with internal tasks where a mistake is inconvenient, not catastrophic. Ticket triage. Data enrichment. Report drafting.
3. Expand only after stability. Once reliability, cost, and quality are stable for the initial scope, add more tools or more complex workflows. One step at a time.

This pacing is unglamorous. It’s also the only approach I’ve seen work consistently.

Agents act, so the bar is higher

Agents are powerful. They’re also the highest-risk AI feature you can ship. Every other AI feature is advisory: the model suggests, the user decides. An agent acts. That means every bug, every hallucination, every misunderstanding has real consequences.

Treat agents as systems engineering, not prompt engineering. Define the blast radius. Build the constraints. Invest in the observability. Ship slow.

The teams that move carefully are the ones still running agents in production six months later. The teams that rush are the ones writing postmortems.

Everyone can write a Terraform file. The problem starts around month six, when you have fifteen modules nobody remembers writing, state files that three people are afraid to touch, and a staging environment that has drifted so far from production that testing there is theater.

I’ve seen this pattern at multiple organizations, and I wrote about IaC patterns back in 2018 . The IaC starts clean. Then it grows. Then it becomes the thing everyone is scared of. The patterns below are what I’ve seen work to prevent that decay.

Module Design: One Job Per Module

The single most important structural decision is module scope. A module should do one thing. Not “provision a service and its database and its DNS and its monitoring.” One thing.

Here is what a focused module looks like for an application service:

# modules/ecs-service/main.tf

variable "service_name" {
  type        = string
  description = "Name of the ECS service"
}

variable "cluster_arn" {
  type = string
}

variable "container_image" {
  type        = string
  description = "Docker image URI"
}

variable "cpu" {
  type    = number
  default = 256
}

variable "memory" {
  type    = number
  default = 512
}

variable "desired_count" {
  type    = number
  default = 2
}

variable "subnet_ids" {
  type = list(string)
}

variable "security_group_ids" {
  type = list(string)
}

resource "aws_ecs_service" "this" {
  name            = var.service_name
  cluster         = var.cluster_arn
  task_definition = aws_ecs_task_definition.this.arn
  desired_count   = var.desired_count
  launch_type     = "FARGATE"

  network_configuration {
    subnets         = var.subnet_ids
    security_groups = var.security_group_ids
  }
}

resource "aws_ecs_task_definition" "this" {
  family                   = var.service_name
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = var.cpu
  memory                   = var.memory

  container_definitions = jsonencode([{
    name  = var.service_name
    image = var.container_image
    portMappings = [{
      containerPort = 8080
      protocol      = "tcp"
    }]
  }])
}

Clean inputs, clean outputs, one responsibility. The database, the DNS record, the monitoring: those are separate modules composed at a higher level. When the ECS module changes, you know exactly what’s affected.

The moment a module becomes a grab bag, it stops being reusable. I’ve seen “service” modules with 40 input variables and 200 lines of conditional logic. Nobody wants to touch them. Nobody trusts them. They become legacy infrastructure inside your infrastructure code.

Repository Layout

Your repo structure should reflect ownership, not technology. A layout I’ve used successfully:

infrastructure/
  modules/
    ecs-service/
    rds-instance/
    cloudfront-distribution/
  environments/
    production/
      main.tf
      terraform.tfvars
    staging/
      main.tf
      terraform.tfvars
    dev/
      main.tf
      terraform.tfvars

Each environment directory composes the same modules with different variables. This makes differences between environments explicit and small instead of hidden and surprising.

State Management Isn’t a Detail

State management is where IaC projects actually fail. Not in the HCL. In the state.

Rules that have saved me from pain:

Remote state with locking. Always. No exceptions. Two people running terraform apply at the same time against the same state file is how you get a 3am incident.

terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "production/api-service/terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Segment state by blast radius. Don’t put your entire production environment in one state file (I covered splitting state by blast radius last year). When you run terraform plan and it wants to touch 200 resources, you aren’t going to review that carefully. Nobody does. Split by service or component so each plan is small and reviewable.

Encrypt everything. State files contain secrets whether you want them to or not. Database passwords, API keys, connection strings: they end up in state. Encryption at rest is the minimum.

At a real-time messaging platform I worked with, we segmented state per service per environment. A change to the messaging service’s production infrastructure touched exactly that service’s state. Nothing else. This made reviews fast, plans small, and rollbacks contained.

Environment Promotion Over Rebuilds

The safest deployment model is promotion. A change that passed in dev should be the exact same change in staging, which should be the exact same change in production. Not “similar.” Identical.

# environments/production/main.tf

module "api_service" {
  source = "../../modules/ecs-service"

  service_name       = "api-service"
  cluster_arn        = var.cluster_arn
  container_image    = var.api_image  # promoted from staging
  cpu                = 512
  memory             = 1024
  desired_count      = 4
  subnet_ids         = var.private_subnet_ids
  security_group_ids = [aws_security_group.api.id]
}

The module source is the same across environments. Only the variables change. When you promote, you update the image tag and any environment-specific sizing, and the rest stays locked.

Avoid long-lived branches per environment. I’ve seen teams with a staging branch and a production branch that diverged months ago. Nobody knows what the differences are. Nobody wants to find out. Use one branch, different variable files.

Testing: Layers, Not Afterthoughts

IaC testing is closer to software testing than most teams realize. Three layers that work:

Static analysis. Run terraform validate, tflint, and a policy tool like OPA or Sentinel in CI. These catch syntax errors, deprecated patterns, and policy violations before anyone runs a plan. Fast and cheap.

# policy/require_encryption.rego

package terraform

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  not resource.change.after.server_side_encryption_configuration
  msg := sprintf("S3 bucket %s must have encryption enabled", [resource.address])
}

With AWS provider v4 and later, bucket encryption lives in a separate aws_s3_bucket_server_side_encryption_configuration resource, so adapt a rule like this to the provider version you run.

Plan review. Every change gets a terraform plan output attached to the PR. Reviewers look at the plan, not just the code. The code tells you what you intended. The plan tells you what will actually happen. These aren’t always the same thing.

Integration tests for critical modules. For modules that provision core infrastructure, use something like Terratest to actually create and destroy resources in a test account. This is slow and expensive, so reserve it for the modules where a mistake costs real money.

Policy as Code

Policy documents nobody reads aren’t policy. Automated checks that block bad changes are policy.

Encode your security baselines, encryption requirements, and network boundaries as code. Run them in CI before any apply. When a developer opens a PR that creates an unencrypted S3 bucket, the pipeline catches it, not a security audit six months later.

Managing Drift

Drift is inevitable. Someone will fix something in the console during an incident. An auto-scaling event will leave resources in a state Terraform didn’t expect. External automation will modify tags.

The answer isn’t “never touch the console.” The answer is detect drift early and reconcile it regularly. Run terraform plan on a schedule. Alert on unexpected changes. Treat drift like tech debt: small amounts are fine, accumulated drift is dangerous.

Secrets: Keep Them Out

Secrets don’t belong in .tfvars files, in version control, or in state files if you can avoid it. Use a secrets manager and reference values at runtime:

data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = "production/api-service/db-password"
}

resource "aws_db_instance" "main" {
  # ...
  password = data.aws_secretsmanager_secret_version.db_password.secret_string
}

Mark outputs as sensitive. Review plan output for accidental exposure. This isn’t paranoia. I’ve seen database credentials committed to state files in shared S3 buckets with overly broad access policies.

Treat Terraform Like Application Code

The teams I’ve seen manage infrastructure well at scale all share the same trait: they treat their Terraform code with the same rigor they treat their application code. Reviews, tests, clear ownership, and a healthy fear of clever abstractions. Boring infrastructure is reliable infrastructure.

By now, the pattern is familiar. You start with a few teams, each running their own pipelines and infrastructure. It works. Then you grow to twenty teams. Each one has reinvented service templates, built slightly different CI pipelines, and solved the same security problems in incompatible ways. Suddenly you have twenty bespoke deployment processes, none of which are documented well enough for anyone outside the team to understand.

That isn’t a failure of DevOps principles. It’s a scale problem, and platform engineering is the pragmatic fix. In 2020 I called platform engineering DevOps with a rebrand . At twenty teams, the rebrand earns its keep.

What Platform Engineering Actually Is

Strip away the conference talks and vendor pitches, and platform engineering is simple: a team that builds the paved road so product teams don’t have to pave their own.

The platform team builds opinionated defaults for the boring stuff: service scaffolding, deployment pipelines, observability, security baselines. Product teams use those defaults and focus on product work. If a team needs to deviate, they can, but they start from a working foundation instead of a blank page.

At a large consumer platform, the infrastructure team had elements of this approach. The teams that used the supported path shipped faster and had fewer incidents. The teams that went custom spent meaningful time on plumbing that had nothing to do with their actual domain.

The Golden Path Is a Contract

I like the “golden path” framing because it makes the deal explicit. Follow this path, and the platform team supports you end to end. Deviate, and you own the consequences.

The point is sustainability. A platform team can’t support infinite variation. Opinionated defaults reduce the support surface and let the platform team actually invest in quality instead of fighting fires across twenty different setups.

A good golden path covers the boring lifecycle: create a service, deploy it, observe it, roll it back. Make each step predictable and fast. If “create a new service” takes a developer two days of config and ticket filing, your golden path isn’t golden. It’s bureaucracy with a nice name.

Treat It Like a Product (Or Don’t Bother)

This is where most platform efforts fail. They build internal tooling with an infrastructure team’s mindset: here is the tool, use it, file a ticket if it breaks.

That doesn’t work. An internal platform needs product thinking. Who are the users? What workflows are painful? Where is the friction? What does adoption look like?

If developers avoid your platform, you have a product problem, and more training won’t fix it. Either the platform doesn’t solve a real pain point, or it creates more friction than it removes.

Adoption is the only metric that matters early on. Everything else (consistency, reliability, cost savings) follows from teams actually choosing to use the thing.

What Goes Into a Platform

Keep it focused. An internal platform typically needs three capabilities:

A developer entry point. Somewhere that surfaces service ownership, docs, and supported workflows. This can be a portal , a CLI, or even a well-maintained repo with templates. The format matters less than the discoverability.

Self-service provisioning. Environments, pipelines, databases, queues: the resources teams need to ship. If provisioning requires a ticket and a three-day wait, you haven’t solved the problem.

Shared guardrails. Observability, security controls, cost defaults. Things that every team needs but nobody wants to build from scratch. Bake them into the golden path so compliance is the default, not an afterthought.

Build vs. Buy

Build what encodes your internal standards and unique workflows. Buy what’s commoditized. Most organizations don’t need a custom CI system or a bespoke container registry. They do need custom service templates, deployment policies, and integration glue that ties everything together in a way that makes sense for their stack.

The real cost is the ongoing maintenance and integration work, more than the license fee. Every tool you adopt becomes a dependency you have to keep running.

The Team Matters

A platform team needs infrastructure depth, software engineering skill, and (this gets overlooked) the ability to write decent documentation. A platform nobody can figure out is a platform nobody uses.

Avoid the ticket-queue trap. If the platform team’s primary interface is a Jira board where developers file requests and wait, you’ve built an ops team with a fancier name. The whole point is self-service.

Platform engineering is DevOps growing up, acknowledging that “everyone does everything” doesn’t scale, and building the shared foundations that let teams move fast without reinventing the wheel every sprint.

This debate generates more heat than it deserves. I’ve worked in both models: a monorepo at my infrastructure startup where every change could touch everything, and polyrepos at larger companies where dozens of teams needed autonomy. Both worked. Both had pain points. The difference was never the repo structure itself. It was whether the structure matched how the code actually changed.

Three Questions That Decide It

Forget “monorepo vs. polyrepo” as an identity choice. Ask these questions instead:

1. How often do changes span multiple services or libraries?
2. How independent are your team release cycles?
3. How much build infrastructure are you willing to own?

That’s basically it. The answers determine the tradeoff surface.

The Tradeoff Table

Neither column is universally better. It depends on your situation.

When Monorepo Wins

Monorepo shines when coupling is real, not aspirational. If your backend API change requires a frontend update and a shared library bump, doing that across three repos with three PRs and a release dance is painful. A monorepo makes it one commit with one review.

It also works when you want consistency. Shared linting, shared CI, shared tooling: all easier when there’s one source of truth. At one company with fifteen microservices, we moved to a monorepo and immediately found three services using different logging libraries for no reason. Consolidation happened naturally once the code was visible.

The cost is real though. You need smart builds. Running all tests for every change in a repo with fifty services isn’t viable. Tools like Bazel, Nx, or Turborepo exist for this reason, but they aren’t free to set up or maintain. The repository itself becomes infrastructure, and someone has to own it.

When Polyrepo Wins

Polyrepo wins when teams are genuinely independent. If your payments team and your notifications team share an API contract but otherwise never touch each other’s code, separate repos give them freedom to move at their own pace. Different languages, different deploy schedules, different testing strategies: all fine.

Permissions are simpler too. Contractors working on one service don’t need to see the entire codebase. Compliance teams are happier when access boundaries are clean.

The cost is coordination. Cross-repo changes require versioning , compatibility planning, and discipline. Shared code gets forked, diverges, and becomes a source of subtle bugs. I’ve seen organizations with thirty repos where “shared utilities” existed in five slightly different versions across teams, each with its own bugs.

The Hybrid Reality

Most organizations I’ve worked with end up somewhere in the middle. A few domain-aligned monorepos (one for the core platform, one for data pipelines, one for customer-facing apps) with polyrepo isolation between domains. This is fine as long as it’s deliberate.

The danger is the accidental hybrid: some code is in the monorepo because someone put it there, some is in separate repos because someone else preferred that, and nobody wrote down why. That’s how you get the worst of both worlds.

Decision Checklist

Answer these:

• Cross-service changes happen weekly or more? Lean monorepo.
• Teams need independent release schedules? Lean polyrepo.
• You have (or will build) dedicated build tooling capacity? Monorepo is feasible.
• Permissions and isolation are a hard requirement? Polyrepo is simpler.
• Shared code is a core part of delivery speed? Monorepo makes it visible and maintainable.

If you’re split 3-2 either way, you’re probably fine with either model. Pick one and commit to owning the tradeoffs.

Migration Is Possible but Expensive

I’ve helped teams move in both directions. It’s never a weekend project. You need a clear boundary map, a dependency inventory, and a staged migration plan. Treat it like a product initiative with dedicated ownership, not a “we’ll do it gradually” background task. Gradual migrations without ownership die quietly and leave you in the worst-of-both-worlds hybrid.

On day three, during a traffic spike, the container hit 4 GB. Kubernetes evicted it. The replacement pod landed on the same node, inherited the same traffic, and the cycle repeated. The node started thrashing. Other pods on that node, including a critical payment service, got evicted too. What started as one leaky container turned into a 45-minute partial outage.

The fix was three lines of YAML. The investigation took most of a day.

The One Thing You Need to Understand

CPU is compressible. When a container exceeds its CPU limit, Kubernetes throttles it. The process slows down but keeps running.

Memory isn’t compressible. When a container exceeds its memory limit, the kernel kills it. OOMKilled. The process restarts from scratch.

That asymmetry drives almost every resource management decision. CPU limits trade throughput for predictability. Memory limits are a hard safety net. Treat them differently.

Requests vs Limits

Requests tell the scheduler how much capacity to reserve. They’re a guarantee. The scheduler won’t place your pod on a node unless that capacity is available.

Limits tell the runtime when to intervene. CPU limits trigger throttling. Memory limits trigger OOMKill.

A cluster can overcommit limits: the sum of all limits can exceed the node capacity. It can’t overcommit requests. This is why requests matter more for scheduling stability.

Setting Them

Start with measurements, not guesses. If your monitoring shows a Go service using 150m CPU on average and 200 MB of memory, set requests near those values:

resources:
  requests:
    cpu: "200m"
    memory: "256Mi"
  limits:
    memory: "512Mi"

Notice: no CPU limit. For latency-sensitive services, CPU limits can cause throttling spikes even when the node has spare capacity. The CFS throttling behavior in Linux is well-documented and surprising: a service using 180m of CPU can still get throttled with a 200m limit because of how the quota is applied per scheduling period.

I default to: set CPU requests, skip CPU limits, always set memory limits. That is a change from what I recommended in 2020 . If you need CPU limits for noisy-neighbor isolation, set them generously: 2x to 3x the request.

Measuring and Iterating

Don’t set values once and forget them. Traffic patterns change. Code changes. Dependencies change.

Look at these metrics over at least two weeks:

• Container CPU usage (the actual usage, not the request)
• Memory working set (not RSS; working set is what Kubernetes uses for eviction decisions)
• Throttling events (container_cpu_cfs_throttled_seconds_total)

Set requests around the P50 usage. Set memory limits around the P99 plus a buffer. If you’re seeing throttling events on a latency-sensitive service, either raise the CPU limit or remove it.

Tools like VPA (Vertical Pod Autoscaler) can generate recommendations. Use them as input, not as autopilot. I’ve seen VPA recommendations that would have cut memory limits below the application’s startup footprint. Trust but verify.

QoS Classes

Kubernetes assigns a quality-of-service class based on how you configure requests and limits:

• Guaranteed: requests equal limits for both CPU and memory. Last to be evicted.
• Burstable: requests set but not equal to limits. Most common. Middle priority for eviction.
• BestEffort: no requests, no limits. First to be evicted. Never appropriate for production.

The team from my opening story ran BestEffort pods on shared nodes. When the node ran out of memory, the BestEffort pods died first, as designed. But the leaky pod kept landing back on the same node, and the pressure spread to the other pods running there.

Guardrails

Two cluster-level mechanisms prevent the “no resources specified” scenario:

LimitRange sets default requests and limits for containers that don’t specify their own. Set this on every namespace. The defaults don’t need to be perfect. They need to prevent BestEffort pods from accidentally running in production.

ResourceQuota caps the total resource usage in a namespace. This prevents one team from consuming the cluster. Set quotas per team namespace and review them quarterly.

At a large consumer platform, we enforced both. LimitRange provided sane defaults, and ResourceQuota kept growth predictable. Teams that needed more resources had to justify it, which had the side effect of surfacing inefficient services early.

The Three Failure Modes

OOMKilled. Memory limit is too low, or the application has a leak. Check the limit against actual usage. If usage grows linearly over time, it’s a leak: fix the code, not the limit.

CPU throttling. Shows up as latency spikes with available CPU on the node. The container is being throttled by CFS even though the node isn’t overloaded. Raise or remove the CPU limit.

Pending pods. Requests are too large for available capacity, or scheduling constraints (node selectors, taints, affinity rules) are too restrictive. Either reduce requests, add capacity, or relax the constraints.

All three are observable. All three are fixable. The usual problem is that nobody looks at the metrics until an incident forces it.

The Minimum

1. Set memory limits on every container. No exceptions.
2. Set CPU requests based on measured usage.
3. Apply LimitRange defaults on every namespace.
4. Apply ResourceQuota on team namespaces.
5. Monitor throttling, OOMKill events, and pending pods.
6. Review resource settings quarterly or after significant traffic changes.

Resource management isn’t glamorous work. It’s the kind of thing that only gets attention after an outage. But three lines of YAML and fifteen minutes of looking at Grafana dashboards can prevent the kind of cascading failure that ruins your weekend.

I’ve hardened clusters across multiple organizations and I keep seeing the same gaps I wrote about in 2019 . This post walks through the layers that matter, with config you can actually apply. Some of this comes from taking part in cyber-defense exercises, where “assume breach” was the operating model and defense in depth was the only design that survived contact with reality.

Start with a threat model (seriously)

I know. Nobody wants to do this part. But every security control you skip because “we’re not a bank” is a control an attacker won’t encounter. You don’t need a 40-page document. You need answers to three questions:

1. What data would hurt you if it leaked?
2. Who has access to your cluster, and how?
3. If a container gets compromised, what can the attacker reach?

If you can’t answer these quickly, start there before touching any YAML.

Pod security: the non-negotiable baseline

Every container should run with the least privilege possible. Full stop. Here is what that looks like in practice:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000
  containers:
    - name: app
      image: myregistry/app:v1.2.3@sha256:abc123...
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"
        limits:
          cpu: "500m"
          memory: "256Mi"

Key points:

• runAsNonRoot: true: if your app needs root, fix the app. I’ve seen exactly one legitimate case for running as root in a non-system container in the last three years.
• Drop ALL capabilities, then add back only what’s needed. Most applications need zero Linux capabilities.
• Read-only root filesystem: forces you to use volumes for writable paths, which is where you want writes anyway.
• Resource limits: not just for cost. An unbound container can starve the node and affect every other workload on it.
• Pin image digests: latest isn’t a version. Pin to a SHA256 digest for anything that matters.

PodSecurityPolicy was deprecated in 1.21 and is scheduled for removal in 1.25. Replace it. Pod Security Admission (beta in 1.23) implements three levels: privileged, baseline, restricted. At minimum, enforce baseline cluster-wide and restricted on sensitive namespaces:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

For more granular control, OPA Gatekeeper or Kyverno let you write custom policies. I prefer Kyverno for readability, but both work.

Network policies: default deny, then allow

Kubernetes networking is flat by default. Every pod can talk to every other pod. That’s fantastic for an attacker who compromises one service and wants to pivot laterally.

Start with a default deny policy on every namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then explicitly allow what’s needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:  # DNS
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

A critical gotcha: NetworkPolicy only works if your CNI supports it. Flannel doesn’t enforce network policies. Calico, Cilium, and Weave do. Verify enforcement in every environment, including staging. I’ve seen teams spend weeks writing policies that were silently ignored because the CNI was wrong.

Don’t forget egress. Ingress gets all the attention, but an attacker who compromises a pod and can make outbound HTTPS calls to the internet has an exfiltration path. Restrict egress to known destinations.

RBAC: strict and boring

Good RBAC is boring. It should be so restrictive that nobody notices it until they try to do something they shouldn’t.

• One service account per workload. Not one per namespace. Not the default service account.
• Scope roles to namespaces and specific resources. Cluster-wide roles are for platform teams, not applications.
• Disable automatic service account token mounting for pods that don’t need to talk to the API server:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
automountServiceAccountToken: false

For human access, use OIDC with your identity provider. Short-lived tokens from kubectl via OIDC are vastly better than long-lived kubeconfig certificates. Nobody should have cluster-admin in production unless they’re actively performing cluster operations, and even then, consider just-in-time access.

Secrets: they aren’t encrypted

This surprises people. Kubernetes Secrets are base64-encoded in etcd. That’s encoding, not encryption. Anyone with etcd access can read every secret in your cluster.

Fix this in layers:

1. Encrypt etcd at rest. Configure EncryptionConfiguration on the API server with aescbc or secretbox providers.
2. Use an external secrets manager. HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager with an operator like External Secrets. The secret value never lives in etcd.
3. Scope access. Only the service accounts that need a secret should be able to read it. RBAC applies to secrets too.
4. Rotate. If you can’t rotate a secret without a deployment, fix your rotation process before you fix the secret.

Never bake secrets into container images. Never put them in ConfigMaps. Never log them. These sound obvious, but I find violations in almost every cluster audit.

Control plane hygiene

The API server is the front door to everything. Lock it down:

• Disable anonymous authentication (--anonymous-auth=false).
• Enable audit logging. You need to know who did what and when.
• Restrict API server access to trusted networks. If your API server is reachable from the public internet, you have a bigger problem than this blog post covers.
• Keep etcd on a private network, TLS-encrypted, and only reachable by the API server.
• Treat kubeconfig files like production database credentials. Because that’s what they are.

If you’re on a managed service (EKS, GKE, AKS), the provider handles some of this. But “managed” doesn’t mean “secure.” Review what they expose and what they lock down. I’ve seen managed clusters with surprisingly permissive defaults.

Image supply chain

Stop using latest. Stop using mutable tags. Every production image should be:

• Built from a minimal base (distroless, Alpine, or scratch for Go binaries).
• Scanned for vulnerabilities in CI before it reaches a registry.
• Signed, ideally with cosign or Notary.
• Pulled from a private registry with access controls.

If you’re pulling public images directly into production, you’re trusting that Docker Hub (or whoever) hasn’t been compromised. That trust should make you uncomfortable.

Node-level protections

Nodes are part of your trust boundary. A compromised node means compromised pods.

• Use a minimal OS (Bottlerocket, Flatcar, Talos). Less software means fewer attack vectors.
• Patch regularly. Automate node rotation if you can.
• Lock down the kubelet API. Disable anonymous kubelet auth.
• Apply seccomp profiles. The RuntimeDefault profile is a good starting point and blocks a significant number of syscalls:

securityContext:
  seccompProfile:
    type: RuntimeDefault

Runtime detection tools like Falco are worth considering. They won’t prevent an attack, but they can alert you when something unexpected happens: a shell spawning inside a container, an unusual network connection, a binary that shouldn’t exist being executed.

Monitoring and incident response

Hardening without visibility is just hoping. You need:

• Centralized audit logs from the API server.
• Alerts on failed authentication attempts and privilege escalations.
• Monitoring for unexpected image pulls or outbound connections.
• A written incident response playbook. Not a novel. A one-page runbook that answers: who gets paged, what gets isolated, how do we preserve evidence.

Write the playbook before you need it. During an actual incident isn’t the time to figure out who has cluster access.

The operating discipline

None of this works as a one-time project. Security is operational:

• All cluster config changes go through code review. GitOps is your friend here.
• Stay on supported Kubernetes versions. Running an EOL version means running with known unpatched vulnerabilities.
• Test your controls in staging. I’ve seen policies that worked perfectly in staging and failed silently in production because the namespace labels were wrong.
• Run backup and restore drills. Not “we have Velero installed.” Actually restore from backup and verify the cluster works.

Kubernetes hardening is a set of defaults you enforce continuously, across pods, network, identities, secrets, and the control plane. When those defaults are solid, incidents become containable instead of catastrophic.

DORA metrics (deployment frequency, lead time for changes, change failure rate, time to restore) are good because they measure system outcomes. Not individual output. Not lines of code. Not story points. System outcomes. The moment you point them at individual engineers, you’ve already lost the plot.

The four metrics, briefly

The 2021 State of DevOps report also tracks reliability as a fifth measure, but the original four are what most teams mean by “DORA.”

Deployment frequency: how often code reaches production. Higher is generally better because it implies smaller batches and faster feedback. But a team deploying empty feature flags ten times a day isn’t actually faster.

Lead time for changes: the time from commit to production. This one is revealing. When lead time is high, the bottleneck is almost never “engineers are slow.” It’s reviews, environments, approval gates, flaky tests. Infrastructure problems disguised as people problems.

Change failure rate: the percentage of deployments that cause incidents or need rollback. The quality counterbalance to speed. If you’re deploying constantly but breaking things, you aren’t winning.

Time to restore: how quickly you recover after a failure. I care about this one the most. Your system will break. The question is whether you can fix it in minutes or hours.

The measurement trap

Here is where teams go wrong. They start measuring before they agree on definitions.

What counts as a deployment? Does a config change count? A feature flag flip? What about a deployment to staging that auto-promotes? Is a deployment “to production” when it hits the first canary or when it reaches 100% of traffic?

What counts as a failure? Only P1 incidents? Any rollback? A hotfix that ships within an hour?

If you don’t write these definitions down and get agreement, every team will interpret the metrics differently. Then someone will compare teams, the numbers will be meaningless, and trust in the whole system collapses.

I watched exactly this happen at one company. Two teams reported wildly different deployment frequencies. Turned out one team counted every Kubernetes pod restart as a deployment. The other only counted manual releases. Same dashboard, completely different realities.

Definitions, medians, and team-level numbers

Use rolling 30-day medians, not averages. Averages lie when you have one outlier incident that takes a week to resolve. Medians are boring and that’s the point.

Keep the metrics at the team or service level. Never at the individual level. The second you rank individuals by lead time, someone will start gaming commit timestamps. I’ve seen it happen.

Track trends against a team’s own baseline. Comparing Team A to Team B is almost always misleading. Teams have different codebases, different customer profiles, different risk tolerances. A payments team deploying twice a week with zero failures is doing better than a marketing team deploying daily with a 15% failure rate, even though the dashboard would suggest otherwise.

Automate collection. If someone has to manually tag deployments or log incidents for the metrics to work, the data will be inconsistent within a month. Pull from your CI/CD system, your version control, and your incident tracker. If those systems don’t have the events you need, fix that first.

The traps to avoid

Don’t use DORA for performance reviews. I can’t say this enough. The moment engineers know their deployment frequency affects their bonus, they will split PRs into meaningless fragments. You’ll get higher numbers and worse software.

Don’t optimize one metric in isolation. Deployment frequency without change failure rate is just recklessness. Lead time without quality is just speed. They work as a set or not at all.

Don’t hide incidents. If the change failure rate shows up in some leadership scorecard, teams will reclassify incidents as “expected behavior” or “not caused by the deployment.” I’ve seen teams debate for twenty minutes whether something was “really an incident” just to protect their number.

Don’t expect precision. These are directional signals. Trends matter. A lead time that drops from days to hours over a quarter is meaningful. The difference between 2.3 hours and 2.7 hours is noise. Treat it as noise.

Pair the numbers with conversations

Pair DORA metrics with qualitative feedback. Ask teams: “What’s slowing you down this sprint?” The metrics should confirm what teams already feel. If the numbers say everything is fine but engineers are miserable, the numbers are wrong, or more likely they’re measuring the wrong thing.

Use the metrics to fund improvements. Lead time is high because the test suite takes 40 minutes? That’s a concrete investment case. Change failure rate is climbing? Maybe the team needs better staging environments, not a talking-to.

DORA is a compass. It tells you roughly which direction you’re heading. It doesn’t tell you where every pothole is. Use it that way and it’s genuinely valuable. Overfit to the numbers and you’ll optimize for the dashboard while the actual engineering culture rots underneath.

I wrote about Terraform patterns back in 2019 when I was mostly dealing with teams of 5-15 engineers sharing a handful of state files. Since then I’ve helped scale Terraform at organizations with hundreds of engineers, dozens of AWS accounts, and thousands of managed resources.

Some of what I wrote in 2019 held up. Some of it was naive. Here is the updated version.

What Held Up

Split state by ownership and blast radius. This is still the single most important Terraform pattern. One state file per component, scoped by team ownership and failure domain. Networking is separate from application infrastructure. Shared data stores are separate from service-specific resources.

The reason hasn’t changed: a terraform plan that touches resources owned by three different teams is a plan nobody wants to review and nobody wants to approve. Small states, clear ownership, independent deploys.

Remote state with locking is non-negotiable. S3 + DynamoDB for AWS. Still the standard baseline in 2021. Encrypt state at rest, restrict access to the minimum set of IAM roles, and never store state locally.

terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "prod/networking/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Explicit cross-component dependencies. Publish outputs. Consume them through terraform_remote_state data sources or (better) through SSM parameters or a service registry. Don’t assume. Don’t hardcode. If many stacks need the same value, treat it as a versioned contract.

What I Do Differently Now

Modules Are a Product, Not a Library

In 2019 I treated modules as code reuse. Wrong framing. Modules at scale are a contract between the platform team and everyone else. The interface matters more than the implementation.

A good module in 2021:

module "service" {
  source  = "company/ecs-service/aws"
  version = "~> 2.1"

  name        = "orders-api"
  environment = "production"
  # everything else has sane defaults
}

Narrow input surface. Strong defaults. Documented outputs. Pinned versions. If a product team has to read the module source to use it, the module has failed.

I now recommend running an internal module registry. Terraform Cloud has one built in. If you’re self-hosting, a simple Git-based registry with tagged releases works. The point is discoverability and versioning. Without it, module consumption is copy-paste and you lose the ability to push improvements to consumers.

Workspaces Are Still a Trap

I was cautious about workspaces in 2019. I’m more forceful now: don’t use them for environment separation. Directories per environment are uglier but dramatically safer.

With directories, each environment has its own backend config, its own variable files, and its own state. You can review a production change without worrying about accidentally picking the wrong workspace. You can apply to staging without touching production. The blast radius is obvious from the file path.

Workspaces hide this separation behind a CLI flag. I’ve seen production outages caused by terraform workspace select picking the wrong environment. Twice at different companies. Both times the engineer was experienced and careful. The tool made a catastrophic mistake easy.

Use workspaces for ephemeral environments (PR preview environments, load test clusters) where the lifecycle is short and the blast radius is low.

CI Is the Only Operator

In 2019 I said Terraform should run in CI. In 2021 I say nobody should have terraform apply credentials on their laptop. Period.

The workflow:

1. Developer opens a PR with infrastructure changes
2. CI runs terraform plan and posts the output as a PR comment
3. A reviewer reads the plan (the actual plan output, not just the code diff)
4. Merge to main triggers terraform apply
5. Short-lived credentials via OIDC federation. No long-lived AWS keys anywhere.

This gives you an audit trail, consistent environments, peer review of actual changes, and no “I ran apply from my laptop and I’m not sure what happened” incidents.

Policy Enforcement Is Table Stakes

In 2019, policy enforcement felt optional. In 2021, after watching too many “someone opened 0.0.0.0/0 on a production security group” incidents, it’s mandatory.

OPA with Conftest is my go-to for self-hosted setups. Terraform Cloud has Sentinel. Either works. The goal is the same: block known-bad patterns before they reach production.

Policies I enforce everywhere:

• No public ingress on security groups without explicit approval
• All resources must have required tags (team, environment, service)
• No inline IAM policies; use managed policies
• S3 buckets must have encryption and versioning enabled
• RDS instances must not be publicly accessible

These aren’t business logic. They’re guardrails that prevent the easy mistakes.

Secrets Management Got Better

State files contain sensitive data. This was true in 2019 and it’s still true in 2021. The difference is that the tooling around it improved.

Use a secrets manager (AWS Secrets Manager, HashiCorp Vault) and pass sensitive values at runtime. Mark outputs as sensitive so they don’t leak in plan output or CI logs. Never put credentials, API keys, or certificates in .tfvars files.

output "database_password" {
  value     = aws_db_instance.main.password
  sensitive = true
}

Drift Detection Isn’t Optional

Terraform assumes it’s the only thing changing your infrastructure. It isn’t. Console clicks, auto-scaling events, AWS service updates: drift happens.

Run terraform plan on a schedule (daily for production, weekly for staging) and alert on unexpected changes. This catches two things: manual changes that should have been codified, and resource changes that indicate a problem.

The Operational Routine

Terraform isn’t deploy-and-forget. It needs maintenance.

• Weekly: Review plan output for drift in production stacks
• Monthly: Update provider versions (pin them, but don’t let them go stale for months)
• Quarterly: Audit state access, review module usage, clean up orphaned resources
• Every PR: Review the plan output, not just the HCL diff

What I Would Tell 2019 Me

Invest in the module registry earlier. It’s the highest-leverage thing you can build for Terraform at scale. Every team that consumes a well-designed module is a team that isn’t reinventing infrastructure patterns.

Stop fighting workspaces. Just use directories. The extra files are worth the safety.

Get apply off of laptops on day one, not “when we have time.” The incident that happens before you migrate to CI will be more expensive than the migration itself.

Terraform scales fine. The syntax isn’t the hard part. Coordination, ownership, and safety are the hard parts. Get those right and the HCL takes care of itself.

I’ve looked closely at platform teams in a number of large organizations. The pattern is depressingly consistent. There’s a “platform team” somewhere in the org chart. They have built some Terraform modules. Maybe a Jenkins pipeline or two. Possibly a Backstage instance that three people use.

Meanwhile, developers still open Jira tickets to get a database provisioned. Lead time from “I need an environment” to “I have an environment” is measured in days. Sometimes weeks.

The platform team is busy. They are just busy building the wrong thing.

The Five Levels

I use a maturity model when I assess platform teams. Not because I love frameworks, but because it gives me and the team a shared language for “here’s where you are” and “here’s where the payoff is.”

Level 1: Ad hoc. Everything is manual. Knowledge lives in people’s heads. Getting a new environment means finding the one person who knows how. Deployments happen on Tuesdays because that’s when the ops team has capacity. This is where most startups are and where some surprisingly large companies still live.

Level 2: Standardized. Infrastructure as code exists. CI pipelines exist. But the platform team gates everything. Developers can build, but they can’t deploy or provision without filing a request. The automation is real, but the bottleneck just moved.

Level 3: Self-service. Developers can provision environments, deploy services, and roll back on their own, within guardrails. This is where the platform starts paying for itself. Wait time drops. Consistency improves. The platform team stops being a ticket queue and starts being an enablement function.

Level 4: Product-minded. The platform has a roadmap based on user feedback. There are golden paths for common service types. An internal CLI or portal handles the 80% case. Progressive delivery is standard. The platform team runs NPS surveys and actually reads them.

Level 5: Optimized. Data drives changes to platform defaults. Automated remediation handles known failure modes. Cost, reliability, and security metrics improve continuously. This is rare. I’ve seen maybe two organizations here.

Where Most Enterprises Actually Are

Based on what I’ve seen in large, telecom-scale organizations, most enterprises are at Level 2. Some pockets are at Level 3. Almost nobody is at Level 4.

The gap between Level 2 and Level 3 is the most expensive gap in enterprise software delivery. It’s the difference between “we have automation” and “developers can actually use it without waiting.” Every day a developer spends waiting for provisioning is a day they aren’t shipping features. Multiply that by hundreds of developers and the cost is staggering.

Why Platform Teams Get Stuck

Three patterns I see repeatedly:

Building tools instead of workflows. A team builds a slick Terraform module for spinning up EKS clusters. Great. But developers still need to file a ticket to get the module run for them. The tool exists. The self-service doesn’t. The developer experience didn’t actually change.

Centralizing instead of enabling. The platform team takes on more ownership instead of giving ownership away safely. They become the bottleneck for every infrastructure change. They are drowning in operational work and have no time to build the self-service layer that would free them.

Ignoring their users. Platform teams that never talk to developers build platforms that developers avoid. I’ve walked into organizations where three competing shadow platforms exist because the official one didn’t solve actual problems.

How to Move Up

Level 1 to 2: Document and automate the basics

Write down how to get from code to production. Adopt infrastructure as code for core services. Get CI pipelines running. Define standard environments. This isn’t glamorous work, but it’s the foundation.

Level 2 to 3: Kill the ticket queue

This is the most impactful transition. Give developers templates and modules they can run themselves. Automate deployments with guardrails instead of gates that require human approval for every change. Bake observability in by default so developers don’t have to think about it. Clarify who owns what.

The goal is simple: a developer should be able to go from “I want to deploy” to “it’s deployed” in minutes, not days.

Level 3 to 4: Treat it like a product

Build an internal CLI or portal around the most common workflows. Publish golden paths and reduce unnecessary choices. Measure developer satisfaction. Run feedback sessions. Build a roadmap based on what developers actually need, not what the platform team thinks is interesting.

Level 4 to 5: Let data drive

Automate remediations for known incident types. Tune platform defaults based on real usage patterns. Track cost and performance per service. This is continuous improvement territory; it never ends.

The Assessment

When I assess a platform, I score six dimensions independently:

1. Infrastructure provisioning and environment consistency
2. Delivery workflows and release safety
3. Observability and operational feedback
4. Security controls and policy enforcement
5. Developer experience and self-service
6. Platform team operating model and ownership

Each one gets a 1-5. I don’t average them. I look at the lowest two scores because that’s where the biggest drag on delivery lives. A platform with great CI/CD but terrible provisioning still has developers waiting.

Metrics That Show It’s Working

Forget vanity metrics. These tell you if your platform is working:

• Lead time from commit to production. If this is longer than an hour for most services, something is broken.
• Time to provision a new environment. If this takes more than a day, your self-service isn’t actually self-service.
• Deployment frequency. Teams that trust their platform deploy more often.
• Developer satisfaction. Ask them. Regularly. Act on what they say.

The best platform I’ve seen wasn’t the one with the most tools. It was the one where a new developer could deploy their first change to production on day one. Everything else followed from that principle.

It’s an Organizational Problem

Most platform immaturity is organizational. The platform team doesn’t have a mandate to remove ticket queues. Leadership doesn’t treat developer experience as a first-class investment. The team is understaffed for the scope they’re expected to cover.

If you’re a platform engineer reading this and nodding, show this to your VP of Engineering. The conversation needs to happen at the level where headcount and priorities are decided. No amount of Terraform modules fixes a staffing problem.

The worst part: when I cross-referenced with our flag metadata, only about 200 had an owner listed. The rest were orphans. Nobody knew who created them, what they controlled, or whether they were safe to remove. One flag had been set to “false” for 14 months. The code path behind it was unreachable. But nobody dared delete it because nobody knew what it did.

That’s what feature flags look like at scale when you don’t have discipline. And I’m writing this post because I don’t want you to end up there.

Flags are great. Stale flags are debt.

The pitch for feature flags is solid. Separate deployment from release. Ship code to production behind a flag. Roll out gradually . Roll back instantly. Run experiments. Gate features by customer plan. Use kill switches during incidents.

All of that works. I’m a believer. Flags have saved me multiple times during incidents. Flipping a kill switch is faster than deploying a rollback. Having a gradual rollout catch a regression at 5% of traffic instead of 100% is valuable.

The problem isn’t the concept. The problem is lifecycle management. Flags are easy to create. Nobody wants to remove them.

Categorize your flags or drown

After the 847-flag incident, we introduced four categories:

Release flags are short-lived. Control rollout of a new feature. Expected lifetime: days to weeks. Must be removed once the feature is fully rolled out.

Experiment flags are medium-lived. Power A/B tests. Expected lifetime: duration of the experiment. Must be retired with a clear outcome documented.

Operational flags are potentially long-lived. Kill switches, circuit breakers, graceful degradation. These can stay, but they need periodic review.

Permission flags are permanent. Gate access by customer plan or contract. These are really entitlements, not feature flags, and should be treated differently in your code.

The categories matter because the cleanup rules are different. A release flag that’s still around after a month is a problem. An operational kill switch that’s been around for a year is fine.

The rules that saved us

After the cleanup, we implemented a simple policy. Every new flag must have:

1. An owner (the person who will remove it)
2. A type (release, experiment, operational, permission)
3. An expected removal date (except operational and permission flags)
4. A documented default value and failure mode

If any of those are missing, the flag can’t be created. We enforced this in our flag management wrapper.

type Flag struct {
    Name       string
    Owner      string
    Type       FlagType
    RemoveBy   time.Time // zero for operational/permission
    DefaultVal bool
    FailMode   FailMode  // FailOpen or FailClosed
}

func (s *FlagService) Create(f Flag) error {
    if f.Owner == "" {
        return errors.New("flag must have an owner")
    }
    if f.Type == Release && f.RemoveBy.IsZero() {
        return errors.New("release flags must have a removal date")
    }
    // ...
}

We also added a weekly report: flags past their removal date, flags with no evaluations in the last 30 days, and flags where the “off” path has never been exercised in production. That last one is subtle but important: if you’ve never seen the fallback path in production, you don’t actually know it works.

Fail-open or fail-closed

This is the decision most teams skip. What happens when the flag service is down?

For a payment flow, you probably want fail-closed. Better to show an error than process a payment through an untested code path.

For a UI experiment, fail-open makes sense. Show the default experience. Nobody notices.

At one enterprise, the flag service went down for 20 minutes and their checkout flow broke because every flag defaulted to “false”, including the flag that enabled their only working checkout flow. They’d never tested what “all flags off” looked like. Don’t be that team.

The cleanup problem is a people problem

Technical solutions help. Expiration dates, automated reports, lint rules that flag dead code paths. But the real problem is incentives.

Nobody gets credit for removing a feature flag. Shipping a new feature is visible. Cleaning up after it isn’t. So flags accumulate, like any other unowned tech debt .

On that platform, we made flag cleanup part of the definition of done. A feature isn’t finished when it’s rolled out to 100%. It’s finished when the flag is removed and the old code path is deleted. We tracked “flag debt” on our engineering dashboard alongside other health metrics.

At larger organizations, I recommend a quarterly flag review. Pull up the full inventory. For every flag without a recent evaluation, ask: is this still needed? For every flag past its removal date, assign cleanup as a sprint task. It’s unglamorous work. It prevents the kind of mess that makes your codebase feel hostile to new engineers.

Keep it simple

The best feature flag setup I’ve seen is also the simplest. Boolean flags with deterministic evaluation. Clear categories. Mandatory ownership. Aggressive cleanup.

The worst setups are the ones with complex targeting rules, flags that depend on other flags, and evaluation logic spread across three different layers. If a single flag requires a whiteboard to explain, break it into smaller flags or put the complexity in application code where it can be tested properly.

Feature flags are a powerful tool. They’re also a maintenance commitment. Treat them like production configuration, because that’s what they are.

It isn’t a methodology. It means instrumenting your code before you ship it. That’s the whole thing.

And yet I keep walking into systems, at startups and enterprises alike, where observability was bolted on after the first production fire. The logs are a mess. Every service logs differently. Metrics exist for some endpoints but not others. Nobody has traces. The dashboards are either empty or full of vanity charts that nobody looks at.

The actual problem

When observability comes last, every team invents their own approach. Service A logs JSON with request_id. Service B logs plain text with req_id. Service C doesn’t log request IDs at all. You find this out at 2am during an outage while trying to correlate a failure across three services.

I’ve lived this. At the fintech startup we had a period where our financial data pipeline logs were useless for debugging cross-service issues because every team had picked their own field names. Fixing that retroactively took weeks. If we’d agreed on a format before writing the services, it would’ve taken an afternoon.

What “ODD” actually means in practice

Before you write a feature, answer three questions:

1. How will I know this is working correctly in production?
2. How will I know this is slow?
3. How will I know this is broken?

If you can’t answer those, you’re not ready to write the code. That’s the whole framework. No acronym needed.

Structured logs or nothing

Your logs should be structured JSON with consistent field names across every service. Non-negotiable.

{"level":"info","event":"order_created","request_id":"abc-123","order_id":"ord-456","duration_ms":42}

Every log line gets a request_id. Every log line gets an event name. Every log line gets a level. If you’re logging fmt.Println("something happened") in production Go code, we need to talk.

Pick your field names once. Write them down. Enforce them in code review. This is boring work. It pays off every single incident.

Metrics: RED and stop

For services, use RED metrics . Rate, Errors, Duration. For every endpoint. That’s your baseline.

I see teams go wild with custom metrics on day one. Thirty metrics per service, half of them never queried. Meanwhile they’re missing basic error rate tracking on their most critical endpoint. Start with RED. Add custom metrics when you have a specific question that RED can’t answer.

One thing that will absolutely burn you: high-cardinality labels. If you’re putting user IDs or full URLs into metric labels, you’re building a cost bomb. I saw one team’s Prometheus storage costs triple in a month because someone added a path label that included query parameters. Keep labels to things like method, status_code, service. Low and predictable.

Traces aren’t optional

Distributed tracing used to feel like a luxury. It’s not. If you’re running more than two services, you need traces. Full stop.

Every inbound request starts a trace. Every outbound call propagates the trace context. This is a few lines of middleware in Go. It’s trivial. And it’s the difference between “I think the problem is in the payment service” and “I can see the exact call that took 4 seconds.”

Sample if you need to for cost reasons. But sample consistently. Don’t sample 100% on staging and 1% on production and then wonder why you can never find the trace you need.

Make it part of code review

This is where it sticks or falls apart. If observability isn’t in your code review checklist, it won’t happen.

When I review a PR that adds a new endpoint, I look for:

• Does the handler emit RED metrics?
• Are key events logged with stable fields?
• Does the trace propagate to downstream calls?
• Are the labels safe?

If the answer is no, the PR isn’t ready. Same as missing tests. Same as missing error handling. Observability isn’t a follow-up ticket. It ships with the feature.

Alerts that don’t suck

Most alerts are terrible. They fire on every blip, train everyone to ignore them, and then nobody notices when something actually breaks.

Alert on symptoms, not causes. Alert on “error rate is above X% for Y minutes,” not “one request returned a 500.” Better yet, use SLO-based alerts . Set an error budget. Alert when you’re burning through it too fast. This single change cut our alert noise at my infrastructure startup by something like 80%.

I have a rule when I start working with a new team: before I recommend anything, I ask the team to show me how code gets from a developer’s laptop to production. Every step. Every tool. Every credential.

The answer is almost always incomplete. Not because people are hiding things, but because nobody has mapped the full path recently. That incomplete picture is where security gaps live.

DevSecOps is the practice of embedding security checks into the workflow developers already use. You can’t buy it. If it slows them down, they’ll circumvent it. If it’s invisible and fast, they’ll barely notice it’s there. That’s the target.

Layer 1: the developer’s machine

The cheapest place to catch a security issue is before it ever leaves the developer’s laptop. Pre-commit hooks are the first line.

Here is what I install on every project:

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: gitleaks
        name: gitleaks
        entry: gitleaks --path=. --unstaged -v
        language: system
        pass_filenames: false
  - repo: https://github.com/hadolint/hadolint
    rev: v2.1.0
    hooks:
      - id: hadolint
        args: ['--ignore', 'DL3008']
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.48.0
    hooks:
      - id: terraform_validate
      - id: terraform_tfsec

Gitleaks catches secrets before they hit Git history. Hadolint catches Dockerfile anti-patterns. Terraform validation and tfsec catch infrastructure misconfigurations.

The critical design principle: these hooks must run in under 10 seconds. If a pre-commit hook takes 30 seconds, developers disable it. I’ve seen this happen at every organization I’ve worked with. Speed isn’t optional.

For Go projects specifically, I also add go vet, staticcheck, and gosec as pre-commit checks. Gosec catches common security issues like SQL injection patterns and hardcoded credentials.

Layer 2: CI pipeline security stages

Pre-commit hooks are a safety net, not a gate. The real enforcement happens in CI. Here is the pipeline structure I implement, using GitHub Actions as the example:

# .github/workflows/security.yml
name: Security Checks
on: [push, pull_request]

jobs:
  secrets-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - uses: zricethezav/gitleaks-action@v1.5.0

  sast:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run gosec
        uses: securego/gosec@master
        with:
          args: '-exclude-generated ./...'
      - name: Run Semgrep
        uses: returntocorp/semgrep-action@v1
        with:
          config: >-
            p/owasp-top-ten
            p/golang

  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run Trivy for dependencies
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'

  container-scan:
    runs-on: ubuntu-latest
    needs: [sast]
    steps:
      - uses: actions/checkout@v2
      - name: Build image
        run: docker build -t app:${{ github.sha }} .
      - name: Scan image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'app:${{ github.sha }}'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'

Four jobs: secrets scanning, static analysis, and dependency scanning run in parallel, and container scanning runs once SAST passes. There’s no point scanning an image if the code has critical issues.

The key decision is the gating policy. I use a two-tier approach:

• Critical and High severity: block the pipeline. The PR can’t merge.
• Medium and Low: create a tracking issue automatically, assign it to the team, set a remediation deadline. Don’t block the pipeline.

This is the balance that makes DevSecOps sustainable. If you block on everything, teams stop caring about the signal because it’s always red. If you block on nothing, the findings pile up and nobody fixes them.

At one telecom company, we went from zero automated security checks to full pipeline coverage in three weeks. The first week was brutal: the initial scan found over 200 findings across their repositories. We triaged them into buckets, fixed the critical ones immediately, and created a backlog for the rest. Within a month, new PRs were consistently clean.

Layer 3: infrastructure and runtime controls

Static analysis catches what it can see. It can’t see runtime behavior, configuration drift, or privilege escalation. That’s where policy-as-code and runtime controls come in.

I use Open Policy Agent (OPA) with Gatekeeper for Kubernetes-based deployments, as part of the baseline hardening every cluster should get. Here is a constraint that prevents containers from running as root:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPAllowedUsers
metadata:
  name: require-non-root
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system"]
  parameters:
    runAsUser:
      rule: MustRunAsNonRoot

And one that requires resource limits on every container:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
  name: require-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    cpu: "2"
    memory: "2Gi"

These constraints are enforced at admission time. If a deployment tries to run a container as root, the API server rejects it. No exceptions. No “we’ll fix it later.” The cluster enforces the policy.

For network policies, I default to deny-all and explicitly allow what is needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then each service gets a specific policy that allows only its known dependencies. This is tedious to set up and worth every minute. When a service gets compromised, lateral movement is limited by the network policy. I first saw this pattern in cyber-defense exercises. The principle is the same regardless of scale. Default deny. Explicit allow. No implicit trust.

Layer 4: triage and ownership

This is where most DevSecOps implementations die. The tools work. The scans run. The findings pile up. Nobody owns them.

My triage system is simple:

• Critical: fix within 24 hours. Page the on-call if necessary. These are actively exploitable issues in production code.
• High: fix within one sprint. Track in the regular backlog. Team lead owns the follow-up.
• Medium: fix within 30 days. Batch these into security-focused work blocks.
• Low: fix when convenient, or accept the risk with documentation.

Every finding gets an owner. Not a team: a person. “The security team will handle it” is how findings go to die.

I also set up a weekly triage meeting. Fifteen minutes. Review new findings, check progress on open items, and escalate anything that’s stuck. At one company, this meeting reduced the median age of open vulnerabilities from 47 days to 11 days in two months. Not because we added more tools. Because someone was consistently asking “is this fixed yet?”

Secrets management

Secrets in source control are the most common security finding I see. Hardcoded API keys, database passwords in config files, AWS credentials in test fixtures. It happens everywhere.

The fix is structural, not behavioral. Don’t tell developers “be careful with secrets.” Give them a system that makes it hard to leak secrets in the first place.

• All secrets live in a vault (HashiCorp Vault, AWS Secrets Manager, whatever fits your stack).
• Applications pull secrets at runtime, never at build time.
• CI credentials are scoped to individual jobs and rotate automatically, the same rule I apply when hardening build pipelines .
• Pre-commit hooks catch anything that slips through.
• Git history scanning runs on the full repository, not just the diff. Because secrets removed in a later commit are still in the history.

What I’ve learned from doing this repeatedly

The technical setup is the easy part. The hard part is culture.

Developers aren’t the enemy of security. They’re the delivery mechanism for security. If you treat them as adversaries (locking everything down, blocking every build, requiring approvals for trivial changes), they’ll find workarounds. Shadow builds. Personal accounts. Manual deployments that skip the pipeline entirely.

The organizations that succeed with DevSecOps are the ones that make secure the default and make the default fast. Secure container base images that are pre-approved. Templates that include security configs out of the box. Scanning that runs in parallel with tests instead of after them.

Security should feel like good tooling. When it does, adoption is automatic.

Almost none of them are ready for any of it.

I say this as someone who has watched ML projects go sideways in multiple organizations. I keep seeing the same pattern. A data science team trains a model in a notebook. It works great in evaluation. Someone manually deploys it. It slowly degrades over months because nobody is monitoring whether the real world still looks like the training data. Eventually a stakeholder notices the predictions are garbage and the team scrambles to retrain.

That’s not an MLOps problem. That’s a “we don’t even have the basics” problem.

What MLOps actually requires

ML systems aren’t just model files. They are data pipelines , training code, feature stores, serving infrastructure, and monitoring, all tightly coupled in ways that traditional software isn’t. A code change might not matter. A data distribution shift always matters. The output is probabilistic. Degradation is silent.

MLOps is the discipline that keeps this entire system reliable. But discipline implies prerequisites.

Can your team reproduce a model from six months ago? Not just the code, but the exact dataset, the feature definitions, the hyperparameters, the environment? If no, you’re not ready for MLOps. You’re ready for version control.

Can you answer “what data was this model trained on?” If that question requires someone to check their memory or dig through a shared drive, you’re not ready for automated retraining pipelines. You’re ready for data documentation.

Can you tell when a model’s predictions are getting worse? Not when a stakeholder complains, but when the drift actually starts? If you have no monitoring on prediction distributions or feature distributions, you’re not ready for canary deployments of models. You’re ready for basic observability.

MLOps v0: where to start

Here is what I tell teams who want MLOps:

Pick your single most important model. The one that actually affects revenue or user experience. Write down its data dependencies. All of them. Where does the training data come from? Who owns those sources? What happens if a schema changes upstream?

Then add three things:

1. Dataset versioning. When you train, snapshot the data. Tag it. Store it alongside the model artifact. You need to be able to say “model v7 was trained on dataset snapshot X from date Y.”

2. A release checklist. Before a model goes to production, someone checks it against a known baseline on held-out data. Latency and memory usage are within serving targets. Known edge cases are tested. This is a manual checklist. That’s fine. Manual and consistent beats automated and nonexistent.

3. One monitoring signal. Pick one metric that reflects whether the model is doing its job in the real world. Not offline accuracy; something tied to a business outcome. Monitor it. Set an alert threshold. When it degrades, retrain.

That’s it. That’s your MLOps v0. It’s not glamorous. It won’t impress anyone at a conference. But it prevents the silent failure mode that kills most ML projects.

When you’re actually ready for more

Once those basics are reliable (and I mean actually reliable, not “we set it up three months ago and haven’t checked since”), then you can start adding automation. Automated retraining triggers. Feature stores. Shadow deployments. A/B testing infrastructure. Model registries with governance controls.

But every layer you add assumes the previous layer works. Automated retraining is useless if your data isn’t versioned. A feature store is overhead if your feature definitions aren’t documented. A/B testing is theater if you can’t measure business outcomes.

The MLOps vendor market is exploding right now. Everyone is selling platforms. Most of them solve problems that come after the problems most teams actually have.

Fix the boring stuff first. The tooling can wait.

I’ve been working with companies on their infrastructure stories for a while now. I keep seeing the same pattern. Someone reads a blog post about how Spotify or Netflix built an internal developer platform. They get excited. They rebrand their ops team. Nothing actually changes.

The naming problem

Our industry has a compulsive need to mint new labels. We did it with “DevOps” when we meant “ops engineers who can code.” We did it with “SRE” when we meant “on-call but with better job titles.” Now we’re starting to do it with “platform teams” and “internal developer platforms” when we mean “build internal tools so developers stop filing tickets.”

The underlying idea is fine. Good, even. Build self-service paths. Automate the repetitive stuff. Treat your internal tooling like a product. None of this is new. It’s what good infrastructure teams have done for years, and what I tried to build at the fintech startup . We just keep repackaging it.

What a platform has to do

Strip away the branding and you get a short list. Your developers should be able to spin up a service, deploy it, and observe it without waiting on another team. That’s it. That’s the whole thing.

The hard part was never the concept. It’s the execution. I’ve seen companies build elaborate internal platforms that nobody uses because the team built what they thought was cool instead of what developers actually needed. Classic product mistake, applied to infrastructure.

The teams that get it right have three habits:

They start embarrassingly small. One good deploy pipeline beats a half-finished portal with a service catalog. Ship the boring stuff first.

They talk to their users. Your users are the other engineers. If you’re not sitting with them, watching them struggle with your tooling, you’re guessing. And you’re probably guessing wrong.

They don’t mandate adoption. If your platform is better, people will use it. If you have to force adoption, that’s feedback. Listen to it.

The cynical take

Part of me expects the “platform” rebrand to become a way to hire for the same roles at higher salaries. New title, new comp band. I’d get it. Supply and demand.

But the other part of me worries that every time we rename a discipline, we lose institutional knowledge. Junior engineers will think platform teams were invented last year. They miss the decades of operations wisdom that came before. They reinvent mistakes that were solved in 2012.

Do the boring work

If you’re building an internal platform right now, my advice is simple. Ignore the conference talks. Ignore the vendor pitches about developer portals. Go sit with a developer on your team. Watch them deploy something. Count the manual steps. Automate the worst one. Repeat.

That’s platform engineering. It’s also DevOps. It’s also just… good engineering.

Call it whatever you want. Just do the work.

I’ve been working with distributed teams for most of this year. Some are five people across three time zones. Some are thirty across eight. The pattern I keep seeing: they either have zero observability or they went full enterprise cargo cult and now nobody can find anything.

There’s a middle ground.

Why distributed teams feel the gaps first

In an office, someone notices something is slow. They say it out loud. Someone else goes “oh yeah, I deployed ten minutes ago.” Problem found in under a minute.

Remote? That same issue sits in a Slack thread for 45 minutes while people in different time zones wake up, read context, and try to figure out what changed. I’ve watched this happen. Repeatedly.

More tools won’t fix it. What fixes it is making your systems capable of answering basic questions without requiring a human to be online at the right moment. That’s the gap between monitoring and observability .

Three questions. That’s it:

1. Is this thing broken right now?
2. What changed recently?
3. Where do I look next?

If your setup can answer those, you’re ahead of most teams I’ve worked with.

Enterprise observability isn’t your observability

Google has an entire SRE organization. They built custom everything. When you read their SRE book and try to implement the same stack with your team of eight, you end up with:

• A Prometheus instance nobody configured alerts for
• Grafana dashboards copied from a blog post that don’t match your services
• Jaeger running but with 0.1% sampling so traces are useless when you actually need them
• An ELK stack eating 40% of your infrastructure budget

I’ve seen this exact setup at three different companies this year. Not exaggerating.

What you actually need

Here’s my stack recommendation for a small distributed team. Opinionated, yes. But it works.

Logs: Structured JSON to a managed service. Loki if you’re cheap. Papertrail if you want simple. The key is structured, not the tool. Every log line should look roughly like this:

{
  "level": "error",
  "msg": "payment failed",
  "service": "checkout",
  "request_id": "7f3c2c4d",
  "user_id": "u_123",
  "error": "card_declined",
  "duration_ms": 340
}

Same field names. Every service. No exceptions. The request_id alone will save you hours of debugging per incident. I can’t stress this enough. Propagate it through HTTP headers, queue messages, background jobs. Everywhere.

Metrics: Prometheus + Grafana. Still the best bang for buck. But don’t build 30 dashboards. Build one per service. Three panels:

1. Request rate and error rate (tells you if something is broken)
2. Latency percentiles (tells you if it’s degrading)
3. Recent deploys and config changes overlaid on the graphs (tells you what caused it)

That’s your dashboard. If a new engineer can’t look at it and understand the health of the service in 30 seconds, strip it down further.

Traces: Jaeger or Zipkin, but only if you have more than two services. If you’re a monolith with a database and a cache, traces are overhead you don’t need yet. Just use request IDs in your logs. Seriously.

When you do need traces, bump sampling to at least 10% on critical paths. 0.1% default sampling means you’ll never have a trace for the request that actually broke.

Alerts: Less is more. Every alert that pages someone at 3am and isn’t actionable is erosion of trust. Once your team stops trusting alerts, you’ve lost. They’ll start muting channels. I’ve seen it happen.

My rule: every alert needs three things.

• A condition that’s actually abnormal (not “CPU above 60%”)
• A link to the relevant dashboard
• A link to a runbook that says what to do first

If you can’t write those three things for an alert, the alert shouldn’t exist.

The request_id sermon

I keep coming back to this because it’s the single highest-leverage thing you can do.

At the fintech startup we had services talking to services talking to queues talking to workers. When something went wrong without correlation IDs, the debugging process was: check this log, then that log, then maybe this other log, hope the timestamps line up, piece it together manually. Took forever.

After we standardized on a single request_id header propagated everywhere? Same investigation. One search. Done.

The implementation is trivial. Middleware that reads X-Request-ID from incoming requests. Generates a UUID if missing. Passes it along. Logs it on every line. Takes an afternoon to implement across your whole stack.

An afternoon of work for months of saved debugging time. That’s the kind of trade I like.

Runbooks: the unsexy high-leverage tool

Nobody wants to write runbooks. I get it. But here’s the scenario: it’s 2am in your time zone. The alert fires. The person on call is in a different country. They’ve been on the team for three weeks.

Without a runbook, they’re messaging people, waiting for responses, guessing. With a runbook, they open it, follow the steps, and either fix it or know exactly who to escalate to.

Keep them short. Keep them next to the code. Update them after every incident. A runbook that says “check the database connection pool, then check Redis, then check the upstream API timeout” is worth more than a 50-page incident response process document nobody has read.

Mistakes I keep seeing

Collecting everything. Storage is cheap. Cardinality explosions aren’t. I watched a team’s Prometheus instance fall over because they added a user_id label to a counter. Millions of time series. Dead monitoring system. During an outage. Ironic.

Dashboard graveyards. Thirty dashboards, twenty-eight of which nobody has looked at in months. Two of which are actually useful but you can’t remember which ones. Delete aggressively .

Happy path instrumentation only. Your error paths need more instrumentation than your happy paths. The happy path works. You know this because nobody is complaining. The error paths are where surprises live.

Separate conventions per team. One team calls it user_id, another calls it userId, a third calls it uid. Now your cross-service queries are a mess. Pick a convention. Enforce it in code review. This is boring work that pays off enormously.

What to measure about your observability itself

One meta-metric I track: time from “something seems wrong” to “I know what changed and where to look.” If that number is going down over time, your observability is working. If it’s flat or going up, you’re adding complexity without adding clarity.

The other one: how many alerts fired this week that didn’t need a human response? If it’s more than 20%, you have a noise problem.

Start here

If you’re starting from scratch with a small distributed team, do this in order:

1. Structured JSON logs with a shared request_id. One week of work, max.
2. One Grafana dashboard per service with the three panels I mentioned. Another week.
3. Three to five alerts that are actually actionable. A few days.
4. Short runbooks for those alerts. A day.

That’s a month of work spread across your team. After that, you have a system that answers the three questions. Everything else (traces, SLOs , error budgets, custom metrics) can wait until you feel the pain.

Don’t let perfect be the enemy of “I can actually debug production at 2am without waking up three people.”

Then I needed matrix builds across three OSes, caching that doesn’t invalidate every other commit, gated production deploys, and secrets that don’t leak into fork PRs. That’s where it got interesting. These are the patterns that survived, each learned by breaking something at least once.

Matrix builds: cover ground without burning minutes

The matrix strategy is the single best feature in Actions. One job definition, multiple OS and runtime combinations. But the defaults will burn through your free minutes fast if you’re not careful.

Here’s what our test workflow looks like:

name: ci
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ${{ matrix.os }}
    timeout-minutes: 15
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: [12, 14]
        exclude:
          - os: windows-latest
            node: 12
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci
      - run: npm test

Few things to note.

fail-fast: false. I used to leave this at the default (true). Then we’d get a flaky Windows test that killed the whole matrix, and we’d lose the Linux and macOS results entirely. Set it to false. Let everything run. You want the full picture.

Exclude dead combos. We dropped Node 10 entirely and excluded Node 12 on Windows because we don’t ship Windows builds on that version. Every combination you exclude saves real minutes. We were running 9 combinations, trimmed it to 5. The monthly bill noticed.

timeout-minutes on everything. I learned this one the hard way. A hung test on macOS once ran for 6 hours before I noticed. GitHub’s default timeout is also 6 hours. That’s not a default, that’s a trap. Set 15 minutes. If your tests take longer than that, you have a different problem.

Caching: the difference between 8 minutes and 90 seconds

Caching npm dependencies turned our builds from “go get coffee” to “it’s already done.” But the cache key design matters more than you’d think.

- name: Cache node modules
  uses: actions/cache@v1
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

- name: Install dependencies
  run: npm ci

The restore-keys fallback is critical. Without it, any change to package-lock.json means a complete cache miss and a full install. With the fallback, you get the last close-enough cache and npm ci only fetches the diff. Went from 4-minute installs to about 20 seconds on cache hit.

One gotcha: actions/cache@v1 has a 5GB limit per repo. We hit it after a few weeks because we were caching node_modules directly. Big mistake: cache the npm cache directory instead and let npm ci do the linking. Switched to caching ~/.npm and the size dropped by 80%.

Job orchestration: build once, deploy from artifacts

This is the pattern I’m most opinionated about. Never rebuild in your deploy step. Build once, upload the artifact, download it in deploy. Guarantees you deploy exactly what you tested.

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v2
      - name: Cache npm
        uses: actions/cache@v1
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-
      - run: npm ci
      - run: npm run build
      - run: npm test
      - uses: actions/upload-artifact@v2
        with:
          name: dist
          path: dist/

  deploy-staging:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: dist
      - name: Deploy to staging
        run: ./scripts/deploy.sh staging
        env:
          DEPLOY_TOKEN: ${{ secrets.STAGING_DEPLOY_TOKEN }}

  deploy-production:
    needs: deploy-staging
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: dist
      - name: Deploy to production
        run: ./scripts/deploy.sh production
        env:
          DEPLOY_TOKEN: ${{ secrets.PROD_DEPLOY_TOKEN }}

Watch artifact storage. Artifacts are kept for 90 days and there’s no per-artifact retention setting yet. You don’t need three months of build artifacts. You need maybe a week for debugging. We hit 12GB of stored artifacts before I noticed.

The needs chain (build -> deploy-staging -> deploy-production) gives you a natural pipeline. Staging has to pass before production even starts. Actions has no built-in approval gate, which is why our real production deploys go through a manual trigger (below). That keeps us at continuous delivery for prod: the artifact is always ready, the last step is a human decision.

Trigger filtering: stop running CI on README changes

This was annoying before I figured it out. Every docs edit, every README typo fix, every changelog update triggered a full CI run. Waste.

on:
  push:
    branches: [main]
    paths:
      - 'src/**'
      - 'package.json'
      - 'package-lock.json'
      - '.github/workflows/**'
  pull_request:
    paths:
      - 'src/**'

Only run when code actually changes. I also include .github/workflows/** in the push paths because I want CI to test itself when I modify the workflow files. Learned that one after pushing a broken workflow that I didn’t catch for two days because the paths filter excluded the workflows directory.

For production deploys, we use workflow_dispatch, which GitHub shipped this month, so nobody accidentally deploys on merge:

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Deploy target'
        required: true
        default: 'staging'

Manual button in the Actions tab. Feels old-school. Works perfectly.

Secrets and fork PRs: the security stuff that bit me

The rule nobody states clearly enough: fork pull requests run your workflows, but secrets aren’t injected. This is correct behavior and you shouldn’t try to work around it.

I see people “fixing” the fork PR problem anyway, by finding some other way to hand secrets to a job that runs the PR’s code. Don’t. Any setup where untrusted PR code runs with access to your secrets is a supply chain attack waiting to happen.

Our approach:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: npm ci
      - run: npm test
      # No secrets needed for tests

  deploy:
    needs: test
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Deploy
        run: ./scripts/deploy.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Tests run on PRs without secrets. Deploy only runs on push to main, where secrets are available and the code is trusted. Simple separation.

Also: always mask secrets in logs.

- name: Mask token
  run: echo "::add-mask::${{ secrets.API_TOKEN }}"

GitHub does this automatically for secrets referenced directly, but if you derive a value from a secret (substring, base64 encode, whatever), the derived value isn’t masked. Found that out when a deploy URL containing an embedded token showed up in our build logs. Fun morning.

Self-hosted runners: probably not yet

We considered self-hosted runners for builds that need access to our internal Docker registry. Decided against it. The maintenance overhead of patching, isolating, and monitoring runner machines wasn’t worth it for our team size. We just push to a public registry and pull in deploy.

If you do need them:

jobs:
  build:
    runs-on: self-hosted
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v2
      - run: make build

Just know that self-hosted runners don’t get automatic cleanup between jobs like hosted runners do. Previous job’s files, environment variables, running processes: all still there. You need to handle that yourself or you’ll get extremely confusing cross-contamination bugs.

Four months in

None of this is groundbreaking. But I spent a couple months learning each of these the hard way, so maybe this saves you the same debugging sessions. GitHub Actions is good in 2020. The YAML is verbose. The ecosystem is young. But the fundamentals are solid and the GitHub integration is unbeatable.

Between running infrastructure at my infrastructure startup and working with other teams, there’s one pattern I see in almost every cluster I look at: resource settings are either copy-pasted from a tutorial, set to absurdly high values “just to be safe,” or missing entirely.

The result is always the same. Either you’re paying for 3x the compute you actually need , or your pods are getting evicted during the one traffic spike that matters.

Requests vs limits: two different jobs

People treat these as the same thing. They’re not.

Requests are what the scheduler sees. When you set requests.cpu: 200m, you’re telling Kubernetes “this pod needs at least 200 millicores to run, find me a node that has that available.” The scheduler uses requests to bin-pack pods onto nodes.

Limits are runtime caps. The kernel enforces these. Hit your CPU limit and you get throttled. Hit your memory limit and you get OOM-killed. Very different consequences.

resources:
  requests:
    cpu: "200m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "512Mi"

One thing that trips people up: if you set a limit but skip the request, Kubernetes sets the request equal to the limit. If you set neither, congratulations, you’ve got a BestEffort pod. First to die when the node gets tight.

CPU throttling vs OOM kills

This is the part that bites people.

CPU is compressible. Your container hits its CPU limit? It gets throttled. Slower responses, higher latency, but it stays alive. Annoying but survivable.

Memory isn’t compressible. Your container exceeds its memory limit? The kernel kills it. No negotiation. The pod restarts (if your restart policy allows it), your users see errors, and you get an alert.

I’ve seen teams set memory limits too close to actual usage, then wonder why they’re getting random OOM kills during garbage collection spikes. Leave real headroom on memory. I’d rather waste 50Mi per pod than deal with cascading restarts.

QoS classes matter more than you think

Kubernetes assigns every pod a Quality of Service class based on its resource settings. The class shapes eviction order when a node runs out of resources.

Guaranteed: requests equal limits for both CPU and memory, in every container. These pods get evicted last. Use this for anything you actually care about in production.

Burstable: requests are set but limits are higher. Most pods in real clusters end up here. Fine for most workloads, but know that you’re in the middle of the eviction queue.

BestEffort: no requests, no limits, no guarantees. First to get evicted. I only use this for batch jobs I don’t care about.

The mistake I see constantly: teams running critical APIs as BestEffort because nobody set resource fields in the deployment spec. Then a memory spike on the node evicts those pods before the logging agent that has Guaranteed QoS. Backwards.

How I actually size resources

I don’t guess. I measure, then set values based on real data.

Start with kubectl top to get a quick snapshot:

kubectl top pods -n production
kubectl top pods -n production --containers

But a snapshot isn’t enough. You need percentile data over days or weeks. Pull p50, p75, p95, and p99 from whatever metrics system you’re running: Prometheus, Datadog, whatever.

Then I set values like this:

• CPU request: p75 usage plus a small buffer. This is your “normal operating range.”
• Memory request: p95 usage plus a buffer. Memory spikes are less forgiving.
• CPU limit: 2-3x the request. Let it burst for short periods.
• Memory limit: request plus enough headroom for GC spikes and transient allocations. But not so much that a leak runs unchecked.

resources:
  requests:
    cpu: "200m"      # p75 + headroom
    memory: "300Mi"  # p95 + headroom
  limits:
    cpu: "500m"      # room for bursts
    memory: "450Mi"  # room for GC, not for leaks

Review these after any significant traffic change. Resource settings should evolve with your workload, not stay frozen from the day someone first deployed the service.

Autoscaling helps but isn’t magic

HPA (Horizontal Pod Autoscaler) scales replicas based on CPU utilization relative to your requests. If your requests are wrong, your scaling behavior is wrong. Garbage in, garbage out.

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

VPA (Vertical Pod Autoscaler) is interesting but still beta. I run it in recommendation mode only: let it tell me what it thinks the requests should be, then I decide whether to apply that.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"

I don’t trust VPA to mutate running pods in production yet. Maybe next year.

LimitRanges and ResourceQuotas: the guardrails you need

Every namespace in a shared cluster should have a LimitRange. Full stop. Without one, any developer can deploy a pod with no resource settings, and now you’ve got a BestEffort pod sitting in production waiting to get evicted.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      default:
        cpu: "500m"
        memory: "256Mi"

This gives every container sane defaults even if the developer forgot to set them. It’s a safety net, not a replacement for proper resource specs.

ResourceQuotas cap total consumption per namespace. Essential for multi-tenant clusters where you don’t want one team eating the whole cluster.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"

What to actually monitor

Set it and forget it doesn’t work here. Watch these:

• CPU throttling rate: if it’s consistently high, your CPU limit is too low
• OOM kill count: any non-zero number means your memory limit needs attention
• Request-to-usage ratio: if you’re requesting 4x what you use, you’re wasting cluster capacity
• Allocatable vs requested: tells you how much room the cluster actually has
• Eviction events: if pods are getting evicted, something is wrong with your resource math

I check these weekly for the clusters I manage. It takes ten minutes and prevents most resource-related incidents before they happen. It belongs on the same boring checklist as probes and upgrades.

Nobody writes conference talks about setting CPU requests correctly. Get it wrong anyway and a BestEffort pod gets evicted at 3am and takes your API gateway with it.

I broke production with a Terraform typo last year. Changed a security group rule, fat-fingered a CIDR block, and opened port 22 to the internet for about forty minutes. Nobody caught it in review. Our CI pipeline at the time was terraform plan and vibes.

That was the wake-up call. I spent the next six months building out infrastructure testing at my infrastructure startup, trying every approach I could find. Unit tests for IaC. Policy engines. Full integration runs with real resources. Some of it was worth the effort. Some wasn’t.

Three Approaches, Three Different Jobs

The mistake I see most teams make is treating “infrastructure testing” as one thing. There are three fundamentally different approaches, and they catch fundamentally different problems.

That table is the whole thesis. Each layer fills in what the previous one misses.

Static and Unit Tests: The Fast Floor

These are your terraform fmt, terraform validate, tflint, kubeval checks. They run in seconds. They cost nothing. They should be on every single commit.

terraform fmt -check -recursive
terraform validate
tflint terraform/
kubeval --strict k8s/*.yaml

What surprised me: these catch more than you’d think. At my infrastructure startup, roughly 40% of our infra PRs had at least one issue caught at this layer. Mostly formatting, occasionally an actual invalid reference or deprecated resource argument.

What they won’t catch: anything that’s syntactically valid but semantically wrong. That security group I opened to the world? Perfectly valid HCL. terraform validate had no opinion.

Policy Tests: Where the Real Value Is

This is the layer I underestimated. OPA with Conftest lets you write rules against Terraform plan JSON, Kubernetes manifests, basically any structured config.

The SSH rule that would have saved me forty minutes of panic:

package terraform

deny[msg] {
  r := input.resource_changes[_]
  r.type == "aws_security_group_rule"
  r.change.after.cidr_blocks[_] == "0.0.0.0/0"
  r.change.after.from_port == 22
  msg := "SSH must not be open to the world"
}

Small. Specific. Tied to an actual incident. That’s the pattern.

I tried building comprehensive policy libraries. Dozens of rules covering every possible misconfiguration. Bad idea. Teams bypass large generic policy sets. They add exceptions, disable checks, or just stop running them. The policies that stick are the ones written after something went wrong. Each one represents a scar.

We also run Checkov as a complement. It only came out in December, but its list of built-in checks for common misconfigs is growing fast. Good for catching things you haven’t been burned by yet. But it’s a floor, not a ceiling.

My rule of thumb: if a scanner catches it, great. If it doesn’t, and it bit you, write a policy for it. Your policy library should grow from incidents, not from a wish list.

Integration Tests: Expensive and Worth It (Sometimes)

Integration tests create real AWS resources, run assertions against them, then tear everything down. Terratest is what we use. Go-based, works well with our existing stack.

These are the only tests that catch networking and IAM problems. You can validate a Terraform plan all day and it won’t tell you that your NAT gateway routing is broken or that your RDS instance can’t actually reach the VPC endpoint it needs.

But they’re slow. Our VPC module integration test takes about twelve minutes. Our EKS test takes twenty-five. And they cost money. Not a lot per run, but it adds up if you’re running them on every PR.

We don’t integration test every module. That’s the trap. Test the ones where a static check can’t tell you if it works. Our S3 bucket module? Unit tests and policy checks are enough. Our networking module? Needs the real thing.

The cleanup problem is real. Terratest has defer terraform.Destroy() but it’s not bulletproof. We’ve had orphaned resources from failed tests sit around for weeks. I wrote a nightly cleanup job that tags test resources and nukes anything older than 24 hours. Unsexy but necessary.

How We Wire It Together

Our CI pipeline at my infrastructure startup:

1. Pre-commit hooks: terraform fmt, basic linting. Runs locally in seconds.
2. PR checks: terraform validate, tflint, Conftest policies, Checkov. Blocks merge on failure. Takes about 90 seconds.
3. Nightly on main: Integration tests for critical modules. Results posted to Slack. Takes about 45 minutes total.
4. Post-deploy: Smoke tests hitting real endpoints. Health checks, latency thresholds, basic connectivity. Alerts on failure.

Fast checks on every commit, expensive checks on a schedule. That split is what makes the pipeline scale with the codebase . I’ve seen teams try to run Terratest on every PR and give up within a month because the feedback loop is too slow.

Practical Stuff That Matters

A few things I wish someone had told me six months ago:

• Tag everything with a test identifier. When tests fail mid-run and leave resources behind, you need to find them.
• Set hard cost limits. We capped our test AWS account at $200/month. Hit it once. Fixed the test that was spinning up m5.4xlarge instances.
• Never put real credentials in test fixtures. Sounds obvious. I’ve reviewed code where people did it anyway.
• Store plan outputs. When something breaks in production, being able to diff against the last tested plan is invaluable.

Where We Stand After Six Months

Static checks are table stakes. There’s no excuse not to have them. Policy tests are the highest ROI layer and the one most teams skip. Integration tests are worth it for complex modules but not for everything.

If I were starting from scratch tomorrow, I’d add static checks on day one, write my first policy rule on day two after reading our incident history, and add integration tests only for the modules that scare me.

Perfect coverage was never the goal. The goal is knowing that the categories of mistakes that have hurt you before won’t hurt you again.

The adoption argument is finished

Nobody is debating whether to use Kubernetes anymore. That conversation ended sometime mid-2019. Every team I’ve worked with lately has clusters in production. The question now is whether they’re running it well. Spoiler: most aren’t.

The real work in 2020 is day-two operations: upgrades, multi-tenant governance, cost control, and not breaking things on Friday afternoon. Boring stuff. Important stuff.

I’ve been living this reality since I started building cloud infrastructure tooling after a founder program. The gap between “we have Kubernetes” and “we operate Kubernetes well” is enormous. That gap is where most of the pain lives.

Managed control planes win. Stop fighting it.

If you’re still self-managing your control plane without a very specific reason, I don’t understand what you’re doing. Cloud providers handle patching, availability, and scaling. The operational cost of doing it yourself is absurd for most teams.

Yes, there are exceptions. Air-gapped environments. Weird regulatory constraints. Hybrid topologies that managed services can’t reach. But these are exceptions, not defaults. The default in 2020 should be managed.

GitOps becomes the normal way to deploy

This is probably my highest-confidence prediction. GitOps moves from “cool thing the early adopters do” to just how you deploy. Argo CD and Flux are solid. The model is proven.

Pull-based delivery, audit trails in git, rollbacks that don’t require someone SSHing into a bastion host at 2 a.m. It works. Multi-cluster GitOps is still rough, but single-cluster is basically solved.

If you’re starting fresh in 2020 and not using GitOps, you’re choosing to have a worse time later.

Security shifts from checkboxes to enforcement

RBAC and network policies should already be in place. If they’re not, that’s a different conversation (and you should be worried). The 2020 shift is toward policy as code: admission control that actually prevents bad configurations from reaching the cluster.

OPA Gatekeeper is the one to watch here:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-team-label
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["team", "cost-center"]

Simple example, but the idea matters. Don’t just document policies. Enforce them automatically so humans can’t skip them when they’re in a rush.

Service mesh is overrated (for now)

Hot take that shouldn’t be a hot take: most teams don’t need a service mesh in 2020. The mTLS and traffic shaping benefits are real, but the operational overhead is significant. I’ve watched teams spend weeks debugging Istio sidecars instead of shipping features.

For critical services with strict security requirements, use a mesh. For your internal CRUD APIs and batch jobs, you’re adding complexity for conference talk credibility.

eBPF-based networking like Cilium is interesting and may eventually take over some of what sidecars do, but it’s early. Check back in 2021.

Running databases on k8s: still mostly don’t

The operator ecosystem is improving. Zalando’s Postgres operator, Strimzi for Kafka: these are credible projects. CSI is maturing. But “technically possible” and “good idea for your team” are very different things.

From my time at the fintech startup handling strict data requirements, I learned that stateful workloads need operational confidence you only get through battle-testing. Most teams haven’t battle-tested their k8s storage layer enough to bet production data on it. Give it another year or two before it goes mainstream.

Internal platforms are the underrated trend

This is the one nobody’s hyping enough. Kubernetes is powerful and terrible as a developer interface. The teams that win in 2020 will build internal platforms that hide the complexity. Opinionated templates. Self-service environments. Clear boundaries between app teams and platform teams.

The point is to let developers ship faster without accidentally creating cost or security problems.

What won’t happen

Service mesh everywhere won’t happen. Neither will databases migrating wholesale to k8s, or the mythical “NoOps” future some people keep promising.

Kubernetes in 2020 is about operating it well. Teams that treat operations, security, and developer experience as actual product work, not afterthoughts, will be fine. Everyone else will keep filing the same incident reports they filed last year.

No deploy tool gives you zero downtime on its own. It’s a discipline across code, database migrations, and infrastructure, and the hardest part is the schema changes, not the YAML.

What Zero Downtime Means

Two things. That’s it.

1. Capacity never drops below what traffic needs.
2. Running code stays compatible with in-flight requests and existing data.

If both hold, your deploy is invisible to users. If either breaks, you have an outage regardless of how fancy your canary setup is.

Database Migrations Are the Hard Part

Rolling updates, blue-green, canary: pick your favorite. They all handle the application layer fine. The hard part is always the database.

A schema change that locks a table or drops a column still in use will take you down no matter what deployment pattern you’re running. I learned this the painful way with Postgres at the fintech startup, where our users table had enough rows that a naive ALTER TABLE would hold a lock for minutes.

The fix is boring, and I covered it in more depth in database migrations without downtime . Expand, migrate, contract. In that order, across separate deploys.

-- Deploy 1: Expand (add the new column)
ALTER TABLE users ADD COLUMN phone VARCHAR(20);

-- Deploy 2: Migrate (backfill in batches by id range, not one giant UPDATE)
UPDATE users SET phone = legacy_phone
WHERE id > 0 AND id <= 1000 AND phone IS NULL;
-- ...repeat for the next id range until done

-- Deploy 3: Contract (drop old column after all code uses the new one)
ALTER TABLE users DROP COLUMN legacy_phone;

Three deploys for one column rename. Annoying? Yes. But nobody notices. That’s the point.

On MySQL, tools like gh-ost or pt-online-schema-change do background table copies to avoid long locks. On Postgres, the work is mostly careful DDL: nullable columns without defaults, CREATE INDEX CONCURRENTLY, a short lock_timeout, and backfills in batches (Postgres UPDATE has no LIMIT, so batch by primary key range). Worth learning if you’re running anything with real traffic.

Pick a Rollout Pattern and Keep It Simple

Rolling Updates

The default and usually the right choice. Set maxUnavailable: 0 so you never lose capacity during the rollout.

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 25%

This works when old and new versions can run side by side. Which means your APIs need to be backward compatible. If that sounds like extra work, it is. But it’s the kind of extra work that prevents 3 a.m. pages.

Canary

Send a small slice of traffic to the new version. Watch your error rate. If it’s clean, ramp up. If it spikes, roll back automatically.

The key word there is “automatically.” A canary release without automated rollback is just a smaller blast radius with manual intervention. You might as well flip a coin.

Blue-Green

Run two environments, switch traffic when the new one is verified. Fast rollback, but you’re paying for double infrastructure. Fine for critical services. Overkill for most things.

The Stuff Between the Deploys

The patterns above are table stakes. What actually makes deploys boring (in a good way) is the plumbing around them.

Readiness vs. liveness probes. These aren’t the same thing. Liveness says “is this process alive.” Readiness says “can this process handle traffic right now.” If your readiness check passes before caches are warm and dependencies are verified, you’re sending users to a half-alive instance.

Graceful shutdown. When Kubernetes sends SIGTERM (see my Kubernetes production checklist ), your app needs to stop accepting new connections, finish in-flight requests, then exit. Sounds obvious. Roughly half the Go services I’ve reviewed get this wrong, usually by not waiting long enough for the load balancer to drain.

Connection draining. Set a realistic termination grace period. If your longest request takes 30 seconds, a 5-second grace period is going to drop connections.

DNS TTLs. Lower them before a cutover, raise them after. I’ve seen teams spend hours debugging a “failed deploy” that was actually stale DNS.

My Pre-Deploy Checklist

The questions I ask before every deploy:

• Can the new code read data written by the old code? Can the old code read data written by the new code?
• Does the database migration need a lock? How long?
• Do the readiness checks actually verify dependencies, or do they just return 200?
• If this deploy goes wrong, can I roll back in under a minute?

If any answer is “I’m not sure,” that’s where the work is. Not in the deploy tool.

Why this stays hard

Zero downtime deploys aren’t hard technically. They’re hard culturally. They require everyone on the team to think about backward compatibility, schema evolution, and graceful degradation before they write the code. Not after.

The teams I’ve seen do this well, at the fintech startup, at startups in my founder-program cohort, and on other teams I’ve worked with, all share one trait. Deploys are boring. Nobody watches them. Nobody holds their breath. The monitoring catches problems and the rollback is automatic.

By the time I started building my own infrastructure startup in a founder program this year, I’d already learned this lesson the hard way. The problem is never Terraform itself. The problem is that infrastructure grows quietly. You add a few resources here, a new environment there, and one morning you run terraform plan and it takes eleven minutes and touches things you forgot existed. That’s the moment you realize your Terraform setup doesn’t scale. And by then, untangling it is painful.

Stop dumping everything into one state file. Split by domain, pin everything, automate plan-and-apply, and treat -target like a fire extinguisher. The rest of this post is what I wish I’d known before the first time a plan returned 200+ changes because someone refactored a module without understanding what depended on it.

The breaking point

Small Terraform setups are a joy. You describe what you want, you apply it, it exists. Beautiful.

At scale, the problems are predictable:

• Plans take forever. API rate limits turn a 30-second plan into a 10-minute gamble.
• State becomes a bottleneck. Two engineers run apply at the same time. One wins, one gets a lock error and has to wait.
• Blast radius grows silently. A change to a security group touches the same state as your database config. One bad merge and you’re having a very bad afternoon.
• Drift accumulates. Resources nobody has looked at in months quietly diverge from what Terraform thinks they are.

None of these kill you immediately. They just make everything slower and scarier until one day someone is afraid to run apply at all.

Split state by domain, not by convenience

The single most impactful thing you can do is break your state apart. Not by team, not by project. By domain.

terraform/
|-- network/
|-- security/
|-- data/
|-- kubernetes/
`-- services/
    |-- api/
    `-- web/

Each directory gets its own state backend. Each one can be planned and applied independently. A change to your VPC config doesn’t require Terraform to also reconcile your Kubernetes cluster and all your application services.

For environments, use separate directories and backends. I know workspaces exist. I’ve used them. They work fine when it’s just you. They get confusing fast when three engineers are working across staging and production and nobody remembers which workspace they’re in. Separate directories are boring. Boring is good.

Remote state with locking. Always.

If you’re running Terraform at scale and your state is local, stop reading this and go fix that first.

terraform {
  backend "s3" {
    bucket         = "terraform-state"
    key            = "network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

DynamoDB for locking. Encryption on. Versioning on the S3 bucket so you can recover when (not if) someone corrupts state. This is infrastructure for your infrastructure. Treat it seriously.

Cross-state references

When stacks need to talk to each other, use remote state data sources. Keep the outputs stable: changing an output name in your network stack shouldn’t break your services stack.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "terraform-state"
    key    = "network/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_subnet" "app" {
  vpc_id     = data.terraform_remote_state.network.outputs.vpc_id
  cidr_block = "10.0.20.0/24"
}

Think of outputs as your stack’s API. You wouldn’t change an API contract without versioning. Same principle.

Modules: small, versioned, boring

I’ve seen Terraform modules that try to do everything. A “vpc” module that also creates subnets, NAT gateways, route tables, security groups, and somehow a Lambda function. Don’t do this.

Keep modules focused on one thing.

module "vpc" {
  source = "./modules/vpc"
  cidr   = "10.0.0.0/16"
}

module "public_subnet" {
  source = "./modules/subnet"
  vpc_id = module.vpc.vpc_id
  cidr   = "10.0.1.0/24"
}

Pin versions. Pin everything. Terraform version, provider versions, module versions.

terraform {
  required_version = "~> 0.12"
}

provider "aws" {
  version = "~> 2.28"
  region  = "us-east-1"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.15.0"
}

I learned this one when an unpinned provider upgrade changed the default behavior of an AWS resource and our plan suddenly wanted to destroy and recreate a production database. Nobody was hurt. My heart rate was.

The review workflow

At the fintech startup I watched us go from “just apply it” to a proper review flow, and the number of infrastructure incidents dropped to near zero. The process isn’t exciting:

1. Branch for the change.
2. terraform fmt and terraform validate locally.
3. CI runs plan and posts the output to the PR.
4. The team that owns that stack reviews the plan.
5. apply runs only after approval, then the PR merges.

We used Atlantis for this and it worked well:

# atlantis.yaml
version: 3
projects:
- name: network
  dir: terraform/network
  workflow: default

workflows:
  default:
    plan:
      steps:
      - init
      - plan
    apply:
      steps:
      - apply

The plan output is the thing being reviewed, not just the code diff. A four-line HCL change can produce a 200-line plan. The plan is what matters.

The stuff that will save you at 2 AM

Plan files. Always save the plan and apply from the saved file. What was reviewed should be exactly what gets applied.

terraform plan -out=plan.tfplan
terraform apply plan.tfplan

-target is for emergencies only. It’s tempting to use -target to apply just one resource when you’re in a hurry. Don’t make it a habit. It skips dependency resolution and creates drift. Use it when something is on fire, then follow up with a full plan to clean up.

Drift detection. Schedule terraform plan -detailed-exitcode to run regularly. Surface drift before it surprises you during an incident. Exit code 2 means there’s drift. Pipe that into an alert.

terraform plan -detailed-exitcode
# Exit 0: no changes
# Exit 1: error
# Exit 2: changes detected (drift)

Parallelism tuning. Default is 10. You can bump it to 20 for faster applies, but watch for API rate limit errors. More parallelism isn’t always faster.

Guard the gates

Treat Terraform like application code. Lint it. Format it. Run policy checks before it reaches production.

# .pre-commit-config.yaml
repos:
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.19.0
  hooks:
  - id: terraform_fmt
  - id: terraform_validate
  - id: terraform_tflint

Pre-commit hooks catch the easy mistakes. For the hard ones, like someone trying to open port 22 to the world or creating an unencrypted S3 bucket, use Sentinel or OPA. Policy as code is one of those things that feels like overhead until the first time it stops a bad change from reaching production.

Own your stacks

Define CODEOWNERS by directory. The network team reviews network changes. The platform team reviews Kubernetes changes. Nobody should be able to merge infrastructure changes without review from someone who understands the blast radius.

Keep documentation minimal and close to the code. A README in each stack directory with the inputs, outputs, and any non-obvious decisions. Not a wiki that nobody updates.

Terraform scales fine. The part that doesn’t scale is humans making ad hoc decisions about where things go and how changes flow. The IaC patterns I wrote about last year still hold; the state layout is where most teams need to start.

At the fintech startup, the number of services and repos grew fast. We hit a point where onboarding a new engineer meant a full day of tribal knowledge transfer: which repo to clone first, which env vars to set, which Docker Compose file was current. The ad-hoc scripts that got us through the first year became the bottleneck. That’s when a platform starts to make sense. Not before. If your developers are already shipping fast with scripts and Makefiles, a shiny platform might just slow them down.

The Two Approaches

Ad-hoc tooling is what every team starts with. Makefiles, shell scripts, a README that someone updates when they remember. It’s fast to create, easy to understand, and perfectly tuned to the team that wrote it.

An internal platform is what you build when the ad-hoc approach stops scaling. A curated set of tools, templates, and workflows that standardize the path from code to production.

Right now, a lot of the conference conversation favors platforms. I think that framing skips the part where most teams aren’t ready for one.

Where Ad-Hoc Tooling Wins

Small teams. Early-stage products. Anything where the domain is still shifting.

When I was at a mobility startup building our backend, we had four engineers. A Makefile with targets for build, test, deploy-staging, and deploy-prod covered about 90% of our workflow. Total maintenance cost: maybe an hour a month. Everyone understood every line because everyone had written at least one target.

Ad-hoc tooling wins when:

• The team is under ten engineers and everyone knows the stack.
• Requirements change weekly, so a rigid platform would just get in the way.
• The “platform” would have exactly one consumer team. That’s a library, not a platform.
• You can still onboard a new engineer in under two hours with a README and a pair programming session.

The advantage is speed and ownership. Nobody files a ticket to change a Makefile. Nobody waits for “the platform team” to prioritize a feature.

Where Ad-Hoc Tooling Breaks Down

It breaks down at the seams. Not within one team, but across teams.

The symptoms are consistent. I’ve seen them at every company past about fifteen engineers:

• New hires take days to get a working local environment because every team’s setup script assumes different things.
• “Works on my machine” becomes a weekly standup topic.
• Three teams have three different CI configs that do roughly the same thing with slightly different bugs.
• Nobody knows how to deploy the service they inherited from the team that left.

The failure mode isn’t that any single script is bad. It’s that the collection of scripts has no coherence. Each one optimizes for its author’s context, and that context drifts.

Where a Platform Earns Its Keep

A platform earns its keep when it removes decisions that slow people down without removing decisions that matter.

Good platform DX looks like this:

• platform init scaffolds a new service with CI, monitoring, and deployment already wired up. The developer writes business logic on day one.
• git push triggers a pipeline that builds, tests, and deploys to staging. No YAML archaeology required.
• Logs, metrics, and traces are correlated by default. When something breaks at 2am, the on-call engineer doesn’t need to be an expert in the observability stack.

Bad platform DX looks like this:

• A portal with a service catalog that nobody updates.
• Mandatory migration to “the platform way” with no clear improvement in daily workflow.
• A platform team that ships features on their own roadmap instead of fixing the paper cuts developers actually report.

I’ve been in a founder-program cohort this year, building a cloud infrastructure startup, and one thing the startup environment hammers home is this: your internal customers are just as fickle as external ones. If the platform isn’t obviously better than what developers already have, they will route around it. Every time.

The Comparison That Matters

Here is how I think about the tradeoff in practice:

Onboarding time. Ad-hoc: depends entirely on documentation quality, which is usually poor. Platform: can be excellent if platform init and the paved-road defaults work. Can be terrible if the platform itself needs onboarding.

Speed of change. Ad-hoc: instant for the team that owns the script, impossible for everyone else. Platform: slower to change but the change applies everywhere.

Reliability. Ad-hoc: varies wildly per team. Platform: consistent baseline, but a platform bug affects everyone simultaneously.

Cost. Ad-hoc: hidden in every team’s sprint. Platform: visible headcount on a dedicated team. Leadership often prefers the visible cost, which is ironic because it’s usually higher.

Autonomy. This is the one people underweight. Developers care about autonomy. A platform that takes away the ability to customize a build step or add a deployment hook will face resistance no matter how polished it is. The best platforms I’ve seen treat defaults as suggestions and overrides as first-class features.

My Rule of Thumb

If you have fewer than three teams consuming shared infrastructure, invest in better scripts and a good README. A Makefile with ten well-named targets and a one-page setup guide will outperform a half-built platform every time.

If you have more than three teams and you’re seeing the same problems solved differently in each one, start building a platform. But start with the one workflow that causes the most pain, not with a portal and a service catalog.

One platform rollout I watched work started with deployment. Just deployment. One command to deploy any service to staging, with automatic rollback if health checks failed. That single workflow converted the skeptics faster than any roadmap slide could have. Then we added CI standardization. Then observability. Each step earned trust before the next one started.

Where platform teams fail

Most internal platforms I’ve seen fail because the team builds what it finds interesting instead of what developers need. I’ve been guilty of this. At one point I spent two weeks building a slick CLI for environment management when the actual pain point was that our staging database kept running out of disk space.

Talk to your developers before you build anything. Sit with them. Watch them work. The friction you need to fix is almost never the friction you assumed.

Good DX, whether it comes from a platform or a pile of scripts, is measured by how fast a developer goes from idea to running code in production, without fear.

Nobody reads the 40-page incident response PDF until the breach is already happening. The plan that survives first contact is the one your team has rehearsed, and it has to work at 2am with half your team asleep. I wrote about incident response for startups a few years ago; this is what I’d add now.

The problem with most IR plans

Every company I’ve worked with has some version of an incident response document. At the fintech startup, we had one. At a mobility startup, we had one. They were fine documents. Well-structured. Thorough. And completely useless the first time something real happened.

The issue isn’t the content. The issue is that nobody has practiced it. An IR plan is like a fire drill: the value is in everyone having walked the route, more than in the map of the exits.

During a cyber-defense exercise I participated in, the teams that performed best weren’t the ones with the most sophisticated tooling. They were the ones where every person knew their role cold. The analyst knew to preserve logs before touching anything. The incident commander knew to start a timeline before asking questions. The comms lead had templates ready. Muscle memory.

Severity: stop overthinking it

I’ve seen teams spend 20 minutes debating whether something is a SEV-2 or SEV-3 while the attacker is still active. Here is a simple rule that works for startups and mid-size teams:

SEV-1: You know or strongly suspect data has been accessed by someone who shouldn’t have it. Customer data, credentials, financial records. Page everyone.

SEV-2: Something is clearly wrong (unauthorized access, exploitation of a known vuln, weird lateral movement), but you don’t know the scope yet. Page the security lead and an incident commander.

SEV-3: Suspicious activity that needs investigation but shows no evidence of compromise. Next business hours is fine.

That’s it. Three levels. The distinction between 1 and 2 is “do we know what they got?” If yes, SEV-1. If no, SEV-2. Stop debating and start working.

The first 30 minutes decide everything

I have a strong opinion on this: the first 30 minutes of an incident determine whether you spend three days cleaning up or three weeks. Here is what has to happen in that window:

Minute 0-5: Acknowledge and assign. Someone claims the incident commander role. Not “can someone look at this?” but an explicit “I’m IC for this.” Open a dedicated channel. Name it something obvious like incident-2019-07-15.

Minute 5-15: Triage and preserve. Before you touch anything, snapshot what you can. Disk images, memory dumps, log exports. On one team I worked with, we learned this the hard way when someone rebooted a compromised container to “fix it” and wiped the volatile evidence we needed. Never again.

The key questions at this stage:

• Is the attacker still active right now?
• What systems are potentially in scope?
• What is the worst-case data exposure?

Minute 15-30: Contain. Isolate affected hosts. Revoke compromised credentials. Block known-bad IPs. Kill active sessions from suspicious accounts. The goal is to stop the bleeding; understanding everything comes later.

A useful mental model: containment is a tourniquet. You apply it fast and ugly. Eradication is surgery. That comes later.

Four roles that matter

Skip the org chart with 12 named roles. For a team under 50 engineers, you need four:

Incident Commander (IC): Owns the incident. Makes decisions when there’s disagreement. Controls the communication cadence. The IC coordinates and doesn’t debug. The single hardest skill here is resisting the urge to jump into a terminal. I struggle with this every time.

Technical Lead: Drives the investigation. Validates containment. Recommends eradication steps. This person lives in the logs and the systems.

Scribe: Documents everything with timestamps. Decisions, actions, findings, who said what. This role feels bureaucratic until you need to write the post-mortem or talk to a regulator. Then it’s the most valuable person in the room.

Comms Lead: Handles internal updates and external messaging. Gets legal involved early if disclosure might be required. At a startup, this is often the CEO or co-founder. Fine. Just make sure they aren’t also trying to be the IC.

Investigation: follow the trail, not your assumptions

The biggest mistake in incident investigation is anchoring on the first hypothesis. “It was probably a leaked API key” becomes the only thing anyone investigates, while the actual entry point, a vulnerable dependency three services deep, goes unexamined.

Ask these questions systematically:

• How did initial access occur?
• What was the dwell time? (How long were they in before we noticed?)
• Was there lateral movement?
• What data was accessed, modified, or exfiltrated?

Where to look:

• Application logs (request patterns, auth failures, unusual API calls)
• Cloud audit logs (IAM changes, new resources, API activity)
• Network flow data (unexpected outbound connections, data transfer volumes)
• Auth logs (session creation, privilege escalation, MFA bypasses)

Build your timeline as you go. Not after. Reconstructing a timeline from memory 48 hours later is fiction writing, not forensics.

Evidence: the chain-of-custody problem

This is the part most startup engineers have never thought about. If your incident turns into a legal matter, and breaches involving customer data often do, you need evidence that holds up.

Practical rules:

• Snapshot before you fix. Always.
• Hash everything you collect. SHA-256 is fine.
• Record who collected what, when, and where it’s stored.
• Don’t modify originals. Work on copies.

A minimal evidence log looks like this:

evidence:
  item: api-server-memory.raw
  collected_by: law
  collected_at: 2019-07-15T10:45:00Z
  hash: sha256:3e0bfd4a4c8662dbe8f663b96b7e5e1bc7bd8281ec39abcffb342d056c88194e
  storage: s3://incident-evidence/2019-07-15/
  notes: captured before container restart

Is this overkill for most startups? Maybe. But the one time you need it and don’t have it, you will wish you had spent the extra five minutes.

Communication: silence isn’t a strategy

Internal communication during an incident should follow a predictable cadence. SEV-1 gets updates every 30 minutes. SEV-2 every hour. Use a simple template:

Incident: checkout-auth-bypass
Severity: SEV-1
Status: containment
IC: law
Last update: 2019-07-15 11:00 UTC
Summary: Unauthorized API access via expired OAuth tokens.
         Revoked all active sessions. Investigating scope.
Next update: 11:30 UTC

External communication is harder and scarier. My advice: get legal involved before you send anything to customers. Not because lawyers make things better, but because disclosure obligations vary by jurisdiction and getting it wrong creates more problems than the breach itself. GDPR gives you 72 hours to notify the supervisory authority. US obligations vary by state and sector; New York’s DFS cybersecurity rule, for example, also sets a 72-hour clock for covered financial companies. Know your obligations before the incident, not during.

The post-mortem: where the real value lives

Run a blameless post-mortem within 48 hours. Not a week later. Not “when things calm down.” Memory degrades fast.

Structure it simply:

1. Timeline: What happened, in order, with timestamps.
2. Root cause: Not “human error.” The systemic reason. Why was it possible for a human to make that error?
3. What worked: Reinforce the things that went right.
4. What broke: Be honest about gaps.
5. Action items: With owners and deadlines. No orphaned tasks.

The post-mortem is how you convert an expensive incident into durable organizational learning. If your team is afraid to write honest post-mortems, that’s a bigger problem than any security incident.

Preparation: the boring stuff that saves you

None of this works if you haven’t done the groundwork:

• Run tabletop exercises quarterly. Describe a scenario. Walk through the response. Find the gaps before they find you. Keep each one to 30 minutes. Low overhead. High value.
• Test your backups. Not “verify they exist.” Actually restore from them. I’ve seen teams discover their backup pipeline had been silently failing for weeks.
• Ensure logging is actually on. Audit trails, cloud API logs, application logs. If it isn’t being collected, it doesn’t exist during an incident.
• Keep a contact list current. Who do you call at your cloud provider? Your legal counsel? Your insurance carrier? Your biggest customer’s security team? Write it down.

Practice before the first breach

Most startups won’t invest seriously in incident response until after their first real breach. That’s human nature. But the gap between “we have a plan” and “we’ve practiced the plan” is the gap between a manageable incident and a company-threatening one. The plan doesn’t need to be perfect. It needs to be practiced.

At the fintech startup, we built a deployment system that was technically sound and architecturally clean. Nobody used it. Engineers kept SSHing into boxes and running scripts manually because our “platform” added steps instead of removing them. That was humbling.

The mistake was obvious in hindsight. We built what we thought engineers should want instead of solving what actually slowed them down.

Platforms Are Products

This is the part most engineering leaders get backwards. You wouldn’t ship a product feature without talking to users. But somehow, platform teams spin up for months in isolation, emerge with a Kubernetes abstraction layer, and then act surprised when adoption is flat.

Your developers are your users. They have deadlines, context-switching overhead, and zero patience for tools that add friction. If your platform doesn’t make their Tuesday afternoon measurably better, it’s shelf-ware.

At the mobility startup, we got this right. We started by sitting with engineers and listing every manual step in their deploy process. Not what we imagined the pain points were. What they actually were. Turned out the biggest time sink wasn’t CI/CD at all. It was environment provisioning. Engineers were burning hours configuring staging environments that drifted from production within a week.

So we fixed that one thing first. Standardized environments, one command to spin up, automatic teardown. Adoption was immediate because it solved a real problem engineers hated.

Paved Roads, Not Walls

The best platforms are opinionated but not authoritarian. You set strong defaults, make the common case trivial, and leave an escape hatch for the 10% of services that genuinely need something different.

The trap is mandating usage. The moment you force teams onto your platform without it being obviously better, you’ve created resentment and shadow infrastructure. I’ve seen this happen. Engineers are resourceful. They will find workarounds, and those workarounds will be worse than whatever you were trying to replace.

The Only Metric That Matters Early On

Forget deployment frequency dashboards and adoption funnels. Early on, there’s one question: would engineers choose your platform if it wasn’t required?

If the answer is no, you have a product problem, not an adoption problem. No amount of migration mandates or executive sponsorship fixes bad developer experience.

Ship Small, Listen Hard

Start with the most painful thing. Fix it. Ship it. Watch what happens. Then find the next most painful thing. That’s it. That’s the strategy.

I’m three weeks into the founder-program cohort now, starting my infrastructure startup from scratch. Even at this stage, with just me writing the code, I’m thinking about what “the default path to production” looks like. Not because I need a platform. Because the habits you set in week one calcify fast, and undoing bad tooling decisions at 20 engineers is a nightmare I’ve already lived through while scaling an engineering team .

Build for the workflow your team already has. Then make the default path undeniably better.

At the fintech startup, our “deployment pipeline” for the first few months was me running kubectl apply from my MacBook at 11pm. I had a checklist in Notes.app. It worked until I forgot step 4 one night and took down the price feed for 45 minutes.

The outage wasn’t catastrophic, but it made me realize that if I got hit by a bus, nobody else could deploy. The entire process lived in my head and my shell history.

GitOps fixed that. It took a while, and it stuck.

What GitOps is

GitOps is a principle: Git is the single source of truth for what your infrastructure should look like, and a controller continuously reconciles reality to match.

That’s it. Everything else is implementation detail.

The word gets thrown around a lot in 2019, mostly by vendors trying to sell dashboards. Strip away the marketing and you get three rules:

1. Declare your desired state in files checked into Git
2. Run a controller that watches those files and applies changes automatically
3. Never touch the cluster directly

If a change isn’t in Git, it didn’t happen. Full stop.

Why pull-based beats push-based

Most CI/CD pipelines are push-based. Jenkins builds your image, then runs kubectl apply against the cluster. The problem: Jenkins now needs cluster credentials, network access, and the ability to mutate production state. Your CI system becomes a god-mode attack surface.

Pull-based GitOps flips this. The controller runs inside the cluster. It watches a Git repo and pulls changes when it sees a new commit. The cluster reaches out to Git, not the other way around.

Push model:  CI --> kubectl apply --> Cluster
Pull model:  Cluster <-- controller watches <-- Git repo

The pull model wins on security first. No cluster credentials in CI. No network path from build servers to production. And because the controller keeps comparing the cluster with Git, drift shows up instead of accumulating quietly.

Setting it up: a real example

Here is the repo layout I settled on after trying three different approaches. It’s for Go microservices on Kubernetes , so it’s Kustomize-flavored.

infra/
  apps/
    api-gateway/
      base/
        deployment.yaml
        service.yaml
        kustomization.yaml
      overlays/
        staging/
          kustomization.yaml
          replicas.yaml
        production/
          kustomization.yaml
          replicas.yaml
          hpa.yaml
    auth-service/
      base/
        ...
      overlays/
        ...
  platform/
    namespaces/
    network-policies/
    rbac/

The apps/ directory is owned by service teams. The platform/ directory is owned by me (or whoever is on infra that week). This separation matters more than any specific tool choice.

The base deployment

Nothing exotic here. A standard Kubernetes deployment with explicit resource requests, health checks, and a pinned image tag. No latest. Ever.

# apps/api-gateway/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
        - name: api-gateway
          image: registry.example.com/api-gateway:v0.8.3
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5

The production overlay

Production gets more replicas, an HPA, and a specific image tag. The overlay touches only what differs.

# apps/api-gateway/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
  - ../../base
  - hpa.yaml
patchesStrategicMerge:
  - replicas.yaml
images:
  - name: registry.example.com/api-gateway
    newTag: v0.8.3

This is the file that changes on every deploy. A version bump is a one-line diff in a pull request. Reviewable. Revertable. Boring.

Choosing a controller: Flux vs Argo CD

In early 2019, the two serious options are Flux and Argo CD. I’ve run both.

Flux is simple. It watches a Git repo, syncs to a cluster, done. Installing it is one Helm chart that runs a Deployment pointed at your repo:

helm repo add weaveworks https://weaveworks.github.io/flux
helm install --name flux \
  --set git.url=git@github.com:example/infra.git \
  --set git.branch=master \
  --set git.path=apps/ \
  --namespace flux \
  weaveworks/flux

Flux feels like a Unix tool. It does one thing. If you want a UI or multi-cluster visibility, look elsewhere.

Argo CD is heavier but more featureful. You define Applications as custom resources:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-gateway
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/infra
    targetRevision: master
    path: apps/api-gateway/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true

Automated sync with prune: true makes a merged commit the deploy, deletions included. And when someone runs a manual kubectl edit in production, the UI marks the application OutOfSync, so drift is visible to the whole team. That alone is worth the setup cost.

My recommendation: start with Flux if you have one cluster and want simplicity. Move to Argo CD when you need multi-cluster, a UI for the team, or you’re tired of explaining to people what state the cluster is in.

We started with Flux. Moved to Argo CD three months later. No regrets on either decision.

The deployment workflow

Here is how a deploy actually works once this is set up:

# Developer bumps the image tag after CI builds a new version
git checkout -b deploy/api-gateway-v0.8.4
vim apps/api-gateway/overlays/production/kustomization.yaml
# Change newTag: v0.8.3 -> v0.8.4
git add -A && git commit -m "deploy: api-gateway v0.8.4"
git push origin deploy/api-gateway-v0.8.4
# Open PR, get review, merge

After merge, the controller picks up the change on its next poll (every few minutes by default; the interval is configurable). The rollout happens. Monitoring confirms. Done.

Compare that to the old world:

# The old way -- do not do this
kubectl set image deployment/api-gateway \
  api-gateway=registry.example.com/api-gateway:v0.8.4 \
  -n production
# Hope you remember what you just did
# Hope the next person knows too

The first approach gives you an author, a reviewer, a timestamp, a diff, and a one-command rollback (git revert). The second gives you nothing.

Secrets: the hard part

Secrets are where GitOps gets uncomfortable. You want everything in Git, but you can’t commit plaintext credentials. In 2019, the options are:

Mozilla SOPS encrypts values in-place using KMS or PGP. The file structure stays readable, only the values are encrypted. This is what I use.

# secrets.enc.yaml (encrypted with SOPS)
apiVersion: v1
kind: Secret
metadata:
  name: api-gateway-secrets
data:
  DATABASE_URL: ENC[AES256_GCM,data:abc123...,type:str]
  API_KEY: ENC[AES256_GCM,data:def456...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:eu-west-1:123:key/abc-def

Bitnami Sealed Secrets encrypts secrets with a cluster-side key. Safe to commit the encrypted form. Decryption only happens inside the cluster.

External secret managers such as Vault or AWS Secrets Manager. The secret reference lives in Git, the actual value lives elsewhere. More moving parts, but better for large orgs.

Pick one. Don’t commit plaintext. I’ve seen production database passwords in public GitHub repos more times than I want to admit.

Handling emergencies

Sometimes you need to bypass the process. A P1 incident at 3am isn’t the time to open a pull request.

The rule: fix it now, commit it immediately after.

# 1. Fix the fire
kubectl rollout undo deployment/api-gateway -n production

# 2. Immediately capture in Git (before you forget or go back to sleep)
git checkout -b hotfix/api-gateway-rollback
# Update manifests to match the rolled-back state
git commit -m "hotfix: rollback api-gateway to v0.8.3 (incident #47)"
git push && # open PR

# 3. Write a note in the incident channel explaining why you bypassed GitOps

If you skip step 2, the controller will eventually reconcile and re-deploy the broken version. I learned this the hard way: we had a demo the next morning and I rolled back manually without committing. Woke up to the broken version redeployed by Flux at 6am. Fun times.

What drift detection tells you

When you first enable GitOps, the controller will fight you. It will revert manual changes, flag resources you forgot about, and generally make noise.

Good. That noise is telling you things:

• A developer ran kubectl edit because the PR process felt too slow. Fix the process.
• A resource exists in the cluster that isn’t in Git. Either add it to Git or delete it.
• Someone created a secret manually during an incident. Capture it properly.

Drift detection is your audit system running in real time. Treat every drift event as a process fix to make, and don’t suppress it.

Repo structure: mono vs split

I tried three approaches before settling:

Mono-repo (app + infra together). Simple for tiny teams. Falls apart when you have more than ~5 services because every deploy touches the same repo and branch protection gets weird.

Split repos (one for app, one for infra). Clean ownership boundaries. The infra repo becomes the system of record. Cross-cutting changes require two PRs, which is annoying but forces you to think about coupling. This is what I recommend for most teams.

Per-app manifests in app repos. Each service owns its deployment config. Great for autonomy, terrible for platform-wide changes like “update the sidecar version on every service.” You will end up writing a bot to open 30 PRs.

For a team of 3-10 engineers, split repos with Kustomize overlays is the sweet spot. You can always migrate later; your manifests are just files in Git.

What I got wrong

A few mistakes worth sharing:

I over-automated image updates. I set up Flux’s auto-image-update feature, which watches a registry and commits new tags automatically. Sounds great. In practice, it deployed a broken build at 2am because CI had a flaky test that passed on a bad commit. Explicit tag bumps via PRs are slower but safer. I turned off auto-updates within a week.

I didn’t enforce branch protection early enough. For the first month, anyone could push directly to master on the infra repo. One accidental force-push later, I added required reviews and status checks. Should have been day one.

I skipped the README. New engineers had no idea how to deploy. The process was obvious to me because I built it. It wasn’t obvious to anyone else. A one-page doc explaining “how to deploy a new version” saved hours of Slack questions.

Getting started

If you’re reading this and still deploying with kubectl from a laptop, here is what I would do tomorrow:

1. Create an infra repo. Move your manifests there.
2. Install Flux and point it at the repo.
3. Enable branch protection on master. Require one review.
4. Deploy something small through a PR. Watch it sync.
5. Delete kubectl access for your CI system. If CI can’t touch the cluster, it won’t.

You can do steps 1-4 in an afternoon. Step 5 takes courage, but it’s the one that makes the whole thing stick.

This is the checklist I use, not a getting-started guide. Everything here exists because I’ve been paged at 3am for the opposite.

Upgrades Should Be Boring

If your Kubernetes upgrade process feels heroic, something is wrong. Two years of running clusters taught me that the boring parts are the hard parts. Stay on a supported minor version. Have a rollback plan. Run it on a schedule, not as a fire drill.

I keep a dead-simple rule: if we’re more than one minor version behind, it’s a P1. Not because something is broken, but because something will break and we’ll be debugging with stale docs and missing patches.

Resource Requests and Limits. On Everything.

This is the single most impactful thing you can do. No exceptions.

resources:
  requests:
    cpu: "250m"
    memory: "512Mi"
  limits:
    cpu: "1000m"
    memory: "1Gi"

Requests are your scheduling guarantee. Limits are your safety net. Without both, you’re one misbehaving pod away from cascading node pressure. I’ve seen a single Go service with no memory limit take down a shared node at the fintech startup because someone forgot to close a channel and goroutines piled up. That was a fun Friday evening.

Size these from actual metrics, not guesswork. The metrics server and Prometheus give you enough signal. Start conservative, then adjust.

Namespaces With Actual Guardrails

Namespaces without resource quotas are just labels. Enforce boundaries at the namespace level so one team’s experiment can’t starve another team’s production workload.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi

Probes Are Mandatory

Not optional. Not “we’ll add them later.” Mandatory.

Liveness probes restart broken processes. Readiness probes keep unready pods out of the Service’s endpoints, so they get no traffic. Without them, you’re routing traffic to containers that are technically running but functionally dead.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

One thing I always add: make /healthz and /ready do different things. Your liveness check should be “is the process alive.” Your readiness check should be “can this instance serve traffic right now”: database connected, caches warm, dependencies reachable.

Handle Shutdown Properly

Your container will get SIGTERM’d. Handle it. Stop accepting new work, drain in-flight requests, exit clean. Set terminationGracePeriodSeconds to match your actual drain time, not the default 30 seconds.

In Go (which is what I write most services in), this means catching the signal and giving your HTTP server a context-bounded shutdown. It’s ten lines of code. There’s no excuse.

Secrets Deserve Respect

Two rules:

1. Mount secrets as files, not environment variables. Env vars leak into crash dumps, child processes, and logging. Files don’t.
2. Enable encryption at rest in etcd. If you’re on a managed provider, verify this is actually on. Don’t assume.

For anything beyond toy projects, use an external secret manager. Kubernetes native secrets are base64-encoded, not encrypted. Big difference.

Default-Deny Networking

Without network policies, every pod can talk to every other pod. That’s not a cluster, it’s a flat network with extra steps.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Start with deny-all, then explicitly allow what’s needed, the same default-deny idea behind the zero trust setup we built . Yes, it’s more work upfront. But “every service can reach every other service plus the internet” isn’t a security posture I’m comfortable with, and you shouldn’t be either.

My order of priorities is always the same: security, stability, performance. In that order. Network policies fall squarely in bucket one.

RBAC and Pod Security

Run as non-root. Drop capabilities. Use read-only root filesystems where you can. Enforce these with PodSecurityPolicy so it’s not just a suggestion.

For RBAC: if any service account has cluster-admin, fix that first. Service accounts get the verbs and resources they need. Nothing more.

Roll Out Safely

Use rolling updates with sensible surge and unavailable settings. Add Pod Disruption Budgets so node drains and maintenance don’t accidentally kill your service. Spread replicas across nodes and zones with anti-affinity rules.

None of this is glamorous. All of it prevents 2am pages.

Observability That Helps

Three things:

Structured logs. JSON, consistent fields, correlation IDs. If your logs are unstructured strings, your incident response is grep and prayer.

Application metrics tied to SLOs. Service-level objectives tell you whether users are happy; vanity dashboards don’t. Expose the metrics with Prometheus and build dashboards that answer questions.

Consistent labels. Every resource gets app, team, version, environment. Those labels are what make cost allocation, debugging, and incident response possible at any scale beyond one person.

Back Up etcd. Test the Restore.

A backup you’ve never restored is a hope, not a backup. Schedule restores. Verify them. The five minutes this takes will save you from the worst day of your career.

The Pattern

The teams I’ve seen run Kubernetes well aren’t doing anything clever. They do the basics consistently and spend their creativity on the product instead of firefighting infrastructure.

Declarative or go home

Imperative scripts describe steps. Declarative config describes outcomes. Huge difference. Terraform handles ordering, diffs, idempotency. You just tell it what you want.

resource "aws_instance" "web" {
  ami           = "${var.ami_id}"
  instance_type = "t3.micro"

  tags = {
    Name = "web"
  }
}

At the fintech startup we stopped writing bash provisioning scripts the week we adopted Terraform. Never looked back.

Immutable over mutable. Always.

Replace, don’t patch. When config changes, tear down and rebuild. Sounds wasteful until you realize in-place mutations are where drift lives. Rollback becomes “redeploy the previous version” instead of “figure out which of 47 manual tweaks broke something.”

If it’s not in git, it doesn’t exist

I mean this literally. We reject any infrastructure change that isn’t a PR. No exceptions. The diff is the documentation.

Repo layout

We use a monorepo. One repo, all infra. Refactors are easier, search is easier, onboarding is easier. The risk is blast radius, which is what small state files and good reviews are for.

Structure by concern, not by team:

terraform/
  global/
  network/
  data/
  services/
    api/
    web/

Each directory maps to its own state file. Smaller state means faster plans and smaller explosions when someone makes a mistake.

Modules that don’t suck

A few rules I enforce:

One job per module. A network module. A database module. Not an everything-module with 30 variables and a prayer.

Explicit inputs, focused outputs. If your module takes 15 required variables, it’s doing too much. Outputs should be stable values other modules can depend on without breaking.

Sensible defaults. Most consumers should be able to use your module with minimal config. Overrides available, not required.

Compose, don’t inherit. Build bigger systems by wiring small modules together. Adding flags to one giant module is how you get a 2000-line main.tf that nobody wants to touch.

Pin your versions. Unpinned modules are a time bomb. We learned this the hard way when an upstream module update silently changed our security group rules. Pin everything, upgrade deliberately.

Environments

One directory per environment. Shared modules underneath. Each environment gets its own state.

# environments/production/main.tf
module "app" {
  source         = "../../modules/app"
  environment    = "production"
  instance_count = 3
}

Environments differ by variables, not by code. Promotion is dev to staging to production, same Terraform, different tfvars. Simple.

Workspaces? Fine for side projects. For a real team, explicit directories win. Workspaces hide differences and make it too easy to apply to the wrong environment.

CI pipeline

Treat infra code like app code. Our pipeline:

terraform fmt -check
terraform init
terraform validate
terraform plan -no-color

Plan output goes into the PR as a comment. Apply is gated behind manual approval for production. Fast automation, human control where it counts.

Drift detection

People will click things in the console. It happens. We run scheduled plans and alert on any diff. The goal is getting manual fixes back into code before they become invisible tech debt. Nobody gets punished.

Secrets and state

Three things I’m strict about:

Secrets never go in .tf files. Use AWS Secrets Manager, Vault, injected env vars. Whatever. Just not in your repo.

State files are sensitive. They contain plaintext values for things you’d rather keep private. Remote backend, encryption on, locking on.

terraform {
  backend "s3" {
    bucket         = "terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Least privilege for CI. Your automation role shouldn’t be able to delete your account. Scope it tight to the resources it manages. Broad permissions hide mistakes until they become outages.

The boring part is the point

Good IaC is boring. Small diffs, predictable plans, easy reviews, easy rollbacks. That’s the goal. At the fintech startup this approach has kept our infra stable through multiple scaling phases, and I’d use the same patterns anywhere else.

PodSecurityPolicy is real now

When I wrote the first post, PodSecurityPolicy existed, but almost nobody used it. That changed fast. We rolled it out at the fintech startup in Q1, and it caught issues immediately. A dev tried to deploy a container with hostNetwork: true for a quick port-forwarding hack. PSP rejected it. Before PSP, that would have gone through and sat there until someone noticed, or didn’t.

The enforcement model is rough. You define policies, bind them via RBAC, and hope the interaction between multiple policies does what you think it does. We spent a solid week debugging why certain pods wouldn’t schedule. But the alternative, trusting every deploy not to escalate privileges, is worse.

We moved past Alpine

In the 2017 post I showed an Alpine-based final image. We’ve since moved most Go services to scratch or distroless. No shell. No package manager. No /etc/passwd. An attacker who gets code execution inside one of these containers gets almost nothing. They can’t spawn a shell because there isn’t one.

The tradeoff is debugging. You can’t kubectl exec into a scratch container and poke around. We lean on better structured logging instead. Worth it.

Image signing went from nice-to-have to required

Docker Content Trust plus Notary. We gate all production pulls on signature verification now. This came out of a near-miss: a CI pipeline pulled from an internal registry without verifying the image hadn’t been tampered with. Nobody tampered with it. But nobody would have known if they had.

export DOCKER_CONTENT_TRUST=1
docker trust sign registry.internal/api:2.4.1

Simple. We should have done it a year ago.

Network policies still matter more than service meshes

Istio is getting a lot of attention, and it hit 1.0 at the end of July. We’re still not running it. NetworkPolicy with Calico does what we need: default-deny, explicit allows, and no extra control plane to babysit. When Istio matures, maybe. For now, we keep it simple.

Secrets: from Kubernetes Secrets to Vault

We switched from Kubernetes Secrets to Vault-backed dynamic credentials for database access. Short-lived tokens that rotate automatically. The ops overhead was real (Vault isn’t a simple system), but the security posture improvement was immediate. No more long-lived database passwords sitting in etcd.

Where we are

Eight months of tightening. Every service runs non-root, read-only filesystem, all capabilities dropped. PSP enforces it cluster-wide. Images are signed. Network policies are default-deny. Secrets come from Vault.

Most of it’s YAML and policy files. But the gap between “we know what to do” and “we do it everywhere” is where breaches live. We’re closing that gap, one deploy at a time.

I spent forty minutes SSH-ing into boxes, tailing logs, and grepping for exceptions. Nothing. The pipeline just… stopped processing. It took me another hour to discover the root cause: a third-party API we depended on had started returning empty 200 responses instead of actual data. Our monitoring checked for errors. It checked for timeouts. It never checked for “success that contains nothing useful.”

That night changed how I think about production systems.

Monitoring Checks Boxes. Observability Answers Questions.

Monitoring is built around things you already know can go wrong . Threshold crossed, alert fires, runbook engaged. It works great for predictable problems: disk filling up, CPU pegged, error rate spiking. We had all of that at the fintech startup. Grafana dashboards everywhere. PagerDuty wired up. It felt safe.

But distributed systems don’t fail in predictable ways. They fail in weird, combinatorial, never-seen-this-before ways. And that’s where monitoring falls apart. You can’t write an alert for a failure mode you haven’t imagined yet.

Observability flips the model. Instead of predefining what questions the system can answer, you instrument it richly enough that you can ask new questions on the fly. During an incident. At 2 AM. Without deploying anything.

The difference: monitoring tells you something is wrong. Observability helps you figure out why.

Three Signals, One Story

After the empty-200 incident, I started rebuilding our telemetry around three pillars.

Metrics give you the bird’s-eye view. Aggregated numbers over time. At the fintech startup, we track request rates, error rates, and latency distributions for every service using Prometheus.

http_requests_total{method="GET", endpoint="/api/stories", status="200"} 15234
http_request_duration_seconds_bucket{endpoint="/api/stories", le="0.5"} 421

Metrics are cheap to store and great for alerting. But they can’t tell you why a specific request failed or what user it affected. They’re the smoke detector, not the fire investigator.

Logs capture the details. Each event, each error, each decision the code made. We moved early to structured JSON logs because free-form strings are nearly useless at scale.

{"timestamp":"2018-07-09T10:30:45Z","level":"error","service":"ingestion","request_id":"abc123","message":"Empty response from provider","provider":"reuters","duration_ms":340}

That structured format meant we could query logs in Kibana instead of grepping through them. That alone made incident response faster.

Traces show you the journey of a single request across services. This was the missing piece for us. The fintech startup’s architecture involves an ingestion service talking to NLP processors talking to a ranking engine talking to the API layer. When something is slow, you need to see the whole chain.

Trace abc123
- POST /ingest/batch (120ms)
  - NLP enrichment (45ms)
  - Relevance scoring (30ms)
  - db.write stories (35ms)

We started with Zipkin. Not perfect, but suddenly I could see that our NLP service was adding 200ms of latency on certain content types. That was invisible before.

The Glue: Correlation IDs

Each signal alone is useful. Together, connected by a shared identifier, they’re powerful.

The workflow that matters: a latency alert fires from metrics. You pull up the trace for a slow request. The trace shows a specific span taking too long. You jump to the logs for that span using the same request ID and find the exact query and error.

X-Request-ID: 5f3c9e86c2c84e1b
X-B3-TraceId: 4d1e00a3b9bd1d42
X-B3-SpanId: 6df3a1c2b93f6b1a

We made it a rule: every log line includes the request ID, and every trace propagates context headers. No exceptions. It took weeks to retrofit across all our services, but the payoff during the next incident was immediate. Instead of an hour of archaeology, I had a clear thread to pull.

Instrument the Critical Path First

When I started adding instrumentation at the fintech startup, the temptation was to instrument everything. Don’t do that. You’ll drown in data and your storage costs will spike.

Start with what matters: the critical path your users depend on.

For us that meant inbound API handlers, the news ingestion pipeline, database calls, and every external API dependency. We used RED (rate, errors, duration) for our services and USE (utilization, saturation, errors) for infrastructure resources.

Tracing has a sampling problem. Capturing every single request is expensive. We settled on keeping 100% of errors and sampling successful requests at about 5%. During incidents, we crank sampling up. Good enough for debugging, affordable enough to run continuously.

Designing Systems That Can Be Debugged

Observability isn’t something you bolt on after the fact. It’s a design choice.

Structured logs over free-form strings. Always. If you can’t query it, it’s useless when you’re under pressure at 2 AM.

Watch your cardinality. Metrics with unbounded label values (like user IDs) will destroy your Prometheus instance. High-cardinality data belongs in logs and traces, not metrics.

Add context that matters. A trace that only shows timings is half the story. Include the request type, the tenant, the result count. When something goes wrong, that context is the difference between a five-minute fix and a two-hour hunt.

Our Stack in 2018

For anyone building this out now, this is what we run:

• Metrics: Prometheus with Alertmanager, Grafana for dashboards
• Logs: Fluentd piping into Elasticsearch, Kibana for exploration
• Tracing: Zipkin with OpenTracing instrumentation

OpenTracing is getting solid adoption and keeps us from being locked into one vendor. OpenCensus is emerging as an alternative worth watching. Both are pushing toward a world where trace context propagation just works out of the box.

There are commercial options that bundle everything behind one query layer. We looked at them. For our scale, the open source stack made more sense. That calculus changes depending on team size and how much operational overhead you can absorb.

What That 2 AM Incident Taught Me

That silent pipeline failure exposed a blind spot in how I thought about production readiness. Having dashboards isn’t the same as having understanding.

We still have incidents at the fintech startup. But now when something breaks in a way we’ve never seen before, we have the telemetry to figure it out fast.

The principles behind SRE are good. I don’t dispute that. What drives me up the wall is watching teams of fifteen people set up elaborate error budget policies and multi-stage canary deployments when they have three services and a hundred users. You don’t need Google’s playbook. You need common sense with a bit of structure.

What SRE gets right

The core idea is dead simple: treat reliability as a feature, not a side effect. Measure it. Set targets. Make trade-offs explicit instead of pretending you can ship fast and never break anything.

At the fintech startup, I started by picking our most critical service, the real-time financial data API, and defining one SLO for it. Not twelve. One. “99.9% of requests succeed under 200ms.” That single number gave us a shared language between engineering and product. When product wanted to ship something risky, we could point at the error budget and have a real conversation instead of gut-feel arguments.

That’s the part people skip. The error budget isn’t a fancy dashboard widget. It’s a negotiation tool. Budget healthy? Ship bold. Budget burning? Slow down. Simple.

SLOs aren’t a religion

This is where the cargo-culting starts. I’ve seen teams spend weeks building SLO tracking infrastructure before they even know what their users care about. They’ll measure fourteen signals and set targets for all of them because the Google SRE book said so.

Pick the one or two things that matter to your users. For us it was latency and availability of the news feed. That’s it. We didn’t need saturation metrics for a service running on three nodes. We didn’t need a formal SLA document when our “SLA” was “don’t let the paying customers notice.”

Start with what hurts. The rest can wait.

Toil is real, but so is over-engineering

The toil concept is the best thing to come out of SRE thinking. Manual, repetitive work that scales with the system and produces no lasting value. Kill it. Automate it. Eliminate the need for it entirely if you can.

At the fintech startup we had a deployment process that involved SSH-ing into machines and running scripts in a specific order. Classic toil. We automated it with a simple CI pipeline. Nothing fancy. No custom deployment orchestrator, no multi-stage canary with automatic rollback. A pipeline that ran tests and deployed if they passed.

The temptation is always to build the Rolls-Royce version. Resist it. Automate the thing that’s wasting the most time right now. Move on. Come back later if you need more.

Monitoring: stop alerting on nonsense

I had an alert that fired when CPU hit 80%. It went off constantly. Nobody cared. It trained the whole team to ignore alerts, which is the single worst outcome a monitoring system can produce.

We ripped out the noise and replaced it with alerts tied to user impact . P99 latency above our SLO threshold for ten minutes? That pages someone. CPU is high but latency is fine? That’s a graph you check during business hours.

The four golden signals (latency, traffic, errors, saturation) are a fine starting framework. But “framework” is the key word. You don’t need all four on day one. You need the ones that tell you when users are hurting.

Incidents and postmortems

This part most teams do need, regardless of scale. When something breaks, have a plan that isn’t “the person who knows the most panics the hardest.”

We kept it simple. Mitigate first, investigate later. Communicate during, not after. And when it’s over, write up what happened without blaming anyone. Blameless postmortems aren’t some warm-fuzzy HR initiative. They’re how you actually learn things, because people tell the truth when they’re not afraid of getting fired.

The part nobody talks about: follow through on the action items. I’ve seen more postmortem documents gather dust than I care to admit. If you’re not going to fix the thing, don’t bother writing it down.

Gradual rollouts are just good sense

Deploy to 1%, watch the metrics, expand to 10%, watch again, go to 100%. This isn’t rocket science. It’s the engineering equivalent of “taste the soup before serving it.” But you don’t always need automated canary analysis and a dedicated deployment pipeline to do it. Sometimes a feature flag and a pair of eyes on a dashboard is plenty.

Rollback capability matters more than rollout sophistication. Can you get back to the last known good state quickly? If yes, you’re in decent shape. If no, fix that before you build anything else.

SRE gives you a vocabulary for thinking about reliability, and that’s valuable. Treating the Google SRE book like scripture and implementing every practice wholesale because a conference speaker said so is not.

At the fintech startup, we run our infrastructure on Kubernetes. We’ve evaluated operators, built internal tooling around them, and dealt with the sharp edges firsthand.

What an Operator Is

Dead simple concept. An operator is a controller that watches a custom resource, compares desired state to actual state, and reconciles the difference. Same pattern as built-in K8s controllers, just applied to your specific application.

You define a CRD like this:

apiVersion: example.com/v1
kind: Database
metadata:
  name: orders-db
spec:
  engine: postgres
  version: "10"
  replicas: 3

The operator sees that and creates the StatefulSets, Services, ConfigMaps, backup jobs, whatever the system needs. Intent-driven. Declarative. Very Kubernetes-native.

The real value is Day 2 operations. Scaling, rolling upgrades, backups, failover, drift correction. The stuff that happens after you deploy. That’s where operators shine and where manual runbooks fall apart at 3am.

Where the Hype Breaks Down

What the conference talks skip: writing a good operator is hard. Writing a bad one is easy. And a bad operator running with cluster-level RBAC is a liability.

I’ve seen teams spend weeks building custom operators for problems that a well-written Helm chart and a cron job would have solved. The pattern is seductive (“we’ll encode all our operational knowledge into code!”) but the implementation details are brutal.

Your reconciliation loop has to be idempotent. Truly idempotent. Not “works most of the time” idempotent. You need proper owner references for garbage collection. You need finalizers if you touch anything outside the cluster. You need status conditions so someone debugging at 2am can see what’s going on. Most first-attempt operators skip half of this.

Build vs. Adopt (Usually Adopt)

At the fintech startup, we evaluated building custom operators early on. For most of our use cases, adopting existing ones was the right call. The etcd operator and the Prometheus operator come from CoreOS, which introduced the operator pattern in 2016, and they have real production hours behind them. Our custom needs weren’t special enough to justify the maintenance burden.

Build your own only if your system is genuinely proprietary, your operational rules can’t be expressed with existing operators, or you need tighter control than third-party code allows. Otherwise, use what exists.

If You Write One Anyway

A few things will save you:

• Idempotent reconciliation. Compute desired state from the CR every single time. No “run once” actions without guards.
• Owner references. Let K8s garbage collection do its job.
• Status fields. If kubectl describe can’t tell me what’s broken, your operator is incomplete.
• Least privilege RBAC. Operators often end up with way too many permissions. Start minimal.
• Conservative defaults. Easier to loosen limits than to recover from an operator that went aggressive on scaling.

Test against a real cluster. Unit tests alone won’t catch the weird timing issues that only show up when the API server is under load.

Where I Land

Operators are the right abstraction for complex stateful workloads on Kubernetes. They’re not magic, and they’re not always the right tool. Most teams I talk to would be better served by adopting a mature community operator than by building one from scratch. We still keep our own databases outside the cluster .

The pattern will mature, and the tooling for writing operators will get better. Right now, in 2018, approach with clear eyes about what you’re signing up for.

I wrote about our first year with Kubernetes a year ago. The tone was “we survived and it was worth it.” That’s still true. But year two taught me that survival and maturity are very different things.

The networking tax

I said networking was hard last year. Understatement.

We started with a basic CNI and no network policies. Fine for a while. Then we added more services, more teams started deploying, and suddenly every pod could talk to every other pod. No segmentation. No audit trail of who talks to what. Fixing this retroactively was brutal. You can’t just flip on network policies without mapping every legitimate connection first. We spent two weeks doing traffic analysis before we could write a single policy.

DNS was the other trap. Kube-dns worked great at our initial scale. Ten services, light traffic, no complaints. Then we added a data pipeline that hammered DNS with lookups on every request. Intermittent failures. Slow responses that looked like application bugs. Took us days to trace it back to DNS resolution under load. The fix was trivial: ndots configuration and a local DNS cache. The debugging wasn’t.

Treat networking as its own project. First-class. With a budget. Not something you’ll “figure out later.”

Resource requests stopped being guesses

Year one, our resource requests were vibes. Somebody would eyeball a service, pick a number, ship it. We got away with it because we had plenty of headroom.

Year two, the cluster got denser. Services started competing for resources. OOMKills on one service, wasted capacity on another. The scheduler was doing exactly what we told it to. The problem was that we told it garbage.

We started profiling every service under realistic load in staging. Measured actual memory and CPU usage over a week. Then set requests based on the P95, not the average, not a guess. Kept memory limits close to requests. Dropped CPU limits entirely unless a service had a known runaway pattern.

resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"

That YAML block looks boring. It’s also the difference between a stable cluster and a 3 AM page. The numbers matter. Profile them.

Stateful workloads: still nope (mostly)

We tried running Elasticsearch inside the cluster for about a month. Storage behavior was unpredictable. Backup and restore required custom tooling. Recovery after a node failure was slow and manual. We moved it back to managed infrastructure and haven’t looked back.

Postgres and Redis stayed outside the cluster all of year two. I know people run stateful workloads on Kubernetes successfully. I also know it requires an operational investment we weren’t ready to make. Managed services aren’t free. They just shift the cost from your on-call to your invoice. For us, that trade was obviously correct.

PodDisruptionBudgets: the lesson I learned the hard way

Back to that 3 AM page. A single node drain shouldn’t be able to take down a production service. That’s the whole point of running multiple replicas. But without a PodDisruptionBudget, Kubernetes will happily drain every pod from a node regardless of what service they belong to. If all your replicas happen to land on the same node (and the scheduler will do this if your anti-affinity rules are weak), one drain takes you to zero.

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

We now treat PDBs as part of the deployment contract. You don’t get to ship a production service without one. Period.

Upgrades aren’t routine

Upgraded the cluster twice in year two. Both times were more work than expected.

Deprecations broke manifests. API versions we were using got dropped with one minor release warning. Node draining surfaced resource issues we’d been ignoring. Control plane changes rippled into unexpected places. One upgrade changed default admission controller behavior and three deploys failed before we figured out why.

The process that worked: read every release note cover to cover. Rehearse in staging with production-like load. Not “run the tests.” Push real traffic through it. Schedule a real maintenance window. Don’t YOLO a cluster upgrade on a Friday afternoon. (We did this once. Never again.)

Multiple clusters helped. We moved to a setup where we could upgrade one cluster while the other kept serving. Longer to provision, but the blast radius dropped to near zero.

Git as the source of truth

This was already working in year one, but by year two it became non-negotiable culture. Every manifest in version control. Every deployment triggered by a merge. No kubectl apply from someone’s laptop. No exceptions.

The payoff isn’t just auditability. It’s rollbacks. Something breaks, you revert a commit. No guessing what changed. No “who ran what command in production.” The answer is always in the git log. This single practice eliminated more incidents than any other operational improvement we made.

RBAC and secrets: the wake-up call

Year one, everyone had cluster-admin. Year two, we finally locked it down. Created service accounts per application, scoped access narrowly, and spent an uncomfortable afternoon discovering how many things broke when we removed broad permissions. Every one of those breakages was a security hole we’d been ignoring.

Secrets were the other wake-up call. Base64 isn’t encryption. It’s encoding. Anyone with API read access could decode every secret in the cluster. We moved to pulling secrets from Vault at deploy time and encrypting etcd at rest. Should have done it in month one. Didn’t.

Java, DNS, and autoscaling

Some workloads need special treatment. Our Java services ignored container memory limits by default and allocated memory like they owned the whole node. Took a few OOMKills to learn that the JVM needs explicit container-aware flags.

DNS bit us again here. Services making too many lookups created a bottleneck that looked like network latency. Horizontal pod autoscaling only worked correctly once our base resource requests were accurate. Garbage in, garbage scaling decisions out.

What I’d tell someone starting today

Use managed Kubernetes. Don’t run your own control plane unless you have a very specific reason. GKE is mature, and AWS and Azure both announced managed offerings late last year. The time you save goes directly into product work.

Invest in developer experience from day one. The platform can be rock solid and still feel terrible if developers can’t get a fast local feedback loop. Minikube, namespaced dev environments, whatever works. Budget time for it early.

Don’t touch service mesh until the basics are boring. I’ve seen teams adopt Istio before they had working health checks. Get networking, observability, and resource management right first. Then decide if you need the complexity.

Year three

The cluster is boring now. Boring in the good way. Deploys happen multiple times a day without drama. Incidents are rarer and smaller. The platform does what we need it to do.

But it took two full years of grinding to get here. None of that’s glamorous. None of it makes for a good conference talk. It’s just the work.

Kubernetes is a platform that rewards the teams willing to operate it seriously. Two years in, I’m convinced it’s the right foundation. I’m also convinced that most of the value comes from the boring operational work that nobody wants to do.

That was March. By December, we had a small platform team that actually worked. Getting there was messy.

The problem was obvious

We had product engineers writing their own deployment scripts. Plural. Each service had its own way of getting to production. Some used shell scripts checked into the repo. One team had a guy who just SSH’d into the box. Logging was inconsistent. Monitoring was whatever someone remembered to set up. When something broke at 2 AM, diagnosing it meant guessing which service did what and where its logs lived.

This is fine when you have three services. We had more than three services.

What we built

I kept the scope ruthless. Four areas, nothing else.

Developer experience. CI/CD pipelines , a shared staging environment, deployment tooling that didn’t require tribal knowledge. This was the first thing we tackled because it was the loudest pain point.

Runtime infrastructure. Databases, caches, message queues. Standardized. Documented. Not six different Postgres configurations floating around.

Observability. Centralized logging and monitoring . One place to look when things go wrong. This alone saved us hours every incident.

Security defaults. Secrets management that wasn’t “put it in an environment variable and hope.” Auth tooling that product teams could plug into without reading a novel.

Treat it like a product or it dies

What I didn’t expect: building the platform was the easy part. Getting people to use it was the actual job.

We had product engineers with muscle memory. They knew their janky deploy scripts. They didn’t trust our new pipeline. Fair enough. So I stopped thinking of the platform as infrastructure and started thinking of it as a product. Our users were the product teams. If they didn’t adopt what we built, we failed. Full stop.

This meant sitting with teams. Watching them work. Asking dumb questions like “why did you just do that manually?” and then going back and automating whatever that was. It meant writing docs that were actually useful, not docs that checked a box. It meant having a Slack channel where people got answers fast.

Self-service was non-negotiable. If a team needed a new database, they should get it in minutes. Not file a ticket. Not wait for me to wake up. The moment we became a ticket queue, we became the bottleneck we were trying to eliminate.

How we built the team

I started alone, then pulled in one more person with strong ops background. That was the right call. You need someone who has been paged at 3 AM and someone who knows how to build tooling that doesn’t feel like punishment to use. Pure infra people build things that work but nobody can figure out. Pure app developers build things that look nice but fall over under load.

We stayed small. Two people, then three by Q4. Attached to engineering leadership, sitting next to the product teams. Physically close. That proximity matters more than any process. When you overhear someone complaining about deploys, you fix deploys. When you’re in a different building, you build things nobody asked for.

Paved roads over mandates

We never mandated anything. I hate mandates. Instead, we built paved roads, to borrow Netflix’s term: a service template, a default pipeline config, a monitoring setup that worked out of the box. You could ignore it all and do your own thing. But the paved road was so much easier that nobody bothered.

That’s the trick. Make the right thing the easy thing. Don’t write policies. Write code that makes the policy unnecessary.

We exposed platform capabilities through simple APIs. Want a database? Here’s a YAML spec, submit it, done. Want to deploy? Push to master. Want logs? They are already where you expect them.

apiVersion: platform.example.com/v1
kind: Database
metadata:
  name: billing-db
spec:
  engine: postgresql
  size: small

Nothing fancy. Deliberately boring. Boring infrastructure is good infrastructure.

What I would do differently

I waited too long to write documentation. The first three months, everything lived in my head and in Slack threads. That doesn’t scale, even at our size. Write the docs early, even if they are rough.

I also underestimated how much time support would eat. By September, I was spending most of my week answering questions and helping teams migrate. Almost no time left to build new capabilities. We fixed this by setting explicit office hours and protecting two full days per week for building. Should have done that from day one.

Where this goes next

We’re a small company. This setup works for our size. If the fintech startup doubles its engineering team, the platform group will need to split into focused sub-teams: developer experience, infrastructure, security. But not yet. Right now the right move is staying lean and staying close to the people we serve.

A platform team is a product team that happens to build infrastructure. The moment you forget the product part, you’re just an ops team with a fancier name.

Containers Aren’t a Security Boundary

I need to say this bluntly because I keep hearing it wrong: a container isn’t a VM. It isn’t a sandbox. It shares a kernel with the host. If something breaks out, the blast radius is the whole node. Full stop.

At the fintech startup we learned this the hard way during an early Kubernetes migration . We had a service running as root inside its container (“just for debugging,” someone said), and a misconfigured volume mount gave it read access to the host’s /etc. Nobody exploited it. We caught it in review. But it sat there for two weeks before anyone noticed, and that scared me more than an actual breach would have.

A principle from cyber-defense exercises that applies everywhere: assume the perimeter is already broken. Then design so that a breach in one layer doesn’t cascade. Containers are just another layer. Treat them that way.

Ship Less, Expose Less

The single highest-ROI thing you can do for container security is shrink your images. Every package you include is attack surface. Every binary is a tool an attacker can use post-exploitation. Multi-stage builds are your best friend here.

FROM golang:1.9 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o service .

FROM alpine:3.6
RUN adduser -D appuser
USER appuser
COPY --from=builder /app/service /service
ENTRYPOINT ["/service"]

This is almost exactly what we use at the fintech startup for our Go services. Builder stage compiles everything. Final image has Alpine, a non-root user, and the binary. That’s it. No curl, no bash, no wget. If an attacker gets code execution inside this container, they have almost nothing to work with.

Two more things that matter:

Pin to digests, not tags. alpine:3.6 can change underneath you. alpine@sha256:abc123... can’t. We had a CI build pass on Tuesday and fail on Thursday because the base image got a patch that broke our libc expectations. Tags are lies. Digests are truth.

Scan continuously. Not just at build time. Vulnerabilities get published weeks or months after an image ships. We run Clair against our registry on a cron. Anything with a critical CVE gets flagged and the owning team has 48 hours to patch or explain why not.

Runtime: Where Mistakes Become Incidents

A clean image means nothing if you run it with --privileged. I’ve seen production deployments where containers had SYS_ADMIN capabilities because someone copy-pasted a Stack Overflow answer during a late-night deploy. That one capability basically gives you root on the host.

Strip everything. Start from zero and add back only what you need.

securityContext:
  runAsNonRoot: true
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

Three lines of YAML. That’s it. runAsNonRoot stops the “just for debugging” excuse. readOnlyRootFilesystem prevents an attacker from dropping binaries. Dropping all capabilities means no mount, no raw sockets, no ptrace. If your app genuinely needs a specific capability, like NET_BIND_SERVICE to bind port 443, add that one back explicitly and document why.

Pair this with seccomp profiles if your runtime supports them. Default Docker seccomp blocks around 44 syscalls. For most web services, you can block even more. We profiled our API gateway’s syscall usage over a week, built a whitelist, and cut the available syscall surface by 60%.

The other half is detection. You need to know when something unexpected happens inside a container. A new shell process. An outbound connection to an IP that isn’t in your dependency list. A binary that wasn’t in the original image. We keep this simple: sysdig + alerting rules. Nothing fancy. But it works because we actually look at the alerts.

Network Segmentation: Kill Flat Networks

Default Kubernetes networking is flat. Every pod can talk to every other pod. This is the container equivalent of putting every server in the same VLAN with no firewall rules. Insane, but it’s the default.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-frontend
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - port: 8080

This policy says: the API pod accepts traffic from frontend pods on port 8080. Everything else is denied. Lateral movement goes from trivial to hard: the attacker now has to compromise the frontend service too.

At the fintech startup we started by mapping every service-to-service dependency, then wrote deny-all policies and added explicit allows. Took about a week for our cluster. Found three services that were talking to things they had no business talking to, leftovers from an old feature that got half-removed. Network policies are a security tool and an architecture audit tool.

Service meshes with mutual TLS are starting to appear (Istio, Linkerd), but they’re early. We’re watching, not adopting yet. For secrets, keep them out of your images. Kubernetes Secrets with encryption at rest (alpha since 1.7), or Vault if you want something more robust. Never ENV a database password into a Dockerfile. I’ve seen it. In production. At companies you’ve heard of.

RBAC and Audit Logging

Kubernetes RBAC isn’t optional. Neither is PodSecurityPolicy. These are the controls that prevent a developer with deploy access from accidentally (or intentionally) escalating to cluster-admin. Scope permissions to namespaces. Give teams access to their stuff and nothing else.

Audit logging closes the loop. When something goes wrong, and it will, you need to answer: who did what, when, and to which resource. Without audit logs, incident response is guesswork. With them, it’s a timeline.

A Checklist, Applied Every Time

None of this is exotic. Slim images, pinned builds, non-root containers, read-only filesystems, dropped capabilities, network policies, RBAC, audit logs. It’s a checklist. The hard part is doing it consistently, across every service, on every deploy, even when someone says “we’ll fix it after launch.”

Stack these controls and a container escape becomes a dead end instead of a highway to your entire infrastructure.

That incident cost us a client conversation I never want to have again.

At the fintech startup, we were five people for a long time. Incident response was simple: someone noticed a problem, yelled about it, and whoever was closest fixed it. Worked great. Then we grew. More services, more people, more time zones. That informal model didn’t scale. It just produced chaos with more participants.

Roles Kill Ambiguity

The single biggest improvement we made was assigning roles during incidents. Not permanently, per incident.

Incident commander. One person owns coordination. They decide severity, pull in the right people, and keep things moving. They don’t debug. They run the show.

Technical lead. The person actually investigating and fixing the problem. They talk to the incident commander, not to Slack at large.

Comms lead. Someone handles customer updates and internal stakeholders. Separating this from the technical work was huge for us. Engineers hate writing status updates mid-crisis, and it shows.

Scribe. Captures the timeline. What happened when, what was tried, what worked. Without this, your postmortem is fiction.

For smaller incidents, one person wears multiple hats. Fine. But for anything SEV-1, staff every role. Context switching during a crisis is where mistakes compound.

Severity Levels Are a Decision Framework

We use four levels. SEV-1 is full outage or data loss: all hands, immediate external comms. SEV-2 is partial degradation: on-call team owns it with escalation paths. SEV-3 hits a subset of users: on-call handles it. SEV-4 goes through normal workflow.

The numbering matters less than everyone agreeing on what warrants waking someone up at 3 AM versus filing a ticket. Without that agreement, you get two failure modes: either every alert is treated as a fire drill, or real emergencies get a slow response because the team is desensitized.

On-Call That Doesn’t Destroy People

I’ve seen on-call rotations burn out good engineers in months. We run week-long rotations with explicit handoffs and a secondary on-call for escalation. The handoff is a real conversation, not a calendar entry.

Three rules we follow:

Response time expectations are written down. Not “ASAP”, actual numbers. Acknowledge within ten minutes, active investigation within thirty.

Noisy alerts get fixed or deleted. Every alert that wakes someone up without being actionable erodes trust in the system. We review alert volume monthly.

Compensation exists. Time off after a rough night. No one should eat the cost of a 3 AM page out of team loyalty.

The First Five Minutes

When a page fires, the response should be boring. Acknowledge it. Assess severity. Open a dedicated channel. Page the people who need to be there. Start a timeline with what you know, even if what you know is almost nothing.

This sounds obvious. It isn’t. Without a checklist, people skip steps when adrenaline kicks in. They jump straight to debugging and forget to tell anyone what’s happening.

Mitigation Before Root Cause

This one took me too long to learn. When something is broken, your first job is to stop the bleeding. Roll back the deploy. Restart the service. Disable the feature. Redirect traffic. Whatever makes the user impact stop.

Root cause analysis is important. It’s also a luxury you earn after users stop being affected. I’ve watched engineers spend thirty minutes debugging while customers churned. Fix it first, understand it later.

Postmortems That Change Things

We do blameless postmortems for every significant incident. Timeline, impact, root cause, contributing factors, action items. The blameless part matters. If people think they’ll get punished, they’ll hide what actually happened, and your postmortem becomes useless.

What I care about more than the postmortem document: action items that ship. Every action item gets an owner and a deadline. We review them in our regular engineering meetings. An item is closed when the change is deployed, not when someone says they’ll get to it.

I can’t overstate this. If your postmortem action items rot in a backlog, you will have the same incident again. Guaranteed.

Patterns Over Individual Fires

Track your incidents. Frequency by service, time to detect, time to resolve, on-call load per team. Individual incidents teach you something. Patterns teach you where to invest. If one service generates 60% of your pages, you have an engineering priority problem, and no on-call tweak will fix it.

Pick Tools Your Team Will Use

I don’t care which paging system you run. PagerDuty, Opsgenie, whatever. I care that your team actually uses it consistently. Same for your status page, your incident channels, your postmortem docs. The best incident management tool is the one nobody has to be reminded to check.

We keep it simple at the fintech startup. Dedicated Slack channels per incident, a status page for external comms, and a shared doc template for postmortems. Nothing fancy. All of it used every time.

Predictability Is the Goal

Systems break. What separates good teams from drowning teams is predictability in the response: everyone knows their role, everyone knows the severity, communication flows to the right places, and the things you learn actually make it into production.

Every talk I’ve seen on chaos engineering opens with the same slide: Netflix’s Chaos Monkey. And then everyone in the audience nods, thinks “cool, but we’re not Netflix,” and goes back to hoping their services don’t fall over on a Friday afternoon.

I get it. I was there too. At the fintech startup we had a small engineering team, a handful of microservices, and real users depending on real-time financial data. Not exactly the scale where you hire a dedicated Site Reliability team. But that’s precisely why we needed chaos experiments. We couldn’t afford a mystery outage at 2 AM with no runbook and no idea which service just took everything else down with it.

What chaos engineering is

Strip away the branding and chaos engineering is dead simple. You pick something that could break. You guess what will happen when it does. You break it. Then you check if you were right.

That’s the whole thing. Hypothesis, experiment, observation. Science fair stuff.

The part most people skip is the hypothesis. They just kill a pod and watch the fireworks. Without a hypothesis, that’s sabotage. The hypothesis is what makes it useful. “If service A goes down, service B should degrade gracefully and return cached results within 500ms.” Now you have something testable. Now you’ll actually learn something when the answer is no.

Start with a game day before any tool

Forget Chaos Monkey and the tooling around it. At least for now.

What worked for us was embarrassingly low-tech. We’d pick a Thursday afternoon, gather around a whiteboard, and ask: “What happens if Postgres becomes unreachable for 30 seconds?” Then we’d talk through it. Just talk. Who gets paged? What errors do users see? Does the queue back up? Does it recover automatically or does someone need to SSH in and restart something?

Half the time, the answer was “nobody knows.” That’s the finding. That’s the value. You don’t even need to touch a terminal yet.

When you’re ready to actually inject failures, staging is your friend. We ran our first real experiments there. Killed processes, added latency with tc, dropped DNS. Found out our health checks were lying to us . Found out one service had a retry loop that would hammer a dead dependency 200 times a second. In staging. Before any customer saw it.

The failures that teach you the most

Kill a process. See if it restarts. See if anyone gets alerted. You’d be surprised how many teams discover their alerting doesn’t work during their first chaos experiment. We did.

Add network latency. 100ms, then 500ms, then 2 seconds. Watch what happens to upstream callers. Timeouts configured wrong will cascade faster than you can type kubectl get pods.

# Kill a process, see if your system notices
kill -9 $PID

# Add 100ms latency, see who complains
tc qdisc add dev eth0 root netem delay 100ms

Starve a service of memory or CPU. We had a data ingestion job at the fintech startup that would occasionally eat all available memory on the box. We only discovered this pattern because we simulated it first. Fixed it with proper resource limits before it hit production hard.

Cut off a dependency. What does your app do when Redis is gone? If the answer is “500 errors for everyone,” you’ve got work to do. If the answer is “falls back to the database, slightly slower,” you’re in good shape.

The only rule: you must be able to stop it

This is non-negotiable. If you can’t kill the experiment in seconds, don’t run it. Blast radius has to be small. Rollback has to be instant. Everyone on the team has to know the experiment is happening.

We had a simple rule: no experiments without a Slack message first, and no experiments that touch more than one service at a time. Boring, sure. But nobody got surprised, and we never turned a controlled test into a real incident.

Progress looks like: staging first, then one production instance, then a few, then wider. If staging experiments are still surprising you, you’re not ready for production. Stay there. There’s no shame in it.

Make it boring

The goal is for chaos experiments to become routine. Wednesday afternoon, we break something. Thursday morning, we file tickets for what we found. Next week, we verify the fixes. Repeat.

The best outcome is when experiments stop being interesting. When you kill a service and everything just… works. Traffic reroutes. Alerts fire. Dashboards light up. Recovery happens automatically. Boring. Beautiful.

Resilience means knowing exactly what happens when things break. Any team can get there. You just have to be willing to break things on purpose first.

Classified government environments are rigid, process-heavy, approval-driven. Everything gated. Everything slow. For classified systems, that makes sense.

Fintech isn’t classified systems.

At the fintech startup we ship multiple times a day . We handle financial data, user accounts, payment flows. The attack surface is real. But if I made every deploy wait for a manual security review, we’d ship once a week. Maybe. And the devs would hate me. Rightfully so.

So I stopped pretending manual review scales and started building security into the pipeline itself.

The Problem with Manual Gates

A manual security review at a startup doing continuous delivery looks like this: a Slack message saying “hey can you look at this PR?” followed by me context-switching from whatever I’m actually working on, skimming the diff, and approving it because I’ve got six other things on fire.

That’s security theater.

Manual review is slow. It’s inconsistent. I catch different things depending on whether I’ve had coffee. It doesn’t leave an audit trail beyond a thumbs-up emoji. And it absolutely can’t cover every commit when you’re pushing code dozens of times a day.

The goal is to stop wasting security engineers’ time on things a script can catch.

What I Built: Layered Pipeline Checks

Security in a pipeline comes in layers. You want cheap, fast checks early and heavier analysis later. The key is that none of it requires a human to manually trigger.

Pre-commit: catch secrets before they leave the laptop. This is the single highest-value automation I’ve ever set up. One leaked API key in a public repo and you’re having a very bad week. At the fintech startup, dealing with financial APIs, a leaked key could mean real money walking out the door.

#!/bin/sh
## .git/hooks/pre-commit
./scripts/scan-secrets.sh || exit 1

Dead simple. Runs in under a second. Saves you from the kind of incident that makes the news.

PR checks: block the merge if it’s dirty. Static analysis, dependency audit, container scan. All automated. All required to pass. Never allow_failure: true, which turns a gate into a suggestion.

security-checks:
  script:
    - ./scripts/sast.sh
    - ./scripts/dependency-audit.sh
    - ./scripts/container-scan.sh
  allow_failure: false

Cyber-defense exercises drill in the idea of “deny by default.” Same principle here. The merge doesn’t happen unless the checks are green. No exceptions, no “I’ll fix it later” PRs.

Post-merge: the heavy stuff. DAST against staging. Security-focused integration tests. Things like “can an unauthenticated user hit this admin endpoint?” and “are the CORS headers actually set right?” Static analysis can’t catch these. You need a running application.

In production, runtime monitoring and log analysis close the loop. Security becomes a continuous signal instead of a checkbox someone ticked three sprints ago.

Making It Stick

Most teams screw up here: they turn on everything at once, get 400 findings on the first run, and the entire dev team starts ignoring the output within a week.

Don’t do that.

Start with one check. Make it reliable. Make the failure messages useful. Instead of “vulnerability found”, say “this dependency has a known RCE, upgrade to version X, here’s the CVE link.” The difference between a useful pipeline and an annoying one is whether the developer knows what to do when it fails.

At the fintech startup I rolled checks out one at a time over a few months. Secret scanning first. Then dependency auditing. Then SAST. Each one tuned, false positives suppressed, team trained on the output before adding the next. Boring, and effective.

Build a suppression process too. Some findings are false positives. Some are accepted risks. That’s fine. But track the suppressions and review them. A suppression list that only grows is a red flag.

Humans on the Hard Problems

Scanners catch known patterns. SQL injection signatures, outdated libraries, hardcoded credentials. Important stuff. But they can’t think.

Threat modeling is a human job. So is reviewing the authentication architecture of a new feature, and penetration testing that simulates an attacker’s creativity. Incident response when something real happens at 2 AM is very much a human job.

The whole point of automating the repetitive checks is to free up time for this work. In cyber-defense exercises, dedicated teams handle threat analysis without getting bogged down running vulnerability scanners by hand. Same principle at a startup, except the “dedicated team” is me and maybe one other person. Which is exactly why automation matters even more.

The Payoff

Before automation, security at the fintech startup was me trying to review everything and inevitably missing things. After, every single commit gets checked. Every dependency gets audited. Every container gets scanned. And I spend my time on architecture reviews and threat modeling instead of eyeballing diffs.

The pipeline doesn’t get tired. It doesn’t get distracted. It doesn’t approve a PR because it’s Friday afternoon and everyone wants to go home.

It falls apart the second you split into services.

I learned this the hard way. We had decent Grafana dashboards . Reasonable alerts. Then we started breaking things into microservices and deploying multiple times a day. A request would fail somewhere in a chain of five services and our dashboards would just… look fine. Every individual service reported healthy metrics. The problem lived in the gaps between them.

That’s the core issue with monitoring. It answers questions you already thought to ask. Latency on this endpoint? Sure. Error rate on that queue? Got it. But the failure you actually hit in production is the one you never predicted. Your dashboards have no panel for it.

Observability is a different mindset

Observability means you instrument your system so you can ask new questions after something breaks. Not just “is it up” but “why did this specific user’s request take 8 seconds at 3am on Tuesday.”

Three signals make this work: metrics for trends and alerts, logs for detail, and traces for stitching a single request across every service it touches. Separately they are useful. Together they let you follow one request from the edge to the failure.

Structured logging is the foundation. Stop writing free-text log lines. Make every log entry a JSON object with a trace ID, service name, version, and whatever fields you actually need to filter on.

{
  "timestamp": "2017-03-20T10:23:45Z",
  "level": "error",
  "message": "Payment processing failed",
  "trace_id": "abc123",
  "service": "payment-service",
  "version": "2.1.3"
}

Then propagate context. Generate a trace ID at the edge and carry it through every downstream call. This is the single most important thing you can do. Without it you’re just grepping logs and praying.

X-Trace-Id: abc123
X-Span-Id: def456
X-Parent-Span-Id: ghi789

How incidents change

Alert fires. Pull the trace. Follow the slow span to the source. Read the logs for that trace. Confirm with metrics whether it’s one user or everyone. Done. Five minutes instead of forty-five minutes of bouncing between dashboards while your Slack channel fills with “any update?”

Same thing with performance. User says it’s slow. Pull their trace. Compare to a healthy one. See the difference: a cache miss, an extra database round-trip, a third-party call timing out. You fix the actual cause instead of guessing.

The tools don’t matter that much

Prometheus, InfluxDB, ELK , Zipkin: pick whatever fits your stack. Commercial platforms can save time. But the tooling isn’t the hard part. The hard part is disciplined instrumentation. Consistent field names. Trace IDs everywhere. Every team following the same conventions.

No purchase gets you there. You stop staring at dashboards waiting for red and start asking questions about behavior you didn’t expect. When every team ships services independently, that’s the only way I’ve found to stay sane.

Worth it? Yes. Painless? Not even close.

The stuff that pays off immediately

The declarative model is the real win. You describe what you want, Kubernetes converges toward it. No more runbooks for “what if the service dies on box 3.” It just comes back. Deployments get boring, which is exactly what you want deployments to be.

Resource requests and limits are the other quick win, but only once you actually set them properly. We ran for weeks without limits on one of our data ingestion services. It worked great until it didn’t, and then it took down two neighbors on the same node.

Don’t skip this part:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Those numbers are a starting point. Profile under real load. Revisit quarterly. The defaults are lies.

Rolling updates deserve a mention too. Going from “schedule maintenance window, hold breath, deploy, pray” to incremental rollouts with automatic rollback changed how the whole team thinks about shipping. People deploy more often because it feels safe. That feedback loop alone justified the migration.

Where it bites you

Networking. Every single time.

Service discovery looks dead simple in the docs. Then you hit DNS caching issues, or a network policy blocks something you didn’t expect, or your ingress controller does something creative with header forwarding. Kubernetes hides the network behind a nice abstraction until that abstraction leaks, and then you’re reading iptables rules at midnight.

We had a bug where intermittent 503s on one service turned out to be a kube-dns cache TTL mismatch with our upstream provider. Took three days to find. The fix was two lines of config. That’s Kubernetes networking in a nutshell.

Resource tuning is the slow burn. Set requests too low and the scheduler packs your nodes like a clown car, then starts evicting pods when things get tight. Set limits too low and your service gets CPU-throttled under load, which looks exactly like an application bug from the outside. The only way through is profiling under production traffic patterns and adjusting. There’s no shortcut here.

Stateful workloads are possible, but we have kept Postgres and Redis outside the cluster so far. Storage classes, PV lifecycle, backup and restore: it all works, but the operational surface area is large. If you’re just getting started, keep your databases where they are. Move them in later when you actually understand what you’re signing up for.

Debugging gets weird. Something fails and the cause could be the application code, the container image, the scheduler, the network plugin, the node itself, or some combination. I’ve seen a deployment fail because a node had a full disk from container image garbage collection not running. Good luck finding that without decent logging and tracing. kubectl describe is your best friend, but sometimes your best friend doesn’t know the answer either.

The hard lessons

Treating Kubernetes like a black box works until you need to upgrade the cluster. Or recover from an etcd failure. Or figure out why the scheduler won’t place a pod. Someone on your team needs to actually understand the control plane. Not “watched a conference talk” understand. “Can read the API server logs and make sense of them” understand.

Running your own control plane is expensive, and the currency is attention more than money. If your cloud provider offers managed Kubernetes, take the deal. We spent months on control plane operations that could have been spent on product work. Google Container Engine is solid enough now. If your provider has no managed option yet, budget real engineering time for the control plane instead of pretending it’s free.

YAML sprawl is the death of a thousand paper cuts. You start with clean manifests for three services. Six months later you have forty files that are 80% identical and nobody remembers which copy is canonical. Helm helps, but be careful. I’ve seen teams replace YAML sprawl with template sprawl, which is somehow worse because now you need to understand Go templates to deploy a config change.

Secrets handling is genuinely bad out of the box. Kubernetes Secrets are base64-encoded. That’s encoding, not encryption. Anyone with API access can read them. We ended up pulling secrets from Vault at deploy time and treating the built-in Secrets as a transport layer. If you’re storing database passwords as Kubernetes Secrets and calling it done, stop.

Local development is the thing nobody budgets time for. Your cluster isn’t a laptop. Developers need fast feedback without blowing up shared environments. Minikube and isolated namespaces help, but they need investment. We lost a solid week of productivity before we built a decent local dev story.

Habits that actually stuck

Git is the source of truth. Every manifest in version control. Deployments triggered by merges. No kubectl apply from someone’s laptop in production. This gives you an audit trail, makes rollbacks trivial, and kills snowflake configurations dead.

Health checks as production code. A bad liveness probe will kill your service faster than a bad deploy. We had a probe that checked a database connection with a 1-second timeout. Database had a slow moment, probe failed, Kubernetes restarted the pod, pod came up and hit the still-slow database, probe failed again. Restart loop. Cascading failure from a health check. Write your probes carefully.

Kubernetes-aware monitoring. Watching application metrics is necessary but not sufficient. You need to see what the scheduler is doing, how the nodes look, what the control plane health is. Without that, incidents are guesswork. With it, incidents become analysis.

Should you adopt it?

If you have multiple services, deploy frequently, and need real scheduling and resilience, yes. The investment pays off.

If you have a small team, a monolith , or two services behind a load balancer, probably not yet. A simpler orchestrator like Swarm is a gentler start. Kubernetes doesn’t remove operational complexity. It trades one kind for another, and the new kind requires specific expertise. A team that’s struggling with basic deployments won’t be saved by Kubernetes. They’ll just struggle with more abstraction layers.

Where it leaves me

Kubernetes feels less like magic now and more like a system I can reason about. That’s the milestone that matters: I understand why it works, and I understand why it breaks.

The teams I’ve seen succeed treat Kubernetes as a product they operate, with on-call, with runbooks, with regular investment. The teams that struggle treat it like something they installed once and forgot about. Same as any other infrastructure, really. The tool doesn’t care about your intentions. It cares about your configuration.

The problem wasn’t tooling. The problem was noise. So we deleted 42 of those panels. Kept five metrics. Built real alerts around them. Our on-call engineer started sleeping again.

Most monitoring is waste

Here is what I’ve learned running production systems : if you’re watching more than a handful of metrics, you’re watching none of them. Human attention doesn’t scale. Twenty dashboards with eight panels each means you have zero dashboards, because nobody is actually reading them under pressure.

The instinct after an outage is always “we need more visibility.” Wrong. You need better visibility. Those are different things.

Five numbers, nothing else

For a startup backend serving real users, these are the only metrics I care about on a daily basis:

1. Request latency at p95 and p99. Not average. Averages lie. A service averaging 40ms with a p99 at 3 seconds is broken for one request in a hundred, and the user behind it is filing the support ticket. We tracked this per service and alerted on it.

2. Error rate by type. Not just “5xx count.” Break it down. A spike in 401s is a different problem than a spike in 503s. One is probably a bad deploy or a client bug. The other is your database falling over. The distinction matters because the response is completely different.

3. Saturation of your bottleneck resource. For us that was PostgreSQL connection pool utilization. For you it might be memory, disk I/O, or worker threads. The point is: know which resource will kill you first and watch that one. Not all resources. That one.

4. Request throughput. Traffic going up is good. Traffic dropping to zero at 2pm on a Tuesday is very bad. This metric is less about the number and more about detecting sudden changes. A 50% drop in requests is an incident whether or not errors are firing.

5. Deployment markers. Not a metric in the traditional sense, but I overlay every deploy on our Grafana dashboards. Half the incidents I’ve investigated started with “someone shipped something.” Correlating metrics with deploys is the fastest diagnostic shortcut I know.

That’s it. Five things.

Alerts should wake you up for a reason

We had a rule: if an alert fires and the on-call engineer can’t take a meaningful action within five minutes, delete the alert. Brief CPU spikes? Not actionable. Disk at 80%? Only actionable if there’s a runbook. “Something might be wrong” isn’t an alert. It’s anxiety.

Our paging alerts were:

• p99 latency above threshold for 5 minutes
• Error rate above 1% for 3 minutes
• Primary database connection pool above 90%

Three alerts. That was it for paging. Everything else went to a Slack channel that people checked during business hours.

The discipline is in what you remove

After I stripped our monitoring down, the team pushed back. “What if we miss something?” My answer: we were already missing things. Forty-seven panels meant nobody looked at any of them carefully. Five panels meant the on-call engineer actually understood the state of the system at a glance.

Monitoring isn’t a collection problem. It’s an attention problem. Pick the five metrics that reflect what your users experience, alert only where someone can act, and delete everything else.

The evaluation

At the mobility startup we run location-tracking services, payment processing, and a fleet management backend. All containerized. When we outgrew single-host Docker Compose , I needed an orchestrator that could schedule across hosts, handle rolling deploys, and not require a dedicated platform team to babysit.

I spent three weeks running Swarm, Kubernetes, and Mesos with Marathon side by side on identical three-node clusters. Same workload: our core API, a background worker, and a Redis instance. Here is what I found.

The comparison

My take

Kubernetes is going to win this market. The abstraction model is right. Pods, services, and declarative desired-state are the correct primitives for running containers at scale. The community momentum is unmistakable. Every infrastructure vendor is building Kubernetes integrations, not Swarm or Mesos ones.

But the operational cost in late 2016 is genuinely high. I burned two full days debugging etcd quorum problems on a three-node cluster. Networking with flannel required reading source code, not documentation. RBAC is immature. Monitoring the control plane itself is a project unto itself.

Swarm is the practical answer for small teams today. We used it at the mobility startup before migrating to Kubernetes, and that stretch was productive. The API is Docker’s API. The mental model is simple. You can ship with it and migrate later without regret.

Mesos is the right choice if you already run it or if you genuinely need to schedule containers alongside Spark jobs and batch workloads on the same cluster. For a startup running pure container workloads, Mesos is overbuilt. The ZooKeeper dependency alone adds operational surface area that a small team doesn’t need.

What we actually did

We started on Swarm and shipped to production on it. When we hit the limits of Swarm’s rolling deploy controls and needed better service discovery for inter-service communication, we migrated to Kubernetes. The migration was painful but contained. It took about a week of focused work.

I would make the same decision again. Swarm bought us time to ship features instead of fighting infrastructure. Kubernetes gave us a platform we could grow into. Trying to start with Kubernetes on day one with our team size would have been a mistake.

Pick for your team, not the hype

Pick based on your team, not the feature matrix. Swarm if you need to ship now with minimal operational burden. Kubernetes if you can absorb the learning curve and want the platform that will matter in two years. Mesos if you’re already running it or have workloads that genuinely require its scheduling model.

Kubernetes is going to dominate. That doesn’t mean you have to adopt it today.

Security culture isn’t a project

I’ve watched teams try to buy their way into security. They purchase a scanner, run a penetration test once a year, and call it done. Then a junior engineer hardcodes an API key, pushes it to a public repo, and suddenly the scanner doesn’t matter.

Security culture is the habit that prevents that push in the first place.

One team I worked with handled sensitive user data, including payment information. There’s no version of “we’ll fix it later” that regulators or users will accept. Security had to be foundational from the start, not bolted on after the first scare.

What cyber-defense exercises taught me

A principle I carry from cyber-defense exercises: the biggest lesson was organizational. The teams that performed well were the ones where every person understood the threat model and acted on it without waiting for permission. Tooling mattered less than that.

That stuck with me. Security at scale is a culture problem first.

The policies I actually enforce

Theory is cheap. Here are the specific rules I’ve implemented across teams.

No secrets in code. Ever. We use environment variables and a secrets manager. The CI pipeline fails if it detects anything that looks like a key or credential in the codebase. This is automated, not optional, and not overridable without a written justification that I personally review.

Every pull request gets a security question. Not a full threat model. Just one question: “What is the worst thing that happens if this code is exploited?” If the author can’t answer it, the PR doesn’t merge. This forces engineers to think about attack surface as part of their daily work, not as a separate exercise.

Least privilege by default. New services start with zero permissions and add only what they need. New employees get read access to the repositories they work on and nothing else. Escalation requires a request and a reason. I review access quarterly and revoke anything that isn’t actively justified.

Dependency updates aren’t optional. We track dependencies weekly. Known vulnerabilities get patched within 48 hours for critical severity, one week for high. This is a policy, not a suggestion. I’ve delayed feature work to meet these windows.

Incident response is rehearsed. We run a tabletop exercise every quarter. Not a checkbox drill. A real scenario where I throw a curveball halfway through to see how the team adapts. The ones who have done this three or four times respond to real incidents with calm instead of panic.

The hard part is consistency

Any team can write a security policy document. The hard part is enforcing it on the days when you’re behind on a deadline and the shortcut is right there.

I’ve blocked releases. I’ve told product managers that a feature would ship a week late because the authentication flow wasn’t reviewed. Those conversations are uncomfortable. They are also the moments that define whether your culture is real or performative.

Engineers watch what leadership does, not what leadership says. If the CTO merges a PR that skips the security review because “we need to ship,” every engineer on the team learns that security is negotiable. It takes one exception to undo months of habit building.

Security champions scale your coverage

I can’t review every line of code. No security team can. The answer isn’t to hire more security people. It’s to make every engineer a little bit dangerous.

The model I like: one engineer on each team acts as a security champion. Not a full-time role. Maybe 10-15% of their time. They attend a monthly session to review recent vulnerabilities in the stack, discuss attack patterns relevant to the business, and update the threat model.

These champions become the first line of defense in code reviews. They catch the obvious issues before they reach me, and they raise the questions that nobody else on the team would think to ask.

The key is that being a security champion is respected. It counts in performance reviews. It isn’t extra work on top of their real job. It’s part of their real job.

Make the secure path the easy path

If doing the right thing is harder than doing the wrong thing, people will do the wrong thing. That’s ordinary human behavior.

We built internal libraries that handle authentication, input validation, and encryption correctly. When an engineer needs to call an external API, they use our wrapper that handles TLS verification, credential injection, and audit logging automatically. The secure path is also the path with less code to write.

We templated our infrastructure so that new services deploy with network isolation, encrypted storage, and logging enabled by default. An engineer has to go out of their way to deploy something insecure. That friction is intentional.

Incidents are data, not blame

When something goes wrong, and it will, the response determines whether your culture strengthens or collapses.

We run blameless postmortems , because blame destroys information flow. If engineers fear punishment, they hide mistakes. Hidden mistakes compound. The breach that takes down a company is rarely the first failure. It’s the tenth failure that nobody reported because the first person who reported one got burned.

Every postmortem produces exactly two outputs: a timeline of what happened, and a list of specific changes to prevent recurrence. Not “be more careful.” Specific changes. A new automated check. A revised permission. A policy update.

Security is a daily decision

Security culture is discipline. It’s the same decision made correctly a thousand times, even when it’s inconvenient. Especially when it’s inconvenient.

You build it by enforcing the standards you set, including on yourself.

So we set up ELK . Elasticsearch, Logstash, Kibana. The pitch was compelling: open source, flexible, great full-text search, a plugin for everything. We stood up a three-node cluster, pointed Logstash at it, gave the team Kibana dashboards. For the first few weeks, it felt like a superpower.

Then the cluster started misbehaving. And I spent the next several months learning a painful lesson about the gap between “powerful” and “worth the operational cost.”

If you don’t have someone who wants to babysit Elasticsearch full time, don’t run ELK yourself. Hosted Elasticsearch, Graylog, or even Splunk will save you more engineering hours than they cost.

The Operational Tax Nobody Warns You About

Elasticsearch isn’t a database you deploy and forget. It’s a distributed system that demands constant attention. Shard allocation, index rotation and retention, JVM heap tuning, split-brain prevention, disk watermarks. Every one of these will bite you, and they will bite you at 3 AM.

At the mobility startup, our log volume was moderate. Maybe a few gigabytes a day. Nothing that should stress a three-node cluster. But Elasticsearch doesn’t care about your expectations. It cares about index design, merge policies, and whether you remembered to set bootstrap.mlockall. We spent more time keeping the logging infrastructure healthy than we spent on some of our actual product services.

The worst part is that when your logging system goes down, you lose visibility into everything else at the same time. Your safety net disappears exactly when you need it most.

What ELK Actually Gets Right

I’m not going to pretend it’s all bad. Elasticsearch’s search is genuinely excellent. When the cluster is healthy, the ability to run arbitrary queries across millions of log lines with sub-second response times is hard to match. Logstash can parse almost any log format into structured fields. Kibana dashboards gave our product team visibility they never had before.

The ecosystem is real. Beats shippers are lightweight, community plugins cover most integrations, and the documentation is solid. If you have the operational muscle, ELK is the most flexible open source logging stack available.

What I’d Do Differently

If I were starting over, I wouldn’t self-host Elasticsearch for logging. Full stop.

Hosted Elasticsearch removes the worst of the operational burden. You keep the same query model, the same Kibana dashboards, the same integrations. You lose some control over cluster configuration and you pay more per gigabyte, but you gain back the engineering hours you were burning on cluster babysitting. For most teams, that tradeoff is obvious.

Graylog is worth a look if you want Elasticsearch search without the full DIY build. It wraps Elasticsearch in a more opinionated log management experience with built-in alerting and stream routing. Less flexible than raw ELK, but faster to get running and easier to keep running.

Splunk is the enterprise answer. Powerful, mature, battle-tested. Also expensive enough to make your finance team flinch. If budget isn’t the constraint, Splunk is a safe bet. For a startup, it rarely makes sense.

Cloud provider logging is the lowest-effort option. AWS CloudWatch Logs, for example, integrates deeply with everything else in AWS and requires zero operational overhead. The query capabilities are basic compared to Elasticsearch, but basic is often enough. You can always export to something more powerful later.

Structured Logging Matters More Than the Tool

Regardless of which aggregation tool you pick, the single best investment is structured logging. A JSON log line with consistent fields turns debugging from archaeology into search.

{
  "timestamp": "2016-09-05T10:23:45Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123",
  "message": "Payment processing failed",
  "error": "Connection timeout",
  "customer_id": "cust_456"
}

Carry a trace ID through every service. Use consistent field names. Emit JSON instead of free-form text. These decisions pay off no matter what sits behind the ingestion pipeline. If you do nothing else, do this.

Put a Buffer in Front of Ingestion

One more thing I learned the hard way: put a queue between your log shippers and your aggregation layer. Kafka, Redis, even a simple file buffer. Traffic spikes will happen. Deploys will happen. If your pipeline has no buffer, you drop logs during the exact moments you need them most.

Pick your logging battles

ELK is powerful software with brutal operational costs. Most teams underestimate how much work Elasticsearch is to run, and they find out at the worst possible time. If you can afford someone who genuinely enjoys tuning JVM garbage collection and shard allocation, go for it. Otherwise, pay for a hosted solution or pick a simpler tool.

The problem with “just run the migration”

One system I worked on was a high-volume data API. Its clients didn’t pause because the engineering team needed a maintenance window, and a feed that goes dark is a feed that loses subscribers. When I took over database operations there, the existing pattern was to schedule migrations at 2 AM on Sundays, take the API offline, run the DDL, and hope the application came back cleanly. It worked until it didn’t.

The breaking point was a column type change on a table with 40 million rows of historical data. The migration ran for 47 minutes. During that time the table was locked, the API returned errors, and a downstream consumer silently switched to a stale cache it never recovered from. We spent the next two days cleaning up data consistency issues.

After that I decided every migration would be zero-downtime or it wouldn’t ship.

The expand-and-contract pattern

The core idea is simple. Instead of making a breaking change in one step, you split it into phases where the schema is always compatible with whatever application code is currently running.

Expand: add the new structure alongside the old one. Both coexist.

Migrate: deploy code that writes to both old and new, reads from new. Backfill historical data.

Contract: once every running instance uses the new structure, remove the old one.

This pattern handles nearly every schema change. The details vary, but the rhythm stays the same.

Adding a column safely

The simplest case. A nullable column with no default doesn’t rewrite the table in PostgreSQL. It’s metadata-only and takes a lock for milliseconds.

ALTER TABLE orders ADD COLUMN processed_date DATE;

Fast. Safe. Old code ignores the column. New code starts writing to it.

If you need a NOT NULL constraint, don’t add it in the same statement. PostgreSQL will scan the entire table to verify the constraint, holding an ACCESS EXCLUSIVE lock the whole time. Instead:

-- Step 1: add nullable column
ALTER TABLE orders ADD COLUMN processed_date DATE;

-- Step 2: backfill in batches (see below)

-- Step 3: add constraint without full table scan
ALTER TABLE orders ADD CONSTRAINT orders_processed_not_null
  CHECK (processed_date IS NOT NULL) NOT VALID;

-- Step 4: validate separately (only takes a SHARE UPDATE EXCLUSIVE lock)
ALTER TABLE orders VALIDATE CONSTRAINT orders_processed_not_null;

The NOT VALID trick is critical on large tables. It tells PostgreSQL to enforce the constraint on new writes immediately but skip the full table scan. The VALIDATE step runs later with a weaker lock that doesn’t block writes. On our 40-million-row tables, this was the difference between a 200-millisecond migration and a 20-minute outage.

Renaming a column

You don’t rename columns in production. Full stop.

What you actually do is expand-and-contract. Here is the concrete sequence, shown on an example orders table where price becomes unit_price:

-- Phase 1: Expand
ALTER TABLE orders ADD COLUMN unit_price NUMERIC(18,8);

Deploy code that writes to both price and unit_price. Reads come from unit_price with a fallback to price.

-- Phase 2: Backfill
UPDATE orders SET unit_price = price
WHERE unit_price IS NULL
AND id BETWEEN 1 AND 100000;

-- Repeat in batches...
UPDATE orders SET unit_price = price
WHERE unit_price IS NULL
AND id BETWEEN 100001 AND 200000;

Once backfill is complete and every application instance reads from unit_price:

-- Phase 3: Contract
ALTER TABLE orders DROP COLUMN price;

Three deployments minimum. That’s the cost. The benefit is zero downtime and a clean rollback at every step.

Changing a column type

Same pattern, different details. We had a case where an external identifier was stored as INTEGER but needed to become TEXT to support a new ID format.

-- Expand
ALTER TABLE products ADD COLUMN external_id_new TEXT;

-- Backfill
UPDATE products SET external_id_new = external_id::TEXT
WHERE external_id_new IS NULL
AND id BETWEEN 1 AND 50000;

Deploy dual-write code. Cut over reads. Drop the old column.

The temptation is to use ALTER COLUMN ... TYPE TEXT which rewrites the entire table under an exclusive lock. On a table with millions of rows that’s actively being queried, that isn’t an option.

Batched backfills that don’t kill the database

Large backfills are where most zero-downtime migrations go wrong. A single UPDATE ... WHERE condition on 40 million rows will generate enormous WAL, bloat the table, and hold locks that block concurrent operations.

The pattern I use:

#!/bin/bash
set -e
batch_size=5000
max_id=$(psql -Atc "SELECT COALESCE(MAX(id), 0) FROM orders")

for ((current_id = 0; current_id < max_id; current_id += batch_size)); do
  psql -c "UPDATE orders
           SET unit_price = price
           WHERE id > $current_id
             AND id <= $current_id + $batch_size
             AND unit_price IS NULL"
  sleep 0.1  # breathe
done

Each psql call is its own transaction, so every batch commits and releases its locks before the next one starts. A DO block can’t do that: it runs as a single transaction from start to finish.

The pause matters. Without it, you saturate I/O and replication lag spikes. On that system our replicas served read traffic for dashboards and analytics. A backfill that caused 30 seconds of replication lag would show stale data to every user watching a dashboard. We settled on batches of 5,000 rows with 100ms pauses. The backfill took longer but production stayed healthy.

Indexes without locking

CREATE INDEX on PostgreSQL takes a SHARE lock on the table, which blocks writes for the duration of the build. On a large table, that can be minutes.

CREATE INDEX CONCURRENTLY idx_orders_processed ON orders(processed_date);

CONCURRENTLY builds the index without blocking writes. It takes longer and does two table scans instead of one, but it’s the only option for production tables. There are two caveats worth knowing:

First, it can’t run inside a transaction. If your migration tool wraps everything in a transaction, you need to handle this case separately.

Second, if it fails partway through, it leaves an invalid index behind. Find it in pg_index and drop it before retrying.

SELECT indexrelid::regclass, indisvalid
FROM pg_index
WHERE NOT indisvalid;

Foreign keys without blocking

Adding a foreign key constraint with ADD CONSTRAINT ... FOREIGN KEY scans the whole table while holding a SHARE ROW EXCLUSIVE lock on both tables, which blocks writes for the duration. Same trick as NOT NULL:

ALTER TABLE orders
  ADD CONSTRAINT fk_orders_product
  FOREIGN KEY (product_id) REFERENCES products(id)
  NOT VALID;

ALTER TABLE orders VALIDATE CONSTRAINT fk_orders_product;

The first statement adds the constraint for new writes only, taking a brief lock. The second validates existing rows with a much weaker lock.

The deployment sequence

The order matters. Getting it wrong means you have a window where the application and schema disagree.

1. Run expand migration. New columns, new indexes (concurrently), new constraints (NOT VALID). Schema is now compatible with both old and new code.

2. Deploy new application code. Writes to both old and new columns. Reads from new with fallback to old.

3. Run backfill. In batches, during low traffic if possible. Monitor replication lag and query latency.

4. Verify completeness. SELECT COUNT(*) FROM orders WHERE unit_price IS NULL should be zero.

5. Deploy read-cutover code. Reads exclusively from new columns. Still writes to both.

6. Run contract migration. Drop old columns, drop temporary constraints. Schema is clean.

Each step is independently reversible. If the backfill causes problems, pause it. If the new code has bugs, roll back the deployment. The old schema is still there. This is the entire point.

What this costs

More deployments. More code that handles two schemas simultaneously. Migrations that used to be one PR become three or four. Application code carries temporary dual-write logic that gets cleaned up in the contract phase.

It’s more work. But on that system, the alternative was telling API consumers that we needed a maintenance window during their business hours. That conversation never goes well.

The discipline pays off in a different way too. When every migration follows expand-and-contract, deployments become boring . Nobody pages you at 2 AM for a schema change. Nobody holds their breath during a release. The process is mechanical and predictable.

The best migration is the one nobody notices.

The Night That Changed My Mind

I got paged at 1 AM because an EC2 instance hosting our vehicle-availability API had died. No big deal, I thought. Spin up a new one. Except nobody could tell me the exact instance type, the security group rules, or the IAM role attached to it. The wiki said one thing. The console said another. The CloudFormation stack that supposedly managed it had drifted so far from reality that applying it would have torn down half the VPC.

We recovered in about four hours. Four hours for one instance. That was the moment I decided our infrastructure couldn’t live in people’s heads anymore.

What We Were Running Before

The mobility startup’s AWS setup was a patchwork. Some resources lived in a CloudFormation stack written by a contractor who had since left. Other pieces were created by hand in the console during late-night debugging sessions and never documented. A handful of bash scripts in a repo called infra-scripts handled things like spinning up staging environments, but they were brittle and nobody trusted them enough to run without reading every line first.

CloudFormation itself wasn’t the problem exactly. The problem was that we treated it as a one-time setup tool instead of a living definition. We would write a template, deploy it, then immediately start making manual changes on top. Within weeks the template was fiction.

Why Terraform Won

I evaluated three options: doubling down on CloudFormation, adopting Ansible for infrastructure provisioning, or trying Terraform. Terraform was still young in mid-2016, somewhere around version 0.6, but it had two properties that mattered to me.

First, the declarative model. You describe what you want, not the steps to get there. After years of imperative scripts where ordering and idempotency were my problem, this was a relief. I could define a VPC, subnets, security groups, and instances in HCL and let Terraform figure out the dependency graph.

Second, the plan step. Before Terraform touches anything, it shows you exactly what it intends to do. Create this, modify that, destroy the other. At the mobility startup we were a small team and we couldn’t afford surprises. The plan became our pre-flight checklist.

resource "aws_instance" "vehicle_api" {
  ami           = "ami-0a2e1043"
  instance_type = "t2.small"
  subnet_id     = "${aws_subnet.private.id}"

  vpc_security_group_ids = [
    "${aws_security_group.api.id}",
  ]

  iam_instance_profile = "${aws_iam_instance_profile.vehicle_api.name}"

  tags {
    Name        = "vehicle-api-${var.environment}"
    Service     = "vehicle-availability"
    Environment = "${var.environment}"
    ManagedBy   = "terraform"
  }
}

That snippet is close to our actual vehicle-availability API definition. Nothing clever. Readable. Reviewable. If someone asks how production is configured, the answer is in Git.

The Migration

I didn’t attempt a big bang. That would have been reckless with a small engineering team and live users on the platform.

Week one: all new infrastructure goes into Terraform. No exceptions. If you need a new security group, you write HCL.

Week two through four: I started bringing existing resources under Terraform. Terraform 0.6 has no import command, so this is the tedious part: write the HCL for a resource, get it into state, run plan, and iterate until plan shows no changes. For our setup it meant adopting roughly forty resources across the VPC, subnets, route tables, security groups, instances, RDS, and an Elastic Load Balancer.

I did this environment by environment. Development first, where mistakes were cheap. Then staging. Production came last, after I had already made every dumb mistake in a place where it didn’t matter. By the time I brought the production VPC under Terraform, the process was mechanical.

Modules helped keep things sane. I built a small VPC module and a service module. Every environment became a composition of those two building blocks with different variables. The upfront effort paid back immediately in consistency. Staging actually looked like production for the first time.

State: The Sharp Edge

Terraform’s state file is where it tracks the mapping between your HCL and real AWS resources. It’s powerful and it’s dangerous. Early on I kept state in a local file and committed it to Git. That worked fine for a solo operator but fell apart as soon as a second engineer touched the repo.

We moved state to S3 remote state within the first month. Terraform has no state locking yet, so two people applying at once is a real risk: concurrent applies can corrupt state and leave you in a worse position than having no IaC at all. The state file also contains secrets in plain text, things like RDS passwords and API keys. Encryption at rest on the S3 bucket and tight IAM policies aren’t optional. We treat state like production credentials because that’s what it contains.

What Changed Operationally

The biggest shift was cultural. Before Terraform, infrastructure changes happened whenever someone had console access and a reason. After Terraform, every change went through a pull request. Run terraform plan, paste the output into the PR, get a review, merge, apply. Simple process. Hard to skip.

We caught real problems this way. A plan once showed that changing a subnet CIDR would destroy and recreate every instance in the subnet. That would have been a fifteen-minute outage discovered at 2 AM instead of during a calm code review on a Tuesday afternoon.

Recovery got faster too. When that same vehicle-availability API instance died again two months later, rebuilding it was a terraform apply away. Five minutes instead of four hours. That alone justified the migration.

Trade-offs

Terraform isn’t free. The learning curve is real, especially around state management and provider quirks. HCL is readable but limited. Complex logic feels awkward compared to a real programming language. And in mid-2016, the provider ecosystem still has gaps. We hit bugs. We read source code. We worked around things.

But the alternative was worse. The alternative was guessing at 1 AM.

Where We Stand

Every piece of the mobility startup’s AWS infrastructure is now defined in Terraform and stored in version control. Environments are reproducible. Changes are reviewed. Recovery is predictable. The migration took about six weeks of focused work alongside normal feature development.

If your infrastructure still lives in consoles and half-remembered scripts, make the move. Terraform isn’t perfect, but the cost of not knowing what you’re running is higher than you think. You will find out exactly how high at 1 AM on a Tuesday.

Deploying Many Times a Day

At a mobility startup we deploy our backend services multiple times a day. Not because we decided to be trendy. Because our business requires it. When you’re running a shared mobility platform, a broken deploy means vehicles that don’t unlock, payments that don’t process, and users who open a competitor’s app. The cost of a bad release is immediate and measurable.

That urgency forced us to take continuous deployment seriously much earlier than I expected. We were a small team, moving fast, and the only way to ship that fast without constant fires was to build discipline into the pipeline itself.

CD Is a Discipline Problem

Most teams that fail at continuous deployment fail because they treat it as a tooling problem. They install Jenkins, wire up a webhook, and wonder why production keeps breaking on Friday afternoon.

The tooling is the easy part. Jenkins works. GitLab CI is getting better by the month. The hard part is the set of habits that make frequent deploys safe. Habits like writing tests that actually catch regressions. Habits like reviewing changes with deployment risk in mind. Habits like keeping every change small enough to reason about when something goes wrong at midnight.

I’ve watched teams with excellent pipelines ship broken code because nobody enforced the discipline around the pipeline. And I’ve watched teams with mediocre tooling ship reliably because their habits were strong. Discipline beats tooling every time.

The Pipeline We Actually Run

Our Jenkins setup is straightforward. On every push, it runs the unit tests. If those pass, it builds the Docker image, tags it with the commit SHA , and pushes it to our private registry. Integration tests run against that image. If everything is green, it deploys to staging automatically. Production deploys happen after a one-click approval (strictly speaking, that makes our production step continuous delivery), and the click takes about ten seconds because the person approving has already seen the diff, the test results, and the staging behavior.

The whole cycle from push to production takes under fifteen minutes on a good day. That speed matters. When the feedback loop is short, developers catch their own mistakes before context switches away. When the loop is long, commits pile up, responsibility diffuses, and nobody knows which change caused the problem.

We also run a nightly full regression suite that catches the slower, more expensive test cases. Those don’t block individual deploys, but a red nightly build stops all production deploys the next morning until someone fixes it. No exceptions.

Tests You Can Trust

A continuous deployment pipeline is only as strong as the tests it runs. Flaky tests are poison. A test that fails randomly teaches the team to ignore failures. Once that habit forms, real failures get waved through too.

We spent weeks stabilizing our test suite before we trusted it to gate production deploys. That meant removing tests that depended on timing, isolating tests that shared state, and replacing slow end-to-end tests with faster contract tests where possible. It was unglamorous work. It was also the single highest-leverage investment we made in our deployment process.

The rule is simple: if a test fails, it means something is wrong with the code, not with the test. Any test that violates that rule gets fixed or deleted. There’s no middle ground.

Small Changes, Fast Rollback

Every change we deploy is small. Not because we have a policy document that says so. Because small changes are easier to review, easier to test, and critically easier to roll back.

Rollback isn’t an afterthought in our process. It’s a first-class operation. We keep the previous three Docker images tagged and ready. Rolling back means pointing the load balancer at the previous image. It takes less than a minute. We practice it regularly, not just when things are on fire.

The worst continuous deployment failures I’ve seen happened because rollback was theoretical. Someone wrote a wiki page describing the rollback steps, nobody ever tested them, and when the moment came, the steps didn’t work. A rollback plan that hasn’t been exercised isn’t a plan. It’s a wish.

Monitoring That Closes the Loop

Deploying fast without watching the result is just shipping bugs faster. After every production deploy, we watch error rates, response latency, and a handful of business metrics for at least ten minutes. If anything moves in the wrong direction, we roll back first and investigate second.

This is where most teams cut corners. They deploy, see green in the pipeline, and move on. But the pipeline only tells you the tests passed. It doesn’t tell you that the new code path is three times slower under real traffic, or that an edge case in the mobile client is now returning 500s.

We use basic Grafana dashboards with alerts that fire if error rate or p99 latency crosses a threshold within the first fifteen minutes after deploy. Nothing sophisticated. Just enough signal to catch the obvious problems before users do.

When CD Doesn’t Fit

I’m not a purist about this. Some systems shouldn’t deploy on every green build. Anything that touches payment processing gets an extra review gate. Schema migrations go through a separate, more careful process. And we don’t pretend that mobile app releases can follow the same cadence as backend services when the App Store review cycle exists.

The point of continuous deployment isn’t to deploy everything all the time. It’s to make deployment a non-event for the systems where speed matters. For everything else, continuous delivery, where the artifact is always ready but the final step is manual, is perfectly fine.

Ship the habits first

After months of running this process, what matters most is the team’s relationship with production. When developers own the deploy and own the monitoring , they write smaller changes and better tests, and they think about rollback before they write the feature.

If you want continuous deployment, start with discipline. Make your tests trustworthy. Make your changes small. Make rollback fast and practiced. Make monitoring non-negotiable. The tooling will follow. Jenkins, GitLab CI, or whatever comes next won’t matter if the habits aren’t there first.

I pulled up the logs and saw authenticated requests hitting an internal endpoint we had deprecated months ago but never torn down. The requests were coming from a session token that belonged to an engineer who had left the company six weeks earlier. His credentials should have been revoked. They weren’t. Somewhere between the offboarding checklist and our actual IAM setup, we had a gap.

The next four hours taught me more about incident response than any exercise or drill ever could. Not because the technical problem was hard. It wasn’t. The deprecated endpoint was read-only, the data it exposed was limited, and the token hadn’t been used for anything beyond what looked like automated probing. But the organizational response was chaos. Nobody knew who was supposed to make decisions. Our CEO was calling me every ten minutes asking for updates I didn’t have. One engineer started revoking tokens across the board, which knocked real users offline. Another started restoring from backups before we understood what had actually happened.

We contained it within a few hours. The blast radius was small. But the recovery took three days because of the collateral damage from our own panic. The token revocation took down a payment processing integration. The premature backup restore overwrote six hours of legitimate transaction data. We spent more time cleaning up our response than we spent on the actual incident.

That night changed how I think about security incidents. The technical compromise was a footnote. The damage came from not having a plan: people acting fast without acting together.

The lesson I took from it: the plan doesn’t need to be sophisticated. It needs to exist, it needs to be practiced, and it needs to be clear about who does what. That’s the bar. Most startups don’t clear it.

Here is what I’ve learned.

The line between event and incident

Draw it before you need it. A security event is suspicious activity that might be nothing. A security incident is confirmed or strongly suspected unauthorized access that demands immediate action.

When you’re unsure, treat it as an incident and downgrade later. This is a lesson reinforced in cyber-defense exercises, and it applies just as much to a twelve-person startup. You can walk back an overreaction. You can’t walk back a delayed containment that let an attacker pivot to your production database.

The distinction matters because it determines who gets woken up and how fast. If everything is an incident, people stop responding. If nothing is, you miss the real ones.

Ownership before the alarm goes off

At our mobility startup, the problem wasn’t that nobody cared. Everyone cared. That was the problem. Five people all making independent decisions in a crisis is worse than one person making imperfect decisions with authority.

You need three roles defined ahead of time:

Incident commander. This person makes decisions and keeps the response coordinated. They don’t need to be the most technical person. They need to be calm, organized, and empowered to say “stop” when someone is about to make things worse.

Technical lead. This person drives the investigation and remediation. They decide what logs to pull, what systems to isolate, and what the containment strategy looks like.

Communications lead. This person keeps internal and external messaging accurate. At a five-person startup, this might be the CEO. At a fifty-person company, it might be someone in ops or legal.

At a small startup, one person wears multiple hats. Fine. But the ownership has to be explicit. Write it down. Put it in the wiki. And for the love of everything, make sure the contact information works at 11 PM on a Tuesday. Test it quarterly. Phone numbers change. People leave. On-call lists rot faster than you think.

Severity in plain language

Fancy severity matrices are a waste of time at a startup. You need four levels that everyone understands without looking anything up:

Critical. Active data exfiltration or production systems are compromised. Drop everything.

High. Confirmed unauthorized access, scope unclear. Assemble the response team now.

Medium. Suspected compromise, limited scope. Investigate within hours, not days.

Low. Suspicious activity that needs a closer look. Triage it during business hours.

The most important thing about your severity levels is that people actually use them. If your incident commander has to consult a decision tree to figure out whether an incident is a P1 or a P2, you have failed.

Containment is where most startups break

Containment is the step where panic does the most damage. The goal is to stop the bleeding without cutting off the patient’s blood supply.

Revoke compromised credentials immediately. Isolate affected systems from the network, but preserve evidence while you do it. Cut off active data exfiltration paths. If malware is involved, isolate the system and keep the sample for analysis.

Every containment action has a tradeoff. Revoking all tokens is safe but might take down production. Isolating a server preserves evidence but removes capacity. These are real decisions with real business impact. Make them intentionally, communicate the impact, and document why you chose what you chose.

The engineer at my startup who revoked every token in the system wasn’t wrong to act fast. He was wrong to act alone, without telling anyone, during an active incident. By the time I realized what had happened, our payment integration was down and we were fielding angry support tickets alongside the security investigation. Two fires instead of one.

Investigation: slow down before you speed up

Once the immediate bleeding stops, resist the urge to fix everything at once. Investigation is about building a timeline and understanding the blast radius.

Pull logs across authentication, application, and network layers. Preserve forensic images before you start making changes. Build a timeline of attacker activity. Collect indicators of compromise so you can sweep for related access you might have missed.

Don’t rush to restore service before you understand how the attacker got in. I’ve seen startups bring compromised systems back online with the same vulnerability still open because they were in a hurry to get back to normal. That’s a second incident waiting to happen.

Eradication and recovery

Eradication means removing every foothold. Close the vulnerability that allowed the initial access. Remove any malware or backdoors. Reset credentials that could have been compromised. Verify that no unauthorized accounts were created.

Attackers who know what they are doing leave multiple persistence mechanisms. A revoked token isn’t enough if they also dropped an SSH key or created a service account. Validate your cleanup thoroughly.

Recovery means rebuilding from known-good sources when possible. Restore data from clean backups if needed. Bring systems back gradually and watch closely for signs of re-entry. The monitoring should be tighter after an incident, not looser.

Communication during a crisis

Internal communication should be regular, calm, and honest. Keep leadership aligned on severity and business risk. Inform affected teams. Separate confirmed facts from open questions in every update. Nothing erodes trust faster than a retraction.

External communication should be coordinated with legal counsel, especially if customer data is involved. Move quickly, but don’t speculate. Explain what happened, what data was affected, what you’re doing about it, and what customers should do. If you’re in fintech or health, map your regulatory notification requirements now. Not during the incident. Now.

At the shared mobility startup, we got this part right almost by accident. Our CEO was a former lawyer and insisted on reviewing every external statement. It slowed us down by an hour. It saved us from saying something incorrect that would have required a correction, which always looks worse than a slight delay.

Documentation as you go

Write it down in real time. Decisions, actions, evidence, timestamps. You will need this record for compliance, for legal protection, and for learning. It also prevents confusion when responders rotate or when you need to brief someone new.

Use a shared document. A Google Doc is fine. A dedicated incident management tool is better. The format matters less than the habit. If you aren’t documenting during the incident, you won’t remember accurately afterward. Memory under stress is unreliable.

Preparation that costs almost nothing

Know your critical assets. What systems hold sensitive data? What access paths could cause maximum damage? If you can’t answer these questions in under a minute, you aren’t ready.

Build detection that actually works. Centralized logs . Alerts on unusual patterns. At minimum, you should know when someone authenticates from a new location or when an API sees traffic outside normal bounds. This doesn’t require expensive tooling. It requires attention.

Write draft communication templates. Under pressure, writing coherent messages is harder than you think. Have templates for internal updates, customer notifications, and regulatory disclosures. Fill in the specifics during the incident, but don’t start from a blank page.

Run tabletop exercises. Pick a plausible scenario: stolen laptop, exposed database, compromised former employee credentials. Walk through the response as a team. These discussions take an hour and reveal gaps that would cost you days during a real incident.

The former employee scenario isn’t hypothetical. It’s what hit us. If we had run that exercise once, someone would have asked whether our offboarding process actually revoked all access. We would have checked. We would have found the gap. The incident would never have happened.

Cheap insurance

Incident response planning is the cheapest insurance a startup can buy. A simple playbook, three named roles, and one tabletop exercise per quarter will put you ahead of most companies your size.

You will still have incidents. The goal is to respond with discipline instead of panic, contain the damage instead of amplifying it, and come out the other side with a team that trusts each other more.

Puppet

Puppet’s declarative DSL is solid. The agent-server model gives you continuous enforcement, which compliance-heavy orgs love. But you’re running agent infrastructure on every node, learning a custom DSL that isn’t quite a programming language, and debugging catalog compilation errors that give you nothing useful.

For a 500-node fleet with a dedicated infra team and compliance requirements? Puppet earns its keep. For anything smaller, it’s overhead you don’t need.

Chef

Chef lets you write real Ruby. That’s its greatest strength and its biggest trap. I’ve watched teams build incredibly clever Chef cookbooks that nobody else could maintain. The testing story is good if you invest in it, but most teams don’t.

If your infra team already thinks in Ruby, Chef can be powerful. If they don’t, you’re paying the Ruby learning tax on top of the config management learning tax. That’s two taxes too many.

Ansible

Ansible is YAML, SSH, no agents. You write a playbook, you run it, machines converge. Done.

- hosts: web
  tasks:
    - name: install nginx
      apt:
        name: nginx
        state: present

That’s readable by anyone on the team. The new hire understands it on day one. The on-call engineer at 3am can follow it. That matters more than any feature comparison chart.

The tradeoff is real: Ansible is push-based, not continuously enforcing. At serious scale you’ll bolt on Ansible Tower for scheduling and reporting. But for most teams, “run the playbook” is the right model.

My take

The config management wars of 2016 are mostly a distraction. Docker and immutable infrastructure are already eating into this space. But you still need something for the hosts that run your containers, for your legacy systems, for the stuff that isn’t going anywhere.

For that, I want the tool with the least ceremony. The one where I spend time on the actual infrastructure problem, not on the tool’s own complexity.

That’s Ansible.

Pick it, version your playbooks, write roles that are boring and readable, and get back to work that actually matters.

DevOps isn’t a role. It isn’t Jenkins. It isn’t even a set of practices you can copy from a blog post. It’s a cultural shift where the people who build software and the people who run it share the same goals, the same pain, and the same wins. Fix the incentives that make dev and ops fight each other first. Tools come second.

I learned this the hard way at a mobility startup.

What Happened at the Mobility Startup

When I joined, we had a classic split. Three backend engineers shipping features as fast as possible. One ops person, call him Max, who kept the lights on. The engineers would push code on Friday afternoon, Max would get paged at 2 a.m. Saturday, and Monday morning would start with a blame session disguised as a standup.

Max was good at his job. The engineers were good at theirs. The system between them was broken.

The fix wasn’t a tool. It was a conversation. I put the entire team in a room and said: starting next week, the person who ships the code carries the pager for it. Not forever, just for the first 48 hours after deploy. If your code wakes someone up, that someone is you.

The reaction was roughly what you would expect. The engineers hated it. Max loved it. But within two weeks, something shifted. Deploys got smaller. Engineers started writing health checks without being asked. Someone added a runbook for the payment service, unprompted. The Friday afternoon deploys stopped entirely.

Nobody told them to do any of that. The incentive structure changed, and the behavior followed.

Start With Shared Pain, Not Shared Tools

That experience taught me something I keep coming back to: if you want DevOps culture, start by making everyone feel the consequences of their decisions.

Developers who never see production alerts write code that’s hard to operate. Ops people who never sit in planning meetings treat every deploy as a threat. The gap is empathy.

The cheapest thing you can do is cross-pollinate. Have a developer shadow on-call for a week. Have the ops person sit in sprint planning. Run a postmortem together after the next incident and focus on the system, not the person. I’ve never seen a team do this and not come out with a better understanding of each other.

Pick a Pilot Team and Let Them Own It

Don’t try to transform the whole company at once. Pick one team. Give them ownership of their deploy pipeline and their on-call. Let them figure out what works. Measure what happens.

At the mobility startup, that pilot was the team running the vehicle availability service. Small blast radius, clear metrics, leadership willing to experiment. They went from deploying once a week to deploying daily within a month. Not because I told them to, but because shorter cycles meant smaller changes meant fewer pages.

The rest of the engineering org watched this happen. Nobody had to sell them on DevOps after that. They just asked when they could start.

The Technical Stuff Matters, But It Isn’t First

Once the culture is moving, you need the technical foundations to support it. Infrastructure as code so environments are reproducible. A CI pipeline so every commit gets tested. Monitoring that answers “is this thing working” without requiring tribal knowledge.

In 2016, that means Ansible or Chef for config management, Jenkins or CircleCI for builds, and something like the ELK stack or Graphite for observability. The specific tools matter less than the principle: everything should be automated, version-controlled, and visible to the whole team.

I’ve seen teams with perfect pipelines and terrible culture. The pipeline doesn’t fix trust. It just makes deploys faster. If the team still throws code over the wall and blames each other when it breaks, a faster pipeline just accelerates the dysfunction.

Traps I’ve Walked Into

The hero trap. At the mobility startup, Max was the hero. He could fix anything at 3 a.m. That felt like a strength until he took a week off and nobody knew how to restart the payment gateway. Heroes are a single point of failure wearing a cape. Spread the knowledge.

Tool obsession. We spent two weeks evaluating orchestration platforms when the real problem was that nobody knew what was running in production. A shared spreadsheet would have been more useful than Kubernetes at that point. Solve the problem you actually have.

Burnout disguised as ownership. If you give developers pager duty without reducing their feature load, you aren’t building DevOps culture. You’re just distributing the suffering more evenly. Shared responsibility means shared capacity planning too.

It’s a Slow Game

DevOps culture doesn’t happen in a quarter. At the mobility startup, it took about six months before the dev-vs-ops tension actually went away. The team went from monthly releases with weekend war rooms to daily deploys that nobody thought twice about. But it was gradual, messy, and full of setbacks.

The wins compound though. Smaller deploys mean fewer incidents. Fewer incidents mean less firefighting. Less firefighting means more time for the work that actually matters. Once that flywheel starts turning, it’s hard to stop.

If you’re starting from zero, that’s fine. Most teams are. Start with the incentives, not the tools. Make people feel what production actually looks like. Pick a small team and let them prove the model. The culture will follow the work.

We Broke Production Three Times in the First Month

At a mobility startup, we moved our backend services into Docker containers because we were tired of deploy scripts that worked on one machine and failed on another. The promise was simple: build once, run anywhere. That part was true. What nobody told us was how many new ways we would find to break things.

The first week, a developer pushed a container built on their laptop. It had a different glibc than the host. Segfault in production. The second week, we ran out of disk because nobody was cleaning up old images. The third week, a container ran as root and wrote a config file to a bind mount that the host process couldn’t read back. Three incidents, three categories of failure, all in 30 days.

Every one of those was our fault. Docker did exactly what we told it to do. We just hadn’t learned to tell it the right things yet.

Dockerfiles That Actually Work

Most Dockerfile tutorials produce images that are 800MB and run as root. That’s fine for a demo. It isn’t fine when you’re pushing 15 deploys a day across 4 services.

Here is the pattern we settled on for our Go services at the mobility startup:

# Dockerfile.build: compiles the binary in a throwaway Go image
FROM golang:1.5

WORKDIR /go/src/github.com/acme/location-svc
COPY . .
RUN go get -d ./... \
    && CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo \
       -ldflags="-s -w" -o /app ./cmd/server

# Dockerfile: the runtime image, Alpine plus the binary
FROM alpine:3.3

RUN apk add --no-cache ca-certificates tzdata \
    && adduser -D -u 1000 appuser

COPY app /usr/local/bin/app

USER appuser
EXPOSE 8080
ENTRYPOINT ["app"]

docker build -t location-svc-build -f Dockerfile.build .
docker create --name extract location-svc-build
docker cp extract:/app ./app
docker rm extract
docker build -t acme/location-svc:$(git rev-parse --short HEAD) .

A few things to notice. One Dockerfile can’t compile the code and then leave the compiler behind, so we split the job in two: a throwaway Go image builds the binary, and a short script copies it into the runtime image. Only the static binary ends up on the Alpine image, which keeps it under 20MB for most of our services. We create a non-root user. We strip debug symbols with -ldflags="-s -w". We use the exec form of ENTRYPOINT, not a shell wrapper, so the binary runs as PID 1 and receives signals directly.

For our Python services (we had one analytics pipeline that wasn’t worth rewriting), the pattern looked different:

FROM python:3.5-slim

RUN groupadd -r app && useradd -r -g app -d /app -s /sbin/nologin app

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
RUN chown -R app:app /app

USER app
EXPOSE 5000
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "2", "wsgi:app"]

Same principles, plus dependencies installed before the code is copied, so Docker caches that layer. Non-root user. Minimal base image. Explicit port. No latest tag anywhere.

Image Tagging: latest Isn’t a Version

We tag every image with the git commit SHA. Not the branch name, not latest, not a date string. The commit SHA.

docker build -t acme/location-svc:$(git rev-parse --short HEAD) .
docker push acme/location-svc:$(git rev-parse --short HEAD)

When something breaks at 2 AM, the first question is: what is running right now? With SHA tags, you can answer that in seconds. You can diff the exact code. You can roll back to the previous SHA. You can see who merged what. latest tells you nothing.

We store image metadata as labels too:

ARG GIT_SHA
ARG BUILD_DATE
LABEL build.sha="${GIT_SHA}" \
      build.date="${BUILD_DATE}" \
      build.ci="jenkins"

It sounds like overkill. It isn’t. During an incident last month, we traced a memory leak to a specific commit because the container label told us exactly what was deployed.

docker-compose for Local Development

Our services talk to PostgreSQL, Redis, and each other. Getting all of that running locally used to mean a two-page setup guide that was always out of date. Now it’s a single file:

version: '2'
services:
  location-svc:
    build: ./services/location
    ports:
      - "8080:8080"
    environment:
      - DATABASE_URL=postgres://app:dev@db:5432/app?sslmode=disable
      - REDIS_URL=redis://cache:6379
    depends_on:
      - db
      - cache

  trip-svc:
    build: ./services/trip
    ports:
      - "8081:8080"
    environment:
      - DATABASE_URL=postgres://app:dev@db:5432/app?sslmode=disable
      - LOCATION_SVC_URL=http://location-svc:8080
    depends_on:
      - db
      - location-svc

  db:
    image: postgres:9.5
    environment:
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=dev
      - POSTGRES_DB=app
    volumes:
      - pgdata:/var/lib/postgresql/data

  cache:
    image: redis:3.0-alpine

volumes:
  pgdata:

New engineers run docker-compose up and have a working environment in under two minutes. No installing Go. No configuring Postgres. No hunting for the right Redis version. The compose file is checked into the repo, so it stays current.

One thing we learned the hard way: depends_on doesn’t mean “wait until the service is ready.” It means “start this container after that one.” Postgres takes a few seconds to accept connections. Your application will crash if it tries to connect immediately. We added retry logic to every service’s database connection code. Not elegant, but reliable.

Networking: Stop Hardcoding IPs

Docker’s default bridge network assigns IPs dynamically. If you hardcode an IP in a config file, it will work until it doesn’t, and then you will spend an hour figuring out why service A can’t find service B.

On a single host, Docker Compose gives you DNS resolution for free. Service names resolve to container IPs. http://location-svc:8080 just works. That’s good enough for development and for small production setups.

For multi-host, the story is more complicated. We use overlay networks, which let containers on different hosts communicate as if they were on the same network. It works, but it adds latency. We measured an extra 1-2ms per hop on our setup. For most services that’s noise. For the real-time vehicle location tracking, it was noticeable. We kept that service on a single host with host networking.

Logs: stdout or Nothing

Every one of our services logs to stdout. No log files inside the container. No custom logging directories. stdout.

log.SetOutput(os.Stdout)
log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds | log.LUTC)

Docker captures stdout and makes it available via docker logs. We run a log forwarder on each host that ships container logs to a central ELK stack. Each log line includes the container ID, image name, and service name as structured fields. When something goes wrong, we can filter by service, time range, and container in Kibana within seconds.

The mistake we made early was letting one service log at DEBUG level in production. It wrote 2GB of logs in an hour, filled up the Docker log driver’s buffer, and slowed down every other container on the host. We now enforce log level via environment variables and default to WARN in production.

Resource Limits Aren’t Optional

A container without resource limits will happily consume all available memory on the host and take everything else down with it. We learned this when our analytics service hit a data spike and OOM-killed the Postgres container running on the same host.

Every container gets explicit limits:

location-svc:
  mem_limit: 256m
  memswap_limit: 256m
  cpu_shares: 512

The memswap_limit matching mem_limit prevents the container from using swap. If it runs out of memory, it dies. That sounds harsh, but a container thrashing on swap is worse than a container that restarts cleanly. Our health checks pick up the restart within seconds.

Security: Run as Non-Root or Don’t Ship

I mentioned this in the Dockerfile section but it bears repeating. Every container runs as a non-root user. No exceptions. If a library or tool requires root, we find a different library or tool.

Beyond the user, we also:

• Use --read-only filesystem where possible, mounting only specific writable directories for temp files
• Drop all capabilities and add back only what is needed: --cap-drop=ALL --cap-add=NET_BIND_SERVICE
• Never mount the Docker socket into a container (I’ve seen this in tutorials and it terrifies me)
• Scan images for known vulnerabilities before pushing to our private registry

Our private registry runs behind TLS with basic auth. Every push and pull is authenticated. The registry itself runs in a container, which is a fun bit of recursion, but it works.

Health Checks and Restarts

Every service exposes a /healthz endpoint that returns 200 if the service can reach its dependencies:

func healthHandler(w http.ResponseWriter, r *http.Request) {
    if err := db.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        fmt.Fprintf(w, "db: %v", err)
        return
    }
    w.WriteHeader(http.StatusOK)
    fmt.Fprint(w, "ok")
}

Our load balancer polls this endpoint every 5 seconds. If a container fails three consecutive checks, it gets pulled from the rotation. Docker’s restart policy (--restart=on-failure:5) handles restarting the process. If it fails 5 times in a row, it stays down and pages us. At that point, something is fundamentally wrong and automatic restarts won’t fix it.

What I Would Tell Myself Six Months Ago

Start with one service. Not the most important one and not the least important one. Pick something that deploys frequently and has good test coverage. Containerize it, run it in production for two weeks, and fix every sharp edge you find. Then do the next one.

Don’t try to containerize your database early. We keep Postgres on a dedicated host with proper backups, WAL archiving, and monitoring. The application containers are stateless and disposable. The database is neither of those things.

Write your Dockerfiles like they will be read by someone debugging a production incident at 3 AM. Because they will be.

Treat image builds as CI artifacts. Build them in CI, tag them with the commit, push them to a registry, deploy from the registry. Never build on a developer laptop and push to production. We burned ourselves on that one.

Docker changed how we ship software at the mobility startup. Deploys went from 20 minutes of SSH-and-pray to 45 seconds of pull-and-start. But the speed only came after we did the unglamorous work of writing good Dockerfiles, setting up log aggregation, adding health checks, and enforcing resource limits. The container is the easy part. The ops around it is the job.