They aren’t.

What Are You Actually Scanning

A container image is a layer cake of risk. The base OS, the system packages, the application dependencies, your code, and whatever config and secrets accidentally ended up in there. Scanning just the OS packages – which is what most default setups do – covers maybe 30% of the attack surface.

A useful scan covers:

• Base image and OS packages
• Application dependencies (npm, pip, Go modules, whatever)
• Misconfigurations – running as root, exposed ports, writable filesystems
• Secrets embedded in image layers (it happens more than you think)

If your scanner only does the first one, you have a false sense of security. Which is worse than no scanner at all, because at least without a scanner you know you’re exposed.

Where to Scan

CI, before merge. This is where fixes are cheap. Block on critical findings. Let medium and low findings through with a warning. If you block on everything, developers will route around the scanner. I’ve seen it happen at three different companies. The scanner becomes a rubber stamp or, worse, gets disabled.

Registry, on a schedule. An image that was clean last Tuesday might have a critical CVE by Friday. Rescan images in the registry weekly at minimum. Alert on new critical findings. This is the part most teams skip, and it’s the part that matters most for long-lived images.

Admission control. Prevent unscanned images from running in production. This doesn’t need to be complicated – a simple OPA policy or Kyverno rule that rejects images without a scan annotation. The goal is to close the gap between “we scanned it” and “we’re running it.”

Prioritization Over Volume

A scanner that reports 500 findings isn’t helping. It’s generating noise. Prioritize by four factors:

1. Exploitability. Is there a known exploit in the wild? A theoretical vulnerability in a library you don’t use isn’t urgent.
2. Exposure. Is this component internet-facing or buried behind three layers of internal networking?
3. Criticality. Is this your payment service or your internal metrics dashboard?
4. Fix availability. If there’s no patch, what exactly do you want the team to do about it?

Handle exceptions with documented waivers – owner, reason, expiry date. No permanent exceptions. If a waiver expires and nobody renews it, the finding blocks again.

Base Images Are Dependencies

Treat them like one. Pin versions. Set a monthly upgrade cadence. Use minimal bases – distroless or alpine – not because they’re trendy but because fewer packages means fewer CVEs means less noise.

I’ve seen teams running Ubuntu 20.04 base images with 200+ packages when their Go binary needs exactly zero of them. That isn’t a base image. That’s an attack surface donation.

Sign Your Images

Scanning tells you what’s inside an image. Signing tells you it’s the image you built and nobody tampered with it between your CI pipeline and your cluster. Use cosign, use Notary, use whatever – but sign your artifacts and verify on pull.

After the SolarWinds and codecov incidents, “trust but don’t verify” isn’t a defensible position anymore.

The Minimum Viable Pipeline

1. Scan in CI with Trivy or Grype. Block on critical, warn on high.
2. Rescan in your registry weekly. Alert on new criticals.
3. Admission control in Kubernetes. Reject unscanned images.
4. Pin and regularly update base images.
5. Sign images in CI. Verify at deploy.

That’s it. Five steps. None of them are hard individually. The hard part is maintaining the discipline to triage findings, update waivers, and keep base images current. Discipline over heroics, as always.

Kubernetes out of the box is optimized for convenience, not security. Lock down pods, network, RBAC, secrets, and the control plane as separate layers. Assume every layer will be tested. My national cyber-defense background taught me defense in depth isn’t a buzzword – it’s the only model that survives contact with reality.

Kubernetes ships with defaults designed to get your workloads running. Not to resist a motivated attacker. Not to survive a leaked credential. Not to contain a compromised container. If you’re running a production cluster with default settings, you’re running a cluster that trusts everything by default. That should scare you.

I’ve hardened clusters across multiple organizations and I keep seeing the same gaps. This post walks through the layers that matter, with config you can actually apply. Some of this comes from my time working with national cyber-defense systems, where “assume breach” wasn’t a thought experiment – it was the operating model.

Start with a threat model (seriously)

I know. Nobody wants to do this part. But every security control you skip because “we’re not a bank” is a control an attacker won’t encounter. You don’t need a 40-page document. You need answers to three questions:

1. What data would hurt you if it leaked?
2. Who has access to your cluster, and how?
3. If a container gets compromised, what can the attacker reach?

If you can’t answer these quickly, start there before touching any YAML.

Pod security: the non-negotiable baseline

Every container should run with the least privilege possible. Full stop. Here is what that looks like in practice:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000
  containers:
    - name: app
      image: myregistry/app:v1.2.3@sha256:abc123...
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"
        limits:
          cpu: "500m"
          memory: "256Mi"

Key points:

• runAsNonRoot: true – if your app needs root, fix the app. I’ve seen exactly one legitimate case for running as root in a non-system container in the last three years.
• Drop ALL capabilities – then add back only what’s needed. Most applications need zero Linux capabilities.
• Read-only root filesystem – forces you to use volumes for writable paths, which is where you want writes anyway.
• Resource limits – not just for cost. An unbound container can starve the node and affect every other workload on it.
• Pin image digests – latest isn’t a version. Pin to a SHA256 digest for anything that matters.

PodSecurityPolicy is deprecated as of 1.21 and removed in 1.25. Replace it. Pod Security Admission (beta in 1.23) implements three levels – privileged, baseline, restricted. At minimum, enforce baseline cluster-wide and restricted on sensitive namespaces:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

For more granular control, OPA Gatekeeper or Kyverno let you write custom policies. I prefer Kyverno for readability, but both work.

Network policies: default deny, then allow

Kubernetes networking is flat by default. Every pod can talk to every other pod. That’s fantastic for an attacker who compromises one service and wants to pivot laterally.

Start with a default deny policy on every namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then explicitly allow what’s needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:  # DNS
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

A critical gotcha: NetworkPolicy only works if your CNI supports it. Flannel doesn’t enforce network policies. Calico, Cilium, and Weave do. Verify enforcement in every environment, including staging. I’ve seen teams spend weeks writing policies that were silently ignored because the CNI was wrong.

Don’t forget egress. Ingress gets all the attention, but an attacker who compromises a pod and can make outbound HTTPS calls to the internet has an exfiltration path. Restrict egress to known destinations.

RBAC: strict and boring

Good RBAC is boring. It should be so restrictive that nobody notices it until they try to do something they shouldn’t.

• One service account per workload. Not one per namespace. Not the default service account.
• Scope roles to namespaces and specific resources. Cluster-wide roles are for platform teams, not applications.
• Disable automatic service account token mounting for pods that don’t need to talk to the API server:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
automountServiceAccountToken: false

For human access, use OIDC with your identity provider. Short-lived tokens from kubectl via OIDC are vastly better than long-lived kubeconfig certificates. Nobody should have cluster-admin in production unless they’re actively performing cluster operations, and even then, consider just-in-time access.

Secrets: they aren’t encrypted

This surprises people. Kubernetes Secrets are base64-encoded in etcd. That’s encoding, not encryption. Anyone with etcd access can read every secret in your cluster.

Fix this in layers:

1. Encrypt etcd at rest. Configure EncryptionConfiguration on the API server with aescbc or secretbox providers.
2. Use an external secrets manager. HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager with an operator like External Secrets. The secret value never lives in etcd.
3. Scope access. Only the service accounts that need a secret should be able to read it. RBAC applies to secrets too.
4. Rotate. If you can’t rotate a secret without a deployment, fix your rotation process before you fix the secret.

Never bake secrets into container images. Never put them in ConfigMaps. Never log them. These sound obvious, but I find violations in almost every cluster audit.

Control plane hygiene

The API server is the front door to everything. Lock it down:

• Disable anonymous authentication (--anonymous-auth=false).
• Enable audit logging. You need to know who did what and when.
• Restrict API server access to trusted networks. If your API server is reachable from the public internet, you have a bigger problem than this blog post covers.
• Keep etcd on a private network, TLS-encrypted, and only reachable by the API server.
• Treat kubeconfig files like production database credentials. Because that’s what they’re.

If you’re on a managed service (EKS, GKE, AKS), the provider handles some of this. But “managed” doesn’t mean “secure.” Review what they expose and what they lock down. I’ve seen managed clusters with surprisingly permissive defaults.

Image supply chain

Stop using latest. Stop using mutable tags. Every production image should be:

• Built from a minimal base (distroless, Alpine, or scratch for Go binaries).
• Scanned for vulnerabilities in CI before it reaches a registry.
• Signed, ideally with cosign or Notary.
• Pulled from a private registry with access controls.

If you’re pulling public images directly into production, you’re trusting that Docker Hub (or whoever) hasn’t been compromised. That trust should make you uncomfortable.

Node-level protections

Nodes are part of your trust boundary. A compromised node means compromised pods.

• Use a minimal OS (Bottlerocket, Flatcar, Talos). Less software means fewer attack vectors.
• Patch regularly. Automate node rotation if you can.
• Lock down the kubelet API. Disable anonymous kubelet auth.
• Apply seccomp profiles. The RuntimeDefault profile is a good starting point and blocks a significant number of syscalls:

securityContext:
  seccompProfile:
    type: RuntimeDefault

Runtime detection tools like Falco are worth considering. They won’t prevent an attack, but they can alert you when something unexpected happens – a shell spawning inside a container, an unusual network connection, a binary that shouldn’t exist being executed.

Monitoring and incident response

Hardening without visibility is just hoping. You need:

• Centralized audit logs from the API server.
• Alerts on failed authentication attempts and privilege escalations.
• Monitoring for unexpected image pulls or outbound connections.
• A written incident response playbook. Not a novel. A one-page runbook that answers: who gets paged, what gets isolated, how do we preserve evidence.

Write the playbook before you need it. During an actual incident isn’t the time to figure out who has cluster access.

The operating discipline

None of this works as a one-time project. Security is operational:

• All cluster config changes go through code review. GitOps is your friend here.
• Stay on supported Kubernetes versions. Running an EOL version means running with known unpatched vulnerabilities.
• Test your controls in staging. I’ve seen policies that worked perfectly in staging and failed silently in production because the namespace labels were wrong.
• Run backup and restore drills. Not “we’ve Velero installed.” Actually restore from backup and verify the cluster works.

Kubernetes hardening isn’t a checklist you complete once. It’s a set of defaults you enforce continuously, across pods, network, identities, secrets, and the control plane. When those defaults are solid, incidents become containable instead of catastrophic.

Your image scan is green. Congratulations. You’ve answered “what’s in this container.” You haven’t answered “what is this container doing right now.” Runtime security is how you answer that second question – through least privilege, kernel-level controls, network policy, and behavioral detection. This post walks through exactly how we do it at Decloud, with real configs.

I’ve had this conversation too many times. Someone shows me a green Trivy scan and says “we’re secure.” No. You know what’s in the image. You have no idea what it’s doing.

At Decloud we run a lot of containers. Hundreds of services. And after spending time in national cyber-defense, I came away with one conviction that shapes everything: assume the perimeter is already breached. The question isn’t whether something gets in. It’s how much damage it can do once it’s there.

That’s runtime security. Not a product. Not a checkbox. A set of controls that limit what containers can do, watch what they’re doing, and react when behavior goes sideways.

What image scanning doesn’t catch

Static scanning is useful. I’m not dismissing it. But it’s blind to:

• Zero-days. By definition your scanner doesn’t know about them yet.
• Stolen credentials being used inside a running container.
• Lateral movement. A compromised container talking to your database.
• Data exfiltration over DNS or unexpected outbound connections.
• Runtime drift. Someone kubectl exec’d in and installed curl. Your image is clean. Your container isn’t.

The mental model shift is important. Scanning is about contents. Runtime security is about behavior.

What “suspicious” actually looks like in production

Forget movie-hacker nonsense. In real production clusters, compromise looks boring:

• A shell spawns in a container that should only run a Go binary.
• A container starts making outbound connections to IPs it’s never talked to.
• Something writes to /etc/passwd or drops a new binary in /tmp.
• CPU spikes because someone’s mining Monero in your API pod.
• A container that’s been running for 40 days suddenly starts probing the Kubernetes API.

None of these trip an image scanner. All of them should wake someone up.

Start with the Dockerfile. Seriously.

Before you install Falco or write a single policy, harden the image itself. This is where most teams skip steps and then wonder why their runtime security is noisy.

Here’s what a production Dockerfile should look like:

FROM golang:1.15-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /server .

FROM scratch
COPY --from=builder /server /server
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
USER 65534:65534
ENTRYPOINT ["/server"]

Key things here: multi-stage build so no compiler or tools in the final image. scratch base so there’s literally no shell, no package manager, no curl, no nothing. Non-root user. If an attacker gets code execution in this container, they land in an empty filesystem with no tools and no root. Good luck.

At Decloud we moved almost all Go services to scratch or distroless. It eliminates entire categories of post-exploitation. Can’t spawn a shell if there’s no shell.

Pod security context: the non-negotiable baseline

Every pod in your cluster should have this. No exceptions.

securityContext:
  runAsNonRoot: true
  runAsUser: 65534
  runAsGroup: 65534
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

readOnlyRootFilesystem is the one people push back on. “But my app writes temp files!” Fine. Mount a single emptyDir for /tmp and nothing else. Don’t give up the whole filesystem because one library wants to write a cache file.

volumeMounts:
  - name: tmp
    mountPath: /tmp
volumes:
  - name: tmp
    emptyDir:
      sizeLimit: 64Mi

At Decloud, we enforce this via OPA Gatekeeper. If your pod spec requests privileged: true or doesn’t drop all capabilities, the admission controller rejects it. Period.

Seccomp: the kernel-level lockdown most people ignore

Here’s the thing about Linux containers – they share the host kernel. Every syscall your container makes goes through the same kernel as every other container on that node. Seccomp lets you whitelist which syscalls a container is allowed to make.

The default Docker seccomp profile blocks about 44 syscalls. That’s a start. But for a well-behaved Go HTTP server, you need maybe 40-50 syscalls total out of the 300+ available. Everything else should be denied.

Here’s a stripped-down seccomp profile we use for stateless API services:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": ["SCMP_ARCH_X86_64"],
  "syscalls": [
    {
      "names": [
        "accept4", "bind", "brk", "clone", "close",
        "connect", "epoll_create1", "epoll_ctl", "epoll_pwait",
        "exit_group", "fcntl", "fstat", "futex", "getpeername",
        "getpid", "getsockname", "getsockopt", "listen",
        "madvise", "mmap", "mprotect", "munmap", "nanosleep",
        "openat", "read", "recvfrom", "rt_sigaction",
        "rt_sigprocmask", "rt_sigreturn", "sched_yield",
        "sendto", "setsockopt", "sigaltstack", "socket",
        "tgkill", "write"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Apply it in your pod spec:

securityContext:
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/api-server.json

This is defense in depth in the truest sense. Even if an attacker gets arbitrary code execution, they can’t call ptrace, can’t load kernel modules, can’t mount filesystems. The kernel says no. I picked this habit up from national cyber-defense work – you don’t rely on one layer. You assume each layer will fail and build the next one.

Building these profiles by hand is tedious. We generate them by running the app under strace in a staging environment, collecting the syscalls it actually uses, and then adding a small buffer. Not glamorous. Works.

Network policy: because default Kubernetes networking is terrifying

Out of the box, every pod in a Kubernetes cluster can talk to every other pod. Let that sink in. Your frontend can talk to your payment service. Your batch job can reach your secrets manager. It’s a flat network with no segmentation.

Network policies fix this. Here’s what we apply to every namespace at Decloud:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

This denies everything by default. Then we whitelist specific flows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: gateway
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector:
            matchLabels:
              name: kube-system
      ports:
        - protocol: UDP
          port: 53

That last rule is DNS. People forget DNS and then spend an hour wondering why their app can’t resolve anything. Ask me how I know.

The egress rules are the important ones for runtime security. If a compromised container can’t make outbound connections, the attacker can’t exfiltrate data or download tools. It’s boring. It works.

Falco: watching what your containers actually do

Falco hooks into the kernel via eBPF (or a kernel module) and watches syscalls in real time. It’s the closest thing to “what is this container actually doing right now.”

Here are the rules we run in production at Decloud:

- rule: Shell in non-shell container
  desc: Detect shell spawned in a container that shouldn't have one
  condition: >
    spawned_process and container
    and proc.name in (sh, bash, zsh, dash, csh)
    and not container.image.repository in (debug-tools, ops-shell)
  output: >
    Shell started in production container
    (user=%user.name container=%container.name
    image=%container.image.repository command=%proc.cmdline)
  priority: CRITICAL

- rule: Write below /bin or /usr
  desc: Detect new binary written to system paths
  condition: >
    open_write and container
    and (fd.name startswith /bin/ or fd.name startswith /usr/bin/
    or fd.name startswith /usr/sbin/)
  output: >
    Binary written in container
    (file=%fd.name container=%container.name image=%container.image.repository)
  priority: CRITICAL

- rule: Outbound connection to unexpected port
  desc: Detect container connecting to non-standard ports
  condition: >
    outbound and container
    and not fd.sport in (80, 443, 5432, 6379, 8080, 8443, 9090)
    and not container.image.repository in (monitoring-agent)
  output: >
    Unexpected outbound connection
    (container=%container.name port=%fd.sport dest=%fd.sip)
  priority: WARNING

Two things I’ve learned the hard way about Falco rules:

Be specific. Generic rules like “alert on any process spawn” will bury you in noise within hours. You’ll disable the alerts and then you’re back to flying blind.

Tune per workload. Your Nginx sidecar legitimately spawns worker processes. Your Go API server doesn’t. Same rule, different context. We maintain per-service rule overrides and review them quarterly.

Incident response: what happens when something fires

Detection without response is a hobby, not a security program. Here’s our playbook at Decloud:

1. CRITICAL alert fires. PagerDuty wakes someone up. Shell in a production container or binary write to system paths – someone looks at it within minutes.

2. Capture state before killing. kubectl logs, kubectl describe pod, network connection dumps. If you kill the pod first, you’ve destroyed your forensics.

3. Network isolate. Apply a deny-all network policy to the specific pod. The container is still running but can’t talk to anything. This buys time.

4. Investigate. Was it a legitimate deploy? A developer debugging? Or actual compromise? Most of our critical alerts turn out to be config mistakes. That’s fine. I’d rather investigate 10 false positives than miss one real incident.

5. Kill and rotate. If it’s real, kill the pod, rotate every credential it had access to, and check the blast radius.

One rule we follow strictly: no automated kills on low-confidence alerts. I’ve seen teams auto-restart pods on anomaly detection. They ended up killing healthy services during a deploy because the new version had slightly different syscall patterns. Humans in the loop for destructive actions.

The compounding effect

None of these controls is a silver bullet on its own. A seccomp profile doesn’t prevent credential theft. Network policy doesn’t stop a shell spawn. Falco doesn’t harden your kernel surface.

But stack them together and the attacker’s job gets exponentially harder:

• Scratch image: no tools to use post-exploitation.
• Non-root, dropped capabilities: can’t escalate privileges.
• Read-only filesystem: can’t persist or install anything.
• Seccomp: can’t make dangerous syscalls.
• Network policy: can’t phone home or move laterally.
• Falco: any anomalous behavior triggers an alert.

Each layer assumes the previous one failed. That’s not paranoia. That’s engineering.

From my national cyber-defense days: you don’t build a castle with one wall. You build concentric rings where each ring assumes the outer one is already breached. Container runtime security is the same idea, just with YAML instead of stone.

Serverless wins at low, bursty traffic. Containers win at sustained load. The crossover happens sooner than most people think. I’ve run workloads on both sides of that line and the difference in cost can be 3-5x if you pick wrong.

Everyone’s doing serverless now. Every conference talk. Every blog post. Every startup pitch deck mentions Lambda like it’s a personality trait.

I get it. At Decloud we help companies sort out their cloud infrastructure, and serverless comes up in almost every conversation. Half the time it’s the right call. The other half, someone read a blog post and now their entire API runs on Lambda with 400ms cold starts and a monthly bill that makes no sense.

So here’s the honest breakdown. No hype, no anti-hype. Just the math.

The comparison nobody wants to make

People treat serverless vs containers like a religious debate. It’s not. It’s arithmetic.

That last row matters more than people admit.

Where serverless genuinely wins

I’ll give credit where it’s due. For certain workloads, serverless is unbeatable.

Event processing. S3 upload triggers a function, function processes the file, done. No server sitting around waiting. This is the original Lambda use case and it’s still the best one. We use this pattern at Decloud for processing customer infrastructure snapshots.

Webhooks and integrations. Glue code between services. Receives a payload, transforms it, passes it along. Runs maybe 200ms. Happens a few thousand times a day. Perfect fit. Running a container for this is like hiring a full-time employee to check the mailbox.

Cron jobs that run under 15 minutes. Cleanup tasks, report generation, health checks. A Lambda on a CloudWatch schedule is simpler than managing a cron server or scheduling containers.

Genuinely unpredictable traffic. If you can’t forecast whether you’ll get 10 requests or 10,000 in the next hour, serverless handles that gracefully. Containers need lead time to scale.

Where serverless falls apart

Here’s the part that gets me uninvited from serverless meetups.

Sustained API traffic. If your API handles steady traffic – say 500+ requests per second, consistently – you’re paying a premium for Lambda that buys you nothing. The per-invocation cost adds up fast. I’ve seen teams cut their compute bill by 60-70% by moving a stable API from Lambda to Fargate. Not a theoretical number. Actual invoices.

Anything that needs database connections. This one drives me crazy. Lambda spins up instances independently. Each one opens its own database connection. You go from 10 concurrent executions to 500 during a traffic spike and suddenly your Postgres is drowning in connections. Yes, RDS Proxy exists now. It helps. It’s also another managed service you’re paying for to solve a problem containers don’t have.

Latency-sensitive paths. Cold starts. I know, provisioned concurrency exists. But provisioned concurrency is just… running a container with extra steps. You’re paying to keep Lambda instances warm. At that point, what are you even doing?

Complex request processing. If your function needs to do three API calls, a database write, and a cache update, that 200ms function becomes 800ms. You’re paying for all that wall-clock time. A container doing the same work with persistent connections and warm caches does it in 150ms.

The real cost comparison

Let me get specific. Rough numbers for a simple API endpoint, US East, mid-2020 pricing:

1 million requests/day, 200ms average duration, 256MB memory:

• Lambda: ~$250/month (invocations + duration)
• Fargate (2 tasks, 0.5 vCPU, 1GB): ~$60/month

That’s 4x. At 5 million requests/day with the same profile, the gap widens.

10,000 requests/day, bursty, same specs:

• Lambda: ~$3/month
• Fargate (1 task minimum): ~$30/month

Flipped completely. Serverless is 10x cheaper at low volume.

The crossover point for a typical web API sits somewhere around 200K-500K requests per day, depending on duration and memory. Below that, serverless. Above that, containers. This isn’t gospel – measure your own workload – but it’s a reasonable starting point.

The real problem is lock-in

Something nobody talks about enough: at 50+ Lambda functions with API Gateway, Step Functions, SQS triggers, DynamoDB streams, and EventBridge rules, you haven’t built an application. You’ve built an AWS application. Every piece of business logic is coupled to a specific AWS service.

Containers running your own code with standard libraries? Move them to GCP, Azure, your own hardware, whatever. The portability isn’t theoretical. I’ve done it.

With Decloud we see this regularly. Companies come to us wanting to optimize or migrate, and the ones running serverless-heavy architectures have a much harder time. Not impossible. Just harder and more expensive to change.

My actual recommendation

Stop asking “should we use serverless?” and start asking “what does the traffic look like for this specific endpoint?”

• Bursty, low-volume, event-driven? Lambda. Don’t overthink it.
• Steady traffic above a few hundred requests per second? Containers.
• Mixed? Use both. Nobody said you have to pick one.

The teams that do this well treat serverless as a tool, not an architecture. They use it where the math works and containers where it doesn’t. No ideology. Just invoices.

That’s the whole secret. Look at the bill. Do the math. Deploy accordingly.

PodSecurityPolicy is real now

When I wrote the first post, PodSecurityPolicy existed, but almost nobody used it. That changed fast. We rolled it out at the fintech startup in Q1, and it caught issues immediately. A dev tried to deploy a container with hostNetwork: true for a quick port-forwarding hack. PSP rejected it. Before PSP, that would have gone through and sat there until someone noticed — or didn’t.

The enforcement model is rough. You define policies, bind them via RBAC, and hope the interaction between multiple policies does what you think it does. We spent a solid week debugging why certain pods wouldn’t schedule. But the alternative — trusting every deploy not to escalate privileges — is worse.

We moved past Alpine

In the 2017 post I showed an Alpine-based final image. We’ve since moved most Go services to scratch or distroless. No shell. No package manager. No /etc/passwd. An attacker who gets code execution inside one of these containers gets almost nothing. They can’t spawn a shell because there isn’t one.

The tradeoff is debugging. You can’t kubectl exec into a scratch container and poke around. We handle this with ephemeral debug containers and better structured logging. Worth it.

Image signing went from nice-to-have to required

Docker Content Trust plus Notary. We gate all production pulls on signature verification now. This came out of a near-miss: a CI pipeline pulled from an internal registry without verifying the image hadn’t been tampered with. Nobody tampered with it. But nobody would have known if they had.

export DOCKER_CONTENT_TRUST=1
docker trust sign registry.internal/api:2.4.1

Simple. We should have done it a year ago.

Network policies still matter more than service meshes

Istio is getting a lot of attention. We’re still not running it. NetworkPolicy with Calico does what we need: default-deny, explicit allows, and it doesn’t add another control plane to babysit. When Istio matures, maybe. For now, we keep it simple.

The thing nobody talks about

Secrets. We switched from Kubernetes Secrets to Vault-backed dynamic credentials for database access. Short-lived tokens that rotate automatically. The ops overhead was real — Vault isn’t a simple system — but the security posture improvement was immediate. No more long-lived database passwords sitting in etcd.

Where we are

Eight months of tightening. Every service runs non-root, read-only filesystem, all capabilities dropped. PSP enforces it cluster-wide. Images are signed. Network policies are default-deny. Secrets come from Vault.

None of it’s glamorous. Most of it’s YAML and policy files. But the gap between “we know what to do” and “we actually do it everywhere” is where breaches live. We’re closing that gap, one deploy at a time.

I wrote about our first year with Kubernetes a year ago. The tone was “we survived and it was worth it.” That’s still true. But year two taught me that survival and maturity are very different things.

The networking tax

I said networking was hard last year. Understatement.

At the fintech startup we started with a basic CNI and no network policies. Fine for a while. Then we added more services, more teams started deploying, and suddenly every pod could talk to every other pod. No segmentation. No audit trail of who talks to what. Fixing this retroactively was brutal. You can’t just flip on network policies without mapping every legitimate connection first. We spent two weeks doing traffic analysis before we could write a single policy.

DNS was the other trap. Kube-dns worked great at our initial scale. Ten services, light traffic, no complaints. Then we added a data pipeline that hammered DNS with lookups on every request. Intermittent failures. Slow responses that looked like application bugs. Took us days to trace it back to DNS resolution under load. The fix was trivial — ndots configuration and a local DNS cache. The debugging wasn’t.

Treat networking as its own project. First-class. With a budget. Not something you’ll “figure out later.”

Resource requests stopped being guesses

Year one, our resource requests were vibes. Somebody would eyeball a service, pick a number, ship it. We got away with it because we had plenty of headroom.

Year two, the cluster got denser. Services started competing for resources. OOMKills on one service, wasted capacity on another. The scheduler was doing exactly what we told it to — the problem was that we told it garbage.

We started profiling every service under realistic load in staging. Measured actual memory and CPU usage over a week. Then set requests based on the P95, not the average, not a guess. Kept memory limits close to requests. Dropped CPU limits entirely unless a service had a known runaway pattern.

resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"

That YAML block looks boring. It’s also the difference between a stable cluster and a 3 AM page. The numbers matter. Profile them.

Stateful workloads: still nope (mostly)

We tried running Elasticsearch inside the cluster for about a month. Storage behavior was unpredictable. Backup and restore required custom tooling. Recovery after a node failure was slow and manual. We moved it back to managed infrastructure and haven’t looked back.

Postgres and Redis stayed outside the cluster all of year two. I know people run stateful workloads on Kubernetes successfully. I also know it requires an operational investment we weren’t ready to make. Managed services aren’t free — they just shift the cost from your on-call to your invoice. For us, that trade was obviously correct.

PodDisruptionBudgets: the lesson I learned the hard way

Back to that 3 AM page. A single node drain shouldn’t be able to take down a production service. That’s the whole point of running multiple replicas. But without a PodDisruptionBudget, Kubernetes will happily drain every pod from a node regardless of what service they belong to. If all your replicas happen to land on the same node — and the scheduler will do this if your anti-affinity rules are weak — one drain takes you to zero.

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

We now treat PDBs as part of the deployment contract. You don’t get to ship a production service without one. Period.

Upgrades aren’t routine

Upgraded the cluster twice in year two. Both times were more work than expected.

Deprecations broke manifests. API versions we were using got dropped with one minor release warning. Node draining surfaced resource issues we’d been ignoring. Control plane changes rippled into unexpected places — one upgrade changed default admission controller behavior and three deploys failed before we figured out why.

The process that worked: read every release note cover to cover. Rehearse in staging with production-like load. Not “run the tests” — actually push traffic through it. Schedule a real maintenance window. Don’t YOLO a cluster upgrade on a Friday afternoon. (We did this once. Never again.)

Multiple clusters helped. We moved to a setup where we could upgrade one cluster while the other kept serving. Longer to provision, but the blast radius dropped to near zero.

Git as the source of truth

This was already working in year one, but by year two it became non-negotiable culture. Every manifest in version control. Every deployment triggered by a merge. No kubectl apply from someone’s laptop. No exceptions.

The payoff isn’t just auditability. It’s rollbacks. Something breaks, you revert a commit. No guessing what changed. No “who ran what command in production.” The answer is always in the git log. This single practice eliminated more incidents than any other operational improvement we made.

RBAC and secrets: the wake-up call

Year one, everyone had cluster-admin. Year two, we finally locked it down. Created service accounts per application, scoped access narrowly, and spent an uncomfortable afternoon discovering how many things broke when we removed broad permissions. Every one of those breakages was a security hole we’d been ignoring.

Secrets were the other wake-up call. Base64 isn’t encryption. It’s encoding. Anyone with API read access could decode every secret in the cluster. We moved to pulling secrets from Vault at deploy time and encrypting etcd at rest. Should have done it in month one. Didn’t.

Java, DNS, and autoscaling

Some workloads need special treatment. Our Java services ignored container memory limits by default and allocated memory like they owned the whole node. Took a few OOMKills to learn that the JVM needs explicit container-aware flags.

DNS bit us again here. Services making too many lookups created a bottleneck that looked like network latency. Horizontal pod autoscaling only worked correctly once our base resource requests were accurate — garbage in, garbage scaling decisions out.

What I’d tell someone starting today

Use managed Kubernetes. Don’t run your own control plane unless you have a very specific reason. The managed offerings are mature enough now and the time you save goes directly into product work.

Invest in developer experience from day one. The platform can be rock solid and still feel terrible if developers can’t get a fast local feedback loop. Minikube, namespaced dev environments, whatever works — budget time for it early.

Don’t touch service mesh until the basics are boring. I’ve seen teams adopt Istio before they had working health checks. Get networking, observability, and resource management right first. Then decide if you need the complexity.

Year three

The cluster is boring now. Boring in the good way. Deploys happen multiple times a day without drama. Incidents are rarer and smaller. The platform does what we need it to do.

But it took two full years of grinding — profiling resources, debugging DNS, writing network policies, learning the hard way about PDBs, locking down RBAC, building upgrade processes. None of that’s glamorous. None of it makes for a good conference talk. It’s just the work.

Kubernetes is a platform that rewards the teams willing to operate it seriously. Two years in, I’m convinced it’s the right foundation. I’m also convinced that most of the value comes from the boring operational work that nobody wants to do.

Containers ship code. They don’t ship security. Harden the image, lock the runtime, segment the network, or accept that one escape owns your entire node.

Containers Aren’t a Security Boundary

I need to say this bluntly because I keep hearing it wrong: a container isn’t a VM. It isn’t a sandbox. It shares a kernel with the host. If something breaks out, the blast radius is the whole node. Full stop.

At the fintech startup we learned this the hard way during an early Kubernetes migration. We had a service running as root inside its container — “just for debugging,” someone said — and a misconfigured volume mount gave it read access to the host’s /etc. Nobody exploited it. We caught it in review. But it sat there for two weeks before anyone noticed, and that scared me more than an actual breach would have.

A principle from national cyber-defense that applies everywhere: assume the perimeter is already broken. Then design so that a breach in one layer doesn’t cascade. Containers are just another layer. Treat them that way.

Ship Less, Expose Less

The single highest-ROI thing you can do for container security is shrink your images. Every package you include is attack surface. Every binary is a tool an attacker can use post-exploitation. Multi-stage builds are your best friend here.

FROM golang:1.9 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o service .

FROM alpine:3.6
RUN adduser -D appuser
USER appuser
COPY --from=builder /app/service /service
ENTRYPOINT ["/service"]

This is almost exactly what we use at the fintech startup for our Go services. Builder stage compiles everything. Final image has Alpine, a non-root user, and the binary. That’s it. No curl, no bash, no wget. If an attacker gets code execution inside this container, they have almost nothing to work with.

Two more things that matter:

Pin to digests, not tags. alpine:3.6 can change underneath you. alpine@sha256:abc123... can’t. We had a CI build pass on Tuesday and fail on Thursday because the base image got a patch that broke our libc expectations. Tags are lies. Digests are truth.

Scan continuously. Not just at build time. Vulnerabilities get published weeks or months after an image ships. We run Clair against our registry on a cron. Anything with a critical CVE gets flagged and the owning team has 48 hours to patch or explain why not.

Runtime: Where Mistakes Become Incidents

A clean image means nothing if you run it with --privileged. I’ve seen production deployments where containers had SYS_ADMIN capabilities because someone copy-pasted a Stack Overflow answer during a late-night deploy. That one capability basically gives you root on the host.

Strip everything. Start from zero and add back only what you need.

securityContext:
  runAsNonRoot: true
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

Three lines of YAML. That’s it. runAsNonRoot stops the “just for debugging” excuse. readOnlyRootFilesystem prevents an attacker from dropping binaries. Dropping all capabilities means no mount, no raw sockets, no ptrace. If your app genuinely needs a specific capability — like NET_BIND_SERVICE to bind port 443 — add that one back explicitly and document why.

Pair this with seccomp profiles if your runtime supports them. Default Docker seccomp blocks around 44 syscalls. For most web services, you can block even more. We profiled our API gateway’s syscall usage over a week, built a whitelist, and cut the available syscall surface by 60%.

The other half is detection. You need to know when something unexpected happens inside a container. A new shell process. An outbound connection to an IP that isn’t in your dependency list. A binary that wasn’t in the original image. We keep this simple — sysdig + alerting rules. Nothing fancy. But it works because we actually look at the alerts.

Network Segmentation: Kill Flat Networks

Default Kubernetes networking is flat. Every pod can talk to every other pod. This is the container equivalent of putting every server in the same VLAN with no firewall rules. Insane, but it’s the default.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-frontend
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - port: 8080

This policy says: the API pod accepts traffic from frontend pods on port 8080. Everything else is denied. Lateral movement goes from trivial to impossible unless the attacker also compromises the frontend service.

At the fintech startup we started by mapping every service-to-service dependency, then wrote deny-all policies and added explicit allows. Took about a week for our cluster. Found three services that were talking to things they had no business talking to — leftover from an old feature that got half-removed. Network policies are a security tool and an architecture audit tool.

Service meshes with mutual TLS are starting to appear — Istio, Linkerd — but they’re early. We’re watching, not adopting yet. For secrets, keep them out of your images. Kubernetes Secrets with encryption at rest, or Vault if you want something more robust. Never ENV a database password into a Dockerfile. I’ve seen it. In production. At companies you’ve heard of.

RBAC and Audit Logging

Kubernetes RBAC isn’t optional. Neither is PodSecurityPolicy. These are the controls that prevent a developer with deploy access from accidentally (or intentionally) escalating to cluster-admin. Scope permissions to namespaces. Give teams access to their stuff and nothing else.

Audit logging closes the loop. When something goes wrong — and it will — you need to answer: who did what, when, and to which resource. Without audit logs, incident response is guesswork. With them, it’s a timeline.

The Real Point

None of this is exotic. Slim images, pinned builds, non-root containers, read-only filesystems, dropped capabilities, network policies, RBAC, audit logs. It’s a checklist, not a research project. The hard part isn’t knowing what to do. It’s actually doing it consistently, across every service, on every deploy, even when someone says “we’ll fix it after launch.”

Stack these controls and a container escape becomes a dead end instead of a highway to your entire infrastructure.

Worth it? Yes. Painless? Not even close.

The stuff that pays off immediately

The declarative model is the real win. You describe what you want, Kubernetes converges toward it. No more runbooks for “what if the service dies on box 3.” It just comes back. Deployments get boring, which is exactly what you want deployments to be.

Resource requests and limits are the other quick win, but only once you actually set them properly. We ran for weeks without limits on one of our data ingestion services. It worked great until it didn’t, and then it took down two neighbors on the same node.

Don’t skip this part:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Those numbers are a starting point. Profile under real load. Revisit quarterly. The defaults are lies.

Rolling updates deserve a mention too. Going from “schedule maintenance window, hold breath, deploy, pray” to incremental rollouts with automatic rollback changed how the whole team thinks about shipping. People deploy more often because it feels safe. That feedback loop alone justified the migration.

Where it bites you

Networking. Every single time.

Service discovery looks dead simple in the docs. Then you hit DNS caching issues, or a network policy blocks something you didn’t expect, or your ingress controller does something creative with header forwarding. Kubernetes hides the network behind a nice abstraction until that abstraction leaks, and then you’re reading iptables rules at midnight.

We had a bug at the fintech startup where intermittent 503s on one service turned out to be a kube-dns cache TTL mismatch with our upstream provider. Took three days to find. The fix was two lines of config. That’s Kubernetes networking in a nutshell.

Resource tuning is the slow burn. Set requests too low and the scheduler packs your nodes like a clown car — then starts evicting pods when things get tight. Set limits too low and your service gets CPU-throttled under load, which looks exactly like an application bug from the outside. The only way through is profiling under production traffic patterns and adjusting. There’s no shortcut here.

Stateful workloads are possible but honestly, we kept Postgres and Redis outside the cluster for most of the year. Storage classes, PV lifecycle, backup and restore — it all works, but the operational surface area is large. If you’re just getting started, keep your databases where they are. Move them in later when you actually understand what you’re signing up for.

Debugging gets weird. Something fails and the cause could be the application code, the container image, the scheduler, the network plugin, the node itself, or some combination. I’ve seen a deployment fail because a node had a full disk from container image garbage collection not running. Good luck finding that without decent logging and tracing. kubectl describe is your best friend, but sometimes your best friend doesn’t know the answer either.

The hard lessons

Treating Kubernetes like a black box works until you need to upgrade the cluster. Or recover from an etcd failure. Or figure out why the scheduler won’t place a pod. Someone on your team needs to actually understand the control plane. Not “watched a conference talk” understand. “Can read the API server logs and make sense of them” understand.

Running your own control plane is expensive. Not in money — in attention. If your cloud provider offers managed Kubernetes, take the deal. We spent months on control plane operations that could have been spent on product work. The managed offerings are solid enough now. Pay the tax, move on.

YAML sprawl is the death of a thousand paper cuts. You start with clean manifests for three services. Six months later you have forty files that are 80% identical and nobody remembers which copy is canonical. Helm or Kustomize help, but be careful — I’ve seen teams replace YAML sprawl with template sprawl, which is somehow worse because now you need to understand Go templates to deploy a config change.

Secrets handling is genuinely bad out of the box. Kubernetes Secrets are base64-encoded. That’s encoding, not encryption. Anyone with API access can read them. We ended up pulling secrets from Vault at deploy time and treating the built-in Secrets as a transport layer. If you’re storing database passwords as Kubernetes Secrets and calling it done, stop.

Local development is the thing nobody budgets time for. Your cluster isn’t a laptop. Developers need fast feedback without blowing up shared environments. Minikube and isolated namespaces help, but they need investment. We lost a solid week of productivity before we built a decent local dev story.

Habits that actually stuck

GitOps, before we called it that. Every manifest in version control. Deployments triggered by merges. No kubectl apply from someone’s laptop in production. This gives you an audit trail, makes rollbacks trivial, and kills snowflake configurations dead.

Health checks as production code. A bad liveness probe will kill your service faster than a bad deploy. We had a probe that checked a database connection with a 1-second timeout. Database had a slow moment, probe failed, Kubernetes restarted the pod, pod came up and hit the still-slow database, probe failed again. Restart loop. Cascading failure from a health check. Write your probes carefully.

Kubernetes-aware monitoring. Watching application metrics is necessary but not sufficient. You need to see what the scheduler is doing, how the nodes look, what the control plane health is. Without that, incidents are guesswork. With it, incidents become analysis.

Should you adopt it?

If you have multiple services, deploy frequently, and need real scheduling and resilience — yes. The investment pays off.

If you have a small team, a monolith, or two services behind a load balancer — probably not yet. Kubernetes doesn’t remove operational complexity. It trades one kind for another, and the new kind requires specific expertise. A team that’s struggling with basic deployments won’t be saved by Kubernetes. They’ll just struggle with more abstraction layers.

A year later

Kubernetes feels less like magic now and more like a system I can reason about. That’s the real milestone. Not “it works” but “I understand why it works, and I understand why it breaks.”

The teams I’ve seen succeed treat Kubernetes as a product they operate — with on-call, with runbooks, with regular investment. The teams that struggle treat it like something they installed once and forgot about. Same as any other infrastructure, really. The tool doesn’t care about your intentions. It cares about your configuration.

Docker won. Kubernetes is next. Go keeps earning its place. I was wrong about serverless timing and right about containers. Building two startups simultaneously taught me more about operational discipline than any conference talk ever could.

2016 felt like the year the industry collectively decided to stop debating containers and start running them. For me it was more specific than that. It was the year I stopped treating infrastructure as a thing to fight and started treating it as a thing to design.

Docker finally stopped being a toy

I spent most of 2015 defending Docker to skeptics. “It works on my machine” jokes were still the dominant reaction. By mid-2016, that conversation was over. Docker went mainstream. Not just for dev environments – for production workloads, across teams that had never touched a Dockerfile before.

At a mobility startup, we containerized everything. Location tracking, payment processing, fleet management. Docker Compose got us off the ground fast, and when we outgrew single-host scheduling, the migration path was clear. The tooling matured enough in 2016 that containers stopped being a bet and started being a default.

What changed wasn’t Docker itself – it was the ecosystem around it. Registries got better. Logging and monitoring caught up. CI pipelines learned to build and push images without a week of YAML wrestling. The boring parts got boring, which is when you know adoption is real.

Kubernetes momentum was undeniable

I wrote about this earlier this year after evaluating Swarm, Kubernetes, and Mesos side by side. My conclusion then: Kubernetes is going to win, but the operational tax in late 2016 is genuinely painful.

That assessment held up. We ran Swarm at the mobility startup for three months before migrating to Kubernetes, and both decisions were correct. Swarm gave us speed. Kubernetes gave us a real platform. The CNCF formation, Google’s backing, and the velocity of community tooling made the trajectory obvious.

What I underestimated was how fast managed Kubernetes would become available. GKE was already solid. By early 2017, the “Kubernetes is too hard to operate” argument will start losing force. That’s a good thing.

Go kept earning its place

We chose Go for our backend services at both the mobility startup and the fintech startup, and 2016 validated that decision repeatedly. The language grew in adoption. The tooling stayed excellent. The deployment story – single static binary, no runtime dependencies – kept saving us time in production.

Go 1.7 shipped with context in the standard library, which was a practical win for anyone building HTTP services with cancellation and timeouts. The ecosystem filled in noticeably. More battle-tested libraries for common patterns. Better database drivers. gRPC support matured.

The trade-offs I wrote about earlier remain real. No generics. Verbose error handling. A younger ecosystem than Java or Python. But for the kind of services we build – concurrent, networked, performance-sensitive – Go in 2016 was the right tool. Not the trendiest. The right one.

What I got wrong

I thought serverless would matter more by now. Lambda launched in 2014, and by early 2016 I expected it to be a serious architecture option for startups. It wasn’t. The tooling was immature, cold starts were painful, and the debugging experience was bad enough to kill productivity. Serverless in 2016 was interesting to watch but not interesting to use for anything beyond simple event handlers.

I also underestimated how long microservices confusion would persist. I assumed the “monolith first” message from DHH and others would settle things. It didn’t. Teams kept splitting services prematurely, then drowning in operational complexity they weren’t staffed to handle. This is still happening as I write this.

What I got right

Containers as the default deployment unit. Called it in 2015, and 2016 delivered. Kubernetes as the orchestration winner. Called it in October, and the trajectory only accelerated. Go as a serious backend language for startups. We bet on it early and it paid off consistently.

The one I’m most satisfied about: investing in operational discipline over feature velocity. At both the mobility startup and the fintech startup, I pushed hard for automated deploys, rollback capability, and monitoring before we added features. That discipline saved us multiple times this year. One outage at the mobility startup that would have been a multi-hour firefight turned into a three-minute rollback because we had the infrastructure in place.

The fintech angle

At the fintech startup, the ML side of the business grew significantly in 2016. TensorFlow matured, and our team started using it for financial content relevance scoring. The accessibility of ML tooling improved noticeably – pre-trained models, cloud GPUs, better documentation. But the gap between “can run a model” and “understands what the model is doing” remained wide. We hired for ML literacy, not just ML capability, and that distinction mattered.

Looking forward

2017 will be about consolidation. Kubernetes will keep winning and the operational burden will drop as managed offerings improve. Go will gain traction in infrastructure tooling – I expect to see more CLIs, more Kubernetes operators, more network services written in Go. Serverless will get better but will remain niche for another year at least.

The thing I care about most heading into 2017: security as a first-class engineering concern, not a compliance checkbox. GDPR is coming. The breach headlines keep getting worse. At both companies I pushed for security review in the development workflow, not as a gate at the end. That investment will compound.

Discipline over heroics. That was the lesson of 2016. The teams that shipped reliably weren’t the ones with the best engineers. They were the ones with the best habits.

We tested all three at a mobility startup. Kubernetes won on capability but nearly lost on operational cost. If you have a small team and need something running next week, pick Swarm. If you’re building for the next two years, accept the pain and invest in Kubernetes now.

The evaluation

At the mobility startup we run location-tracking services, payment processing, and a fleet management backend. All containerized. When we outgrew single-host Docker Compose, I needed an orchestrator that could schedule across hosts, handle rolling deploys, and not require a dedicated platform team to babysit.

I spent three weeks running Swarm, Kubernetes, and Mesos with Marathon side by side on identical three-node clusters. Same workload: our core API, a background worker, and a Redis instance. Here is what I found.

The comparison

My take

Kubernetes is going to win this market. The abstraction model is right. Pods, services, and declarative desired-state are the correct primitives for running containers at scale. The community momentum is unmistakable. Every infrastructure vendor is building Kubernetes integrations, not Swarm or Mesos ones.

But the operational cost in late 2016 is genuinely high. I burned two full days debugging etcd split-brain on a three-node cluster. Networking with flannel required reading source code, not documentation. RBAC is immature. Monitoring the control plane itself is a project unto itself.

Swarm is the honest answer for small teams today. We used it at the mobility startup for three months before migrating to Kubernetes, and those three months were productive. The API is Docker’s API. The mental model is simple. You can ship with it and migrate later without regret.

Mesos is the right choice if you already run it or if you genuinely need to schedule containers alongside Spark jobs and batch workloads on the same cluster. For a startup running pure container workloads, Mesos is overbuilt. The ZooKeeper dependency alone adds operational surface area that a small team doesn’t need.

What we actually did

We started on Swarm, shipped to production, and ran it for three months. When we hit the limits of Swarm’s rolling deploy controls and needed better service discovery for inter-service communication, we migrated to Kubernetes. The migration was painful but contained. It took about a week of focused work.

I would make the same decision again. Swarm bought us time to ship features instead of fighting infrastructure. Kubernetes gave us a platform we could grow into. Trying to start with Kubernetes on day one with our team size would have been a mistake.

Pick for your team, not the hype

Pick based on your team, not the feature matrix. Swarm if you need to ship now with minimal operational burden. Kubernetes if you can absorb the learning curve and want the platform that will matter in two years. Mesos if you’re already running it or have workloads that genuinely require its scheduling model.

Kubernetes is going to dominate. That doesn’t mean you have to adopt it today.

Docker won’t fix your ops. It will make bad ops louder and faster. But if you treat images like release artifacts, lock down your Dockerfiles, and build real observability around your containers, it’s the best deployment tool available right now.

We Broke Production Three Times in the First Month

At a mobility startup, we moved our backend services into Docker containers because we were tired of deploy scripts that worked on one machine and failed on another. The promise was simple: build once, run anywhere. That part was true. What nobody told us was how many new ways we would find to break things.

The first week, a developer pushed a container built on their laptop. It had a different glibc than the host. Segfault in production. The second week, we ran out of disk because nobody was cleaning up old images. The third week, a container ran as root and wrote a config file to a bind mount that the host process couldn’t read back. Three incidents, three categories of failure, all in 30 days.

Every one of those was our fault. Docker did exactly what we told it to do. We just hadn’t learned to tell it the right things yet.

Dockerfiles That Actually Work

Most Dockerfile tutorials produce images that are 800MB and run as root. That’s fine for a demo. It isn’t fine when you’re pushing 15 deploys a day across 4 services.

Here is the pattern we settled on for our Go services at the mobility startup:

FROM golang:1.6-alpine AS builder

RUN apk add --no-cache git ca-certificates

WORKDIR /src
COPY go.sum go.src ./
RUN go get -d ./...

COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app ./cmd/server

FROM alpine:3.3

RUN apk add --no-cache ca-certificates tzdata \
    && adduser -D -u 1000 appuser

COPY --from=builder /app /usr/local/bin/app

USER appuser
EXPOSE 8080
ENTRYPOINT ["app"]

A few things to notice. The multi-stage build keeps the final image small – under 20MB for most of our services. We copy dependency files first and run go get before copying the rest of the source. That means Docker caches the dependency layer, and rebuilds only take seconds when application code changes. We create a non-root user. We strip debug symbols with -ldflags="-s -w". We use ENTRYPOINT instead of CMD so the container process is PID 1 and receives signals properly.

For our Python services (we had one analytics pipeline that wasn’t worth rewriting), the pattern looked different:

FROM python:3.5-slim

RUN groupadd -r app && useradd -r -g app -d /app -s /sbin/nologin app

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
RUN chown -R app:app /app

USER app
EXPOSE 5000
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "2", "wsgi:app"]

Same principles. Dependencies first. Non-root user. Minimal base image. Explicit port. No latest tag anywhere.

Image Tagging: latest Isn’t a Version

We tag every image with the git commit SHA. Not the branch name, not latest, not a date string. The commit SHA.

docker build -t acme/location-svc:$(git rev-parse --short HEAD) .
docker push acme/location-svc:$(git rev-parse --short HEAD)

When something breaks at 2 AM, the first question is: what is running right now? With SHA tags, you can answer that in seconds. You can diff the exact code. You can roll back to the previous SHA. You can see who merged what. latest tells you nothing.

We store image metadata as labels too:

LABEL build.sha="${GIT_SHA}" \
      build.date="${BUILD_DATE}" \
      build.ci="jenkins"

It sounds like overkill. It isn’t. During an incident last month, we traced a memory leak to a specific commit because the container label told us exactly what was deployed.

docker-compose for Local Development

Our services talk to PostgreSQL, Redis, and each other. Getting all of that running locally used to mean a two-page setup guide that was always out of date. Now it’s a single file:

version: '2'
services:
  location-svc:
    build: ./services/location
    ports:
      - "8080:8080"
    environment:
      - DATABASE_URL=postgres://app:dev@db:5432/app?sslmode=disable
      - REDIS_URL=redis://cache:6379
    depends_on:
      - db
      - cache

  trip-svc:
    build: ./services/trip
    ports:
      - "8081:8080"
    environment:
      - DATABASE_URL=postgres://app:dev@db:5432/app?sslmode=disable
      - LOCATION_SVC_URL=http://location-svc:8080
    depends_on:
      - db
      - location-svc

  db:
    image: postgres:9.5
    environment:
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=dev
      - POSTGRES_DB=app
    volumes:
      - pgdata:/var/lib/postgresql/data

  cache:
    image: redis:3.0-alpine

volumes:
  pgdata:

New engineers run docker-compose up and have a working environment in under two minutes. No installing Go. No configuring Postgres. No hunting for the right Redis version. The compose file is checked into the repo, so it stays current.

One thing we learned the hard way: depends_on doesn’t mean “wait until the service is ready.” It means “start this container after that one.” Postgres takes a few seconds to accept connections. Your application will crash if it tries to connect immediately. We added retry logic to every service’s database connection code. Not elegant, but reliable.

Networking: Stop Hardcoding IPs

Docker’s default bridge network assigns IPs dynamically. If you hardcode an IP in a config file, it will work until it doesn’t, and then you will spend an hour figuring out why service A can’t find service B.

On a single host, Docker Compose gives you DNS resolution for free. Service names resolve to container IPs. http://location-svc:8080 just works. That’s good enough for development and for small production setups.

For multi-host, the story is more complicated. We use overlay networks, which let containers on different hosts communicate as if they were on the same network. It works, but it adds latency. We measured an extra 1-2ms per hop on our setup. For most services that’s noise. For the real-time bike location tracking, it was noticeable. We kept that service on a single host with host networking.

Logs: stdout or Nothing

Every one of our services logs to stdout. No log files inside the container. No custom logging directories. stdout.

log.SetOutput(os.Stdout)
log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds | log.LUTC)

Docker captures stdout and makes it available via docker logs. We run a log forwarder on each host that ships container logs to a central ELK stack. Each log line includes the container ID, image name, and service name as structured fields. When something goes wrong, we can filter by service, time range, and container in Kibana within seconds.

The mistake we made early was letting one service log at DEBUG level in production. It wrote 2GB of logs in an hour, filled up the Docker log driver’s buffer, and slowed down every other container on the host. We now enforce log level via environment variables and default to WARN in production.

Resource Limits Aren’t Optional

A container without resource limits will happily consume all available memory on the host and take everything else down with it. We learned this when our analytics service hit a data spike and OOM-killed the Postgres container running on the same host.

Every container gets explicit limits:

location-svc:
  mem_limit: 256m
  memswap_limit: 256m
  cpu_shares: 512

The memswap_limit matching mem_limit prevents the container from using swap. If it runs out of memory, it dies. That sounds harsh, but a container thrashing on swap is worse than a container that restarts cleanly. Our health checks pick up the restart within seconds.

Security: Run as Non-Root or Don’t Ship

I mentioned this in the Dockerfile section but it bears repeating. Every container runs as a non-root user. No exceptions. If a library or tool requires root, we find a different library or tool.

Beyond the user, we also:

• Use --read-only filesystem where possible, mounting only specific writable directories for temp files
• Drop all capabilities and add back only what is needed: --cap-drop=ALL --cap-add=NET_BIND_SERVICE
• Never mount the Docker socket into a container (I’ve seen this in tutorials and it terrifies me)
• Scan images for known vulnerabilities before pushing to our private registry

Our private registry runs behind TLS with basic auth. Every push and pull is authenticated. The registry itself runs in a container, which is a fun bit of recursion, but it works.

Health Checks and Restarts

Every service exposes a /healthz endpoint that returns 200 if the service can reach its dependencies:

func healthHandler(w http.ResponseWriter, r *http.Request) {
    if err := db.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        fmt.Fprintf(w, "db: %v", err)
        return
    }
    w.WriteHeader(http.StatusOK)
    fmt.Fprint(w, "ok")
}

Our load balancer polls this endpoint every 5 seconds. If a container fails three consecutive checks, it gets pulled from the rotation. Docker’s restart policy (--restart=on-failure:5) handles restarting the process. If it fails 5 times in a row, it stays down and pages us. At that point, something is fundamentally wrong and automatic restarts won’t fix it.

What I Would Tell Myself Six Months Ago

Start with one service. Not the most important one and not the least important one. Pick something that deploys frequently and has good test coverage. Containerize it, run it in production for two weeks, and fix every sharp edge you find. Then do the next one.

Don’t try to containerize your database early. We keep Postgres on a dedicated host with proper backups, WAL archiving, and monitoring. The application containers are stateless and disposable. The database is neither of those things.

Write your Dockerfiles like they will be read by someone debugging a production incident at 3 AM. Because they will be.

Treat image builds as CI artifacts. Build them in CI, tag them with the commit, push them to a registry, deploy from the registry. Never build on a developer laptop and push to production. We burned ourselves on that one.

Docker changed how we ship software at the mobility startup. Deploys went from 20 minutes of SSH-and-pray to 45 seconds of pull-and-start. But the speed only came after we did the unglamorous work of writing good Dockerfiles, setting up log aggregation, adding health checks, and enforcing resource limits. The container is the easy part. The ops around it’s the job.