More end-to-end tests isn’t the answer. Push coverage down: unit and component tests for speed, contract tests for boundaries, and a thin layer of integration tests for the flows that actually matter. If your test suite takes 30 minutes, it isn’t a test suite – it’s a deployment blocker.

Microservices promise independent deployability. What they deliver is independent failure modes that span network boundaries, team boundaries, and time zones. A contract change in Service A breaks a downstream flow in Service B, and nobody finds out until staging – or worse, production.

The instinct is to add more end-to-end tests. That instinct is wrong. E2E tests are slow, brittle, and expensive to maintain. The teams I’ve seen ship confidently do less at the top of the testing stack and more at the bottom.

Where Microservices Actually Break

They break at the seams. Not inside the service – inside the service, a monolith and a microservice look the same. They break at the HTTP boundary, the message schema, the shared database assumption, the “this field is always present” contract that nobody wrote down.

At a large consumer platform, we had a service that returned a JSON field as a string. A downstream consumer parsed it as an integer. Worked fine for months because the values happened to be numeric. Then someone added a UUID-based identifier. The downstream service started throwing parse errors in production. No unit test caught it. No integration test covered that path. A contract test would have caught it on the first PR.

The Testing Stack

Think of it as a funnel. Wide at the bottom, narrow at the top.

Unit Tests

Cover business logic, edge cases, error handling. These run in milliseconds, on every commit, with no external dependencies. If your unit tests need a database connection, they aren’t unit tests.

This is where the bulk of your coverage should live. 70-80% of your test count. At Decloud, we had a rule: if a bug reaches production, the fix must include a unit test that reproduces it. That rule alone improved our coverage more than any top-down testing initiative.

Component Tests

Exercise a single service with its internals wired together. Stub or fake external dependencies. Use testcontainers for databases and message brokers when you need realistic behavior without a full environment.

This is where you validate that your HTTP handler correctly parses the request, calls the right service method, and returns the right response code. The external dependencies are controlled, so the test is deterministic and fast.

Contract Tests

This is the layer most teams skip, and it’s the layer that prevents the most microservice-specific bugs.

A contract test verifies that a provider still meets the expectations of its consumers. The consumer defines what it expects – which endpoints, which fields, which status codes. The provider runs those expectations in its own CI pipeline. If a change breaks a consumer expectation, the build fails.

Pact is the standard tool here. It isn’t perfect, but it closes the gap between “Service A changed its response format” and “Service B found out about it three days later in staging.”

If you adopt one testing practice from this post, make it contract tests.

Integration Tests

Reserved for cross-service flows that are too risky to simulate. Order placement, payment processing, the critical paths where real money or real user data is involved.

Keep the list short. Five to ten flows, covering the business-critical happy paths and the most dangerous failure modes. If the list grows to fifty, you have an environment management problem disguised as a testing strategy.

End-to-End Tests

A handful of user journeys that represent revenue, compliance, or safety risk. Run them on a schedule or behind a merge gate, not on every commit.

E2E tests are where I’ve seen the most waste. Teams build sprawling browser-driven suites that take 45 minutes to run, fail intermittently due to timing issues, and nobody trusts the results. Kill the flaky ones. Keep the critical ones. Accept that E2E is a smoke test, not a safety net.

Test Data

The quiet killer. Shared test databases with stale fixtures. Service A’s tests depend on data that Service B’s tests modified. Nondeterministic failures that nobody can reproduce locally.

Generate test data programmatically. Keep fixtures minimal. If you need production-like data, scrub it, subset it, and version it. Determinism matters more than realism.

At a large consumer platform, we moved from shared test databases to per-test database instances using testcontainers. Test isolation improved. Flaky test rate dropped by half. The extra CI time was worth it.

CI That Matches the Layers

• Unit and component tests: every commit, every PR. Must be fast (under 5 minutes for the full suite).
• Contract tests: every PR, run in both consumer and provider pipelines.
• Integration tests: on merge to main or behind a manual gate.
• E2E tests: scheduled runs or pre-release gates.

If your fastest test layer takes more than 5 minutes, fix it before adding more tests. Developer feedback loops matter more than coverage percentages.

The Shape That Works

Push coverage down. Most of your tests should be fast, deterministic, and runnable on a laptop. Contract tests protect the seams where microservices actually break. Integration and E2E tests cover the thin slice of flows where simulation isn’t enough.

The teams that ship confidently aren’t the ones with the most tests. They’re the ones with the right tests in the right layers, running fast enough that developers actually wait for them.

Most distributed systems advice reads like a textbook. This is the shortlist I actually use. Timeouts, retries, circuit breakers, sagas, outbox/inbox, and backpressure – applied with discipline, not ceremony.

I’ve built and operated distributed systems at Verizon, AT&T, Decloud, and most recently at a large consumer platform. The failure modes are remarkably consistent across all of them. Network partitions, cascading timeouts, retry storms, stale caches. The systems that survive aren’t the clever ones. They’re the ones with boring, well-applied patterns.

This isn’t a catalog. It’s the set of patterns I keep reaching for, along with real tradeoffs and code where it helps.

Failure Handling

Timeouts and Deadlines

Every remote call without a timeout is a bug waiting to happen. I learned this the hard way at Decloud – a single downstream service that started responding in 30 seconds instead of 300 milliseconds brought down our entire checkout flow. No timeout, no deadline propagation, no circuit breaker. Just threads piling up until the JVM ran out of memory.

Propagate deadlines end to end. If the user’s request has 2 seconds left, the downstream call should know that. In Go, this is context.WithTimeout and it works beautifully:

func fetchUser(ctx context.Context, id string) (*User, error) {
    ctx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, "GET", userServiceURL+"/"+id, nil)
    if err != nil {
        return nil, err
    }

    resp, err := httpClient.Do(req)
    if err != nil {
        return nil, fmt.Errorf("fetch user %s: %w", id, err)
    }
    defer resp.Body.Close()

    var u User
    return &u, json.NewDecoder(resp.Body).Decode(&u)
}

The caller sets the budget. The callee respects it. Simple.

Retry With Backoff and Jitter

Retries fix transient errors. Immediate retries create storms. I’ve seen a single retry loop without jitter generate enough traffic to keep a recovering service down for an extra twenty minutes.

Exponential backoff with full jitter. Cap the attempts. Only retry on errors that are actually transient – a 400 isn’t transient, a 503 probably is.

func retryWithBackoff(ctx context.Context, maxAttempts int, fn func() error) error {
    var err error
    for attempt := 0; attempt < maxAttempts; attempt++ {
        if err = fn(); err == nil {
            return nil
        }

        if attempt == maxAttempts-1 {
            break
        }

        backoff := time.Duration(1<<uint(attempt)) * 100 * time.Millisecond
        jitter := time.Duration(rand.Int63n(int64(backoff)))
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(jitter):
        }
    }
    return fmt.Errorf("after %d attempts: %w", maxAttempts, err)
}

Circuit Breaker

After a threshold of errors, stop calling the failing dependency. Just stop. Return a degraded response, a cached value, or an honest error. Let the dependency recover without your traffic making things worse.

The mental model: closed (normal), open (failing, short-circuit), half-open (testing recovery). The implementation doesn’t need to be complex. A counter, a timestamp, and a threshold.

Bulkheads

At a large consumer platform, we had a service that talked to six downstream dependencies through the same HTTP client pool. One slow dependency drained the pool and every other call started timing out too. Classic.

Separate connection pools per dependency. Separate goroutine budgets. The Titanic metaphor is overused but accurate – bulkheads keep one leak from sinking the whole ship.

Rate Limiting

A predictable 429 is better than an unpredictable timeout. Always. Apply rate limits at the edge and between services. I’ll talk more about this in a future post.

Consistency and Transactions

Sagas

Distributed transactions across service boundaries don’t work. Full stop. Two-phase commit sounds great in a database textbook and falls apart the moment you have services owned by different teams with different SLAs.

Sagas replace a global transaction with local transactions and compensating actions. Two flavors:

Choreography: services react to events. Decentralized, but the flow gets hard to trace once you have more than four or five steps. Debugging a choreographed saga during an incident is an exercise in grep and prayer.

Orchestration: a coordinator drives the flow. Easier to understand, easier to audit, easier to debug. The coordinator becomes a critical path, so it needs to be durable. I default to orchestration unless the flow is truly simple.

Outbox and Inbox

You can’t publish an event and update a database atomically with two separate calls. The outbox pattern solves this: write the event to a table in the same transaction as your state change. A separate publisher reads the outbox and sends the events.

The flip side is duplicate delivery. The inbox pattern handles that – record processed event IDs at the consumer and skip repeats. Pair them together. This is the backbone of every reliable event pipeline I’ve built.

Idempotency

Make writes idempotent. Use client-provided idempotency keys. Store the result keyed by that token. If the same request shows up twice, return the same result without doing the work again.

This isn’t optional in a system with retries. If you have retries (and you should), you need idempotency.

CQRS and Read Models

Strong consistency across services is expensive and usually unnecessary. Separate your write model from your read model. Accept eventual consistency for queries, search, and reporting. The read model can be denormalized, optimized, and rebuilt without touching the write path.

Messaging

Events

Events decouple services in time and deployment. Two styles worth knowing:

• Event notification: “something happened, look it up if you care.” Lightweight, but consumers need access to the source.
• Event-carried state: “something happened, here are the details.” Heavier payload, but consumers are self-sufficient.

Keep schemas versioned and backward compatible. Breaking an event schema in production is one of those things you only do once.

Work Queues

Queues buffer work and let you process at a steady rate. Multiple consumers on the same queue give you horizontal scaling for free. Set visibility timeouts, handle retries explicitly, and always have a dead-letter queue for messages that can’t be processed.

Coordination

Leader Election

Some work must be done by exactly one node. Scheduling, cleanup, deduplication. Use leader election with lease-based expiration. If the leader dies, the lease expires, and someone else takes over.

I’ve seen teams try to avoid leader election by distributing coordination across all nodes. It always ends with split-brain bugs that take weeks to reproduce.

Distributed Locks

Use sparingly. A distributed lock should protect a small, short-lived critical section. If you find yourself holding a lock for seconds, you probably need a different design – partition by key or use a single-writer pattern.

Scaling

Sharding and Consistent Hashing

Partition by key. When you add or remove nodes, consistent hashing minimizes the data that moves. Keep shard ownership explicit and routing predictable. Implicit sharding is debugging hell.

Backpressure

When overloaded, shed work instead of collapsing. Bounded queues, rate limits, and explicit flow control. The system that says “no” gracefully is more reliable than the system that says “yes” and falls over.

This is a cultural thing too. Engineers need to be comfortable returning errors under load instead of trying to serve every request.

Observability

Correlation IDs through every log and event. Latency percentiles (not averages). Error rates per dependency. Queue depth trends. Alert on symptoms – elevated latency, increasing error rates – not on individual errors.

Without observability, you’re guessing. During an incident, guessing is expensive.

The Default Kit

If I’m starting a new service, this is what goes in on day one:

• Timeouts and deadline propagation on every remote call
• Retries with exponential backoff and jitter for transient errors
• Circuit breakers and bulkheads around dependencies
• Idempotency keys for writes, inbox deduplication for events
• Outbox pattern for reliable event publishing
• Rate limiting and backpressure at boundaries
• Correlation IDs and service-level metrics

None of this is novel. That’s the point. Distributed systems are messy by nature. Patterns make the mess predictable. Discipline over heroics.

I remain skeptical.

The problem federation solves

Federation exists because a single GraphQL server becomes a coordination bottleneck when multiple teams contribute to it. Schema changes need cross-team approval. A single deploy touches unrelated domains. One bad resolver takes down the whole API.

Federation fixes this by splitting the graph into subgraphs owned by different teams, composed behind a gateway. Each team deploys independently. The gateway stitches it together.

This is a real problem. At scale. With many teams. With a large, actively-evolving API surface.

The problem is: most teams aren’t there

Every team I’ve talked to this year that’s considering federation has fewer than 30 engineers. Most have fewer than 15. They don’t have a coordination bottleneck because they don’t have enough teams to create one.

What they have is a single GraphQL server that works. And someone read an Apollo blog post and now there’s a ticket to “federate the graph.”

Federation introduces a gateway with a query planner. It introduces composition – a build step where subgraph schemas are merged, and breaking changes are detected. It introduces entity resolution across subgraphs, which means you need to think about N+1 fetches at the gateway layer. It introduces distributed tracing across subgraph boundaries. It introduces versioning and compatibility rules for shared types.

All of that’s manageable. None of it’s free.

When it actually makes sense

You need federation when:

• Multiple teams (3+) independently contribute to the API
• Schema changes in one domain regularly block another team
• You need independent deployment of API segments
• The API surface is large enough that one team can’t own it

If those are true, federate. The organizational benefits are worth the operational cost.

If you’re a single team, or two teams that talk to each other daily, a monolithic GraphQL server with good module boundaries is simpler, faster to debug, and cheaper to operate. You don’t need a query planner. You need a code review.

The N+1 problem gets worse

GraphQL already has an N+1 problem that DataLoader solves at the resolver level. Federation moves the N+1 problem up to the gateway. The query planner decides how to fetch entities across subgraphs, and it’s easy to end up with serial round-trips that destroy your latency.

You can mitigate this with batched entity resolution and careful schema design. But it’s another thing to get right, monitor, and debug. In a monolithic GraphQL server, a slow resolver is one function call away. In a federated graph, it’s a network hop through a query planner into a subgraph you might not own.

My recommendation

If you’re using GraphQL today and it’s working, keep it. Don’t federate because a blog post told you to.

If you’re genuinely hitting coordination problems – teams blocking each other on schema changes, deploys causing cross-domain risk, ownership unclear – then federation is a reasonable solution. Invest in the composition pipeline, the observability, and the entity resolution patterns. Treat it as a significant infrastructure investment.

And if you’re still deciding whether to adopt GraphQL at all, I’ll repeat what I said last year: REST with good API design and versioning covers most use cases. GraphQL is worth it when you have diverse clients with genuinely different data needs. Federation is worth it on top of that only when you have the team scale to justify the complexity.

Most teams should stop adding layers and start shipping features.

Your gateway should be boring. Pick one pattern that matches your clients, keep it thin (auth, rate limits, routing), and treat config changes like code deploys. Most gateway disasters come from stuffing business logic where it doesn’t belong.

I’ve built or inherited API gateways at every company I’ve worked with. At Decloud we ran a single edge gateway in front of about a dozen services. At a large telecom, I walked into a gateway that had become a second application server – hundreds of Kong plugins, custom Lua transformations, business rules nobody could trace back to a ticket. It took three engineers two months to untangle.

The gateway itself was fine. The pattern choice was the problem.

When you actually need one

Not every system needs an API gateway. If you have one service and one client, a gateway is overhead. You need a gateway when you have multiple services exposed externally, or when you have distinct client types (web, mobile, partner APIs) that need different response shapes.

At Decloud we hit that threshold around service number four. Before that, we just had nginx doing TLS termination and basic routing. That was fine. The moment we needed per-client rate limits and centralized auth token validation, we added a proper gateway. Not before.

The patterns worth knowing

Edge gateway

Single entry point. All external traffic flows through it. This is where most teams should start and where many teams should stop.

You get centralized auth, rate limiting, and routing. Your services stay behind a private network. The config is one place, the logs are one place, the failure mode is one place. I like boring. Boring is debuggable.

At Decloud, our edge gateway handled TLS termination, JWT validation, rate limits, and request ID injection. That’s it. We resisted every request to add “just one more transformation.” Every single time someone wanted to add response shaping or field filtering at the gateway layer, the answer was no. Put it in a service.

Backend for Frontend (BFF)

A separate gateway per client type. Your mobile app gets one gateway that returns compact payloads. Your web app gets another. Your partner API gets a third.

I ended up recommending this at a telecom where the mobile team and web team were in a constant fight over the shape of API responses. The mobile team needed tiny payloads. The web team wanted rich nested objects. One gateway couldn’t make both happy without turning into a mess of conditional logic.

BFFs solve this cleanly. Each team owns their gateway, shapes responses for their client, and deploys independently. The tradeoff is more gateways to operate. Worth it when the alternative is a god-gateway that tries to please everyone.

Service mesh ingress

If you’re already running Istio or Linkerd inside your cluster, your mesh’s ingress gateway can handle the edge. You get mTLS, traffic shifting, and telemetry for free because the mesh is already doing that internally.

I’d only go this route if the mesh is already in place and working. Don’t adopt a service mesh just to get a gateway. That’s backwards.

What belongs in the gateway

Keep this list short. I mean it.

• TLS termination
• Auth token validation (not the auth logic itself, just “is this token valid”)
• Rate limiting
• Request ID injection
• Consistent error formatting
• Basic routing

A typical route config should be dead simple:

routes:
  - path: /api/orders
    upstream: orders-service
    timeout_ms: 3000
    rate_limit_per_minute: 120
    auth: required
    add_headers:
      X-Request-ID: $request_id

If your gateway config requires a senior engineer to review, it’s too complex.

What doesn’t belong in the gateway

This is the part people get wrong.

• Business validation. (“Is this order amount valid?” No. That’s a service concern.)
• Response aggregation across multiple services. (That’s a BFF or a dedicated orchestration service.)
• Data transformation beyond trivial header manipulation.
• Anything that requires state.

I watched a team spend six months debugging production issues that all traced back to a Kong plugin doing field-level authorization. The plugin had its own config, its own cache, its own failure modes. It was invisible to application-level monitoring. When it failed, the errors looked like they came from downstream services.

Don’t do this.

Operational discipline

A gateway is in every request path. Treat it accordingly.

Timeouts aren’t optional. At one company, I found a gateway with no timeout configuration. A slow downstream service caused requests to pile up at the gateway until it ran out of connections. The entire platform went down because one service was having a bad day. Set timeouts. Set them tight. 3-5 seconds for most APIs. If an operation takes longer, it probably shouldn’t be synchronous.

Retries need limits. One retry with jitter is usually fine. Aggressive retry policies create thundering herds. I’ve seen a gateway with 3 retries and no backoff take a struggling service from “slow” to “completely dead” in under a minute.

Config changes are deploys. Version your gateway config. Review it. Roll it out gradually. A bad config change at the gateway layer is an outage for every service behind it.

Choosing a technology

In 2021, I see three real options:

Kong if you want plugin-driven API management and fast setup. Good ecosystem, reasonable operational story. Watch out for plugin sprawl.

Envoy if you want deep traffic control and high performance. Steeper learning curve. Better fit for teams that already think in terms of proxies and traffic policy.

AWS API Gateway if you’re all-in on AWS and want managed infrastructure. Less flexibility, but less operational burden. Fine for most REST APIs.

I’ve used all three. For most teams, Kong or Envoy behind a load balancer covers everything you need. If you’re on AWS and don’t want to operate infrastructure, the managed option is perfectly reasonable.

The real lesson

The best gateways I’ve seen are the ones nobody talks about. They’re boring. They do five things well. They have clean configs, good logs, and predictable failure modes.

The worst gateways are the ones that became “the platform.” Business logic, custom transformations, complex routing rules that only one person understands. Once a gateway reaches that point, you’re not operating a gateway anymore. You’re operating a distributed monolith with extra latency.

Keep it thin. Keep it boring. Your future self will thank you.

Stop treating gRPC like REST with a different wire format. Design protos for evolution from day one, always set deadlines, use interceptors for everything cross-cutting, and test with real servers. I share the exact patterns we use at Decloud with Go code you can steal.

We moved Decloud’s internal APIs to gRPC about eight months ago. Before that, everything was REST with hand-rolled JSON serialization, version mismatches everywhere, and a weekly ritual of “why is the client sending a string where we expect an int.” Moving to gRPC fixed the type safety problem. It also introduced a whole new category of problems I didn’t anticipate.

This is what I wish someone had written down before we started. Real patterns, real Go code, real mistakes.

Why we picked gRPC (and where we didn’t)

Short version: internal service-to-service calls where latency matters and both sides are services we control. That’s it. That’s the use case.

We kept REST for:

• Anything browser-facing. gRPC-Web exists but it’s a subset and adds a proxy layer. Not worth it for us.
• Third-party integrations. Nobody wants to learn your proto schema.
• Simple CRUD admin tools. curl is a better debugger than grpcurl and I don’t care what anyone says.

The real win was code generation. We’ve Go, Python, and a small Rust service. Generating typed clients from one .proto file eliminated an entire class of integration bugs. Before gRPC, every language had its own hand-written client that drifted independently. Now drift is a compilation error.

Proto design: get this wrong and you’ll pay for years

Field numbers are forever. I mean that literally. Once you ship a proto, those field numbers are carved into every binary that’s ever been compiled against it. Get the schema design wrong early and you’re living with it or doing a painful migration.

Here’s what our service definitions actually look like at Decloud:

syntax = "proto3";

package decloud.nodes.v1;

import "google/protobuf/timestamp.proto";

service NodeService {
  rpc GetNode(GetNodeRequest) returns (Node);
  rpc ListNodes(ListNodesRequest) returns (ListNodesResponse);
  rpc WatchNodeStatus(WatchNodeStatusRequest) returns (stream NodeStatusEvent);
}

message GetNodeRequest {
  string node_id = 1;
}

message ListNodesRequest {
  int32 page_size = 1;
  string page_token = 2;
  NodeFilter filter = 3;
}

message ListNodesResponse {
  repeated Node nodes = 1;
  string next_page_token = 2;
}

message NodeFilter {
  repeated string regions = 1;
  NodeStatus status = 2;
}

message Node {
  string id = 1;
  string hostname = 2;
  NodeStatus status = 3;
  google.protobuf.Timestamp created_at = 4;
  google.protobuf.Timestamp last_heartbeat = 5;
  NodeResources resources = 6;

  reserved 7, 8;
  reserved "legacy_provider";
}

message NodeResources {
  int64 cpu_millicores = 1;
  int64 memory_bytes = 2;
  int64 disk_bytes = 3;
}

enum NodeStatus {
  NODE_STATUS_UNSPECIFIED = 0;
  NODE_STATUS_PROVISIONING = 1;
  NODE_STATUS_READY = 2;
  NODE_STATUS_DRAINING = 3;
  NODE_STATUS_OFFLINE = 4;
}

message WatchNodeStatusRequest {
  string node_id = 1;
}

message NodeStatusEvent {
  string node_id = 1;
  NodeStatus previous = 2;
  NodeStatus current = 3;
  google.protobuf.Timestamp occurred_at = 4;
}

A few things worth noting:

Every RPC gets its own request and response messages. Even if GetNodeRequest only has one field today. You will add fields later. If you use a raw string as your request type, you’ve locked yourself out of adding filters, field masks, or anything else without a breaking change.

Enums start with _UNSPECIFIED = 0. Proto3 defaults to zero. If you put a meaningful value at zero, you can’t distinguish “the client explicitly set this” from “the client didn’t set it.” We learned this the hard way with a status enum that defaulted to ACTIVE at zero. Debugging phantom active nodes wasn’t fun.

Reserve removed fields. See that reserved 7, 8 and reserved "legacy_provider"? Those are fields we removed during a redesign. Without the reservation, someone could reuse field number 7 for a completely different type and corrupt data in old clients that still have the old schema cached.

Use Timestamp not int64. I’ve seen people use epoch millis as int64 fields. Looks fine until you’re debugging across time zones and nobody remembers whether it’s seconds or milliseconds or which epoch.

Go server implementation

Here’s a stripped-down but realistic server. This is close to what we actually run:

package main

import (
	"context"
	"log"
	"net"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/health"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/status"

	pb "github.com/decloud/api/nodes/v1"
)

type nodeServer struct {
	pb.UnimplementedNodeServiceServer
	store NodeStore
}

func (s *nodeServer) GetNode(ctx context.Context, req *pb.GetNodeRequest) (*pb.Node, error) {
	if req.GetNodeId() == "" {
		return nil, status.Error(codes.InvalidArgument, "node_id is required")
	}

	node, err := s.store.Get(ctx, req.GetNodeId())
	if err != nil {
		return nil, status.Errorf(codes.Internal, "store lookup failed: %v", err)
	}
	if node == nil {
		return nil, status.Errorf(codes.NotFound, "node %q not found", req.GetNodeId())
	}

	return node, nil
}

func (s *nodeServer) WatchNodeStatus(req *pb.WatchNodeStatusRequest, stream pb.NodeService_WatchNodeStatusServer) error {
	if req.GetNodeId() == "" {
		return status.Error(codes.InvalidArgument, "node_id is required")
	}

	events := s.store.Subscribe(req.GetNodeId())
	defer s.store.Unsubscribe(req.GetNodeId(), events)

	for {
		select {
		case <-stream.Context().Done():
			return nil
		case event, ok := <-events:
			if !ok {
				return nil
			}
			if err := stream.Send(event); err != nil {
				return err
			}
		}
	}
}

func main() {
	lis, err := net.Listen("tcp", ":9090")
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}

	srv := grpc.NewServer(
		grpc.ChainUnaryInterceptor(
			loggingInterceptor,
			recoveryInterceptor,
		),
	)

	pb.RegisterNodeServiceServer(srv, &nodeServer{store: NewNodeStore()})

	hsrv := health.NewServer()
	healthpb.RegisterHealthServer(srv, hsrv)
	hsrv.SetServingStatus("decloud.nodes.v1.NodeService", healthpb.HealthCheckResponse_SERVING)

	log.Printf("listening on :9090")
	if err := srv.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

A few patterns here that took us a while to settle on:

Embed UnimplementedNodeServiceServer. This is the forward-compatible pattern. When you add a new RPC to the proto, the server still compiles. It returns Unimplemented for the new method until you add the handler. Without this, adding a method to the proto breaks every server binary.

Register the health service. This isn’t optional. Kubernetes liveness and readiness probes need it. Our deploy pipeline rejects services that don’t register the gRPC health check. No exceptions.

Chain interceptors. Logging, recovery, metrics – all cross-cutting concerns go in interceptors. Not in every handler. The ChainUnaryInterceptor API landed relatively recently and it’s much cleaner than the old single-interceptor pattern where you’d nest them manually.

Client patterns that don’t break at 3am

The client side is where most people get sloppy. Here’s what we enforce:

func newNodeClient(addr string) (pb.NodeServiceClient, func(), error) {
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	conn, err := grpc.DialContext(ctx, addr,
		grpc.WithTransportCredentials(loadTLSCredentials()),
		grpc.WithDefaultCallOptions(
			grpc.MaxCallRecvMsgSize(4*1024*1024),
		),
		grpc.WithChainUnaryInterceptor(
			retryInterceptor(3, 100*time.Millisecond),
		),
	)
	if err != nil {
		return nil, nil, fmt.Errorf("dial %s: %w", addr, err)
	}

	cleanup := func() { conn.Close() }
	return pb.NewNodeServiceClient(conn), cleanup, nil
}

func getNodeWithDeadline(client pb.NodeServiceClient, nodeID string) (*pb.Node, error) {
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

	node, err := client.GetNode(ctx, &pb.GetNodeRequest{NodeId: nodeID})
	if err != nil {
		st := status.Convert(err)
		switch st.Code() {
		case codes.NotFound:
			return nil, nil // not an error, just missing
		case codes.DeadlineExceeded:
			return nil, fmt.Errorf("node lookup timed out after 3s")
		default:
			return nil, fmt.Errorf("node lookup failed: %s", st.Message())
		}
	}
	return node, nil
}

Always set deadlines. Every single call. No exceptions. A call without a deadline is a goroutine leak waiting to happen. We had an incident where a downstream service hung and our caller accumulated ~40,000 goroutines because nobody set a timeout. The fix was one line. The outage was three hours.

Handle status codes explicitly. Don’t just check err != nil. A NotFound is fundamentally different from Internal. Your retry policy, your alerting, your user-facing message – all different depending on the code. status.Convert(err) gives you the code. Use it.

Retry with backoff, but only on the right codes. We retry on Unavailable and DeadlineExceeded. We do not retry on InvalidArgument or NotFound. Retrying a bad request is just a faster way to burn your quota.

Interceptors: the gRPC middleware pattern

This is where gRPC really shines versus REST. Interceptors are typed, composable, and they work identically for unary and streaming RPCs. Here’s our logging interceptor:

func loggingInterceptor(
	ctx context.Context,
	req interface{},
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (interface{}, error) {
	start := time.Now()
	resp, err := handler(ctx, req)
	duration := time.Since(start)

	code := codes.OK
	if err != nil {
		code = status.Code(err)
	}

	log.Printf("method=%s code=%s duration=%s",
		info.FullMethod, code, duration)

	return resp, err
}

func recoveryInterceptor(
	ctx context.Context,
	req interface{},
	info *grpc.UnaryServerInfo,
	handler grpc.UnaryHandler,
) (resp interface{}, err error) {
	defer func() {
		if r := recover(); r != nil {
			log.Printf("panic in %s: %v", info.FullMethod, r)
			err = status.Errorf(codes.Internal, "internal error")
		}
	}()
	return handler(ctx, req)
}

The recovery interceptor has saved us more than once. A nil pointer dereference in a handler used to crash the entire server. Now it returns Internal and logs the panic. The server stays up. Other RPCs keep working. We still fix the bug, but we don’t page the on-call team at 3am for a panic in a non-critical endpoint.

Error handling: the part everyone gets wrong

I’ve seen codebases where every gRPC error is codes.Internal. That’s like every HTTP response being a 500. Useless for clients. Useless for monitoring.

Our rule is simple: pick the status code that tells the client what to do.

The key insight: Internal means “the server has a bug.” If you’re returning Internal for a missing resource or bad input, you’re lying to your monitoring system. Your alerts will fire for things that aren’t server bugs. Alert fatigue follows.

Proto evolution: the non-obvious rules

Adding fields is safe. Removing them isn’t. That much is obvious. The non-obvious part:

Changing a field from string to bytes is wire-compatible but semantically different. We did this once with a field that held a UUID. Wire format was identical. But the generated Go code changed from string to []byte and broke every caller at compile time. “Wire compatible” and “API compatible” are different things.

Removing a field without reserved is a time bomb. Six months later, someone reuses that field number for a different type. Old clients that haven’t recompiled send the old type. The new server interprets it as the new type. Data corruption that only manifests in production with old client versions. Good luck debugging that.

Version at the package level. When you need a breaking change:

// Old clients still work
package decloud.nodes.v1;

// New clients use v2
package decloud.nodes.v2;

Run both versions side by side. Migrate clients one at a time. Kill v1 when the last client is gone. We track this with a Grafana dashboard that shows request counts per proto package version.

Testing: skip the mocks

We don’t mock gRPC clients. We spin up a real grpc.Server in tests with bufconn:

func setupTest(t *testing.T) pb.NodeServiceClient {
	t.Helper()

	lis := bufconn.Listen(1024 * 1024)
	srv := grpc.NewServer()
	pb.RegisterNodeServiceServer(srv, &nodeServer{store: NewMemoryStore()})

	go func() { srv.Serve(lis) }()
	t.Cleanup(func() { srv.GracefulStop() })

	conn, err := grpc.DialContext(
		context.Background(), "",
		grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
			return lis.DialContext(ctx)
		}),
		grpc.WithTransportCredentials(insecure.NewCredentials()),
	)
	if err != nil {
		t.Fatalf("dial bufconn: %v", err)
	}
	t.Cleanup(func() { conn.Close() })

	return pb.NewNodeServiceClient(conn)
}

func TestGetNode(t *testing.T) {
	client := setupTest(t)

	// Happy path
	node, err := client.GetNode(context.Background(), &pb.GetNodeRequest{NodeId: "node-1"})
	if err != nil {
		t.Fatalf("unexpected error: %v", err)
	}
	if node.GetHostname() != "worker-01.decloud.dev" {
		t.Errorf("hostname = %q, want %q", node.GetHostname(), "worker-01.decloud.dev")
	}

	// Not found
	_, err = client.GetNode(context.Background(), &pb.GetNodeRequest{NodeId: "nonexistent"})
	if status.Code(err) != codes.NotFound {
		t.Errorf("code = %v, want NotFound", status.Code(err))
	}

	// Validation
	_, err = client.GetNode(context.Background(), &pb.GetNodeRequest{})
	if status.Code(err) != codes.InvalidArgument {
		t.Errorf("code = %v, want InvalidArgument", status.Code(err))
	}
}

bufconn gives you an in-memory listener. No ports, no flaky tests from port conflicts, no network overhead. But you still test the full gRPC stack: serialization, interceptors, status codes, deadlines. Mocking the client interface skips all of that. We found real bugs with bufconn that mocks would have hidden – serialization of oneof fields, deadline propagation through interceptors, and metadata handling.

Load balancing: the HTTP/2 gotcha

gRPC runs on HTTP/2, which multiplexes requests over a single TCP connection. This means traditional L4 load balancers don’t work. All requests go to whichever backend received the connection. One hot server, N-1 idle servers.

We use client-side balancing with a service registry. The gRPC resolver API lets you plug in your own discovery:

conn, err := grpc.Dial(
	"dns:///nodes.decloud.internal:9090",
	grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)

The dns:/// scheme tells the resolver to use DNS. round_robin distributes across all A records. For us this works because we run headless Kubernetes services that return pod IPs.

If you’re behind Envoy or Istio , make sure they’re configured for HTTP/2, not HTTP/1.1 upgrade. I’ve seen service meshes silently downgrade gRPC to HTTP/1.1 and nobody notices until streaming breaks.

What I’d do differently

Eight months in, here’s what I’d change if we started over:

Invest in proto linting from day one. We use buf now but adopted it late. Early protos have inconsistent naming, missing field reservations, and enum values that don’t follow the TYPE_NAME_VALUE pattern. Fixing these in a live system is painful.

Start with buf instead of raw protoc. The protoc plugin ecosystem is a maze. buf handles code generation, linting, and breaking change detection in one tool. Should have started there.

Don’t over-decompose services . Our first instinct was one proto per entity. Node service. Deployment service. Billing service. Network service. That’s fine in theory. In practice, most operations touch three or four services, so every user-facing action became a cascade of RPCs. We’ve since consolidated the ones that always move together.

gRPC is a great tool for internal APIs. But it’s a tool, not a religion. Use it where it helps. Use REST where that’s simpler. The goal is shipping working software, not architectural purity.

Stop. You probably don’t need microservices. If you do, the strangler pattern is the only sane approach, your shared database is the real problem, and any team that can’t operate a healthy monolith will absolutely drown in a distributed system.

I’ve been building the backend for Decloud in Go since joining a deep-tech founder program earlier this year. Before that I ran engineering at a fintech startup and a mobility startup. Across all three, the microservices question came up. In two of those cases, the correct answer was “not yet.” Here’s what I’ve learned about knowing the difference.

Do you actually have a monolith problem?

Most teams that want microservices have a process problem dressed up as an architecture problem. Slow deploys, blocked releases, painful coordination – these feel like monolith issues, but usually they’re symptoms of missing automation, weak tests, or unclear ownership.

Ask yourself:

• Are multiple teams genuinely blocked by shared release cycles? Not annoyed. Blocked.
• Do different parts of your system need fundamentally different scaling or uptime guarantees?
• Do clear product boundaries already exist in your codebase, or are you hoping microservices will create them?

If you answered “no” to most of these, you don’t need microservices. You need a better monolith. Invest in CI/CD, write tests, add feature flags, and stop doing synchronized deploys.

The uncomfortable prerequisite

Here’s the part nobody wants to hear. Before you can successfully run microservices, your monolith needs to already be healthy. Specifically:

• Automated tests that run fast and actually catch regressions
• Deploys measured in minutes, not hours
• Centralized logging and alerting that someone actually looks at
• Clear module boundaries in the codebase
• Feature flags for safe releases

If you can’t do these things with one service, you won’t magically do them with twelve. Microservices multiply your operational surface area. Every gap becomes a canyon.

At the fintech startup we had a monolith handling market data, user portfolios, and news aggregation. The temptation to split was strong. But our deploy pipeline was slow and our test coverage had gaps. Splitting would have made both problems worse. We fixed the fundamentals first. Most of the pressure to split evaporated once deploys were fast and modules were cleanly separated.

When it’s actually time

Sometimes it’s genuinely time. At Decloud, we’re building a cloud infrastructure product where the billing service has completely different scaling characteristics than the provisioning engine. Billing needs to be rock-solid and auditable. Provisioning needs to be fast and horizontally scalable. Different teams own them. That’s a real reason to split.

The strangler pattern is the only approach I’d recommend:

Phase 1: All traffic hits the monolith
Phase 2: /billing routes to billing service, everything else stays
Phase 3: Most traffic hits services, small core remains

You extract one domain at a time. The rest stays stable. You can reverse any step. Progress is visible. This isn’t glamorous work. It takes months. But it’s the kind of work that actually ships.

Pick your first extraction carefully

Your first service should be boring. Not your most critical business logic – that’s your worst option. Pick something that:

• Has a small API surface
• Changes frequently (so you see the benefits fast)
• Has a clear owner
• Has obvious scaling needs separate from the rest

At the mobility startup, if we had gone the microservices route, the GPS tracking ingestion pipeline would have been the obvious first candidate. High throughput, clear boundary, independent scaling needs, and a single team owned it. The user management system? Terrible candidate. It touched everything.

The database is the real problem

Everybody focuses on splitting code. The hard part is splitting data.

Shared databases are what turn “microservices” into a distributed monolith. If two services read from the same table, you haven’t decoupled anything. You’ve just added network hops to your coupling.

Before you extract a service:

1. Map every table to its consumers. All of them.
2. Assign a single owner per dataset.
3. Expose shared data through an API, not a shared connection string.
4. Run dual-writes temporarily during migration. Emphasis on temporarily.

A transitional read view can buy you time:

CREATE VIEW payments_order_read AS
SELECT id, total_cents, currency, user_id, status
FROM orders
WHERE status IN ('pending_payment', 'paid');

But set a date to kill it. Transitional things that don’t have end dates become permanent things.

Integration: keep it boring

For service-to-service communication, my strong preference is gRPC for synchronous calls and a durable message queue for async. In Go, the gRPC tooling is excellent and the generated clients mean one less thing to get wrong.

A few rules:

• Every sync call needs a timeout, a retry budget, and a circuit breaker. No exceptions.
• Async handlers must be idempotent. You will get duplicate messages. Plan for it.
• Request IDs for tracing. Everywhere. Non-negotiable.

Skip the complex choreography patterns until your observability is genuinely mature. Start with simple orchestration. You can get clever later. You probably won’t need to.

How you know it worked

You’ll know the migration is working when deployment frequency goes up per service, not when you have more services. More services with the same deploy cadence just means you added complexity for free.

Track lead time from commit to production. Track how often deploys fail and need rollback. Track incidents caused by cross-service dependencies. If those numbers aren’t improving, the migration isn’t helping regardless of how clean the architecture diagram looks.

Microservices can genuinely unlock velocity when the conditions are right. But “our monolith feels messy” isn’t one of those conditions. Fix your house before you build an addition.

Istio is genuinely impressive technology. It’s also a complexity bomb that most teams have no business adopting.

What it actually does

The pitch is simple. You have microservices, they all need retries, timeouts, mTLS, and observability. Instead of implementing that in every service, you push it to a sidecar proxy layer. Istio manages those proxies. Fine.

In practice you get three things:

Traffic control. Route by version, headers, percentage. Canary deployments become a YAML change instead of a code change.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
  - api
  http:
  - route:
    - destination:
        host: api
        subset: v1
      weight: 90
    - destination:
        host: api
        subset: v2
      weight: 10

Mutual TLS without touching application code. This is genuinely nice. You flip a policy and suddenly everything’s encrypted in transit.

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: default
  namespace: production
spec:
  peers:
  - mtls:
      mode: PERMISSIVE

Uniform observability. Every request goes through Envoy, so you get consistent metrics and traces everywhere. Request rates, error rates, latency percentiles, distributed traces — all without instrumenting each service individually.

That’s the good stuff. And if you stopped reading here, you’d think Istio is a no-brainer.

How the thing actually works

The architecture is a control plane plus a data plane. Data plane: Envoy sidecar proxies injected into every pod, intercepting all traffic. Control plane: four components — Pilot (routing), Mixer (policy and telemetry), Citadel (certificates), Galley (config processing).

App -> Envoy -> network -> Envoy -> App

Four control plane components for what is essentially a proxy configurator. That should tell you something about the operational surface area you’re signing up for.

The part nobody talks about at conferences

Here’s what I found evaluating this at the fintech startup.

Every pod now has a sidecar. That’s extra CPU, extra memory, extra things that can fail. We saw meaningful resource overhead. Not catastrophic, but not nothing — and it scales linearly with your pod count.

Timeout alignment is a nightmare. You set a 2-second timeout at the mesh layer, but the upstream service has a 5-second timeout, and the downstream expects responses in 1 second. Now you’ve got three layers of timeout logic interacting in ways that are genuinely hard to reason about. We spent more time debugging timeout cascades than we saved by having mesh-level retries.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
  - api
  http:
  - timeout: 2s
    retries:
      attempts: 2
      perTryTimeout: 1s
    route:
    - destination:
        host: api
        subset: v1

Partial adoption is worse than no adoption. If half your services are in the mesh and half aren’t, you’ve got blind spots everywhere. Policies don’t apply uniformly. Your observability has gaps. You’re paying the complexity cost without getting the full benefit.

Egress traffic silently bypasses your policies unless you explicitly configure egress rules. We found this out the fun way.

And upgrades? Every Istio upgrade changes CRDs, defaults, sometimes both. You rehearse them or you regret it.

When it’s actually worth it

Look, if you’re running dozens of services on Kubernetes, you need consistent traffic policy across all of them, and you have the team to operate the mesh — Istio delivers. The RBAC model is solid:

apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRole
metadata:
  name: api-reader
  namespace: production
spec:
  rules:
  - services:
    - api.production.svc.cluster.local
    methods: ["GET", "POST"]
    paths: ["/api/*"]
---
apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
  name: api-reader-binding
  namespace: production
spec:
  subjects:
  - user: "cluster.local/ns/production/sa/frontend"
  roleRef:
    kind: ServiceRole
    name: api-reader

The observability story — golden signals per service, distributed traces, service topology maps — is genuinely better than bolting together per-service monitoring.

But if you’re running five services? Ten? Just write a shared library for retries and use Prometheus directly. You don’t need this.

My actual advice

Start in staging. Not production. Not “let’s just try it on one production namespace.” Staging.

kubectl apply -f install/kubernetes/istio-demo.yaml
kubectl label namespace staging istio-injection=enabled

Get comfortable with the abstractions. DestinationRule for defining service subsets, VirtualService for traffic splitting. Understand that these are two separate resources that reference each other and both need to be correct or nothing works.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api
spec:
  host: api
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

Start with permissive mTLS. Watch your metrics. Only move to strict when you’re confident nothing breaks. Then add RBAC rules one service pair at a time.

The teams I’ve seen fail with Istio all did the same thing: they adopted everything at once because the demo looked cool. The teams that succeeded treated it like any other piece of infrastructure — incrementally, skeptically, with rollback plans and runbooks.

Istio is a powerful tool. Whether it’s the right tool for you is a different question entirely. At the fintech startup, we got value from it — eventually. But “eventually” involved a lot of late nights and some colorful Slack messages that I won’t reproduce here.

Microservices don’t have a security perimeter. They have dozens. Treat every internal hop as hostile, enforce identity everywhere, encrypt everything in transit, and keep secrets out of code. The rest is details.

When we broke the fintech startup’s monolith into services, the first thing I noticed was how much implicit trust we had been leaning on. One process, one memory space, one set of credentials. Easy. Comfortable. Gone.

Microservices replace that single boundary with a mesh of network calls, each one a potential point of compromise. Every background job, every inter-service HTTP call, every gRPC stream – all of it’s attack surface now. National cyber-defense drills one thing into you: assume the network is compromised. That mindset translates directly to microservices.

This post is what I wish I’d had when we started. Practical patterns, not theory.

Start With a Threat Model

Before writing a single line of security code, sit down and think about what can go wrong. Not in the abstract. Specifically.

At the fintech startup we built our threat model around a simple premise: any internal network segment can be observed or spoofed. Paranoid? Maybe. But it forces you to build real defenses instead of relying on the warm blanket of a private VPC.

The risks that keep me up at night:

• Stolen or replayed tokens granting access long after they should
• Lateral movement – one compromised service becoming a beachhead into everything
• Over-privileged service accounts that can read data they have no business touching
• Sensitive data leaking through logs, traces, or error messages
• Secrets baked into container images or checked into git

If your threat model doesn’t scare you a little, it’s not honest enough.

Authentication: Who Are You?

Gateway Auth for External Traffic

Centralize user authentication at the API gateway. Validate the token once, attach identity headers, forward to internal services over trusted channels. Done.

Client -> API Gateway -> Services
          validate token
          attach identity headers

This keeps individual services simple. They don’t need to know about OAuth flows or token validation libraries. They just read a header. The critical trade-off: internal services must reject anything that didn’t come through the gateway. If a service accepts direct traffic, you’ve defeated the entire pattern.

Service-to-Service Identity

This is where most teams get sloppy. Internal calls between services need identity too. “It’s internal” isn’t an authentication strategy.

Two options that work:

Mutual TLS. Both sides present certificates from a trusted internal CA. Authenticates the caller, authenticates the receiver, encrypts the wire. This is the gold standard. We used it at the fintech startup for anything touching financial data.

Short-lived service tokens. JWT or similar, with a tight expiry and explicit audience claim. Works well when you can’t run a full mesh or when edge systems call internal APIs.

Pick one. Enforce it everywhere. No exceptions for “low-risk” services – those are the ones attackers pivot through.

Authorization: What Can You Do?

Authentication tells you who’s calling. Authorization decides if they’re allowed. These are separate concerns and you should keep them that way.

Centralized Policy Service

A dedicated authorization service that evaluates allow/deny decisions. You send it “service X wants to do Y on resource Z” and it answers.

Service -> AuthZ Service -> Decision

Good for complex, frequently changing rules. Good for audit trails. Bad for latency-sensitive paths – you’re adding a network hop to every decision. At the fintech startup we used this for anything involving user data access decisions. The latency hit was worth the consistency.

Embedded Policy Evaluation

Each service evaluates policy locally using a shared library or rules engine. No network hop. Fast. The downside is keeping policies in sync across dozens of services. One stale deployment and you’ve got inconsistent authorization.

Use this for latency-critical paths where the rules are stable and well-understood.

Token-Embedded Permissions

Stuff roles or scopes into the JWT itself. Simple, no extra calls needed. But tokens are snapshots – if you revoke a permission, every unexpired token still carries the old grants. Keep expiry times short. Minutes, not hours.

Works for coarse-grained access control. Falls apart when you need fine-grained, data-specific rules.

Encrypt Everything in Transit

Not just external traffic. All of it. Service-to-service, service-to-database, service-to-cache. Everything.

“But it’s a private network.” I don’t care. Private networks get breached. Network segmentation gets misconfigured. A single compromised host with tcpdump running will capture every unencrypted call in the segment.

Mutual TLS or a service mesh handles this with minimal code changes. Encryption alone doesn’t replace authorization, but it kills passive eavesdropping and makes man-in-the-middle attacks dramatically harder.

Field-Level Encryption for Sensitive Data

Some fields need protection beyond transport encryption. Payment card numbers, national IDs, health data – encrypt these at the application layer. If an intermediate proxy logs the request body or a tracing system captures the payload, the sensitive fields are still opaque.

We learned this the hard way at the fintech startup when a debug log captured a full API response including user financial preferences. Transport encryption didn’t help because the log was written on the receiving end.

Secrets Management

Hardcoded secrets are a gift to attackers. Secrets in environment variables are only slightly better – they show up in process listings, crash dumps, and container inspection output.

What actually works:

• A real secrets manager. Vault, AWS Secrets Manager, whatever. Not a config file.
• Short-lived credentials that expire before they can be exfiltrated and reused.
• Rotation on a schedule and immediately after any incident.
• No developer workstation has production secrets by default. Full stop.

The running process fetches secrets at runtime. Nothing is baked into the image. Nothing lives in source control. If I can find your database password in a git history, your security posture is theater.

Defense in Depth

No single control is enough. Layer them.

Network Segmentation

Default-deny between services. If service A doesn’t need to talk to service B, block it. Use network policies based on service identity, not IP addresses. IPs change. Service names don’t.

Input Validation on Every Boundary

Even internal calls. Especially internal calls. A compromised service sending malformed data to a downstream service shouldn’t be able to trigger a buffer overflow or SQL injection. Strict schemas, fail fast on anything unexpected.

This was hammered into us during national cyber-defense exercises. The perimeter isn’t the only place attacks happen. Assume any input can be hostile.

Least Privilege Everywhere

If a service only reads from a database table, its credentials shouldn’t allow writes. If a service only calls two other services, its network policy should block everything else. Shared credentials across services are a lateral movement highway.

Resilience Controls as Security

Circuit breakers, rate limits, and timeouts aren’t just reliability features. They’re security controls. A compromised service trying to exfiltrate data through a downstream API gets stopped by rate limits. A denial-of-service attempt gets contained by circuit breakers.

Observability Without Leaking

Log authentication failures. Log authorization decisions. Log admin actions. Use consistent request IDs so you can trace a request across services during an incident.

But – and this matters – don’t log the sensitive data itself. Log metadata. Redact by default. I’ve seen security logging implementations that were themselves a data breach waiting to happen because they captured full request bodies “for debugging.”

The goal is answering four questions fast: who did what, when, and from where.

What I’d Tell You Over Coffee

Microservices security isn’t a product you buy or a checklist you complete. It’s a set of boring, consistent patterns applied everywhere. Mutual TLS, short-lived tokens, least privilege, secrets in a vault, encrypted transit, validated inputs, observable decisions.

None of this is glamorous. Most of it’s plumbing. But I’ve seen what happens when that plumbing leaks, both in military contexts and in production systems handling real user data. The organizations that stay safe are the ones that got the fundamentals right and kept them right, not the ones that bought the fanciest tools.

Build it into the platform so individual services get security by default. Make the secure path the easy path. That’s the whole game.

Here’s what a mesh actually does: it jams a proxy next to every service instance, routes all traffic through it, and lets a control plane push config to those proxies. You get consistent retries, mutual TLS without app changes, and automatic metrics collection. Sounds great on a slide deck.

The reality is uglier. Every sidecar eats memory and CPU. Tens of megabytes per pod, plus steady overhead. At the fintech startup, we run enough services that this cost isn’t trivial, and we’re not even that big. For a mobility startup – a side project with maybe eight services – it would be absurd. You’re adding latency on every single request too. An extra hop. Maybe a few milliseconds. Doesn’t sound like much until your p99 latency budget is already tight and you just burned a chunk of it on infrastructure plumbing.

Then there’s the operational weight. A mesh isn’t something you install and forget. It’s a new control plane. New failure modes. When a request fails, is it the app? The mesh policy? The proxy? Good luck debugging that at 2am when your team is already stretched thin keeping Kubernetes itself stable.

I keep asking one question: what specific problem are you solving that you can’t solve with something simpler?

Need retries and circuit breaking? A small client library handles that. Need edge traffic control? An API gateway. Need service-to-service encryption? Application-level TLS works fine if you own the services. Need observability? Structured logging and an APM tool get you surprisingly far.

A mesh starts making sense when you have dozens of services, a complex communication graph, and genuine policy drift you can’t manage any other way. Or when security compliance demands uniform mTLS and you physically can’t retrofit every app. Those are real use cases. But most teams I talk to have 10-15 services and are adopting a mesh because it feels like the right thing to do. That’s cargo culting.

If you’re dead set on it, Linkerd is the saner choice right now. Smaller footprint, narrower scope, less to break. Istio is ambitious but heavy. Consul Connect makes sense if you’re already deep in the HashiCorp ecosystem. But honestly? Start without any of them. Add the mesh when the pain is specific and measurable. Not before.

Keep it simple. You can always add complexity later. Removing it is the hard part.

One Tuesday afternoon the enrichment service got slow. Not down. Slow. Response times climbed from 40ms to 1200ms. Because every upstream service was waiting synchronously, the entire chain backed up. Users stopped getting price alerts. The dashboard froze. We had a four-service outage caused by one service being a bit sluggish.

That was the week I started ripping out synchronous calls and replacing them with events.

Events vs. commands – this matters

An event says “this happened.” Past tense. Immutable. PriceUpdated, BikeUnlocked, UserRegistered. A command says “please do this.” It can be rejected. It can fail.

The difference isn’t pedantic. At a mobility startup, when a user unlocked a bike, we emitted a BikeUnlocked event. The billing service, the map service, the analytics pipeline – they all consumed that event independently. None of them could say “no, don’t unlock it.” That decision was already made. They just reacted.

Commands flow differently. ChargeCreditCard can fail. ReserveBike can be rejected if inventory is zero. Mixing these up causes real bugs. I’ve seen teams emit a PaymentProcessed event before the payment actually went through. Don’t do that.

The broker in the middle

The whole thing works because of a message broker sitting between producers and consumers. We used Kafka at the fintech startup – those price feeds generated serious volume and we needed ordering guarantees per instrument. At the mobility startup it was RabbitMQ. Smaller scale, simpler ops, good enough.

The broker decouples everything. The price feed doesn’t know or care that six different services consume its events. It publishes and moves on. Each consumer reads at its own pace. One goes down? The others keep running. You deploy a new analytics consumer next month? Just subscribe. The producer never changes.

This is the core win. Services stop depending on each other directly. They deploy independently. They fail independently. They scale independently.

Patterns I’ve actually used

Pub-sub is the bread and butter. At the fintech startup, a single PriceUpdated event triggered watchlist checks, portfolio recalculations, alert evaluations, and analytics writes. Four systems, zero direct calls between them.

Event sourcing we used for audit-critical flows. Instead of storing “current balance is X,” you store every event that led to X. You can replay the whole history. You can answer “what was the state at 3pm yesterday?” without building a time-travel feature. The downside: schema evolution is painful, and once your event streams get long you need snapshots or replay takes forever.

CQRS we adopted for the watchlist feature. Writes went through a command model – add stock, remove stock, set alert threshold. Reads came from a denormalised projection optimised for fast lookups. Different data shapes, different scaling needs. But you live with eventual consistency. A user adds a stock and for a few hundred milliseconds the read model hasn’t caught up. You handle that in the UI or you accept it.

Sagas we used at the mobility startup for the ride lifecycle. BikeUnlocked kicks off. If the ride ends normally, RideCompleted triggers billing. If something goes wrong – GPS lost, bike reported damaged – compensating events fire to reverse or adjust charges. No distributed transactions. Each step is its own event, each failure has a defined recovery path.

Where the real work lives

Event design is everything. A bad event is one that forces the consumer to call back to the producer for context. At the fintech startup we learned this the hard way. Our first PriceUpdated event had just the instrument ID and the new price. The notification service needed the user’s alert thresholds to decide whether to fire. So it called the user service. Synchronously. Defeating the entire point.

We fixed it by enriching the event. Not with everything, but with enough:

{
  "event": "PriceUpdated",
  "version": 3,
  "instrument_id": "AAPL",
  "price": 143.50,
  "exchange": "NASDAQ",
  "timestamp": "2017-04-09T14:30:00Z",
  "previous_close": 142.80
}

Version the events from day one. You will change the schema. You will add fields, rename things, realise you forgot something critical. If you don’t version, you break every consumer on every change.

Idempotency is non-negotiable. Kafka gives you at-least-once delivery. That means duplicates. Your consumers must handle processing the same event twice without corrupting state. Idempotency keys, deduplication windows, whatever works – but you can’t skip this.

Dead letters. When a consumer fails to process an event, it has to go somewhere visible. Not silently dropped. Not retried infinitely until it blows up the queue. A dead letter topic with alerting. We caught a serialisation bug at the mobility startup within minutes because the dead letter queue spiked. Without it, we would have lost ride events silently.

When to use it and when not to

Go event-driven when multiple services need to react to the same change, when you need audit trails, when you want services to evolve independently, when eventual consistency is acceptable. Financial data pipelines? Perfect fit. IoT-style location updates from a fleet of bikes? Great.

Don’t use it for simple CRUD apps. Don’t use it if your team can’t handle the operational overhead – you’re running a broker now, monitoring consumer lag, managing schemas. And definitely don’t use it if you need strict synchronous guarantees everywhere. A checkout flow where the user needs an immediate “payment confirmed” is a bad candidate for fire-and-forget events.

What I’d tell myself before starting

Pick the right broker for your volume and ordering needs. Kafka if you need partitioned ordering and high throughput. RabbitMQ if you want simpler operations and flexible routing. Don’t overthink it early – you can migrate later, and you probably will.

Start with one bounded context. Don’t try to event-drive the entire system in a quarter. We started with the price feed pipeline at the fintech startup. Got it stable. Understood the operational model. Then expanded.

The complexity doesn’t disappear. It moves. Instead of debugging synchronous call chains, you’re debugging event flows, consumer lag, and ordering anomalies. Different problems, not fewer problems. But the system bends instead of breaking, and at 2am that difference matters a lot.

If you have fewer than four teams and your domain boundaries are still shifting, you almost certainly don’t need microservices. You need a clean monolith and the discipline to keep it modular.

The split that cost us three months

At a mobility startup we had a Go monolith that handled bike availability, user accounts, payments, and ride tracking. It was about 40k lines, well-tested, deployed in under two minutes. It worked.

Then we decided to extract payments into its own service. The reasoning sounded right: payments are sensitive, they have different scaling characteristics, and we wanted to isolate failures. Classic microservices pitch.

What actually happened: we spent three months building the extraction. We needed a new deploy pipeline, a message contract between services, retry logic for the network boundary, and a way to keep ride state consistent when the payment service was slow or down. We went from a function call that took microseconds to a network call that introduced latency, partial failures, and a new category of bugs we had never dealt with before.

The team was four engineers. We could have spent those three months shipping features. Instead we shipped infrastructure.

The real cost of splitting early

Microservices solve an organizational problem, not a technical one. When you have multiple teams that need to ship independently without stepping on each other, services aligned to team boundaries are powerful. That’s a real benefit.

But most teams I see adopting microservices don’t have that problem. They have five to ten engineers, a shared codebase, and a product that’s still changing shape weekly. At that stage, splitting into services means:

More deploy pipelines to maintain. Each service needs its own CI, its own monitoring, its own alerting. That’s real maintenance overhead for a small team.

Distributed data pain. A query that was a simple SQL join becomes a cross-service call. A transaction that was ACID becomes an eventually consistent workflow. At the mobility startup, a single “end ride and charge user” operation went from one database transaction to a choreography of events across two services with compensating actions for failure cases. The code tripled in complexity.

Testing gets hard. A monolith test suite runs in one process. Service tests require either mocking everything (which hides real bugs) or running integration environments that need their own care. We went from go test ./... taking 30 seconds to an integration suite that took 12 minutes and broke for infrastructure reasons as often as code reasons.

Debugging gets hard. A stack trace in a monolith tells you exactly what happened. A distributed trace across services requires correlation IDs, centralized logging, and tracing infrastructure. In 2016, that tooling is still immature.

When a monolith is the right call

A monolith isn’t a dirty word. A well-structured monolith with clear package boundaries, explicit interfaces between modules, and owned data per module gives you most of the architectural benefits of services without the operational tax.

At the fintech startup we kept the backend as a single deployable for much longer than conventional wisdom suggested. Financial news ingestion, NLP processing, user-facing API, all in one app. The key was strict internal boundaries. The NLP module exposed a clean interface. The API layer never reached into ingestion internals. Data ownership was explicit even though it shared a database.

This let us move fast. A new feature that touched multiple concerns was a single PR, a single deploy, a single rollback if something went wrong. We weren’t coordinating releases across services or debugging network failures between components that used to be function calls.

The modular monolith

The pattern I keep coming back to is the modular monolith. One deployable unit, strict internal boundaries, explicit interfaces between modules. You get the code organization benefits without paying the distributed systems tax.

The key disciplines:

• Each module owns its data. No reaching into another module’s tables.
• Modules communicate through defined interfaces, not by importing each other’s internals.
• Dependencies between modules are visible and intentional.

This isn’t easy. It requires code review discipline and a team that cares about boundaries. But it’s dramatically simpler than operating a fleet of services, and it keeps the option to extract a service later when you have a concrete reason.

When you actually need services

There are legitimate reasons to split. I’ve seen three that hold up in practice:

Radically different scaling needs. If one component handles 100x the traffic of everything else, separating it can save real money and improve reliability. At the mobility startup, the bike location tracking service eventually did need to be separate because it processed GPS updates at a rate that would have required over-provisioning the entire monolith.

Independent team ownership. When you have genuinely separate teams that are blocked by coordinated releases, aligning service boundaries to team boundaries unblocks delivery. But this is a team problem, not a technology problem. If you have one team, you don’t have this problem.

Hard compliance boundaries. PCI, SOX, specific regulatory requirements that mandate isolation. These are real constraints, not aspirational architecture goals.

Notice what isn’t on the list: “because Netflix does it” or “because we might need to scale someday.” Premature optimization applied to architecture is just as wasteful as premature optimization applied to code.

If you’re going to split, do it incrementally

If you have a monolith and genuine reasons to extract a service, don’t rewrite. Extract one piece. Run it alongside the monolith. Learn everything about operating two things instead of one. Then decide if the trade-off was worth it before extracting the next piece.

The operational foundations you need are the same ones that make a monolith healthy: good logging, monitoring, deployment automation, and incident response. Build those first. They pay off regardless of your architecture.

The bottom line

Microservices are a trade-off, not an upgrade. They buy organizational independence at the cost of operational complexity. For most teams I talk to, the honest answer is: you aren’t big enough to need them yet, and your monolith isn’t the thing slowing you down.

Ship features. Keep your boundaries clean. Split when the pain of coordination is real and measured, not hypothetical. That’s the job.