Quick take
You don’t need Datadog’s enterprise tier. You need structured logs, one good dashboard per service, alerts that don’t cry wolf, and a request_id on everything. That’s 80% of it.
I’ve been working with distributed teams for most of this year. Some are five people across three time zones. Some are thirty across eight. The pattern I keep seeing: they either have zero observability or they went full enterprise cargo cult and now nobody can find anything.
There’s a middle ground. I want to talk about that.
The actual problem with distributed teams
In an office, someone notices something is slow. They say it out loud. Someone else goes “oh yeah, I deployed ten minutes ago.” Problem found in under a minute.
Remote? That same issue sits in a Slack thread for 45 minutes while people in different time zones wake up, read context, and try to figure out what changed. I’ve watched this happen. Repeatedly.
The fix isn’t more tools. It’s making your systems capable of answering basic questions without requiring a human to be online at the right moment.
Three questions. That’s it:
- Is this thing broken right now?
- What changed recently?
- Where do I look next?
If your setup can answer those, you’re ahead of most teams I’ve worked with.
Enterprise observability isn’t your observability
Google has 10,000 SREs. They built custom everything. When you read their SRE book and try to implement the same stack with your team of eight, you end up with:
- A Prometheus instance nobody configured alerts for
- Grafana dashboards copied from a blog post that don’t match your services
- Jaeger running but with 0.1% sampling so traces are useless when you actually need them
- An ELK stack eating 40% of your infrastructure budget
I’ve seen this exact setup at three different companies this year. Not exaggerating.
What you actually need
Here’s my stack recommendation for a small distributed team. Opinionated, yes. But it works.
Logs: Structured JSON to a managed service. Loki if you’re cheap. Papertrail if you want simple. The key is structured, not the tool. Every log line should look roughly like this:
{
"level": "error",
"msg": "payment failed",
"service": "checkout",
"request_id": "7f3c2c4d",
"user_id": "u_123",
"error": "card_declined",
"duration_ms": 340
}
Same field names. Every service. No exceptions. The request_id alone will save you hours of debugging per incident. I can’t stress this enough. Propagate it through HTTP headers, queue messages, background jobs. Everywhere.
Metrics: Prometheus + Grafana. Still the best bang for buck. But here’s the thing – don’t build 30 dashboards. Build one per service. Three panels:
- Request rate and error rate (tells you if something is broken)
- Latency percentiles (tells you if it’s degrading)
- Recent deploys and config changes overlaid on the graphs (tells you what caused it)
That’s your dashboard. If a new engineer can’t look at it and understand the health of the service in 30 seconds, strip it down further.
Traces: Jaeger or Zipkin, but only if you have more than two services. If you’re a monolith with a database and a cache, traces are overhead you don’t need yet. Just use request IDs in your logs. Seriously.
When you do need traces, bump sampling to at least 10% on critical paths. 0.1% default sampling means you’ll never have a trace for the request that actually broke.
Alerts: Less is more. Every alert that pages someone at 3am and isn’t actionable is erosion of trust. Once your team stops trusting alerts, you’ve lost. They’ll start muting channels. I’ve seen it happen.
My rule: every alert needs three things.
- A condition that’s actually abnormal (not “CPU above 60%”)
- A link to the relevant dashboard
- A link to a runbook that says what to do first
If you can’t write those three things for an alert, the alert shouldn’t exist.
The request_id sermon
I keep coming back to this because it’s the single highest-leverage thing you can do.
At the fintech startup we had services talking to services talking to queues talking to workers. When something went wrong without correlation IDs, the debugging process was: check this log, then that log, then maybe this other log, hope the timestamps line up, piece it together manually. Took forever.
After we standardized on a single request_id header propagated everywhere? Same investigation. One search. Done.
The implementation is trivial. Middleware that reads X-Request-ID from incoming requests. Generates a UUID if missing. Passes it along. Logs it on every line. Takes an afternoon to implement across your whole stack.
An afternoon of work for months of saved debugging time. That’s the kind of trade I like.
Runbooks: the unsexy high-leverage tool
Nobody wants to write runbooks. I get it. But here’s the scenario: it’s 2am in your time zone. The alert fires. The person on call is in a different country. They’ve been on the team for three weeks.
Without a runbook, they’re messaging people, waiting for responses, guessing. With a runbook, they open it, follow the steps, and either fix it or know exactly who to escalate to.
Keep them short. Keep them next to the code. Update them after every incident. A runbook that says “check the database connection pool, then check Redis, then check the upstream API timeout” is worth more than a 50-page incident response process document nobody has read.
Mistakes I keep seeing
Collecting everything. Storage is cheap. Cardinality explosions aren’t. I watched a team’s Prometheus instance fall over because they added a user_id label to a counter. Millions of time series. Dead monitoring system. During an outage. Ironic.
Dashboard graveyards. Thirty dashboards, twenty-eight of which nobody has looked at in months. Two of which are actually useful but you can’t remember which ones. Delete aggressively.
Happy path instrumentation only. Your error paths need more instrumentation than your happy paths. The happy path works. You know this because nobody is complaining. The error paths are where surprises live.
Separate conventions per team. One team calls it user_id, another calls it userId, a third calls it uid. Now your cross-service queries are a mess. Pick a convention. Enforce it in code review. This is boring work that pays off enormously.
What to measure about your observability itself
One meta-metric I track: time from “something seems wrong” to “I know what changed and where to look.” If that number is going down over time, your observability is working. If it’s flat or going up, you’re adding complexity without adding clarity.
The other one: how many alerts fired this week that didn’t need a human response? If it’s more than 20%, you have a noise problem.
Start here
If you’re starting from scratch with a small distributed team, do this in order:
- Structured JSON logs with a shared
request_id. One week of work, max. - One Grafana dashboard per service with the three panels I mentioned. Another week.
- Three to five alerts that are actually actionable. A few days.
- Short runbooks for those alerts. A day.
That’s a month of work spread across your team. After that, you have a system that answers the three questions. Everything else – traces, SLOs, error budgets, custom metrics – layer it on when you feel the pain, not before.
Don’t let perfect be the enemy of “I can actually debug production at 2am without waking up three people.”