Your eval suite passes. Your staging environment looks good. Your AI feature will still break in production because real users do things your test set never imagined. Shadow it, canary it, measure it, and make every rollout reversible. Evidence before confidence.

I wrote about testing in production back in 2019. The core thesis hasn’t changed: staging lies to you. What has changed is that AI makes the lying worse.

Traditional software either works or it doesn’t. The test passes or fails. The API returns the right data or throws an error. AI features exist in a gray zone where the output is almost always plausible, sometimes correct, and occasionally dangerous. Your test suite can’t cover this space. Production can.

Why offline evals aren’t enough

Every AI project should have an eval suite . I’ve been saying this for over a year. But evals test known scenarios. Production surfaces the unknown ones.

Real users send inputs your test set never imagined. They misspell things. They paste in multi-language text. They include personally identifiable information that triggers different model behavior. They ask questions that are ambiguous in ways your eval prompts aren’t.

At one company, their AI support agent passed every eval with flying colors. In production, users started treating it like a search engine – pasting in order numbers and expecting it to look up status. The model happily hallucinated order details instead of saying “I can’t do that.” The eval suite had no test case for “user treats chatbot like a database query tool.” Production found it in the first hour.

Shadow mode first

Before any AI change touches a real user, shadow it. Run the new version in parallel with the current one, compare outputs, and log everything. The user only sees the current version.

Here’s the pattern I use in Go:

type ShadowRunner struct {
	current   ModelClient
	candidate ModelClient
	logger    *ShadowLogger
}

func (s *ShadowRunner) Execute(ctx context.Context, req Request) (Response, error) {
	// Current model serves the user
	resp, err := s.current.Complete(ctx, req)

	// Candidate runs in background -- never blocks the user
	go func() {
		candidateCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
		defer cancel()

		candidateResp, candidateErr := s.candidate.Complete(candidateCtx, req)
		s.logger.LogComparison(ShadowResult{
			RequestID:      req.ID,
			CurrentOutput:  resp,
			CandidateOutput: candidateResp,
			CandidateErr:   candidateErr,
			Match:          s.compareOutputs(resp, candidateResp),
		})
	}()

	return resp, err
}

The shadow logger captures every comparison. I review divergences daily during the shadow period. If the candidate produces different outputs, I want to understand whether those differences are improvements, regressions, or neutral changes.

The shadow period should last at least a week. Longer for high-traffic services. The goal is to see enough real-world input diversity to have confidence in the change.

Canary with kill switches

Once shadow results look good, move to a canary deployment . Route a small percentage of real traffic to the new version and monitor closely .

type CanaryRouter struct {
	current     ModelClient
	candidate   ModelClient
	percentage  atomic.Int32
	qualityGate *QualityGate
}

func (c *CanaryRouter) Route(ctx context.Context, req Request) (Response, error) {
	if c.shouldCanary(req.UserID) {
		resp, err := c.candidate.Complete(ctx, req)
		if err != nil || !c.qualityGate.Check(resp) {
			// Automatic fallback to current
			return c.current.Complete(ctx, req)
		}
		return resp, err
	}
	return c.current.Complete(ctx, req)
}

func (c *CanaryRouter) shouldCanary(userID string) bool {
	hash := fnv.New32a()
	hash.Write([]byte(userID))
	return int(hash.Sum32()%100) < int(c.percentage.Load())
}

The QualityGate is the part most teams skip. It checks the candidate response against basic quality criteria before serving it. If the response fails the gate, the user gets the current version transparently. No harm done.

I start at 1%. Watch for a day. If quality signals hold, move to 5%. Then 25%. Then 100%. Each step gets at least a few hours of observation. If anything looks off at any step, roll back to the previous percentage. No drama.

The hash-based routing is important: the same user always gets the same version within a rollout step. This prevents confusing experiences where the same user gets different quality outputs on consecutive requests.

What to measure during rollout

Three categories of signals, checked at every rollout step:

Quality signals. Task success rate on your eval set. But also: user re-prompts (did they have to ask again?), abandonment rate (did they give up?), explicit negative feedback. These are the signals your eval suite can’t give you.

Safety signals. Refusal rate. Policy trigger count. Anything flagged by your content filters. If the candidate model refuses more or fewer requests than the current one, investigate before expanding.

Operational signals. Latency p50 and p95 by workflow. Token usage. Cost per request. Error rates. A model change that improves quality but doubles cost might not be a net win. Make that trade-off explicit.

type RolloutMetrics struct {
	Version         string
	QualityScore    float64
	RefusalRate     float64
	P50Latency      time.Duration
	P95Latency      time.Duration
	CostPerRequest  float64
	ErrorRate       float64
	UserRepromptRate float64
}

func (m *RolloutMetrics) PassesGate(baseline RolloutMetrics) bool {
	if m.QualityScore < baseline.QualityScore*0.95 {
		return false // quality regression > 5%
	}
	if m.ErrorRate > baseline.ErrorRate*1.5 {
		return false // error rate increase > 50%
	}
	if m.P95Latency > baseline.P95Latency*2 {
		return false // latency doubled
	}
	return true
}

These thresholds aren’t magic numbers. They’re product decisions. A 5% quality regression might be acceptable if cost drops by 40%. A latency doubling might be fine for a background task but fatal for a chat interface. Define them before the rollout starts, not during.

The one-change rule

Never change the model and the prompt at the same time. If quality drops, you won’t know which change caused it. This sounds obvious. I’ve watched four different teams make this mistake in the last three months.

Ship the prompt change. Measure. Ship the model change. Measure. If you must change both, do the prompt first because it’s cheaper to roll back.

Same goes for retrieval changes, system message changes, and tool configuration changes. One variable at a time. Anything else is debugging in the dark.

Holdout baselines

Keep a small, stable slice of traffic permanently on a known-good version. This is your holdout. It tells you whether quality changes are due to your changes or due to shifts in user behavior, input distribution, or upstream data.

Without a holdout, slow regressions look like normal variance. You won’t notice a 2% quality drop per week because no individual week looks bad. But your holdout will show the cumulative drift loud and clear.

What matters

Testing AI in production isn’t reckless. Shipping AI without testing it in production is reckless. Offline evals give you a baseline. Shadow mode gives you confidence. Canaries give you safety. Holdouts give you ground truth.

Every rollout should be reversible, measurable, and attributable to a single change. That isn’t a testing philosophy. That’s engineering discipline applied to a system that fails in ways your test suite can’t anticipate.

An AI-powered feature is in production. Uptime: 99.9%. Latency: nominal. Error rate: near zero. Dashboards are green. Everyone is happy.

Except the answers are wrong 15% of the time, and nobody knows because nothing is measuring answer quality. The system is healthy. The outputs are not.

This is the fundamental gap in AI observability . Traditional monitoring tells you whether the service is running. It does not tell you whether the service is useful.

Why AI systems fail silently

A classic API returns structured data. If the response is malformed, you get a parse error. If the logic is wrong, a test catches it. The failure modes are usually loud and obvious.

AI systems fail quietly. The model returns a perfectly formatted response with a confident tone and completely wrong content. The HTTP status is 200. The latency is fine. The JSON is valid. And the user just got told that their refund was processed when it wasn’t.

At a fintech startup, we had a similar problem with our financial news summarization pipeline, long before the current AI wave. The summaries looked plausible but occasionally attributed quotes to the wrong CEO or mixed up fiscal quarters. The system was “working” by every operational metric. The outputs were unreliable. We caught it only because a user complained, not because monitoring flagged it.

The lesson stuck with me. You can’t monitor AI like you monitor a REST API. You need different signals.

The signals that actually matter

I use a simple framework with five categories. If you are not tracking all five, you have blind spots.

Traceability. For every response, you need to know: which model, which prompt version, which retrieved context, which tool calls. If you can’t reconstruct why the model said what it said, you can’t debug a bad answer. You’re just guessing. I store a trace object alongside every response that includes model ID, prompt hash, retrieval IDs, and tool call logs. When something goes wrong, the trace is the first thing I pull.

Quality signals. This is the hard one. You need some measure of whether the output was good. Heuristic checks catch obvious failures: empty responses, responses that are too long or too short, and responses that contain known-bad patterns. Sampled evaluation catches the subtle failures: a human or a second model scores a random slice of outputs against a rubric. Neither is perfect. Together they cover enough ground.

Cost per outcome. Not cost per request, cost per successful outcome. A system that gets it right on the first try costs less than one that needs three retries and a human escalation. Track the full cost of getting to a good answer, including retries, fallbacks, and human review. This number will surprise you.

Safety and policy. Refusal rates, blocked content, policy trigger counts. If your refusal rate spikes, something changed – either the inputs or the model behavior. If it drops to zero, something might be wrong too. These are canary signals.

Operational basics. Latency percentiles by workflow (not globally – global averages hide everything), error rates with reason codes, token usage trends. The same stuff you track for any API, but broken down by the AI-specific dimensions that matter.

The prompt versioning problem

Here is something that bites almost every team. Someone changes a prompt. Quality drops. Nobody connects the two events because the prompt change was not tracked alongside the quality metrics.

Treat prompts as production code. Version them. Deploy them through your normal release process. Tag every response with the prompt version that produced it. When quality dips, the first question should be: what changed since the last known-good state?

I version prompts in the same repo as the service code. A prompt change gets a PR, a review, and a run against the eval suite before it hits production. It sounds like overkill until the first time it prevents a regression. Then it sounds obvious.

Keep it lean

The temptation is to build a dashboard for everything. Do not. Start with the minimum set of signals that lets you answer one question: “A user reported a bad answer. Can I explain why it happened and prevent it from happening again?”

If you can answer that question end-to-end, your observability is good enough. If you can’t, no amount of dashboards will save you.

Log the trace. Track quality. Version your prompts. Measure cost per outcome, not cost per request. That’s the baseline. Everything else is optimization.

Treat LLM calls like you treat database calls in OpenTelemetry: trace them, measure them, and alert on quality drift – not just errors. Your Grafana dashboard showing 200 OK and low latency means nothing if the model is hallucinating.

Two weeks after we shipped the transaction categorization feature at a fintech company, OpenAI quietly changed something in their model inference. Our API calls still returned 200. Latency was normal. The logs looked clean. But accuracy dropped from 89% to 86% overnight, and the confidence score distribution shifted in a way that pushed more borderline results above our threshold.

We caught it because we had quality monitoring in place. If we’d been relying on traditional application monitoring – error rates, latency percentiles, throughput – we’d have missed it entirely. The system was “healthy” by every standard metric. It was just less correct.

This is the fundamental problem with LLM observability. The output can be fluent, well-formatted, and completely wrong. Your existing monitoring stack doesn’t know the difference.

What LLM observability actually means

For traditional services, observability is the RED metrics: Rate, Errors, Duration. For LLMs, you need those plus an entire quality dimension that doesn’t exist in normal software.

I think about it in three layers:

Infrastructure layer. The basics. API availability, latency (p50, p95, p99), error rates by type (rate limits, timeouts, 500s), token usage, cost. If you’re using OpenTelemetry – and you should be – this maps cleanly onto standard span attributes.

Model behavior layer. This is the new part. Output quality, confidence distributions, format compliance, fallback rates, and drift detection. These signals require domain-specific instrumentation, not just HTTP metrics.

Product outcome layer. User actions on model output: acceptance, edits, rejections, re-prompts. This is where you learn whether the model is actually helping or just producing plausible noise.

How we instrumented it

I’m a big OpenTelemetry advocate. We use it at a fintech company for everything, and extending it to LLM calls was a natural fit. The key insight: treat each LLM call as a span in your trace, with custom attributes for the things you need to measure.

Here’s the shape of what we log for every LLM call:

span.name: "llm.completion"
span.attributes:
  llm.model: "gpt-3.5-turbo-0613"
  llm.prompt_template: "tx-categorize-v3"
  llm.prompt_tokens: 847
  llm.completion_tokens: 124
  llm.total_tokens: 971
  llm.temperature: 0.0
  llm.confidence_score: 0.91
  llm.output_valid: true
  llm.fallback_used: false
  llm.cost_usd: 0.0015

The prompt template version is critical. Without it, you can’t correlate quality changes with prompt changes. We version our prompts like we version our API: tx-categorize-v3 tells us exactly which instructions the model received.

We also log a hashed version of the input (for cardinality analysis) and the full output (for sampling and review). The full input gets logged to a separate, access-controlled store because transaction data is sensitive. Don’t dump PII into your standard telemetry pipeline. I’ve seen teams do this. Don’t be that team.

The quality metrics that actually matter

After running this for a couple of months, here’s what I actually look at:

Accuracy on the eval set. We run our 200-transaction eval set daily against the live model. This is the canary. If this number drops, something changed – either in our code, our prompts, or the model itself. We alert at a 2-point drop sustained for two consecutive runs.

Confidence score distribution. Not the average – the distribution. A shift in the shape of the histogram tells you more than the mean. When the model gets less confident on average, it means the input distribution has changed or the model itself has drifted.

Fallback rate. What percentage of requests hit our rules-based fallback instead of using the model output? This is a compound signal: it reflects both model quality and input quality. A spike in fallback rate is always worth investigating.

User correction rate. How often do users change the model’s output? This is the ground truth. Automated metrics are approximations. User corrections are direct feedback. We track this weekly and use it to update our eval set.

Cost per successful categorization. Not cost per API call. Cost per result that the user accepted without correction. This metric keeps us honest about whether quality improvements are actually cost-effective.

Dashboards for different audiences

I built three dashboards. Each one serves a different question.

The ops dashboard shows error rates, latency, rate limit hits, and cost. This is for on-call. It answers: “Is the LLM integration working?” Standard SRE stuff, just with LLM-specific dimensions.

The quality dashboard shows accuracy trends, confidence distributions, fallback rates, and eval set results. This is for the engineering team. It answers: “Is the model producing good results?” This is the one we check every morning.

The product dashboard shows user correction rates, acceptance rates, and task completion metrics. This is for the PM. It answers: “Is this feature helping users?” This is the one that justifies the feature’s existence.

The temptation is to build one mega-dashboard. Resist it. Different audiences need different views, and combining them creates noise that ensures nobody looks at any of it.

Alerting without drowning

The biggest mistake I see: alerting on every metric at static thresholds. LLM behavior is inherently variable. You’ll page yourself into exhaustion.

What works better: alert on sustained deviations from a rolling baseline. Our eval accuracy fluctuates between 87% and 91% day to day. That’s normal. An alert at 85% absolute threshold would fire once and be ignored. An alert on “3+ consecutive days below the 7-day moving average minus 2 points” catches real regressions while ignoring noise.

For cost, we alert on day-over-day percentage increases above 50%. This catches both sudden spikes (runaway retry loops) and gradual creep (prompt bloat) depending on the time window.

The honest summary

LLM observability isn’t a new discipline. It’s regular observability plus quality measurement. If you’re already doing OpenTelemetry tracing, adding LLM spans is straightforward. If you’re already running eval sets, automating them on a daily cadence is a small step.

The hard part isn’t the tooling. The hard part is accepting that “the service is up” isn’t the same as “the service is working.” For LLMs, the gap between those two statements is where all the interesting failures live.

Monitor the quality. Version your prompts. Run your eval set daily. Everything else is optimization.

OpenTelemetry tracing hit 1.0 this year and it’s the real deal. Adopt it now for tracing. Be cautious with metrics – the API isn’t finalized. Ignore logs for now. Deploy the Collector as your telemetry gateway, standardize your resource attributes on day one, and configure sampling before you drown in data. The vendor lock-in argument alone makes this worth the migration effort.

I’m tired of observability vendor lock-in. Every organization I work with has a different combination of Datadog, New Relic, Jaeger, Prometheus, and three logging tools. Switching backends means rewriting instrumentation across dozens of services. Correlating traces with metrics requires duct tape and prayer.

OpenTelemetry fixes this. One instrumentation standard, any backend. It’s the most important infrastructure project nobody is talking about enough.

What Is Actually Stable Right Now

As of November 2021, here is the honest state:

Tracing: production-ready. The API and SDKs hit 1.0. Go, Java, Python, JavaScript, .NET all have stable implementations. Context propagation works across HTTP, gRPC , and most messaging systems. This is safe to adopt today.

Metrics: getting there. The metrics API isn’t finalized. SDKs are in various stages of beta. You can start experimenting but I wouldn’t bet a production monitoring pipeline on it yet. Give it six months.

Logs: early. On the roadmap. Not a reason to adopt OTel today. Keep your existing log pipeline.

This maturity gap matters for planning. Don’t try to adopt all three signals at once. Start with tracing, add metrics when the API stabilizes, leave logs alone.

Start With the Collector

The single best decision you can make is deploying the OpenTelemetry Collector before you instrument a single service. The Collector sits between your applications and your backends. Applications export to it via OTLP. It forwards to whatever backend you use.

Why this matters: when you inevitably switch observability vendors (and you’ll), you change the Collector config. Not your application code. Not a hundred services. One config file.

A basic Collector config for forwarding traces to Jaeger:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger-collector:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

Deploy this as a sidecar or a standalone service. I prefer standalone in most cases – easier to manage, easier to scale, and you avoid coupling the Collector lifecycle to your application pods.

Instrumenting a Go Service

Here is what basic OTel tracing looks like in Go. I use this as my starting template for new projects.

Set up the trace provider at application startup:

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, fmt.Errorf("creating OTLP exporter: %w", err)
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("orders-api"),
            semconv.ServiceVersionKey.String("1.4.2"),
            attribute.String("environment", "production"),
        )),
        sdktrace.WithSampler(
            sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)),
        ),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})
    return tp, nil
}

Then instrument your HTTP handlers:

func handleOrder(w http.ResponseWriter, r *http.Request) {
    ctx, span := otel.Tracer("orders-api").Start(r.Context(), "handleOrder")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.customer_id", r.Header.Get("X-Customer-ID")),
    )

    order, err := processOrder(ctx, r)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        http.Error(w, "order processing failed", http.StatusInternalServerError)
        return
    }

    span.SetAttributes(attribute.String("order.id", order.ID))
    json.NewEncoder(w).Encode(order)
}

The ctx parameter carries trace context through your call stack. Pass it everywhere. Every downstream call that receives this context becomes part of the same trace. This is how you get end-to-end visibility across services .

Sampling: Configure This on Day One

I can’t stress this enough. Default sampling is 100% – every request gets traced. That’s fine for development. In production with any real traffic, you’ll generate terabytes of trace data and your observability bill will make your CFO cry.

Set up parent-based sampling with a ratio. 10% is a good starting point for most services. Critical paths can be sampled at higher rates.

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Parent-based means if an upstream service already decided to sample this request, downstream services honor that decision. This keeps traces complete instead of fragmented.

Resource Attributes: The Thing Everyone Gets Wrong

Resource attributes are metadata attached to every span your service produces. Service name, environment, version. They’re how you filter and correlate traces across your entire fleet.

Get these right from day one. I’ve seen migrations stall for months because teams used inconsistent service names and nobody could query across services.

Standardize at minimum:

• service.name – unique, lowercase, hyphenated
• service.version – semver, from your build
• deployment.environment – production, staging, development
• service.namespace – team or domain grouping

Enforce these through the Collector. Use the resource processor to inject defaults and reject spans missing required attributes.

The Migration Path

Don’t try to migrate everything at once. This is the sequence I follow:

Week 1-2: Deploy the Collector. Configure it to export to your current backend. No application changes yet.

Week 3-4: Instrument one new service (or a non-critical existing one) with OTel. Verify traces show up in your backend. Fix any context propagation gaps at service boundaries.

Month 2: Migrate 2-3 critical services from vendor SDKs to OTel. Run both in parallel for a week to verify data parity.

Month 3+: Expand coverage. Set up dashboards and alerts on the new pipeline. Once confident, remove vendor SDK instrumentation.

This sequence keeps your existing observability intact while you build confidence in the new pipeline. Nobody loses visibility during the migration.

Pitfalls I’ve Hit

Inconsistent service naming. One team calls it orders-api, another calls it OrdersAPI, a third calls it orders. Now you can’t query across services. Solve this with a naming convention doc and Collector-level enforcement.

Missing context propagation at message queues . HTTP propagation works out of the box. Kafka, RabbitMQ, SQS – you need to manually inject and extract trace context from message headers. If you skip this, your traces end at the queue boundary and you lose visibility into async processing.

High-cardinality attributes. Putting user IDs, request IDs, or full URLs as span attributes sounds useful until your trace backend is indexing millions of unique values and your storage costs explode. Use low-cardinality attributes for filtering. Put high-cardinality data in span events or logs.

OpenTelemetry is the right bet for 2021 and beyond. The tracing story is solid. The Collector architecture is sound. Adopt it incrementally, get your conventions right early, and you’ll never have to rewrite instrumentation for a vendor switch again.

It’s not a methodology. It’s just: instrument your code before you ship it. That’s it. That’s the whole thing.

And yet I keep walking into systems – at startups, at enterprises, everywhere – where observability was bolted on after the first production fire. The logs are a mess. Every service logs differently. Metrics exist for some endpoints but not others. Nobody has traces. The dashboards are either empty or full of vanity charts that nobody looks at.

The actual problem

When observability comes last, every team invents their own approach. Service A logs JSON with request_id. Service B logs plain text with req_id. Service C doesn’t log request IDs at all. You find this out at 2am during an outage while trying to correlate a failure across three services.

I’ve lived this. At the fintech startup we had a period where our financial data pipeline logs were useless for debugging cross-service issues because every team had picked their own field names. Fixing that retroactively took weeks. If we’d agreed on a format before writing the services, it would’ve taken an afternoon.

What “ODD” actually means in practice

Before you write a feature, answer three questions:

1. How will I know this is working correctly in production?
2. How will I know this is slow?
3. How will I know this is broken?

If you can’t answer those, you’re not ready to write the code. That’s the whole framework. No acronym needed.

Structured logs or nothing

Your logs should be structured JSON with consistent field names across every service. Non-negotiable.

{"level":"info","event":"order_created","request_id":"abc-123","order_id":"ord-456","duration_ms":42}

Every log line gets a request_id. Every log line gets an event name. Every log line gets a level. If you’re logging fmt.Println("something happened") in production Go code, we need to talk.

Pick your field names once. Write them down. Enforce them in code review. This is boring work. It pays off every single incident.

Metrics: RED and stop

For services, use RED metrics. Rate, Errors, Duration. For every endpoint. That’s your baseline.

I see teams go wild with custom metrics on day one. Thirty metrics per service, half of them never queried. Meanwhile they’re missing basic error rate tracking on their most critical endpoint. Start with RED. Add custom metrics when you have a specific question that RED can’t answer.

One thing that will absolutely burn you: high-cardinality labels. If you’re putting user IDs or full URLs into metric labels, you’re building a cost bomb. I saw one team’s Prometheus storage costs triple in a month because someone added a path label that included query parameters. Keep labels to things like method, status_code, service. Low and predictable.

Traces aren’t optional

Distributed tracing used to feel like a luxury. It’s not. If you’re running more than two services, you need traces. Full stop.

Every inbound request starts a trace. Every outbound call propagates the trace context. This is a few lines of middleware in Go. It’s trivial. And it’s the difference between “I think the problem is in the payment service” and “I can see the exact call that took 4 seconds.”

Sample if you need to for cost reasons. But sample consistently – don’t sample 100% on staging and 1% on production and then wonder why you can never find the trace you need.

Make it part of code review

This is where it sticks or falls apart. If observability isn’t in your code review checklist, it won’t happen.

When I review a PR that adds a new endpoint, I look for:

• Does the handler emit RED metrics?
• Are key events logged with stable fields?
• Does the trace propagate to downstream calls?
• Are the labels safe?

If the answer is no, the PR isn’t ready. Same as missing tests. Same as missing error handling. Observability isn’t a follow-up ticket. It ships with the feature.

Alerts that don’t suck

Most alerts are terrible. They fire on every blip, train everyone to ignore them, and then nobody notices when something actually breaks.

Alert on symptoms, not causes. Alert on “error rate is above X% for Y minutes,” not “one request returned a 500.” Better yet, use SLO-based alerts. Set an error budget. Alert when you’re burning through it too fast. This single change cut our alert noise at Decloud by something like 80%.

Stop making this complicated

The observability vendor ecosystem wants you to believe this is complex. It’s not. Structured logs, RED metrics, distributed traces, and alerts that fire on actual problems. Agree on conventions. Enforce them in review. Ship them with every feature.

That’s observability-driven development. No manifesto required.

That’s not a contradiction. I’ve been running Linux in production since before containers were a thing. The idea of safely running custom programs inside the kernel – attaching to tracepoints, kprobes, uprobes, without writing a kernel module or rebooting anything – is genuinely exciting. When I first ran bpftrace against a production system and got per-process syscall counts in real time with near-zero overhead, I understood the appeal immediately.

bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

That one-liner gives you more insight into what your system is actually doing than most monitoring stacks costing six figures a year. The kernel verifier checks your program for safety. Data flows through maps and perf buffers. No agent bloat. No sampling artifacts. Just direct observation at the source.

So why am I skeptical?

Because the gap between “this is technically possible” and “my team can operate this in production” is enormous. And the eBPF community seems uninterested in acknowledging that gap.

The promise is real

I don’t want to undersell what eBPF enables. Traditional monitoring gives you counters, logs, and coarse sampling. Fine for dashboards. Terrible for understanding why a specific request took 800ms when the p50 is 12ms. eBPF lets you attach instrumentation at the exact point where something happens. Syscall latency. TCP retransmits by destination. Filesystem I/O by process. All with filtering done in-kernel so you aren’t drowning user space in data.

For container-dense environments – which is everything I work with these days – the ability to map kernel events to cgroups and namespaces is a game changer. Short-lived processes that vanish before your log collector notices? eBPF sees them.

The reality check

Here’s my problem. Every conference talk shows eBPF solving elegant debugging puzzles. Nobody talks about the operational burden.

Kernel version compatibility is a real issue. eBPF features vary across kernel versions, and the enterprise Linux distributions I see in production aren’t exactly bleeding edge. A program that works on kernel 5.10 might not work on 4.18. BTF (BPF Type Format) availability is inconsistent. CO-RE (Compile Once, Run Everywhere) helps but isn’t universally supported yet.

Then there’s the expertise problem. Writing eBPF programs isn’t like writing application code. You need to understand kernel internals, verifier constraints, and the performance implications of your hook points. Most engineering teams I work with can’t spare someone to become the eBPF specialist. They need tools that work out of the box.

BCC, bpftrace, and the growing ecosystem of pre-built tools help. Brendan Gregg’s work has been invaluable. But “install bcc-tools and run execsnoop” is a long way from “build a production observability pipeline backed by eBPF.”

Where I land

eBPF is infrastructure technology. It’ll become the foundation that observability vendors build on. Cilium is already proving this for networking. The profiling tools are getting there. Give it two or three more years and it will be invisible plumbing that powers your monitoring stack.

But right now, in early 2021, if someone tells me they’re building their observability strategy around eBPF, I ask two questions: what kernel version are you running, and who on your team understands the verifier? If they can’t answer both, they should start with existing tools – opensnoop, tcpconnect, biolatency – and build intuition before writing custom programs.

The technology is sound. The ecosystem is maturing. I’m watching closely. I’m just not rewriting my monitoring stack around it today.

You don’t need Datadog’s enterprise tier. You need structured logs, one good dashboard per service, alerts that don’t cry wolf, and a request_id on everything. That’s 80% of it.

I’ve been working with distributed teams for most of this year. Some are five people across three time zones. Some are thirty across eight. The pattern I keep seeing: they either have zero observability or they went full enterprise cargo cult and now nobody can find anything.

There’s a middle ground. I want to talk about that.

The actual problem with distributed teams

In an office, someone notices something is slow. They say it out loud. Someone else goes “oh yeah, I deployed ten minutes ago.” Problem found in under a minute.

Remote? That same issue sits in a Slack thread for 45 minutes while people in different time zones wake up, read context, and try to figure out what changed. I’ve watched this happen. Repeatedly.

The fix isn’t more tools. It’s making your systems capable of answering basic questions without requiring a human to be online at the right moment.

Three questions. That’s it:

1. Is this thing broken right now?
2. What changed recently?
3. Where do I look next?

If your setup can answer those, you’re ahead of most teams I’ve worked with.

Enterprise observability isn’t your observability

Google has 10,000 SREs. They built custom everything. When you read their SRE book and try to implement the same stack with your team of eight, you end up with:

• A Prometheus instance nobody configured alerts for
• Grafana dashboards copied from a blog post that don’t match your services
• Jaeger running but with 0.1% sampling so traces are useless when you actually need them
• An ELK stack eating 40% of your infrastructure budget

I’ve seen this exact setup at three different companies this year. Not exaggerating.

What you actually need

Here’s my stack recommendation for a small distributed team. Opinionated, yes. But it works.

Logs: Structured JSON to a managed service. Loki if you’re cheap. Papertrail if you want simple. The key is structured, not the tool. Every log line should look roughly like this:

{
  "level": "error",
  "msg": "payment failed",
  "service": "checkout",
  "request_id": "7f3c2c4d",
  "user_id": "u_123",
  "error": "card_declined",
  "duration_ms": 340
}

Same field names. Every service. No exceptions. The request_id alone will save you hours of debugging per incident. I can’t stress this enough. Propagate it through HTTP headers, queue messages, background jobs. Everywhere.

Metrics: Prometheus + Grafana. Still the best bang for buck. But here’s the thing – don’t build 30 dashboards. Build one per service. Three panels:

1. Request rate and error rate (tells you if something is broken)
2. Latency percentiles (tells you if it’s degrading)
3. Recent deploys and config changes overlaid on the graphs (tells you what caused it)

That’s your dashboard. If a new engineer can’t look at it and understand the health of the service in 30 seconds, strip it down further.

Traces: Jaeger or Zipkin, but only if you have more than two services. If you’re a monolith with a database and a cache, traces are overhead you don’t need yet. Just use request IDs in your logs. Seriously.

When you do need traces, bump sampling to at least 10% on critical paths. 0.1% default sampling means you’ll never have a trace for the request that actually broke.

Alerts: Less is more. Every alert that pages someone at 3am and isn’t actionable is erosion of trust. Once your team stops trusting alerts, you’ve lost. They’ll start muting channels. I’ve seen it happen.

My rule: every alert needs three things.

• A condition that’s actually abnormal (not “CPU above 60%”)
• A link to the relevant dashboard
• A link to a runbook that says what to do first

If you can’t write those three things for an alert, the alert shouldn’t exist.

The request_id sermon

I keep coming back to this because it’s the single highest-leverage thing you can do.

At the fintech startup we had services talking to services talking to queues talking to workers. When something went wrong without correlation IDs, the debugging process was: check this log, then that log, then maybe this other log, hope the timestamps line up, piece it together manually. Took forever.

After we standardized on a single request_id header propagated everywhere? Same investigation. One search. Done.

The implementation is trivial. Middleware that reads X-Request-ID from incoming requests. Generates a UUID if missing. Passes it along. Logs it on every line. Takes an afternoon to implement across your whole stack.

An afternoon of work for months of saved debugging time. That’s the kind of trade I like.

Runbooks: the unsexy high-leverage tool

Nobody wants to write runbooks. I get it. But here’s the scenario: it’s 2am in your time zone. The alert fires. The person on call is in a different country. They’ve been on the team for three weeks.

Without a runbook, they’re messaging people, waiting for responses, guessing. With a runbook, they open it, follow the steps, and either fix it or know exactly who to escalate to.

Keep them short. Keep them next to the code. Update them after every incident. A runbook that says “check the database connection pool, then check Redis, then check the upstream API timeout” is worth more than a 50-page incident response process document nobody has read.

Mistakes I keep seeing

Collecting everything. Storage is cheap. Cardinality explosions aren’t. I watched a team’s Prometheus instance fall over because they added a user_id label to a counter. Millions of time series. Dead monitoring system. During an outage. Ironic.

Dashboard graveyards. Thirty dashboards, twenty-eight of which nobody has looked at in months. Two of which are actually useful but you can’t remember which ones. Delete aggressively.

Happy path instrumentation only. Your error paths need more instrumentation than your happy paths. The happy path works. You know this because nobody is complaining. The error paths are where surprises live.

Separate conventions per team. One team calls it user_id, another calls it userId, a third calls it uid. Now your cross-service queries are a mess. Pick a convention. Enforce it in code review. This is boring work that pays off enormously.

What to measure about your observability itself

One meta-metric I track: time from “something seems wrong” to “I know what changed and where to look.” If that number is going down over time, your observability is working. If it’s flat or going up, you’re adding complexity without adding clarity.

The other one: how many alerts fired this week that didn’t need a human response? If it’s more than 20%, you have a noise problem.

Start here

If you’re starting from scratch with a small distributed team, do this in order:

1. Structured JSON logs with a shared request_id. One week of work, max.
2. One Grafana dashboard per service with the three panels I mentioned. Another week.
3. Three to five alerts that are actually actionable. A few days.
4. Short runbooks for those alerts. A day.

That’s a month of work spread across your team. After that, you have a system that answers the three questions. Everything else – traces, SLOs, error budgets, custom metrics – layer it on when you feel the pain, not before.

Don’t let perfect be the enemy of “I can actually debug production at 2am without waking up three people.”

If burning through your error budget doesn’t change how your team ships, you don’t have SLOs. You have decorative charts.

I’ve watched three different teams adopt SLOs in the past year. Two of them ended up with beautiful Grafana dashboards that nobody looked at after the first sprint. The third team actually used their error budget to cancel a feature release and fix a checkout regression instead. Guess which team had fewer incidents in Q4.

The difference wasn’t tooling. It was whether the SLO changed behavior or just measured things.

SLOs are decisions, not dashboards

At the fintech startup, we tracked uptime for our financial data API. 99.9% availability on a fancy status page. Looked great. The problem? Our SLI was measuring HTTP 200s from a health endpoint. Meanwhile, users were getting stale stock prices because our data pipeline was silently lagging by 30 minutes. By our SLO, everything was fine. By our users’ experience, the product was broken.

An effective SLO is a contract between reliability and velocity. It answers one question: can we ship this week, or do we owe the users some stability work first? If it doesn’t influence your sprint planning, kill it.

Measure what your users feel, not what your infra reports

Start with the user journey. Not the Kubernetes dashboard.

These aren’t user-facing indicators:

• CPU utilization
• Pod restart counts
• Database connection pool size

These are:

• Successful checkout completions
• Search results returned under 400ms
• API responses with correct, fresh data

The distinction seems obvious written down, but I still see teams default to infrastructure metrics because they’re easier to collect. Easier isn’t the point. Accurate is.

Start with four signals, then get specific

For most services, you can begin with availability, latency, throughput, and error rate. The classic golden signals. But don’t stop there. Availability that counts health check pings the same as checkout requests is lying to you.

A useful SLI is brutally specific:

• The request: POST /api/v1/checkout from authenticated users
• What counts as success: HTTP status < 500 AND order confirmation generated
• The population: production traffic only, excluding synthetic monitors
• The window: rolling 28 days

That specificity is the difference between an SLI that catches real problems and one that hides them in averages.

Set targets from data, not ambition

I see this constantly: a team picks 99.99% availability because it sounds professional. They’ve been running at 99.2% for six months. The gap between target and reality is so large that the error budget is permanently exhausted, which means the policy attached to it’s permanently triggered, which means everyone ignores it.

A target that’s never met is noise. A target that’s always met is invisible. Neither changes behavior.

Here’s what actually works:

1. Measure your current performance for 2-4 weeks. No changes, just observation.
2. Set the target slightly tighter than current reality. If you’re running at 99.5%, try 99.7%.
3. Adjust quarterly based on data and user feedback. Not based on what the VP saw at a conference.

Different services deserve different targets. Your payment processing endpoint can justify 99.95%. Your internal admin dashboard? 99% is probably generous. During the early Decloud days at a deep-tech founder program, we ran our dev tooling at targets that would horrify a payments team – and that was the right call. We needed to ship fast, not polish internal tools.

Windows matter more than you think

A 99.9% SLO over 30 days gives you about 43 minutes of allowed downtime. The same target over 7 days gives you about 10 minutes. Choose a window that matches how fast your team can actually detect and respond to problems. If your mean time to detect is 20 minutes, a 7-day window at 99.9% is a trap.

Error budgets: the part everyone gets wrong

The math is simple:

error_budget = 1 - SLO_target

A 99.9% target over 30 days means you can tolerate roughly 43 minutes of downtime. A 99.5% target gives you about 3.6 hours. These numbers aren’t interesting by themselves. What makes them powerful is the policy.

Budget healthy (> 50% remaining): Ship normally. Take calculated risks. Run that migration you’ve been planning.

Budget tight (10-50% remaining): Slow down releases. Require extra review on risky changes. Maybe skip the experimental feature flag rollout this week.

Budget burned (< 10% remaining): Stop feature work. The entire team focuses on reliability until the budget recovers.

That third state is where most teams fail. They write the policy, then when the budget actually burns, some product manager argues that the feature is too important to delay. If leadership won’t enforce the budget policy, you don’t have SLOs. You have aspirations.

Track burn rate, not just remaining budget

A single bad deployment can eat your monthly budget in an hour. By the time you notice the remaining budget is low, the damage is done.

burn_rate = errors_in_window / budget_for_window

Alert on burn rate. If you’re consuming budget at 10x the sustainable rate, you want to know in minutes, not at the Monday standup.

Keep it minimal

You don’t need an SLO for every endpoint. Pick the 3-5 user journeys that define whether your product is working. For most B2B SaaS, that’s: login, core workflow, data export, and billing. Everything else is noise at this stage.

Instrumentation comes first. An SLO is just a query on top of good metrics. If you don’t have request counts, status codes, and latency histograms, start there. A YAML definition can be as simple as:

slo:
  name: checkout-availability
  objective: 99.9
  window: 28d
  indicator: success_rate
  filter: "route = /checkout AND source = production"
  success: "status_code < 500 AND order_confirmed = true"

Your dashboard should answer three questions and nothing else: Are we meeting the SLO right now? How much budget is left? How fast are we burning it?

Proof it works

Here’s how you know if your SLOs are working: the last time your error budget got tight, did anything actually change? Did a release get delayed? Did someone shift from feature work to fixing that flaky dependency? Did the on-call rotation get extra support?

If the answer is no, go back to the error budget policy and make it real. Get sign-off from engineering leadership. Write it into your sprint process. Make the consequences automatic, not optional.

SLOs are a decision framework disguised as monitoring. The monitoring part is easy. The decision part is where most teams give up.

Don’t be most teams.

I spent forty minutes SSH-ing into boxes, tailing logs, and grepping for exceptions. Nothing. The pipeline just… stopped processing. It took me another hour to discover the root cause: a third-party API we depended on had started returning empty 200 responses instead of actual data. Our monitoring checked for errors. It checked for timeouts. It never checked for “success that contains nothing useful.”

That night changed how I think about production systems.

Monitoring Checks Boxes. Observability Answers Questions.

Monitoring is built around things you already know can go wrong. Threshold crossed, alert fires, runbook engaged. It works great for predictable problems: disk filling up, CPU pegged, error rate spiking. We had all of that at the fintech startup. Grafana dashboards everywhere. PagerDuty wired up. It felt safe.

But distributed systems don’t fail in predictable ways. They fail in weird, combinatorial, never-seen-this-before ways. And that’s where monitoring falls apart. You can’t write an alert for a failure mode you haven’t imagined yet.

Observability flips the model. Instead of predefining what questions the system can answer, you instrument it richly enough that you can ask new questions on the fly. During an incident. At 2 AM. Without deploying anything.

The difference: monitoring tells you something is wrong. Observability helps you figure out why.

Three Signals, One Story

After the empty-200 incident, I started rebuilding our telemetry around three pillars.

Metrics give you the bird’s-eye view. Aggregated numbers over time. At the fintech startup, we track request rates, error rates, and latency distributions for every service using Prometheus.

http_requests_total{method="GET", endpoint="/api/stories", status="200"} 15234
http_request_duration_seconds_bucket{endpoint="/api/stories", le="0.5"} 421

Metrics are cheap to store and great for alerting. But they can’t tell you why a specific request failed or what user it affected. They’re the smoke detector, not the fire investigator.

Logs capture the details. Each event, each error, each decision the code made. We moved early to structured JSON logs because free-form strings are nearly useless at scale.

{"timestamp":"2018-07-09T10:30:45Z","level":"error","service":"ingestion","request_id":"abc123","message":"Empty response from provider","provider":"reuters","duration_ms":340}

That structured format meant we could query logs in Kibana instead of grepping through them. Game changer for incident response.

Traces show you the journey of a single request across services. This was the missing piece for us. The fintech startup’s architecture involves an ingestion service talking to NLP processors talking to a ranking engine talking to the API layer. When something is slow, you need to see the whole chain.

Trace abc123
- POST /ingest/batch (120ms)
  - NLP enrichment (45ms)
  - Relevance scoring (30ms)
  - db.write stories (35ms)

We started with Zipkin. Not perfect, but suddenly I could see that our NLP service was adding 200ms of latency on certain content types. That was invisible before.

The Glue: Correlation IDs

Each signal alone is useful. Together, connected by a shared identifier, they’re powerful.

Here’s the workflow that actually matters: a latency alert fires from metrics. You pull up the trace for a slow request. The trace shows a specific span taking too long. You jump to the logs for that span using the same request ID and find the exact query and error.

X-Request-ID: 5f3c9e86c2c84e1b
X-B3-TraceId: 4d1e00a3b9bd1d42
X-B3-SpanId: 6df3a1c2b93f6b1a

We made it a rule: every log line includes the request ID, and every trace propagates context headers. No exceptions. It took weeks to retrofit across all our services, but the payoff during the next incident was immediate. Instead of an hour of archaeology, I had a clear thread to pull.

Instrument the Critical Path First

When I started adding instrumentation at the fintech startup, the temptation was to instrument everything. Don’t do that. You’ll drown in data and your storage costs will spike.

Start with what matters: the critical path your users depend on.

For us that meant inbound API handlers, the news ingestion pipeline, database calls, and every external API dependency. We used RED (rate, errors, duration) for our services and USE (utilization, saturation, errors) for infrastructure resources.

Tracing has a sampling problem. Capturing every single request is expensive. We settled on keeping 100% of errors and sampling successful requests at about 5%. During incidents, we crank sampling up. Good enough for debugging, affordable enough to run continuously.

Designing Systems That Can Be Debugged

Observability isn’t something you bolt on after the fact. It’s a design choice.

Structured logs over free-form strings. Always. If you can’t query it, it’s useless when you’re under pressure at 2 AM.

Watch your cardinality. Metrics with unbounded label values (like user IDs) will destroy your Prometheus instance. High-cardinality data belongs in logs and traces, not metrics.

Add context that matters. A trace that only shows timings is half the story. Include the request type, the tenant, the result count. When something goes wrong, that context is the difference between a five-minute fix and a two-hour hunt.

Our Stack in 2018

For anyone building this out now, here’s what we were running:

• Metrics: Prometheus with Alertmanager, Grafana for dashboards
• Logs: Fluentd piping into Elasticsearch, Kibana for exploration
• Tracing: Zipkin with OpenTracing instrumentation

OpenTracing is getting solid adoption and keeps us from being locked into one vendor. OpenCensus is emerging as an alternative worth watching. Both are pushing toward a world where trace context propagation just works out of the box.

There are commercial options that bundle everything behind one query layer. We looked at them. For our scale, the open source stack made more sense. That calculus changes depending on team size and how much operational overhead you can absorb.

What That 2 AM Incident Taught Me

That silent pipeline failure was a gift. It exposed a blind spot in how I thought about production readiness. Having dashboards isn’t the same as having understanding. Monitoring answers the questions you thought to ask. Observability gives you the ability to investigate the questions you didn’t.

We still have incidents at the fintech startup. But now when something breaks in a way we’ve never seen before, we have the telemetry to figure it out fast. That’s the whole point.

It falls apart the second you split into services.

I learned this the hard way at the fintech startup. We had decent Grafana dashboards. Reasonable alerts. Then we started breaking things into microservices and deploying multiple times a day. A request would fail somewhere in a chain of five services and our dashboards would just… look fine. Every individual service reported healthy metrics. The problem lived in the gaps between them.

That’s the core issue with monitoring. It answers questions you already thought to ask. Latency on this endpoint? Sure. Error rate on that queue? Got it. But the failure you actually hit in production is the one you never predicted. Your dashboards have no panel for it.

Observability is a different mindset

Observability means you instrument your system so you can ask new questions after something breaks. Not just “is it up” but “why did this specific user’s request take 8 seconds at 3am on Tuesday.”

Three signals make this work: metrics for trends and alerts, logs for detail, and traces for stitching a single request across every service it touches. Separately they are useful. Together they are a debugging superpower.

Structured logging is the foundation. Stop writing free-text log lines. Make every log entry a JSON object with a trace ID, service name, version, and whatever fields you actually need to filter on.

{
  "timestamp": "2017-03-20T10:23:45Z",
  "level": "error",
  "message": "Payment processing failed",
  "trace_id": "abc123",
  "service": "payment-service",
  "version": "2.1.3"
}

Then propagate context. Generate a trace ID at the edge and carry it through every downstream call. This is the single most important thing you can do. Without it you’re just grepping logs and praying.

X-Trace-Id: abc123
X-Span-Id: def456
X-Parent-Span-Id: ghi789

How incidents actually change

Alert fires. Pull the trace. Follow the slow span to the source. Read the logs for that trace. Confirm with metrics whether it’s one user or everyone. Done. Five minutes instead of forty-five minutes of bouncing between dashboards while your Slack channel fills with “any update?”

Same thing with performance. User says it’s slow. Pull their trace. Compare to a healthy one. See the difference – a cache miss, an extra database round-trip, a third-party call timing out. You fix the actual cause instead of guessing.

The tools don’t matter that much

Prometheus, InfluxDB, ELK, Jaeger, Zipkin – pick whatever fits your stack. Commercial platforms that bundle all three signals save time. But the tooling isn’t the hard part. The hard part is disciplined instrumentation. Consistent field names. Trace IDs everywhere. Every team following the same conventions.

What actually matters

Observability isn’t a product you buy. It’s a practice you build. You stop staring at dashboards waiting for red. You start asking questions about behavior you didn’t expect. That shift – from reactive to exploratory – is the entire point. And in a world where every team is shipping services independently, it’s the only way to stay sane.

The problem wasn’t tooling. The problem was noise. So we deleted 42 of those panels. Kept five metrics. Built real alerts around them. Our on-call engineer started sleeping again.

Most monitoring is waste

Here is what I’ve learned running production systems: if you’re watching more than a handful of metrics, you’re watching none of them. Human attention doesn’t scale. Twenty dashboards with eight panels each means you have zero dashboards, because nobody is actually reading them under pressure.

The instinct after an outage is always “we need more visibility.” Wrong. You need better visibility. Those are different things.

Five numbers, nothing else

For a startup backend serving real users, these are the only metrics I care about on a daily basis:

1. Request latency at p95 and p99. Not average. Averages lie. A service averaging 40ms with a p99 at 3 seconds is broken for one in a hundred users, and that one user is filing the support ticket. We tracked this per-service in Prometheus and set alerts at the Grafana layer.

2. Error rate by type. Not just “5xx count.” Break it down. A spike in 401s is a different problem than a spike in 503s. One is probably a bad deploy or a client bug. The other is your database falling over. The distinction matters because the response is completely different.

3. Saturation of your bottleneck resource. For us that was PostgreSQL connection pool utilization. For you it might be memory, disk I/O, or worker threads. The point is: know which resource will kill you first and watch that one. Not all resources. That one.

4. Request throughput. Traffic going up is good. Traffic dropping to zero at 2pm on a Tuesday is very bad. This metric is less about the number and more about detecting sudden changes. A 50% drop in requests is an incident whether or not errors are firing.

5. Deployment markers. Not a metric in the traditional sense, but I overlay every deploy on our Grafana dashboards. Half the incidents I’ve investigated started with “someone shipped something.” Correlating metrics with deploys cuts your mean time to diagnosis in half.

That’s it. Five things.

Alerts should wake you up for a reason

We had a rule: if an alert fires and the on-call engineer can’t take a meaningful action within five minutes, delete the alert. Brief CPU spikes? Not actionable. Disk at 80%? Only actionable if there’s a runbook. “Something might be wrong” isn’t an alert. It’s anxiety.

Our paging alerts were:

• p99 latency above threshold for 5 minutes
• Error rate above 1% for 3 minutes
• Primary database connection pool above 90%

Three alerts. That was it for paging. Everything else went to a Slack channel that people checked during business hours.

The discipline is in what you remove

After I stripped our monitoring down, the team pushed back. “What if we miss something?” My answer: we were already missing things. Forty-seven panels meant nobody looked at any of them carefully. Five panels meant the on-call engineer actually understood the state of the system at a glance.

Monitoring isn’t a collection problem. It’s an attention problem. Treat it that way.

Five metrics, three alerts, zero noise

Pick the five metrics that reflect what your users experience. Build alerts only for conditions where someone can take immediate action. Delete everything else. Discipline over dashboards.

So we set up ELK. Elasticsearch, Logstash, Kibana. The pitch was compelling: open source, flexible, great full-text search, a plugin for everything. We stood up a three-node cluster, pointed Logstash at it, gave the team Kibana dashboards. For the first few weeks, it felt like a superpower.

Then the cluster started misbehaving. And I spent the next several months learning a painful lesson about the gap between “powerful” and “worth the operational cost.”

If you don’t have someone who wants to babysit Elasticsearch full time, don’t run ELK yourself. Hosted Elasticsearch, Graylog, or even Splunk will save you more engineering hours than they cost.

The Operational Tax Nobody Warns You About

Elasticsearch isn’t a database you deploy and forget. It’s a distributed system that demands constant attention. Shard allocation, index lifecycle, JVM heap tuning, split-brain prevention, disk watermarks. Every one of these will bite you, and they will bite you at 3 AM.

At the mobility startup, our log volume was moderate. Maybe a few gigabytes a day. Nothing that should stress a three-node cluster. But Elasticsearch doesn’t care about your expectations. It cares about index design, merge policies, and whether you remembered to set bootstrap.memory_lock. We spent more time keeping the logging infrastructure healthy than we spent on some of our actual product services.

The worst part is that when your logging system goes down, you lose visibility into everything else at the same time. Your safety net disappears exactly when you need it most.

What ELK Actually Gets Right

I’m not going to pretend it’s all bad. Elasticsearch’s search is genuinely excellent. When the cluster is healthy, the ability to run arbitrary queries across millions of log lines with sub-second response times is hard to match. Logstash can parse almost any log format into structured fields. Kibana dashboards gave our product team visibility they never had before.

The ecosystem is real. Beats shippers are lightweight, community plugins cover most integrations, and the documentation is solid. If you have the operational muscle, ELK is the most flexible open source logging stack available.

What I’d Do Differently

If I were starting over, I wouldn’t self-host Elasticsearch for logging. Full stop.

Hosted Elasticsearch removes the worst of the operational burden. You keep the same query model, the same Kibana dashboards, the same integrations. You lose some control over cluster configuration and you pay more per gigabyte, but you gain back the engineering hours you were burning on cluster babysitting. For most teams, that tradeoff is obvious.

Graylog is worth a look if you want Elasticsearch search without the full DIY build. It wraps Elasticsearch in a more opinionated log management experience with built-in alerting and stream routing. Less flexible than raw ELK, but faster to get running and easier to keep running.

Splunk is the enterprise answer. Powerful, mature, battle-tested. Also expensive enough to make your finance team flinch. If budget isn’t the constraint, Splunk is a safe bet. For a startup, it rarely makes sense.

Cloud provider logging is the lowest-effort option. AWS CloudWatch Logs, for example, integrates deeply with everything else in AWS and requires zero operational overhead. The query capabilities are basic compared to Elasticsearch, but basic is often enough. You can always export to something more powerful later.

Structured Logging Is the Real Win

Regardless of which aggregation tool you pick, the single best investment is structured logging. A JSON log line with consistent fields turns debugging from archaeology into search.

{
  "timestamp": "2016-09-05T10:23:45Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123",
  "message": "Payment processing failed",
  "error": "Connection timeout",
  "customer_id": "cust_456"
}

Carry a trace ID through every service. Use consistent field names. Emit JSON instead of free-form text. These decisions pay off no matter what sits behind the ingestion pipeline. If you do nothing else, do this.

Put a Buffer in Front of Ingestion

One more thing I learned the hard way: put a queue between your log shippers and your aggregation layer. Kafka, Redis, even a simple file buffer. Traffic spikes will happen. Deploys will happen. If your pipeline has no buffer, you drop logs during the exact moments you need them most.

Pick your logging battles

ELK is powerful software with brutal operational costs. Most teams underestimate how much work Elasticsearch is to run, and they find out at the worst possible time. If you can afford someone who genuinely enjoys tuning JVM garbage collection and shard allocation, go for it. Otherwise, pay for a hosted solution or pick a simpler tool. Your on-call engineers will thank you.