// Topics / Observability

Observability

    Testing AI Where It Actually Runs Offline evals are necessary but not sufficient. Here's how I test AI features in production with shadow mode, canaries, and rollback automation -- with Go code. testing ai production Your AI System Looks Healthy. It Is Not. Traditional monitoring will tell you your AI service is up. It won't tell you it's returning confident garbage. Here's what observability actually looks like for AI. observability ai production LLM Observability: Your Existing Monitoring Is Not Enough Traditional monitoring says the service is up. It won't tell you the model started returning garbage last Tuesday. How to actually observe LLM systems. observability llm ai OpenTelemetry in Late 2021: What's Ready and What's Not Tracing is ready. Metrics are getting there. Logs are not. Here's a practical adoption path and the code to back it up. observability go Observability-Driven Development Is Just Instrumenting Your Code ODD sounds fancy. It's not. It means writing logs, metrics, and traces before you ship, not after your first outage. observability development reliability eBPF Is Interesting. I Am Not Sold Yet. eBPF promises kernel-level observability without the pain of kernel modules. The tech is real. The hype-to-adoption ratio concerns me. observability engineering performance Observability for Small Distributed Teams (What Actually Works) Most observability advice is written for 500-engineer orgs. Here's what actually matters when you're a small distributed team trying not to drown in dashboards. observability distributed-systems devops Your SLOs Are Probably Useless (Here's How to Fix Them) Most SLOs are dashboards nobody acts on. Pick indicators that reflect real users, set targets from data, and make error budgets change how your team ships. reliability observability engineering Why Monitoring Wasn't Enough and How We Built Observability at a Fintech Startup After a mystery outage that our dashboards couldn't explain, I rebuilt the fintech startup's telemetry stack around metrics, logs, and traces. Here's what I learned. observability devops distributed-systems Monitoring Is Not Enough Your dashboards look green. Your users say the site is broken. That gap is the whole problem. observability devops distributed-systems Why We Deleted 42 Grafana Panels Most teams monitor too much and alert on the wrong things. Five metrics are enough to run a startup backend. observability devops production Log Aggregation at Scale: ELK vs Alternatives ELK is powerful. It's also a second full-time job. Here's what I learned running it at a mobility startup, and what I'd consider instead. observability databases devops