Observability.
Metrics, logs and traces across four environments.
I built and run our Kubernetes monitoring stack so engineers can investigate failures across services. It connects metrics, logs and traces, with 72 alerts linked to runbooks.
- My role
- Built and operate the monitoring platform
- Context
- Loweconex
- Built with
- Kubernetes · Prometheus · Thanos · Loki · Tempo · Grafana
01
A shared view of service failures
Services ran across four environments, but we had no shared way to investigate what they were doing. Too often, a problem was reported from outside the engineering team before we saw it ourselves.
I deployed Prometheus and Thanos for metrics, Loki for logs, Tempo for traces and Grafana to explore them. We used existing cluster capacity and kept the configuration in Git. Self-hosting kept costs down, with the operational work staying with us.
02
Connecting metrics, traces and logs
A metric exemplar opens the matching trace, and the trace links to the relevant service logs. Log lines include a trace ID to navigate back to the request. I wrote the shared structured logging configuration and OpenTelemetry conventions for our Java services to make those links work.
Each environment has its own stack, and a federated query layer joins them across AWS accounts. Older data moves to object storage, with retention set per signal.
03
Alert thresholds, routing and runbooks
I checked the alert rules against the available metrics and linked 72 alerts to runbooks. Thresholds for the monitoring pipeline came from its recorded history.
Production alerts page on-call at any hour. QA alerts go to Teams, and dev notifications wait for business hours. Inhibition rules suppress related alerts when a broader failure already explains them.
04
Using and maintaining the stack
Incident discussions now usually start with a Grafana link. Engineers can follow requests across services and use the linked runbooks to investigate alerts.
I also maintain retention policies, upgrades and capacity, and investigate failures in the monitoring system itself.
AI gateway
Shared model access with usage tracked per consumer.