ADR-001 · Architecture decision record
Observability stack — build over buy
- Status
- Accepted, 2024 · still in force
- Deciders
- Jack Devlin, platform engineer · Loweconex, a UK IoT platform business
- Context
- Twenty services across four environments, no shared monitoring. Commercial quotes came in near £100k a year; self-hosting runs at about £5k.
- Outcome
- Two years on, an incident starts with someone pasting a Grafana link, and the bill stayed flat while the data grew. Section 6 reviews the decision.
The estate had grown faster than its monitoring
Twenty microservices on Kubernetes, and the tooling around them hadn't kept pace with how quickly the team had grown. Too often the first sign something was wrong came from outside rather than from a dashboard. Buying a commercial platform was the obvious fix, and the quote for it is why we didn't.
So I built it in-house. The stack is unsurprising: Prometheus and Thanos for metrics, Loki for logs, Alertmanager for paging, Grafana for everyone to actually look at.
The pieces are standard; the work was wiring them so people can find what they need while something is breaking.
The options, and the call
- Context
- The headline figures are in the title block. What made it a live decision was that we already had the cluster capacity to run this ourselves, so the quote was buying convenience rather than capability.
- Option A
- A commercial SaaS platform. Fastest to value, no operational burden, per-host and per-GB pricing that grows with exactly the thing you can't control: how much telemetry your developers decide to emit.
- Option B
- Self-host Prometheus, Thanos, Loki, Tempo and Grafana on the existing cluster. Slower to stand up, an operational surface we own, and near-flat cost as the data grows.
- Decision
- Option B, at roughly £5k a year all-in. The cluster capacity was already there, the team knew Kubernetes, and the gap between the two numbers was too wide to argue with.
- Consequence
- Cold data goes to object storage — Thanos for metrics, S3-backed Loki for logs — so we only pay premium prices for the recent data people actually query. That single choice is why the bill stayed flat while the data grew.
- Consequence
- It's mine to fix at 3am. There is nobody on the other end of a support contract, and the stack that tells you what's broken is itself a thing that can break.
- Revisit when
- The team is small enough that a day of my time is worth more than the difference, or the estate grows to where storage costs start tracking the SaaS quote. I'd make the opposite call at a three-person startup without hesitating.
How it fits together
Observability stack · system overview
Select any box in the diagram for a line on what it does. The diagram is wider than this column, so it scrolls sideways.
Prometheus scrapes everything and hands the long tail off to Thanos in S3, so we aren't paying hot-storage prices for data nobody queries.
Loki runs in microservices mode for the same reason. Logs are cheap to generate and expensive to keep. Alertmanager routes by environment: prod pages, dev gets a Teams message in business hours.
What people actually look at
I ended up with 22 dashboards, but most of the traffic goes to maybe five. The rest exist for the once-a-quarter question they answer perfectly. Three I'm happy with:
IoT Gateway throughput
Live request rate, connected devices, vendor-by-vendor performance. The first place anyone looks when an integration partner says something's broken.
Kafka consumer lag
Per-topic, per-group lag with sensible thresholds. Replaced about a dozen ad-hoc kafka-cli queries that used to live in someone's bash history.
Node infrastructure
CPU, memory, disk, network. Nobody looks at it until something is wrong, and then everybody does.
Alerts that don't cry wolf
Every alert passes two tests. A human has to be able to do something about it, and the runbook has to exist before the rule ships.
The runbook isn't fancy: symptom, what to check, common fixes, who to escalate to. Just enough that whoever picks up the page isn't starting from zero.
Routing is by environment more than severity. Prod fires straight to the on-call channel. Dev waits until business hours.
Inhibition rules kill the cascade of follow-on alerts when one root cause takes out a dozen things downstream. Without them, the first real incident would have trained everyone to ignore alerts.
Review — two years on
The saving is what got it approved. Two years on, an incident starts with someone pasting a Grafana link. Nobody has thanked me for the dashboards. They'd notice immediately if they went.
How long the server took to start answering you.
not measurable here
How long before anything at all appeared on screen.
not measurable here
When the main content finished rendering.
not measurable here
How much the page jumped around while loading. 0 is perfectly still.
not measurable here
Measured in your browser, on this visit — not a screenshot of a good day. Thresholds are the Core Web Vitals definitions. This site ships as static files, so a slow first byte is the journey to you, not a server thinking.
Appendix A — reference
- Stack
- Prometheus · Thanos · Loki · Grafana · Alertmanager · Promtail · Kubernetes · Kustomize · S3 · PromQL · LogQL
- Figures
- Services: 20, across 4 envs
- Annual cost: ~£5k all-in
- Alerts: 50+, runbook per rule
- Dashboards: 22 active