Monitoring tells you whether the things you thought to check are healthy. Observability is whether you can answer a question you had not thought of, without shipping new code.
The distinction is practical. A dashboard of CPU and error rate is monitoring. If the question is “why are Android users in Brazil on the new checkout slow?” and nothing you have can answer it, you have monitoring and not observability.
The average is lying to you
- per-call slow rate
- 1%
- page loads that hit at least one
- 18%
A page making 20 calls is as slow as its slowest one, so a 1% tail becomes 18% of page loads. This is why the p99 of a dependency is closer to the average experience of a user than its own average is — and why averaging latency across services hides the thing you need to see.
The three signals, and what each is for
Logs — discrete events with detail. Best for “what exactly happened to this one request”. Expensive at volume, and they are only useful if structured:
log.info('payment.completed', {orderId, amount, durationMs, traceId}); // queryablelog.info(`Payment ${orderId} completed in ${durationMs}ms`); // grep-onlyMetrics — numbers aggregated over time. Cheap, ideal for dashboards and alerts, and they cannot tell you about one request.
Traces — one request’s path across every service, with timings. This is the signal that makes distributed systems debuggable, and the one most teams add last.
Tracing, concretely
A request enters the API gateway, calls auth, then orders, then the database. A trace is that whole path as one tree of timed spans.
The mechanism is a trace ID created at the edge and propagated to every
downstream call — via the W3C traceparent header — with each service recording
spans against it.
The rule that makes this work: propagate the context everywhere, including into message queues and background jobs. A trace that stops at the queue boundary is missing exactly the asynchronous part you cannot otherwise reason about.
Use OpenTelemetry. It is the vendor-neutral standard, it auto-instruments most frameworks, and it means changing backend later is a configuration change.
What to alert on
Symptoms, not causes. “Checkout error rate above 1%” wakes someone for a real problem. “CPU above 80%” wakes them for a machine doing its job.
The four golden signals — latency, traffic, errors, saturation — cover most services and are a good default.
Error budgets. If the SLO is 99.9%, you may be down 43 minutes a month. Alert on the burn rate: consuming a month’s budget in an hour is urgent; a slow trickle is a ticket. This is what stops alerting being either too noisy or too quiet.
Where to start
If you have nothing: structured logs with a request ID, and a way to search them. That alone answers most questions.
Then the four golden signals as metrics, with percentiles.
Then distributed tracing, once you have more than about three services and “which service is slow?” stops being obvious.
Then wide structured events — one rich event per request with every attribute you might want to slice by. This is the modern direction, and it is where the “questions you did not plan for” property actually comes from.