Observability
The ability to measure the internal state of a system by examining its external outputs (logs, metrics, and traces).
Detailed Explanation
Observability goes beyond simple 'monitoring' (checking if a server is up). It involves collecting granular data to understand *why* a system is behaving a certain way. By correlating logs (what happened), metrics (numerical stats), and traces (the path of a single request), developers can debug complex issues in distributed systems more effectively.
Quick Summary
Observability is the ability to ask new questions about a running system without redeploying it. Logs, metrics, and traces (the "three pillars") instrument enough signal that an operator can debug failures they didn't anticipate.
Key Takeaways
- Monitoring answers known questions ("is CPU > 80%?"); observability answers novel ones ("why are p99 latencies on /checkout up since 2:14 PM?").
- Three signal types: logs (discrete events), metrics (numeric time series), traces (causal request paths across services).
- High-cardinality context (user ID, request ID, region, build SHA) is what makes data debuggable; pre-aggregated metrics are not.
- OpenTelemetry is the de-facto standard SDK/protocol; pair it with a backend like Datadog, Honeycomb, Grafana Tempo, or Lightstep.
- Sampling is a cost lever, keep all errors and a fraction of successes; head-based vs. tail-based sampling are the two main strategies.
When to use it
- Debugging production incidents where the failure mode wasn't anticipated by a dashboard.
- Tracing a slow request through 10+ microservices to find the bottleneck.
- Capacity planning from real usage data instead of guesses.
- Investigating customer-specific issues by filtering by user ID across logs/traces/metrics.
Common Mistakes
- Logging unstructured text strings, searchable in a pinch, useless for aggregation. Use structured logs (JSON) with consistent fields.
- Treating metrics as a substitute for traces; you can't reconstruct a single user's bad request from histograms.
- Cardinality explosion: tagging metrics with user IDs blows up storage and cost, that's what logs/traces are for.
- Skipping correlation IDs so logs, traces, and metrics can't be linked back to one request.
Observability, Frequently Asked
Observability vs. monitoring, what's the difference?
Monitoring is about watching predefined signals (uptime, CPU, error rate). Observability is about being able to investigate the unknown. A good system has both: alerting (monitoring) and the ability to drill in when alerts fire (observability).
Do I need all three pillars from day one?
Start with structured logs and a few key metrics (error rate, latency, throughput). Add distributed tracing once you have more than two services calling each other. OpenTelemetry instrumentation gives you all three with similar effort, so it's worth standardizing on it early.
Why is observability so expensive?
High-cardinality data and high data volume, every request generates logs, spans, and metrics. Sampling, retention policies, and shipping only what you need to investigate (vs. everything just in case) are how teams keep costs sane.