Observability
Observability is the ability to understand a system’s internal behavior from its external signals. In modern distributed systems, this usually means logs, metrics, traces, events, profiles, and deployment metadata. The point is not collecting everything. The point is answering production questions quickly.
Signals
Metrics are numerical measurements over time. Logs are timestamped event records. Traces describe the path of a request through services. Events record discrete changes such as deployments, autoscaling, failovers, and feature-flag flips.
Correlation matters. A trace ID in logs, metrics with service and version labels, and deployment events on dashboards can turn an outage from archaeology into diagnosis.
Useful Instrumentation
Good telemetry represents user harm and system saturation. Request count, error rate, latency, and resource saturation are a starting point. Business-specific signals are often more important: failed checkouts, delayed messages, missing invoices, or authorization denials.
Cardinality and Cost
Observability systems can become expensive or unusable when labels have uncontrolled cardinality. A metric labeled by user ID, request ID, or raw URL can create millions of time series. The result is high cost and slow queries. Good instrumentation chooses labels that support diagnosis without exploding storage: service, route template, status class, region, version, and dependency are common examples.
Logs have similar tradeoffs. Structured logs are easier to query than free text, but sensitive fields must be redacted. Production logs should not contain access tokens, personal data, or secrets. Observability is part of security.
Tracing in Practice
Distributed tracing is strongest when it follows user requests across service boundaries. A trace can show that the API was fast until it called inventory, which waited on a database, which retried a locked query. Without traces, each team may see only its local span of the failure and blame the network.
Instrumentation should be consistent. If half the services propagate trace context and half drop it, the most important incidents will have broken evidence.
Observability as Design Feedback
If a system is impossible to observe, it is often poorly modularized. Missing service boundaries, unclear ownership, and unstructured error handling all appear as telemetry gaps. Adding observability can expose architecture debt.
Study Task
Pick one user journey and design its telemetry. Include the top-level SLI, trace spans, log fields, key metrics, deployment annotations, and the dashboard an on-call engineer would open first.
Operational Review
Observability should be reviewed with real questions, not only signal inventories. Can the team answer why latency rose after the last deployment? Can it identify which customer segment is failing? Can it connect a trace to logs and the exact artifact version? Can it see whether a background job is delayed rather than merely running?
The review should include privacy and retention. Collecting more telemetry is not automatically better. Sensitive data, excessive cardinality, and indefinite retention create cost and risk. A good observability system collects enough context to diagnose failure while respecting data boundaries.
Finally, review whether telemetry changes behavior. If dashboards are impressive but never used during incidents, they may be decorative. The best signals appear in alerts, runbooks, release checks, and post-incident learning.