Observability Engineering

WHAT IT IS. Observability is the property of a system whose internal state can be interrogated from its outputs. Monitoring tells you THAT something is wrong against questions you wrote in advance; observability lets you ask WHY — a question you could not have predicted — without shipping new code to ask it. The raw material is high-cardinality, high-dimension events: traces that follow one request across every service, structured logs that carry the trace ID, metrics with the labels that matter. OpenTelemetry (CNCF, linked below) is the vendor-neutral standard that makes the three one instrumentation concern instead of three procurement decisions. WHY IT EXISTS. The debugging cost of a distributed system is dominated by the questions you cannot yet ask: "why is checkout slow for this one customer, on this path, since this deploy?" If answering that requires adding a log line and redeploying, the answer arrives after the outage is over. Unified telemetry with consistent context turns novel questions into queries. The second, less discussed reason is cost discipline: telemetry volume grows superlinearly with the estate, and without deliberate sampling and retention policy it becomes a bill nobody budgeted — observability includes observing itself. WHEN TO USE IT. Before you need it — instrumentation retrofitted mid-incident is too late by definition. The trigger is a second service: the moment a request crosses a process boundary, logs alone stop being sufficient and trace context becomes the only honest way to follow it. WHO OWNS IT. Each service team owns its instrumentation — what it emits, with what dimensions — because only they know their failure modes. A platform or observability group owns the pipes: the collectors, the storage, the query layer, the sampling and retention policy, and the cost. Alerting sits at the intersection: symptom-based, SLO-backed pages belong to the service team; pipeline health belongs to the platform. WHAT GOOD LOOKS LIKE. An engineer can open a trace from a user-visible symptom (a slow checkout, a failed request) and walk it across every service boundary with consistent context, then pivot to that span's logs without a second query language. Debuggability latency — time from "huh, that's weird" to an evidence-backed answer — is minutes, and the team measures it. Alerts page on user-visible symptoms and budget burn, not on CPU. Cardinality is treated as a feature, with cost controls deliberate rather than accidental. HOW TO IMPLEMENT. Instrument one critical path end to end with OpenTelemetry traces — propagate context explicitly across every boundary, including the queue you forgot. Correlate: structured logs carrying trace and span IDs. Then metrics with the dimensions incidents actually slice by (route, status class, deploy version). Only then dashboards — one per service from a shared template — and alerts on the SLOs. Sampling and retention policy come with the bill. HOW TO ASSESS YOURSELF. The production readiness assessment scores monitoring coverage, tracing, log management, and alerting against their citations: /assess. The maturity ladder below runs from dark services (L1) to interrogable systems with measured debuggability (L5). Related: /sre consumes this telemetry as SLOs and error budgets, /incidentops lives on it when things burn.