Skip to main content
OpsRoadmaps
Roadmap library3 phases · 8 steps · 7 dependenciesLast reviewed 2026-09-19

Observability Maturity

From scattered logs to answerable questions: OpenTelemetry-instrumented services with propagated traces, shared dashboards and SLO-alerts, then cost policy and time-to-answer as managed metrics.

Every step is executable work: what, why, prerequisites, implementation, validation you can check, and the failure modes to avoid. Dependencies make the order explicit — a step marked Requires waits on those steps completing first.

8 steps

Phase 1

Unified telemetry

One instrumentation standard across services: OTel SDKs, trace context that survives hops, logs that carry trace ids.

  1. Adopt the OpenTelemetry SDK in every service

    P0M

    Observability EngineeringPlatform Engineering

    Full step — what, why, how, failure modes
    WHAT. OTel SDK installed and exporting traces and metrics from every production service — one vendor-neutral standard, however the backend evolves. WHY. Per-language bespoke instrumentation cannot follow a request across services and locks you to whatever tool you hand-rolled for; OTel makes telemetry a platform property instead of a per-service habit. PREREQUISITES. None — start with the highest-traffic service. IMPLEMENTATION. Add the OTel SDK via the service template or base image so new services inherit it, auto-instrument the frameworks (HTTP, DB drivers) first, export to your existing backend (most ingest OTLP natively), and exclude health-check noise at the source. TOOLS. OpenTelemetry SDKs and auto-instrumentation agents; an OTLP-capable backend. COMMON FAILURE MODES. Sampling everything to 1 percent at rollout — traces exist but never reproduce the rare bug; start at 10-100 percent on low-volume paths. EXPECTED IMPACT. Telemetry stops being a per-service lottery, and backend changes stop meaning re-instrumenting the fleet. EVIDENCE. The catalog-vs-telemetry diff output and one service's trace list.

    Validation — before you call it done

    Every service in the catalog reports traces and metrics to the backend (a catalog-vs-telemetry diff of 0 missing services), and each shows at least request-rate and error-rate as auto-instrumented metrics.

  2. Propagate trace context across every boundary

    P0MRequires: Adopt the OpenTelemetry SDK in every service

    Observability Engineering

    Full step — what, why, how, failure modes
    WHAT. W3C trace-context headers (traceparent) forwarded across all service-to-service calls — HTTP, queues, and scheduled jobs — so one trace follows one logical request. WHY. A trace that dies at the first queue hop answers nothing about where latency went; propagation is the difference between tracing and per-service logs with correlation guesses. PREREQUISITES. OTel SDKs in the services on both sides of each boundary. IMPLEMENTATION. Enable context propagation in HTTP clients and servers (usually automatic once the SDK is present), inject/extract context into message headers for queues, propagate job metadata for scheduled work, and verify with a synthetic request that the trace id survives each hop. TOOLS. W3C trace context; your queue client's header hooks; the trace UI. COMMON FAILURE MODES. Forgetting the queue — HTTP hops work, and the trace still splits at the most interesting boundary. EXPECTED IMPACT. The question where did those 800ms go becomes answerable by looking instead of reasoning. EVIDENCE. The multi-service trace screenshot with the queue span visible.

    Validation — before you call it done

    A synthetic end-to-end request produces 1 unbroken trace spanning every service it touched (including 1 queue hop), verified in the backend's trace view.

  3. Emit structured logs with trace ids

    P1MRequires: Adopt the OpenTelemetry SDK in every service

    Observability Engineering

    Full step — what, why, how, failure modes
    WHAT. JSON logs (or structured events) from every service, each entry carrying trace id, span id, service, and severity — queryable fields, not grep fodder. WHY. When a trace pins the slow request, the next question is what did it say — and unstructured logs answer that with a person scrolling; the trace id is the join key that makes logs and traces one system. PREREQUISITES. OTel SDK present to supply the trace context cheaply. IMPLEMENTATION. Switch each service's logger to structured output (most frameworks do this natively), auto-attach trace/span ids from the active context via the OTel logging bridge, adopt a small common field set (service, env, severity, message, plus the trace pair), and ship to a log store that indexes the fields. TOOLS. pino/structlog/zap or equivalent; OTel log bridge; a field-indexed log store. COMMON FAILURE MODES. Structured logs that still pack the payload into one message string — structured envelope, unstructured contents; log the fields you will query. EXPECTED IMPACT. Logs become a queryable extension of traces instead of a separate silo. EVIDENCE. The trace-to-logs query demonstrated on the 3 samples.

    Validation — before you call it done

    For 3 sampled traces, every log line those requests produced is retrievable by trace id in under 30 seconds, with fields (not regex) doing the filtering.

Phase 2

Ask anything

Telemetry nobody looks at is inventory: critical paths traced end to end, dashboards people share, alerts on symptoms.

  1. Trace the money path end to end

    P1MRequires: Propagate trace context across every boundary

    Observability EngineeringSRE

    Full step — what, why, how, failure modes
    WHAT. The one journey that pays the bills (browse-to-buy, search-to-book) instrumented with hand-written spans at each business step — not just framework auto-spans. WHY. Auto-instrumentation sees technology boundaries (http, sql) but is blind to business meaning; named spans for checkout started, payment authorized, order persisted make the trace readable to the business, not just the author of the service. PREREQUISITES. SDKs and propagation working along that path. IMPLEMENTATION. Add spans with business-meaningful names at each step of the primary revenue path, tag them with the domain identifiers (order id, cart id), and check the resulting trace tells the story of one order without needing the source open. TOOLS. Manual span APIs; domain id attributes. COMMON FAILURE MODES. Span-per-line-of-code noise — span per business step is the goal; everything else is auto-instrumentation's job. EXPECTED IMPACT. Latency and failure conversations start from the customer journey, not the infrastructure diagram. EVIDENCE. The annotated trace and the new-engineer read-through note.

    Validation — before you call it done

    A sampled trace of the money path shows each business step as a named span with domain ids attached (3-plus spans for a typical purchase), and a new engineer can state where time went in under 5 minutes without opening the code.

  2. Alert on symptoms with burn-rate windows, not raw thresholds

    P0MRequires: Trace the money path end to end

    Observability EngineeringSRE

    Full step — what, why, how, failure modes
    WHAT. Alerts defined from SLOs and user-felt symptoms (error-ratio burn, latency budget) — paging only when users are harmed, ticketing for the rest. WHY. CPU at 81 percent at 3 a.m. is not an emergency — users failing checkout is; symptom alerts page proportional to harm, which is the only sustainable pager policy (the SRE monitoring chapter and burn-rate practice). PREREQUISITES. Metrics and the money-path spans providing user-level signals. IMPLEMENTATION. For each tier-1 service: define the SLI from existing telemetry, set alert windows on burn rate (fast page, slow ticket), migrate or delete the inherited cause-based alerts for the same services, and document each alert's meaning and first response in a linked runbook. TOOLS. Prometheus/Alertmanager or backend SLO alerting; the runbook repo. COMMON FAILURE MODES. Keeping the old alerts running in parallel — dual alert streams is how on-call learns to trust neither. EXPECTED IMPACT. The pager speaks only when users hurt, and every page starts with a runbook. EVIDENCE. The alert config diff and the 30-day page log with symptom mapping.

    Validation — before you call it done

    Every page fired in the last 30 days maps to a user-felt symptom with its runbook link, and the pre-existing cause-threshold pages for those services are gone (alert-config diff shown).

  3. Build shared service dashboards, not personal ones

    P1SRequires: Trace the money path end to end

    Observability Engineering

    Full step — what, why, how, failure modes
    WHAT. One dashboard per service (red-rate, latency, saturation, plus the business step counts), generated from the template, owned by the service team, linked from the catalog. WHY. Personal dashboards die with their author's vacation and hide the incident's shape from everyone else; shared ones are where the team converges during an incident and between them. PREREQUISITES. SDK metrics flowing from the services. IMPLEMENTATION. Define the standard service dashboard in the template (the four golden signals plus domain counters), provision it automatically per service from the catalog, delete or privatize the graveyard of personal panels, and link each service's dashboard in the catalog entry. TOOLS. Your metrics backend's dashboard-as-code (Grafana provisioning et al.); the service catalog. COMMON FAILURE MODES. Dashboards with 40 panels — nobody finds the one that matters at 3 a.m.; four signals and the business line, nothing else above the fold. EXPECTED IMPACT. Every service has one obvious place to look, and on-call inherits answers instead of archaeology. EVIDENCE. The catalog export with dashboard links and one opened dashboard.

    Validation — before you call it done

    Every service in the catalog links a working shared dashboard showing all 4 golden signals, and each has a named owning team in the dashboard header.

Phase 3

Observability as a product

The steady state: measured time-to-answer, and a sampling/retention policy that treats telemetry cost as a design constraint.

  1. Measure time-to-answer for hard questions

    P2SRequires: Build shared service dashboards, not personal ones

    Observability Engineering

    Full step — what, why, how, failure modes
    WHAT. A small set of standing questions (which orders failed payment yesterday, what caused the p99 spike, who is affected by this bug) answered on the clock monthly — the elapsed time from question to supported answer. WHY. Observability's real output is answered questions; dashboards and retention are inputs, and time-to-answer is the metric that tells you which improvements actually paid. PREREQUISITES. Telemetry, dashboards, and trace-to-logs joins all live. IMPLEMENTATION. Pick 3 recurring questions (one from each of: business, latency, incident-forensics), time answering them each month with the current tooling, log the elapsed minutes and the friction encountered, and pick next month's improvement from the friction list. TOOLS. A stopwatch; the existing telemetry stack. COMMON FAILURE MODES. Timing a softball — the questions must be the ones that actually hurt; cherry-picked questions produce a beautiful, meaningless trend. EXPECTED IMPACT. Telemetry investment gets judged by the only output that matters: answers, faster. EVIDENCE. The timing log with friction notes.

    Validation — before you call it done

    Three consecutive monthly timings exist for all 3 questions, and the current month's median is lower than the first month's.

  2. Set a sampling and retention policy with cost accounting

    P2MRequires: Build shared service dashboards, not personal ones

    Observability EngineeringFinOps

    Full step — what, why, how, failure modes
    WHAT. A written policy: head sampling rates per path (100 percent for errors and low-volume critical paths, less for high-volume health traffic), retention tiers per signal, reviewed against its own monthly cost line. WHY. Observability cost compounds silently — 100 percent traces forever is a future outage in the bill; a policy makes the tradeoffs explicit instead of discovered at budget time. PREREQUISITES. Telemetry volume large enough to matter on the bill and the anomaly hooks from the cost program if present. IMPLEMENTATION. Enable tail-sampling or head rules that keep all error traces and sample successes (e.g., keep 100 percent of errors, 10 percent of clean high-volume paths), set retention per signal (e.g., metrics 13 months, traces 14-30 days, logs 30-90 days by tier), document it, and add telemetry cost as a line in the monthly showback. TOOLS. OTel collectors (tail sampling); backend retention settings; the showback report. COMMON FAILURE MODES. Sampling away the errors to save money — the exact traces you will need are the ones the policy must always keep. EXPECTED IMPACT. Telemetry cost becomes a deliberate dial instead of a quarterly surprise, without losing the traces that matter. EVIDENCE. The policy doc and the showback line.

    Validation — before you call it done

    The policy doc states rates and retention per signal type, telemetry spend is visible as its own showback line, and the policy's next review date is within 6 months.

Sources

Start from where YOU are

This roadmap is the general path. An assessment tells you which steps you can skip and which are critical — then generates a version of this plan from your actual findings, tracked to done in your workspace.

Assess where you standBrowse the roadmap library