Skip to main content
Reliability & Runtime OpsLast reviewed 2026-09-20

SRE

The discipline that treats reliability as a software problem: SLOs define how unreliable you may be, error budgets turn that into policy, and engineering time replaces toil.

Overview

Read the full SRE guidance
WHAT IT IS. Site Reliability Engineering is Google's answer to a hard question: when your product team wants to ship constantly and your users want the service to never break, who decides? SRE decides with numbers. A Service Level Objective states exactly how unreliable the service is allowed to be in a window (99.9% of requests successful in 30 days). The remaining allowance — the error budget — is a resource the team spends on change. When the budget burns too fast, releases freeze until reliability recovers; when it is healthy, nobody argues about deploys. WHY IT EXISTS. "Reliable" and "fast" are usually settled by whoever shouts loudest. SRE replaces the argument with a negotiated, written contract — the error-budget policy — that both product and engineering signed. It also names the silent tax on operations: toil, the manual, repetitive, automatable work that grows until it consumes the team. SRE caps toil explicitly and spends the freed time building systems that make the service run itself. WHEN TO USE IT. The moment a service has users who lose money, trust, or patience when it fails. Before that, SLOs are ceremony. It scales down honestly: a three-person team can run one SLO and a one-paragraph budget policy; that IS SRE at small scale. What does not scale down is skipping the policy — an SLO without a consequence attached is a dashboard, not a contract. WHO OWNS IT. Service teams own their SLIs, SLOs, and the budget they spend; a reliability or platform group owns the shared machinery — telemetry, alerting, the release-freeze automation — and acts as an internal contractor with explicit engagements, not a permanent owner of other teams' pages. Product management co-signs the error-budget policy because it trades their release velocity. WHAT GOOD LOOKS LIKE. Every tier-1 service has a user-felt SLI (not CPU, but successful checkouts), an SLO baselined from real traffic, a written budget policy that has actually paused a release at least once, alerts that page only on budget burn or explicit user symptoms, and a toil ledger showing manual operations shrinking quarter over quarter. Google's SRE book and workbook (linked below) are the primary sources; the error-budget chapter is the one to read twice. HOW TO IMPLEMENT. Pick the one service whose outage would hurt most. Define one SLI from existing telemetry, set the SLO where current reality is (not where you wish it were), write the one-paragraph policy — "burned budget this week means releases hold until it recovers, here is who can waive it" — and wire one multi-window burn-rate alert. One service done honestly beats five done as paperwork. HOW TO ASSESS YOURSELF. The production readiness assessment scores SLO management, alerting quality, and toil directly — see /assess. The maturity ladder below stages L1 (no SLOs, firefighting) to L5 (budgets steer roadmap). Related: /observability-engineering supplies the telemetry SLOs depend on, /incidentops runs the failure side, /resilience-engineering extends budgets into rehearsed failure.

Capabilities

  • Alerting

    Symptom-based, SLO-backed alerts that page a human only when a user-visible promise is burning.

  • Capacity Planning

    Headroom against measured peak load, with rehearsed scaling procedures.

  • Error Budget Governance

    A written policy that trades release velocity for reliability when an error budget burns.

  • Incident Response

    Detecting, coordinating, and resolving unplanned interruptions with defined roles and severities.

  • SLO Management

    Defining, measuring, and governing Service Level Objectives and their error budgets.

  • Toil Management

    Measuring manual operational work, capping it as a budgeted share, and automating it away deliberately.

Practices

  • Blameless Postmortems

    Incident reviews that fix systems, not people — with tracked action items.

  • Error Budget Policy

    A written contract trading release velocity for reliability when budgets burn.

  • Toil Budgeting

    Measuring and capping manual operational work to fund automation.

Metrics

  • Alert Noise Ratio

    Share of pages that required no action.

    Unit: %. Good direction: down.

  • Availability

    Achieved uptime against the defined SLO window.

    Unit: %. Good direction: up.

  • Error Budget Burn Rate

    Speed at which the error budget is being consumed.

    Unit: × budget/hour. Good direction: down.

  • MTTD — Mean Time to Detect

    Average time from fault to detection by a human or automated signal.

    Unit: minutes. Good direction: down.

  • MTTR — Mean Time to Restore

    Average time from incident start to service restoration.

    Unit: hours. Good direction: down.

  • SLO Coverage

    Share of tier-1 services with a defined, measured SLO.

    Unit: %. Good direction: up.

  • Toil Ratio

    Share of engineering time spent on manual, repetitive operational work.

    Unit: %. Good direction: down.

Tools

  • Alertmanager

    Prometheus's alert routing layer: deduplication, grouping, silencing, and escalation.

    Visit site ↗
  • Grafana

    Dashboards and visualization for metrics, logs, and traces.

    Visit site ↗
  • PagerDuty

    On-call scheduling, escalation, and incident mobilization.

    Visit site ↗
  • Prometheus

    Open-source metrics collection and alerting toolkit.

    Visit site ↗

Architecture patterns

Reference architectures and their trade-offs live in the blueprints library.

Maturity

  1. L1 ReactiveNo SLOs — firefightingrequired stage

    Reliability is an opinion; outages are surprise; toil is unmeasured.

  2. L2 RepeatableFirst SLOs defined

    Tier-1 services have SLIs and targets baselined from real traffic; budgets not yet enforced.

  3. L3 StandardizedBudget policy in actual userequired stage

    A written error-budget policy has real consequences — it has held or freed a release at least once.

  4. L4 MeasuredToil budgeted and shrinking

    Toil is measured, capped as a share of time, and the freed capacity visibly funds automation.

  5. L5 OptimizedReliability steers roadmap

    Budget health and toil trends input directly into planning; alerts page only on symptoms.

Maturity for sre is measured, not guessed — every score traces to your answers. See how maturity is scored.

Put it to work

Sources