Resilience Engineering

WHAT IT IS. Resilience engineering starts from an honest premise: in complex systems, failure is not an anomaly to prevent but a condition to survive. The discipline designs for it on three levels. Architecturally: redundancy whose components do not share failure modes (two AZs, not one AZ with a warm standby), backpressure and load shedding instead of unbounded queues, and graceful degradation — a checkout that still works without recommendations beats a checkout that is down. Organizationally: rehearsal, because a failover that has never been exercised is a hypothesis. Culturally: studying how work actually succeeds near the edge (the resilience-engineering literature calls this work-as-done versus work-as-imagined) instead of auditing it against the diagram. WHY IT EXISTS. Redundancy on paper is the signature failure of this discipline: the second node existed, the failover procedure existed, and the first real event found the two systems shared a database, a credential, or a person. Complexity makes such couplings invisible until they fire. NIST SP 800-34 (contingency planning, linked below) and AWS's reliability pillar provide the citable scaffolding: recovery objectives per service (RTO/RPO), rehearsed plans, and tested dependencies. The practice that binds them is the game day — deliberately failing a dependency under controlled conditions, which is chaos engineering's experimental method (Principles of Chaos, linked below) applied on a schedule. WHEN TO USE IT. As soon as an outage would cost something you care about — revenue, reputation, or a compliance letter. The honest sequencing: before multi-region, know your dependencies; before DR, test your backups' restores; before chaos experiments, have the observability to detect what you broke. Each layer assumes the one beneath it. WHO OWNS IT. Distributed by design: each service team owns its failure modes, its degradation modes, and its recovery objectives; leadership owns the rehearsal culture, because game days are the first thing a deadline eats and only management can make them un-skippable. The platform group runs the rehearsal calendar and the dependency map's tooling — not the resilience itself, which cannot be centralized. WHAT GOOD LOOKS LIKE. Every tier-1 service has stated RTO/RPO that the business has actually agreed to, and a failover or restore has been exercised within the quarter with the clock recorded. Capacity headroom against peak times a safety factor is known, watched, and rehearsed for scaling. The dependency map includes the vendor you forgot you depended on — the email API, the DNS provider. Degraded modes are designed, documented, and rehearsed: the team knows what the product looks like at 60% function. Post-incident, the question asked is "how did the system make that easy to do," not "who touched it." HOW TO IMPLEMENT. Map tier-1 dependencies first — including the third parties — because everything else depends on knowing them. Define RTO/RPO per service with the business. Then schedule the first game day: pick one dependency, break it in a controlled window, measure time-to-detect and time-to-recover, and fix what the rehearsal exposed. Automate nothing about the first one; the manual run IS the discovery of what automation is missing. NIST 800-34's contingency-planning structure maps the whole ladder if you need the formal version. HOW TO ASSESS YOURSELF. The production readiness assessment scores redundancy, DR readiness, backup/restore, capacity planning, and performance validation with citations: /assess. On the maturity ladder below, unrehearsed best intentions is L1; designed-for-degradation and studied, not assumed, is L5. Related: /sre quantifies how much failure is affordable, /incidentops runs the failures that still happen, /cloudops owns the failure domains themselves.