Checklists / chaos-experiment-safety
Chaos Experiment Safety Checklist
The four things that separate an experiment from a self-inflicted outage: a written hypothesis, a bounded blast radius, a tested abort, and responders who know it is happening.
8 steps · 7 required · last reviewed (next review 2027-03-20) · updated 2026-09-20
Related focus areas:Chaos Engineering
Roadmaps covering the same ground:Incident Management MaturityResilience and DR Improvement
Run this before every experiment, including the small ones and especially the ones you have run before. An experiment without these four properties is an outage you caused on purpose, and the difference is entirely in the preparation.
If the system has known unowned alerts or no incident process, stop: injecting faults there produces incidents rather than learning.
A hypothesis about steady-state behaviour is written down before injecting anything
requiredPredicting the outcome first is where most of the value is; without it you get an anecdote.
Steady state is defined as a measurable signal, not a feeling
requiredThe experiment must be able to conclude something, which requires a number to compare against.
Blast radius is bounded to the smallest useful scope
requiredOne instance or a traffic fraction first; widen only after the small version behaves as predicted.
An abort condition is defined and the stop mechanism has been tested
requiredDiscovering that the stop button does not work is the worst possible finding to make mid-experiment.
The people on call know the experiment is running and when
requiredA surprise experiment is indistinguishable from an incident and wastes a night proving that.
The experiment runs at a time chosen for recovery capability, not convenience
requiredThe point of choosing the moment is having the right people available and awake.
The result will be recorded whether or not it is surprising
requiredUnrecorded boring results are how the same experiment gets run three times by three teams.
A follow-up owner is named for any finding before the experiment starts
optionalFindings without an owner become a list, and the list is the reason the practice gets cancelled.
Run it against your real gaps
A checklist is most useful when you know which controls you are actually missing. A free OpsRoadmaps assessment scores your controls with explainable findings and builds a roadmap from the results.
Assess my stackChecklists document self-reported practice. Items were last reviewed 2026-09-20 — adapt them to your own environment.