Kubernetes CrashLoopBackOff

• kubectl get pods shows CrashLoopBackOff with a climbing RESTARTS column; the workload's availability drops or never comes up after a rollout.
• Alerts: KubePodCrashLooping firing; service-level 5xx or capacity loss downstream if the pod backs a serving workload.

• Sev-3 for a redundant deployment with healthy siblings absorbing traffic; Sev-1 for a singleton (the thing simply does not exist while it crash-loops) or when the crash-looping pod holds a leader lease others wait on.

• Do NOT delete the pod yet — its lastState holds the exit code and the previous logs; deleting destroys the evidence.
• If a rollout just started and the previous ReplicaSet was healthy, kubectl rollout undo restores service first; triage the bad revision afterwards.
• Freeze CI/CD for that service so a "fix" deploy does not stack on an unknown cause.

• kubectl describe pod: exit code and reason in lastState.terminated (Error/OOMKilled), and whether probes kill it before it is ready.
• Application logs of the PREVIOUS container (--previous), not the current one — the current one usually has nothing useful.
• Did anything change? Rollout event, configmap/secret edit, secret rotation expiry, node migration to a different image pull policy.

1. Application exits on boot: bad config, missing env var, unreachable dependency at startup (most common). 2. OOMKilled: container memory limit lower than the app's steady-state footprint. 3. Liveness probe too aggressive: kills the pod during slow start (JVM warmup, migrations) — RESTARTS climbs while the app would have become healthy. 4. Command/args error: wrong entrypoint, missing file, wrong user permissions on a mounted volume.

• Exit code 1 with an application error in previous logs → config/dependency problem: fix the config, no probe changes needed.
• Exit code 137 + OOMKilled reason → raise the memory limit to observed usage × 1.5, then investigate the leak if usage still grows.
• Exit code 137 but NOT OOMKilled → liveness probe is killing it: check probe timeout/failureThreshold vs startup time.
• Exit code 126/127 → entrypoint problem: wrong command or missing binary — check the image tag actually deployed.

• Config/boot error: correct the configmap/secret, then restart the pod deliberately (kubectl delete pod after evidence captured).
• OOMKilled: raise resources.limits.memory; for a leak, add a restart policy (vertical autoscaler or cron) as a stopgap and file the leak as a bug with a heap dump.
• Probe kills: add a StartupProbe or lower liveness failureThreshold; never "fix" this by deleting the liveness probe.
• Roll out with maxUnavailable: 0 so a bad revision cannot take the last healthy pod with it.

• RESTARTS stops climbing for 10+ minutes and the pod reports Ready.
• The workload's own health endpoint and golden signals (latency/errors) are green — Ready pods can still be broken inside.
• If OOM was the cause: memory graph shows the new limit has ≥30% headroom at peak.

• Service owners for app-level boot failures (they own the config); platform team for node-level or image-registry issues; if a leader-lease singleton is down, page the on-call for that system explicitly.

• Startup probes on every slow-starting workload so liveness never races initialization.
• Memory limits set from observed usage, not guesses; alert at 80% of limit.
• kubectl rollout undo drilled in game days — rollback under pressure must be muscle memory.

• KubePodCrashLooping and container restart rate per deployment.
• container_memory_working_set_bytes vs limit (OOM early warning).
• Rollout failure alerts (revision stuck below desired availability).

• Kubernetes pod lifecycle and container states
• Container exit codes reference