Kubernetes OOMKilled

• Pod events show OOMKilled (exit code 137); the container may restart (CrashLoopBackOff) or the job simply fails.
• kubectl describe pod shows lastState.terminated.reason: OOMKilled with the exit code; memory graphs show usage climbing to the limit and then the cliff.
• Node stays healthy (the kernel's cgroup OOM killer acted on the CONTAINER, not the node) — no node-level signal unless pods are being evicted too.

• Sev-2 for a stateless deployment with siblings absorbing traffic; Sev-1 for single-replica services, jobs whose failure blocks a pipeline, or when the OOM is a leak that will kill every replica in sequence (rolling death).
• Sev-3 for a batch job that can rerun cleanly — but recurring OOMKill of the same job is data-loss-adjacent if mid-run state exists.

• Do NOT just raise the limit before capturing the evidence: kubectl top history (or your metrics store) around the kill is what tells leak from under-provisioning.
• If a rollout introduced it, kubectl rollout undo first — a regression to a known-good memory profile beats live-tuning a leak.
• If the pod holds caches (local page cache, model weights), check whether the cache grew (config) or the working set grew (leak) before touching limits.

• Usage vs limit: was memory climbing linearly over hours/days (leak) or plateauing just over the limit (limit too low)?
• One pod or all pods of the workload? All pods hitting a flat ceiling = sizing problem; one pod diverging = per-instance load or a leak triggered by a request pattern.
• Did the application recently change (image, config, traffic mix)? JVM flags, memory-hungry dependency, larger payloads.

1. Limit set below steady-state usage (the classic: limit copied from a hello-world manifest) — usage plateaus at/below the old footprint. 2. Memory leak in the application (native libs, caches without bounds, listeners never detached) — usage climbs linearly and every replica dies in sequence. 3. Traffic or payload growth (bigger responses, more concurrent requests) — usage tracks load shape. 4. Runtime mis-sizing: JVM heap larger than the container limit (-Xmx close to the limit leaves no room for metaspace/threads/off-heap).

• Plateau just over the limit, stable across replicas → sizing: raise limit to observed peak × 1.5 and set the request honestly (a).
• Linear climb over hours/days, survives restarts → leak: raise the limit as a stopgap to buy alert time, capture a heap dump, file the bug (b).
• Climb tracks traffic → capacity: scale replicas or bound concurrency; re-derive limit from peak+headroom (c).
• JVM: exit right after start, heap near limit → fix -Xmx to ~75% of the container limit (d).

• (a) Re-size: set resources.requests.memory to real steady-state and limits.memory to peak × 1.5; deploy and WATCH the graph — memory sizing is an experiment, not a guess with more zeros.
• (b) Leak: temporary limit bump + alert at 80% of the new limit; heap dump before the next kill (jmap / node inspector / Go pprof as fits); the fix is a code change, never a permanent limit ratchet.
• (c) Capacity: add replicas, bound request concurrency or payload size; for caches, set explicit max-size/eviction instead of "the OS will handle it".
• (d) Runtime flags: container-aware runtimes default sanely — if you set -Xmx explicitly, keep it ≤75% of the limit; remove flags you copied from a 2018 blog post.
• If the workload genuinely needs more memory than a node can give, that is a platform conversation (bigger node class or redesign), not a bigger fudge factor.

• 7 days of memory graphs with peak usage ≤70% of the new limit (headroom for traffic bursts and GC spikes).
• Zero OOMKilled events in that window; restart counts stable.
• Requests reflect reality: node memory allocation (sum of requests) still leaves the node below its allocatable — no shifted node-pressure evictions.

• Service owners for application leaks and sizing (they own the profile); platform team for node-class limits; data owners if a batch job is dying mid-run and re-running has side effects.
• Hand-off state: usage graphs around the kills, limits (old/new), leak-vs-sizing verdict, heap dump if captured.

• Alert at 80% of container memory limit (working set) — that is the OOM early-warning tier.
• Size limits from observed metrics (p99 + burst), review on every major traffic change; requests = steady-state so scheduling is honest.
• Golden manifests per runtime: JVM with container-aware flags, caches with bounds — defaults that fail safe, not fail loudly.
• Soak tests (hours, not minutes) before promoting workloads with new dependencies.

• container_memory_working_set_bytes vs limit per container.
• OOM kill counter per node/container (container_oom_events_total or cgroup events).
• Restart counts and CrashLoopBackOff alerts as the downstream symptom.

• Resource management for pods and containers
• Debugging OOMKilled pods (kubectl troubleshooting)