Kubernetes Node NotReady

• kubectl get nodes shows NotReady (or Unknown) for a node; Ready condition's lastHeartbeatTime is minutes old.
• The node gains the node.kubernetes.io/not-ready taint; pods without a toleration get evicted after their tolerationSeconds (default 300s for the system default).
• Pending pods accumulate if the remaining nodes lack capacity; node-level dashboards (allocatable pods, kubelet uptime) flatline.

• Sev-3 for a worker in a large cluster with headroom — the scheduler absorbs it, nobody notices but the on-call.
• Sev-1 when the node runs a singleton or stateful pod without a toleration window, when remaining capacity cannot absorb the evictions, or — worst case — the NotReady node is a control-plane member and etcd quorum is at risk.

• Do NOT cordon or drain MORE nodes "to be safe" — draining reduces capacity exactly when you need it. Touch only the broken node.
• Check etcd health FIRST if the node is control-plane: a not-ready worker is an availability problem; lost etcd quorum is a cluster.
• If the node hosts singleton/stateful workloads, raise their tolerationSeconds or apply a temporary toleration so eviction does not outrun your diagnosis.

• One node or several? Several at once points at the network, the control plane, or a bad node-image rollout — different playbook.
• Does the node respond to SSH? Reachable-but-NotReady is usually kubelet/container-runtime; unreachable is a host/cloud problem.
• kubectl describe node: the Ready condition's reason (KubeletNotReady, ContainerRuntimeNotReady) names the failing component before you log in.

1. Kubelet stopped or wedged (OOM, failed upgrade, cgroup stall) — most common on long-lived manually-managed nodes. 2. Container runtime (containerd) dead or unresponsive — kubelet reports NotReady when it cannot ping the runtime. 3. Node under disk/memory pressure evicting the kubelet's own overhead, or the node image filled /var with logs. 4. Network partition: the node works locally but cannot reach the API server (security group, route, DNS). 5. Host/cloud failure: instance stopped, hardware replacement, snapshot hung — the node is simply gone.

• Node SSH-able, kubelet inactive → restart kubelet (a); if it loops, read journalctl for the refusing reason.
• Node SSH-able, kubelet active, runtime errors in logs → restart containerd before anything heavier (b).
• Node SSH-able, disk >90% → free space / expand volume (c); kubelet usually self-recovers once it can write.
• Node unreachable from both SSH and kubelet → cloud-level failure (d): replace the node, do not wait it out beyond your declared threshold.
• Several nodes NotReady at once → stop: suspect network/control plane/upgrade, escalate to the platform on-call with the timeline.

• (a) Kubelet: systemctl restart kubelet on the node; watch kubectl get node flip Ready within ~1 minute. A kubelet that needs regular restarts is a bug — capture journalctl for the follow-up, do not script around it.
• (b) Runtime: systemctl restart containerd then the kubelet; expect pods on the node to restart — announce it in the incident channel BEFORE running it.
• (c) Disk: clear what is safe (rotated logs, dangling images via the runtime's prune — read what it lists before confirming), then grow the volume through the cloud.
• (d) Replace: for autoscaled nodes, delete the instance and let the ASG replace it; for static nodes, cordon + drain the NotReady node (kubectl drain <node> --ignore-daemonsets --delete-emptydir-data — it will mostly time out on unreachable pods; use --force on a confirmed-dead node only), then remove it from the cluster and provision a successor.
• Never "fix" repeated NotReady by raising tolerationSeconds everywhere — that trades eviction latency for silent downtime.

• kubectl get node <node> Ready continuously for 15+ minutes; heartbeat current.
• Pod count on the node back to baseline; no pods Pending cluster-wide; DaemonSets re-scheduled.
• If the node was replaced: the new node Ready, the dead object gone (kubectl get nodes has no ghost entry).

• Platform/cluster team for kubelet, runtime, or node-image problems; cloud on-call for instance-level failures; service owners of anything still Pending after the node recovered. For control-plane NotReady, escalate immediately — that is a cluster-risk incident, not a node one.
• Hand-off state: node name, condition reason, SSH-ability, kubelet journal excerpt, what was restarted/replaced, current pod placement.

• Node auto-repair / health remediation where the platform offers it (GKE auto-repair, ELC autoreplacement, ASG replace-on-health-check).
• Alerts on kube_node_status_ready and on heartbeat AGE (not just current state) — the age alert catches wedged nodes the state alert misses.
• Multiple nodes for every workload class + PodDisruptionBudgets so one NotReady node is a non-event, not an outage.
• Keep node images small and /var on a dedicated volume with a disk-pressure alert at 80%.

• kube_node_status_ready / kube_node_status_condition per node.
• Kubelet uptime and heartbeat age; containerd service health.
• Node disk usage (kubelet_volume_stats_available_bytes or node-exporter equivalents) and Pending-pod counts.

• Kubernetes node conditions
• Troubleshooting clusters