Production-Ready Kubernetes
Take a Kubernetes cluster from works-on-demo to production-grade: supported versions, hardened nodes, right-sized workloads, disruption budgets, and a rehearsed upgrade and restore path.
Every step is executable work: what, why, prerequisites, implementation, validation you can check, and the failure modes to avoid. Dependencies make the order explicit — a step marked Requires waits on those steps completing first.
Phase 1
Cluster foundations
A supported, hardened, policy-bounded cluster — the substrate everything else assumes.
Run a supported version with a deliberate upgrade cadence
P0SFull step — what, why, how, failure modes
WHAT. Pin the cluster to one supported minor version and commit to upgrading at least twice a year, never falling more than one minor behind upstream support. WHY. Kubernetes ships a minor every ~4 months and supports the latest three; a cluster that drifts two minors back stops receiving security patches and turns every upgrade into a multi-version migration nobody has rehearsed. PREREQUISITES. None — this is the entry step. IMPLEMENTATION. Check the current version against the upstream release/support matrix, record it in the platform docs, and put two upgrade windows per year on the team calendar with a named owner. Budget a n-1 ceiling: when upstream announces a version is exiting support, the next window upgrades to it. TOOLS. kubectl; the upstream Kubernetes release page in the team wiki. COMMON FAILURE MODES. Upgrading only when forced by an EOL announcement — those upgrades skip versions and land mid-quarter with no rehearsal time. EXPECTED IMPACT. Security patches keep arriving, and every upgrade stays a one-step, rehearsed change instead of an archaeology project. EVIDENCE. kubectl version output next to the upstream support page, plus the calendar invite series.Validation — before you call it done
Cluster version is within one minor of the latest upstream release, and the next two upgrade windows exist on a shared calendar with owners, each less than 6 months away.
Audit RBAC down to least privilege
P0MFull step — what, why, how, failure modes
WHAT. Enumerate every Role, ClusterRole, and binding; remove wildcard verbs/resources and cluster-admin grants that nobody can justify. WHY. In most clusters, a handful of legacy bindings grant cluster-admin to humans and CI accounts alike; any one of those credentials is full-cluster compromise. PREREQUISITES. None, though it lands best after node hardening so both layers of the same audit close together. IMPLEMENTATION. Dump all bindings with kubectl, map each subject to an owner and a purpose, delete bindings nobody claims within a set window, and replace justifiable broad grants with namespace-scoped roles. Bind human access to groups (SSO-backed), not individuals, and make ServiceAccount tokens short-lived. TOOLS. kubectl auth can-i; an RBAC audit tool such as rbac-lookup or kubiscan. COMMON FAILURE MODES. Auditing only human roles — the stale CI pipeline token with cluster-admin is the classic survivor. EXPECTED IMPACT. A stolen namespace credential can damage its namespace, not the cluster. EVIDENCE. The before/after binding dump plus the break-glass procedure document.Validation — before you call it done
kubectl auth can-i --list as each CI account shows no cluster-scoped write verbs, and zero subjects hold cluster-admin except the break-glass role, which is documented and unused for 90 days.
Give every team a namespace with quotas and limit ranges
P1MRequires: Audit RBAC down to least privilegeFull step — what, why, how, failure modes
WHAT. One namespace per team or service, each carrying a ResourceQuota and a LimitRange so requests and limits are defaults, not opt-ins. WHY. Without quotas, one mis-sized deployment starves every neighbor on the node — noisy-neighbor at cluster scale; without limit ranges, half the workloads ship with no requests at all. PREREQUISITES. RBAC roles exist so namespace boundaries mean something for access too. IMPLEMENTATION. Define the namespace taxonomy (per service or per team), attach a ResourceQuota capping CPU/memory/pod-count per namespace and a LimitRange injecting default requests and limits, then roll existing workloads in by namespace over a migration window. TOOLS. kubectl; admission policy (Kyverno or OPA Gatekeeper) to enforce namespace hygiene at create time. COMMON FAILURE MODES. Quotas set generously enough to never bind — a quota that has never rejected anything is documentation, not a control. EXPECTED IMPACT. Blast radius of resource mistakes shrinks from the cluster to the namespace that made them. EVIDENCE. kubectl get resourcequota,limitrange -A output attached to the platform doc.Validation — before you call it done
Every non-system namespace reports a ResourceQuota and LimitRange object, and kubectl describe quota shows no namespace above 80 percent of any cap.
Harden nodes against the CIS Kubernetes Benchmark
P1LRequires: Run a supported version with a deliberate upgrade cadenceFull step — what, why, how, failure modes
WHAT. Run nodes (control plane where you own it, worker nodes always) against the CIS Kubernetes Benchmark and close the high-impact findings: no SSH key-sharing, minimal host packages, kubelet permissions tightened. WHY. A container escape lands on the node; a node with permissive kubelet flags and a shared SSH key turns one compromised pod into root on the substrate. PREREQUISITES. The version policy step — you want a benchmark pass that stays valid across the next upgrade. IMPLEMENTATION. Run a CIS benchmark scanner (kube-bench or a managed equivalent) against one node group, triage failures into fix-now versus accept-with-note, and roll remediations into the node image or launch template so rebuilt nodes stay compliant. Score the first pass, fix, then re-score. TOOLS. kube-bench; your node image build (Packer or cloud-equivalent). COMMON FAILURE MODES. Fixing findings by hand on running nodes — the next scale-out or rebuild silently reverts everything. Fixes belong in the image. EXPECTED IMPACT. The blast radius of a pod compromise stops at the pod; rebuilds are compliant by construction. EVIDENCE. Two benchmark runs (before and after) plus the exception list with reasons.Validation — before you call it done
A dated benchmark report shows zero failed checks in the level-1 profile except documented exceptions, and the node image build reproduces that result on a fresh node.
Phase 2
Workload reliability
Make the workloads themselves survive node loss, deployments, and probe storms.
Set requests and limits on every container
P0MRequires: Give every team a namespace with quotas and limit rangesFull step — what, why, how, failure modes
WHAT. Every container in production carries CPU and memory requests (scheduling floor) and limits (blast ceiling), sized from observed usage rather than guesses. WHY. Requests drive bin-packing and CPU guarantees; limits bound the damage of a leak. Unbounded or unsized containers are why production clusters die at 2 a.m. on a memory creep. PREREQUISITES. LimitRange defaults exist, but real sizing replaces defaults. IMPLEMENTATION. Pull two weeks of per-container usage from metrics, set requests around the p95 and memory limits around 1.5× the p99, and treat QoS class as a deliberate choice (Guaranteed for tier-1 serving paths). Enforce with an admission policy so new manifests cannot ship unsized. TOOLS. Prometheus / metrics-server for usage; Kyverno or Gatekeeper for the admission gate. COMMON FAILURE MODES. Copying the template's placeholder values everywhere — every service with 100m CPU and a 2Gi limit is a latent eviction. EXPECTED IMPACT. The scheduler places work honestly, evictions become rare and explainable, and node capacity maps to reality. EVIDENCE. The admission policy config plus a sample pod spec showing the sized values and their source metric.Validation — before you call it done
A policy-audit query returns zero production pods with missing requests or limits, and each tier-1 deployment reports QoS class Guaranteed or Burstable by documented choice.
Tune liveness, readiness, and startup probes deliberately
P0SRequires: Set requests and limits on every containerFull step — what, why, how, failure modes
WHAT. Three probes per serving container, each testing a distinct question: startup (still initializing), readiness (safe to receive traffic), liveness (worth restarting). WHY. A missing readiness probe means a cold pod takes user traffic; an aggressive liveness probe means a slow-but-healthy pod gets killed in a loop — turning load into an outage. PREREQUISITES. Requests and limits sized, so probe timeouts behave predictably under contention. IMPLEMENTATION. Add a health endpoint that checks real dependencies (readiness: can this pod serve?) and wire startup probes for slow booting services with generous failureThreshold. Liveness checks process health only — not downstream services, or you build a restart cascade. TOOLS. kubectl describe pod events; your service framework's health endpoint. COMMON FAILURE MODES. Pointing liveness at the database — the database blips and every pod across every service restarts simultaneously. EXPECTED IMPACT. Rollouts stop serving from cold pods, and dependency blips stop cascading into restart storms. EVIDENCE. The deployment YAML probe blocks plus the test-day timeline showing the endpoint flip.Validation — before you call it done
Chaos-verifying one tier-1 deployment: killing its dependency drops readiness (pod removed from endpoints within 15 seconds) without triggering a single liveness restart.
Run tier-1 workloads multi-replica with anti-affinity
P0MRequires: Tune liveness, readiness, and startup probes deliberatelyFull step — what, why, how, failure modes
WHAT. Every tier-1 deployment runs at least two replicas spread across nodes (and zones where offered) via topology spread constraints or anti-affinity. WHY. A single-replica deployment makes every node drain, every eviction, and every crash a user-visible incident; anti-affinity is what makes two replicas actually mean two failure domains. PREREQUISITES. Requests sized — the scheduler can only spread pods it can place. IMPLEMENTATION. Set replicas to at least two for tier-1 deployments, add topologySpreadConstraints keyed on hostname (and zone for multi-zone clusters) with whenUnsatisfiable: DoNotSchedule, and confirm autoscaling floors never drop below two. TOOLS. kubectl drain in a rehearsal window; topologySpreadConstraints in the deployment spec. COMMON FAILURE MODES. Two replicas on one node — replicas without topology constraints are redundancy on paper. EXPECTED IMPACT. Node maintenance stops being an incident, and single-pod crashes become non-events. EVIDENCE. Pod placement output and the drain drill's dashboard snapshot.Validation — before you call it done
kubectl get pods -o wide shows the tier-1 replicas on distinct nodes, and draining one node in a rehearsal keeps the service's error rate flat.
Add Pod Disruption Budgets so drains cannot black-hole traffic
P1SRequires: Run tier-1 workloads multi-replica with anti-affinityFull step — what, why, how, failure modes
WHAT. A PDB per tier-1 workload bounding how many pods voluntary disruption (drains, upgrades, autoscaler) may take away at once. WHY. Without a PDB, a node upgrade evicts everything at once and the drain command happily takes the last replica; with one, the upgrade waits for the workload to stay available. PREREQUISITES. Multi-replica with healthy readiness probes — a PDB counts ready pods. IMPLEMENTATION. Set minAvailable so at least one replica above zero stays serving (typical: 2 replicas with minAvailable 1, or maxUnavailable 1), attach it to the same label selector as the deployment, and verify eviction behavior during the drain rehearsal rather than trusting the YAML. TOOLS. kubectl get pdb; the drain rehearsal from the previous step. COMMON FAILURE MODES. minAvailable equal to replicas — the PDB blocks all drains forever and someone deletes it at 1 a.m. EXPECTED IMPACT. Cluster maintenance and upgrades become capacity-aware instead of availability-agnostic. EVIDENCE. The PDB object and the drain timeline from the multi-replica rehearsal.Validation — before you call it done
A drain of a node hosting the workload is throttled while its PDB reports Disruptions allowed: 0, and the service error rate stays flat (no sustained 5xx above 0.1 percent) through the whole drain.
Phase 3
Operational readiness
Scale on signal, and prove you can rebuild — the two capabilities that make the cluster operable long-term.
Back up cluster state and rehearse the restore
P1MRequires: Run a supported version with a deliberate upgrade cadenceResilience EngineeringCloudOps
Full step — what, why, how, failure modes
WHAT. Velero (or platform equivalent) backups of namespaces, config, and persistent volumes on a schedule — plus one timed restore rehearsal per quarter. WHY. Clusters are cattle, but their state (CRDs, secrets refs, PVCs) is not; the first time you need Velero cannot be the first time you run it. PREREQUISITES. Version and node practices stable, so the restored state matches a current cluster shape. IMPLEMENTATION. Install Velero with object-storage backing, schedule full namespace backups daily with 30-day retention, exclude what etcd owns natively, and rehearse: restore one namespace with PVCs into a scratch cluster and time it end to end. TOOLS. Velero; object storage with lifecycle rules; a scratch cluster for drills. COMMON FAILURE MODES. Backing up but never restoring — the backup corpus quietly rots (expired credentials, full buckets) and you find out during the incident. EXPECTED IMPACT. Cluster loss becomes a 60-minute rebuild with a name attached, not an existential event. EVIDENCE. Velero backup schedule status and the dated restore-drill log with the wall-clock time.Validation — before you call it done
The quarterly rehearsal restores a namespace with working volumes into a scratch cluster in under 60 minutes, and the last 7 scheduled backups all show Completed.
Wire HPA and node autoscaling to real signal
P2MRequires: Add Pod Disruption Budgets so drains cannot black-hole trafficFull step — what, why, how, failure modes
WHAT. Horizontal Pod Autoscaler on CPU or a custom metric for tier-1 deployments, plus cluster autoscaling so unschedulable pods create nodes rather than waiting. WHY. Manual capacity is either wasteful or a queue-time outage; HPA plus cluster autoscaler makes the system absorb load changes inside minutes instead of inside a page. PREREQUISITES. Requests set (HPA percentages are relative to requests) and PDBs in place (autoscaler drains respect them). IMPLEMENTATION. Enable HPA on each tier-1 deployment with minReplicas at the multi-replica floor, scale on CPU first (custom metrics later), tune stabilization windows to the service's load shape, and run the cluster autoscaler with a node budget so scaling is bounded. TOOLS. HPA + metrics-server or Prometheus adapter; cluster-autoscaler or the managed equivalent. COMMON FAILURE MODES. Scaling on CPU for a service whose bottleneck is a connection pool — the pods multiply and the pool stays full. EXPECTED IMPACT. Traffic spikes become a cost line on the bill instead of an incident in the channel. EVIDENCE. HPA event log and the load-test dashboard overlay.Validation — before you call it done
A load test doubling traffic scales the deployment out within 5 minutes with no 5xx increase, and scaling back in respects the PDB.
Rehearse upgrades end to end on a schedule
P2LRequires: Run a supported version with a deliberate upgrade cadence, Harden nodes against the CIS Kubernetes Benchmark, Add Pod Disruption Budgets so drains cannot black-hole trafficPlatform EngineeringCloudOpsDevOps
Full step — what, why, how, failure modes
WHAT. Each minor-version upgrade is exercised first on a staging cluster with the production workload set, then applied in production inside a rehearsal window, timed and logged. WHY. An upgrade is the largest routine change a cluster takes; unrehearsed, it surfaces API deprecations, image runtime breaks, and admission-plugin surprises directly on production traffic. PREREQUISITES. Version policy windows on the calendar and PDBs in place so drains sequence correctly. IMPLEMENTATION. Clone production manifests onto staging, run the upgrade there, run the smoke suite, then upgrade one production node pool at a time watching error rates between pools. Keep a written rollback: cordon, drain to the old pool, and roll the control plane back before the final pool commits. TOOLS. Staging cluster; managed upgrade controls (EKS/GKE/AKS) or kubeadm per the platform. COMMON FAILURE MODES. Letting the managed control plane auto-upgrade while node pools lag past the skew policy — version skew failures are subtle and load-dependent. EXPECTED IMPACT. The twice-a-year upgrade becomes routine maintenance with a rehearsal trail instead of a held-breath event. EVIDENCE. Upgrade runbooks with dated completion logs, staging smoke results, and the error-rate overlay for each pool cutover.Validation — before you call it done
2 consecutive production upgrades completed inside their windows with 0 minutes of error-budget burn attributable to the upgrade, each preceded by a staging pass with the smoke suite green.
Sources
- Tier 2 — AuthoritativeKubernetes — Production Environment Setup ↗— Kubernetes(verified 2026-09-19)
- Tier 2 — AuthoritativeKubernetes — Configuring a Pod Disruption Budget ↗— Kubernetes(verified 2026-09-19)
- Tier 2 — AuthoritativeAWS Well-Architected — Reliability Pillar ↗— AWS(verified 2026-09-19)
Start from where YOU are
This roadmap is the general path. An assessment tells you which steps you can skip and which are critical — then generates a version of this plan from your actual findings, tracked to done in your workspace.