Production Kubernetes Platform

ARCHITECTURE A managed control plane (EKS, GKE or AKS) with separate node groups for system workloads and application workloads, so a noisy application cannot starve the ingress controller or the metrics pipeline. Ingress terminates at a cloud load balancer in front of an in-cluster controller. Workloads are namespaced per team with resource quotas; platform components live in their own namespaces. Delivery is pull-based: a Git repository holds the desired state and a reconciler applies it (see /gitops). Secrets come from a managed secret store through a CSI driver or external-secrets, never from committed manifests. Separate clusters per environment. Namespace-only separation between production and non-production means a cluster-wide failure — an upgrade, a CRD change, a misapplied policy — takes both at once. ASSUMPTIONS This shape assumes you have more than one team deploying independently, that someone is accountable for the platform as a product, and that you can absorb a Kubernetes control-plane upgrade every few months. It assumes container images are already built and signed elsewhere (see /buildops). If a single team runs a handful of services, this is the wrong shape and a managed container runtime will cost far less to operate. Kubernetes repays a coordination cost you may not have. SECURITY RBAC scoped per namespace with no default cluster-admin bindings. Pod security admission at "restricted" for application namespaces, with exceptions recorded rather than granted globally. Network policy default-deny between namespaces — a cluster without network policy is a flat network with extra steps. Workload identity federated to the cloud IAM so pods get short-lived credentials instead of stored keys. Image admission that verifies provenance, and a registry you control. OBSERVABILITY Metrics via the Prometheus ecosystem with a remote write target so retention is not the cluster's problem. Logs shipped off-cluster — a cluster that loses its own logs during an incident is the expected failure mode, not an edge case. Distributed tracing through OpenTelemetry with consistent service and owner metadata, which is also what makes correlation possible later (see /aiops). Alert on symptoms users feel, not on pod restarts. A CrashLoopBackOff that never affected a request is a ticket, not a page. AVAILABILITY Multiple replicas across availability zones with pod anti-affinity and PodDisruptionBudgets, so a voluntary disruption cannot take the last replica. Cluster autoscaling with headroom for a zone loss rather than exact capacity. Control-plane availability is the provider's, and your recovery objective must account for the case where it is degraded and you cannot schedule. Test the failover you designed. Node drains and zone loss are the two worth rehearsing (see /chaos-engineering). COST CONSIDERATIONS The control plane is a flat fee and rarely the issue. The cost is idle node capacity, over-requested resources and cross-zone data transfer. Requests set far above real usage are the single most common source of waste, because the scheduler reserves what you asked for. Right-size from observed usage, use spot capacity for tolerant workloads with a clear fallback, and attribute cost per namespace so teams see their own footprint (see /finops). IAC GUIDANCE Cluster and node groups in Terraform or OpenTofu; in-cluster resources through the reconciler, not through the same Terraform run. Mixing them produces a state file that fights the controller. Keep cluster add-ons versioned and upgrade them deliberately — add-on drift is what makes a control-plane upgrade frightening. PRODUCTION READINESS Before real traffic: requests and limits set, liveness and readiness probes that test different things, PodDisruptionBudgets present, network policy default-deny, secrets from a manager, images pinned by digest, a tested rollback, and an owner who is paged. The Kubernetes production checklist at /checklists covers this item by item, and /assess will tell you whether the practices behind it exist.