Production Kubernetes is less about getting a cluster to start and more about making the platform predictable under failure. A useful design separates responsibilities: control plane, worker capacity, networking, storage, ingress, identity, policy and observability.
Reference architecture
Clients → DNS / WAF / Load Balancer → Ingress / Gateway → Services → Workloads → Data services
A typical production topology uses three control-plane nodes for quorum, a dedicated worker pool, redundant ingress, a CNI that supports policy enforcement, and an observability stack outside the critical request path.
Control plane
The API server, scheduler, controller manager and etcd form the control plane. For high availability, avoid a single control-plane node and distribute members across independent failure domains where the infrastructure permits it.
kubectl get nodes -o wide
kubectl get --raw='/readyz?verbose'
kubectl -n kube-system get pods -o wide
etcd considerations
Protect etcd as a state database, not just another process. Backups should be automated, encrypted, copied outside the cluster and tested through restore drills.
Networking and policy
The network design should answer four questions clearly:
- How do pods reach each other?
- How does external traffic enter?
- How is east-west traffic restricted?
- How is network behavior observed?
Cilium is a strong option when eBPF-based networking, network policy and deep observability are important. The exact CNI choice should still match the team's operational capability and platform constraints.
kubectl get networkpolicies -A
kubectl get svc -A
kubectl get ingress -A
Observability
Metrics, logs and traces should be designed before incidents happen. At minimum, collect infrastructure saturation, Kubernetes object health, workload health, request behavior and control-plane signals.
| Signal | Typical use |
|---|---|
| Metrics | capacity, saturation, SLOs |
| Logs | event detail and debugging |
| Traces | request-path diagnosis |
| Events | Kubernetes lifecycle context |
Upgrade strategy
Treat upgrades as a platform change with a rehearsed path. Test the target release in a representative environment, validate APIs for removals, upgrade the control plane before workers, and preserve rollback/recovery options.
kubectl version
kubectl api-resources
kubectl get --raw /metrics | head
Production checklist
Before calling a cluster production-ready, confirm that identity, RBAC, network policy, secrets handling, backup, monitoring, alerting, upgrade procedure and disaster recovery are all owned by named operational processes.