Kubernetes
Master Kubernetes: container orchestration, cluster management, deployments, and cloud-native architecture.
What is Kubernetes?
Kubernetes is a control system for running containerized workloads across a cluster of machines. You declare an intended state through API objects: six application replicas, a stable service endpoint, a storage claim, or a rollout policy. Controllers continuously observe the cluster and take actions that move reality toward that declaration.
Kubernetes matters when teams need repeatable scheduling, service discovery, rollout, recovery, policy, and capacity management across many workloads. It does not make an application reliable automatically. Teams still own truthful health signals, resource contracts, data durability, security, upgrades, and the behavior users see during failure.
The invariant to remember
The cluster may replace any individual Pod, but the workload contract must preserve service and data while desired and observed state converge. A restarted container is not proof that a request succeeded, data recovered, or the release is safe.
Start with Docker if images, containers, and process isolation are unfamiliar.
Understand the reconciliation loop first
Kubernetes is declarative. Clients write desired objects through the API; controllers and node agents repeatedly compare those objects with what is actually running.
1 API object
Declare desired state
Submit a versioned manifest with identity, replicas, image digest, requests, probes, placement, configuration, and rollout policy.
2 API server
Persist and validate
Authenticate, authorize, apply admission policy, validate the object, and persist cluster state through the control plane.
3 Scheduler
Schedule work
Choose nodes that satisfy resource requests, affinity, taints, storage, topology, and policy constraints.
4 Kubelet
Run and observe
Pull images, start containers, mount configuration and volumes, execute probes, and report Pod status from each node.
5 Controllers
Reconcile again
Replace missing replicas, progress rollouts, update endpoints, and retry until observed state matches the declaration or progress stops.
Reconciliation is asynchronous. A successful API write means the declaration was accepted; it does not mean every dependent resource is ready.
See how control plane and workers cooperate
One declarative API, many cooperating loops
The API is the coordination boundary. Workloads should never depend on a particular Pod identity unless a stateful contract explicitly provides it.
Front door
API server
Exposes the cluster API, runs authentication and admission, validates objects, and coordinates state changes.
Cluster state
etcd
Stores control-plane state. Backups, encryption, quorum health, and recovery procedures are cluster-owner responsibilities.
Decide and reconcile
Scheduler and controllers
Assign Pods to nodes and drive Deployments, Jobs, Nodes, endpoints, and other resources toward desired state.
Execute on nodes
Kubelet and runtime
Reconcile assigned Pods, start containers, attach volumes, run probes, and report status.
Route traffic
Service data plane
Maps stable service identity to changing ready endpoints through the cluster's network implementation.
Learn the workload building blocks
Execution unit
Pod
One or more tightly coupled containers share a network namespace and volumes. Pods are replaceable; their local filesystem and IP are normally ephemeral.
Stateless rollout
Deployment
Owns ReplicaSets and progresses a versioned set of interchangeable Pods through rolling replacement.
Stable discovery
Service
Selects ready endpoints behind a stable virtual address or name. It does not retry application operations or make them idempotent.
Finite work
Job and CronJob
Run work to completion under restart, retry, parallelism, deadline, and history policies. The job itself must tolerate repeated execution.
Stable identity
StatefulSet
Provides ordered identities and per-replica storage claims, while the application still owns replication, quorum, backup, and recovery semantics.
Node-local agent
DaemonSet
Runs a Pod on matching nodes for networking, storage, security, or telemetry functions and consumes capacity on every selected node.
Plan allocatable capacity and disruption together
Node capacity is not fully schedulable. The operating system, kubelet, networking, storage agents, DaemonSets, image pulls, and failure recovery need headroom. Resource requests tell the scheduler what a Pod reserves; limits control runtime behavior and can cause throttling or termination.
Plan schedulable capacity for failure, not steady state
Change the workload, node pool, Pod requests, replica count, reserve, and disruption budget. Kubernetes can only reconcile within capacity you actually leave available.
Scheduling verdict
The pool preserves room for reconciliation and one-node loss
Keep topology spread, autoscaler response time, image pulls, daemon overhead, and real memory behavior in the test plan.
6.0 cores
19% of allocatable CPU
32.0 cores
20% reserved before scheduling
Fits
Up to 3 replicas per surviving node
11 Pods
1 voluntary disruptions allowed
Replica distribution
2.4 Pods per node on average
High average utilization is not efficient when one node or zone loss makes Pods permanently Pending. Size for the failure and rollout overlap you promise to survive.
Give each health signal one job
Protect a slow-starting process from premature liveness checks. Once startup succeeds, normal liveness and readiness behavior begins.
Answer whether this Pod should receive new traffic now. Include dependencies required to serve safely, but avoid making one optional dependency remove every replica.
Detect a process that cannot recover without restart. Do not restart healthy instances because a shared downstream is unavailable.
Measure what users experience: success, latency, freshness, queue age, or business outcomes. Probe health alone cannot approve a release.
Drill a failed rollout before production
Deployment strategy, traffic control, probes, topology, and rollback signals work as one release contract. A rolling update is simple and efficient, a canary limits exposure, and blue-green creates a clean environment boundary at the cost of duplicate capacity.
Loading rollout lab
Preparing failure scenarios...
Treat placement as a reliability decision
request
Scheduling contract
CPU, memory, accelerators, storage, affinity, and policy determine where a Pod can fit.
maxSkew
Failure spread
Topology constraints limit how unevenly replicas land across nodes and zones.
PDB
Voluntary disruption
A disruption budget constrains drains and evictions; it cannot stop an involuntary machine failure.
surge
Rollout overlap
New and old replicas coexist during rollout, so capacity must include the selected surge policy.
Test the placement policy against real events
- Drain one node while a rollout is active and verify enough ready replicas remain.
- Remove a zone and confirm topology spread, storage attachment, and autoscaling behavior.
- Saturate one resource dimension and inspect why Pods stay Pending.
- Verify priority and preemption do not evict a more important workload unexpectedly.
- Measure image-pull and node-provisioning time inside the recovery objective.
Separate configuration, secrets, and durable data
ConfigMaps distribute non-secret configuration. Secret objects distribute sensitive bytes through the Kubernetes API, but a complete design still needs encryption, least-privilege access, external source-of-truth integration, rotation, audit, and safe delivery into the process.
PersistentVolumeClaims request storage from a storage class. They do not create application replication, point-in-time recovery, cross-zone availability, or tested restore procedures.
Preserve configuration identity through releases
- Use immutable names or checksums so a Pod revision identifies the exact configuration it loaded.
- Avoid mutable image tags; deploy image digests and retain the previous revision.
- Rotate secrets without exposing them in manifests, logs, environment dumps, or debugging output.
- Back up application data and control-plane state independently, then rehearse restores.
Secure the workload and cluster boundaries
Before scheduling
Identity and admission
Use least-privilege RBAC and service accounts. Enforce image provenance, allowed registries, Pod security, required labels, and resource policy at admission.
Inside the Pod
Runtime isolation
Run as a non-root user, drop capabilities, use read-only filesystems where practical, constrain syscalls, and isolate untrusted workloads strongly.
Between workloads
Network policy
Default-deny ingress and egress, then allow identity- and purpose-specific paths. Verify the selected network implementation enforces the policy.
Artifact trust
Supply chain
Pin digests, generate provenance and inventories, scan continuously, sign trusted artifacts, and make exception ownership explicit.
Operate the cluster as a dependency
Watch the control plane and node fleet
- API availability and latency, admission failures, controller queue depth, and etcd health
- Scheduling latency, Pending reasons, allocatable headroom, node readiness, pressure, and evictions
- Deployment progress, unavailable replicas, probe transitions, restarts, and crash loops
- Service endpoint counts, network errors, DNS latency, and policy denials
- Storage attachment, capacity, latency, snapshot age, and restore success
Watch each workload through user outcomes
- Request success, tail latency, queue age, saturation, freshness, and business SLOs
- Candidate-versus-baseline release metrics with automated halt and rollback thresholds
- Resource request accuracy, throttling, out-of-memory termination, and autoscaler lag
- Certificate, secret, image, configuration, and dependency version age
Cluster upgrades, API deprecations, node-image changes, and add-on compatibility need staged rollout and rollback just like application releases.
Review the production checklist
Build these controls
- Measure requests and leave capacity for rollout, node loss, and autoscaler delay.
- Use startup, readiness, and liveness probes for their distinct contracts.
- Spread replicas across failure domains and test node and zone loss.
- Pin images and configuration, define disruption budgets, and automate release rollback.
- Apply least privilege, admission policy, network policy, and supply-chain verification.
- Prove backups, restores, cluster upgrades, and stateful failover under realistic conditions.
Avoid these traps
- Treating a restarted Pod as proof that a user operation recovered.
- Packing steady state so tightly that reconciliation has nowhere to schedule replacements.
- Giving every probe the same dependency check or restarting on shared dependency failure.
- Assuming a PersistentVolumeClaim is a backup or a StatefulSet is database replication.
- Using privileged containers, mutable tags, broad cluster-admin access, or unrestricted egress.