Kubernetes Fundamentals

Kubernetes (K8s) automates the deployment, scaling, and management of containerised applications across a cluster of machines.

---

Why Kubernetes?

Running containers in production at scale requires:

Need	What Kubernetes Provides
Keep containers running	Automatic restarts, self-healing
Scale up/down	Horizontal Pod Autoscaler
Zero-downtime deploys	Rolling updates, canary deployments
Service discovery	Built-in DNS, Services
Load balancing	Service load balancing
Secret management	Secrets and ConfigMaps
Storage	Persistent Volumes
Multi-machine scheduling	Scheduler places Pods optimally

---

Kubernetes Cluster Architecture

Kubernetes Architecture, Control Plane & Pod Lifecycle Simulator

CONTROL PLANE NODES (MASTER):

kube-apiserver

etcd Key-Value Store

kube-scheduler

kube-controller-manager

WORKER NODES (COMPUTE):

kubelet Agent

kube-proxy

Container Runtime (containerd)

Subsystem Component Inspection

kube-apiserver

Role: API Gateway & Validation

Central hub. All kubectl calls & internal agents authenticate & validate YAML schemas via apiserver.

---

Control Plane Components

kube-apiserver

The front door to Kubernetes. Every action (CLI, UI, controller) goes through the API server, which validates inputs and persists cluster state to etcd.

• All state changes go through the API server
• Authenticates and authorises every request
• Exposes the Kubernetes REST API

etcd

A distributed key-value store that holds the entire cluster state.

Everything in Kubernetes is stored in etcd:
  /registry/pods/default/my-pod
  /registry/deployments/default/my-deployment
  /registry/services/default/my-service
  /registry/secrets/default/my-secret

• Strongly consistent (Raft consensus)
• Only the API server reads/writes to etcd
• Back this up — losing etcd = losing your cluster

kube-scheduler

Decides which Node a newly created Pod should run on through a two-phase pipeline:

Unscheduled Pod (spec.nodeName is empty)
                     │
                     ▼
       ┌───────────────────────────┐
       │   Phase 1: Filtering      │  Eliminates incompatible nodes (e.g., insufficient CPU/RAM,
       │      (Predicates)         │  taints without tolerations, port conflicts, nodeSelector mismatch)
       └─────────────┬─────────────┘
                     │ (Candidate Nodes)
                     ▼
       ┌───────────────────────────┐
       │   Phase 2: Scoring        │  Ranks feasible nodes from 0 to 100 based on weighted strategies
       │      (Priorities)         │  (e.g., ImageLocality, NodeResourcesBalancedAllocation, Spread)
       └─────────────┬─────────────┘
                     │ (Top Scoring Node)
                     ▼
       ┌───────────────────────────┐
       │   Phase 3: Binding        │  Writes binding back to kube-apiserver: spec.nodeName = "node-02"
       └───────────────────────────┘

• Filtering (Predicates): Checks hard constraints. Nodes lacking requested CPU/RAM, nodes missing required labels (nodeSelector), or nodes with un-tolerated taints are discarded.
• Scoring (Priorities): Ranks surviving candidates. Prefers nodes that already have the container image cached (ImageLocality), nodes that balance CPU and memory usage evenly, or nodes that spread pods across failure domains (Availability Zones).
• Binding: Creates a Binding object via kube-apiserver, populating the Pod's spec.nodeName. The scheduler itself does not touch worker nodes directly.

kube-controller-manager

Runs continuous control loops — constantly reconciles the actual state of the cluster with the desired state stored in etcd.

Controller	What it does	Under-the-Hood Mechanism
ReplicaSet controller	Ensures exact number of Pod replicas exist	Watches Pod creation/deletion; creates or deletes Pods to balance spec.replicas
Deployment controller	Manages declarative rollouts & rollbacks	Creates and orchestrates underlying ReplicaSets during rolling updates
Node lifecycle controller	Monitors worker node health and handles evictions	Watches node heartbeat leases in kube-node-lease; marks nodes NotReady after 40s and initiates Pod evictions after default 300s
EndpointSlice controller	Tracks healthy Pod IPs backing each Service	Populates EndpointSlice objects consumed by kube-proxy
Job / CronJob controller	Manages batch and scheduled workloads	Spawns Pods to completion, tracking retry counts and exit codes

---

Worker Node Components

kubelet

The primary node daemon. Watches the API server for Pods assigned to its node and drives the local container runtime to achieve the Pod spec:

kube-apiserver (Pod scheduled: spec.nodeName = "worker-01")
                      │
           Watch API stream notification
                      ▼
                   kubelet
                      │
         gRPC calls via CRI (Unix Domain Socket)
                      ▼
            Container Runtime (containerd / CRI-O)
            ├── 1. Pull Image (ImageService)
            ├── 2. Create Pod Sandbox & Pause Container (RuntimeService)
            ├── 3. Execute CNI Plugins (Setup veth & IP)
            └── 4. Launch Application Containers via runc
                      │
        Probing & Liveness / Readiness monitoring
                      ▼
kubelet reports PodStatus & Node Lease back to kube-apiserver

• CRI Integration: Talks to container runtimes via standard gRPC over /run/containerd/containerd.sock or /run/crio/crio.sock.
• Pod Sandbox Lifecycle: Always launches a lightweight pause container first to hold the shared network (net) and IPC namespaces before starting application containers.
• Health Probing: Periodically executes liveness, readiness, and startup probes; triggers container restarts or EndpointSlice removal upon failure.

kube-proxy

Maintains node routing rules to direct Service traffic (ClusterIP, NodePort) to target Pod IPs:

Service "orders-svc" (ClusterIP: 10.96.10.50:80)
                      │
       Client Pod initiates TCP connection
                      ▼
        Kernel Routing (programmed by kube-proxy)
        ├── iptables Mode: Sequential rule evaluation with random probability selection
        └── IPVS Mode: Linux IP Virtual Server kernel hash table (O(1) lookup latency)
                      ▼
Packet DNAT'd directly to target Pod IP: 10.244.2.14:8080

Container Runtime (CRI-Compliant)

Executes the actual container processes inside isolated Linux namespaces and cgroups:

Runtime	Architecture & Characteristics
containerd	Industry standard, lightweight daemon spun off from Docker; executes containers via containerd-shim and runc.
CRI-O	Purpose-built by Red Hat specifically for Kubernetes CRI; minimal footprint with zero extraneous tooling.
Docker (dockerd)	Deprecated as direct K8s runtime in v1.20, removed in v1.24. Docker images remain 100% compatible via OCI format.

---

The Kubernetes API

Everything in Kubernetes is a resource — a typed object stored in etcd and managed via the API.

apiVersion: apps/v1        # API group + version
kind: Deployment           # Resource type
metadata:
  name: my-api             # Resource name
  namespace: default       # Logical partition
  labels:
    app: my-api
    version: "1.0"
spec:                      # Desired state
  replicas: 3
  ...
status:                    # Actual state (written by K8s, read-only)
  readyReplicas: 3
  ...

API Groups

core/v1          → Pod, Service, ConfigMap, Secret, PersistentVolume
apps/v1          → Deployment, ReplicaSet, StatefulSet, DaemonSet
batch/v1         → Job, CronJob
networking.k8s.io/v1 → Ingress, NetworkPolicy
rbac.authorization.k8s.io/v1 → Role, RoleBinding, ClusterRole
autoscaling/v2   → HorizontalPodAutoscaler
storage.k8s.io/v1 → StorageClass

---

Namespaces

Virtual clusters within a physical cluster. Scope resources and permissions.

kubectl get namespaces
# NAME              STATUS
# default           Active   ← default if not specified
# kube-system       Active   ← K8s internal components
# kube-public       Active   ← Publicly readable resources
# kube-node-lease   Active   ← Node heartbeat leases

# Create namespace
kubectl create namespace production
kubectl create namespace staging

# Work in a namespace
kubectl get pods -n production
kubectl get all -n production

# Set default namespace for current context
kubectl config set-context --current --namespace=production
kubectl get pods   # Now defaults to production

# Resources that are NOT namespaced (cluster-wide):
# Nodes, PersistentVolumes, ClusterRoles, StorageClass, Namespace itself

---

Declarative vs Imperative

# Imperative — tell K8s what to DO
kubectl run my-pod --image=nginx
kubectl create deployment my-deploy --image=nginx --replicas=3
kubectl expose deployment my-deploy --port=80

# Declarative — tell K8s what STATE you want (preferred!)
kubectl apply -f deployment.yaml    # Create or update
kubectl delete -f deployment.yaml   # Delete what's in the file

Always use declarative YAML files in production. They're version-controlled, reviewable, and idempotent.

---

Reconciliation Loop (The Core Concept)

The fundamental design pattern of Kubernetes is the Level-Triggered Reconciliation Loop:

                  ┌──────────────────────────────┐
                  │    Desired State in etcd     │
                  │ (spec: replicas: 3, img: v2) │
                  └──────────────┬───────────────┘
                                 │
                   Read desired  │  Write actual
                                 ▼
+-------------------------------------------------------------------------+
|                    Kube Controller Reconciliation Loop                  |
|                                                                         |
|   1. OBSERVE   ───► Query API server for current live cluster state     |
|         │                                                               |
|         ▼                                                               |
|   2. ANALYZE   ───► Compute diff = (Desired State - Actual State)       |
|         │                                                               |
|         ▼                                                               |
|   3. ACT       ───► Issue imperative mutations to close the delta       |
+-------------------------------------------------------------------------+

Why Level-Triggered (Not Edge-Triggered)?

• Edge-Triggered (Event-based): Systems that act purely on changes (e.g. "Pod X died") risk permanent state desynchronization if an event is dropped due to a network partition or controller crash.
• Level-Triggered (State-based): Kubernetes controllers observe the current level (what exists right now) regardless of how many intermediate events occurred. If a controller restarts or network reconnects, the very next loop execution inspects observedGeneration vs generation, detects any divergence, and drives the cluster toward the target state.

End-to-End Self-Healing Lifecycle: Node Failure

[Worker Node 2 Dies]
        │
        ├── 0s: Node stops renewing heartbeat in kube-node-lease namespace
        ├── 40s (node-monitor-grace-period):
        │       Node Lifecycle Controller marks Node 2 as "NotReady"
        │       Tolerations kick in (node.kubernetes.io/not-ready:NoExecute)
        ├── 300s (pod-eviction-timeout):
        │       Pods on Node 2 marked as Terminating
        ├── +100ms:
        │       ReplicaSet Controller observes: readyReplicas (2) < spec.replicas (3)
        │       ReplicaSet Controller posts 1 new Pod spec to API server (spec.nodeName is null)
        ├── +200ms:
        │       kube-scheduler detects unbound Pod
        │       Filtering: Eliminates Node 2 (NotReady)
        │       Scoring: Evaluates Node 1 vs Node 3 (CPU/RAM headroom, spread)
        │       Binding: Assigns Pod to Node 3
        └── +2s:
                kubelet on Node 3 invokes containerd CRI to pull image and launch container.
                Cluster returns to desired state (3/3 replicas).

---

Local Kubernetes Options

Tool	Best For	Notes
minikube	Learning, local dev	Single-node VM/container, easy setup
kind	CI testing, multi-node dev	Runs K8s nodes as Docker containers — ultra fast bootstrap
k3s	Lightweight production, edge/IoT	Full certified K8s, replaces etcd with SQLite/etcd, minimal RAM (~512MB)
Docker Desktop	macOS/Windows dev	Single-click enable, embedded single-node cluster
MicroK8s	Ubuntu dev & homelabs	Snap-installed, production-grade single or multi-node

# minikube quickstart
minikube start --driver=docker --cpus=4 --memory=8g --kubernetes-version=v1.30.0
minikube status
minikube dashboard          # Open K8s visual dashboard
minikube tunnel             # Expose LoadBalancer services on localhost

# kind (Kubernetes in Docker)
kind create cluster --name dev --config kind-config.yaml  # Multi-node simulation
kind get clusters
kind load docker-image myapp:v1 --name dev               # Inject local Docker image without registry push
kind delete cluster --name dev

---

🧠 Senior/Staff Interview Playbook: How to Ace Kubernetes Architecture & Control Plane Interviews

1. The Senior Framing Strategy: First 60 Seconds

When asked to explain Kubernetes architecture or design a multi-tenant cluster, a Staff/Principal engineer immediately decouples the declarative control plane from the underlying node data plane:

*"I frame Kubernetes not merely as a container orchestrator, but as a distributed, level-triggered reconciliation state engine.

The architecture breaks down into four physical invariants:

1. Single Source of Truth (etcd): Strongly consistent CP consensus via Raft; all mutations are serialised through kube-apiserver with optimistic concurrency control (resourceVersion).
2. Decoupled Asynchronous Control Loops: Controllers never issue synchronous direct commands to workers; they observe state diffs and write desired specifications. The scheduler only binds names (spec.nodeName), leaving workload enactment to the node agent.
3. Autonomous Node Agents (kubelet): Each node runs an independent daemon driving local container runtimes via gRPC CRI and CNI plugins; nodes continue executing active workloads even during complete control plane loss.
4. Day-2 Lifecycle & Operational Boundaries: Cluster upgrades are one-way due to etcd schema version locks; production topologies treat clusters as disposable cattle rather than hand-reared pets."*

---

2. Classic Interviewer Traps & Principal Rebuttals

Trap 1: "What happens to currently running pods if the entire Kubernetes control plane (all apiserver and etcd nodes) goes down?"

• Junior/Mid Answer: "The cluster goes down and all pods stop running because Kubernetes is dead."
• Principal Rebuttal:

  *"Running container workloads continue executing completely uninterrupted.

  The Kubernetes control plane (kube-apiserver, etcd, kube-scheduler, kube-controller-manager) lives strictly in the management plane. The data plane consists of containerd, runc, and the Linux kernel processes running on worker nodes.

  Worker node packet forwarding (via iptables, IPVS, or eBPF) and active application processes do not depend on real-time API server connectivity.

  However, the cluster enters an un-orchestrated frozen state: auto-scaling stops, crashed pods are not rescheduled, failed nodes are not evicted, and deployments cannot be updated until etcd quorum and API servers recover."*

Trap 2: "Why can't we downgrade a Kubernetes cluster to a previous minor version if an upgrade introduces critical bugs?"

• Junior/Mid Answer: "We can downgrade the kubelet package and reinstall the older API server binary."
• Principal Rebuttal:

  *"Kubernetes does not support automated or in-place cluster downgrades.

  During a minor version upgrade (e.g. v1.23 to v1.24), kube-apiserver permanently migrates etcd storage versions for API groups, rewriting internal serialized protocol buffers to newer schemas and stripping deprecated fields/labels (such as node-role.kubernetes.io/master).

  An older API server binary cannot deserialize data written in a schema version it has no knowledge of. Furthermore, CRDs, conversion webhooks, and kubelet state machines are one-way transitions.

  Resilient enterprises maintain blue-green cluster deployments or automated cluster reconstruction pipelines from GitOps repositories rather than attempting an in-place downgrade."*

Trap 3: "Why is Kubernetes level-triggered rather than edge-triggered (event-driven)?"

• Junior/Mid Answer: "Level-triggered is easier to write with while loops."
• Principal Rebuttal:

  *"In distributed systems, edge-triggered (event-based) systems are inherently brittle: if an edge notification (e.g. PodFailedEvent) is dropped due to network partitions, buffer overflow, or a controller restart, the system permanently loses sync with reality.

  Kubernetes is strictly level-triggered (state-based): controllers operate on the current level (observed state) versus desired level (etcd spec).

  Even if 1,000 intermediate events are lost during a 10-minute network partition, the very next reconciliation pass compares observedGeneration against generation, detects the delta, and issues mutations to converge state. This guarantees self-healing and eventual consistency under arbitrary network and process failures."*

Trap 4: "Why did Kubernetes deprecate and remove Docker in v1.24?"

• Junior/Mid Answer: "Docker was too slow, so Kubernetes replaced it with containerd."
• Principal Rebuttal:

  *"Docker daemon (dockerd) was built as a human-centric developer tool with CLI, volume management, and swarm features, rather than a headless container engine.

  Because Docker did not implement the Kubernetes Container Runtime Interface (CRI) gRPC specification, Kubernetes had to maintain an in-tree adapter called dockershim inside the kubelet codebase. Maintaining dockershim created massive technical debt, extra memory overhead, and hindered CRI feature innovation.

  Removing dockershim allowed kubelet to communicate directly via standard gRPC CRI to lightweight, dedicated runtimes like containerd or CRI-O. OCI container images built via Docker remain 100% compatible."*

---

3. Mid-Level vs. Principal Candidate Cheat Sheet

Architectural Concept	Mid-Level Candidate Says	Principal Candidate Demonstrates
Control Plane Failure	"If etcd crashes, all running containers die."	Distinguishes management plane from data plane; running containers continue operating via local runc/kernel FIB, but reconciliation and scheduling are frozen.
Cluster Scaling	"Add more worker nodes with more CPU and RAM."	Evaluates etcd Raft consensus write serialization throughput, API server watch cache memory pressure, and informers/EndpointSlice overhead.
Cluster Downgrades	"Just roll back the Kubernetes version like an app deployment."	Explains etcd schema storage version lock-in, CRD webhook irreversibility, and designs disposable cattle clusters with GitOps rebuilds.
Node Scheduling	"The scheduler runs pods on the node with the most free memory."	Details the 3-phase pipeline (Filtering predicates $\to$ Scoring priorities $\to$ Binding API write) and topology spread constraints across availability zones.
Reconciliation Logic	"Controllers listen for events and react to them."	Explains level-triggered convergence: observing actual state, diffing against etcd desired state, and driving idempotent mutations.

---

Senior Architect Interview Questions & Deep Answers

1. What are the components of the Kubernetes control plane and what does each do?

• kube-apiserver: Stateless REST gateway and sole component that directly talks to etcd. Handles authentication, authorization (RBAC), admission controllers (Mutating/Validating webhooks), and schema validation.
• etcd: Distributed B-Tree key-value store using the Raft consensus algorithm. Serves as the single source of truth for all cluster declarative state.
• kube-scheduler: Assigns unscheduled Pods to optimal worker nodes via a two-stage pipeline: Filtering (Predicates) and Scoring (Priorities).
• kube-controller-manager: Bundles continuous control loops (Deployment, ReplicaSet, Node Lifecycle, EndpointSlice) that drive actual cluster state toward desired state.
• cloud-controller-manager: Abstracts cloud-provider-specific logic (provisioning AWS NLB/ALBs, EBS storage volumes, or VPC routes).

2. What is etcd and why is it critical?

etcd is a strongly consistent (CP under CAP theorem) distributed key-value store that stores the complete cluster metadata and spec history. If etcd loses quorum (e.g. 2 nodes fail in a 3-node cluster), the control plane becomes read-only and no new pods can be scheduled, updated, or deleted. Worker node data planes (running containers) will continue operating, but cluster orchestration is completely paralyzed.

3. What is the role of the kubelet on a worker node?

The kubelet is the node agent that registers the node with the API server, watches for Pod assignments (spec.nodeName == this_node), and commands the local container runtime via gRPC over the Container Runtime Interface (CRI). It orchestrates the pause container, network namespaces, volume mounts, executes liveness/readiness probes, and continuously reports PodStatus and node heartbeat leases back to the control plane.

4. Explain the Kubernetes reconciliation loop.

It is an infinite level-triggered control loop operating on: Observe $\to$ Analyze $\to$ Act. Rather than reacting only to discrete transient events, the controller continuously observes the live cluster state, computes the difference relative to the desired specification in etcd, and invokes declarative or imperative API mutations to eliminate the delta. This guarantees idempotency and self-healing even after process crashes or dropped network packets.

5. What is kube-proxy and what does it do?

kube-proxy is a network daemon running on each worker node responsible for implementing the Service abstraction (Virtual IPs). It watches the API server for Service and EndpointSlice updates and translates traffic destined for a Service's ClusterIP to one of its healthy backend Pod IPs using either:

• iptables mode: Creates sequential packet filter rules with random weight probabilities (can cause CPU latency bottlenecks at 10,000+ services).
• IPVS mode: Uses Linux kernel Netfilter hash tables with $O(1)$ lookup complexity, supporting sophisticated load-balancing algorithms (Round Robin, Least Connection).

6. What is a Namespace in Kubernetes?

A Namespace is a logical partition within a single physical Kubernetes cluster that provides a scope for object names, resource quotas (ResourceQuota), limit ranges (LimitRange), and Role-Based Access Control (RoleBinding). It does not provide network isolation by default; Pods across different namespaces can communicate freely unless restricted by explicit NetworkPolicy rules.

7. What is the difference between imperative and declarative resource management?

• Imperative (kubectl run, kubectl create): Tells Kubernetes what actions to execute step-by-step. Hard to track in Git, non-idempotent, prone to configuration drift.
• Declarative (kubectl apply -f manifest.yaml): Tells Kubernetes what end state you desire. The API server uses 3-way merge patching (comparing local file, live cluster state, and last-applied configuration annotation), enabling GitOps, peer-reviewed infrastructure code, and idempotent convergence.

8. What happens when a node dies — how does Kubernetes recover?

1. The dead node fails to renew its lease in kube-node-lease.
2. After node-monitor-grace-period (default 40s), the Node Lifecycle Controller transitions the node status to NotReady.
3. After pod-eviction-timeout (default 5 minutes), the controller sets a deletion timestamp on the node's pods.
4. The workload controllers (e.g. ReplicaSet) notice missing healthy replicas and submit replacement Pod objects to the API server without a nodeName.
5. kube-scheduler schedules these replacement Pods to surviving healthy nodes with adequate resource headroom.
6. Once scheduled, kubelet on the destination node pulls images, mounts volumes, and starts the pods, restoring required capacity.

9. What is the kube-scheduler responsible for?

The kube-scheduler assigns unscheduled Pods to the most appropriate worker node without executing the workload itself. It does this via:

1. Filtering: Discards nodes that do not satisfy hard constraints (insufficient allocatable CPU/RAM, unscheduled taints, node affinity rules, port conflicts).
2. Scoring: Assigns a score (0–100) to each surviving node using weighted priority functions (NodeResourcesBalancedAllocation, ImageLocalityPriority, Pod topology spread).
3. Binding: Sends a Binding API request to the API server setting spec.nodeName to the winning node.

10. Why was Docker deprecated as a Kubernetes container runtime?

Docker daemon (dockerd) was built as an end-user developer tool, not a raw container runtime. It communicated via REST API rather than the Kubernetes gRPC Container Runtime Interface (CRI), requiring the Kubernetes project to maintain a heavy in-tree translation layer called dockershim inside kubelet. In Kubernetes 1.24, dockershim was removed to eliminate redundant translation hops, reduce cluster node memory/CPU bloat, and allow kubelet to interface directly with dedicated CRI runtimes (containerd or CRI-O). Images built using Docker adhere to the OCI specification and run unmodified on modern Kubernetes clusters.