Kubernetes Scaling, Rollouts and Recovery

Overview#

Scaling is the operational answer to load, rollouts are the answer to change, and rollback is the answer to a rollout that was wrong. All three act on the same object, the Deployment’s replica count and pod template, and all three are visible through the same commands. This part does each on the api Deployment from earlier parts, then hands the replica count to a HorizontalPodAutoscaler and puts the cluster under enough load to make it move.

The rollout and rollback mechanics are the ones the CI/CD Engineering path built its deployment strategies and rollback parts on; those articles carry the full walkthroughs with output, and this part summarises them from the operator’s side rather than repeating them. The HPA section is new and was run end to end.

Prerequisites

• The cluster and api Deployment (with CPU requests set) from part 1, and the api Service from part 4
• kubectl 1.32 or newer
• Internet access to install metrics-server

Scaling by hand#

kubectl scale edits spec.replicas; the ReplicaSet does the rest:

kubectl scale deployment/api --replicas=5 && kubectl rollout status deployment/api
kubectl get deploy api
kubectl scale deployment/api --replicas=2
kubectl get deploy api

deployment "api" successfully rolled out
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    5/5     5            5           11m
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    2/2     2            2           11m

Scaling up waits for readiness, so rollout status is the right thing to block on. Scaling down terminates pods with the grace period from part 2, and the pods it removes leave their Service’s EndpointSlice before they stop, which is what makes scale-down invisible to clients when the application handles SIGTERM. What manual scaling does not do is stay: the next kubectl apply of a manifest with replicas: 3 puts it back, and so does an HPA (below). Manual scaling is for incidents and experiments; the declared number, or the autoscaler, is the steady state.

Rollouts and rollback, from the operator’s seat#

A rollout is any change to the pod template: a new image, a new environment variable, a changed resource request. The Deployment creates a new ReplicaSet and shifts replicas across according to strategy, and four commands cover everything an operator needs:

kubectl set image deployment/api api=ci-demo:1.4.2      # or kubectl apply with the new template
kubectl rollout status deployment/api --timeout=120s    # blocks; exit 1 on failure
kubectl rollout history deployment/api                  # revisions, with change-cause if you set it
kubectl rollout undo deployment/api                     # back to the previous revision

Three operational facts about them, each demonstrated with output in the CI/CD path:

• maxUnavailable: 0 with a readiness probe means a bad image never takes capacity. The deployment strategies article shows a rollout to a non-existent tag that left all three old pods serving while one new pod sat in ErrImagePull.
• rollout status is the detection. It exits non-zero when the rollout does not complete within --timeout or progressDeadlineSeconds, and that exit code is what a deploy job should fail on.
• rollout undo is a rollout of the previous template, fast because the old ReplicaSet and its image are still there; it appears in rollout history as a new revision. The rollback article walks through it, and through the part undo cannot touch, the database.

Set kubernetes.io/change-cause on every rollout (kubectl annotate deployment/api kubernetes.io/change-cause="…") so rollout history is readable during an incident, and keep revisionHistoryLimit above zero so there is something to undo to.

Autoscaling with a HorizontalPodAutoscaler#

An HPA adjusts spec.replicas between a minimum and a maximum to hold a metric near a target. The metric in the common case is CPU utilisation as a percentage of the pods’ CPU request, which is why part 1 set a request on api and why an HPA on a Deployment without requests does nothing.

metrics-server#

The HPA reads pod metrics from the metrics API, which kind does not ship. Install metrics-server and, on kind only, let it skip kubelet certificate verification:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch -n kube-system deployment metrics-server --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl rollout status -n kube-system deployment/metrics-server
kubectl top nodes
kubectl top pods -l app=api

deployment "metrics-server" successfully rolled out
NAME                        CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
k8s-ops-lab-control-plane   77m          0%       714Mi           8%
k8s-ops-lab-worker          37m          0%       266Mi           3%
k8s-ops-lab-worker2         34m          0%       219Mi           2%
NAME                   CPU(cores)   MEMORY(bytes)
api-6c5dc7ffff-d65d7   2m           21Mi
api-6c5dc7ffff-k8zxw   2m           11Mi
api-6c5dc7ffff-mm4js   2m           10Mi

The release at the time was metrics-server v0.9.0. kubectl top working is the precondition for everything below; if it returns “Metrics API not available”, stop and fix that first.

The autoscaler#

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

kubectl apply -f hpa.yaml; sleep 20
kubectl get hpa api

NAME   REFERENCE        TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
api    Deployment/api   cpu: 3%/50%   2         6         2          20s

Idle, the pods use 3 % of their request and the HPA holds the minimum. The first run of this experiment used the request from part 1 (50m) and a modest load generator, and the utilisation never left single digits: the Node process serves a trivial request in well under a millisecond of CPU, and a handful of sequential clients cannot make two pods busy. To make the autoscaler visible, the request was lowered to 25m (kubectl set resources deployment/api --requests=cpu=25m) and the load generator was made parallel. That is a lab adjustment, not a recommendation; in production the request is whatever the service genuinely needs.

Load#

Eight pods, each running twenty parallel curl loops against the Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loadgen
spec:
  replicas: 8
  selector:
    matchLabels: { app: loadgen }
  template:
    metadata:
      labels: { app: loadgen }
    spec:
      containers:
        - name: curl
          image: curlimages/curl:8.14.1
          command:
            [
              'sh',
              '-c',
              'seq 1 20 | xargs -P 20 -I{} sh -c "while true; do curl -s -o /dev/null http://api/; done"',
            ]

kubectl apply -f loadgen.yaml
for t in 30 60 90 120; do sleep 30
  echo "t+${t}s: $(kubectl get hpa api --no-headers | awk '{print "replicas="$7}') running=$(kubectl get pods -l app=api --no-headers | grep -c Running)"
done

t+30s: replicas=2  running=2
t+60s: replicas=2  running=4
t+90s: replicas=6  running=6
t+120s: replicas=6  running=6

At sixty seconds four pods were already running while the HPA’s summary still said two: the controller had acted and the status column lagged by a reconcile. By ninety seconds it had scaled again.

At the end of the run, with the load still on:

kubectl get hpa api
kubectl top pods -l app=api
kubectl describe hpa api | grep SuccessfulRescale

NAME   REFERENCE        TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
api    Deployment/api   cpu: 346%/50%   2         6         6          27m
NAME                   CPU(cores)   MEMORY(bytes)
api-5ccbf74699-54mwr   81m          12Mi
api-5ccbf74699-5cv6z   78m          14Mi
api-5ccbf74699-clgf9   88m          12Mi
api-5ccbf74699-gb5kr   87m          12Mi
api-5ccbf74699-lcgvq   92m          19Mi
api-5ccbf74699-qhr2w   93m          12Mi
Normal  SuccessfulRescale  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal  SuccessfulRescale  horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target

Read the numbers. Each pod is using 80–93m against a 25m request, which is the 346 % in the TARGETS column; the HPA’s target was 50 %, so it scaled to the maximum in two steps (2 → 4 → 6; the default scale-up policy allows doubling every 15 seconds) and stopped there because maxReplicas said so. The describe events are the authoritative record of what it did and why; the get summary can lag them.

Scale-down#

Remove the load and keep watching:

kubectl delete deployment loadgen
for i in $(seq 1 16); do sleep 30; echo "t+$((i*30))s: $(kubectl get hpa api --no-headers | awk '{print "replicas="$7}')"; done

t+30s: replicas=6
...
t+360s: replicas=6
t+390s: replicas=2
t+420s: replicas=2
Normal  SuccessfulRescale  horizontal-pod-autoscaler  New size: 2; reason: All metrics below target

Six minutes at six replicas with no load, then straight back to the minimum. That is stabilizationWindowSeconds, which defaults to 300 for scale-down: the HPA uses the highest recommendation from the last five minutes, so a brief dip in load does not thrash the replica count. Scale-up has no such window by default, which is the asymmetry you want: fast to add capacity, slow to remove it. Both are tunable under spec.behavior; the defaults are reasonable for a stateless service and wrong for one with slow cold starts, where a longer scale-down window and a smaller scale-up step avoid oscillation.

Basic recovery moves#

Each of these is a kubectl command an operator runs during an incident, and each maps to something earlier in this path:

• Pods crashing after a rollout → kubectl rollout undo (above), then read the logs of the failed ReplicaSet’s pods before they are gone (part 3).
• Not enough capacity → kubectl scale up, or check why the HPA did not (describe hpa: metrics missing, maxReplicas reached, requests unset).
• One bad pod → kubectl delete pod; the controller replaces it. For a StatefulSet pod, wait for the controller unless the node is confirmed dead (part 1).
• Node pressure or maintenance → kubectl cordon stops new scheduling, kubectl drain evicts pods respecting PodDisruptionBudgets. Neither was exercised here; both are the subject of a future article on disruption and affinity.

Security Considerations#

• --kubelet-insecure-tls on metrics-server disables verification of the kubelet’s serving certificate. It is correct for kind, whose kubelets have self-signed certificates, and wrong anywhere else.
• kubectl scale and rollout undo change production with one command. RBAC on deployments/scale and on update of Deployments is what separates on-call from everyone else.

Troubleshooting#

Cleanup#

kubectl delete hpa api
kubectl delete deployment loadgen --ignore-not-found
kubectl scale deployment/api --replicas=3

The whole lab cluster can go when the path is done: kind delete cluster --name k8s-ops-lab.

Conclusion#

scale, rollout status, rollout history and rollout undo are the operator’s four verbs for capacity and change, and an HPA is the same scale verb run by a controller against a metric you chose. The numbers in this part (3 % idle, 346 % under load, 2 → 4 → 6 in ninety seconds, 6 → 2 after the five-minute window) are what a working autoscaler looks like, and the reasons it might not work are all in describe hpa. That closes the path: workloads chosen, health and resources configured, failures debugged, requests traced, capacity managed. The Secure Kubernetes path is the companion for everything these workloads are allowed to do.

References#

• Kubernetes: HorizontalPodAutoscaler
• Kubernetes: HPA walkthrough
• metrics-server
• Kubernetes: scaling a Deployment
• Kubernetes: safely drain a node